This research article investigated the rotor-casing contact phenomena of a high-speed rotor-bearing system under mass unbalance situation. Here, large deflection rotor-bearing model with a flexible shaft, a rigid disk loaded with an unbalance mass and contact phenomenon between a disk and casing of the rotating system has been used. A nonlinear mathematical model of a rotating system has been developed to study the steady-state stability by elucidating the state of bifurcation and route to chaos phenomena. The steady-state bifurcations and plausible route to chaos phenomena have been explored by numerically solving the governing motion upon gradually varying the design parameters i.e., flexible co-efficient, friction, shaft speed and unbalance mass. The rub-impact onto this large deflection model along with mass unbalance advocates the changes in dynamic state and ill-fated instability where the system losses its structural stability due to the presence of critical bifurcation and sensitivity toward chaotic responses. The present system enables a complex dynamic evolving with periodic, multi-period periodic solutions, sequences of period-doubling to chaotic motion on a gradual change in design variables. The results obtained will contribute to predict and identify the parametric conditions at which the system is safe from any contact between casing and rotor–disk. © 2021 Elsevier Ltd