The present article treats the nonlinear dynamic analysis of a lightweight flexible rotor–disk–bearing system with geometric eccentricity and mass unbalance. A large deflection model has been derived to represent a nonlinear flexible rotor–bearing system to study the bifurcation, stability, and route to chaos. This mathematical model includes a bidirectional flexible shaft characterized by nonlinear curvature and gyroscopic effect, geometric eccentricity, a rigid disk crooked with unbalance mass, and nonlinear flexible bearings. A perturbation technique has been used to obtain a set of nonlinear algebraic equations that govern the overall dynamics of the system. The system stability has been studied by investigating the bifurcation and route to chaos upon changing the design parameters such as geometric eccentricity, mass unbalance, and disk parameters under the resonance conditions. The present system exhibits a complex behavior traveling with periodic, quasi-periodic, period doubling and chaotic behavior on a gradual change in design variables. The system loses its stability due to S–N bifurcation, which leads to a sudden jump in the response amplitude. These complex behaviors have been studied in detail with the illustration of time history, phase trajectories, bifurcation diagrams, and Poincaré’s map for each category. Qualitative assessment of bifurcation diagrams has been studied to explore the boundaries of the stable and unstable behaviors and essential dynamics of the systems. Special attention to predict its rich dynamics to highlight the route to chaos as a future diagnostic tool has been explored. The presented results offer significant understanding of the dynamic performances and its critical operating conditions of a rotor system subjected to geometric eccentricity and mass imbalance. © 2019, Springer-Verlag GmbH Austria, part of Springer Nature.