The present research studies the vibration and bifurcation analysis of a spinning rotor-disk-bearing system to carefully scrutinize the dynamic stability under extrinsic mass unbalance and pulsating axial load. The shaft is flexible and taken into account the geometrical nonlinearities due to large elastic deformation in bending. The rotor is supported by flexible bearings, which are modeled as an equivalent spring-damper system having linear and nonlinear stiffness elements. Equation of motion of the rotating system, which includes flexible shaft, rigid disk, and flexible bearing, is derived using extended Hamilton principle with the assumption of the Euler's beam theory. We studied initially the modal analysis to determine the modal parameters, i.e. natural frequency and mode shapes prior to the investigation of the dynamics of the system. Further, we developed the bifurcation diagram for steady-state solutions and to study the subsequent dynamic stability and verify with the findings solved numerically. The interactive behavior among the nonlinear shaft-bearing, axial load and an unbalance has been analyzed. Numerical simulation tools, i.e. frequency–response characteristics, time history, phase trajectories, and Poincaré map have been used to highlight the presence of nonlinear phenomena and its important role towards evaluating the system dynamics and subsequent stability. This current research showed that flexible bearings stabilize the system as a result of increasing the restoring force. Analyzing the bifurcation diagrams of pulsating axial load, we found that the system exhibits complex phenomena as multiple but stable periodic orbits leading to period doubling. The present system is highly vulnerable to catastrophic failure due to the S–N and Pitchfork bifurcation. The present research enables the notability of axial load and mechanical unbalance on the overall system behavior and stability in real working conditions. © IMechE 2020.