This paper describes a step forward in calculating the nonlinear frequencies and resultant dynamic behavior of high speed rotor bearing system with mass unbalance. Nonlinear strongly coupled equations of motion has been formulated based on strain energy and kinetic energy equations for shaft, disk and unbalance mass with shaft undergoing large bending deformations. Here gyroscopic effects of disk as well as mass unbalance are also considered while vibration effect along the shaft axis is ignored. Time history and FFT analysis for finding the fundamental frequencies for the rotating are portrayed under variation of shaft diameter, frequency of the shaft speed, geometric nonlinearity and disk location. The present research shows an interesting development that the initial conditions are playing an important role in finding the nonlinear frequencies and this variation is strongly due to the presence of nonlinear geometric coupling. In addition, response analysis of the system has been developed due to mass unbalanced using time history. This paper enables an understanding and realization of operating zones of rotational speed of the shaft by which the excessive vibration can easily be avoided due to the resonant conditions occurred as the natural frequencies come closer to the frequency of the rotational speed. © 2016 The Authors.