A mathematical model incorporating higher order deformation in bending is developed to investigate the nonlinear behavior of rotor. Transverse harmonic base excitation is imparted to rotor system and Euler-Bernoulli beam theorem is applied with effects such as rotary inertia, gyroscopic effect, higher order large deformations, rotor mass unbalance and dynamic axial force. Discretization of the kinetic and strain (deformation) energies of the rotor system is done using the Rayleigh-Ritz method. Second order coupled nonlinear differential equations of motion are obtained using Hamilton's principle. Nonlinear dynamic response of the rotor system is obtained by solving above equations using the method of multiple scales. This response is examined for resonant condition. It is concluded that nonlinearity due to higher order deformations and variations in the values of different parameters like mass unbalance and shaft diameter significantly affects the dynamic behavior of the rotor system. It is also observed that the external harmonic excitation greatly affects the dynamic response. © 2014 published by EDP Sciences.