Investigation of critical speeds and nonlinear free vibration analysis of a highly flexible rotor system have been studied. The rotary inertia and gyroscopic effect combined with inextensible geometric condition for pinned-guided shaft element have been taken into account to develop the governing equation of motion. The closed-form mathematical expressions have been derived for determining both linear and nonlinear natural frequencies and their behavioral patterns have been simultaneously demonstrated through time histories, FFTs and Poincare's maps upon changing the control parameters. The nonlinear natural frequencies have been found to be higher by about 4–6% as compared to the findings obtained via linear analysis. For lower spin speed of the shaft, both linear and nonlinear forward natural frequencies have been found to be dominant. The influence of rotary inertia on natural frequencies by considering Euler and Rayleigh beam elements, separately has been reported. The system response has been observed to be either periodic or quasi-periodic depending on the spin speed. Outcomes which were not explored earlier enable significant theoretical understanding of free vibration analysis and whirling speeds of rotating system which are of great practical importance for investigating further dynamic performance. © 2017