Based on the Reynolds averaged Navier-Stokes (RANS) equations and the time-averaged continuity equation, a theory of turbulent shear flow over an undulating sand bed is developed addressing the instability criterion of plane sand beds in free-surface flows leading to the formation of sand waves. In the analysis, the integration of RANS equations leads to generalized Saint Venant equations, in which the time-averaged streamwise velocity is characterized by a power law obtained from turbulence closure, treating the curvilinear streamlines by the Boussinesq approximation. As a consequence, the modified pressure distribution has a departure from the traditionally linear hydrostatic pressure distribution. The instability analysis of a plane sand bed yields the curves of the Froude number versus nondimensional wave number, determining an instability zone for which at lower Froude numbers (less than 0.8), the plane bed becomes unstable with the formation of dunes; whereas at higher Froude numbers, the plane bed becomes unstable with the formation of standing waves and antidunes. For higher Froude numbers, the experimental data for antidunes lie within the unstable zone; while for lower Froude numbers, the same is found for dunes with some experimental scatter. © 2009 The American Physical Society.