Autonomous driving is an active area of research. An important aspect of this problem is recognizing the gestures made by humans, both inside and outside the vehicle. In this paper, we present the Auto-G database that comprises different hand gestures for autonomous driving. The database encompasses several challenges such as occlusion, low resolution, motion blur, illumination variation, extreme pose variations, along with the presence of multiple gestures within a frame. We also propose an end-to-end pipeline for hand detection and gesture recognition. The proposed pipeline achieves a frame gesture recognition accuracy of 90.23% on the proposed Auto-G database. © 2019 IEEE.