Roads form a critical part of any region's infrastructure. Their constant monitoring and maintenance is thus essential. Traditional monitoring mechanisms are heavy-weight, and hence have insufficient coverage. In this paper, we explore the use of crowd-sourced intelligent measurements from commuters' smart-phone sensors. Specifically, we propose a deep-learning based approach to road surface quality monitoring, using accelerometer and GPS sensor readings. Through extensive data collection of over 36 hours on different kinds of roads, and subsequent evaluation based on this, we show that the approach can achieve high accuracy (98.5%) in a three-way classification of road surface quality. We also show how the classification can be extended to a finer grained 11-point scale of road quality. The model is also efficient: it can be implemented on today's smart-phones, thus making it practical. Our approach, called RoadCare, enables several useful smart-city applications such as spatio-temporal monitoring of the city's roads, early warning of bad road conditions, as well as choosing the "smoothest" road route to a destination. © 2020 ACM.