Biometric recognition of newborns, toddlers, and pre-school children aims is an important research challenge with applications in identifying newborn swapping, missing kids, and disbursing benefits. In this research, we propose a representation learning algorithm to extract unique and invariant features from face images of newborns and toddlers, to design an efficient face recognition algorithm. Specifically, we propose a deep learning model which applies class-based penalties while learning the filters of a convolutional neural network. The proposed CNN architecture achieves a rank-1 identification accuracy of 62.7% for single gallery newborn face recognition and 85.1% for single gallery toddler face recognition, forming state-of-the-results for both the databases. Comparison with several existing algorithms also showcases the effectiveness of the proposed algorithm on both the databases. © 2018 IEEE.