Biometric recognition of newborn babies is an opportunity for the realization of several useful applications, such as improved security against swapping and abduction, accurate census, and effective drug delivery. This paper explores the possibility of using face recognition toward an affordable and friendly biometric modality for newborns. The paper proposes an autoencoder-based feature representation followed by problem specific distance metric learning via one-shot similarity with one class-online support vector machine. The largest publicly available database of newborns collected from various sources to study face recognition is introduced. Several existing face recognition approaches and commercial systems are also evaluated on a common benchmark protocol. The efficacy of the proposed algorithm is evaluated under both verification and identification settings. With multiple galleries, rank-1 identification accuracy of 78.5% and verification accuracy of 63.4% at 0.1% false accept rate are achieved. © 2016 IEEE.