This chapter focuses on recognizing faces with variations in aging and disguise. In the proposed approach, mutual information based age transformation algorithm registers the gallery and probe face images and minimizes the variations in facial features caused due to aging. Further, gallery and probe face images are decomposed at different levels of granularity to extract non-disjoint spatial features. At the first level, face granules are generated by applying Gaussian and Laplacian operators to extract features at different resolutions and image properties. The second level of granularity divides the face image into vertical and horizontal regions of different sizes to specifically handle variations in pose and disguise. At the third level of granularity, the face image is partitioned into small grid structures to extract local features. A neural network architecture based 2D log polar Gabor transform is used to extract binary phase information from each of the face granules. Finally, likelihood ratio test statistics based support vector machine classification approach is used to classify the granular information. The proposed algorithm is evaluated on multiple databases comprising of disguised faces of real people, disguised synthetic face images, faces with aging variations, and disguised faces of actors and actresses from movie clips that also have aging variations. These databases cover a comprehensive set of aging and disguise scenarios. The performance of the proposed algorithm is compared with existing algorithms and the results show that the performance of the proposed algorithm is significantly better than existing algorithms. © 2014 Springer-Verlag Berlin Heidelberg.