Last decade has witnessed rapid growth for the popularity of Convolutional Neural Networks (CNNs), in detecting and classifying objects. The self trainable nature of CNNs makes them the strongest candidate as a classifier and a feature extractor. However, many of the existing CNN architectures fail recognizing texts or objects under input rotation and scaling. This paper introduces an elegant approach, 'Scale and Rotation Corrected CNN (SRC-CNN)' for scale and rotation invariant text recognition, exploiting the concept of principal component of characters. Prior to training and testing with baseline CNN, 'SRC-CNN' maps each character image to a reference orientation and scale, which is again derived from the character image itself. SRC-CNN is capable of recognizing characters in a document, even though they differ in orientation and scale greatly. The proposed method does not demand any training with samples which are scaled or rotated. The performance of proposed approach is validated on different character data sets like MNIST, MNIST_rot_12k and English alphabets and compared with state of the art rotation invariant classification networks. SRC-CNN is a generalized approach and can be extended for rotation and scale invariant classification of many other datasets as well, choosing any appropriate baseline CNN. Here we have demonstrated the generality of the proposed SRC-CNN on MNIST Fashion data set and found to perform well in rotation and scale invariant classification of objects as well. This paper demonstrates how the basic PCA based rotation and scale invariant image recognition can be integrated to CNN for achieving better rotational and scale invariances in classification. © 2018 ACM.