The paper presents a novel script independent CRF based inferencing framework for character recognition. In this framework we consider a word as a sequence of connected components. The connected components are obtained using different binarization schemes and different possible sequences are considered using a tree structure. CRF uses contextual information to learn perfect primitive sequences and finds the most probable labeling of the sequence of primitives using multiple hypothesis tree to form the correct sequence of alphabets. This approach is particularly suitable for degraded printed document images as it considers multiple alternate hypotheses for correct decision. © 2013 IEEE.