This paper addresses the problem of segmenting handwritten annotations on scientific research papers. The motivation of this work is to geometrically segment the complex cases of handwritten annotations, including marks, cuts and special symbols along, with the regular text. Our work particularly focuses on documents that have multi-oriented handwritten [1] annotations rather than annotations in controlled scenario [2]. Spectral Partitioning is adopted as the segmentation scheme to separate the printed text and annotations. A new feature Envelope Straightness is developed and included in our feature set. This leads to an improvement of accuracy over the state-of-the-art features. The experiments are performed on two datasets: 40 documents authored by two writers from IAM dataset, comprising only printed and handwritten text and a self created dataset of 40 scientific papers from various proceedings annotated by a reader, comprising varied types of annotations. In the framework of spectral partitioning, our feature set has achieved a recall of 98.39% for printed text and precision of 85.40% for handwritten annotations on our dataset. For IAM dataset our feature set has achieved a recall of 81.89% for printed text and a precision of 69.67% for handwritten annotations. The results achieved on both dataset are better compared with results obtained using [3] [1]. © 2015 IEEE.