This paper deals with several new methodologies and concepts in the area of rough set theoretic granular computing which are then applied in video tracking. A new concept of neighborhood granule formation over images is introduced here. These granules are of arbitrary shapes and sizes unlike other existing granulation techniques and hence more natural. The concept of rough-rule base is used for video tracking to deal with the uncertainties and incompleteness as well as to gain in computation time. A new neighborhood granular rough rule base is formulated which proves to be effective in reducing the indiscernibility of the rule-base. This new rule-base provides more accurate results in the task of tracking. Two indices to evaluate the performance of tracking are defined. These indices do not need ground truth information or any estimation technique like the other existing ones. All these features are demonstrated with suitable experimental results. © 2015, Springer Science+Business Media Dordrecht.