This paper presents a robust shape-based on-line tracker for simultaneously tracking the motion of both hands, that is robust to cases of background clutter, other moving objects, occlusions of one hand by the other, and a wide range of illumination variations. The tracker is based on an on-line predictive Eigen Tracking framework. This framework allows efficient tracking of articulate objects, which change in appearance across views. We show results of successful tracking across all possible cases of motion dynamics of both hands during occlusion, and a wide range of illumination conditions. © 2004 IEEE.