This paper describes a novel appearance based scheme for extraction and representation of video objects. The tracking algorithm used for video object extraction is based upon a new eigen-space update scheme. We propose a scheme for organisation of video objects in an appearance based hierarchy. Appearance based hierarchy is constructed using a new SVD based eigen-space merging algorithm. The hierarchy enables approximate query resolution. Experiments performed on a large number of video sequences have yielded promising results. © 2002 IEEE.