We present a novel content-based re-ranking scheme for enhancing the precision of video retrieval on the web. We use ontology specified knowledge of the video domain to map user queries to domain-based concepts. The user preferences are learned implicitly from the web logs of users' interaction with a video search engine. A ranking SVM is trained for each concept to learn the ranking function which incorporates user preferences for the concept. The videos are represented by a set of ingeniously derived contentbased features which are based on MPEG-7 descriptors. Our re-ranking scheme thus effectively re-ranks results for new text queries submitted to our video retrieval system, leading to better satisfaction of the users' information need. © 2007 IEEE.