To enable seamless integration of video information on the semantic web, we require that the knowledge of a video domain be formally specified in an ontology. We present a novel approach for defining video domain concepts in an ontology using properties that can be observed from the media. We use the ontology specified knowledge for recognizing concepts relevant to a video scene by making observations for the media properties of concepts as well as making inferences from other ontological concept definitions and relations. For this purpose we introduce new language constructs to OWL (Web Ontology Language), which are used to specify the inherently uncertain nature of media observations. The new constructs also allow additional semantics concerned with the association of media properties with concepts. We propose the use of Bayesian network as the reasoning mechanism for doing inferencing tasks in the presence of uncertainty. The video is annotated with the relevant concepts defined in the ontology. These conceptual annotations are used to create hyperlinks in the video collection. © 2006 IEEE.