In law enforcement applications such as surveillance and forensics, video is often presented as evidence. It is therefore of paramount importance to establish the authenticity and reliability of the video data. This paper presents an intelligent video authentication algorithm which integrates learning based Support Vector Machine classification with Singular Value Decomposition watermarking. During video capture and storage, intrinsic local correlation information is extracted from the frames and embedded in the frames at local levels. Tamper detection and classification is performed using the inherent video information and embedded correlation information. The proposed algorithm is independent of the choice of watermark and does not require any key to store. Further, it is robust to global tampering such as frame addition and removal, local attacks such as object alteration and can differentiate between acceptable operations and malicious tampering. Experiments are performed on an extensive database which contains non-tampered videos and videos with several types of tampering. The results show that the proposed algorithm outperforms existing video authentication algorithms. © 2008 Springer Science+Business Media, LLC.