This paper addresses the problem of local change detection in an underwater environment. A spatio-contextual framework based frame difference algorithm has been proposed, which uses five consecutive video frames to extract the object. This is a two-stage method. In the first stage, frame differencing is carried out using the spatio-contextual framework, whereas in the second stage a knowledge-based analysis is proposed to detect the moving objects region. The results obtained using the proposed method is compared with the existing background subtraction methods and found to be a far better approach. The performance of the proposed scheme is evaluated by the help of three evaluation measures, such as Precision, Recall, and F-measure. © 2017 IEEE.