Although visual object tracking algorithms are capable of handling various challenging scenarios individually, none of them are robust enough to handle all the challenges simultaneously. For any online tracking by detection method, the key issue lies in detecting the target over the whole frame and updating systematically a target model based on the last detected appearance to avoid the drift phenomenon. This paper aims at proposing a novel robust tracking algorithm by fusing the frame level detection strategy of tracking, learning, detection with the systematic model update strategy of Kernelized Correlation Filter tracker. The risk of drift is mitigated by the fact that the model updates are primarily driven by the detections that occur in the spatial neighborhood of the latest detections. The motivation behind the selection of trackers is their complementary nature in handling tracking challenges. The proposed algorithm efficiently combines the two state-of-the-art tracking algorithms based on conservative correspondence measure with strategic model updates, which takes advantages of both and outperforms them on their short ends by virtue of other. Extensive evaluation of the proposed method based on different metrics is carried out on the data sets ALOV300++, Visual Tracker Benchmark, and Visual Object Tracking. We demonstrated its performance in terms of robustness and success rate by comparing with state-of-the-art trackers. © 1992-2012 IEEE.