Though visual object tracking algorithms are capable of handling various challenging scenarios individually, still none of them is robust enough to handle all the challenges simultaneously. This paper aims at proposing a novel robust tracking algorithm by elegantly fusing the frame level detection strategy of Tracking Learning & Detection (TLD) with systematic model update strategy of Kernelized Correlation Filter tracker (KCF). The motivation behind the selection of trackers is their complementary nature in handling tracking challenges. The proposed algorithm efficiently combines the two tracking algorithms based on conservative correspondence measure with strategic model updates, which takes advantages of both and outperforms them on their short-ends by the virtue of other. The proposed fusion approach is quite general and any complimentary tracker (not just KCF) can be fused with TLD to leverage the best performance. Extensive evaluation of the proposed method based on different metrics is carried out on the datasets ALOV300++, Online Tracking Benchmark (OTB) and Visual Object Tracking (VOT2015) and demonstrated its superiority in terms of robustness and success rate by comparing with state-of-the-art trackers. © 2016 ACM.