Smart video surveillance of indoor and outdoor scenes is a challenging task for modern surveillance systems. Different imaging conditions like bad illumination, adverse weather, etc., makes the surveillance process difficult. Recently, researchers have proposed smart surveillance systems with additional features for more accurate monitoring of events, but not much attention is paid to improve the system such that the monitoring process consumes as minimum resources as possible. In this paper, we propose a novel surveillance system that enhances visibility in adverse weather conditions and summarizes the captured videos automatically to reduce storage space. As the summarization process is based on the events in a scene, video interpretation becomes fast and easy. We propose perceptual features that can be used for more meaningful and robust summarization of the video than the existing summarization algorithms. We test the system for both indoor and outdoor scenes and show that the system works well even with multiple moving objects and complex motions. © 2017 IEEE.