This paper presents a genre-specific modeling strategy capable of improving the task of content based video classification and the speed of data retrieval operations. With the ever increasing growth of video data it is important to classify video shots into groups based on its content. For that reason, it is of primary concern to design systems that could automatically classify videos into different genres based on its content. We consider the genre recognition task as a classification problem. We use support vector machines to perform the classification task and propose an improved video classification method. The experimental results show that genre-specific modeling of features can significantly improve the performance. Results have been compared with two contemporary works on video classification, to demonstrate the superiority of our proposed framework. © 2013, Springer-Verlag London.