In this paper, we have proposed a fuzzy rule based system for classification of video into semantic categories. The classification scheme uses an evolutionary learning methodology to evolve a fuzzy system for use in the classification process. This evolved fuzzy class tier has the inherent capability to tackle variations and ambiguities invariably present in the video data. A novel fuzzy theoretic sheme has been suggested for extraction of key frames from a given video after shot segmentation. Frame based temporal features and spatial features obtained from key frames have been used in the classification system. We have developed an experimental system for categorization of sports video. The experimental system has yielded reasonably correct recognition results for a large number of samples.