Drought stress detection involves multi-modal image analysis with high spatio-temporal resolution. Identification of digital traits that characterizes drought stress response (DSR) is challenging due to high volume of image based features. Also, the labelled data that categorizes DSR are either unavailable or subjectively developed, which is a low-throughput and error-prone task. Therefore, we propose a novel framework that provides an automated scoring of DSR based on multi-trait fusion. k-means clustering was used to extract latent drought clusters and the relevant traits were identified using Support Vector Machine-Recursive Feature Extraction (SVM-RFE). Using these traits, SVM based DSR classification model was constructed. The framework has been validated on visible and thermal shoot images of rice plants, yielding 95% accuracy. Various imaging modalities can be integrated with the proposed framework, thus making it scalable as no prior information about the DSR was assumed. © Springer Nature Singapore Pte Ltd. 2018.