Automatic recognition of digital modulation schemes is becoming an active research area in many covert operations. It has many military applications where surveillance and electronic warfare requires a type of modulation in intercepted signal to prepare jamming signals. Most of the approaches are based on modulated signal's component, but the modulation type can be best identified with the use of constellation diagram. The proposed technique is able to recognize M-QAM, M-ASK, and M-PSK modulation scheme in Additive White Gaussian Noise (AWGN) environment. As the constellation points form clusters in the I-Q plane, the order of the modulation can be obtained by estimating the correct number of clusters, which is calculated by OPTICS algorithm. The least square error has been calculated using linear regression from the obtained constellation points, to identify either ASK or PSK and QAM. The error is least for ASK which differentiates ASK from PSK and QAM. To identify between the PSK and QAM, k-means clustering is employed to find the number of centroids equal to order of modulation estimated by OPTICS. With the difference in maximum and minimum absolute value of the centroids, PSK or QAM is recognized. The proposed method shows an improvement in the classification accuracy which reaches 100% using 1024 symbols at 20 dB compared to 98.89%, 98.05%, and 98% when using more complex classifiers like Support Vector Machine, Naive Bayes Classifier, KNN respectively. The method used is unsupervised whereas most of the methods in the literature require training phase to set the thresholds or weights for final model to detect modulation type. This algorithm is also implemented in LabVIEW, and tested on real-time signals. An intelligent system is made which does not require any knowledge of symbol rate, carrier frequency, and any training phase to set thresholds, and detects the type of modulation blindly in real time. Modulated RF signals are generated by NI PXIe-5673 (RF transmitter). NI PXI 5600 is used to downconvert RF signal and NI PXI-5142 (100 MS/s OSP digitizer) is used to sample the downverted signal. © 2017 Elsevier Ltd