Shoulder injuries are very common in sports and certain labour intensive occupations. While some injuries are minor and full recovery is within 1-2 weeks, some major injuries requires the person to consult a physiotherapist and follow an exercise plan for months for full recovery. With the advent of consumer accessible motion capture (MoCap) technologies, the task of conducting daily rehabilitation routine and evaluation which was previously done by a trained physiotherapist can be now done with computers which can be set up in home. In this paper, we propose a system for medical rehabilitation of patients suffering from shoulder injuries. We use Hidden Markov Models (HMM) for recognition and a histogram-based comparison for computing the accuracy score. The Microsoft Kinect sensor is used to obtain 3D coordinates of human joints. Important features are extracted from the skeletal coordinates which are then quantized into 16 intermediate upper-body poses. The temporal patterns of these upper-body poses are modelled by training an HMM for each exercise. Our system recognizes different exercises performed by the patient and assigns an accuracy score for each exercise carried out in a session. It intends to help the patient by keeping track of daily exercise routine, advising for improvements and maintaining records for doctor to access. © 2015 IEEE.