Temporal segmentation of complex human action videos into action primitives plays a pivotal role in building models for human action understanding. Studies in the past have introduced unsupervised frameworks for deriving a known number of motion primitives from action videos. Our work focuses towards answering a question: Given a set of videos with humans performing an activity, can the action primitives be derived from them without specifying any prior knowledge about the count for the constituting sub-actions categories? To this end, we present a novel community detection-based human action segmentation algorithm. Our work marks the existence of community structures in human action videos where the consecutive frames around the key poses group together to form communities similar to social networks. We test our proposed technique over the stitched Weizmann dataset and MHADI01-s motion capture dataset and our technique outperforms the state-of-the-art techniques of complex action segmentation without the count of actions being pre-specified. © 2018 IEEE.