Development of low-cost robots with the capability to detect and avoid obstacles along their path is essential for autonomous navigation. These robots have limited computational resources and payload capacity. Further, existing direct range-finding methods have the trade-off of complexity against range. In this paper, we propose a vision-based system for obstacle detection which is lightweight and useful for low-cost robots. Currently, monocular vision approaches used in the literature suffer from various environmental constraints such as texture and color. To mitigate these limitations, a novel algorithm is proposed, termed as Pyramid Histogram of Oriented Optical Flow (p-HOOF), which distinctly captures motion vectors from local image patches and provides a robust descriptor capable of discriminating obstacles from nonobstacles. A support vector machine (SVM) classifier that uses p-HOOF for real-time obstacle classification is utilized. To avoid obstacles, a behavior-based collision avoidance mechanism is designed that updates the probability of encountering an obstacle while navigating. The proposed approach depends only on the relative motion of the robot with respect to its surroundings, and therefore is suitable for both indoor and outdoor applications and has been validated through simulated and hardware experiments. © 2018 World Scientific Publishing Company.