Stability and optimality are the two foremost re-quirements for robotic systems that are deployed in critical operations and are to work for long hours or under limited energy resources. To address these, in this work we present a novel Lyapunov stability based discrete-time optimal kinematic control of a robot manipulator using actor-critic (AC) framework. The robot is actuated using optimal joint-space velocity control input to track a time-varying end-effector trajectory in its task space. In comparison to the existing near-optimal kinematic control solutions for robot manipulator under AC framework, proposed controller exhibits guaranteed analytical stability. We derive a novel critic weight update law based on Lyapunov stability, thus ensuring that the weights are updated along the negative gradient of Lyapunov function. This eventually ensures closed-loop system stability and convergence to the optimal control in discrete-time. Extensive simulations are performed on a 3D model of 6-DoF Universal Robot (UR) 10 in Gazebo, followed by implementation on real UR 10 robot manipulator to show the efficacy of the proposed scheme. © 2020 IEEE.