As a certain case in the domain of human actions, hand gestures can be expressed by the motion of user's hand to provide nature interaction in many applications. In this paper we proposed a real-time hand gesture recognition system based on robust hand tracking from depth image sequences. Using hidden markov models (HMM) with varying states, gesture models are trained online along with user's feedback, and the real-time classification is taken simultaneously. A gesture may be falsely classified as the models are trained insufficiently at beginning, in which case we provide a feedback and update the gesture model with this gesture sample. The performance of the system can always be improved by more updating, and in our experiment we give an appropriate result after a reasonable number of samples are used for training. © 2012 IEEE.