Recently, the recognition of posture and gesture has been widely used in fields such as medical treatment and human–computer interaction. Previous research into the recognition of posture and gesture has mainly used human skeletons and an RGB-D camera. The resulting recognition methods utilize models of the human skeleton, with different numbers of joints. The processing of the resulting large amounts of feature data needed to recognize a gesture leads to the recognition being delayed. To overcome this issue, we designed and developed a system for learning and recognizing postures and gestures. This paper proposes a gesture recognition method with enhanced generality and processing speed. The proposed method consists of feature collection part, feature optimization part, and a posture and gesture recognition part. We have verified the solution proposed in this paper through the learning and subsequent recognition of 29 postures and 8 gestures.