Human activity recognition (HAR) by machine learning finds wide applications ranging from posture monitoring for healthcare and rehabilitation to suspicious or dangerous actions detection for security surveillance. Infrared cameras such as Microsoft Kinect and wearable sensors have been the two most adopted devices for collecting data for measuring the bodily movements. These two types of sensors generally are categorized as contactless sensing and contact sensing respectively. Due to hardware limitation, each of the two sensor types has their inherent limitations. One most common problem associating with contactless sensing like Kinect is the distance and indirect angle between the camera and the subject. For wearable sensor, it is limited in recognizing complex human activities. In this paper, a novel data fusion framework is proposed for combining data which are collected from both sensors with the aim of enhancing the HAR accuracy. Kinect is able to capture details of bodily movements from complex activities, but the accuracy is dependent heavily on the angle of view; wearable sensor is relatively primitive in gathering spatial data but reliable for detecting basic movements. Fusing the data from the two sensor types enables complimenting each other by their unique strengths. In particular, a new scheme using incremental learning with decision table coupled with swarm-based feature selection is proposed in our framework for achieving fast and accurate HAR by fusing data of two sensors. Our experiment results show that HAR accuracy could be improved from 23.51% to 68.35% in a case of almost 90 degrees slanted view of Kinect sensing while a wearing sensor is used at the same time. The swarm feature selection in general is shown to enhance the HAR performance compared to standard feature selection method. The experiment results reported here contribute to the possibilities of using hybridized sensors from the machine learning perspective.