Many multi-object tracking methods have been developed to solve the computer vision problem which has been attracting significant attentions. In this paper, a novel convolutional neural networks with frame-pair input method for multi-object tracking is presented. It is found that our object tracking methods trained using two successive frames tend to predict the centers of searching windows as the locations of tracked targets. CNN features and color histogram features are extracted as appearance features to measure similarities between objects which used for Tracklets. Kalman Filter and Hungarian algorithm are used to create tracklets association which indicates the location of tracked targets. Specifically, we construct a novel sampling strategy for off-line training. Experiments on the popular challenging datasets show that the proposed tracking system performs on par with recently developed generic multi object tracking methods, but with much less memory. In addition, our tracking system can run in a speed of over 80 (30) fps with a GPU (CPU), much faster than most deep neural networks based trackers. We found that simply improving detection performance can lead to much better multiple object tracking results.