RGBT tracking incorporates thermal infrared data to achieve more accurate visual tracking. However, the efficiency of RGBT tracking may be diminished by some bottlenecks, such as thermal crossover, illumination variation and occlusion. To address the aforementioned problems, we propose a fully-convolutional Siamese-based Multi-modal Feature Fusion Network (SiamMFF) that integrates RGB and thermal features. In our work, visible and infrared images are initially processed by the Multi-Modal Feature Fusion framework (MFF) at the search and template sides, respectively. Then, the attribute-aware fusion module is introduced to conduct feature extraction and fusion for the major challenge attributes. In particular, we design a skip connections guidance module to prevent the propagation of noise and to enrich the feature information so that we can improve the tracker's discriminative ability for modality-specific challenges. The proposed SiamMFF method has been evaluated in a great number of trials on two benchmark datasets GTOT and RGBT234, and the precision rate and success rate can reach 90.5%/73.6% and 81.2%/57.3%, respectively, demonstrating the superiority of our method over existing state-of-the-art methods.