In this work, we focus on real-time 3D reconstruction or localization and mapping for outdoor scene using an aerial vehicle called quadcopter. Quadcopter provides the advantages of high flexibility and wide view field in spatial movement. However, existing feature-based and direct methods (using dense or semi-dense approach) are not suitable for outdoor environment, in which multiple challenging scenarios arise such as lighting variance, jittering views, high-speed and non-smooth flight trajectory. The main reason is that the existing methods rely on the assumption of brightness constancy across multiple images and only raw pixel intensities are employed for direct image alignment. In order to tackle these scenarios, a novel method called Feature-based Direct Tracking and Mapping (FDTAM) is proposed, which i) incorporates an efficient binary feature descriptor into direct image alignment module to tackle the challenging scenarios, such as drifting issue under lighting variance problem; ii) applies semi-dense approach to obtain high reconstruction quality; iii) provides a framework with low computational complexity for real-time reconstruction. Compared to other state-of-the-art feature-based and direct methods, our proposed method is shown to tackle the challenging scenarios and improve the accuracy and robustness even in CPU (rather than GPU) platform.