Direction information of the palmprint provides one of the most promising features for palmprint recognition. However, more existing direction-based methods only extract the surface direction features from raw palmprint images and ignore the informative latent direction feature of the convolution layer of palmprint images. In this paper, we propose a novel double-layer direction extraction method for palmprint recognition. The method first extracts the apparent direction from the surface layer of a palmprint. Then, it further exploits the latent direction features from the energy map layer of the apparent direction. Lastly, by using the multiplication and addition schemes, the apparent and latent direction features are pooled as the histogram feature descriptor for palmprint recognition. The proposed method achieves state-of-the-art performance on four benchmark palmprint databases, namely the PolyU, IITD, GPDS and CASIA palmprint databases. In particular, the latent energy direction feature shows a promising performance for noisy palmprint image recognition.