Varying illumination is a huge challenge of face recognition. The variation caused by varying illumination in the face appearance can be much larger than the variation caused by personal identity. The high frequency signal component in image represents the detail characteristic of the face, and for the reason of being influenced scarcely by varying illumination, this signal component can be used as illumination invariance features in face recognition. However, the definition of the high frequency signal component is blurry, and it is impossible to separate this component from the face image exactly. Because of using the different decomposition methods and different decomposition parameters, high frequency component has been dispersed in decomposed detail images that characterize themselves by containing different scale frequency signal component. This paper proposes a framework to fuse that high frequency signal components in multi-scale detail images using adaptive weight. This novel framework is an open structure, and any method of getting illumination invariance feature can be applied on this framework. The experiment based on three open face databases shows the framework proposed by this paper can get remarkable performance. © 2012 World Scientific Publishing Company.