Image similarity measure has been widely used in pattern recognition and computer vision. We usually face challenges in terms of rotation and scale changes. In order to overcome these problems, an effective similarity measure which is invariant to rotation and scale is proposed in this paper. Firstly, the proposed method applies the log-polar transform to eliminate the rotation and scale effect and produces a row and column translated log-polar image. Then the obtained log-polar image is passed to hierarchical kernels to eliminate the row and column translation effects. In this way, the output of the proposed method is invariant to rotation and scale. The theoretical analysis of invariance has also been given. In addition, an effective template sets construction method is presented to reduce computational complexity and to improve the accuracy of the proposed similarity measure. Through the experiments with several image data sets we demonstrate the advantages of the proposed approach: high classification accuracy and fast.