Nowadays, numerous corporations (such as Google, Baidu, etc.) require an efficient and effective search algorithm to crawl out the images with queried objects from databases. Moreover, privacy protection is a significant issue such that confidential images must be encrypted in corporations. Nevertheless, decrypting and then classifying millions of encrypted images becomes a heavy burden to computation. In this paper, we proposed an encrypted image classification framework based on multi-layer extreme learning machine that is able to directly classify encrypted images without decryption. Experiments were conducted on popular handwritten digits and letters databases. Results demonstrate that the proposed framework is secure, efficient and accurate for classifying encrypted images.