Due to the redundancy of imaging systems, multispectral and panchromatic images are of higher spatial resolutions and characterized by different attributes, and are often fused together for accurate land-cover mapping. In this work, we propose a novel framework via adaptive multi-scale convolutional neural networks and perceptual loss function for multispectral and panchromatic images classification. In the proposed scheme, adaptive multi-scale convolutional neural networks are designed to capture the scale information of objects adaptively. And perceptual loss function is constructed via non-local spectral and structural similarities to suppress the interference of unbalanced illumination and speckle noises. A corresponding iteration optimization algorithm is presented to solve the proposed perceptual loss function. Experimental results conducted on three datasets indicate that the proposed framework performs better than the state-of-the-art methods.