Convolutional neural networks (CNNs) have proven to be an effective way for deep feature extraction. However, multispectral and panchromatic images are susceptible to illumination unevenness and noise, and the default cross entropy loss function consider only the local information, resulting in misclassification. In this paper, we propose a novel super-pixel-level deep neural networks for multispectral and panchromatic images classification, and define a novel percep-tualloss function via non-local spectral and structure similarity to suppress the interference of unbalanced light and noise. We also propose the corresponding iteration optimization algorithm in this paper. Experimental results show that the proposed method performs better than the state-of-the-art methods.