Image classification is a popular and challenging topic in the computer vision field. On the basis of advances in neuroscience, this paper proposes a sparse-based neural response feature extraction method for image classification. The approach extracts discriminative and invariant representations of images by alternating between non-negative sparse coding and maximum pooling operation with effectiveness. Additionally, effective template selection methods are proposed to further enhance the performance of the algorithm. In comparison with traditional hierarchical methods, our proposed model accounts for the neural processing of visual cortex in human brain, which appears to gain more beneficial discriminative and robust properties for image classification tasks. A variety of benchmarks are used to evaluate the algorithm. The experiment results demonstrate that our proposed algorithm achieves quite excellent or state-of-the-art performance compared with other popular methods.