Image classification is one important task in image processing and pattern recognition. Traditional image classification methods commonly transform the image into a vector. However, in essence, image is a matrix data and using vector instead of image loses the correlations of the matrix data. To address this problem, we propose a graph-based multiple rank regression model (GMRR), which employs multiple-rank left and right projecting vectors to regress each matrix data to its label for each category. To exploit the discriminating structure of the data space, a class compactness graph is constructed to constrain these left and right projecting vectors. The extensive experimental results on image classification have demonstrated the effectiveness of our proposed method. (c) 2018 Elsevier B.V. All rights reserved.