Sparse representation-based classification model has been widely applied into hyperspectral image (HSI) classification. Its mechanism is based on the assumption that the nonzero coefficients in the sparse representation mainly lie in the correct class-dependent low-dimensional subspace. However, the high similarity of pixels between some different classes exists in the HSI, which makes the classification process very unstable. In this paper, we propose a sparse representation based on the set-to-set distance (SRSTSD) for HSI classification. Through utilizing the set-to-set distance, the spatial information is incorporated into the sparse representation-based model. Moreover, to further exploit the spatial structure of the pixel, we also propose a patch-based SRSTSD (PSRSTSD) model. Experimental results demonstrate that our proposed methods can achieve excellent classification performance.