This paper presents a patch-set-based sparse representation for image set classification. Compared with image-based image set representation, our patch-set-based representation is alignment free and thus has an advantage for tasks like video-based face recognition, image-set-based object recognition, and video-based hand gesture recognition, where precious alignment is usually difficult or even impossible due to large variance in view angle or pose. Specifically, to bypass the alignment issue, we propose to adopt the patch-based image set representation by dividing each image within each set into patches, then we cluster all the training patches into multiple clusters and classify the test patches based on the cluster centers of training patches. The labels of test patches within each cluster are inferred from a patch-set-based sparse representation for classification, and the labels of all test patches from all the clusters are then aggregated to predict a single label for the test set. Experimental results on video-based face recognition data sets (CMU-MoBo and YouTube Celebrities), image-set-based object recognition data set (ETH-80), and video-based hand gesture recognition data set (Kinect Hand Gestures) demonstrate that our proposed method consistently outperforms all existing ones, and the improvement is very significant on the YouTube Celebrities and Kinect Hand Gesture data sets. Moreover, we also quantitatively show the robustness of our method to misalignment on the Mutli-PIE data set. © 2016 IEEE.