Conventional models for hyperspectral image (HSI) classification usually demand a substantial quantity of labeled training data. However, when labeled training HSIs are unavailable or have different distributions from test HSIs, many classification models tend to exhibit significant performance declines. For the cross-scene HSI classification task, unsupervised domain adaptation (UDA) technique has been widely developed. Existing discrepancy-based or adversarial-based UDA methods may fail to learn discriminative class boundaries when large class distribution shift (CDS) exists. To alleviate this limitation, we propose the model named Spatially Enhanced Refined Classifier (SERC), which includes a coarse classifier and a refined classifier. The refined classifier constructs a memory module to fuse global-spatial and spectral information simultaneously, and employs the neighborhood aggregation technique to generate the refined predictions. The refined predictions are then transferred to pseudo-labels to train the coarse classifier. Thereby, a mutually reinforcing relationship between the two classifiers is established. Furthermore, we propose the Class Distribution Match (CDM) strategy to further alleviate the serious CDS problem. Notably, SERC doesn't require additional training parameters, which are commonly employed in existing UDA methods. Despite its simplicity, SERC achieves outstanding results. Our method has been extensively evaluated on three public HSI datasets and has shown superior performance compared to state-of-the-art (SOTA) approaches. The source code can be found at https://github.com/Wangzs0228/SERC.