Background: Cognitive impairment is a major health concern in older adults. Few studies have examined the association between environmental factors and cognitive impairment, especially in high altitude areas. In this study, the prevalence of cognitive impairment in older adults living in high altitude was compared with those living in low altitude areas. Methods: This was a comparative study conducted at Qinghai (high altitude group), and Guangzhou (low altitude group), China. Cognition, depressive symptoms and quality of life (QOL) were assessed using the Montreal Cognitive Assessment (MoCA), Patient Health Questionnaire (PHQ-9) and WHO Quality of Life brief version–WHOQOL-BREF, respectively. Results: Altogether, 644 older adults (207 in Qinghai and 437 in Guangzhou) completed the assessment. The prevalence rate of cognitive impairment was 94.7% (95% CI: 91.6–97.7%) in older adults living in the high altitude area, while the corresponding figure was 89.2% (95% CI: 86.3–92.1%) in the low altitude area. After controlling for covariates, the high altitude group appeared more likely to have cognitive impairment (OR = 2.92, 95% CI: 1.23–6.91, P = 0.015) compared with the low altitude group. Within the high altitude group sample, multinomial logistic regression analysis revealed that older age (aged 74 and above) was significantly associated with higher risk of severe cognitive impairment (OR = 3.58, 95%CI: 1.44–8.93, P = 0.006), while higher education level (secondary school and above) was associated with decreased risk of moderate cognitive impairment (OR = 0.43, 95%CI: 0.22–0.85, P = 0.006). Within the high altitude group, QOL did not differ significantly between normal/mild, moderate and severe cognitive impairment subgroups across physical [F = 1.83, P = 0.163], psychological [F = 1.50, P = 0.225], social [F = 2.22, P = 0.111] and environmental domains [F = 0.49, P = 0.614]. Conclusion: This study found that cognitive impairment was more common among older adults living in the high altitude area. Regular screening and appropriate interventions should be provided to older adults in need.