Resilience refers to the capacity to function well despite adversity and facilitates adaptation to various stressors, including pain. Previous studies have operationalized resilience via questionnaires or task performance behaviors that do not always correlate with one another and tested extreme (lower vs. higher) resilience subgroup differences that do not permit the assessment of nonlinear associations between resilience and brain matter volume. To address these limitations, we identified high (HPR, N = 21), moderate (MPR, N = 20), and low (LPR, N = 16) pain resilience subgroups from a trait resilience questionnaire as well as behavioral performance on a laboratory pain task. Subsequently, resilience subgroup differences in gray matter volume (GMV) and behavior responses to a novel numerical interference task (NIT) were assessed. Behavioral results indicated the LPR subgroup was slower in responding to NIT trials accompanied by pain compared to MPR, and especially, HPR subgroups. Voxel-based morphology analyses indicated the LPR subgroup had more GMV in the left postcentral gyrus than did MPR and HPR subgroups. LPR and MPR subgroups displayed larger GMV in the right inferior temporal gyrus than did the HPR subgroup. Nonlinear relations reflecting less regional GMV in the right inferior frontal gyrus (IFG)/putamen, orbitofrontal cortex and left IFG/insula were also observed for the MPR subgroup compared to LPR and HPR subgroups. In sum, lower pain resilience is characterized, in part, by comparatively greater GMV in pain processing regions and reduced GMV in “resilience” regions, though regional GMV also has several nonlinear associations with resilience.