Background: China’s rapidly aging population presents challenges for cognitive health and mental well-being among the older adults. This study examines how the number of children affects cognitive function in middle-aged and older adults and whether depressive symptoms mediate this relationship. Methods: This study analyzed data from waves 1 to 5 (2011–2020) of the China Health and Retirement Longitudinal Study (CHARLS), involving 5932 participants aged 45 and older. Participants were grouped by the number of children: childless, only child and multiple children. We used Logarithmic Generalized Linear Models (LGLMs) to explore the relationships among the number of children, depressive symptoms, and cognitive function. Indirect effect coefficients and 95% bias-corrected and accelerated confidence intervals (BCaCI) were estimated using Simultaneous Equation Models (SEM) with three-stage least squares (3SLS) and the bootstrap method to assess the mediating effect of depressive symptoms. Results: In middle-aged and older adults, a negative association was observed between the number of children and overall cognitive functioning (all p < 0.01). This association remained significant even after adjusting for covariates in groups with three (β = −0.023, p < 0.05) and four or more children (β = −0.043, p < 0.001). Conversely, the positive association between the number of children and depression also persisted after adjusting for covariates, although it weakened as the number of children increased (all p < 0.01). Depressive symptoms consistently correlated negatively with overall cognitive function (p < 0.001) and partially mediated the relationship between the number of children and cognitive function (pMe = 20.36%, p < 0.05). The proportion of the mediating effect attributed to depression was more pronounced in middle-aged and older adults who had experienced the loss of children (pMe = 24.31%) or had two children (pMe = 25.39%), with stronger mediating effects observed in males (pMe = 48.84%) and urban residents (pMe = 64.58%). Conclusions: The findings indicate that depressive symptoms partially mediate the relationship between the number of children and cognitive function in middle-aged and older adults in China. These results highlight the significance of considering mental health factors when studying cognitive function in this demographic. Notably, in families without children and those with two children, depressive symptoms play a crucial role in explaining the decline in cognitive function.