To identify the chemical components responsible for the anti-hyperglycemic effect of Cyclocarya paliurus (Batal.) Iljinsk (Juglandaceae) leaves, an ethanol extract (CPE) and a water extract (CPW) of C. paliurus leaves, as well as their total flavonoids (CPF), triterpenoids (CPT) and crude polysaccharides (CPP), were prepared and assessed on streptozotocin (STZ)-induced diabetic mice. After being orally administrated once a day for 24 days, CPF (300 mg/kg), CPP (180 mg/kg), or CPF+CPP (300 mg/kg CPF + 180 mg/kg CPP) treatment reversed STZ-induced body weight and muscle mass losses. The glucose tolerance tests and insulin tolerance tests suggested that CPF, CPP, and CPF+CPP showed anti-hyperglycemic effect in STZ-induced diabetic mice. Furthermore, CPF enhances glucose-stimulated insulin secretion in MIN6 cells and insulin-stimulated glucose uptake in C2C12 myotubes. CPF and CPP suppressed inflammatory cytokine levels in STZ-induced diabetic mice. Additionally, CPF and CPP improved STZ-induced diabetic nephropathy assessed by H&E staining, blood urea nitrogen content, and urine creatinine level. The molecular networking and Emperor analysis results indicated that CPF showed potential anti-hyperglycemic effects, and HPLC–MS/MS analysis indicated that CPF contains 3 phenolic acids and 9 flavonoids. In contrast, CPT (650 mg/kg) and CPC (300 mg/kg CPF + 180 mg/kg CPP + 650 mg/kg CPT) did not show antihyperglycemic effect. Taken together, polysaccharides and flavonoids are responsible for the antihyperglycemic effect of C. paliurus leaves, and the clinical application of C. paliurus need to be refined.