To study the effect of hyperbranched polyester with different kinds of terminal groups on the thermomechanical and dielectric properties of silica–epoxy resin composite, a molecular dynamics simulation method was utilized. Pure epoxy resin and four groups of silica–epoxy resin composites were established, where the silica surface was hydrogenated, grafted with silane coupling agents, and grafted with hyperbranched polyester with terminal carboxyl and terminal hydroxyl, respectively. Then the thermal conductivity, glass transition temperature, elastic modulus, dielectric constant, free volume fraction, mean square displacement, hydrogen bonds, and binding energy of the five models were calculated. The results showed that the hyperbranched polyester significantly improved the thermomechanical and dielectric properties of the silica–epoxy composites compared with other surface treatments, and the terminal groups had an obvious effect on the enhancement effect. Among them, epoxy composite modified by the hyperbranched polyester with terminal carboxy exhibited the best thermomechanical properties and lowest dielectric constant. Our analysis of the microstructure found that the two systems grafted with hyperbranched polyester had a smaller free volume fraction (FFV) and mean square displacement (MSD), and the larger number of hydrogen bonds and greater binding energy, indicating that weaker strength of molecular segments motion and stronger interfacial bonding between silica and epoxy resin matrix were the reasons for the enhancement of the thermomechanical and dielectric properties.