To study the effect of hyperbranched polyester with different kinds of terminal groups on the thermomechanical and dielectric properties of silica–epoxy resin composite, a molecular dynamics simulation method was utilized. Pure epoxy resin and four groups of silica–epoxy resin composites were established, where the silica surface was hydrogenated, grafted with silane coupling agents, and grafted with hyperbranched polyester with terminal carboxyl and terminal hydroxyl, respectively. Then the thermal conductivity, glass transition temperature, elastic modulus, dielectric constant, free volume fraction, mean square displacement, hydrogen bonds, and binding energy of the five models were calculated. The results showed that the hyperbranched polyester significantly improved the thermomechanical and dielectric properties of the silica–epoxy composites compared with other surface treatments, and the terminal groups had an obvious effect on the enhancement effect. Among them, epoxy composite modified by the hyperbranched polyester with terminal carboxy exhibited the best thermomechanical properties and lowest dielectric constant. Our analysis of the microstructure found that the two systems grafted with hyperbranched polyester had a smaller free volume fraction (FFV) and mean square displacement (MSD), and the larger number of hydrogen bonds and greater binding energy, indicating that weaker strength of molecular segments motion and stronger interfacial bonding between silica and epoxy resin matrix were the reasons for the enhancement of the thermomechanical and dielectric properties.

In this article, pure epoxy resin and silica–epoxy nanocomposite models were established to investigate the effects of hyperbranched polyester on microstructure and thermomechanical properties of epoxy resin through molecular dynamics simulation. Results revealed that the composite of silica can improve the thermomechanical properties of nanocomposites, including the glass transition temperature, thermal conductivity, and elastic modulus. Moreover, the thermomechanical properties were further enhanced through chemical modification on the silica surface, where the effectiveness was the best through grafting hyperbranched polyester on the silica surface. Compared with pure epoxy resin, the glass transition temperature of silica–epoxy composite modified by silica grafted with hyperbranched polyester increased by 38 K. The thermal conductivity increased with the increase of temperature and thermal conductivity at room temperature increased to 0.4171 W/(m·K)⁻¹ with an increase ratio of 94.3%. Young’s modulus, volume modulus, and shear modulus all fluctuated as temperature rise with a down overall trend. They increased by 44.68%, 29.52%, and 36.65%, respectively, when compared with pure epoxy resin. At the same time, the thermomechanical properties were closely related to the microstructure such as fractional free volume (FFV), mean square displacement (MSD), and binding energy. Silica surface modification by grafting hyperbranched polyester reduced the FFV value and MSD value most and strengthened the combination of silica and epoxy resin matrix the best, resulting in the best thermomechanical properties.