Filled high thermal conductivity epoxy composite solves the problem of the low thermal conductivity of the epoxy resin itself, but the addition of the thermal conductive filler reduces the mechanical properties of the composite, which limits its application in the field of high voltage insulation. In this work, carboxyl-terminated butadiene nitrile liquid rubber (CTBN) was used to toughen the boron nitride-epoxy hybrid system, and the effects of different contents of CTBN on the mechanical properties, thermal conductivity, glass transition temperature, thermal stability, and dielectric properties of the composites were investigated. The results showed that when the content of CTBN was 5–15 wt.%, the CTBN formed a dispersed island structure in the epoxy resin matrix. The toughness of the composite increased by about 32%, the breakdown strength was improved, and the thermal conductivity was about 160% higher than that of pure epoxy resin. As the CTBN content increased, the glass transition temperature and thermal stability of the composite decreased and the dielectric constant and the dielectric loss increased. When the CTBN content is 10–15 wt.%, a toughened epoxy composite material with better comprehensive properties is obtained.