In practical applications, semi-flexible pavement materials need to achieve a balance between mechanical properties, high-temperature stability, cracking resistance, and water stability. However, these performance requirements are strongly influenced by the void content of the matrix asphalt mixture, which determines the grouting space and the composite structure of semi-flexible pavement materials. In this study, matrix asphalt mixtures with three different void contents (20.6%, 24.9%, and 29.2%) were prepared. The effects of the matrix void content on the perfusion characteristics and overall performance of semi-flexible pavement materials were systematically evaluated through perfusion tests, Marshall stability and splitting tests, three-point bending tests, rutting tests, and freeze–thaw splitting tests. Subsequently, the impact of the void ratio on each parameter was analyzed. The findings indicate that, within the tested void content range (20.6%–29.2%), increasing the void ratio of the matrix asphalt mixture continuously improved the mechanical properties of the semi-flexible pavement material: the Marshall stability was enhanced by a factor of up to 4.8, and the splitting strength by up to 1.4 times. Results from the trabecular three-point bending test reveal a more pronounced effect of void fraction on the crack resistance of semi-flexible pavement materials with rising temperatures. At 15 °C, lower void fractions lead to superior crack resistance. Conversely, when the void fraction reaches 24.9% at −10 °C, semi-flexible pavement materials exhibit enhanced crack resistance. Rut dynamic stability and freeze–thaw splitting tests demonstrate that higher void ratios in the matrix correlate with superior resistance to high-temperature deformation and water stability. When the void ratio falls below 24.9%, increasing it significantly enhances resistance to high-temperature deformation, with an improvement range of up to 63.6%. Therefore, 24.9% is identified as the recommended balanced air-void content of the matrix asphalt mixture, at which the semi-flexible pavement material achieves the most balanced comprehensive performance in terms of mechanical properties, high-temperature deformation resistance, water stability, and cracking resistance.