Incorporating phase change material (PCM) into paving materials can regulate the pavement temperature, improve the pavement durability, and mitigate the heat-island effects. In this research, porous fillers were used as the PCM carrier, and the thermal and rheological behaviors of the asphalt mastic with the PCM were evaluated. Two different carrier materials (diatomite and expanded perlite) and four types of PCMs were used in the study. The candidate filler, PCM, and proper blending ratios were determined based on the results of scanning electron microscope image analysis, the filter paper test, and the temperature sweep test. The thermal and rheological behaviors of the mastics with PCMs were further evaluated with different filler replacement ratios. Thermal analysis through a differential scanning calorimetry test, thermal conductivity and volumetric heat capacity test, and real-time temperature performance test were performed on the asphalt mastics. Rheological tests, including the complex shear modulus test, the bending beam rheometer test, and the linear amplitude sweep test, were also conducted. The modified mastics were found to have high heat capacity with the latent heat storage ability. The rheological analyses showed that with the addition of polyethylene glycol, while the low-temperature performance of the asphalt mastics was improved, the performance at intermediate and high temperatures was not adversely affected by the PCM.