The effect of sargassum particle size on the final properties of sargassum-modified asphalt is investigated in this article. Seaweed sargassum particles were first obtained and characterized through elemental analysis, thermogravimetric analysis (TGA), X-ray diffraction, and FTIR spectroscopy. Additionally, pure and sargassum-modified asphalt blends were evaluated through physical and rheological tests such as penetration, softening point, thermal stability, dynamic viscosity, failure temperature, and epifluorescence microscopy. Modified asphalt blends were prepared by the hot mixing technique using different proportions of sargassum particles of two maximum sizes: 500 μm and 850 μm. Incorporating 3.0 wt.% of sargassum particles under 500 μm into the asphalt increased the viscosity of the original binder by a factor of 2.5 and its complex modulus by a factor of 1.9. At the same time, its failure temperature was 11 °C higher than the reference asphalt, which implies an improved viscoelastic behavior and rutting resistance at high temperatures. The study results suggest that the particles under 500 μm were responsible for the most significant effect on the final properties of the asphalt. Moreover, the storage stability test revealed that the modified asphalt blends are stable when the sargassum particle content was kept below 3.0 wt.%. The statistical analysis of the effect of sargassum particle size and concentration on the modified asphalt properties revealed that the rheological behavior is more affected by the modifier particle size; in contrast, the conventional physical properties were more determined by its concentration. Therefore, using low proportions of fine sargassum particles is efficient for improving the physical and rheological properties of the original asphalt, which is not only positive from the asphalt modification technology point of view but also from a sustainable perspective, since seaweed sargassum has become a useless plague in many coastal regions.