Fillers (<0.075 mm) in hot mix asphalt (HMA) play a pivotal role in optimizing bitumen content, filling voids, and improving mechanical performance. In many agricultural countries, large quantities of rice and wheat husk waste are produced, while the road construction industry faces material shortages of conventional filler materials and related performance challenges. This study evaluates the feasibility of using rice husk (RH) and wheat husk (WH) fillers on HMA performance. Unlike previous studies that primarily focused on ash-derived agricultural residues, this work investigates the direct utilization of raw husk materials, eliminating the need for energy-intensive processing. Few studies directly examine the aggregate gradation and binder concentration with respect to rice and wheat husk ash. As a result, the relative effectiveness of these two agricultural waste fillers in improving the volumetric and Marshall properties of asphalt mixtures is yet unknown. Fifteen mixtures with varying bitumen contents (3.0–5.0%) were tested to determine the optimum bitumen content (OBC). Subsequently, modified mixtures were prepared at the OBC using RH and WH fillers at five replacement levels (5.0–15.0%). The Marshall Mix design method was employed to assess stability, flow, density, and air voids content. The control mixture showed a Marshall stability of 14.86 kN, flow of 3.52 mm, density of 2.342 g/cm³, and air voids of 2.9%. At their optimum filler contents (i.e., 10.33% for RH and 10.43% for WH), the modified mixtures achieved higher Marshall stability (14.96 kN and 15.06 kN, respectively), with flow values of 3.51 mm and 2.83 mm, and densities of 2.335 g/cm³ and 2.330 g/cm³. Statistical analysis using ANOVA at the OBC confirmed that RH and WH fillers can be used as alternative fillers in HMA without adversely affecting Marshall performance, while contributing to agricultural waste valorization and resource conservation.