On the basis of a modified three-dimensional (3D) random distribution fiber model, this study further investigates the flexural–tensile rheological behavior and its influence factors of fiber-reinforced asphalt mortar. First, the viscoelastic creep at a temperature of 15 °C for pure asphalt mortar as the control sample are obtained by the beam bending creep test to fit the Burgers constitutive parameters. Second, a 3D numerical model consisting of a homogeneous asphalt mortar matrix with viscoelastic parameters and short and straight fibers with elastic characteristics is built in a cuboid space on the basis of a fiber algorithm to simulate the flexural–tensile rheological behavior using ABAQUS software, and the rheological behavior of the 3D model is consistent with those of the test result. Finally, 3D numerical simulations are conducted to further analyze the effect of fiber factors (e.g., contents, aspect ratios, modulus, and fiber types) on the rheological behavior. Results show that the effect of basalt fiber (BF) compared with steel wool fiber are more significant, and increasing fiber contents and aspect ratios have a positive reinforcement effect on the rheological behavior, where BF content for 0.1%, 0.2%, and 0.3% at 3600 s compared with the control reduced by 37.5%, 53%, and 61.7%, and BF aspect ratios for 30, 40, and 50 compared with that for 20 increased by 4.3%, 16.1%, and 32.9%, respectively, but the change in fiber modulus has a minimal impact.