Objective: Against the backdrop of high-intensity academic evaluations, this study examines the longitudinal associations linking leisure-time physical activity (PA) with context-specific anxiety symptoms among university faculty. It further examines the cross-lagged indirect role of emotional exhaustion (EE) and the contrasting moderating effects of challenge and hindrance academic stressors. Methods: A three-wave cross-lagged panel design spanning six months was employed to track 356 faculty members from five universities in China. Validated scales were utilized to measure leisure-time PA, challenge/hindrance academic stressors, EE, and context-specific anxiety symptoms across three critical academic periods (T1, T2, and T3). Data were analyzed using Mplus 8.3 for longitudinal measurement invariance, cross-lagged path modeling, and Bootstrap mediation and moderation tests. Results: After controlling for baseline autoregressive effects, T1 leisure-time PA significantly and negatively predicted T2 EE (β = −0.16, p < 0.01), which in turn positively predicted T3 context-specific anxiety symptoms (β = 0.31, p < 0.001). The longitudinal indirect pathway linking T1 PA to T3 context-specific anxiety symptoms through T2 EE was statistically significant (95% CI [−0.088, −0.017]). Furthermore, the two types of stressors exhibited contrasting moderating effects on the PA-EE relationship. High challenge stressors strengthened the negative association between PA and subsequent EE (Simple Slope = −0.32, p < 0.001). Conversely, high hindrance stressors were associated with a reversal of the slope, wherein higher PA involvement was associated with higher subsequent EE (Simple Slope = 0.12, p < 0.05), a pattern consistent with what has been termed an “exercise paradox.” Conclusions: In this three-wave observational study, regular leisure-time PA was longitudinally associated with, but did not demonstrably cause, lower subsequent context-specific anxiety symptoms among university faculty, with this association potentially operating through reduced emotional exhaustion. The strength of this association was contingent on the type of academic stress: challenge stressors appeared to strengthen the negative association between leisure-time PA and subsequent EE, whereas high-intensity hindrance stressors were associated with a reversal of the predictive pattern. These findings, given the observational design, are best interpreted as longitudinal predictive patterns rather than causal effects. They suggest that interventions targeting faculty well-being may benefit from combining administrative burden reduction with individual recovery support, though direct experimental verification is needed.