Exploring the correlation of the built environment with metro ridership is vital for fostering sustainable urban growth. Although the research conducted in the past has explored how ridership is nonlinearly influenced by the built environment, less research has focused on the spatiotemporal ramifications of these nonlinear effects. In this study, density, diversity, distance, destination, and design parameters are utilized to depict the “5D” traits of the built environment, while Shapley Additive Explanations with eXtreme Gradient Boosting (XGBoost-SHAP) are adopted to uncover the spatial and temporal features concerning the nonlinear relationship of the built environment with ridership for metro stations located in Xi’an. We conducted a K-means clustering analysis to detect different site clusters by utilizing local SHAP coefficients. The results show that (1) built environment variables significantly influence metro ridership in a nonlinear manner at different periods and thresholds, with the POI facility density being the most critical variable and the other variables demonstrating time-driven effects; (2) the variables of population density and parking lot density exhibit spatial impact heterogeneity, while the number of parks and squares do not present a clear pattern; and (3) based on the clustering results, the metro stations are divided into four categories, and differentiated guidance strategies and planning objectives are proposed. Moreover, the current work offers a more developed insight into the spatiotemporal influence of built environments on metro travel in Xi’an, China, using nonlinear modeling, which has vital implications for coordinated urban–metro development.