The rapid growth of hiking tourism presents a critical challenge for balancing visitor safety with the sustainable management of ecologically fragile mountain environments. Traditional models developed in urban settings struggle to capture the highly non-linear, heterogeneous, and zero-inflated characteristics of wilderness trekking behavior. In order to quantify the nonlinear and threshold-based effects of environmental variables on hikers’ spatial decisions in unstructured wilderness and to identify distinct behavioral regimes for segmented management, this study introduces an explainable machine learning framework to reconstruct hikers’ spatial decision-making in a complex mountainous system in Inner Mongolia, China. Random Forest (RF), XGBoost, and LightGBM were compared in predicting trail density and the Euclidean distance to the nearest trail. Results show that transforming behavioral traces into continuous proximity surfaces dramatically improves model performance, with XGBoost achieving the highest predictive accuracy for Trail_Dist. By integrating the SHapley Additive exPlanations framework, this study moves beyond black-box prediction to reveal the nonlinear mechanisms driving hiker behavior. Key findings include: (1) Nighttime light range exhibits a U-shaped threshold effect as the primary anthropogenic attractor. (2) Elevation shows an exponential inhibitory trend above 1238 m. (3) Strong spatial coupling exists between elevation and slope, alongside a landscape compensation effect where high Normalized Difference Vegetation Index (NDVI) areas attract off-trail movements. This research provides a robust methodological pathway for predicting behavior in unstructured outdoor environments. It offers a scientific foundation for smart scenic area management, including optimized route planning, precise ecological protection zoning, and targeted emergency rescue preparedness.