Although it is well established that thermal environments significantly influence travel behavior, the synergistic effects of points of interest (POI) and thermal environments on behavior remain unclear. This study developed a vision-based outdoor evaluation model aimed at uncovering the driving factors behind human behavior in outdoor spaces. First, Yolo v5 and questionnaires were employed to obtain crowd activity intensity and preference levels. Subsequently, target detection and clustering algorithms were used to derive variables such as POI attractiveness and POI distance, while a validated environmental simulator was utilized to simulate outdoor thermal comfort distributions across different times. Finally, multiple classification models were compared to establish the mapping relationships between POI, thermal environment variables, and crowd preferences, with SHAP analysis used to examine the contribution of each variable. The results indicate that XGBoost achieved the best predictive performance (accuracy = 0.95), with shadow proportion (|SHAP| = 0.24) and POI distance (|SHAP| = 0.12) identified as the most significant factors influencing crowd preferences. By extrapolation, this classification model can provide valuable insights for optimizing community environments and enhancing vitality in areas with similar climatic and cultural contexts.

Microclimate and built environment jointly influence outdoor activities among the elderly. However, existing studies largely focus on a single season or environmental factor, lacking a comprehensive analysis of seasonal variation and multi-factor coupling effects. This paper investigates the seasonal interaction effects of microclimate and built environment on elderly outdoor activities, with implications for elderly-friendly urban design. Using a typical residential neighbourhood in Xi’an as a case, we constructed a multi-source spatio-temporal dataset through high-density microclimate monitoring in winter and summer, fine-grained POI mapping, and computer-vision-based behavioural annotation. Generalised Additive Models (GAM) and SHAP analysis were employed for modelling and mechanism exploration. The results show that: (1) Elderly activity patterns exhibit a fundamental seasonal reversal—characterised as “sun-seeking and wind-avoiding” in winter and “shade-seeking and wind-pursuing” in summer; (2) Environmental factors exhibit marked nonlinear and threshold-dependent influences that vary by season; (3) Microclimate and built environment elements demonstrate synergistic interaction effects, especially pronounced in summer. Quantitatively, GAM and SHAP analyses indicate that the “effective service radius” of Elderly-Friendly POIs (defined as the threshold where positive influence approaches zero) contracted from approximately 45–50 m in winter to 35–40 m in summer, while their peak promotional effect occurred at 20–25 m. Positive POIs exhibited a significantly shorter influence range, and Negative POIs demonstrated negligible distance-dependent effects. This study confirms a “seasonal dynamic interaction” mechanism and proposes the adaptive design strategy of “sunlight and wind-shelter pockets—shade and ventilation corridors,” offering empirical and methodological support for climate-responsive elderly-friendly community planning.