Although it is well established that thermal environments significantly influence travel behavior, the synergistic effects of points of interest (POI) and thermal environments on behavior remain unclear. This study developed a vision-based outdoor evaluation model aimed at uncovering the driving factors behind human behavior in outdoor spaces. First, Yolo v5 and questionnaires were employed to obtain crowd activity intensity and preference levels. Subsequently, target detection and clustering algorithms were used to derive variables such as POI attractiveness and POI distance, while a validated environmental simulator was utilized to simulate outdoor thermal comfort distributions across different times. Finally, multiple classification models were compared to establish the mapping relationships between POI, thermal environment variables, and crowd preferences, with SHAP analysis used to examine the contribution of each variable. The results indicate that XGBoost achieved the best predictive performance (accuracy = 0.95), with shadow proportion (|SHAP| = 0.24) and POI distance (|SHAP| = 0.12) identified as the most significant factors influencing crowd preferences. By extrapolation, this classification model can provide valuable insights for optimizing community environments and enhancing vitality in areas with similar climatic and cultural contexts.