Campus buildings often present hidden safety risks such as falls and wheelchair instabilities, which are closely related to architectural layout, material selection, and accessibility design. This study develops YOLOv11-Safe, an attention-enhanced and geometry-aware framework that functions as both a detection model and a spatial diagnostic tool for building safety assessment. The framework integrates a modified SimAM attention mechanism and a normalized Wasserstein distance (NWD) loss to improve detection accuracy in complex indoor environments, trained on a dataset of 1000 annotated images covering fall and wheelchair accident scenarios. To interpret spatial risk patterns, a Random Forest classifier combined with SHAP analysis was applied to quantify the contribution of five architectural–behavioral variables: body–ground contact ratio (BGCR), accessibility index (AI), event duration (D), body posture angle (PA), and spatial density (SD). Results show that BGCR and AI dominate the risk-level prediction, while D, PA, and SD refine boundary conditions. Scene-based verification further demonstrated that the framework accurately localized unsafe features—such as uneven drainage edges and discontinuous handrails—and translated them into actionable design feedback. The proposed approach thus links deep-learning detection with interpretable spatial analysis, offering a quantitative foundation for evidence-based architectural safety optimization in university campuses.