Accurate detection of individual trees is essential for urban forest management and ecological assessment, yet remains challenging due to the heterogeneous backgrounds, variable sizes of tree crowns, and significant variations across urban scenarios. To address these issues, we propose Tree-SAM, a city-scale individual tree detection architecture built upon the visual foundation model Segment Anything Model (SAM) and equipped with three task-specific modules, i.e., Cross-Correlation Feature Backbone (CCFB), Hierarchical Instance Aggregation Neck (HIAN), and Context-Aware Adaptation Head (CAAH). These modules synergistically fuse general semantics with fine-grained structural cues, enable multi-scale feature aggregation, and adaptively refine predictions based on specific scene contexts. On the GZ-Tree Crown dataset, Tree-SAM achieves F1-scores of 0.762, 0.732, and 0.830, with corresponding AP@50 values of 0.478, 0.454, and 0.526 in the forest, mixed, and urban scenarios, respectively, consistently ranking first across all scenes and demonstrating strong adaptability to diverse intra-city landscapes. Additional evaluations on the BAMFORESTS dataset and the SZ-Dataset further confirm its robustness across varied geographic contexts. Tree-SAM provides a reliable, automated framework for large-scale urban tree mapping, providing reliable data support for urban forest management, carbon stock estimation, and ecological assessment.