The urban environment characterized by various stresses poses challenges to trees. In particular, water deficits and high temperatures can cause immense drought stress to urban trees, resulting in reduced growth and die-off. Drought-tolerant species are expected to be resilient to these conditions and are therefore advantageous over other, more susceptible species. However, the drought tolerance of urban trees in relation to the specific growth conditions in urban areas remains poorly researched. This study aimed to analyze the annual growth and drought tolerance of two common urban tree species, namely small-leaved lime (Tilia cordata Mill. (T. cordata)) and black locust (Robinia pseudoacacia L. (R. pseudoacacia)), in two cities in southern Germany in relation to their urban growing conditions. Marked growth reductions during drought periods and subsequent fast recovery were found for R. pseudoacacia, whereas T. cordata exhibited continued reduced growth after a drought event, although these results were highly specific to the analyzed city. We further show that individual tree characteristics and environmental conditions significantly influence the growth of urban trees. Canopy openness and other aspects of the surrounding environment (water supply and open surface area of the tree pit), tree size, and tree species significantly affect urban tree growth and can modify the ability of trees to tolerate the drought stress in urban areas. Sustainable tree planting of well adapted tree species to their urban environment ensures healthy trees providing ecosystem services for a high quality of life in cities.