Extreme heat exposure has severe negative impacts on humans, and the issue is exacerbated by climate change. Estimating spatial heat stress such as mean radiant temperature (MRT) is currently difficult to apply at city scale. This study constructed a method for estimating the MRT of street canyons using Google Street View (GSV) images and investigated its large-scale spatial patterns at street level. We used image segmentation using deep learning to calculate the view factor (VF) and project panorama into fisheye images. We calculated sun paths to estimate MRT using panorama images from Google Street View. This paper shows that regression analysis can be used to validate between estimated short-wave, long-wave radiation and the measurement data at seven field measurements in the clear-sky (0.97 and 0.77, respectively). Additionally, we compared the calculated MRT and land surface temperature (LST) from Landsat 8 on a city scale. As a result of investigating spatial patterns of MRT in Seoul, South Korea, we found that a high MRT of street canyons (>59.4 °C) is mainly distributed in open space areas and compact low-rise density buildings where the sky view factor is 0.6–1.0 and the building view factor (BVF) is 0.35–0.5, or west-east oriented street canyons with an SVF of 0.3–0.55. However, high-density buildings (BVF: 0.4–0.6) or high-density tree areas (Tree View Factor, TVF: 0.6–0.99) showed low MRT (<47.6). The mapped MRT results had a similar spatial distribution to the LST; however, the MRT was lower than the LST in low tree density or low-rise high-density building areas. The method proposed in this study is suitable for a complex urban environment consisting of buildings, trees, and streets. This will help decision makers understand spatial patterns of heat stress at the street level.