Few studies have reported that the correlation between flow and diffusion characteristics at different spatial scales in urban areas is affected by atmospheric stability conditions. This study established a multi-scale numerical model (from neighborhood to indoor scale) validated by wind tunnel experiments to investigate flow pattern, heat transfer, and pollutant dispersion within different scale urban areas. The atmospheric stability conditions are considered and characterized using the bulk Richardson number (Rib), and then seven Rib scenarios (from −0.21 to 0.79), two pollutant source locations (in- and out-), and two building array types: AL (aligned array) and ST (staggered array) are examined. The findings suggest that the dispersion and dilution effects of pollutants are significantly diminished under thermally stable and AL array conditions. When Rib > 0, the concentration experiences a decline of 31%–36% as Rib decreases, while when Rib < 0, the influence of thermal buoyancy becomes more pronounced, resulting in a reduction in the concentration difference between the AL and ST arrays. Furthermore, the distribution of air exchange rate reveals that increased instability improves the ventilation efficacy, albeit its effect diminishes once it surpasses a critical Rib. This study is helpful to clarify the interrelationship of flow and dispersion phenomena within different scale urban areas, so as to enhance ventilation and alleviate air pollution.