Urban parks reduce air temperatures within parks and surroundings by exerting the cooling island effect, significant for mitigating the urban microclimate. However, the park cooling effect may be influenced by the surrounding building configuration, and this needs to be studied in more detail, in particular, to explore how to maximize the cooling effect of parks by adjusting the surrounding building configuration. Thus, in this study, the effects of building height, building interval, and building orientation on the cooling effect of a small urban park were investigated using field measurements and ENVI-met numerical simulations. The results demonstrated that (1) building height, building interval, and building orientation all impact the park cooling effect, but their impacts vary. (2) Building height had the strongest effect on the park cooling intensity, and adjusting building height provided the maximum park cooling intensity (1.2 °C). (3) Building orientation had the most effect on the park cooling distance, 100 m downwind of the park. (4) The park cooling effect is best when the surrounding buildings were parallel to the prevailing wind direction, and the park cool island has the greatest intensity and range. This study can guide decision-makers in optimizing building configuration to maximize the park cooling effect.