Traditional villages are crucial for the sustainable development of both urban and rural areas, and identifying their spatial patterns is key to guiding village construction and promoting urban–rural integration. This research selected 274 traditional Huizhou villages located in the upper basin of the Xin’an River. It examined how the four main factors—construction period, geography, ecology, and social and economic development—shape and influence each other. By incorporating an optimal parameters-based geographical detector model, this study further explored the driving mechanisms behind spatial differentiation. The villages exhibit a “one belt, two cores, and multiple dispersion” pattern, with Shexian and Yixian counties as hot gathering areas of traditional villages. Population migration, internal growth, and external cultural and commercial exchanges drove village formation in three stages. Spatial distribution favors locations with gentle slopes, sunny aspects, proximity to water, suitable climates, convenient transportation, and distance from crowded areas. Topography, water systems, and external communication are key drivers, while the synergy between water systems and transportation is particularly significant. This study concludes that water systems have the greatest influence on village spatial patterns, recommending watersheds as regional boundaries and advocating a clustering development model for planning and protection efforts.