With the global economic restructuring and the consequent population mobility, urban shrinkage has become a common phenomenon. The Qinling–Daba Mountains, a zone with a key ecological function in China, have long experienced population decline and functional degradation. Clarifying the dynamics and influencing factors of urban shrinkage plays a vital role in supporting the sustainable development of the region. This study, using permanent resident population growth rates and nighttime light data, classified cities in the region into four spatial patterns: expansion–growth, intensive growth, expansion–shrinkage, and intensive shrinkage. It further examined the spatial characteristics of shrinkage across four periods (2005–2010, 2010–2015, 2015–2020, and 2020–2022). A Geographically and Temporally Weighted Regression (GTWR) model was applied to examine core influencing factors and their spatiotemporal heterogeneity. The results indicated the following: (1) The dominant pattern of urban shrinkage in the Qinling–Daba Mountains shifted from expansion–growth to expansion–shrinkage, highlighting the paradox of population decline alongside continued spatial expansion. (2) Three critical indicators significantly influenced urban shrinkage: the number of students enrolled in general secondary schools (X₅), the per capita disposable income of urban residents (X₇), and the number of commercial and residential service facilities (X₁₂), with their effects exhibiting significant spatiotemporal heterogeneity. Temporally, X₁₂ was the most influential factor in 2005 and 2010, while in 2015, 2020, and 2022, X₅ and X₇ became the dominant factors. Spatially, X₇ significantly affected both eastern and western areas; X₅’s influence was most pronounced in the west; and X₁₂ had the greatest impact in the east. This study explored the patterns and underlying drivers of urban shrinkage in underdeveloped areas, aiming to inform sustainable development practices in regions facing comparable challenges.