China has set up ecological protection and high-quality development of the Yellow River Basin as its national strategy. However, the fragile natural ecosystem and intensive human disturbances pose challenges to it. This study evaluates habitat quality change and analyzes its drivers in a representative county of this region, aiming to provide scientific basis for ecological protection and sustainable development. We took Liulin, a representative county of middle Yellow River Basin as the study area and evaluated the spatiotemporal variation of habitat quality from 2000 to 2020 with the InVEST model. Further, the influencing factors of habitat quality pattern were explored using GeoDetector, and their gradient ranges dominating the habitat quality change were determined by gradient analysis. The results showed that: (1) Areas of low and medium-low habitat quality grades were distributed interactively in the whole county; medium grade areas were scattered in the northeast and southwest parts of the county; and medium-high and high grades area were distributed sporadically along the Yellow River and its branches. (2) Habitat quality of the county almost unchanged from 2000 to 2010. However, from 2010 to 2020, with the rapid expansion of construction land (increased by 9.62 times), the area proportion of medium, medium-high, and high habitat quality grades decreased from 7.01% to 5.31%, while that of low and medium-low habitat quality grades increased from 92.99% to 94.69%. (3) The habitat quality was influenced by multiple natural-human factors. The main influencing factor was land use, followed by elevation. (4) Most changes of habitat quality occurred in areas with lower elevation, gentler slope, and higher vegetation coverage, which were affected by intensive human activities. These results suggest that in future land use policy making and the construction land expansion in Liulin County should be restricted, and differentiated ecological protection and restoration strategies should be implemented in areas with different habitat quality.