Qilian Mountain National Park (QLMNP) is a biodiversity hotspot with great agriculture and tourism resources. With the expansion of human activities, a few areas of the park are experiencing massive landscape transformation, and these areas are also highly ecologically sensitive. Nevertheless, there are substantial differences in the human activities and natural resources of various communities around QLMNP, resulting in heterogeneous landscape degradation. Hence, this study explores the extent and drivers of spatial heterogeneity in landscape fragmentation associated with ecologically vulnerable communities in QLMNP. Multiple ring buffer analysis and geographically weighted regression (GWR) were used to analyze the relationships between landscape fragmentation and variables of human activities and facilities to identify the main factors influencing landscape fragmentation in different regions. The results reveal that human disturbance had a stronger relationship with landscape fragmentation in QLMNP than natural factors do. Among the drivers of landscape fragmentation, the distribution of residential areas and the extension of agricultural land were found to have more significant impacts than tourism. Expansion of cropland had a greater impact on the eastern part of the national park, where overgrazing and farming require further regulation, while tourism affected the landscape fragmentation in the central area of the national park. The point-shaped human disturbance had a larger impact than the linear disturbance. The study findings can be used to formulate a comprehensive plan to determine the extent to which agriculture and tourism should be developed to avoid excessive damage to the ecosystem.