Diet plays a crucial role in regulating individuals’ lifestyles and is closely related to health. The intake of animal-sourced foods (ASF) provides the human body with high-quality protein and various micronutrients. This study aimed to investigate whether the diversity of animal foods has a positive impact on the health-related quality of life (HRQoL) among residents. The data came from the Shaanxi baseline survey of the Northwest Chinese Regional Ethnic Cohort Study, which recruited more than 100 thousand participants aged 35 to 74 from five provinces between June 2018 and May 2019. A total of 39,997 participants in Shaanxi (mean age: 50 years; 64% women) were finally included in this current study. The animal source food diet diversity score (ASFDDS) was established based on the frequency of consuming pork, mutton, beef, poultry, seafood, eggs, pure milk, and yogurt. The physical component score (PCS) and mental component score (MCS), ranging from 0 to 100 on the 12-Item Short Form Survey (SF-12), were used to assess participants’ HRQoL. Better PCS/MCS was defined as scores higher than the 90th percentile. The results showed that men had a higher intake of ASF and ASFDDS than women. After adjusting for potential confounders, compared with those who never or rarely consumed animal foods, the likelihood of having better PCS and MCS increased by 16% (OR = 1.16, 95%CI: 1.01–1.34) and 24% (OR = 1.24, 95%CI: 1.03–1.448), respectively, in men with an ASFDDS ≥ 2. In women, a 34% increase (OR = l.34, 95%CI: 116–l.54) likelihood for better PCS was observed for an ASFDDS ≥ 2, but no association was observed for MCS. Increasing each specific animal source’s food intake was associated with better PCS after adjusting for all covariates. However, for MCS, positive associations were only observed in seafood consumption among men and eggs among women. Restricted cubic splines showed a substantial dose-response association between intake frequency of animal-source foods and PCS, both in men and women. The study suggests that a diverse intake of animal-sourced foods can potentially improve the HRQoL of Chinese adults.