A balanced diet is essential to achieve and maintain good health. In this study, we assessed diet quality of middle aged and elderly people based on Chinese Diet Balance Index-07 (DBI-07) and explored the associations between DBI-07 and anemia. Data analyzed for this study was from the 2010–2012 National Nutrition and Health Survey in Yunnan province, southwest China (n = 738, aged 50–77 years). Dietary recalls over there consecutive days were done in a face-to-face interview. The scores of DBI-07 for each component and three DBI-07 indicators ((Lower Bound Score (LBS), Higher Bound Score (HBS), Diet Quality Distance (DQD)) were calculated according to compliance with the Dietary Guidelines for Chinese residents. Hemoglobin (Hb) concentration was determined using the cyanmethemoglobin method. Univariate and multivariate linear regression models were used to explore the associations between DBI-07 indicators and anemia, as well as scores of DBI-07 components and Hb level. The sample included 336 men and 402 women. Inadequate intakes of vegetables, fruits, dairy, soybean, eggs, fish and excessive intakes of cereals, meat, cooking oil, salt were both common. 91.3% of the participants had moderate or high levels of inadequate food intake, while 37.7% had moderate or high levels of excessive food intake. The mean Hb was 14.2 ± 1.7 g/dL, with a prevalence of anemia of 13.0%. Subjects with high LBS and DQD were more likely to be anemic (all p < 0.05). After adjustment for potential confounders, there were positive correlations between Hb level and the intakes of vegetables and soybean (β_vegetables = 1.04, p < 0.01; β_soybean = 0.82, p = 0.04). In conclusion, dietary imbalance and anemia are common in middle aged and elderly population in southwest China and inadequate intakes of vegetables and soybean may increase the risk of anemia.