The Chinese government aims to deal with poverty by 2020 for people living in ethnic and rural regions, including mountainous ethnic regions with the highest concentration of poverty and chronic poverty. Based on a sustainable livelihood Framework, five capitals and 33 evaluation indices of livelihood were built, and 13 counties’ resources of the Aba Tibetan and Qiang Autonomous Prefecture were compared in order to calculate the degree of poverty. Topographic factors index of settlement sites (TFIS) were constructed by eight topographic factors, and diagnoses of the dominant factors of differentiation of 2699 settlements were calculated by using the geographical detector model to establish the poverty alleviation policies and models for different regions. The results showed that the livelihood capital evaluation indices were different (0.56–1.88), and natural capitals (mean value 1.56) had obvious advantages, but physical (mean value 0.56), financial (mean value 0.78), and human capital were lower (mean value 0.93), limiting the rate of transforming the ecological resources advantage into the economy. In the TFIS, the settlement points indicate topographic factors of natural breakpoint classification superposition, including elevation, slope, relief amplitude, surface incision, variance coefficient in elevation, surface roughness, distance to roads, and distance to rivers. These are within the 8–34 range, and their power determinant value to TFIS are 0.02, 0.70, 0.77, 0.76, 0.51, 0.66, 0.06, and 0.09. Livelihood capital evaluation indices and TFIS classification one (8–14) are positively correlated, and negative correlation (22–26 and 27–34) is at the 0.05 level. The county's poverty alleviation measures and development under different livelihood indices and TFIS indicate that the ecotourism industry has become the inevitable choice for promoting rapid and coordinated development of economy, society, and the environment in ethnic regions.