Cultural heritage tourism (CHT) has become an important part of the tourism industry. Therefore, it is essential to evaluate the sustainability of CHT destinations for tourism development. The majority of existing studies have examined the destination sustainability of ecotourism with the considerations of three stakeholder groups, namely, resource administration, tourist, and local community. However, the impact of tourism enterprises on destination sustainability has been constantly ignored in CHT-related studies. To fill these gaps, this study aims at developing an indicator system for comprehensively assessing the sustainability of CHT destinations. Specifically, this study identified an evaluation indicator system based on the Delphi technique and applied it to the context of Pingyao Ancient City in China. Results demonstrated that scores obtained by all four stakeholder groups ranged between 3.0 and 4.0 and the overall sustainability score was only 63.94, indicating that Pingyao was in the “somewhat sustainable” state according to the criteria (0−25, unsustainable; 25−50, somewhat unsustainable; 50−75, somewhat sustainable; 75−100, sustainable) specified in previous studies. A more balanced outcome should be achieved where each stakeholder’s interests are fully considered, but not at the expense of the others. The findings are of great significance for heritage tourism literature and sustainable destination management.