Building energy retrofits aim to reduce the energy consumption of existing facilities. The most significant impact of any energy retrofit project is during the operation phase of a building, when a building’s users are directly impacted. This study focuses on the building operation phase by evaluating the social dimension of sustainability in energy retrofit projects. This study aimed to identify, categorize, and organize the social sustainability criteria of energy retrofit projects in existing buildings. A three-stage methodology was used that selected qualified experts in the targeted population, employed a Delphi technique to identify criteria for measuring social sustainability in energy retrofit projects, and used concept mapping to organize the identified social criteria. The study identified 19 social sustainability criteria that can be categorized in six clusters: occupants’ health and comfort impact, society enhancement, cultural and community education, project stakeholder enhancement, building quality and technology enhancement, and socioeconomic growth. An empirical framework was developed for considering the impact level of social sustainability criteria in energy retrofit projects, including three levels: building, community, and society. The proposed framework can be used by practitioners who seek to consider or evaluate the social impacts of energy retrofits as well as policy makers who investigate the social aspects or social costs of energy reduction programs.