It is not unusual for a place to wait for years before being inscribed on the World Heritage List, but Sarnath—the place where the Buddha delivered his first sermon—has been on UNESCO’s “tentative list” for close to 25 years. As a sacred place for Buddhist pilgrimages, Sarnath continues to attract thousands of visitors annually and yet, remains under-developed, unlike other pilgrim towns or religious tourism destinations. This paper examines the reasons for the stunted growth of Sarnath. The findings are based on fieldwork conducted in Sarnath in 2019. The analysis of stakeholder interviews suggests several reasons for Sarnath not being able to capitalize on its religious and cultural heritage for tourism-led development. It was found that the protection of the site as an archaeological park by the Archaeological Survey of India (ASI), which has been controlling development surrounding the park as well as prohibiting the performance of any rituals, have been the key endogenous factors that have contributed to the relative under-development of a Buddhist pilgrimage economy in the town. A handful of monasteries are where Buddhist followers stay and perform their pilgrimage rituals, rendering them as enclaves. Private-sector accommodation is also limited to around 15 hotels and guesthouses. Administratively, Sarnath does not have an independent governance structure. It is governed as just one ward (an electoral constituency) by the Municipal Corporation of Varanasi and is, thus, always under the shadow of Varanasi city. Varanasi is one of the most significant Hindu sacred cities and, hence, visitors stay there for much longer durations to soak in cultural offerings, including religious ceremonies around the Ganga River; a visit to Sarnath is secondary and often limited to a half-day tour. Moreover, the archaeological park at Sarnath serves a recreational purpose as a picnic site for domestic visitors. The exogenous factors mean that the interests of Sarnath (as a Buddhist site) are hardly acknowledged by the Hindu city of Varanasi. This paper argues that the multi-layered contestations that exist at the site level, the town–ward level, and between visitors and managers have further contributed to the poor state of heritage and tourism in Sarnath.