The COVID-19 vaccine has been administered to over 200 countries and regions. With the unprecedented vaccination scale and speed, vaccination correlated mental health issues should be paid precise attention to. This study aims to assess the association between socio-demographic factors and mental health following vaccination and to analyze the mediation effect of vaccine hesitancy. This study recruited 2112 individuals who took two doses of the COVID-19 vaccine in Shanghai. Structural equation modeling was performed to assess factors associated with anxiety and depression of the vaccinated individuals and the underlying mechanism. The results yielded that vaccine hesitancy partially mediated/suppressed the effect from gender and employment status to anxiety/depression and fully mediated the effects from education to anxiety/depression. This study advanced the understanding of mental health disparity among different socio-demographic groups after vaccination and the impact of vaccine hesitancy on the vaccinated population’s mental health. The finding offered insights into the possible mental vulnerability of people holding a hesitant attitude before vaccination and suggested that vaccine hesitancy played a crucial role in people’s mental health after vaccination. Health promotion programs can target vaccine hesitancy to prevent unfavorable mental health consequences among specific populations.