Introduction: Evidence exists that individual-level sociodemographic factors contribute to vaccine hesitancy, but it is unknown how community-level factors affect COVID-19 booster dose hesitancy. The current study aims to fill this knowledge gap by comparing data from a nationwide survey on COVID-19 vaccine hesitancy with a community-level indicator, i.e., the Distressed Communities Index (DCI). Methods: Attitudes toward vaccinations, vaccine literacy, COVID-19 vaccine confidence index, and trust were measured using a 48-item, psychometrically valid and reliable survey tool. In this study, 2138 survey participants residing in the United States were divided into quintiles of varying community distress levels based on their zip codes using the DCI. Data were analyzed through Chi-square, one-way ANOVA, and post hoc analysis with Tukey’s test. Results: A significantly higher proportion of participants from the distressed communities had lower trust than their prosperous counterparts (26.6% vs. 37.6%, p < 0.001). On the contrary, participants from the prosperous communities had significantly higher vaccine confidence index scores than those in distressed communities (2.22 ± 1.13 vs. 1.70 ± 1.01, p < 0.001). Conclusions: These findings affirm the importance of developing community-level interventions to promote trust in COVID-19 vaccinations and increase booster dose uptake. From these results, future studies can examine the efficacy of various community-level interventions.