COVID-19 vaccine acceptance and refusal vary across countries and among different socio-demographic groups. This study investigates hesitancy related to the COVID-19 vaccine and the associated factors in the rural-community-dominated Jazan Province, Saudi Arabia. A cross-sectional study through an online questionnaire was conducted from February to April 2021 to investigate the extent of vaccine hesitancy related to the COVID-19 vaccine and the associated factors in the Jazan region. A Chi-squared test and post hoc analysis were conducted to analyze the statistical significance of the association between variables. Of the 569 participants who completed the online questionnaire, the majority were males (81.5%) and had a university education (72.6%). Of the participants, more than one-third (36.9%) were hesitant to vaccinate. Concern about adverse side effects following vaccination was the most reported reason for vaccine hesitancy (42.6%), followed by beliefs that the vaccine was unsafe or ineffective (15.5%). The data analysis revealed that people who lived in cities in Jazan Province or those who did not have a family history of COVID-19 infection were more likely to be vaccine hesitant. It is more important than ever to develop and implement community-based strategies to address vaccine hesitancy, especially in rural areas.