The aim of the current study was to evaluate the influence of severe radiation-induced lymphopenia (RIL) on the outcomes of esophageal cancer (EC). A systematic review and meta-analysis was performed through the PRISMA guideline. Seventeen studies were included in the current systematic review, with eight included in the meta-analyses. Meta-analyses found that severe RIL was associated with lower pathologic complete response (pCR) rate (odds ratio (OR) = 0.44, 95% confidence interval (CI) = 0.30–0.66, I² = 0%), inferior overall survival (OS) (hazard ratio (HR) = 1.50, 95% CI = 1.29–1.75, I² = 6%), and worse progression-free survival (PFS) (HR = 1.70, 95% CI = 1.39–2.07, I² = 0%) of EC patients. The lymphocyte nadir was found during 4–6 weeks after the start of radiotherapy. The leading dosimetric factors associated with severe RIL included larger PTV, higher dose to heart and body, and higher effective dose to the immune cells (EDIC). Clinical risk factors for RIL mainly comprised lower baseline ALC, higher tumor length and clinical stage, and distal EC. In conclusion, severe RIL might be associated with a lower pCR rate and worse OS and PFS of EC patients. Minimizing the dosimetric risk factors, especially in patients with clinical risk factors, might benefit their outcomes.