Background: Androgen deprivation therapy (ADT) combined with radiation therapy benefits intermediate- and high-risk prostate cancer (PC) patients. The optimal ADT duration in combination with high-dose proton beam therapy (PBT) remains unknown. Methods: Intermediate- and high-risk PC patients treated with PBT combined with ADT for various durations were analyzed retrospectively. To assess the relationship between ADT and biochemical relapse-free (bRF) rate, Cox proportional hazards models including T stage, prostate specific antigen (PSA) level, Gleason score (GS), and total radiation dose were used. Results: In the intermediate-risk PC patients (n = 520), ADT use improved bRF (HR 0.49, 95% CI 0.26–0.93; p = 0.029), especially in those with multiple intermediate-risk factors (T2b–2c, PSA 10–20 ng/mL, and GS 7). In the high-risk PC patients (n = 555), a longer ADT duration (>6 months) conferred a benefit for bRF (HR 0.54, 95% CI 0.32–0.90; p = 0.018), which was most apparent in patients with multiple high-risk factors (T3a–4, PSA > 20 ng/mL, and GS ≥ 8) treated with ADT for ≥21 months. Conclusions: Short-term (≤6 months) ADT is beneficial for intermediate-risk PC patients, but likely unnecessary for those with a single risk factor, whereas ADT for >6 months is necessary for high-risk PC patients and ADT for ≥21 months might be optimal for those with multiple risk factors in combination of high-dose PBT.