Prostate Imaging-Reporting and Data System (PI-RADS) has been widely implemented as a diagnostic tool for significant prostate cancer (PCa); less is known about its prognostic value, especially in the setting of primary radiotherapy. We aimed to analyze the association between PI-RADS v. 2.1 classification and risk of metastases, based on a group of 152 patients treated with ultra-hypofractionated stereotactic CyberKnife radiotherapy for localized low or intermediate risk-group prostate cancer. We found that all distant failures (n = 5) occurred in patients diagnosed with a PI-RADS score of 5, and axial measurements of the target lesion were associated with the risk of developing metastases (p < 0.001). The best risk stratification model (based on a combination of greatest dimension, the product of multiplication of PI-RADS target lesion axial measurements, and age) achieved a c-index of 0.903 (bootstrap-validated bias-corrected 95% CI: 0.848–0.901). This creates a hypothesis that PI-RADS 5 and the size of the target lesion are important prognostic factors in early-stage PCa patients and should be considered as an adverse prognostic measure for patients undergoing early treatment such as radiation or focal therapy.