Background/Objectives: The actual daily dose distribution during image-guided radiotherapy (IGRT) for prostate cancer (PCa) deviates from the planned due to positional and volumetric changes in the patient’s body and organs at risk (OAR). This study investigates the difference between planned and delivered dose, by calculating the actual daily dose to bladder and rectum for each radiotherapy fraction. The impact of OAR volumes on dose distribution and the correlation with treatment-related toxicities will be evaluated. Methods: A cone-beam computed tomography (CBCT) based daily dose calculation was performed for a total of 821 CBCT scans of 20 patients with localized PCa treated with IGRT. Each fraction’s dose-volume histogram was analyzed, and toxicities were correlated with OAR dose exposures. Results: Daily dosimetric evaluation showed a significant increase in bladder V65–V70 and rectum V50–V70 compared to planned values (p ≤ 0.003 each). In contrast to bladder Dmean, the rectum Dmean was significantly increased (0.47 vs. 0.55; p < 0.001). For the bladder an exponential dose-volume relationship was demonstrated, while no dose-volume relationship was found for the rectum. On average, patients with increased genitourinary toxicities received significantly increased bladder Dmean (0.44 vs. 0.63, p = 0.014). Conclusions: Conventional dose evaluation via a summation plan rather underestimates daily dosimetric parameters. For adaptive radiotherapy of PCa, volumetric parameters rather than mean doses should be used for daily treatment planning constraints. Because established dose constraints cannot be reduced to a single fraction, reasonable dose constraints should consider daily positional and volumetric changes, rather than relying on a single planning CT.