Background/Objectives: Optic disc drusen (ODD) are an under-recognized cause of optic neuropathy, and the impact of structure–function damage on quality of life (QoL) is poorly defined. We compared systemic risk factors, ocular structure–function, and QoL in adults with ODD versus matched controls and identified determinants of impaired vision-related QoL. Methods: In a tertiary clinic, 60 adults with ultrasonography- or OCT-confirmed ODD were age- and sex-matched 1:1 to 60 controls without ODD. Retrospective clinical and imaging data (BCVA, RNFL thickness, standard automated perimetry) were combined with cross-sectional NEI VFQ-25 and EQ-5D-5L scores. Results: ODD patients more often had hypertension (51.7% vs. 31.7%, p = 0.026) and migraine (38.3% vs. 21.7%, p = 0.046). They showed worse BCVA (0.2 vs. 0.1 logMAR, p < 0.001), thinner RNFL (95.3 vs. 103.8 µm, p < 0.001), more depressed mean deviation (−4.7 vs. −1.3 dB, p < 0.001), and more frequent reproducible visual field defects (68.3% vs. 11.7%, p < 0.001). Vision-specific QoL was reduced (VFQ-25 composite 77.3 ± 11.4 vs. 89.7 ± 8.6, p < 0.001) and generic health status lower (EQ-5D utility 0.8 ± 0.1 vs. 0.9 ± 0.1, p < 0.001). In ODD, worse BCVA, more negative mean deviation and lower EQ-5D were independently associated with poorer VFQ-25 (model R² = 0.57), while older age, thinner RNFL and migraine predicted visual field defects. Conclusions: ODD are associated with substantial visual field loss and clinically meaningful decrements in vision-related and generic QoL.