Carotid artery stenosis (CAS) is a common vascular disease with long-term consequences for the brain. Although CAS is strongly associated with impaired cerebral hemodynamics and neurodegeneration, the mechanisms underlying hemodynamic impairment in the microvasculature remain unknown. In this work, we employed functional near-infrared spectroscopy (fNIRS) to introduce a methodological approach for quantifying the temporal delay of the evoked hemodynamic response. The method was validated during a vasodilatory task (breath-holding) in 50 CAS patients and 20 controls. Our results suggest that the hemodynamic response to breath-holding can be delayed by up to 6 s in the most severe patients, a significant increase from the median 4 s measured for the control group (p = 0.01). In addition, the fraction of brain regions that responded to the task decreased as the CAS severity increased, from a median of 90% in controls to 73% in the most severe CAS group (p = 0.04). The presence of collateral circulation increases the response to breath-holding and decreases the average time delays across the brain, although the number of communicating arteries alone cannot predict these fNIRS-based hemodynamic variables (p > 0.09). Overall, this work proposes a method to quantitatively assess impaired cerebral hemodynamics in CAS patients.