Driving safety depends on the drivers’ attentional focus on the driving task. Especially in complex situations, distraction due to secondary stimuli can impair driving performance. The inhibition of distractors or inadequate prepotent responses to irrelevant stimuli requires cognitive control, which is assumed to be reduced with increasing age. The present EEG study investigated the effects of secondary acoustic and visual stimuli on driving performance of younger and older car drivers in a driving simulator task. The participants had to respond to brake lights of a preceding car under different distraction conditions and with varying task difficulties. Overall, the anticipation of high demanding tasks affected braking response behavior in young and especially in older adults, who showed reduced cognitive control to task-relevant braking stimuli, as reflected by a smaller P3b. In a more easy (perception only) task, simultaneously presented acoustic stimuli accelerated braking response times (RTs) in young and older adults, which was associated with a pronounced P2. In contrast, secondary visual stimuli increased braking RTs in older adults, associated with a reduced P3b. In a more difficult (discrimination) task, braking response behavior was impaired by the presence of secondary acoustic and visual stimuli in young and older drivers. Braking RT increased (and the P3b decreased), especially when the responses to the secondary stimuli had to be suppressed. This negative effect was more pronounced with visual secondary stimuli, and especially so in the older group. In sum, the results suggest an impaired resistance to distractor interference and a reduced inhibition of prepotent responses in older drivers. This was most pronounced when the processing of task-relevant and irrelevant stimuli engage the same mental resources, for example, by sharing the same stimulus modality.