Behavioral evidence shows that individuals with high trait anxiety tend to be distracted by irrelevant stimulation not only for threat-related stimuli but also for non-emotional neutral stimuli. These findings suggest that there may be a general deficit of attentional control in trait anxiety. However, the neural mechanism underlying the anxiety-related deficit in attentional control, especially inhibition function, is still unclear. Here, we examined the attentional processing of the non-emotional neutral distractor on 66 young adults with different levels of trait anxiety, using the ERP indices of attentional selection (N2pc) and top–down inhibition (Pd) in a search task with geometric stimuli. We found that the distractor-evoked N2pc amplitude did not vary with anxiety levels, but increased anxiety was associated with smaller Pds (i.e. worse inhibition). Besides, delayed attentional selection of targets was associated with higher anxiety levels. These correlations of trait anxiety remained significant even after controlling for state anxiety, and state anxiety did not affect the attentional processing of distractors and targets, suggesting that trait anxiety, not current anxiety, affects attentional function. Our findings clarify the mechanism underlying the general attentional deficits in trait anxiety, e.g. reduced distractor inhibition and delayed target selection.