Although many cognitive models in anxiety propose that an impaired top-down control enhances the processing of task-irrelevant stimuli, few studies have paid attention to task-irrelevant stimuli under a cognitive load task. In the present study, we investigated the effects of the working memory load on attention to task-irrelevant stimuli in trait social anxiety. The results showed that as trait social anxiety increased, participants were unable to disengage from task-irrelevant stimuli identical to the memory cue under low and high working memory loads. Impaired attentional disengagement was positively correlated with trait social anxiety. This impaired attentional disengagement was related to trait social anxiety, but not state anxiety. Our findings suggest that socially anxious people have difficulty in disengaging attention from a task-irrelevant memory cue owing to an impaired top-down control under a working memory load.