Working memory capacity is one of the most important cognitive functions influencing individual traits, such as attentional control, fluid intelligence, and also psychopathological traits. Previous research suggests that anxiety is associated with impaired cognitive function, and studies have shown low verbal working memory capacity in individuals with high trait anxiety. However, the relationship between trait anxiety and visual working memory capacity is still unclear. Considering that people allocate visual attention more widely to detect danger under threat, visual working memory capacity might be higher in anxious people. In the present study, we show that visual working memory capacity increases as trait social anxiety increases by using a change detection task. When the demand to inhibit distractors increased, however, high visual working memory capacity diminished in individuals with social anxiety, and instead, impaired filtering of distractors was predicted by trait social anxiety. State anxiety was not correlated with visual working memory capacity. These results indicate that socially anxious people could potentially hold a large amount of information in working memory. However, because of an impaired cognitive function, they could not inhibit goal-irrelevant distractors and their performance decreased under highly demanding conditions.