Our ability to focus attention on task-relevant information and ignore distractions is reflected by differential enhancement and suppression of neural activity in sensory cortex (i.e., top-down modulation). Such selective, goal-directed modulation of activity may be intimately related to memory, such that the focus of attention biases the likelihood of successfully maintaining relevant information by limiting interference from irrelevant stimuli. Despite recent studies elucidating the mechanistic overlap between attention and memory, the relationship between top-down modulation of visual processing during working memory (WM) encoding, and subsequent recognition performance has not yet been established. Here, we provide neurophysiological evidence in healthy, young adults that top-down modulation of early visual processing (< 200 ms from stimulus onset) is intimately related to subsequent WM performance, such that the likelihood of successfully remembering relevant information is associated with limiting interference from irrelevant stimuli. The consequences of a failure to ignore distractors on recognition performance was replicated for two types of feature-based memory, motion direction and color. Moreover, attention to irrelevant stimuli was reflected neurally during the WM maintenance period as an increased memory load. These results suggest that neural enhancement of relevant information is not the primary determinant of high-level performance, but rather optimal WM performance is dependent on effectively filtering irrelevant information through neural suppression to prevent overloading a limited memory capacity.