Visual working memory (VWM) is a remarkable skill dependent on the brain's ability to construct and hold an internal representation of the world for later comparison with an external stimulus. The prefrontal cortex (PFC) and basal ganglia (BG) interact within a cortical and subcortical network supporting VWM. We used scalp electroencephalography in groups of patients with unilateral PFC or BG lesions to provide evidence that these regions play complementary but dissociable roles in VWM. PFC patients show behavioral and electrophysiological deficits manifested by attenuation of extrastriate attention and VWM-related neural activity only for stimuli presented to the contralesional visual field. In contrast, patients with BG lesions show behavioral and electrophysiological VWM deficits independent of the hemifield of stimulus presentation but have intact extrastriate attention activity. The results support a model wherein the PFC is critical for top-down intrahemispheric modulation of attention and VWM with the BG involved in global support of VWM processes.