Spatial attention enhances our ability to detect stimuli at restricted regions of the visual field. This enhancement is thought to depend on the difficulty of the task being performed, but the underlying neuronal mechanisms for this dependency remain largely unknown. We found that task difficulty modulates neuronal firing rate at the earliest stages of cortical visual processing (area V1) in monkey (Macaca mulatta). These modulations were spatially specific: increasing task difficulty enhanced V1 neuronal firing rate at the focus of attention and suppressed it in regions surrounding the focus. Moreover, we found that response enhancement and suppression are mediated by distinct populations of neurons that differ in direction selectivity, spike width, interspike-interval distribution and contrast sensitivity. Our results provide strong support for center-surround models of spatial attention and suggest that task difficulty modulates the activity of specific populations of neurons in the primary visual cortex.