What parts of a visual stimulus produce the greatest neural signal? Previous studies have explored this question and found that the onset of a stimulus's edge is what excites early visual neurons most strongly. The role of inhibition at the edges of stimuli has remained less clear, however, and the importance of neural responses associated with the termination of stimuli has only recently been examined. Understanding all of these spatiotemporal parameters (the excitation and inhibition evoked by the stimulus's onset and termination, as well as its spatial edges) is crucial if we are to develop a general principle concerning the relationship between neural signals and the parts of the stimulus that generate them. Here, we use visual masking illusions to explore this issue, in combination with human psychophysics, awake behaving primate neurophysiology in the lateral geniculate nucleus of the thalamus, and optical recording in the primary visual cortex of anesthetized monkeys. The edges of the stimulus, rather than its interior, generate the strongest excitatory and inhibitory responses both perceptually and physiologically. These edges can be imaged directly by using optical recording techniques. Excitation and inhibition are moreover most powerful when the stimulus turns both on and off (what might be thought of as the stimulus's temporal edges). We thus conclude that there is a general principle that relates the generation of neural signals (excitatory and inhibitory) to the spatiotemporal edges of stimuli in the early visual system.