We used fast, pseudorandom temporal sequences of preferred and antipreferred stimuli to drive neuronal firing rates rapidly between minimal and maximal across the visual system. Stimuli were tailored to the preferences of cells recorded in the lateral geniculate nucleus (magnocellular and parvocellular), primary visual cortex (simple and complex), and the extrastriate motion area MT. We found that cells took longer to turn on (to increase their firing rate) than to turn off (to reduce their rate). The latency difference (onset minus offset) varied from several to tens of milliseconds across cell type and stimulus class and was correlated with spontaneous or driven firing rates for most cell classes. The delay for response onset depended on the nature of the stimulus present before the preferred stimulus appeared, and may result from persistent inhibition caused by antipreferred stimuli or from suppression that followed the offset of the preferred stimulus. The onset delay showed three distinct types of dependence on the temporal sequence of stimuli across classes of cells, implying that suppression may accumulate or wear off with time. Onset latency is generally longer, can be more variable, and has marked stimulus dependence compared with offset latency. This suggests an important role for offset latency in assessing the speed of information transmission in the visual system and raises the possibility that signal offsets provide a timing reference for visual processing. We discuss the origin of the delay in onset latency compared with offset latency and consider how it may limit the utility of certain feedforward circuits.