Smooth pursuit eye movements enable us to focus our eyes on moving objects by utilizing well-established mechanisms of visual motion processing, sensorimotor transformation and cognition. Novel smooth pursuit tasks and quantitative measurement techniques can help unravel the different smooth pursuit components and complex neural systems involved in its control. The maintenance of smooth pursuit is driven by a combination of the prediction of target velocity and visual feedback about performance quality, thus a combination of retinal and extraretinal information that has to be integrated in various networks. Different models of smooth pursuit with specific in- and output parameters have been developed for a better understanding of the underlying neurophysiological mechanisms and to make quantitative predictions that can be tested in experiments. Functional brain imaging and neurophysiological studies have defined motion sensitive visual area V5, frontal (FEF) and supplementary (SEF) eye fields as core cortical smooth pursuit regions. In addition, a dense neural network is involved in the adjustment of an optimal smooth pursuit response by integrating also extraretinal information. These networks facilitate interaction of the smooth pursuit system with multiple other visual and non-visual sensorimotor systems on the cortical and subcortical level. Future studies with fMRI advanced techniques (e.g., event-related fMRI) promise to provide an insight into how smooth pursuit eye movements are linked to specific brain activation. Applying this approach to neurological and also mental illness can reveal distinct disturbances within neural networks being present in these disorders and also the impact of medication on this circuitry.