Traditional inference in neuroimaging consists in describing brain activations elicited and modulated by different kinds of stimuli. Recently, however, paradigms have been studied in which the converse operation is performed, thus inferring behavioral or mental states associated with activation images. Here, we use the well-known retinotopy of the visual cortex to infer the visual content of real or imaginary scenes from the brain activation patterns that they elicit. We present two decoding algorithms: an explicit technique, based on the current knowledge of the retinotopic structure of the visual areas, and an implicit technique, based on supervised classifiers. Both algorithms predicted the stimulus identity with significant accuracy. Furthermore, we extend this principle to mental imagery data: in five data sets, our algorithms could reconstruct and predict with significant accuracy a pattern imagined by the subjects.