Motion-in-depth causes changes in the size of retinal images in addition to producing optic flow patterns. A previous psychophysical study showed that human subjects can perceive expansion motion in texture stimuli that exhibit increases in the scale of image elements but no consistent optic flow pattern. The neural mechanisms by which the scale-change information is processed remain unknown. Here, we measured the responses of cat V1 and the lateral geniculate nucleus (LGN) neurons to a sequence of random images whose spatial frequency spectrum changed over time (i.e., average spatial scale expanded or contracted). We found that V1 neurons exhibit direction sensitivity to scale changes, with more cells preferring expansion than contraction motion. This direction sensitivity can be partly accounted for by the spectrotemporal receptive field of V1 neurons. Comparison of the direction sensitivity between V1 and LGN neurons showed that the sensitivity in V1 may originate from LGN neurons. Repetitive stimulation with expansion or contraction motion can decrease the sensitivity to the adapted direction in V1, and the effect can be transferred interocularly, suggesting that intracortical connections may be critically involved in the adaptation. Together, our results suggest that direction sensitivity to scale change in V1 may contribute to motion-in-depth processing.