Adaptation is a hallmark of sensory function. Adapting optimally requires matching the dynamics of adaptation to those of changes in the stimulus distribution. Here we show that the dynamics of adaptation in the responses of mouse retinal ganglion cells depend on stimulus history. We hypothesized that the accumulation of evidence for a change in the stimulus distribution controls the dynamics of adaptation, and developed a model for adaptation as an ongoing inference problem. Guided by predictions of this model, we found that the dynamics of adaptation depend on the discriminability of the change in stimulus distribution and that the retina exploits information contained in properties of the stimulus beyond the mean and variance to adapt more quickly when possible.