Evolutionary responses to environmental change depend on the time available for adaptation before environmental degradation leads to extinction. Explicit tests of this relationship are limited to microbes where adaptation usually depends on the sequential fixation of de novo mutations, excluding standing variation for genotype-by-environment fitness interactions that should be key for most natural species. For natural species evolving from standing genetic variation, adaptation at slower rates of environmental change may be impeded since the best genotypes at the most extreme environments can be lost during evolution due to genetic drift or founder effects. To address this hypothesis, we perform experimental evolution with self-fertilizing populations of the nematode Caenorhabditis elegans and develop an inference model to describe natural selection on extant genotypes under environmental change. Under a sudden environmental change, we find that selection rapidly increases the frequency of genotypes with high fitness in the most extreme environment. In contrast, under a gradual environmental change selection first favors genotypes that are worse at the most extreme environment. We demonstrate with a second set of evolution experiments that, as a consequence of slower environmental change and thus longer periods to reach the most extreme environments, genetic drift and founder effects can lead to the loss of the most beneficial genotypes. We further find that maintenance of standing genetic variation can retard the fixation of the best genotypes in the most extreme environment because of interference between them. Taken together, these results show that slower environmental change can hamper adaptation from standing genetic variation and they support theoretical models indicating that standing variation for genotype-by-environment fitness interactions critically alters the pace and outcome of adaptation under environmental change.