The idea of magnetic monopoles in spin ice has enjoyed much success at intermediate temperatures, but at low temperatures a description in terms of monopole dynamics alone is insufficient. Recently, numerical simulations were used to argue that magnetic impurities account for this discrepancy by introducing a magnetic equivalent of residual resistance in the system. Here we propose that oxygen deficiency is the leading cause of magnetic impurities in as-grown samples, and we determine the defect structure and magnetism in Y2Ti2O7-δ using diffuse neutron scattering and magnetization measurements. These defects are eliminated by oxygen annealing. The introduction of oxygen vacancies causes Ti(4+) to transform to magnetic Ti(3+) with quenched orbital magnetism, but the concentration is anomalously low. In the spin-ice material Dy2Ti2O7 we find that the same oxygen-vacancy defects suppress moments on neighbouring rare-earth sites, and that these magnetic distortions markedly slow down the long-time monopole dynamics at sub-Kelvin temperatures.