White matter lesions progress over time, but the clinical consequences are widely unknown. Three-hundred twenty-nine elderly community-dwelling volunteers underwent serial magnetic resonance imaging scanning and cognitive testing at baseline and at 3- and 6-year follow-up. We measured the changes in white matter lesion and brain parenchymal volumes. After 6 years, the median increase in white matter lesion load was 0.2 cm3 (interquartile range [IQR], 0.0-0.80 cm3) with a maximum of 31.4 cm3. The median loss of brain volume was 2.3% (IQR, 1.13-3.58%). Increasing white matter lesion volume was correlated with loss of brain volume (p < 0.0001) and performance decline in tests of memory (p = 0.022), conceptualization (p = 0.046), and visuopractical skills (p = 0.005). Associations between changes in white matter lesion load and cognitive functioning were no longer significant when adding change in brain volume to the models, suggesting that cognitive decline related directly to loss of brain substance with progression of lesion burden.