To investigate the relationship between visual acuity and cognitive function with aging, we compared low-vision and normally-sighted young and elderly individuals on a spatial working memory (WM) task. The task required subjects to memorise target locations on different matrices after perceiving them visually or haptically. The haptic modality was included as a control to look at the effect of aging on memory without the confounding effect of visual deficit. Overall, age and visual status did not interact to affect WM accuracy, suggesting that age does not exaggerate the effects of visual deprivation. Young participants performed better than the elderly only when the task required more operational processes (i.e., integration of information). Sighted participants outperformed the visually impaired regardless of testing modality suggesting that the effect of the visual deficit is not confined to only the most peripheral levels of information processing. These findings suggest that vision, being the primary sensory modality, tends to shape the general supramodal mechanisms of memory.