The core clinical feature of semantic dementia is a progressive yet selective degradation of conceptual knowledge. Understanding the cognitive and neuroanatomical basis for this deficit is a key challenge for both clinical and basic science. Some researchers attribute the deficit to damage to pan-modal conceptual representations that are independent of any particular sensory-motor modality and are represented in the ventrolateral anterior temporal lobes. Others claim that damage to modality-specific visual feature representations in the occipitotemporal 'ventral stream' is responsible. In the present study, we tested the hypothesis that concept degradation in semantic dementia involves a combination of these pan-modal and modality-specific elements. We investigated factors influencing knowledge of object concepts by analysing 43 sets of picture-naming data from patients with semantic dementia. We found a strong influence of two pan-modal factors: highly familiar and typical items were named more accurately than less familiar/atypical items at all stages of the disorder. Items associated with rich sensory-motor information were also named more successfully at all stages, and this effect was present for sound/motion knowledge and tactile/action knowledge when these modalities were studied separately. However, there was no advantage for items rich in visual colour/form characteristics; instead, this factor had an increasingly negative impact in the later stages of the disorder. We propose that these results are best explained by a combination of (i) degradation of modality-independent conceptual representations, which is present throughout the disorder and is a consequence of atrophy focused on the ventrolateral anterior temporal lobes; and (ii) a later additional deficit for concepts that depend heavily on visual colour/form information, caused by the spreading of atrophy to posterior ventral temporal regions specialized for representing this information. This explanation is consistent with a graded hub-and-spoke model of conceptual knowledge, in which there is a gradual convergence of information along the temporal lobes, with visual attributes represented in the posterior cortex giving way to pan-modal representations in the anterior areas.