Impaired empathic response has been described in patients following brain injury, suggesting that empathy may be a fundamental aspect of the social behavior disturbed by brain damage. However, the neuroanatomical basis of impaired empathy has not been studied in detail. The empathic response of patients with localized lesions in the prefrontal cortex (n = 25) was compared to responses of patients with posterior (n = 17) and healthy control subjects (n = 19). To examine the cognitive processes that underlie the empathic ability, the relationships between empathy scores and the performance on tasks that assess processes of cognitive flexibility, affect recognition, and theory of mind (TOM) were also examined. Patients with prefrontal lesions, particularly when their damage included the ventromedial prefrontal cortex, were significantly impaired in empathy as compared to patients with posterior lesions and healthy controls. However, among patients with posterior lesions, those with damage to the right hemisphere were impaired, whereas those with left posterior lesions displayed empathy levels similar to healthy controls. Seven of nine patients with the most profound empathy deficit had a right ventromedial lesion. A differential pattern regarding the relationships between empathy and cognitive performance was also found: Whereas among patients with dorsolateral prefrontal damage empathy was related to cognitive flexibility but not to TOM and affect recognition, empathy scores in patients with ventromedial lesions were related to TOM but not to cognitive flexibility. Our findings suggest that prefrontal structures play an important part in a network mediating the empathic response and specifically that the right ventromedial cortex has a unique role in integrating cognition and affect to produce the empathic response.