Attention is increasingly being given to understanding sex difference in psychopathology to better understand the etiology of disorders. This study tests the hypothesis that sex differences in ventral and middle frontal gray volume contribute to sex differences in antisocial personality disorder (APD) and crime. Participants were recruited from temporary employment agencies, consisting of normal controls, substance/alcohol-dependent controls, axis I/II psychiatric controls and individuals with APD. An independent sample of female volunteers was also recruited. Magnetic resonance imaging volumes of superior frontal, middle frontal, inferior frontal, orbital frontal and rectal gyral frontal gray matter, and dimensional scores of APD and criminal behavior were assessed. APD males when compared with male controls showed an 8.7% reduction in orbitofrontal gray volume, a 17.3% reduction in middle frontal gray and a 16.1% reduction in right rectal gray. Reduced middle and orbitofrontal volumes were significantly associated with increased APD symptoms and criminal offending in both males and females. Males as a whole had reduced orbitofrontal and middle frontal gray volume when compared with females, and controlling for these brain differences reduced the gender difference in the antisocial personality/behavior by 77.3%. Findings were not a function of psychiatric comorbidity, psychosocial risk factors, head injury or trauma exposure. Findings implicate structural differences in the ventral and middle frontal gray as both a risk factor for APD and as a partial explanation for sex differences in APD.