While clinical features of panic disorder show significant sexual dimorphism, previous structural MRI studies have not sufficiently controlled for sex when looking at regional brain abnormalities in panic disorder. Using optimized voxel-based morphometry (VBM), regional gray matter volume was compared between 24 patients (male/female: 9/15) with panic disorder and 24 healthy subjects matched for age and sex. Significant gray matter volume reductions were found in the bilateral dorsomedial and right ventromedial prefrontal cortices, right amygdala, anterior cingulate cortex, bilateral insular cortex, occipitotemporal gyrus and left cerebellar vermis in the patients compared with the controls. Among these regions, the VBM revealed significant sexual dimorphism: volume reduction in the right amygdala and the bilateral insular cortex was significantly greater in the males, while reduction in the right superior temporal gyrus was greater in females. Furthermore, a significant reduction in the dorsolateral and ventrolateral prefrontal cortices, thalamus, and parietal cortex was specific to the female patients. The present study demonstrated the morphological changes in extensive brain regions of patients with panic disorder compared with the sex-matched controls. The current results further suggested that the sexually dimorphic clinical phenotypes of panic disorder might have a neurobiological background even at the structural level of the brain.