Is sexual orientation associated with structural differences in the brain? To address this question, 80 homosexual and heterosexual men and women (16 homosexual men and 15 homosexual women) underwent structural MRI. We used voxel-based morphometry to test for differences in grey matter concentration associated with gender and sexual orientation. Compared with heterosexual women, homosexual women displayed less grey matter bilaterally in the temporo-basal cortex, ventral cerebellum, and left ventral premotor cortex. The relative decrease in grey matter was most prominent in the left perirhinal cortex. The left perirhinal area also showed less grey matter in heterosexual men than in heterosexual women. Thus, in homosexual women, the perirhinal cortex grey matter displayed a more male-like structural pattern. This is in accordance with previous research that revealed signs of sex-atypical prenatal androgenization in homosexual women, but not in homosexual men. The relevance of the perirhinal area for high order multimodal (olfactory and visual) object, social, and sexual processing is discussed.