Human total brain size is consistently reported to be approximately 8-10% larger in males, although consensus on regionally specific differences is weak. Here, in the largest longitudinal pediatric neuroimaging study reported to date (829 scans from 387 subjects, ages 3 to 27 years), we demonstrate the importance of examining size-by-age trajectories of brain development rather than group averages across broad age ranges when assessing sexual dimorphism. Using magnetic resonance imaging (MRI) we found robust male/female differences in the shapes of trajectories with total cerebral volume peaking at age 10.5 in females and 14.5 in males. White matter increases throughout this 24-year period with males having a steeper rate of increase during adolescence. Both cortical and subcortical gray matter trajectories follow an inverted U shaped path with peak sizes 1 to 2 years earlier in females. These sexually dimorphic trajectories confirm the importance of longitudinal data in studies of brain development and underline the need to consider sex matching in studies of brain development.