Human evolution is marked by a continued enlargement of the brain. Previous studies on human brain evolution focused on identifying sequence divergences of brain size regulating genes between humans and nonhuman primates. However, the evolutionary pattern of the brain size regulating genes during recent human evolution is largely unknown. We conducted a comprehensive analysis of the brain size regulating gene CASC5 and found that in recent human evolution, CASC5 has accumulated many modern human specific amino acid changes, including two fixed changes and six polymorphic changes. Among human populations, 4 of the 6 amino acid polymorphic sites have high frequencies of derived alleles in East Asians, but are rare in Europeans and Africans. We proved that this between-population allelic divergence was caused by regional Darwinian positive selection in East Asians. Further analysis of brain image data of Han Chinese showed significant associations of the amino acid polymorphic sites with gray matter volume. Hence, CASC5 may contribute to the morphological and structural changes of the human brain during recent evolution. The observed between-population divergence of CASC5 variants was driven by natural selection that tends to favor a larger gray matter volume in East Asians.