In an edaphically heterogeneous area in the Peruvian Amazon, clay soils and nutrient-poor white sands each harbor distinctive plant communities. To determine whether a trade-off between growth and antiherbivore defense enforces habitat specialization on these two soil types, we conducted a reciprocal transplant study of seedlings of 20 species from six genera of phylogenetically independent pairs of edaphic specialist trees and manipulated the presence of herbivores. Clay specialist species grew significantly faster than white-sand specialists in both soil types when protected from herbivores. However, when unprotected, white-sand specialists dominated in white-sand forests and clay specialists dominated in clay forests. Therefore, habitat specialization in this system results from an interaction of herbivore pressure with soil type.