Estrogen may be involved in the development of prostate cancer. The association between genetic polymorphisms of estrogen receptors alpha (ESR1) and beta (ESR2) and prostate cancer risk was examined in a nested case-control study in Washington County, Maryland. Incident prostate cancer cases (n = 269) were matched to one or two controls (n = 440) by age, sex, race, and date of blood donation. Associations between estrogen receptor genotypes or dietary intake and the development of prostate cancer were examined in conditional logistic regression models. Results from this study showed that six single base-pair polymorphisms (SNPs) of ESR1 (rs1801132, rs2077647, rs746432, rs2273206, rs851982, rs2228480) and four SNPs of ESR2 (rs4986938, rs928554, rs8018687, rs number not available for ESR2 5696 bp 3' of STP A>G) were not significantly associated with prostate cancer risk, either by allelic or genotypic frequencies. However, an interactive association with BMI was observed in the relationship between prostate cancer risk and genotypes of ESR2 38 bp 3' of STP G>A (rs4986938) (p = 0.031). An interaction between intake level of phytoestrogen and genotypes of ESR1 Ex1-192G>C (rs746432) and between intake level of phytoestrogen and genotypes of ESR1 Ex8+229G>A (rs2228480) and risk of prostate cancer was observed (p = 0.0009 and p = 0.044, respectively). In conclusion, selected genetic polymorphisms of ESR1 and ESR2, overall, were not associated with prostate cancer risk. However, a variation in risk by BMI and phytoestrogen intake was implicated.