To examine the relationship between apolipoprotein E ε4 (ApoE ε4) and psychiatric symptoms, we compared ε4/ε4, ε3/ε3, and ε3/ε4 subjects. 659 outpatients with memory complaints underwent comprehensive neuropsychiatric assessment interview and neurological examination and ApoE genotyping: 98 were ε4/ε4. 18.4% (n = 18) ε4/ε4, 19.3% (n = 45) ε3/ε4, and 5.4% (n = 14) ε3/ε3 presented with symptoms of anxiety (p = 0.00001). ε4/ε4 patients with mild cognitive impairment (MCI; p < 0.0001) and those with Alzheimer's disease with late onset (p = 0.0175) were the most frequently affected. For anxiety, there were no gender dependent differences in the two homozygous groups, however, in the ε3/ε4 group, anxiety symptoms were evident in 7.3% (n = 8) of the male versus 30.1% (n = 37) of the female ε3/ε4 heterozygotes (p < 0.0001). Depression was found in 20.4% (n = 20) ε4/ε4 and 21.0% (n = 49) ε3/ε4 compared to 17.1% (n = 44) ε3/ε3 (p = 0.5181). Visual hallucinations were reported in 5.1% (n = 5) ε4/ε4 as opposed to 3.8% (n = 9) ε3/ε4 and 2.3% (n = 6) ε3/ε3 (p = 0.5278). We have seen a higher association of anxiety with the ApoE ε4 allele across all stages of disease and what may be a dosing effect in the early stage (MCI) for this ostensible risk, since we see a significantly higher frequency in the ApoE ε4 homozygotes when compared to the heterozygotes.