We tested whether the protective effects of residential greenness on cognitive function differ by APOE ε4 status by using the Chinese Longitudinal Healthy Longevity Survey (CLHLS). We calculated Normalized Difference Vegetation Index (NDVI) using 500 m radii around residential addresses to measure greenness, and the Mini-Mental State Examination (MMSE) to assess cognitive function. We dichotomized the participants into APOE non-ε4 carriers, and APOE ε4 carriers. We applied the generalized estimating equations (GEE) to examine the association between baseline annual average NDVI, APOE ε4 carrier status, and cognitive impairment. Among 6,994 participants, 19.30% were APOE ε4 carriers. Compared to APOE ε4 non-carriers, the APOE ε4 carriers had a 15% higher odds of cognitive impairment (OR: 1.15, 95% CI: 1.05, 1.26). There was an age difference; the protective effect of residential greenness (the highest vs. lowest quartile) on cognitive impairment was observed among the non-ε4 carriers (OR: 0.83, 95% CI: 0.72, 0.95), but not among the ε4 carriers (OR: 1.00, 95% CI: 0.74, 1.34). However, the interaction term between annual average NDVI and APOE ε4 status was not significant (OR: 1.04, 95% CI: 0.97, 1.11). The protective effects of residential greenness on cognitive function differed by APOE ε4 status, which elucidated possible gene-environment interaction mechanisms in which residential greenness may benefit health.