Human faces look more similar to each other when they are presented upside down, leading to an increase in error rate and response time during individual face discrimination tasks. This face inversion effect (FIE) is one of the most robust findings in the face processing literature. Recent neuroimaging studies using adaptation to face identity have shown that the "fusiform face area" was the primary neural source of the behavioral FIE. However, the time course of the FIE, that is, when inversion affects the coding of facial identity in the human brain, remains unclear. Here, we addressed this question by recording event-related potentials (ERPs) on the scalp during an adaptation paradigm with upright and inverted faces. Subjects were presented first with an adapting face stimulus for 3,000 ms, followed by a second face of either the same identity or a different identity. Starting at about 160 ms after stimulus onset, the ERP response to the second face stimulus was markedly reduced over occipitotemporal electrode sites when it was identical to the adapting face, during the N170 time window. When the exact same stimuli were presented upside down, the reduction of signal was smaller and took place about 30 ms later, in line with the behavioral effect of inversion. This result shows that face inversion affects the early encoding of face identity in the occipitotemporal cortex at about 160 ms. Because inversion is known to disrupt massively the integration of facial features, these observations provide indirect evidence that individual faces are processed holistically as early as 160 ms after stimulus onset.