• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Neurosci. Author manuscript; available in PMC Apr 17, 2013.
Published in final edited form as:
PMCID: PMC3516184

Neural substrate of the late positive potential in emotional processing


The late positive potential (LPP) is a reliable electrophysiological index of emotional perception in humans. Despite years of research the brain structures that contribute to the generation and modulation of LPP are not well understood. Recording EEG and fMRI simultaneously, and applying a recently proposed single-trial ERP analysis method, we addressed the problem by correlating the single-trial LPP amplitude evoked by affective pictures with the blood-oxygen-level-dependent (BOLD) activity. Three results were found. First, relative to neutral pictures, pleasant and unpleasant pictures elicited enhanced LPP, as well as heightened BOLD activity in both visual cortices and emotion-processing structures such as amygdala and prefrontal cortex, consistent with previous findings. Second, the LPP amplitude across three picture categories was significantly correlated with BOLD activity in visual cortices, temporal cortices, amygdala, orbitofrontal cortex, and insula. Third, within each picture category, LPP-BOLD coupling revealed category-specific differences. For pleasant pictures, the LPP amplitude was coupled with BOLD in occipitotemporal junction, medial prefrontal cortex, amygdala, and precuneus, whereas for unpleasant pictures, significant LPP-BOLD correlation was observed in ventrolateral prefrontal cortex, insula, and posterior cingulate cortex. These results suggest that LPP is generated and modulated by an extensive brain network comprised of both cortical and subcortical structures associated with visual and emotional processing and the degree of contribution by each of these structures to the LPP modulation is valence-specific.


The event-related potential (ERP) method is used extensively in affective neuroscience. A key feature observed in ERPs evoked by emotionally engaging stimuli is the late positive potential (LPP), which is characterized by an amplitude enhancement for pleasant and unpleasant stimuli, relative to neutral stimuli, and has a centroparietal maximum topography. For affective picture viewing, LPP starts around 300 - 400 ms after picture onset, and is often sustained throughout the duration of picture presentation (Cuthbert et al., 2000). LPP amplitude has been shown to vary systematically with the experienced intensity of the affective picture content (Schupp et al., 2000; Keil et al., 2002) and exhibit abnormal patterns in mood disorders and other psychiatric conditions (Leutgeb et al., 2011; Weinberg and Hajcak, 2011; Jaworska et al., 2012). In parallel, functional magnetic resonance imaging (fMRI) have found that viewing of affective pictures is associated with increased blood-oxygen-level-dependent (BOLD) activity in widespread brain regions, including occipital, parietal, inferotemporal cortices, and amygdala (Breiter et al., 1996; Lang et al., 1998a; Bradley et al., 2003; Norris et al., 2004; Sabatinelli et al., 2005; Sabatinelli et al., 2009), suggesting that emotionally salient content enhances visual stimulus processing by attracting attentional resources (Lang et al., 1998b; Lang and Bradley, 2010). Taken together, if enhanced LPP and BOLD reflect a common underlying mechanism, one might expect a coupling between LPP amplitude and BOLD activity in the above reported regions.

A prior study recording EEG and fMRI from the same subjects but in separate sessions has found that LPP amplitude was positively correlated with BOLD responses in lateral occipital, parietal, and inferotemporal cortices (Sabatinelli et al., 2007a). This study did not examine LPP-BOLD coupling in other higher order emotional processing areas such as prefrontal cortex and deep subcortical structures known to be involved in emotional perception (Sabatinelli et al., 2009). A more recent study using a between-subjects design observed coupling between LPP amplitude and BOLD activity in both deep and anterior structures (Sabatinelli et al., in press), but it is still unclear whether these structures are engaged with the LPP in a category-specific way based on trial-by-trial information within each picture category. The advent of the simultaneous EEG-fMRI recording technique, together with reliable estimation of single-trial ERPs, opens new avenues to address this problem.

We recorded simultaneous EEG-fMRI while subjects passively viewed emotionally arousing and neutral pictures. The single-trial LPP amplitudes were estimated using a recently proposed method (Xu et al., 2009) and then correlated with the single-trial evoked BOLD responses across the entire brain to identify brain structures whose activity is linearly related to the trial-by-trial variation of the scalp-recorded LPP. In addition, in light of a host of prior studies reporting differential engagement of cortical and sub-cortical structures in appetitive versus aversive processing (e.g., Sabatinelli et al., 2007a), we investigated whether trial-by-trial LPP amplitude fluctuations are mediated by different neural generators during different affective states by examining the coupling between LPP amplitude and BOLD within each picture category (pleasant, neutral, unpleasant).



Fifteen healthy volunteers participated in the experiment in exchange of either course credits or a financial incentive of $30. One participant withdrew from the experiment. In addition, data from 3 participants were discarded due to artifacts generated by excessive movement inside the scanner. The remaining 11 participants (7 females, mean age: 20, standard deviation: 2.65) had normal or corrected-to-normal vision. The experimental protocol was approved by the Institutional Review Board of the University of Florida. Informed consent was obtained from all participants prior to the experiment.

Stimuli and procedure

The stimuli consisted of 20 pleasant, 20 neutral, and 20 unpleasant pictures selected from the International Affective Picture System (IAPS, Lang et al., 2008) based on their normative valence and arousal levels. The IAPS picture numbers used in this study are: Pleasant: 4311, 4599, 4610, 4624, 4626, 4641, 4658, 4680, 4694, 4695, 2057, 2332, 2345, 8186, 8250, 2655, 4597, 4668, 4693, 8030. Neutral: 2398, 2032, 2036, 2037, 2102, 2191, 2305, 2374, 2377, 2411, 2499, 2635, 2347, 5600, 5700, 5781, 5814, 5900, 8034, 2387. Unpleasant: 1114, 1120, 1205, 1220, 1271, 1300, 1302, 1931, 3030, 3051, 3150, 6230, 6550, 9008, 9181, 9253, 9420, 9571, 3000, 3069. The selected pictures cover a wide range of contents and normative ratings. The pleasant pictures in general included sport scenes, romance, and erotic couples, whereas the unpleasant pictures incorporated threat, attack scenes, and bodily mutilations. The neutral pictures included landscapes and neutral human-beings. The mean pleasure (valence) rating for pleasant, neutral, and unpleasant pictures was 7.0, 6.3, and 2.8, respectively. The pleasant and unpleasant pictures had similar mean arousal levels (pleasant: 5.8, unpleasant: 5.9), both being higher than neutral pictures (4.2). Pictures were chosen to be similar overall in composition, matched in jpeg size across categories, and comparable in rated complexity, to minimize confounds.

The experimental paradigm was implemented in an event-related fMRI design. Each IAPS picture was centrally displayed on a monitor for 3 seconds followed by a variable (2800 or 4300 ms) interstimulus interval (ITI). All participants completed 5 experimental sessions in which the pictures were repeated in different random orders. The order of picture presentation was also randomized across different participants. A cross was displayed at the center of the screen during the entire experiment to aid fixation. Stimuli were presented on an MR-compatible monitor using E-Prime software (Psychology Software Tools). The monitor was placed outside the scanner bore over the head of the subject. Participants viewed the task presentation in the scanner via a reflective mirror. Before the start of the first experimental session, participants were instructed to maintain eye fixation whenever the fixation cross is present and viewed the pictures without moving their eyes. After the experiment, as an validation, participants were asked to provide their ratings of 12 representative pictures (4 pictures within each category) they had not seen during the experiment along the scales of valence and arousal using a paper and pencil version of the self-assessment manikin (Bradley and Lang, 1994). The entire experiment lasted about 40 minutes.

Simultaneous EEG-fMRI acquisition

MRI data were collected on a 3-T Philips Achieva scanner (Philips Medical Systems, the Netherlands). Two hundred and twelve (212) volumes of functional images were acquired using a gradient-echo echo planar imaging (EPI) sequence during each session (echo time (TE): 30 ms, repetition time (TR): 1.98 s, flip angle: 80°, slice number: 36, field of view (FOV): 224 mm, voxel size: 3.5×3.5×3.5 mm, matrix size: 64×64). The slices were acquired in ascending order and oriented parallel to the plane connecting the anterior and posterior commissure. Slice acquisition was performed within an interval of 1850 ms during each TR, leaving an interval of 130 ms toward the end of the TR where no image acquisition was performed. This image acquisition approach allowed us to visually monitor the EEG recording within each volume during the no-scan period where EEG was not contaminated by gradient switching. A T1-weighted high resolution structural image was also obtained.

EEG data were recorded during the experiment using a 32-channel MR-compatible EEG system (Brain Products GmbH). Thirty-one sintered Ag/AgCl electrodes were placed according to the 10-20 system, and one additional electrode was placed on subject’s upper back to monitor electrocardiograms (ECG). The recorded ECG will be used to detect heartbeat events to be used for the removal of the cardioballistic artifact. The EEG channels were referenced to site FCz during recording. The impedance from all scalp channels was kept below 10 kΩ during experiment as suggested by the manufacturer. EEG signal was recorded with a built-in 0.1~250 Hz band-pass filter and digitized to 16 bit at a sampling rate of 5 kHz. The digitized EEG signal was then transferred to the recording computer via a fiber-optic cable. The EEG recording system was synchronized with the scanner’s internal clock throughout the recording session to ensure the successful removal of the gradient artifact in subsequent analyses.

EEG data preprocessing

Brain Vision Analyzer 2.0 (Brain Products GmbH) was used for data preprocessing. Gradient artifacts in the EEG data were removed using a modified version of the original algorithms proposed by Allen et al. (2000). Briefly, an artifact template was created by segmenting and averaging the data according to the onset of each volume within a sliding window consisting of 41 consecutive volumes, and subtracted from the raw EEG data. To remove the cardioballistic artifact, an average artifact subtraction method (Allen et al., 1998) was used, in which R peaks were detected in the low-pass filtered ECG signal and used to construct a delayed average artifact template over 21 consecutive heartbeat events in a sliding-window approach, which was subtracted from the original EEG signal. The resulting EEG data were then low-pass filtered with the cutoff set at 50 Hz, down-sampled to 250 Hz, and re-referenced to the average reference. These data were then exported to the EEGLAB (Delorme and Makeig, 2004). SOBI (Second Order Blind Identification) (Belouchrani et al., 1993) was performed to further correct for eye-blinking, residual cardioballistic, and movement-related artifacts. Recent work has shown that SOBI is effective in removing the residual cardioballistic artifact (Vanderperren et al., 2010), as well as in separating EEG data into physiologically interpretable components (Tang et al., 2005; Klemm et al., 2009). The artifacts-corrected data were then epoched from -300 ms to 2000 ms with 0 ms being the onset of affective pictures. The prestimulus baseline was defined as -300 to 0 ms. The EEG epochs were averaged within each condition separately to produce the average ERP (AERP). The AERP coming from each subject was further averaged across subjects to produce the grand average ERP.

Single-trial estimation of LPP

Channel Pz was chosen to guide our subsequent EEG-informed fMRI analysis as it showed strong LPP difference between both emotional conditions and the neutral condition (Figure 1B and 1C). The ERP of each trial at Pz was estimated using the Analysis of Single-trial ERP and Ongoing activity (ASEO) method (Xu et al., 2009). ASEO has the following basic steps. First, according to the variable signal plus ongoing activity (VSPOA) generative model (Chen et al., 2006), the recorded EEG data for the r th trial (r =1, 2,…,R) are expressed as:


where sn(t) (n =1, 2 …,N) is the n th ERP component and zr(t) is an autoregressive (AR) process modeling the ongoing activity. Within each individual trial, the n th ERP component is characterized by an amplitude scaling factor brn and a latency shift trn to account for trial-to-trial variability. Second, using a proper initial condition, the ASEO algorithm estimates the waveforms of the ERP components and their associated amplitude scaling factors and latency shifts in an iterative fashion. Third, from scaled and latency-adjusted ERP component estimates, the single-trial ERP was reconstructed on a trial-by-trial basis. Fourth, the LPP amplitude on each trial was obtained by averaging the single-trial ERP amplitude within a time interval around the peak of LPP (see Figures 2 and and3).3). To date ASEO has been applied to study both monkey local field potential data and human EEG data (Wang et al., 2008; Wang and Ding, 2011; Liu et al., 2012). See Xu et al. (2009) for a more detailed description of the ASEO algorithm.

Figure 1
ERP analysis. A) Grand average (n = 11 subjects) ERP showing the LPP at Pz with time zero set to the onset of pictures. B) The scalp topography showing the ERP difference between pleasant and neutral conditions. Here ERP was averaged within the time interval ...
Figure 2
Single-trial ERP analysis. A) Epoched EEG data at Pz from a representative subject after artifact removal. B) Single-trial ERPs estimated using ASEO from data shown in A. C) Raster plot of the EEG data in A (smoothed with a moving average across 5 trials ...
Figure 3
Single-trial LPP dynamics. A) Trial-by-trial LPP amplitude for the pleasant and neutral conditions. B) Trial-by-trial LPP amplitude for the unpleasant and neutral conditions. The horizontal axes in A and B represent the sequential index of picture presentation ...

MRI data analysis

The fMRI data were processed using SPM5 (http://www.fil.ion.ucl.ac.uk/spm/). The first five volumes in an experimental session were discarded to allow the scanner to stabilize. Slice timing was corrected using sinc interpolation to account for differences in acquisition time. The images were then corrected for head movement by spatially realigning the images to the sixth image of each session. Images were further normalized and registered to a standard template within SPM (the Montreal Neurological Institute (MNI) space). The functional volume images were resampled to a spatial resolution of 3×3×3 mm. The transformed images were then smoothed by a Gaussian filter with a full-width at half-maximum (FWHM) of 8 mm. The low frequency temporal drifts were removed from the functional images by applying a high-pass filter with a cut-off frequency of 1/128 Hz, and the global signal was removed by dividing every voxel in a slice by the estimated global signal value.

Two separate general linear models (GLMs) with parametric modulation were constructed to model the relationship between LPP amplitude and BOLD. In the first model, we were mainly interested in examining the overall LPP-BOLD coupling across all three picture categories (i.e. pleasant, neutral, and unpleasant). Therefore, in this model we combined all three picture categories and modeled the resulting single experimental condition with two task-related regressors. The first regressor described the combined condition and consisted of a sequence of boxcar functions with unit height synchronized with the onset of pictures. The width of each boxcar function was set to the duration of picture presentation (3 s). For the second regressor, the height of the boxcar functions in the first regressor varied according to the ASEO-estimated single-trial LPP amplitude, with the mean level of the single-trial LPP amplitudes removed (Figure 3C). This regressor was intended to account for both the between-category and the within-category variability in LPP amplitude and its correlation with BOLD. The two regressors were further convolved with a canonical hemodynamic response function (HRF) before being incorporated into the design matrix. Six regressors describing the subjects’ head movement obtained from image preprocessing were further introduced to account for any movement related artifacts during scan. We referred to our first model as the “full model.” A single contrast was performed based on this model resulting in one statistical map for each subject.

In the second model, we included six task-related regressors, with three corresponding to the three picture categories (pleasant, neutral, and unpleasant), and the other three modeling the relationship between LPP amplitudes and BOLD within each picture category. Since this model captures trial-by-trial coupling between LPP amplitude and BOLD within each picture category, we refer to it as the “within-category model”. The three regressors modeling the three picture categories consisted of sequences of boxcar functions with unit height placed according to the picture onset within the corresponding categories. The width of the boxcar functions remained the same as those in the “full model” (3 s). For regressors modeling trial-by-trial LPP-BOLD coupling, we further scaled the height of the boxcar functions with the corresponding mean removed single-trial LPP amplitudes within each picture category. Similar to the full model described above, the six task-related regressors were convolved with a canonical HRF. Additional six regressors describing subjects’ head movement were also introduced as covariates in the model. The following contrasts were performed based on this model: pleasant vs. neutral, unpleasant vs. neutral, LPP-BOLD coupling within each category (pleasant, neutral, and unpleasant).

Second-level analyses was performed using random effects models based on the statistical maps obtained from the within-subjects analyses to examine reproducible effects across all subjects. For conventional BOLD contrasts, the group level T-maps were thresholded at p < 0.05 (FDR corrected). For LPP-BOLD coupling, because it is derived from the trial to trial variability signal on top of the large picture-evoked response, and this residual variability is generally smaller than the large picture-evoked response and may contain other ongoing brain processes that are not related to the experimental task, the correlation was generally smaller and required a more relaxed statistical threshold. In line with recent studies using EEG-informed fMRI analysis (see, e.g., Debener et al., 2005; Eichele et al., 2005; Bénar et al., 2007; Scheeringa et al., 2011), for LPP-BOLD coupling effects, the group level T-maps were thresholded at p < 0.003 (uncorrected). A cluster-level threshold of k = 5 voxels was further imposed.


ERP analysis

Post-experiment ratings of 12 representative pictures indicate that the subjects correctly distinguished the three categories of pictures (valence: pleasant = 6.5; neutral = 5.3; unpleasant = 2.6; arousal: pleasant = 4.7; neutral = 2.9; unpleasant = 4.0). Figure 1A shows enhanced positivity for both pleasant and unpleasant pictures, relative to neutral pictures, in the grand average ERP at Pz, starting from about 300 ms after picture onset. Since the time interval during which LPP reached a maximum was relatively broad, the LPP amplitude was measured by taking the mean within 300 to 600 ms. A one-way analysis of variance on LPP amplitudes with repeated measures identified a significant picture category-related difference in LPP amplitudes (F(2,20) = 23.11, p < 0.05). As further indicated by the results of post-hoc tests with Bonferroni adjusted significance level, the mean LPP amplitudes for both the pleasant (M = 3.153, SD = 1.733) and unpleasant (M = 3.090, SD = 2.048) pictures were significantly larger than that for the neutral pictures (M = 1.523, SD = 1.684; pleasant vs. neutral: t(10) = 6.26, p < 0.001; unpleasant vs. neutral: t(10) = 5.65, p < 0.001). However, no significant difference was found in LPP amplitudes between the pleasant and unpleasant categories (t(10) = 0.227, p = 0.825). The ERP difference topography further confirmed that the positivity is strongest among parietal channels for both pleasant and unpleasant conditions (Figure 1B and 1C), agreeing with prior ERP studies of emotion and motivation (e.g. Lang and Bradley, 2010). The enhanced positivity was sustained throughout the duration of picture presentation for both pleasant and unpleasant pictures.

Artifact-removed raw EEG data and ASEO-estimated single-trial ERPs at Pz are shown in Figure 2A and 2B for a representative subject. Displayed as raster plots in Figure 2C and Figure 2D, the ASEO-estimated single-trial ERPs improved signal-to-noise ratio, and preserved the trial-by-trial dynamics of the LPP amplitude, which is important because the single-trial LPP amplitudes were used to correlate with BOLD response in subsequent analyses. The validity of the single-trial ERPs can be further supported by averaging the data in Figure 2A and 2B (Xu et al., 2009; Wang et al., 2008; Wang and Ding, 2011). The similarity between ASEO AERP and the original AERP indicated that the algorithm accurately estimated the single-trial ERPs from the raw data (Figure 2E). Figure 3A and 3B display single-trial LPP amplitudes as functions of trial index. From these two figures, one can see that, on average, the single-trial LPP amplitudes for both pleasant and unpleasant pictures are higher than those for the neutral condition, yielding further support for the grand average ERP result in Figure 1. The estimated single-trial LPP amplitudes were used to scale the boxcar functions to examine the relationship between LPP amplitude and BOLD (Figure 3C).

fMRI analysis

The traditional fMRI group level activation maps contrasting 1) pleasant against neutral and 2) unpleasant against neutral picture categories are shown in Figure 4A and 4B. Both pleasant and unpleasant pictures activated the emotion-processing network, encompassing the visual cortices and deep structures. Specifically, relative to neutral pictures, pleasant pictures mainly activated areas in bilateral occipito-temporal junctions, bilateral posterior parietal cortices, medial prefrontal cortex, and left orbital frontal cortex. Other activated areas included fusiform gyrus, lingual gyrus, middle frontal gyrus, supramarginal gyrus, parahippocampal gyrus, and temporal pole (Figure 4A). Unpleasant pictures mainly activated areas such as the bilateral occipito-temporal junctions, bilateral posterior parietal cortices, bilateral ventral lateral prefrontal cortices, left orbital frontal cortex, bilateral amygdalae/hippocampi, insula, and supplementary motor area. Other activated areas included fusiform gyrus, lingual gyrus, supramarginal gyrus, temporal pole, and postcentral cortex (Figure 4B). In general, the activation results agree with a previous report employing a similar experimental protocol (Sabatinelli et al., 2007a), and serve to demonstrate that the quality of the fMRI data is preserved despite the presence of the EEG recording system in the scanner.

Figure 4
Activation maps based on BOLD contrast. A) Pleasant vs. Neutral (P vs. N) condition. B) Unpleasant vs. Neutral (U vs. N) condition. Activations are thresholded at p = 0.05 FDR corrected. PPC: posterior parietal cortex, OFC: orbital frontal cortex, MPFC: ...

Trial-by-trial coupling of LPP and BOLD

To assess the coupling between the LPP amplitude and BOLD, the coefficients for regressors associated with LPP amplitude variations were examined. Using the “full” model, which combined the neutral, pleasant, and unpleasant picture categories as a single regressor to describe the effect of both between- and within-category LPP amplitude variations on BOLD, the single-trial LPP amplitude was positively correlated with evoked BOLD responses in bilateral occipito-temporal junctions, insula, amygdala/hippocampus, temporal poles, and left orbital frontal cortex (Figure 5A).

Figure 5
LPP-BOLD coupling maps where highlighted regions indicate significant correlation between trial-by-trial fMRI response and the corresponding single-trial LPP amplitude. A) Pleasant, Neutral, and Unpleasant combined. B) Pleasant. C) Unpleasant. All maps ...

Using the “within-category” model, which allowed us to examine the coupling between LPP and BOLD within each picture category, it was found that for the neutral condition, no significant coupling between single-trial LPP amplitude and BOLD existed among subjects. For the pleasant condition, the single-trial LPP amplitude was positively correlated with BOLD responses in bilateral occipito-temporal junctions, amygdala, temporal poles, precuneus, right nucleus accumbens (NAcc), medial prefrontal cortex (MPFC), and cerebellum (Figure 5B). For the unpleasant condition, the single-trial LPP amplitude was positively correlated with BOLD responses in bilateral ventral lateral prefrontal cortices, bilateral insula, temporal poles, precuneus, left middle temporal cortex, and left postcentral cortex (Figure 5C). Table 1 listed the MNI coordinates of these regions. It is worth noting that we did not find any structures in which BOLD was negatively correlated with LPP amplitude under the same significance level.

Table 1
Regions showing coupling between LPP amplitude and BOLD


Emotional stimuli evoke a late positive potential (LPP) which is interpreted to signify enhanced attention and visual processing (Bradley, 2009). This signature ERP response is known to be altered in mood disorders and other related psychiatric illnesses (Foti et al., 2010; Leutgeb et al., 2011; Weinberg and Hajcak, 2011; Jaworska et al., 2012; Weymar et al., 2012). Despite the importance of LPP its neural substrate is not clear. ERP source localization is only partly successful (Keil et al., 2002; Sabatinelli et al., 2007a). This problem is addressed here by recording simultaneous EEG and fMRI while subjects viewed IAPS affective pictures. Extracting LPP on a trial-by-trial basis, the overall LPP amplitude variability across three picture categories (pleasant, neutral and unpleasant) was found to be correlated with BOLD responses in an extensive cortical and subcortical network, including visual cortices and deep emotion-processing structures. In addition, consistent with the notion that appetitive and aversive information may engage different neural substrates, the brain areas where BOLD activity was correlated with LPP amplitude during pleasant picture viewing were not the same as those during unpleasant picture viewing.

Methodological considerations

Prior investigation of the association between LPP and evoked BOLD responses relied on recording EEG and fMRI over separate sessions and correlating averaged responses across subjects (Sabatinelli et al., 2007a; Sabatinelli et al., in press). One potential drawback of such an approach is that it is difficult to keep the psychological and biological conditions exactly the same in different recording sessions, and moreover, the correlation between average LPP and BOLD does not reflect their trial-by-trial co-variations and coupling towards individual pictures within each subject. A new technology, simultaneous EEG-fMRI, has become available over the past few years. As has been recently demonstrated (Nagai et al., 2004; Eichele et al., 2005; Debener et al., 2005; Scheeringa et al., 2011), simultaneous EEG-fMRI is capable of overcoming these limitations, and has the potential to allow the interrogation of trial-by-trial associations between the two recording modalities. The present study benefits from another methodological development. Applying a recently proposed single-trial analysis algorithm called ASEO we were able to estimate the LPP amplitude on a trial-by-trial basis. The improved signal-to-noise ratio helps to more fully reveal the brain areas whose BOLD responses are correlated with LPP fluctuations. The ASEO algorithm has been tested in both monkey local field potential data (Wang et al., 2008; Xu et al., 2009) and human scalp EEG data (Fogelson et al., 2008; Wang and Ding, 2011; Liu et al., 2012), and proven useful to address questions arising in a number of contexts, ranging from the proper preprocessing of event-related data for functional connectivity analysis to the temporal dynamics of emotional conditioning.

LPP-BOLD coupling and its theoretical significance

The brain regions where BOLD activities co-vary with single-trial LPP amplitude across three picture categories reflected a joint involvement of the visual system and a network of structures known to be associated with emotional processing. Past source-space modeling of LPP has only been able to identify generators in the visual system, including occipito-temporal, parietal, and inferior temporal cortices (Keil et al., 2002; Sabatinelli et al., 2007a), despite the fact that the amplitude of LPP is closely related to the rated intensity of emotion (Schupp et al., 2000; Keil et al., 2002). In line with a recent study (Sabatinelli et al., in press), the present study extends the prior findings by showing that deep structures such as the insula and the amygdala, along with visual structures, contribute to the generation of LPP and its amplitude modulation. These results provide further evidence supporting the view that emotional pictures naturally attract attentional resources as a result of the engagement of the fundamental motivational system (Cacioppo et al., 1993, 1994; Palomba et al., 1997; Lang et al., 1997, 1998b; Schupp et al., 2000; Pastor et al., 2008; Lang and Bradley, 2010).

The contribution to scalp-recorded potentials by the emotional processing areas may be modulatory and mediated by the visual cortex. It has been hypothesized that when observers view emotionally engaging scenes cortical and deep subcortical structures modulate visual cortex in a re-entrant fashion (Keil et al, 2009; Pessoa and Adolphs, 2010). These structures include the amygdala, insula, and prefrontal cortex (Rotshtein et al., 2001; Phan et al., 2002; Adolphs, 2002; LeDoux, 2003; Zald, 2003; Luo et al., 2007). As evidenced by recent human intracranial studies, amygdala and orbitofrontal cortex show fast responses to the emotional content of stimuli, which would enable them to provide re-entrant feedback to the visual cortices (Oya et al., 2002; Krolak-Salmon et al., 2004), to potentially affect the gain of visual neurons. In addition, the activation of emotion-related BOLD modulations in the amygdala is found to precede that in the fusiform gyrus, the medial occipital gyrus, and the calcarine fissure, consistent with the amygdala’s putative role in initiating the re-entrant interaction (Sabatinelli et al., 2009). Taken together, the positive correlation found in the current study between single-trial LPP amplitude and BOLD activity in the amygdala, insula and areas in prefrontal cortex is supportive of the re-entrant hypothesis of emotional perception.

Category-specific network processing

Restricting to pictures within each valence category revealed category-dependent differences in regions showing LPP and BOLD coupling. For pleasant pictures, LPP amplitude variability was found to be linearly related to BOLD activity in bilateral amygdalae, whereas for unpleasant pictures, this correlation was absent. This finding was further corroborated by a contrast of LPP-BOLD coupling maps between pleasant and unpleasant conditions showing that the amygdala was preferentially engaged in LPP modulation in the pleasant condition. For the unpleasant pictures, the amygdala was found to be activated by a traditional fMRI contrast between unpleasant and neutral conditions, suggesting a rather constant level of amygdala activation on a trial-by-trial basis. Specifically, while the amygdala activity is clearly enhanced by unpleasant pictures as a whole, it does not co-vary with trial-by-trial LPP changes for different pictures within this category. This may reflect a limited response variability of the amygdala for unpleasant scenes or the presence of additional sources of variance that govern the modulation of LPP. A wider selection of unpleasant scenes may help to identify the potential sources of co-variation between electrophysiological measures and amygdala BOLD activity during aversive/defensive engagement.

Whether amygdala responds to pleasant stimuli is debated, although several studies reported amygdala activation during processing of pleasant stimuli, especially when these stimuli have highly arousing erotica content (Lane et al., 1997; O’Doherty et al., 2001; Hamann et al., 2002; Zald, 2003). In the present study, the fact that the amygdala was not activated in a traditional fMRI analysis by contrasting the pleasant with the neutral category may indicate that the overall mean amygdala BOLD response to pleasant pictures was small. Yet, the positive correlation between LPP amplitude and BOLD activities in bilateral amygdalae suggests that the BOLD fluctuations in the amygdala is parametrically related to intensity variations of pleasant emotional content, as measured by the LPP. This finding further demonstrates that combining electrophysiological recordings and functional imaging can yield information not possible with either modality alone.

The LPP-BOLD coupling was found in MPFC and NAcc only for pleasant pictures. Contrasting LPP-BOLD coupling maps between pleasant and unpleasant conditions showed that the MPFC is preferentially coupled with LPP in the pleasant condition. In addition, the MPFC was activated by contrasting pleasant with neutral pictures, but the same region was not activated when comparing unpleasant and neutral pictures. NAcc and MPFC are densely interconnected (Ferry et al., 2000; Roberts et al., 2007) and often show correlated activities in human reward studies (Knutson et al., 2003; Rogers et al., 2004), leading to the view that both NAcc and MPFC are part of the human reward system mediating appetitive behaviors. Several studies have reported involvement of both structures in the perception of pleasant emotional stimuli including attractive faces, romance, and erotica (Aharon et al., 2001; Karama et al., 2002; O’Doherty et al., 2003; David et al., 2005; Ferretti et al., 2005; Sabatinelli et al. 2007b) as well as in vivid imagery of pleasant scenes (Costa et al., 2010). Hence, the observed positive correlation between LPP amplitude and key structures in the reward system may reflect the contribution to the cortical potential by the appetitive system.

For unpleasant pictures, the LPP was correlated with BOLD in insula and adjacent temporal and ventrolateral prefrontal cortices (VLPFC). The same regions were found to be active when contrasting unpleasant with neutral pictures. It has been shown repeatedly that the human insula is involved in tasks that challenge the representation of bodily states as well as processing of emotions (Craig, 2009; Gu et al., 2010; Fan et al., 2011), especially for aversive emotions such as disgust and threat (Phillips et al., 1997, 1998; Adolphs, 2002; Straube and Miltner, 2011). Reliable co-variation of the insula and peri-insula with the LPP during aversive perception demonstrates that these structures contribute to the modulation of cortical potential during aversive events. It also suggests that the aversive/defensive circuitry involved in processing unpleasant pictures is not engaged in an all-or-none fashion, but varies parametrically as a function of aversive motivation, indexed by the LPP amplitude. It is worth noting that insula was not activated when LPP-BOLD coupling maps were contrasted between pleasant and unpleasant conditions. In light of the finding that insula is activated during viewing of highly arousing pleasant stimuli (e.g. erotica; Karama et al., 2002), this may suggest that for pleasant pictures, insula is engaged in LPP modulation but the degree of modulation did not reach the level of statistical significance.

Finally, for both pleasant and unpleasant pictures, BOLD activity in regions within midline parietal cortex is linearly correlated with the LPP. For pleasant pictures, the region that was most correlated with LPP amplitude was within the precuneus, whereas for unpleasant pictures, such correlation occurred in more ventral regions, particularly the posterior cingulate cortex and precuneus. The involvement of these parietal regions is in line with electrophysiological data and with the theoretical notion of “motivated attention” in which features of emotionally arousing scenes attract perceptual processing resources (Keil et al., 2002; Keil et al., in press).


This research was supported by NIH grant MH097320.


  • Adolphs R. Neural systems for recognizing emotion. Curr Opin in Neurobiol. 2002;12:169–177. [PubMed]
  • Aharon I, Etcoff N, Ariely D, Chabris CF, O’Connor E, Breiter HC. Beautiful faces have variable reward value: fMRI and behavioral evidence. Neuron. 2001;32:537–551. [PubMed]
  • Allen PJ, Polizzi G, Krakow K, Fish DR, Lemieux L. Identification of EEG events in the MR scanner: the problem of pulse artifact and a method for its subtraction. Neuroimage. 1998;8:229–239. [PubMed]
  • Allen PJ, Josephs O, Turner R. A method for removing imaging artifact from continuous EEG recorded during functional MRI. Neuroimage. 2000;12:230–239. [PubMed]
  • Belouchrani A, Abed-Meraim K, Cardoso JF, Moulines E. Second order blind separation of temporally correlated sources. Proc Int Conf on Digital Sig Proc. 1993:346–351.
  • Bradley MM, Lang PJ. Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry. 1994;25:49–59. [PubMed]
  • Bradley MM, Sabatinelli D, Lang PJ, Fitzsimmons JR, King W, Desai P. Activation of the visual cortex in motivated attention. Behav Neurosci. 2003;117:369–380. [PubMed]
  • Bradley MM. Natural selective attention: orienting and emotion. Psychophysiol. 2009;46:1–11. [PMC free article] [PubMed]
  • Breiter H, Etcoff N, Whalen P, Kennedy W, Rauch S, Buckner R, Strauss M, Hyman S, Rosen B. Response and habituation of the human amygdala during visual processing of facial expression. Neuron. 1996;17:875–887. [PubMed]
  • Cacioppo JT, Crites SL, Berntson GG, Coles MGH. If attitudes affect how stimuli are processed, should they not affect the event-related brain potential? Psychol Sci. 1993;4:108–112.
  • Cacioppo JT, Crites SL, Gardner WL, Bernston GG. Bioelectrical echoes from evaluative categorizations: I. a late positive brain potential that varies as a function of trait negativity and extremity. J Pers Soc Psychol. 1994;67:115–125. [PubMed]
  • Chen Y, Bressler SL, Knuth KH, Truccolo WA, Ding M. Stochastic modeling of neurobiological time series: power, coherence, Granger causality, and separation of evoked responses from ongoing activity. Chaos. 2006;16:026113-1–8. [PubMed]
  • Costa VD, Lang PJ, Sabatinelli D, Versace F, Bradley MM. Emotional imagery: assessing pleasure and arousal in the brain’s reward circuitry. Hum Brain Mapp. 2010;31:1446–1457. [PMC free article] [PubMed]
  • Craig (Bud) AD. How do you feel – now? The anterior insula and human awareness. Nat Rev Neurosci. 2009;10:59–70. [PubMed]
  • Cuthbert BN, Schupp HT, Bradley MM, Birbaumer N, Lang PJ. Brain potentials in affective picture processing: covariation with autonomic arousal and affective report. Biol Psychol. 2000;52:95–111. [PubMed]
  • David SP, Munafò MR, Johansen-Berg H, Smith SM, Rogers RD, Matthews PM, Walton RT. Ventral striatum/nucleus accumbens activation to smoking-related pictorial cues in smokers and nonsmokers: a functional magnetic resonance imaging study. Biol Psychiatry. 2005;58:488–494. [PubMed]
  • Debener S, Ullsperger M, Siegel M, Fiehler K, von Cramon DY, Engel AK. Trial-by-trial coupling of concurrent electroencephalogram and functional magnetic resonance imaging identifies the dynamics of performance monitoring. J Neurosci. 2005;25:11730–11737. [PubMed]
  • Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods. 2004;134:9–21. [PubMed]
  • Eichele T, Specht K, Moosmann M, Jongsma ML, Quiroga RQ, Nordby H, Hugdahl K. Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci U S A. 2005;102:17798–17803. [PMC free article] [PubMed]
  • Fan J, Gu X, Liu X, Guise KG, Park Y, Martin L, De Marchena A, Tang CY, Minzenberg MJ, Hof PR. Involvement of the anterior cingulate and frontoinsular cortices in rapid processing of salient facial emotional information. Neuroimage. 2011;54:2539–2546. [PMC free article] [PubMed]
  • Ferretti A, Caulo M, Del Gratta C, Di Matteo R, Merla A, Montorsi F, Pizzella V, Pompa P, Rigatti P, Rossini PM, Salonia A, Tartaro A, Romani GL. Dynamics of male sexual arousal: distinct components of brain activation revealed by fMRI. Neuroimage. 2005;26:1086–1096. [PubMed]
  • Ferry AT, Öngür D, An X, Price JL. Prefrontal cortical projections to the striatum in macaque monkeys: evidence for an organization related to prefrontal networks. J Comp Neurol. 2000;425:447–470. [PubMed]
  • Fogelson N, Wang X, Lewis JB, Kishiyama MM, Ding M, Knight RT. Multimodal effects of local context on target detection: evidence from P3b. J Cogn Neurosci. 2008;21:1680–1692. [PMC free article] [PubMed]
  • Foti D, Olvet DM, Klein DN, Hajcak G. Reduced electrocortical response to threatening faces in major depressive disorder. Depress Anxiety. 2010;27:813–820. [PubMed]
  • Gu X, Liu X, Guise KG, Naidich TP, Hof PR, Fan J. Functional dissociation of the frontoinsular and anterior cingulate cortices in empathy for pain. J Neurosci. 2010;30:3739–3744. [PMC free article] [PubMed]
  • Hamann SB, Ely TD, Hoffman JM, Kilts CD. Ecstasy and agony: activation of the human amygdala in positive and negative emotion. Psychol Sci. 2002;13:135–141. [PubMed]
  • Jaworska N, Thompson A, Shah D, Fisher D, Ilivitsky V, Knott V. Acute tryptophan depletion effects on the vertex and late positive potentials to emotional faces in individuals with a family history of depression. Neuropsychobiology. 2012;65:28–40. [PubMed]
  • Karama S, Lecours AR, Leroux J, Bourgouin P, Beaudoin G, Joubert S, Beauregard M. Areas of brain activation in males and females during viewing of erotic film excerpts. Hum Brain Mapp. 2002;16:1–13. [PubMed]
  • Keil A, Bradley MM, Hauk O, Rockstroh B, Elbert T, Lang PJ. Large-scale neural correlates of affective picture processing. Psychophysiology. 2002;39:641–649. [PubMed]
  • Keil A, Sabatinelli D, Ding M, Lang PJ, Ihssen N, Heim S. Re-entrant projections modulate visual cortex in affective perception: directional evidence from Granger causality analysis. Hum Brain Mapp. 2009;30:532–540. [PMC free article] [PubMed]
  • Keil A, Costa V, Smith JC, Sabatinelli D, McGinnis EM, Bradley MM, Lang PJ. Tagging cortical networks in emotion: a topographical analysis. Hum Brain Mapp. doi: 10.1002/hbm.21413. in press. [PMC free article] [PubMed] [Cross Ref]
  • Klemm M, Haueisen J, Ivanova G. Independent component analysis: comparison of algorithms for the investigation of surface electrical brain activity. Med Biol Eng Comput. 2009;47:413–423. [PubMed]
  • Knutson B, Fong GW, Bennett SM, Adams CM, Hommer D. A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: characterization with rapid event-related fMRI. Neuroimage. 2003;18:263–272. [PubMed]
  • Krolak-Salmon P, Hénaff M, Vighetto A, Bertrand O, Mauguière F. Early amygdala reaction to fear spreading in occipital, temporal, and frontal cortex: a depth electrode ERP study in human. Neuron. 2004;42:665–676. [PubMed]
  • Lane RD, Reiman EM, Bradley MM, Lang PJ, Ahern GL, Davidson RJ, Schwartz GE. Neuroanatomical correlates of pleasant and unpleasant emotion. Neuropsychologia. 1997;35:1437–1444. [PubMed]
  • Lang PJ, Simons RF, Balaban MT. Attention and orienting: sensory and motivational processes. London: Psychology Press; 1997.
  • Lang PJ, Bradley MM, Fitzsimmons JR, Cuthbert BN, Scott JD, Moulder B, Nangia V. Emotional arousal and activation of the visual cortex: an fMRI analysis. Psychophysiology. 1998a;35:199–210. [PubMed]
  • Lang PJ, Bradley MM, Cuthbert BN. Emotion, motivation, and anxiety: brain mechanisms and psychophysiology. Biol Psychiatry. 1998b;44:1248–1263. [PubMed]
  • Lang PJ, Bradley MM, Cuthbert BN. International affective picture system (IAPS): affective ratings of pictures and instruction manual Technical Report A-8. University of Florida; Gainesville, FL: 2008.
  • Lang PJ, Bradley MM. Emotion and the motivational brain. Biol Psychol. 2010;84:437–450. [PMC free article] [PubMed]
  • LeDoux J. The emotional brain, fear, and the amygdala. Cell Mol Neurobiol. 2003;23:727–738. [PubMed]
  • Leutgeb V, Schafer A, Schienle A. Late cortical positivity and cardiac responsitivity in female dental phobics when exposed to phobia-relevant pictures. Int J Psychophysiol. 2011;79:410–416. [PMC free article] [PubMed]
  • Liu Y, Keil A, Ding M. Effects of emotional conditioning on early visual processing: temporal dynamics revealed by ERP single-trial analysis. Hum Brain Mapp. 2012;33:909–919. [PubMed]
  • Luo Q, Holroyd T, Jones M, Hendler T, Blair J. Neural dynamics for facial threat processing as revealed by gamma band synchronization using MEG. Neuroimage. 2007;34:839–847. [PMC free article] [PubMed]
  • Nagai Y, Critchley HD, Featherstone E, Fenwick PB, Trimble MR, Dolan RJ. Brain activity relating to the contingent negative variation: an fMRI investigation. Neuroimage. 2004;21:1232–1241. [PubMed]
  • Norris CJ, Chen EE, Zhu DC, Small SL, Cacioppo JT. The interaction of social and emotional processes in the brain. JCogn Neurosci. 2004;16:1818–1829. [PubMed]
  • O’Doherty J, Rolls ET, Francis S, Bowtell R, McGlone F. Representation of pleasant and aversive taste in the human brain. J Neurophysiol. 2001;85:1315–1321. [PubMed]
  • O’Doherty J, Winston J, Critchley H, Perrett D, Burt DM, Dolan RJ. Beauty in a smile: the role of medial orbitofrontal cortex in facial attractiveness. Neuropsychologia. 2003;41:147–155. [PubMed]
  • Oya H, Kawasaki H, Howard MA, Adolphs R. Electrophysiological responses in the human amygdala discriminate emotion categories of complex visual stimuli. J Neurosci. 2002;22:9502–9512. [PubMed]
  • Palomba D, Angrilli A, Mini A. Visual evoked potentials, heart rate responses and memory to emotional pictorial stimuli. Int J Psychophysiol. 1997;27:55–67. [PubMed]
  • Pastor MC, Bradley MM, Löw A, Versace F, Moltó J, Lang PJ. Affective picture perception: emotion, context, and the late positive potential. Brain Res. 2008;1189:145–151. [PMC free article] [PubMed]
  • Pessoa L, Adolphs R. Emotion processing and the amygdala: from a ‘low road’ to ‘many roads’ of evaluating biological significance. Nat Rev Neurosci. 2010;11:773–783. [PMC free article] [PubMed]
  • Phan KL, Wager T, Taylor SF, Liberzon I. Functional neuroanatomy of emotion: a meta-analysis of emotion activation studies in PET and fMRI. Neuroimage. 2002;16:331–348. [PubMed]
  • Phillips ML, Young AW, Senior C, Brammer M, Andrew C, Calder AJ, Bullmore ET, Perrett DI, Rowland D, Williams SC, Gray JA, David AS. A specific neural substrate for perceiving facial expressions of disgust. Nature. 1997;389:495–498. [PubMed]
  • Phillips ML, Young AW, Scott SK, Calder AJ, Andrew C, Giampietro V, Williams SCR, Bullmore ET, Brammer M, Gray JA. Neural Responses to Facial and Vocal Expressions of Fear and Disgust. Proc R Soc of Lond B Biol Sci. 1998;265:1809–1817. [PMC free article] [PubMed]
  • Roberts AC, Tomic DL, Parkinson CH, Roeling TA, Cutter DJ, Robbins TW, Everitt BJ. Forebrain connectivity of the prefrontal cortex in the marmoset monkey (callithrix jacchus): an anterograde and retrograde tract-tracing study. J Comp Neurol. 2007;502:86–112. [PubMed]
  • Rogers RD, Ramnani N, Mackay C, Wilson JL, Jezzard P, Carter CS, Smith SM. Distinct portions of anterior cingulate cortex and medial prefrontal cortex are activated by reward processing in separable phases of decision-making cognition. Biol Psychiatry. 2004;55:594–602. [PubMed]
  • Rotshtein P, Malach R, Hadar U, Graif M, Hendler T. Feeling or features: different sensitivity to emotion in high-order visual cortex and amygdala. Neuron. 2001;32:747–757. [PubMed]
  • Sabatinelli D, Bradley MM, Fitzsimmons JR, Lang PJ. Parallel amygdala and inferotemporal activation reflect emotional intensity and fear relevance. Neuroimage. 2005;24:1265–1270. [PubMed]
  • Sabatinelli D, Lang PJ, Keil A, Bradley MM. Emotional perception: correlation of functional MRI and event-related potentials. Cereb Cortex. 2007a;17:1085–1091. [PubMed]
  • Sabatinelli D, Bradley MM, Lang PJ, Costa VD, Versace F. Pleasure rather than salience activates human nucleus accumbens and medial prefrontal cortex. J Neurophysiol. 2007b;98:1374–1379. [PubMed]
  • Sabatinelli D, Lang PJ, Bradley MM, Costa VD, Keil A. The timing of emotional discrimination in human amygdala, inferotemporal, and occipital cortex. J Neurosci. 2009;29:14864–14868. [PMC free article] [PubMed]
  • Sabatinelli D, Keil A, Frank DW, Lang PJ. Emotion perception: correspondence of early and late event-related potentials with cortical and subcortical functional MRI. Biol Psychol. in press doi: http://dx.doi.org/10.1016/j.biopsycho.2012.04.005. [PMC free article] [PubMed]
  • Scheeringa R, Mazaheri A, Bojak I, Norris DG, Kleinschmidt A. Modulation of visually evoked cortical fMRI responses by phase of ongoing occipital alpha oscillations. J Neurosci. 2011;31:3813–3820. [PubMed]
  • Schupp HT, Cuthbert BN, Bradley MM, Cacioppo JT, Ito T, Lang PJ. Affective picture processing: the late positive potential is modulated by motivational relevance. Psychophysiol. 2000;37:257–261. [PubMed]
  • Straube T, Miltner WHR. Attention to aversive emotion and specific activation of the right insula and right somatosensory cortex. Neuroimage. 2011;54:2534–2538. [PubMed]
  • Tang AC, Sutherland MT, McKinney CJ. Validation of SOBI components from high-density EEG. Neuroimage. 2005;25:539–553. [PubMed]
  • Vanderperren K, De Vos M, Ramautar JR, Novitskiy N, Mennes M, Assecondi S, Vanrumste B, Stiers P, Van den Bergh BRH, Wagemans J, Lagae L, Sunaert S, Van Huffel S. Removal of BCG artifacts from EEG recordings inside the MR scanner: A comparison of methodological and validation-related aspects. Neuroimage. 2010;50:920–934. [PubMed]
  • Wang X, Chen Y, Ding M. Estimating Granger causality after stimulus onset: a cautionary note. Neuroimage. 2008;41:767–776. [PMC free article] [PubMed]
  • Wang X, Ding M. Relation between P300 and event-related theta-band synchronization: a single-trial analysis. Clin Neurophysiol. 2011;122:916–924. [PubMed]
  • Weinberg A, Hajcak G. Electrocortical evidence for vigilance-avoidance in generalized anxiety disorder. Psychophysiol. 2011;48:842–851. [PubMed]
  • Weymar M, Schwabe L, Löw A, Hamm AO. Stress sensitizes the brain: increased processing of unpleasant pictures after exposure to acute stress. J Cogn Neurosci. 2012;24(7):1511–1518. [PubMed]
  • Xu L, Stoica P, Li J, Bressler SL, Shao X, Ding M. ASEO: a method for the simultaneous estimation of single-trial event-related potentials and ongoing brain activities. IEEE Trans Biomed Eng. 2009;56:111–121. [PubMed]
  • Zald DH. The human amygdala and the emotional evaluation of sensory stimuli. Brain Res Rev. 2003;41:88–123. [PubMed]
PubReader format: click here to try


Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


  • MedGen
    Related information in MedGen
  • PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...