NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Murray MM, Wallace MT, editors. The Neural Bases of Multisensory Processes. Boca Raton (FL): CRC Press/Taylor & Francis; 2012.

Cover of The Neural Bases of Multisensory Processes

The Neural Bases of Multisensory Processes.

Show details

Chapter 33Convergence of Auditory, Visual, and Somatosensory Information in Ventral Prefrontal Cortex



Our ability to recognize and integrate auditory and visual stimuli is the basis for many cognitive processes but is especially essential in meaningful communication. Although many brain regions contribute to recognition and integration of sensory signals, the frontal lobes both receive a multitude of afferents from sensory association areas and have influence over a wide region of the nervous system to govern behavior. Furthermore, the frontal lobes are special in that they have been associated with language processes, working memory, planning, and reasoning, which all depend on the recognition and integration of a vast network of signals. Research has also shown that somatosensory afferents reach the frontal lobe and that in specific regions single cells encode somatosensory signals. In this chapter we will focus on the ventrolateral prefrontal cortex (VLPFC), also known as the inferior convexity in some studies, and describe the connectivity of VLPFC with auditory, visual, and somatosensory cortical areas. This connectivity provides the circuitry for prefrontal responses to these stimuli, which will also be described from previous research. The potential function of combined auditory, visual, and somatosensory inputs will be described with regard to communication and object recognition.


The prefrontal cortex receives a widespread array of afferents from cortical and subcortical areas. These include sensory, motor, and association cortex and thalamic nuclei. The extensive innervation of the frontal lobe is nonetheless organized, and particular circuits have been investigated and carefully characterized leading to a better understanding of frontal lobe function based on this connectivity. Although many areas of the frontal lobe receive converging inputs, we will focus on the multisensory innervation of the VLPFC.

33.2.1. Visual Projections to Ventral Prefrontal Cortex

Much of what we know about the cellular functions of the primate prefrontal cortex is based on the processing of visual information. Thus, it is not surprising that many studies have examined projections from visual association cortex to the primate prefrontal cortex. With regard to the frontal lobe, early anatomical studies by Helen Barbas, Deepak Pandya, and their colleagues (Barbas 1988; Barbas and Mesulam 1981; Barbas and Pandya 1989; Chavis and Pandya 1976) examined the innervation of the entire prefrontal mantle by visual association areas. These studies denoted some specificity in the innervation of dorsal, ventral, and medial prefrontal cortices. Barbas was among the first to note that basoventral prefrontal cortices were more strongly connected with extrastriate ventral visual areas, which have been implicated in pattern recognition and feature discrimination, whereas medial and dorsal prefrontal cortices are more densely connected with medial and dorsolateral occipital and parietal areas, which are associated with visuospatial functions (Barbas 1988). This dissociation was echoed by Bullier and colleagues (1996), who found some segregation of inputs to PFC when paired injections of tracers were placed into temporal and parietal visual processing regions. In their study, the visual temporal cortex projected mainly to area 45, located ventrolaterally in the PFC, whereas the parietal cortex sent projections to both ventrolateral PFC (area 45) and dorsolateral PFC (DLPC) (areas 8a and 46) (Schall et al. 1995; Bullier et al. 1996). Tracing and lesion studies by Ungerleider et al. (1989) showed that area TE projected specifically to three ventral prefrontal targets including the anterior limb of the arcuate sulcus (area 45), the inferior convexity just ventral to the principal sulcus (areas 46v and 12) and within the lateral orbital cortex (areas 11, 12o). These projections are via the uncinate fasciculus (Ungerleider et al. 1989). The selective connectivity of ventrolateral PFC areas 12 and 45, which contain object- and face-selective neurons (O’Scalaidhe et al. 1997, 1999; Wilson et al. 1993), with inferotemporal areas TE and TEO was specifically documented by Webster and colleagues (1994). Comparison of TE and TEO connectivity in their study revealed a number of important differences, including the finding that it is mainly area TE that projects to ventrolateral PFC and orbitofrontal areas 11, 12, and 13. These orbital regions have also been associated with visual object functions.

33.2.2. Auditory Projections to Prefrontal Cortex

In early anatomical studies, lesion/degeneration techniques were used to reveal projections from the caudal superior temporal gyrus (STG) to the periprincipalis, periarcuate, and inferior convexity regions of the frontal lobe and from the middle and rostral STG to rostral principalis and orbital regions (Pandya et al. 1969; Pandya and Kuypers 1969; Jones and Powell 1970; Chavis and Pandya 1976). Studies with anterograde and retrograde tracers that were aimed at determining the overall connectivity between the temporal and frontal lobes brought additional specificity (Pandya and Sanides 1973; Galaburda and Pandya 1983; Barbas 1988; Barbas and Mesulam 1981; Barbas and Pandya 1989; Cipolloni and Pandya 1989). Studies of the periprincipalis and arcuate region showed that the anterior and middle aspects of the principal sulcus, including areas 9, 10, and 46, were connected with the middle and caudal STG (Barbas and Mesulam 1985; Petrides and Pandya 1988), whereas area 8 receives projections from mostly caudal STG (Barbas and Mesulam 1981; Petrides and Pandya 1988). Latter studies confirmed the connection of the posterior STG with areas 46, dorsal area 8, and the middle STG with rostral–dorsal 46 and 10, area 9, and area 12 (Petrides and Pandya 1988; Barbas 1992).

Connections of ventrolateral prefrontal areas with auditory association cortex have been considered by several groups. Cytoarchitectonic analysis of the VLPFC suggested that the region labeled by Walker as area 12 in the macaque monkey has similar characteristics as that of human area 47, and was thus renamed in the macaque as area 47/12 by Petrides and Pandya (1988). Analysis of the connections of areas 45 and 47/12 in the VLPFC has shown that they receive innervation from the STG, the inferotemporal cortex, and from multisensory regions within the superior temporal sulcus. Combining physiological recording with anatomical tract tracing, Romanski and colleagues (1999) analyzed the connections of physiologically defined areas of the belt and parabelt auditory cortex and determined that the projections to prefrontal cortex are topographically arranged so that rostral and ventral prefrontal cortex receive projections from the anterior auditory association cortex (areas AL and anterior parabelt), whereas caudal prefrontal regions are innervated by the posterior auditory cortex (areas CL and caudal parabelt; Figure 33.1). Together with recent auditory physiological recordings from the lateral belt (Tian et al. 2001) and from the prefrontal cortex (Romanski and Goldman-Rakic 2002; Romanski et al. 2005), these studies suggest that separate auditory streams originate in the anterior and posterior auditory cortex and target anterior-ventrolateral object, and dorsolateral spatial domains in the frontal lobe, respectively (Romanski 2007), similar to those of the visual system. Ultimately, this also implies that auditory and visual afferents target similar regions of dorsolateral and ventrolateral prefrontal cortex (Price 2008). The convergence of auditory and visual ventral stream inputs to the same VLPFC domain implies that they may be integrated and combined to serve a similar function, that of object recognition.

FIGURE 33.1. Innervation of prefrontal cortex by auditory belt and parabelt injections.


Innervation of prefrontal cortex by auditory belt and parabelt injections. (a) Projections from anterior auditory cortex to ventrolateral prefrontal cortex (VLPFC) are shown with black arrows and projections from caudal auditory cortex to dorsolateral prefrontal (more...)

33.2.3. Somatosensory Connections with Prefrontal Cortex

Previous studies have noted connections between the principal sulcus and inferior convexity with somatosensory cortical areas (Barbas and Mesulam 1985), most notably SII and 7b (Cavada and Goldman-Rakic 1989; Preuss and Goldman-Rakic 1989; Carmichael and Price 1995). Injections that included the ventral bank of the principal sulcus and the anterior part of area 12 resulted in strong labeling of perisylvian somatic cortex including the second somatosensory area (SII) and insular cortex (Preuss and Goldman-Rakic 1989). Anterograde studies have confirmed this showing that area SII has a projection to the inferior convexity and principal sulcus region of the macaque frontal lobe (Cipolloni and Pandya 1999). This region of the PFC overlaps with the projection field of auditory association cortex and visual extrastriate cortex.


33.3.1. Visual Responses

In 1993, Wilson et al. (1993) published a groundbreaking study revealing a physiological dissociation between dorsal and ventral prefrontal cortex (Figure 33.2). In this study, the authors showed that DLPFC cells responded in a spatial working memory task, with single cells exhibiting selective cue and delay activity for discrete eccentric locations. In the same animals, it was shown that electrode penetrations into VLPFC regions, which included the expanse of the inferior convexity (areas 47/12 lateral, 12 orbital, and area 45), revealed single-unit responses to pictures of objects and faces. These VLPFC cells did not respond in the spatial working memory task but did respond in an object-fixation task and an object-conditional association task. Further electrophysiological and neuroimaging studies have demonstrated face selectivity in this same area of VLPFC (O’Scalaidhe et al. 1997, 1999; Tsao et al. 2008), confirming this functional domain separation.

FIGURE 33.2. Lateral brain schematic of visual pathways in nonhuman primate showing dorsal–spatial and ventral–object visual streams that terminate in DLPFC and VLPFC, respectively.


Lateral brain schematic of visual pathways in nonhuman primate showing dorsal–spatial and ventral–object visual streams that terminate in DLPFC and VLPFC, respectively. Wilson et al. (1993) showed that neurons in DLPFC (black) respond (more...)

Although these studies were the first to demonstrate an electrophysiological dissociation between DLPFC and VLPFC, they were not the first to suggest a functional difference and to show the preference for object as opposed to spatial processing in the ventral prefrontal cortex. An earlier study by Mishkin and Manning (1978) showed that lesions of the VLPFC in nonhuman primates interfere with the processing of nonspatial information, including color and form. These ventral prefrontal lesions had a severe and lasting impairment on the performance of three nonspatial tasks, whereas lesions of the principal sulcus had only a transient effect (Mishkin and Manning 1978). Just a few years earlier, Passingham (1975) had also suggested a dissociation between dorsal and ventral PFC. In their study, rhesus monkeys were trained on delayed color matching task and delayed spatial alternation tasks. Lesions of the VLPFC resulted in an impairment only in the delayed color matching task, whereas lesions of the DLPFC only impaired the delayed spatial alternation task. These results, like the Wilson et al. study two decades later, demonstrated a double dissociation of dorsal and ventral PFC and suggested a role in the processing of object features and recognition for the VLPFC.

Further analysis of the properties of cells in the VLPFC was done by Joaquin Fuster and colleagues. In their electrophysiological analysis of ventral prefrontal neurons, they showed that single cells are responsive to simple and complex visual stimuli presented at the fovea (Pigarev et al. 1979; Rosenkilde et al. 1981). The foveal receptive field properties of these cells had first been shown in studies by Suzuki and Azuma (1977), who examined receptive field properties of neurons across the expanse of lateral prefrontal cortex. The receptive fields of neurons in DLPFC were found to lie outside the fovea and favored the contralateral visual field, whereas neurons below the principal sulcus in areas 12/47 and 45 were found to be driven best by visual stimuli shown within the fovea (Suzuki and Azuma 1977). Hoshi et al. (2000) examined the spatial distribution of location-selective and shape-selective neurons during cue, delay, and response periods, and found more location-selective neurons in the posterior part of the lateral PFC, whereas more shape-selective neurons were found in the anterior part, corresponding to area 12/47. Ninokura et al. (2004) found that cells that responded selectively to the physical properties (color and shape) of objects were localized to the VLPFC. These various studies fostered the notion that visual neurons in VLPFC were tuned to nonspatial features including color, shape, and type of object, and had receptive fields representing areas in and around the fovea.

Finally, studies from Goldman-Rakic and colleagues further demonstrated that neurons in the VLPFC were not only responsive to object features, but that some neurons were highly specialized and were face-selective (Wilson et al. 1993; O’Scalaidhe et al. 1997, 1999). The face-selective neurons were found in several discrete regions including an anterior location that appears to be area 12/47, a posterior, periarcuate, location within area 45, and some penetrations into the orbital cortex also yielded face cells. These single-unit responses were further corroborated with functional magnetic resonance imaging (fMRI) data by Tsao and colleagues (2008). In their fMRI, study they showed that three loci within the VLPFC of macaques were selectively activated by faces (Tsao et al. 2008; Figure 33.3). These three locations correspond roughly to the same anterior, posterior, and ventral/orbital locations that O’Scalaidhe et al. (1997, 1999) mapped as being face-responsive in their single-unit recording studies. Demonstration by both methods of visual responsiveness and face selectivity substantiates the notion that the VLPFC is involved in object and face processing.

FIGURE 33.3. (See color insert.


(See color insert.) Activation of macaque prefrontal cortex by faces in the study of Tsao et al. (2008). Shown here are two coronal sections showing “face patches” in VLPFC (activations are yellow), delineated with white arrows. (Reprinted (more...)

33.3.2. Auditory Responses and Function in Prefrontal Cortex

The frontal lobe has long been linked with complex auditory function through its association with language functions and Broca’s area. What we hear and say seems to be important to frontal lobe neurons. In the human brain, the posterior aspects of Broca’s area are thought to be especially involved in the phonetic and motor control of speech, whereas more anterior regions have been shown to be activated during semantic processing, comprehension, and auditory working memory (Zatorre et al. 1992; Paulesu et al. 1993; Buckner et al. 1995; Demb et al. 1995; Fiez et al. 1996; Stromswold et al. 1996; Cohen et al. 1997; Gabrieli et al. 1998; Stevens et al. 1998; Price 1998; Posner et al. 1999; Gelfand and Bookheimer 2003). Examination of prefrontal auditory function in nonhuman primates has not received as much attention as visual prefrontal function. A few studies have investigated the effects of large prefrontal lesions on behavioral task performance of auditory discrimination or mnemonic processing of complex acoustic stimuli. In each of these four studies, relatively large lesions of the lateral PFC were shown to cause an impairment in an auditory go/no-go task for food reward (Weiskrantz and Mishkin 1958; Gross and Weiskrantz 1962; Gross 1963; Goldman and Rosvold 1970). This was taken as evidence of the PFC’s involvement in modality-independent processing especially in tasks requiring inhibitory control (Weiskrantz and Mishkin 1958).

Despite the localization of language function in the human brain to ventral frontal lobe regions and the demonstration that lesions of lateral PFC in nonhuman primates interferes with auditory discrimination, single-cell responses to acoustic stimuli have only been sporadically noted in the frontal lobes of Old and New World monkeys (Benevento et al. 1977; Bodner et al. 1996; Newman and Lindsley 1976; Tanila et al. 1992, 1993; Wollberg and Sela 1980). However, a close look at these studies reveals that few of the studies sampled neurons in ventrolateral and orbitofrontal regions. Most recordings in the past have been confined to the dorsolateral surface of the frontal lobe where projections from secondary and tertiary auditory cortices are sparse. Only one early study recorded from the lateral orbital region in the macaque cortex and found both auditory and visual responses to simple visual flashes and to broadband auditory clicks (Benevento et al. 1977). Furthermore, none of the studies tested neurons systematically with naturalistic and species-relevant acoustic stimuli. Recent approaches to frontal lobe auditory function have utilized naturalistic stimuli, including species-specific vocalizations and have extended the area of investigation to orbital and ventral PFC regions.

33.3.3. Prefrontal Responses to Vocalizations

After establishing the area of the prefrontal cortex that receive dense afferents from early-auditory cortical regions (Romanski et al. 1999a, 1999b), Romanski and Goldman-Rakic, revealed a discrete auditory responsive region in the macaque VLPFC (Romanski and Goldman-Rakic 2002). This VLPFC region has neurons that respond to complex acoustic stimuli, including species-specific vocalizations, and lies adjacent to the object- and face-selective region proposed previously (O’Scalaidhe et al. 1997, 1999; Wilson et al. 1993). Although VLPFC auditory neurons have not been thoroughly tested for directional selectivity, further examination has suggested that they encode complex auditory features and thus respond to complex stimuli on the basis of similar acoustic features (Romanski et al. 2005; Averbeck and Romanski 2006).

Use of a large library of rhesus macaque vocalizations to test auditory selectivity in prefrontal neurons has shown that VLPFC neurons are robustly responsive to species-specific vocalizations (Romanski et al. 2005). A cluster analysis of these vocalization responses did not show a clustering of responses to vocalizations depicting similar functions (i.e., food calls) but demonstrated that neurons tend to respond to multiple vocalizations with similar acoustic morphology (Romanski et al. 2005; Figure 33.4). Neuroimaging in rhesus monkeys has revealed a small ventral prefrontal locus that was active during presentation of complex acoustic stimuli including vocalizations (Poremba and Mishkin 2007). Additional electrophysiological recording studies by Cohen and colleagues have suggested that prefrontal auditory neurons may also participate in the categorization of species-specific vocalizations (Gifford et al. 2005). These combined data are consistent with a role for VLPFC auditory neurons in a ventral auditory processing stream that analyzes the features of auditory objects including vocalizations.

FIGURE 33.4. A vocalization responsive cell in VLPFC.


A vocalization responsive cell in VLPFC. (a) Response to 10 vocalization exemplars is shown in raster/spike density plots. Strongest response was to submissive scream vocalizations and copulation scream vocalizations, which are similar in acoustic features as (more...)

Evidence for object-based auditory processing in the ventral frontal lobe of the human brain is suggested by neuroimaging studies that have detected activation in the VLPFC not only by speech stimuli but by nonspeech and music stimuli (Belin et al. 2000; Binder et al. 2000; Scott et al. 2000; Zatorre et al. 2004) in auditory recognition tasks and voice recognition tasks (Fecteau et al. 2005). The localization of an auditory object processing stream in the human brain to the very same ventral prefrontal region in a nonhuman primate suggests a functional similarity between this area and human language-processing regions located in the inferior frontal gyrus (Deacon 1992; Romanski and Goldman-Rakic 2002).

33.3.4. Somatosensory Responses

Fewer studies have examined the responses of prefrontal neurons to somatosensory stimuli. This may be partly attributable to the lack of an easy association between a known human function for somatosensory stimuli and the frontal lobes, as there is for language and audition. One group, however, has demonstrated responses to somatosensory stimuli in single lateral prefrontal neurons. Recordings in the prefrontal cortex were made while macaque monkeys performed a somatosensory discrimination task (Romo et al. 1999). Neurons were found whose discharge rates varied before and during the delay period between the two stimuli, as a monotonic function of the base stimulus frequency (Figure 33.5). These cells were localized specifically to the VLPFC, also known as the inferior convexity (Romo et al. 1999) within the same general ventral prefrontal regions where object-, face-, and auditory-responsive neurons have been recorded. The feature-based encoding of these cells supports their role in an object-based ventral stream function. In addition to this demonstration of prefrontal neuronal function in a somatosensory task, there is an early lesion study that noted an impairment in a somatosensory alternation task after large lateral prefrontal lesions but not after parietal lesions (Ettlinger and Wegener 1958).

FIGURE 33.5. Single-neuron spike density functions from six different neurons.


Single-neuron spike density functions from six different neurons. Dark bars above each plot indicate times during which the neuron’s firing rate carried significant (P < .01) monotonic signal about base stimulus. (a, c, e) Positive monotonic (more...)

In human neuroimaging studies, it has been shown that the ventral frontal lobe is activated by somatosensory stimulation (Hagen et al. 2002). In their study, two discrete ventral frontal brain regions were responsive to somatosensory stimulation including the posterior inferior frontal gyrus and the orbitofrontal cortex. Additional neuroimaging studies have examined frontal lobe activation during haptic shape perception and discrimination. A recent fMRI study found that several frontal lobe sites were activated during haptic shape-selectivity (Miquee et al. 2008) and during visuo-haptic processing (Stilla and Sathian 2008). Most interesting is the demonstration of vibrotactile working memory activation of human VLPFC areas 47/12 and 45 by Kostopoulos et al. (2007). In their fMRI study, the authors not only demonstrated activity of the VLPFC during a vibrotactile working memory task but also showed functional connectivity with the secondary somatosensory cortex, which was also active in this vibrotactile delayed discrimination task. The area activated, area 47 in the human brain, is analogous to monkey area 12/47, where face and vocalization responses have been recorded (O’Scalaidhe et al. 1997, 1999; Romanski and Goldman-Rakic 2002; Romanski et al. 2005). The anatomical, electrophysiological, and neuroimaging data suggest that somatosensory stimuli may converge in similar VLPFC regions where auditory and visual responsive neurons are found and may combine to participate in object recognition.

33.3.5. Multisensory Responses

The anatomical, physiological, and behavioral data described above show that the ventral frontal lobe receives afferents carrying information about auditory, visual, and somatosensory stimuli. Furthermore, physiological studies indicate that VLPFC neurons prefer complex information and are activated by stimuli with social communication information, that is, faces and vocalizations. Although only one group has examined somatosensory responses in the prefrontal cortex thus far, several imaging studies have shown activation of the ventral frontal lobe with haptic stimulation, which also holds some importance in social communication and also in object recognition.

Although many human neuroimaging studies have posited a role for the frontal lobes in the integration of auditory and visual speech or communication information (Gilbert and Fiez 2004; Hickok et al. 2003; Jones and Callan 2003; Homae et al. 2002), few studies have addressed the cellular mechanisms underlying frontal lobe multisensory integration. An early study by Benevento et al. (1977) made intracellular electrophysiological recordings in the lateral orbital cortex (area 12 orbital) and found that single cells were responsive to simple auditory and visual stimuli (Benevento et al. 1977). Fuster and colleagues recorded from the lateral frontal cortex during an audiovisual matching task (Fuster et al. 2000; Bodner et al. 1996). In this task, prefrontal cortex cells responded selectively to tones, and most of them also responded to colors according to the task rule (Fuster et al. 2000). Gaffan and Harrison (1991) determined the importance of ventral prefrontal cortex in sensory integration by showing that lesions disrupt the performance of cross-modal matching involving auditory and visual objects. Importantly, Rao et al. (1997) have described integration of object and location information in single prefrontal neurons.

A recent study by Romanski and colleagues has documented multisensory responses to combined auditory and visual stimuli in the VLPFC. In this study, rhesus monkeys were presented with movies of familiar monkeys vocalizing while single neurons were recorded from the VLPFC (Sugihara et al. 2006). These movies were separated into audio and video streams, and neural responses to the unimodal stimuli were compared to combined audiovisual responses. Interestingly, about half of the neurons recorded in the VLPFC were multisensory in that they responded to both unimodal auditory and visual stimuli or responded differently to simultaneously presented audiovisual stimuli than to either unimodal stimuli (Sugihara et al. 2006). As has been shown in the superior colliculus, the STS, and auditory cortex, prefrontal neurons exhibited enhancement or suppression (Figure 33.6), and, like the STS, suppression was more commonly observed than enhancement. Multisensory responses were stimulus-dependent in that not all combinations of face-vocalization stimuli elicited a multisensory response. Hence, our estimate of multisensory neurons is most likely a lower bound. If the stimulus battery tested were large enough, we would expect that more neurons would be shown to be multisensory rather than the default, unimodal visual. It was also interesting that face/voice stimuli evoked multisensory responses more frequently than nonface/nonvoice combinations, as in auditory cortex (Ghazanfar et al. 2008) and in the STS (Barraclough et al. 2005). This adds support to the notion that VLPFC is part of a circuit that is specialized for integrating face and voice information rather than integrating all forms of auditory and visual stimuli generically.

FIGURE 33.6. Multisensory neuronal responses in prefrontal cortex.


Multisensory neuronal responses in prefrontal cortex. Responses of two single units are shown in (a) and (b) as raster/spike density plots to auditory vocalization alone (Aud), face alone (Vis), and both presented simultaneously (AV). A bar graph of mean (more...)

Specialization for the integration of communication-relevant audiovisual stimuli in the frontal lobe, and particularly in the VLPFC, is also apparent in the human brain. An fMRI study has shown that area 47 in the human brain is active during the simultaneous presentation of gestures and speech (Xu et al. 2009). In this study, Braun and colleagues found overlap of activation in area 47 when subjects viewed gestures or listened to a voicing of the phrase that fit the gesture. The region of activation in this study of the human brain is a homologous area to that recorded by Sugihara et al., suggesting that this region of the VLPFC is specialized for the multisensory integration of communication-relevant auditory and visual information, namely, gestures (i.e., facial) and vocal sounds.

Thus, there is evidence that auditory, visual, and somatosensory information is reaching the VLPFC, and is converging within areas 12/47 and 45 (Figure 33.7). Furthermore, this information appears to be related most to communication. Although Romo et al. (1999) showed evidence of somatosensory processing related to touch, the innervation of ventral prefrontal cortex includes afferents from the face region of SII (Preuss et al. 1989). This somatosensory face information is arriving at ventral prefrontal regions that receive information about face identity, features, and expression from areas TE and TPO (Webster et al. 1994; O’Scalaidhe et al. 1997, 1999), in addition to auditory inputs that carry information regarding species-specific vocalizations (Romanski et al. 2005).

FIGURE 33.7. Auditory, visual, and somatosensory convergence in VLPFC is shown on a lateral brain schematic of macaque frontal lobe.


Auditory, visual, and somatosensory convergence in VLPFC is shown on a lateral brain schematic of macaque frontal lobe. VLPFC location of vocalization-responsive area (dark gray), visual object- and face-responsive area (light gray), somatosensory-responsive (more...)

33.3.6. Functional Considerations

Although a number of studies have examined the response of prefrontal neurons to face, vocalization, and somatosensory stimuli during passive fixation tasks, it is expected that the VLPFC utilizes these stimuli in more complex processes. There is no doubt that the context of a task will affect the firing of VLPFC neurons. Nonetheless, face and vocalization stimuli are different from typical simple sensory stimuli in that they already carry semantic meaning and emotional valence and need no additional task contexts to make them relevant. A face or vocalization, even when presented in a passive task, will be associated with previous experiences, emotions, and meanings that will evoke responses in a number of brain areas that project to the VLPFC, whereas simple sensory stimuli do not have innate associations and depend only on task contingencies to give them relevance. Thus, responses to face, voice, and other communication-relevant stimuli in prefrontal neurons are the sum total of experience with these stimuli in addition to any task or contextual information presented.

The combination of somatosensory face or touch information, visual face information, and vocalization information could play a number of roles. First, the general process of conjunction allows for the combining of auditory, visual, and/or somatosensory stimuli for many known and, as yet, unknown functions. Thus, the VLPFC may serve a general purpose in allowing complex stimuli related to any of the modalities to be integrated. This may be especially relevant for the frontal lobe when the information is to be remembered or operated on in some way. However, a function more directly suited to the process of communication would be feedback control of articulation. Auditory information that is coded phonologically and mouth or face movements perceived via somatosensory input would be integrated, and then orofacial movements could be adjusted to alter the production of sounds via a speech/vocalization output circuit. The posterior part of the inferior frontal gyrus (Broca’s area) has been shown, via lesions analysis and neuroimaging, to play a role in the production of this phonetic code, or articulatory stream. In contrast, the anterior inferior frontal gyrus may integrate auditory, somatosensory, and visual perceptual information to produce this stream (Papoutsi et al. 2009). The somatosensory feedback regarding positioning of the mouth and face would play an important role in control of articulation. The visual face and auditory vocalization information available to these neurons could provide further information from a speaker that warrants a reply or could provide information about a hand or face during a gesture. Thus, a third function for the combination of auditory, visual, and somatosensory information would be the perception, memory, and execution of gestures that accompany speech and vocalizations.

The VLPFC may also be part of a larger circuit that has been called the mirror neuron system. This system is purported to be involved in the perception and execution of gestures as occurs in imitation (Rizzolatti and Craighero 2004). The VLPFC has reciprocal connections with many parts of the mirror neuron circuit. Finally, convergence of auditory, visual, and haptic information could also be used in face or object recognition especially when one sense is not optimal, and additional information from other sensory modalities is needed to confirm identification. The convergence of these sensory modalities and others may play additional functional roles during a variety of complex cognitive functions.


  1. Averbeck B. B, Romanski L. M. Probabilistic encoding of vocalizations in macaque ventral lateral prefrontal cortex. Journal Neuroscience. 2006;26:11023–11033. [PubMed: 17065444]
  2. Barbas H. Anatomic organization of basoventral and mediodorsal visual recipient prefrontal regions in the rhesus monkey. Journal of Comparative Neurology. 1988;276:313–342. [PubMed: 3192766]
  3. Barbas H. Architecture and cortical connections of the prefrontal cortex in the rhesus monkey. Advances in Neurology. 1992;57:91–115. [PubMed: 1543090]
  4. Barbas H, Pandya D. N. Architecture and intrinsic connections of the prefrontal cortex in the rhesus monkey. Journal of Comparative Neurology. 1989;286:353–375. [PubMed: 2768563]
  5. Barbas H, Mesulam M. M. Organization of afferent input to subdivisions of area 8 in the rhesus monkey. Journal of Comparative Neurology. 1981;200:407–431. [PubMed: 7276245]
  6. Barbas H, Mesulam M. M. Cortical afferent input to the principalis region of the rhesus monkey. Neuroscience. 1985;15:619–637. [PubMed: 4069349]
  7. Barraclough N. E, Xiao D, Baker C. I, Oram M. W, Perrett D. I. Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience. 2005;17:377–391. [PubMed: 15813999]
  8. Belin P, Zatorre R. J, Lafaille P, Ahad P, Pike B. Voice-selective areas in human auditory cortex. Nature. 2000;403:309–312. [PubMed: 10659849]
  9. Benevento L. A, Fallon J, Davis B. J, Rezak M. Auditory–visual interaction in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Experimental Neurology. 1977;57:849–872. [PubMed: 411682]
  10. Binder J. R, Frost J. A, Hammeke T. A, Bellgowan P. S, Springer J. A, Kaufman J. N, Possing E. T. Human temporal lobe activation by speech and nonspeech sounds. Cerebral Cortex. 2000;10:512–528. [PubMed: 10847601]
  11. Bodner M, Kroger J, Fuster J. M. Auditory memory cells in dorsolateral prefrontal cortex. Neuroreport. 1996;7:1905–1908. [PubMed: 8905689]
  12. Buckner R. L, Raichle M. E, Petersen S. E. Dissociation of human prefrontal cortical areas across different speech production tasks and gender groups. Journal of Neurophysiology. 1995;74:2163–2173. [PubMed: 8592204]
  13. Bullier J, Schall J. D, Morel A. Functional streams in occipito-frontal connections in the monkey. Behavioural Brain Research. 1996;76:89–97. [PubMed: 8734045]
  14. Carmichael S. T, Price J. L. Sensory and premotor connections of the orbital and medial prefrontal cortex of macaque monkeys. Journal of Comparative Neurology. 1995;363:642–664. [PubMed: 8847422]
  15. Cavada C, Goldman-Rakic P. S. Posterior parietal cortex in rhesus monkey: II. Evidence for segregated corticocortical networks linking sensory and limbic areas with the frontal lobe. Journal of Comparative Neurology. 1989;287:422–445. [PubMed: 2477406]
  16. Chavis D. A, Pandya D. N. Further observations on corticofrontal connections in the rhesus monkey. Brain Research. 1976;117:369–386. [PubMed: 825194]
  17. Cipolloni P. B, Pandya D. N. Connectional analysis of the ipsilateral and contralateral afferent neurons of the superior temporal region in the rhesus monkey. Journal of Comparative Neurology. 1989;281:567–585. [PubMed: 2708582]
  18. Cipolloni P. B, Pandya D. N. Cortical connections of the frontoparietal opercular areas in the rhesus monkey. Journal of Comparative Neurology. 1999;403:431–458. [PubMed: 9888311]
  19. Cohen J. D, Perlstein W. M, Braver T. S, Nystrom L. E, Noll D. C, Jonides J, Smith E. E. Temporal dynamics of brain activation during a working memory task. Nature. 1997;386:604–608. [PubMed: 9121583]
  20. Cohen Y. E, Theunissen F, Russ B. E, Gill P. Acoustic features of rhesus vocalizations and their representation in the ventrolateral prefrontal cortex. Journal of Neurophysiology. 2007;97:1470–1484. [PubMed: 17135477]
  21. Deacon T. W. Cortical connections of the inferior arcuate sulcus cortex in the macaque brain. Brain Research. 1992;573:8–26. [PubMed: 1374284]
  22. Demb J. B, Desmond J. E, Wagner A. D, Vaidya C. J, Glover G. H, Gabrieli J. D. Semantic encoding and retrieval in the left inferior prefrontal cortex: A functional MRI study of task difficulty and process specificity. Journal of Neuroscience. 1995;15:5870–5878. [PubMed: 7666172]
  23. Ettlinger G, Wegener J. Somaesthetic alternation, discrimination and orientation after frontal and parietal lesions in monkeys. The Quarterly Journal of Experimental Psychology. 1958;10:177–186.
  24. Fecteau S, Armony J. L, Joanette Y, Belin P. Sensitivity to voice in human prefrontal cortex. Journal of Neurophysiology. 2005;94:2251–2254. [PubMed: 15928057]
  25. Fiez J. A, Raife E. A, Balota D. A, Schwarz J. P, Raichle M. E, Petersen S. E. A positron emission tomography study of the short-term maintenance of verbal information. Journal of Neuroscience. 1996;16:808–822. [PubMed: 8551361]
  26. Fuster J. M, Bodner M, Kroger J. K. Cross-modal and cross-temporal association in neurons of frontal cortex. Nature. 2000;405:347–351. [PubMed: 10830963]
  27. Gabrieli J. D. E, Poldrack R. A, Desmond J. E. The role of left prefrontal cortex in language and memory. Proceedings of the National Academy of Sciences. 1998;95:906–913. [PMC free article: PMC33815] [PubMed: 9448258]
  28. Gaffan D, Harrison S. Auditory–visual associations, hemispheric specialization and temporalfrontal interaction in the rhesus monkey. Brain. 1991;114:2133–2144. [PubMed: 1933238]
  29. Galaburda A. M, Pandya D. N. The intrinsic architectonic and connectional organization of the superior temporal region of the rhesus monkey. Journal of Comparative Neurology. 1983;221:169–184. [PubMed: 6655080]
  30. Gelfand J. R, Bookheimer S. Y. Dissociating neural mechanisms of temporal sequencing and processing phonemes. Neuron. 2003;38:831–842. [PubMed: 12797966]
  31. Ghazanfar A. A, Chandrasekaran C, Logothetis N. K. Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. Journal of Neuroscience. 2008;28:4457–4469. [PMC free article: PMC2663804] [PubMed: 18434524]
  32. Gilbert A. M, Fiez J. A. Integrating rewards and cognition in the frontal cortex. Cognitive, Affective and Behavioral Neuroscience. 2004;4:540–552. [PubMed: 15849896]
  33. Goldman P. S, Rosvold H. E. Localization of function within the dorsolateral prefrontal cortex of the rhesus monkey. Experimental Neurology. 1970;27:291–304. [PubMed: 4987453]
  34. Gross C. G. A comparison of the effects of partial and total lateral frontal lesions on test performance by monkeys. Journal of Comparative Physiological Psychology. 1963;56:41–47.
  35. Gross C. G, Weiskrantz L. Evidence for dissociation of impairment on auditory discrimination and delayed response following lateral frontal lesions in monkeys. Experimental Neurology. 1962;5:453–476. [PubMed: 13902195]
  36. Hagen M. C, Zald D. H, Thornton T. A, Pardo J. V. Somatosensory processing in the human inferior prefrontal cortex. Journal of Neurophysiology. 2002;88:1400–1406. [PubMed: 12205161]
  37. Hickok G, Buchsbaum B, Humphries C, Muftuler T. Auditory–motor interaction revealed by fMRI: Speech, music, and working memory in area spt. Journal of Cognitive Neuroscience. 2003;15:673–682. [PubMed: 12965041]
  38. Homae F, Hashimoto R, Nakajima K, Miyashita Y, Sakai K. L. From perception to sentence comprehension: The convergence of auditory and visual information of language in the left inferior frontal cortex. NeuroImage. 2002;16:883–900. [PubMed: 12202077]
  39. Hoshi E, Shima K, Tanji J. Neuronal activity in the primate prefrontal cortex in the process of motor selection based on two behavioral rules. Journal of Neurophysiology. 2000;83:2355–2373. [PubMed: 10758139]
  40. Gifford G. W III, Maclean K. A, Hauser M. D, Cohen Y. E. The neurophysiology of functionally meaningful categories: Macaque ventrolateral prefrontal cortex plays a critical role in spontaneous categorization of species-specific vocalizations. Journal of Cognitive Neuroscience. 2005;17:1471–1482. [PubMed: 16197700]
  41. Jones E. G, Powell T. P. An anatomical study of converging sensory pathways within the cerebral cortex of the monkey. Brain. 1970;93:793–820. [PubMed: 4992433]
  42. Jones J. A, Callan D. E. Brain activity during audiovisual speech perception: An fMRI study of the McGurk effect. Neuroreport. 2003;14:1129–1133. [PubMed: 12821795]
  43. Kostopoulos P, Albanese M. C, Petrides M. Ventrolateral prefrontal cortex and tactile memory disambiguation in the human brain. Proceedings of the National Academy of Sciences of the United States of America. 2007;104:10223–10228. [PMC free article: PMC1891227] [PubMed: 17551017]
  44. Miquee A, Xerri C, Rainville C, Anton J. L, Nazarian B, Roth M, Zennou-Azogui Y. Neuronal substrates of haptic shape encoding and matching: A functional magnetic resonance imaging study. Neuroscience. 2008;152:29–39. [PubMed: 18255234]
  45. Mishkin M, Manning F. J. Non-spatial memory after selective prefrontal lesions in monkeys. Brain Research. 1978;143:313–323. [PubMed: 415803]
  46. Newman J. D, Lindsley D. F. Single unit analysis of auditory processing in squirrel monkey frontal cortex. Experimental Brain Research. 1976;25:169–181. [PubMed: 819284]
  47. Ninokura Y, Mushiake H, Tanji J. Integration of temporal order and object information in the monkey lateral prefrontal cortex. Journal of Neurophysiology. 2004;91:555–560. [PubMed: 12968014]
  48. O’Scalaidhe S. P. O, Wilson F. A. W, Goldman-Rakic P. G. R. Face-selective neurons during passive viewing and working memory performance of rhesus monkeys: Evidence for intrinsic specialization of neuronal coding. Cerebral Cortex. 1999;9:459–475.1. [PubMed: 10450891]
  49. O’Scalaidhe S. P, Wilson F. A, Goldman-Rakic P. S. Areal segregation of face-processing neurons in prefrontal cortex. Science. 1997;278:1135–1138. [PubMed: 9353197]
  50. Pandya D.N, Sanides F. Architectonic parcellation of the temporal operculum in rhesus monkey and its projection pattern. Zeitschrift fuer Anatomie und Entwicklungsgeschichte. 1973;139:127–161. [PubMed: 4197942]
  51. Pandya D. N, Kuypers H. G. Cortico-cortical connections in the rhesus monkey. Brain Research. 1969;13:13–36. [PubMed: 4185124]
  52. Pandya D. N, Hallett M, Kmukherjee S. K. Intra- and interhemispheric connections of the neocortical auditory system in the rhesus monkey. Brain Research. 1969;14:49–65. [PubMed: 4977327]
  53. Papoutsi M, de Zwart J. A, Jansma J. M, Pickering M. J, Bednar J. A, Horwitz B. From phonemes to articulatory codes: An fMRI study of the role of Broca’s area in speech production. Cerebral Cortex. 2009;19:2156–2165. [PMC free article: PMC2722428] [PubMed: 19181696]
  54. Passingham R. Delayed matching after selective prefrontal lesions in monkeys (Macaca mulatta) Brain Research. 1975;92:89–102. [PubMed: 809096]
  55. Paulesu E, Frith C. D, Frackowiak R. S. J. The neural correlates of the verbal component of working memory. Nature. 1993;362:342–5.32. [PubMed: 8455719]
  56. Petrides M, Pandya D. N. Association fiber pathways to the frontal cortex from the superior temporal region in the rhesus monkey. Journal of Comparative Neurology. 1988;273:52–66. [PubMed: 2463275]
  57. Pigarev I. N, Rizzolatti G, Schandolara C. Neurons responding to visual stimuli in the frontal lobe of macaque monkeys. Neuroscience Letters. 1979;12:207–212. [PubMed: 111174]
  58. Poremba A, Mishkin M. Exploring the extent and function of higher-order auditory cortex in rhesus monkeys. Hearing Research. 2007;229:14–23. [PMC free article: PMC2709996] [PubMed: 17321703]
  59. Posner M. I, Abdullaev Y. G, Mc Candliss B. D, Sereno S. C. Neuroanatomy, circuitry and plasticity of word reading. Neuroreport. 1999;10:R12–R23. [PubMed: 10208568]
  60. Preuss T. M, Goldman-Rakic P. S. Connections of the ventral granular frontal cortex of macaques with perisylvian premotor and somatosensory areas: Anatomical evidence for somatic representation in primate frontal association cortex. Journal of Comparative Neurology. 1989;282:293–316. [PubMed: 2708598]
  61. Price C. J. The functional anatomy of word comprehension and production. Trends in Cognitive Sciences. 1998;2:281–288. [PubMed: 21227210]
  62. Price J. L. Multisensory convergence in the orbital and ventrolateral prefrontal cortex. Chemosensory Perception. 2008;1:103–109.
  63. Rao S. C, Rainer G, Miller E. K. Integration of what and where in the primate prefrontal cortex. Science. 1997;276:821–824. [PubMed: 9115211]
  64. Rizzolatti G, Craighero L. The mirror–neuron system. Annual Review of Neuroscience. 2004;27:169–192. [PubMed: 15217330]
  65. Romanski L. M. Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. Cerebral Cortex. 2007;17(S1):i61–i69. [PMC free article: PMC2778283] [PubMed: 17634387]
  66. Romanski L. M, Goldman-Rakic P. S. An auditory domain in primate prefrontal cortex. Nature Neuroscience. 2002;5:15–16. [PMC free article: PMC2793092] [PubMed: 11753413]
  67. Romanski L. M, Averbeck B. B, Diltz M. Neural representation of vocalizations in the primate ventrolateral prefrontal cortex. Journal of Neurophysiology. 2005;93:734–747. [PubMed: 15371495]
  68. Romanski L. M, Tian B, Fritz J, Mishkin M, Goldman-Rakic P. S, Rauschecker J. P. Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nature Neuroscience. 1999b;2:1131–1136. [PMC free article: PMC2778291] [PubMed: 10570492]
  69. Romanski L. M, Bates J. F, Goldman-Rakic P. S. Auditory belt and parabelt projections to the prefrontal cortex in the rhesus monkey. Journal of Comparative Neurology. 1999a;403:141–157. [PubMed: 9886040]
  70. Romo R, Brody C. D, Hernandez A, Lemus L. Neuronal correlates of parametric working memory in the prefrontal cortex. Nature. 1999;399:470–473. [PubMed: 10365959]
  71. Rosenkilde C. E, Bauer R. H, Fuster J. M. Single cell activity in ventral prefrontal cortex of behaving monkeys. Brain Research. 1981;209:375–394. [PubMed: 7225799]
  72. Schall J. D, Morel A, King D. J, Bullier J. Topography of visual cortex connections with frontal eye field in macaque: Convergence and segregation of processing streams. Journal of Neuroscience. 1995;15:4464–4487. [PubMed: 7540675]
  73. Scott S. K, Blank C. C, Rosen S, Wise R. J. Identification of a pathway for intelligible speech in the left temporal lobe. Brain. 2000;12:2400–2406. [PubMed: 11099443]
  74. Stevens A. A, Goldman-Rakic P. S, Gore J. C, Fulbright R. K, Wexler B. E. Cortical dysfunction in schizophrenia during auditory word and tone working memory demonstrated by functional magnetic resonance imaging. Archives of General Psychiatry. 1998;55:1097–1103. [PubMed: 9862553]
  75. Stilla R, Sathian K. Selective visuo-haptic processing of shape and texture. Human Brain Mapping. 2008;29:1123–1138. [PMC free article: PMC3060058] [PubMed: 17924535]
  76. Stromswold K, Caplan D, Alpert N, Rauch S. Localization of syntactic comprehension by positron emission tomography. Brain & Language. 1996;52:452–473. [PubMed: 8653390]
  77. Sugihara T, Diltz M. D, Averbeck B. B, Romanski L. M. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. Journal of Neuroscience. 2006;26:11138–11147. [PMC free article: PMC2767253] [PubMed: 17065454]
  78. Suzuki H, Azuma M. Prefrontal neuronal activity during gazing at a light spot in the monkey. Brain Research. 1977;126:497–508. [PubMed: 405083]
  79. Tanila H, Carlson S, Linnankoski I, Kahila H. Regional distribution of functions in dorsolateral prefrontal cortex of the monkey. Behavioral Brain Research. 1993;53:63–71. [PubMed: 8466668]
  80. Tanila H, Carlson S, Linnankoski I, Lindroos F, Kahila H. Functional properties of dorsolateral prefrontal cortical neurons in awake monkey. Behavioral Brain Research. 1992;47:169–180. [PubMed: 1590947]
  81. Tian B, Reser D, Durham A, Kustov A, Rauschecker J. P. Functional specialization in rhesus monkey auditory cortex. Science. 2001;292:290–293. [PubMed: 11303104]
  82. Tsao D. Y, Schweers N, Moeller S, Freiwald W. A. Patches of face-selective cortex in the macaque frontal lobe. Nature Neuroscience. 2008;11:877–879. [PubMed: 18622399]
  83. Ungerleider L. G, Gaffan D, Pelak V. S. Projections from inferior temporal cortex to prefrontal cortex via the uncinate fascicle in rhesus monkeys. Experimental Brain Research. 1989;76:473–484. [PubMed: 2792241]
  84. Webster M. J, Bachevalier J, Ungerleider L. G. Connections of inferior temporal areas TEO and TE with parietal and frontal cortex in macaque monkeys. Cerebral Cortex. 1994;4:470–483. [PubMed: 7530521]
  85. Weiskrantz L, Mishkin M. Effects of temporal and frontal cortical lesions on auditory discrimination in monkeys. Brain. 1958;80:406–414. [PubMed: 13596473]
  86. Wilson F. A, O’Scalaidhe S. P, Goldman-Rakic P. S. Dissociation of object and spatial processing domains in primate prefrontal cortex. Science. 1993;260:1955–1958. [PubMed: 8316836]
  87. Wollberg Z, Sela J. Frontal cortex of the awake squirrel monkey: Responses of single cells to visual and auditory stimuli. Brain Research. 1980;198:216–220. [PubMed: 7407587]
  88. Xu J, Gannon P. J, Emmorey K, Smith J. F, Braun A. R. Symbolic gestures and spoken language are processed by a common neural system. Proceedings of the National Academy of Sciences of the United States of America. 2009;106:20664–20669. [PMC free article: PMC2779203] [PubMed: 19923436]
  89. Zatorre R. J, Evans A. C, Meyer E, Gjedde A. Lateralization of phonetic and pitch discrimination in speech processing. Science. 1992;256:846–849. [PubMed: 1589767]
  90. Zatorre R. J, Bouffard M, Belin P. Sensitivity to auditory object features in human temporal neocortex. The Journal of Neuroscience. 2004;24:3637–3642. [PubMed: 15071112]
Copyright © 2012 by Taylor & Francis Group, LLC.
Bookshelf ID: NBK92838PMID: 22593866


  • PubReader
  • Print View
  • Cite this Page

Other titles in this collection

Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to PubMed

Similar articles in PubMed

See reviews...See all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...