Brain activation related to the sensory modality. Auditory (in red) and tactile (in green) functional MRI activation in 17 sighted subjects is projected onto a 3-D representation of the right and left hemispheres (RH and LH) of a representative brain of one subject. Only positive differences are shown in the related contrasts. a, Activation maps resulting from the contrasts {Auditory conditions vs Rest}, i.e., {(AID + AL + AD)} and {Tact vs Rest}, i.e., {(TID + TL + TD)}. The activation maps were obtained using RFX with a corrected threshold for multiple comparisons using FDR of q < 0.05 in combination with a cluster size threshold correction of p > 0.05 and superimposed on the individual brain. The auditory conditions activated bilaterally the primary and secondary auditory cortex (BA41, BA42, BA22) (see Tables 2 and 3 for an exhaustive list of the activation foci). The tactile conditions activated bilaterally the primary and secondary somatosensory cortex (BA1, 2, 3, 43) (see Tables 2 and 3 for an exhaustive list of activation foci), although the activation was more extended in the left hemisphere. Overlapping activation is shown in yellow. b, Activation maps resulting from the contrasts {Auditory Conditions − Tactile Conditions}, i.e., {[(AID + AL + AD) > (TID + TL + TD)] ∩ (AID + AL + AD)} and {Tactile Conditions − Auditory Conditions}, i.e., {[(TID + TL + TD) > (AID + A + AD)] ∩ (TID + TL + TD)}. The activation maps were obtained using RFX analyses with an uncorrected threshold of p < 0.005 in combination with a cluster size threshold correction of p < 0.05 and superimposed on the individual brain. Left and right primary and secondary auditory cortices (BA41, BA42, BA22) were specifically activated during the auditory conditions (Tables 2, 3). The left postcentral gyrus (BA43, contralateral to the stimulated hand) was specifically activated during the tactile conditions (Tables 2, 3).

Brain activation related to auditory and tactile “what” and “where” processing. The modality-specific auditory “what” (in red), the modality-specific tactile “where” (in green), the bimodal “what” (in yellow), and the bimodal “where” (in blue) functional MRI activation in 17 subjects is projected onto a 3-D representation of the right and left hemispheres (RH and LH) of a representative brain of one subject. The activation maps resulting from the contrasts {(AID > TID) ∩ (AID > AL)} (modality-specific tactile “what” areas), {(TL > AL) ∩ (TL > TID)} (modality-specific tactile “where” areas), {[(AID + TID) > (AL + TL)] ∩ [(AID > AD) ∩ (TID > TD)]} (bimodal “what” areas), and {[(AL + TL) > (AID + TID)] ∩ [(AL > AD (TL > TD)]} (bimodal “where” areas) were obtained using RFX with a threshold of p < 0.005 (uncorrected for multiple comparisons) and superimposed on the individual brain. Bimodal “what” brain areas were mainly found in the inferior frontal lobe and insula whereas bimodal “where” brain areas were mainly in the parietal lobe. Modality-specific auditory “what” brain areas were in the anterior STG-STS whereas modality-specific tactile “where” brain areas were distributed in the parietal cortex. It is worth noting that activation in the MOG was revealed by this contrast, whereas this region was deactivated to a different extent during all auditory and tactile conditions. No significant modality-specific auditory “where” or tactile “what” activations were found. The lower part of the figure shows the bilateral activation of the insula on the sagittal (SAG), coronal (COR), and transversal (TRA) plane, related to bimodal “what” processing.

Activity profile in the brain areas identified as multisensory “where” brain areas. The BOLD signal is plotted as a function of the task and the modality in the left SPL (x = −21; y = −67; z = 43; cluster size = 600 voxels), the left IPL (x = −39, y = −43; z = 43; cluster size = 666 voxels), the right precuneus (x = 15; y = −67; z = 46; cluster size = 1020 voxels), and the right IPL (x = 42; y = −37; z = 40; cluster size = 151 voxels). The BOLD response is expressed in percentage of signal change. The error bars represent SE. Localization conditions activated parietal regions more than identification conditions in both modalities.

Experimental setup. a, The experimental setup used to deliver vibrotactile stimulation. Vibrotactile stimuli were produced by nonmagnetic, ceramic piezoelectric bending elements placed directly under the fingers of the hand. The benders were driven by tones presented at a high-amplitude value and produced a vibration that corresponded to that driving frequency. Each of the four piezoelectric elements was attached to one of the four fingers with fasteners (Velcro), which was glued to the underside of the piezoelectric element. In addition, the right hand was securely attached to prevent any hand movements during the experiment. b, Stimuli consisted of four pure tones of 40, 80, 160, and 320 Hz (duration = 1000 ms, with a rise/fall time of ~10 ms, and an interstimulus interval of 1000 ms) applied to one of the four fingers of the right hand. The waveform of the 80 Hz stimulus is shown here for illustration. c, For the auditory modality, four piano chords varying in pitch originated from four different sources around the subject: −90°, −30°, 30°, and 90°. d, One of the auditory stimuli originating from a source located at 30°. The signal varied both in intensity and in frequency according to the head-related transfer function calculated from a mean head size. e, Experimental design. The stimulus sequence within one block for the tactile modality is schematized here. During one block (18 s), a sequence of nine stimuli of 1 s duration (separated by an interstimulus interval of 1 s) was delivered on the subject’s fingers. The subjects had to perform a one-back comparison task, i.e., to compare each stimulus with the previous one and to determine whether it was the same or different regarding either its frequency or its location. Eight responses were expected per block of nine stimuli. An identical paradigm was used in the auditory modality.

Brain activation related to the task. Identification (in pink) and localization (in blue) functional MRI activation in 17 sighted subjects is projected onto a 3-D representation of the right and left hemispheres (RH and LH) of a representative brain of one subject. Please note that only activations that reach the surface of the cortex are visible on the mesh. a, Activation maps resulting from the contrasts {identification task – control task (detection)}, i.e., {[(AID + TID) > (AD + TD)] ∩ (AID + TID)} and {localization task – control task (detection)}, i.e., {[(AL + TL) > (AD + TD)] ∩ (AL + TL)}. The activation maps were obtained using RFX analyses with a corrected threshold for FDR of q < 0.05 in combination with a cluster size threshold correction of p < 0.05 and superimposed on the individual brain. The identification conditions activated bilaterally the frontal and parietal cortices, including the middle and inferior frontal gyri and the IPL (see Tables 4 and 5 for an exhaustive list of activation foci). The localization conditions activated the left SPL and IPL, the right IPL and the right precuneus as well as the middle and inferior frontal gyri (BA7, BA40, and BA6, BA9) (see Tables 4 and 5 for an exhaustive list of activation foci). b, Activation maps resulting from the contrasts {Identification – Localization}, i.e., {[(AID + TID) > (AL + TL)] ∩ (AID + TID)} and {Localization – Identification}, i.e., {[(AL + TL) > (AID + TID)] ∩ (AL + TL)}. The activation maps were obtained using RFX analyses with an uncorrected threshold of p < 0.005 in combination with a cluster size threshold correction of p < 0.05 and superimposed on the individual brain. The right inferior frontal gyrus (BA9) as well as the insula bilaterally (not visible in this figure) was specifically activated during the identification conditions (Tables 4, 5). The right IPL (BA40), the right precuneus (BA7), the left SPL (BA7), and the left IPL (BA40) were specifically activated during the localization conditions (Tables 4, 5).

Activity profile in the brain areas identified as multisensory “what” brain areas. The BOLD signal is plotted as a function of the task and the modality in the left (x = −30; y = 17; z = 16; cluster size = 88 voxels) and right (x = 36; y = 23; z = 7; cluster size = 164 voxels) insula, the right IFG (x = 48; y = 11; z = 25; cluster size = 68 voxels), and the right MFG (x = 42; y = 32; z = 28; cluster size = 151 voxels). The BOLD response is expressed as a percentage of signal change. The error bars represent SE. Identification conditions activated inferior middle frontal and insular regions more than localization conditions in both modalities.