Format

Send to

Choose Destination

See 1 citation found by title matching your search:

Neuroimage. 2018 Nov 1;181:734-747. doi: 10.1016/j.neuroimage.2018.07.047. Epub 2018 Jul 25.

Reading the (functional) writing on the (structural) wall: Multimodal fusion of brain structure and function via a deep neural network based translation approach reveals novel impairments in schizophrenia.

Author information

1
The Mind Research Network, 1101 Yale Blvd, Albuquerque, NM, 87106, USA. Electronic address: s.m.plis@gmail.com.
2
The Mind Research Network, 1101 Yale Blvd, Albuquerque, NM, 87106, USA; Intel Corporation, 5000 W Chandler Blvd, Chandler, AZ, 85226, USA.
3
Human Neuroscience Laboratory, Yale University, New Haven, CT, 06520, USA.
4
The Mind Research Network, 1101 Yale Blvd, Albuquerque, NM, 87106, USA.
5
Division of Computer Science and Engineering, CAIIT, Chonbuk National University, Jeonju, South Korea.
6
Department of Psychiatry and Neurosciences, University of New Mexico, Albuquerque, NM, 87131, USA.
7
Courant Institute & Center for Data Science, New York University, New York, NY, 10012, USA.
8
Olin Neuropsychiatry Research Center, Hartford Hospital (IOL Campus), Hartford, CT, USA; Department of Psychiatry and Neurobiology, Yale University School of Medicine, New Haven, CT, USA.
9
The Mind Research Network, 1101 Yale Blvd, Albuquerque, NM, 87106, USA; Department of Psychiatry and Neurosciences, University of New Mexico, Albuquerque, NM, 87131, USA; Dept. of Electrical and Computer Engineering, University of New Mexico, Albuquerque, NM, 87131, USA.

Abstract

This work presents a novel approach to finding linkage/association between multimodal brain imaging data, such as structural MRI (sMRI) and functional MRI (fMRI). Motivated by the machine translation domain, we employ a deep learning model, and consider two different imaging views of the same brain like two different languages conveying some common facts. That analogy enables finding linkages between two modalities. The proposed translation-based fusion model contains a computing layer that learns "alignments" (or links) between dynamic connectivity features from fMRI data and static gray matter patterns from sMRI data. The approach is evaluated on a multi-site dataset consisting of eyes-closed resting state imaging data collected from 298 subjects (age- and gender matched 154 healthy controls and 144 patients with schizophrenia). Results are further confirmed on an independent dataset consisting of eyes-open resting state imaging data from 189 subjects (age- and gender matched 91 healthy controls and 98 patients with schizophrenia). We used dynamic functional connectivity (dFNC) states as the functional features and ICA-based sources from gray matter densities as the structural features. The dFNC states characterized by weakly correlated intrinsic connectivity networks (ICNs) were found to have stronger association with putamen and insular gray matter pattern, while the dFNC states of profuse strongly correlated ICNs exhibited stronger links with the gray matter pattern in precuneus, posterior cingulate cortex (PCC), and temporal cortex. Further investigation with the estimated link strength (or alignment score) showed significant group differences between healthy controls and patients with schizophrenia in several key regions including temporal lobe, and linked these to connectivity states showing less occupancy in healthy controls. Moreover, this novel approach revealed significant correlation between a cognitive score (attention/vigilance) and the function/structure alignment score that was not detected when data modalities were considered separately.

KEYWORDS:

Deep learning; Multimodal fusion; Psychosis; Schizophrenia

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center