Send to

Choose Destination
Front Neurosci. 2014 Nov 18;8:373. doi: 10.3389/fnins.2014.00373. eCollection 2014.

Hybrid fNIRS-EEG based classification of auditory and visual perception processes.

Author information

Cognitive Systems Lab, Institute of Anthropomatics and Robotics, Karlsruhe Institute of Technology Karlsruhe, Germany.
Department of Psychology, Center for Cognition and Brain Studies, The Chinese University of Hong Kong Hong Kong, China ; Temasek Laboratories, National University of Singapore Singapore, Singapore.
Nuffield Department of Clinical Neurosciences, John Radcliffe Hospital Oxford, UK.
Institute for Infocomm Research (I2R), ASTAR Singapore, Singapore.


For multimodal Human-Computer Interaction (HCI), it is very useful to identify the modalities on which the user is currently processing information. This would enable a system to select complementary output modalities to reduce the user's workload. In this paper, we develop a hybrid Brain-Computer Interface (BCI) which uses Electroencephalography (EEG) and functional Near Infrared Spectroscopy (fNIRS) to discriminate and detect visual and auditory stimulus processing. We describe the experimental setup we used for collection of our data corpus with 12 subjects. On this data, we performed cross-validation evaluation, of which we report accuracy for different classification conditions. The results show that the subject-dependent systems achieved a classification accuracy of 97.8% for discriminating visual and auditory perception processes from each other and a classification accuracy of up to 94.8% for detecting modality-specific processes independently of other cognitive activity. The same classification conditions could also be discriminated in a subject-independent fashion with accuracy of up to 94.6 and 86.7%, respectively. We also look at the contributions of the two signal types and show that the fusion of classifiers using different features significantly increases accuracy.


EEG; brain-computer interface; fNIRS; visual and auditory perception

Supplemental Content

Full text links

Icon for Frontiers Media SA Icon for PubMed Central
Loading ...
Support Center