Format

Send to

Choose Destination
Neuroimage. 2017 Jan 15;145(Pt B):166-179. doi: 10.1016/j.neuroimage.2016.10.038. Epub 2016 Oct 29.

Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines.

Author information

1
Parietal project-team, INRIA Saclay-ile de France, France; CEA/Neurospin bât 145, 91191 Gif-Sur-Yvette, France.
2
Rotman Research Institute, Baycrest Health Sciences, Toronto, ON, Canada M6A 2E1; Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada M5S 1A1.
3
CEA/Neurospin bât 145, 91191 Gif-Sur-Yvette, France; Cognitive Neuroimaging Unit, INSERM, Université Paris-Sud and Université Paris-Saclay, 91191 Gif-sur-Yvette, France; Neuropsychology & Neuroimaging team INSERM UMRS 975, Brain and Spine Institute (ICM), Paris.

Abstract

Decoding, i.e. prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within- and across-subject predictions, on multiple datasets -anatomical and functional MRI and MEG- and simulations. Theory and experiments outline that the popular "leave-one-out" strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be favorable to use sane defaults, in particular for non-sparse decoders.

KEYWORDS:

Bagging; Cross-validation; Decoding; FMRI; MVPA; Model selection; Sparse

[Indexed for MEDLINE]

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center