Send to

Choose Destination
Neuroimage Clin. 2014 Apr 13;4:687-94. doi: 10.1016/j.nicl.2014.04.004. eCollection 2014.

Biased binomial assessment of cross-validated estimation of classification accuracies illustrated in diagnosis predictions.

Author information

Cyclotron Research Centre, University of Liège, Liège, Belgium ; Coma Science Group, Neurology Department, University Hospital of Liège, Liège, Belgium.
Complexus Group, Computer Science Department, Universidad Central de Colombia, Bogotá, Colombia.
Department of Physics & Astronomy, Brain and Mind Institute, University of Western Ontario, London, ON, Canada.
Cyclotron Research Centre, University of Liège, Liège, Belgium ; Laboratory of Behavioral and Cognitive Neurology, Stanford University, Palo Alto, USA.
Cyclotron Research Centre, University of Liège, Liège, Belgium.
Cyclotron Research Centre, University of Liège, Liège, Belgium ; Department of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium.


Multivariate classification is used in neuroimaging studies to infer brain activation or in medical applications to infer diagnosis. Their results are often assessed through either a binomial or a permutation test. Here, we simulated classification results of generated random data to assess the influence of the cross-validation scheme on the significance of results. Distributions built from classification of random data with cross-validation did not follow the binomial distribution. The binomial test is therefore not adapted. On the contrary, the permutation test was unaffected by the cross-validation scheme. The influence of the cross-validation was further illustrated on real-data from a brain-computer interface experiment in patients with disorders of consciousness and from an fMRI study on patients with Parkinson disease. Three out of 16 patients with disorders of consciousness had significant accuracy on binomial testing, but only one showed significant accuracy using permutation testing. In the fMRI experiment, the mental imagery of gait could discriminate significantly between idiopathic Parkinson's disease patients and healthy subjects according to the permutation test but not according to the binomial test. Hence, binomial testing could lead to biased estimation of significance and false positive or negative results. In our view, permutation testing is thus recommended for clinical application of classification with cross-validation.


binomial; classification; cross-validation; permutation test

[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Elsevier Science Icon for PubMed Central
Loading ...
Support Center