Format

Send to

Choose Destination
Sci Rep. 2018 Apr 26;8(1):6620. doi: 10.1038/s41598-018-24937-4.

A closer look at cross-validation for assessing the accuracy of gene regulatory networks and models.

Author information

1
Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL, United States of America.
2
Carl R. Woese Institute for Genomic Biology, University of Illinois at Urbana-Champaign, Urbana, IL, United States of America.
3
Department of Statistics, University of Illinois at Urbana-Champaign, Urbana, IL, United States of America.
4
Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL, United States of America. sinhas@illinois.edu.
5
Carl R. Woese Institute for Genomic Biology, University of Illinois at Urbana-Champaign, Urbana, IL, United States of America. sinhas@illinois.edu.

Abstract

Cross-validation (CV) is a technique to assess the generalizability of a model to unseen data. This technique relies on assumptions that may not be satisfied when studying genomics datasets. For example, random CV (RCV) assumes that a randomly selected set of samples, the test set, well represents unseen data. This assumption doesn't hold true where samples are obtained from different experimental conditions, and the goal is to learn regulatory relationships among the genes that generalize beyond the observed conditions. In this study, we investigated how the CV procedure affects the assessment of supervised learning methods used to learn gene regulatory networks (or in other applications). We compared the performance of a regression-based method for gene expression prediction estimated using RCV with that estimated using a clustering-based CV (CCV) procedure. Our analysis illustrates that RCV can produce over-optimistic estimates of the model's generalizability compared to CCV. Next, we defined the 'distinctness' of test set from training set and showed that this measure is predictive of performance of the regression method. Finally, we introduced a simulated annealing method to construct partitions with gradually increasing distinctness and showed that performance of different gene expression prediction methods can be better evaluated using this method.

Supplemental Content

Full text links

Icon for Nature Publishing Group Icon for PubMed Central
Loading ...
Support Center