Format

Send to

Choose Destination
Genome Biol. 2019 Sep 10;20(1):195. doi: 10.1186/s13059-019-1794-0.

Reproducible biomedical benchmarking in the cloud: lessons from crowd-sourced data challenges.

Author information

1
Biomedical Engineering, Oregon Health and Science University, Portland, OR, 97239, USA.
2
Sage Bionetworks, Seattle, WA, USA.
3
IBM Research, Yorktown Heights, NY, USA.
4
Department of Biochemistry and Molecular Biology, The Pennsylvania State University, University Park, State College, PA, USA.
5
University of California, Santa Cruz, Santa Cruz, CA, USA.
6
Institute for Computational Biomedicine, Heidelberg University, Faculty of Medicine and Heidelberg University Hospital, Bioquant, Heidelberg, Germany.
7
Joint Research Center for Computational Biomedicine, RWTH Aachen University, Faculty of Medicine, Aachen, Germany.
8
Ontario Institute for Cancer Research, Toronto, Canada.
9
Departments of Medical Biophysics and Pharmacology & Toxicology, University of Toronto, Toronto, Canada.
10
Departments of Human Genetics and Urology, University of California, Los Angeles, CA, USA.
11
Jonsson Comprehensive Cancer Centre, University of California, Los Angeles, CA, USA.
12
Institute for Precision Health, University of California, Los Angeles, CA, USA.
13
Sage Bionetworks, Seattle, WA, USA. justin.guinney@sagebase.org.
14
Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, 98195, USA. justin.guinney@sagebase.org.

Abstract

Challenges are achieving broad acceptance for addressing many biomedical questions and enabling tool assessment. But ensuring that the methods evaluated are reproducible and reusable is complicated by the diversity of software architectures, input and output file formats, and computing environments. To mitigate these problems, some challenges have leveraged new virtualization and compute methods, requiring participants to submit cloud-ready software packages. We review recent data challenges with innovative approaches to model reproducibility and data sharing, and outline key lessons for improving quantitative biomedical data analysis through crowd-sourced benchmarking challenges.

Supplemental Content

Full text links

Icon for BioMed Central Icon for PubMed Central
Loading ...
Support Center