Format

Send to

Choose Destination
Nat Methods. 2017 Nov;14(11):1063-1071. doi: 10.1038/nmeth.4458. Epub 2017 Oct 2.

Critical Assessment of Metagenome Interpretation-a benchmark of metagenomics software.

Author information

1
Faculty of Technology, Bielefeld University, Bielefeld, Germany.
2
Center for Biotechnology, Bielefeld University, Bielefeld, Germany.
3
Formerly Department of Algorithmic Bioinformatics, Heinrich Heine University (HHU), Duesseldorf, Germany.
4
Department of Computational Biology of Infection Research, Helmholtz Centre for Infection Research (HZI), Braunschweig, Germany.
5
Braunschweig Integrated Centre of Systems Biology (BRICS), Braunschweig, Germany.
6
Mathematics Department, Oregon State University, Corvallis, Oregon, USA.
7
Department of Pediatrics, University of California, San Diego, California, USA.
8
Department of Computer Science and Engineering, University of California, San Diego, California, USA.
9
German Center for Infection Research (DZIF), partner site Hannover-Braunschweig, Braunschweig, Germany.
10
Department of Plant Microbe Interactions, Max Planck Institute for Plant Breeding Research, Cologne, Germany.
11
Cluster of Excellence on Plant Sciences (CEPLAS).
12
Department of Environmental Science, Section of Environmental microbiology and Biotechnology, Aarhus University, Roskilde, Denmark.
13
Department of Microbiology, University of Copenhagen, Copenhagen, Denmark.
14
Department of Science and Environment, Roskilde University, Roskilde, Denmark.
15
Department of Energy, Joint Genome Institute, Walnut Creek, California, USA.
16
Pittsburgh Supercomputing Center, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA.
17
Center for Algorithmic Biotechnology, Institute of Translational Biomedicine, Saint Petersburg State University, Saint Petersburg, Russia.
18
Department of Microbiology and Ecosystem Science, University of Vienna, Vienna, Austria.
19
The ithree institute, University of Technology Sydney, Sydney, New South Wales, Australia.
20
Department of Computer Science, Research Center in Computer Science (CRIStAL), Signal and Automatic Control of Lille, Lille, France.
21
National Centre of the Scientific Research (CNRS), Rennes, France.
22
Department of Computational and Systems Biology, Genome Institute of Singapore, Singapore.
23
Department of Microbiology and Infection, Warwick Medical School, University of Warwick, Coventry, UK.
24
Department of Computer Science, University of Tuebingen, Tuebingen, Germany.
25
Intel Corporation, Hillsboro, Oregon, USA.
26
GenScale-Bioinformatics Research Team, Inria Rennes-Bretagne Atlantique Research Centre, Rennes, France.
27
Institute of Research in Informatics and Random Systems (IRISA), Rennes, France.
28
Department of Molecular Infection Biology, Helmholtz Centre for Infection Research, Braunschweig, Germany.
29
Algorizk-IT consulting and software systems, Paris, France.
30
Joint BioEnergy Institute, Emeryville, California, USA.
31
Graduate Institute of Biomedical Informatics, College of Medical Science and Technology, Taipei Medical University, Taipei, Taiwan.
32
Biological Systems and Engineering Division, Lawrence Berkeley National Laboratory, Berkeley, California, USA.
33
School of Computational Science and Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA.
34
Energy Engineering and Geomicrobiology, University of Calgary, Calgary, Alberta, Canada.
35
Department of Bioinformatics, Institute for Microbiology and Genetics, University of Goettingen, Goettingen, Germany.
36
Genevention GmbH, Goettingen, Germany.
37
Institute of Population Health Sciences, National Health Research Institutes, Zhunan Town, Taiwan.
38
Computational Science Research Center, San Diego State University, San Diego, California, USA.
39
Boyce Thompson Institute for Plant Research, New York, New York, USA.
40
Research Group Bioinformatics (NG4), Robert Koch Institute, Berlin, Germany.
41
Coordination for the Improvement of Higher Education Personnel (CAPES) Foundation, Ministry of Education of Brazil, Brasília, Brazil.
42
Center for Bioinformatics and Computational Biology, University of Maryland, College Park, Maryland, USA.
43
Department of Computer Science, University of Maryland, College Park, Maryland, USA.
44
School of Biology, Newcastle University, Newcastle upon Tyne, UK.
45
Leibniz Institute DSMZ-German Collection of Microorganisms and Cell Cultures, Braunschweig, Germany.
46
Institute of Microbiology, ETH Zurich, Zurich, Switzerland.

Abstract

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

PMID:
28967888
PMCID:
PMC5903868
DOI:
10.1038/nmeth.4458
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Nature Publishing Group Icon for PubMed Central
Loading ...
Support Center