Format

Send to

Choose Destination
Methods Mol Biol. 2019;1910:723-745. doi: 10.1007/978-1-4939-9074-0_24.

Scalable Workflows and Reproducible Data Analysis for Genomics.

Author information

1
Enterome Bioscience, Paris, France.
2
Department of Genetics, Center for Molecular Medicine, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
3
BIMSB Scientific Bioinformatics Platform, Max Delbrück Center for Molecular Medicine, Berlin, Germany.
4
Common Workflow Language Project, Vilnius, Lithuania.
5
KEMRI Wellcome Trust Research Programme, Kilifi, Kenya.
6
Centre for Genomic Regulation (CRG), The Barcelona Institute for Science and Technology, Barcelona, Spain.
7
Life Technologies, Waltham, MA, USA.
8
Institute for Biostatistics and Informatics in Medicine and Ageing Research (IBIMA), Rostock University Medical Center, Rostock, Germany.
9
Laboratory of Nematology, Department of Plant Science, Wageningen University, Wageningen, The Netherlands.
10
Department of Genetics, Center for Molecular Medicine, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands. pjotr2018@thebird.nl.
11
Laboratory of Nematology, Department of Plant Science, Wageningen University, Wageningen, The Netherlands. pjotr2018@thebird.nl.

Abstract

Biological, clinical, and pharmacological research now often involves analyses of genomes, transcriptomes, proteomes, and interactomes, within and between individuals and across species. Due to large volumes, the analysis and integration of data generated by such high-throughput technologies have become computationally intensive, and analysis can no longer happen on a typical desktop computer.In this chapter we show how to describe and execute the same analysis using a number of workflow systems and how these follow different approaches to tackle execution and reproducibility issues. We show how any researcher can create a reusable and reproducible bioinformatics pipeline that can be deployed and run anywhere. We show how to create a scalable, reusable, and shareable workflow using four different workflow engines: the Common Workflow Language (CWL), Guix Workflow Language (GWL), Snakemake, and Nextflow. Each of which can be run in parallel.We show how to bundle a number of tools used in evolutionary biology by using Debian, GNU Guix, and Bioconda software distributions, along with the use of container systems, such as Docker, GNU Guix, and Singularity. Together these distributions represent the overall majority of software packages relevant for biology, including PAML, Muscle, MAFFT, MrBayes, and BLAST. By bundling software in lightweight containers, they can be deployed on a desktop, in the cloud, and, increasingly, on compute clusters.By bundling software through these public software distributions, and by creating reproducible and shareable pipelines using these workflow engines, not only do bioinformaticians have to spend less time reinventing the wheel but also do we get closer to the ideal of making science reproducible. The examples in this chapter allow a quick comparison of different solutions.

KEYWORDS:

Big data; Bioconda; Bioinformatics; CWL; Cloud computing; Cluster computing; Common Workflow Language; Debian Linux; Evolutionary biology; GNU Guix; Guix Workflow Language; MPI; MrBayes; Nextflow; Parallelization; Snakemake; Virtual machine

PMID:
31278683
DOI:
10.1007/978-1-4939-9074-0_24
[Indexed for MEDLINE]

Supplemental Content

Full text links

Icon for Springer
Loading ...
Support Center