Format

Send to

Choose Destination
Molecules. 2017 Dec 1;22(12). pii: E2116. doi: 10.3390/molecules22122116.

An Interface for Biomedical Big Data Processing on the Tianhe-2 Supercomputer.

Author information

1
College of Computer, National University of Defense Technology, Changsha 410073, China. yangxi1016@nudt.edu.cn.
2
College of Computer, National University of Defense Technology, Changsha 410073, China. chengkun_wu@nudt.edu.cn.
3
College of Computer, National University of Defense Technology, Changsha 410073, China. kailu@nudt.edu.cn.
4
National Supercomputing Center of Guangzhou, Guangzhou 510006, China. fangl@genomics.cn.
5
National Supercomputing Center of Guangzhou, Guangzhou 510006, China. zhangyong2@genomics.cn.
6
National Supercomputing Center of Guangzhou, Guangzhou 510006, China. lishengkang@genomics.cn.
7
Beijing Genomics Institute (BGI) Shenzhen, Shenzhen 518083, China. guixin.guo@nscc-gz.cn.
8
School of Data and Computer Science, Sun Yat-Sen University, Guangzhou 510000, China. duyunfei@mail.sysu.edu.cn.

Abstract

Big data, cloud computing, and high-performance computing (HPC) are at the verge of convergence. Cloud computing is already playing an active part in big data processing with the help of big data frameworks like Hadoop and Spark. The recent upsurge of high-performance computing in China provides extra possibilities and capacity to address the challenges associated with big data. In this paper, we propose Orion-a big data interface on the Tianhe-2 supercomputer-to enable big data applications to run on Tianhe-2 via a single command or a shell script. Orion supports multiple users, and each user can launch multiple tasks. It minimizes the effort needed to initiate big data applications on the Tianhe-2 supercomputer via automated configuration. Orion follows the "allocate-when-needed" paradigm, and it avoids the idle occupation of computational resources. We tested the utility and performance of Orion using a big genomic dataset and achieved a satisfactory performance on Tianhe-2 with very few modifications to existing applications that were implemented in Hadoop/Spark. In summary, Orion provides a practical and economical interface for big data processing on Tianhe-2.

KEYWORDS:

Hadoop; Spark; Tianhe-2; big data; genomics big data

PMID:
29194413
PMCID:
PMC6149962
DOI:
10.3390/molecules22122116
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Multidisciplinary Digital Publishing Institute (MDPI) Icon for PubMed Central
Loading ...
Support Center