Format

Send to

Choose Destination
PLoS One. 2016 Sep 29;11(9):e0162721. doi: 10.1371/journal.pone.0162721. eCollection 2016.

SparkText: Biomedical Text Mining on Big Data Framework.

Ye Z1, Tafti AP2,3, He KY4, Wang K5,6, He MM1,2,7.

Author information

1
Biomedical Informatics Research Center, Marshfield Clinic Research Foundation, Marshfield, WI, 54449, United States of America.
2
Center for Human Genetics, Marshfield Clinic Research Foundation, Marshfield, WI, 54449, United States of America.
3
Department of Computer Science, University of Wisconsin-Milwaukee, Milwaukee, WI, 53211, United States of America.
4
Department of Epidemiology and Biostatistics, Case Western Reserve University, Cleveland, OH, 44106, United States of America.
5
Zilkha Neurogenetic Institute, University of Southern California, Los Angeles, CA, 90089, United States of America.
6
Department of Psychiatry, University of Southern California, Los Angeles, CA, 90089, United States of America.
7
Computation and Informatics in Biology and Medicine, University of Wisconsin-Madison, Madison, WI, 53706, United States of America.

Abstract

BACKGROUND:

Many new biomedical research articles are published every day, accumulating rich information, such as genetic variants, genes, diseases, and treatments. Rapid yet accurate text mining on large-scale scientific literature can discover novel knowledge to better understand human diseases and to improve the quality of disease diagnosis, prevention, and treatment.

RESULTS:

In this study, we designed and developed an efficient text mining framework called SparkText on a Big Data infrastructure, which is composed of Apache Spark data streaming and machine learning methods, combined with a Cassandra NoSQL database. To demonstrate its performance for classifying cancer types, we extracted information (e.g., breast, prostate, and lung cancers) from tens of thousands of articles downloaded from PubMed, and then employed Naïve Bayes, Support Vector Machine (SVM), and Logistic Regression to build prediction models to mine the articles. The accuracy of predicting a cancer type by SVM using the 29,437 full-text articles was 93.81%. While competing text-mining tools took more than 11 hours, SparkText mined the dataset in approximately 6 minutes.

CONCLUSIONS:

This study demonstrates the potential for mining large-scale scientific articles on a Big Data infrastructure, with real-time update from new articles published daily. SparkText can be extended to other areas of biomedical research.

Conflict of interest statement

We have the following interests: Kai Wang (K.W.) is a board member and shareholder of Tute Genomics, Inc. There are no patents, products in development or marketed products to declare. This does not alter our adherence to all the PLOS ONE policies on sharing data and materials.

Supplemental Content

Full text links

Icon for Public Library of Science Icon for PubMed Central
Loading ...
Support Center