Format

Send to

Choose Destination
Database (Oxford). 2016 Apr 17;2016. pii: baw051. doi: 10.1093/database/baw051. Print 2016.

A crowdsourcing workflow for extracting chemical-induced disease relations from free text.

Author information

1
Department of Molecular and Experimental Medicine, the Scripps Research Institute, La Jolla, CA 92037, USA.
2
Research Programme on Biomedical Informatics (GRIB), IMIM, UPF, Barcelona, Spain.
3
Department of Molecular and Experimental Medicine, the Scripps Research Institute, La Jolla, CA 92037, USA asu@scripps.edu.

Abstract

Relations between chemicals and diseases are one of the most queried biomedical interactions. Although expert manual curation is the standard method for extracting these relations from the literature, it is expensive and impractical to apply to large numbers of documents, and therefore alternative methods are required. We describe here a crowdsourcing workflow for extracting chemical-induced disease relations from free text as part of the BioCreative V Chemical Disease Relation challenge. Five non-expert workers on the CrowdFlower platform were shown each potential chemical-induced disease relation highlighted in the original source text and asked to make binary judgments about whether the text supported the relation. Worker responses were aggregated through voting, and relations receiving four or more votes were predicted as true. On the official evaluation dataset of 500 PubMed abstracts, the crowd attained a 0.505F-score (0.475 precision, 0.540 recall), with a maximum theoretical recall of 0.751 due to errors with named entity recognition. The total crowdsourcing cost was $1290.67 ($2.58 per abstract) and took a total of 7 h. A qualitative error analysis revealed that 46.66% of sampled errors were due to task limitations and gold standard errors, indicating that performance can still be improved. All code and results are publicly available athttps://github.com/SuLab/crowd_cid_relexDatabase URL:https://github.com/SuLab/crowd_cid_relex.

PMID:
27087308
PMCID:
PMC4834205
DOI:
10.1093/database/baw051
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Silverchair Information Systems Icon for PubMed Central
Loading ...
Support Center