Format

Send to

Choose Destination
J Biomed Inform. 2016 Apr;60:199-209. doi: 10.1016/j.jbi.2016.02.005. Epub 2016 Feb 10.

Is the crowd better as an assistant or a replacement in ontology engineering? An exploration through the lens of the Gene Ontology.

Author information

1
Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94305-5479, United States; Biomedical Informatics Training Program, Stanford University, Stanford, CA 94305-5479, United States.
2
Biomedical Informatics Training Program, Stanford University, Stanford, CA 94305-5479, United States.
3
Institute of Computational Health Sciences, University of California, San Francisco, San Francisco, CA 94143, United States.
4
Department of Pediatrics, Stanford University, Stanford, CA 94305-5479, United States.
5
Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA 91125, United States.
6
Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94305-5479, United States.
7
Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA 94305-5479, United States. Electronic address: musen@stanford.edu.

Abstract

Biomedical ontologies contain errors. Crowdsourcing, defined as taking a job traditionally performed by a designated agent and outsourcing it to an undefined large group of people, provides scalable access to humans. Therefore, the crowd has the potential to overcome the limited accuracy and scalability found in current ontology quality assurance approaches. Crowd-based methods have identified errors in SNOMED CT, a large, clinical ontology, with an accuracy similar to that of experts, suggesting that crowdsourcing is indeed a feasible approach for identifying ontology errors. This work uses that same crowd-based methodology, as well as a panel of experts, to verify a subset of the Gene Ontology (200 relationships). Experts identified 16 errors, generally in relationships referencing acids and metals. The crowd performed poorly in identifying those errors, with an area under the receiver operating characteristic curve ranging from 0.44 to 0.73, depending on the methods configuration. However, when the crowd verified what experts considered to be easy relationships with useful definitions, they performed reasonably well. Notably, there are significantly fewer Google search results for Gene Ontology concepts than SNOMED CT concepts. This disparity may account for the difference in performance - fewer search results indicate a more difficult task for the worker. The number of Internet search results could serve as a method to assess which tasks are appropriate for the crowd. These results suggest that the crowd fits better as an expert assistant, helping experts with their verification by completing the easy tasks and allowing experts to focus on the difficult tasks, rather than an expert replacement.

KEYWORDS:

Crowdsourcing; Gene Ontology; Ontology engineering

PMID:
26873781
PMCID:
PMC4836980
DOI:
10.1016/j.jbi.2016.02.005
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Elsevier Science Icon for PubMed Central
Loading ...
Support Center