Format

Send to

Choose Destination
J Am Med Inform Assoc. 2015 May;22(3):640-8. doi: 10.1136/amiajnl-2014-002901. Epub 2014 Oct 23.

Using the wisdom of the crowds to find critical errors in biomedical ontologies: a study of SNOMED CT.

Author information

1
Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA Biomedical Informatics Training Program, Stanford University, Stanford, California, USA.
2
Biomedical Informatics Training Program, Stanford University, Stanford, California, USA Faculty of Medicine, University of Calgary, Calgary, Canada.
3
Biomedical Informatics Training Program, Stanford University, Stanford, California, USA.
4
Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA.
5
School of Computer Science, University of Manchester, Manchester, UK.
6
Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA Google Inc., Mountain View, California, USA.

Abstract

OBJECTIVES:

The verification of biomedical ontologies is an arduous process that typically involves peer review by subject-matter experts. This work evaluated the ability of crowdsourcing methods to detect errors in SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and to address the challenges of scalable ontology verification.

METHODS:

We developed a methodology to crowdsource ontology verification that uses micro-tasking combined with a Bayesian classifier. We then conducted a prospective study in which both the crowd and domain experts verified a subset of SNOMED CT comprising 200 taxonomic relationships.

RESULTS:

The crowd identified errors as well as any single expert at about one-quarter of the cost. The inter-rater agreement (κ) between the crowd and the experts was 0.58; the inter-rater agreement between experts themselves was 0.59, suggesting that the crowd is nearly indistinguishable from any one expert. Furthermore, the crowd identified 39 previously undiscovered, critical errors in SNOMED CT (eg, 'septic shock is a soft-tissue infection').

DISCUSSION:

The results show that the crowd can indeed identify errors in SNOMED CT that experts also find, and the results suggest that our method will likely perform well on similar ontologies. The crowd may be particularly useful in situations where an expert is unavailable, budget is limited, or an ontology is too large for manual error checking. Finally, our results suggest that the online anonymous crowd could successfully complete other domain-specific tasks.

CONCLUSIONS:

We have demonstrated that the crowd can address the challenges of scalable ontology verification, completing not only intuitive, common-sense tasks, but also expert-level, knowledge-intensive tasks.

KEYWORDS:

SNOMED CT; biomedical ontology; crowdsourcing; ontology engineering

PMID:
25342179
PMCID:
PMC5566196
DOI:
10.1136/amiajnl-2014-002901
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Silverchair Information Systems Icon for PubMed Central
Loading ...
Support Center