Send to

Choose Destination
See comment in PubMed Commons below
J Am Med Inform Assoc. 2012 Sep-Oct;19(5):713-8. doi: 10.1136/amiajnl-2012-000852. Epub 2012 May 12.

Development and evaluation of a crowdsourcing methodology for knowledge base construction: identifying relationships between clinical problems and medications.

Author information

School of Biomedical Informatics, The University of Texas Health Science Center at Houston (UTHealth), Houston, Texas 77030, USA.



We describe a novel, crowdsourcing method for generating a knowledge base of problem-medication pairs that takes advantage of manually asserted links between medications and problems.


Through iterative review, we developed metrics to estimate the appropriateness of manually entered problem-medication links for inclusion in a knowledge base that can be used to infer previously unasserted links between problems and medications.


Clinicians manually linked 231,223 medications (55.30% of prescribed medications) to problems within the electronic health record, generating 41,203 distinct problem-medication pairs, although not all were accurate. We developed methods to evaluate the accuracy of the pairs, and after limiting the pairs to those meeting an estimated 95% appropriateness threshold, 11,166 pairs remained. The pairs in the knowledge base accounted for 183,127 total links asserted (76.47% of all links). Retrospective application of the knowledge base linked 68,316 medications not previously linked by a clinician to an indicated problem (36.53% of unlinked medications). Expert review of the combined knowledge base, including inferred and manually linked problem-medication pairs, found a sensitivity of 65.8% and a specificity of 97.9%.


Crowdsourcing is an effective, inexpensive method for generating a knowledge base of problem-medication pairs that is automatically mapped to local terminologies, up-to-date, and reflective of local prescribing practices and trends.

[Indexed for MEDLINE]
Free PMC Article
PubMed Commons home

PubMed Commons

How to join PubMed Commons

    Supplemental Content

    Full text links

    Icon for Silverchair Information Systems Icon for PubMed Central
    Loading ...
    Support Center