• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of procamiaLink to Publisher's site
AMIA Annu Symp Proc. 2011; 2011: 1506–1513.
Published online Oct 22, 2011.
PMCID: PMC3243169

Search Filter Precision Can Be Improved By NOTing Out Irrelevant Content

Abstract

Background:

Most methodologic search filters developed for use in large electronic databases such as MEDLINE have low precision. One method that has been proposed but not tested for improving precision is NOTing out irrelevant content.

Objective:

To determine if search filter precision can be improved by NOTing out the text words and index terms assigned to those articles that are retrieved but are off-target.

Design:

Analytic survey.

Methods:

NOTing out unique terms in off-target articles and testing search filter performance in the Clinical Hedges Database.

Main Outcome Measures:

Sensitivity, specificity, precision and number needed to read (NNR).

Results:

For all purpose categories (diagnosis, prognosis and etiology) except treatment and for all databases (MEDLINE, EMBASE, CINAHL and PsycINFO), constructing search filters that NOTed out irrelevant content resulted in substantive improvements in NNR (over four-fold for some purpose categories and databases).

Conclusion:

Search filter precision can be improved by NOTing out irrelevant content.

Introduction

Individuals who attempt to find the best evidence by searching online using electronic databases such as MEDLINE search through over 20 million articles from over 5400 journals (http://www.ncbi.nlm.nih.gov/PubMed). A recent survey reported that clinicians have a high interest in using evidence-based information and claim frequent use of MEDLINE to find it [1]. Additionally, a recent study showed that clinicians’ online evidence use increases with patient admissions, supporting the hypothesis that clinicians’ use of evidence is related to direct patient care [2]. Because clinician use of the evidence is related to direct patient care and primary care clinicians have been shown to generate 2 questions for every 3 patients [3], easy and successful access to best evidence is needed to ensure best patient care. A recent qualitative study, however, found that 6 obstacles were encountered when attempting to answer clinical questions with published research evidence [4]. Two obstacles were the excessive time required to find information and the difficulty in selecting an optimal filter to search for information. Many clinicians who support the use of evidence for patient care in principle often believe that they do not have time to find and apply it in practice [5]. When they do try to find the best evidence, research has shown that practitioners do not search the medical literature very effectively [6]. It has been repeatedly shown that clinicians have difficulties with collecting and interpreting medical evidence, communicating accurately with one another, applying recommended medical procedures in a timely fashion, and keeping up-to-date with new advances in health care [57].

If large electronic bibliographic databases such as MEDLINE are to be helpful to end-users, they must be able to retrieve articles that are both scientifically sound and directly relevant to the health problem they are trying to solve, without missing key studies or retrieving excessive numbers of preliminary, irrelevant, outdated, or misleading reports. Our approach and the approach of others to these problems has been to develop search filters (“hedges”) to improve the retrieval of clinically relevant and scientifically sound study reports from MEDLINE and similar databases [818]. These search filters are created by testing the performance of search terms against a gold standard database and stringing top performing terms together using the Boolean “OR”. For example, a highly sensitive search filter for retrieving high quality treatment articles in Ovid MEDLINE is “clinical trial.mp. OR clinical trial.pt. OR random:.mp. OR tu.xs.” [8]. The use of search filters has been shown to increase the relevancy and reduce the volume of information retrieved. The search filters we developed are available for use in PubMed, Ovid, and EBSCO and are called Clinical Queries.

Low precision of these search filters remains a problem with even the best search filters available and increases the work of searchers to find appropriate articles. Various approaches have been proposed to increase precision including searching in journal subsets, adding content terms to the methodologic search filters, machine learning techniques, citation rank methods, and excluding irrelevant content by using the Boolean “NOT”. To our knowledge research has not been published on the impact of “NOTing” out irrelevant content.

The research question addressed in this paper is: Can the precision of the search filters available for use in PubMed, Ovid, and EBSCO Clinical Queries for treatment, diagnosis, prognosis, and etiology be improved by NOTing out the text words and index terms assigned to those articles that are retrieved by the search filter that are off-target (that is the “false positive” articles)? If we find that unique terms are applied to the off-target articles we could attempt to increase precision by “NOTing” out this content having no impact on the sensitivity of the search.

Methods

The McMaster University Clinical Hedges Database contains tagged citations from 170 clinically relevant journals from the publishing year 2000. Six highly trained and calibrated research assistants hand searched the 170 journals and classified each item within these journals for study type (original study, review, general interest paper, case report) and for those that were an original or review for purpose of the study (including treatment, diagnosis, prognosis and etiology). The original and review articles were further evaluated for methods and were considered to be methodology sound if all methods criteria specific to the purpose were met. Using this database and a compiled list of search terms (text words and index terms) search filters were developed and validated to maximize each of sensitivity (the proportion of relevant articles retrieved) and specificity (the proportion of non-relevant articles not retrieved) and to optimize the balance between sensitivity and specificity. Details of our methods for developing and validating these search filters has been previously published [19].

Our search filters were developed to detect original studies of treatment, diagnosis, prognosis and etiology. Using our broad (high sensitivity), narrow (high specificity) and balanced (best balance between sensitivity and specificity) search filters, we determined if precision would be increased by “NOTing” out content that is retrieved in cell B (see Table 1). When determining the operating characteristic of search filters in the Clinical Hedges Database interface each of the 4 cells in Table 1 contains the number of citations retrieved. Within the Clinical Hedges web interface we retrieved the citations noted in cells A and B. Using a program developed in-house we created a list and rank ordered all text words and index terms (from the list of terms tested when developing the search filters) that were unique to citations found in cell B when compared with citations found in cell A. The program that was developed in-house is similar to Microsoft Office Index function. Cell B citation terms needed to be unique when compared with cell A citations to ensure that sensitivity did not decrease when NOTing out content. These unique text words or index terms were appended as a string of “ORed” terms using the Boolean operator “Not”. For example, “randomized controlled trial.pt,mp. NOT (case reports.pt. OR case report:.tw.). The process of developing the ORed string of terms to be NOTed out was automated using software developed in-house. All terms unique to cell B citations were considered starting with the term assigned to most citations. A term was included in the ORed string if the number of citation in cell B decreased when the term was added.

Table 1.
Search Retrieval Cells

The operating characteristics of the revised search filters were determined by testing them in the Clinical Hedges Database and included calculating the sensitivity, specificity, precision (proportion of retrieval that is relevant), and number needed to read (NNR; 1 / precision). The NNR is an index of how many retrieved articles need to be sorted through to find one article that is on-target or relevant to the search.

Results

Databases

All searches were conducted in the Clinical Hedges Database which contains tagged citations from 170 clinical journals published in the year 2000. The MEDLINE version of this database contains a subset of 161 journals (those that were indexed in MEDLINE) with 49,028 tagged citations; EMBASE contains a subset of 135 journals with 43,954 tagged citations; CINAHL contains a subset of 75 journals with 8,493 tagged citations; and PsycINFO contains a subset of 64 journals with 6,301 tagged citations.

Methodologic search filters were originally developed and validated for the purpose categories treatment, diagnosis, prognosis and etiology for MEDLINE and EMBASE; treatment, prognosis and etiology for CINAHL; and treatment for PsycINFO. The numbers of methodologically sound studies in Clinical Hedges Database for each of the electronic databases are shown in Table 2.

Table 2.
Numbers of methodologically sound original studies by database and purpose category

MEDLINE – Treatment

The results for the sensitive, specific and balanced treatment search filters for MEDLINE are shown in Table 3. In all cases precision increased significantly after NOTing out unique search terms. The number needed to read (NNR) improved approximately 1.5-fold for the sensitive and balanced search filters. NNR did not change much for the specific search filter even though the improvement in precision was statistically significant.

Table 3.
Methodologic Search Filters for detecting Original Treatment Studies in MEDLINE

MEDLINE – Diagnosis

The results for the sensitive, specific and balanced diagnosis search filters are shown in Table 4. When searching using the highly sensitive MEDLINE diagnosis search filter precision increased from 1.1% to 4.5% after NOTing out an ORed string of 306 terms. NNR improved almost four-fold from 88 to 23. NNR improved over three-fold for the specific search filter, and almost four-fold for the balanced filter.

Table 4.
Methodologic Search Filters for detecting Original Diagnosis Studies in MEDLINE

MEDLINE – Prognosis

The results for the sensitive, specific and balanced prognosis search filters for MEDLINE are shown in Table 5. In all cases precision increased with improvements in the NNR of almost three-fold for the sensitive search filer, almost 3.5-fold for the specific filter, and over 2.5-fold for the balanced search filter.

Table 5.
Methodologic Search Filters for detecting Original Prognosis Studies in MEDLINE

MEDLINE – Etiology

The results for the sensitive, specific and balanced etiology search filters are shown in Table 6. When searching using the highly sensitive MEDLINE etiology search filter precision was 1.4%. After NOTing out an ORed string of 411 terms precision increased to 3.5% and the NNR improved almost 2.5-fold from 70 to 29. The NNR improved over 3.5-fold for the specific filter, and over 2.5-fold for the balanced search filter.

Table 6.
Methodologic Search Filters for detecting Original Etiology Studies in MEDLINE

EMBASE, CINAHL and PsycINFO

For all purpose categories and for all other electronic databases, EMBASE, CINAHL, and PsycINFO, results were very similar to that of MEDLINE (Table 7). Results were notable when using the database relevant highly sensitive, specific and balanced search filters for prognosis in CINAHL as NNRs improved between 6.5- and nine-fold.

Table 7.
Results for EMBASE, CINAHL and PsycINFO

Discussion

As indicated in our previous papers [818], when searching with the methodologic search filters alone we found that precision was generally low and therefore of concern. This was expected given the low proportion of relevant target articles for a given purpose in a very large, multipurpose database. This means that searchers will continue to need to spend time discarding irrelevant retrievals.

As reported in this paper, we set out to test whether precision would be improved by NOTing out unique search terms for the irrelevant retrieval. To our knowledge this is the first study to empirically test such an approach. We found, in most cases, two- to three-fold decreases in the absolute number of articles that would need to be sorted through to find articles that are on-target. In one case, sensitive prognosis search filter in CINAHL, the NNR improved nine-fold. This improvement is substantive and shows that combining empirically derived search filters for enhancing the retrieval by eliminating irrelevant content with search filters derived for enhancing the retrieval of scientifically sound, clinically relevant articles can have a profound impact on searching. Treatment was the only purpose category where NOTing out irrelevant content did not substantively impact the NNR. The reason for this is likely due to the already relatively high precision figures for these search filters compared with the other purpose categories.

Previous research has shown that physicians only have two minutes or less to find the literature they need [20, 21]. The improvements in NNR shown in this study may make it more feasible for clinicians to find the literature they need at the point of patient care. The NNRs reported in this study will likely be further improved when searching with the addition of content terms.

Other methods have been proposed and studied for improving search precision. For example, we have shown that adding content terms, specifically in the area of mental health, to the methodologic search filters using the Boolean “AND” can improve precision [22]. Machine learning techniques have also been studied to improve the precision of MEDLINE searching. Aphinyanaphongs and collegues [23] applied machine learning techniques to automatically identify high-quality, content-specific articles in internal medicine and compared their performance to our search filters available for use in PubMed Clinical Queries, the same search filters studied in this article. They found that polynomial support vector machine models resulted in at least a doubling of precision for the treatment and etiology search filters, similar precision for prognosis, and not as good performance for diagnosis. Aphinyanaphongs and colleagues [24] have also studied citation counts, impact factors, and non-machine learning methods of improving search precision and found that machine learning techniques outperform these alternative methodologies.

When comparing the results of this study to that of Aphinyanaphongs and colleagues, we report more than a doubling of precision for etiology, prognosis and diagnosis when searching in MEDLINE. Precision improved for treatment but did not double. Unlike machine learning models, the search filters developed in this study can be made available for use in the large electronic databases after they have been validated. The search filters are already in Boolean format which is needed when searching in MEDLINE and similar databases.

It should be noted that the search filters have been modified for use when searching for original studies. It is conceivable that NOTing out content may result in the exclusion of relevant systematic reviews. Additionally, these search filters are complex and will need to be readily available for searchers. We currently have funding from the Canadian Institutes of Health Research to develop a “Super Filters site” where we plan to make these search filters available for use after validating them in another data set.

Conclusion

Search filter precision can be improved by NOTing out irrelevant content.

Acknowledgments

This research was funded by the Canadian Institutes of Health Research. Chris Cotoi and Nicholas Hobson, programmers in the Health Information Unit at McMaster University, developed the software used in this project.

References

1. Sahapong S, Manmart L, Ayuvat D, Potisat S. Information use behavior of clinicians in evidence-based medicine process in Thailand. J Med Assoc Thai. 2009;92:435–41. [PubMed]
2. McCord G, Smucker WD, Selius BA, et al. Answering questions at the point of care: do residents practice EBM or manage information sources? Acad Med. 2007;82:298–303. [PubMed]
3. Green ML, Ciampi MA, Ellis PJ. Residents’ medical information needs in clinic: are they being met? Am J Med. 2000;109:218–23. [PubMed]
4. Ely JW, Osheroff JA, Chambliss ML, Embell MH, Rosenbaum ME. Answering physicians’ clinical questions: obstacles and potential solutions. BMJ. 2005;12:217–24. [PMC free article] [PubMed]
5. Ulvenes LV, Aaslad O, Nylenna M, Kristiansen IS. Norwegian physicians’ knowledge of and opinions about evidence-based medicine: cross-sectional study. PLoS One. 2009;4:e7828. [PMC free article] [PubMed]
6. Balas EA, Stockham MG, Mitchell JA, Sievert ME, Ewigman BG, Boren SA. In search of controlled evidence for health care quality improvement. J Med Syst. 1997;21:21–32. [PubMed]
7. Burke DT, Judelson AL, Schneider JC, DeVito MC, Latta D. Reading habits of practicing physiatrists. Am J Phys Med Rehabil. 2002;81:779–87. [PubMed]
8. Haynes RB, McKibbon KA, Wilczynski NL, Walter SD, Werre SR, Hedges Team Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey. BMJ. 2005;330:1179. Epub 2005 May 13. [PMC free article] [PubMed]
9. Haynes RB, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey. BMJ. 2004;328:1040. Epub 2004 Apr 8. [PMC free article] [PubMed]
10. Wilczynski NL, Haynes RB, Hedges Team Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytical survey. BMC Med. 2004 Jun 9;2:23. [PMC free article] [PubMed]
11. Wilczynski NL, Haynes RB, Hedges Team Developing optimal search strategies for detecting clinically sound causation studies in MEDLINE. AMIA Annu Symp Proc; 2003. pp. 719–23. [PMC free article] [PubMed]
12. Wong SS, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically sound treatment studies in EMBASE. J Med Libr Assoc. 2006;94:41–7. [PMC free article] [PubMed]
13. Wilczynski NL, Haynes RB, Hedges Team EMBASE search strategies for identifying methodologically sound diagnostic studies for use by clinicians and researchers. BMC Med. 2005 Mar 29;3:7. [PMC free article] [PubMed]
14. Wilczynski NL, Haynes RB. Optimal search strategies for detecting clinically sound prognostic studies in EMBASE: an analytic survey. J Am Med Inform Assoc. 2005;12:481–5. [PMC free article] [PubMed]
15. Haynes RB, Kastner M, Wilczynski NL, Hedges Team Developing optimal search strategies for detecting clinically relevant causation studies in EMBASE. BMC Med Inform Decis Mak. 2005 Mar 22;5(1):8. [PMC free article] [PubMed]
16. Wong SS, Wilczynski NL, Haynes RB. Optimal CINAHL search strategies for identifying therapy studies and review articles. J Nurs Scholarsh. 2006;38:194–9. [PubMed]
17. Walker-Dilks C, Wilczynski NL, Haynes RB, Hedges Team Cumulative Index to Nursing and Allied Health Literature search strategies for identifying methodologically sound causation and prognosis studies. Appl Nurs Res. 2008;21:98–103. [PubMed]
18. Eady AM, Wilczynski NL, Haynes RB. PsycINFO search strategies identified methodologically sound therapy studies and review articles for use by clinicians and researchers. J Clin Epidemol. 2008;336:34–40. [PMC free article] [PubMed]
19. Wilczynski NL, Morgan D, Haynes RB, Hedges Team An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Med Inform Decis Mak. 2005 Jun 21;5:20. [PMC free article] [PubMed]
20. Ely JW, Osheroff JA, Ebell MH, et al. Analysis of questions asked by family doctors regarding patient care. Br Med Assoc. 1999;319:358–61. [PMC free article] [PubMed]
21. Alper BS, Stevermer JJ, White DS, Ewigman B. Answering family physicians clinical questions using electronic medical databases. J Fam Pract. 2001;50:960–5. [PubMed]
22. Wilczynski NL, Haynes RB, Team Hedges Optimal search strategies for retrieving mental health content in MEDLINE: analytic survey. Ann Gen Psychiatry. 2006;5:4. [PMC free article] [PubMed]
23. Aphinyanaphongs Y, Tsamardinos I, Statnikov A, Hardin D, Aliferis CF. Text categorization models for high-quality article retrieval in internal medicine. J Am Med Inform Assoc. 2005;12:207–16. [PMC free article] [PubMed]
24. Aphinyanaphongs Y, Statnikov A, Aliferis CF. A comparison of citation metrics to machine learning filters for identification of high quality MEDLINE documents. J Am Med Inform Assoc. 2006;13:446–55. [PMC free article] [PubMed]

Articles from AMIA Annual Symposium Proceedings are provided here courtesy of American Medical Informatics Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...