• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of mlabJournal informationSubscribeSubmissions on the Publisher web siteCurrent issue of BMLA in PMCAlso see JMLA journal in PMC
Bull Med Libr Assoc. Oct 2001; 89(4): 346–352.
PMCID: PMC57963

Evidence-based practice: extending the search to find material for the systematic review

Diane Helmer, M.L.I.S., Information Specialist,1 Isabelle Savoie, M.D., MHA, Medical Consultant,1 Carolyn Green, B.H.Sc. (PT), M.Sc., Research Coordinator,1 and Arminée Kazanjian, Dr. Soc., Principal Investigator and Associate Director1

Abstract

Background: Cochrane-style systematic reviews increasingly require the participation of librarians. Guidelines on the appropriate search strategy to use for systematic reviews have been proposed. However, research evidence supporting these recommendations is limited.

Objective: This study investigates the effectiveness of various systematic search methods used to uncover randomized controlled trials (RCTs) for systematic reviews. Effectiveness is defined as the proportion of relevant material uncovered for the systematic review using extended systematic review search methods. The following extended systematic search methods are evaluated: searching subject-specific or specialized databases (including trial registries), hand searching, scanning reference lists, and communicating personally.

Methods: Two systematic review projects were prospectively monitored regarding the method used to identify items as well as the type of items retrieved. The proportion of RCTs identified by each systematic search method was calculated.

Results: The extended systematic search methods uncovered 29.2% of all items retrieved for the systematic reviews. The search of specialized databases was the most effective method, followed by scanning of reference lists, communicating personally, and hand searching. Although the number of items identified through hand searching was small, these unique items would otherwise have been missed.

Conclusions: Extended systematic search methods are effective tools for uncovering material for the systematic review. The quality of the items uncovered has yet to be assessed and will be key in evaluating the value of the systematic search methods.

INTRODUCTION

Cochrane-style systematic reviews are a value-added tool for health care providers, researchers, policy makers, and consumers. They provide rigorous and comprehensive summaries of research evidence on which appropriate policy or treatment decisions can be based [1]. Researchers, policy makers, or health care providers would have to spend fifty-seven hours a day reading new articles to keep up with the 15,000 new randomized controlled trials (RCTs) published annually [2]. The British Columbia Office of Health Technology Assessment (BCOHTA), the British Columbian representative of the Cochrane Collaboration, is one of the growing numbers of organizations involved in preparing and disseminating systematic reviews. Increasingly, librarians are participating in systematic reviews. Their expertise is used to formulate comprehensive search strategies and to uncover unindexed material for the systematic reviews.

While MEDLINE is considered the premier source for accessing clinical medical information, it has been established that searching MEDLINE alone generally fails to identify all possible studies for inclusion in systematic reviews [3, 4]. A number of guidelines have been produced for conducting systematic review searches that extend beyond MEDLINE [5–10]. Search protocols based on these guidelines generally require that researchers and librarians search the subject-specific and specialized databases (e.g., trial registries), search the gray literature, hand search key journals, scan reference lists, identify conference proceedings, and consult other researchers in the topic area.

While there is some evidence about the effectiveness of hand searching of published journals and trial registries, this evidence is mostly based on comparisons with MEDLINE searches [11–14]. In a pioneering study, the Register of Controlled Trials in Perinatal Medicine was used to assess the efficacy of a MEDLINE search. The authors concluded that a significant number of studies were missed when searching MEDLINE alone compared to the trials database [15]. Other studies discuss the value of trial registries but offer no empirical evidence as to their relative effectiveness [16–18].

Better evidence is available on the effectiveness of personal communication. A survey of obstetricians identified eighteen additional unpublished RCTs not in the Oxford Database of Perinatal Trials. Although the yield was a small percentage of all identified RCTs (0.008%), the survey was especially useful in identifying ongoing trials [19]. McGrath found that of 133 researchers contacted by letter, twenty-two individuals (17%) provided additional material; 53% of the contacted individuals did not reply; and 9% of the letters were returned to sender because of wrong addresses [20]. Issues such as the cooperation of those performing the trials, the perceived invalidity of unpublished trials, the tendency not to mention these trials, and the mobility of the research community have been discussed [21, 22].

This study investigates the effectiveness of various systematic search methods used to uncover additional RCTs for systematic reviews beyond those identified by a search of the mainstream databases (i.e., MEDLINE, Embase/Excerpta Medica). Effectiveness is defined as the proportion of relevant material identified for the systematic review using extended systematic search methods. The following extended systematic search methods are evaluated: searching subject-specific or specialized databases (including trial registries), hand searching of published journals, scanning reference lists, and communicating personally (contacting researchers for relevant material). The gold standard is the total number of RCTs identified by the major databases and the extended systematic search methods combined.

METHODS

To evaluate the effectiveness of the various extended systematic search methods, two different systematic reviews of RCTs conducted at the BCOHTA were selected: (1) a systematic review of RCTs of acupuncture in the treatment of addiction and (2) a systematic review of lipid-lowering drug therapy in the prevention and treatment of coronary heart disease. Using a prospective research design, the retrieved literature items were monitored and analyzed.

For each systematic review, major databases were searched using the OVID platform available through the University of British Columbia (UBC) library system. The time frame for each major database searched differed based on database availability at UBC. This study did not attempt to impose further time-frame limits on any of the major database searches. The major databases searched included MEDLINE, Embase/Excerpta Medica (Embase), HealthStar, and Current Contents. Additional major databases focusing on complementary medicine, such as the Complementary Alternative Medicine (CAM) and the Alternative Medicine (AMED) databases, were also searched to improve the search for the acupuncture topic. Details of the systematic search methods for each systematic review can be found in Tables 1 and 2.

Table thumbnail
Table 1 Extended search: lipid-lowering drugs

Specialized databases were searched using a combination of key words similar to the terms used in the major database search. A wide variety of databases were included in this category such as databases available free of charge over the Internet, Web catalogs, trial registries, and commercial databases (e.g., Dissertation Abstracts).

Key journals were hand searched, including both indexed and unindexed journals, located either through the local academic health sciences library or through Uncover Reveal Table of Contents. Hand-searched journals indexed in MEDLINE or Embase were selected based on the citation frequency from the results of the major database search. Specialized-subject journals, not indexed in MEDLINE or Embase, were sought out and hand searched for relevant material. The reference lists of items that met the inclusion criteria were scanned to locate additional items.

The Internet was used to facilitate the evaluation of personal communication. Various search engines and meta-indexes such as the Organising Medical Networked Information (OMNI) were used to locate organizations and researchers, who were then contacted by letter, telephone, or email. Directories identifying topic-specific research organizations were also used to identify relevant contacts. A list and bibliography of trials already located were sent to identified researchers and organizations, who were invited to forward additional relevant material and to advise of any unlisted trials.

Items meeting specific criteria for each project (i.e., human studies, studies with at least one outcome measure, randomized controlled trials or controlled trials, etc.) were retrieved, entered in a project-specific database using DB/TextWorks, and were included in the final analysis. Each item was coded to describe the systematic search method by which it was found. A second code describing the type of item uncovered (RCT, systematic review of RCTs, etc.) was also assigned to each record in the database. Coding was conducted independently by one librarian and by one researcher. The coding results were compared, and differences were resolved by discussion.

The proportion of RCTs identified by each systematic search method was calculated. The total number of RCTs was the sum of the RCTs uncovered though the major database search and the RCTs identified by the extended systematic search methods. Chi-square analyses of statistical significance were applied to test differences in proportions. The Epi Info statistical package was used [23]. The statistical significance level was set at α < 0.05.

RESULTS

The extended systematic search methods uncovered an additional 302 items for both systematic reviews beyond those identified by searches of the major databases. This represented 29.2% of the total number of retrieved items (Table 3). The extended systematic search methods identified a significantly higher proportion of the citations in the acupuncture than the lipid-lowering project (41.6% versus 26.1% P < 0.05).

Table thumbnail
Table 3 Overall results

The search of specialized databases was the most effective method of identification, providing a total of 96 citations (31.8%), followed by scanning of reference lists with 76 citations (25.2%) and communicating personally with an additional 72 citations (23.8%) (Table 4). Hand searching was the least effective method, though still yielding 19.2% of the citations. The search of specialized databases was the most effective method in the lipid-lowering project, whereas personal communication produced a larger number of citations in the acupuncture project (P < 0.05). Reference list scanning and hand searching were equally effective in both projects.

Table thumbnail
Table 4 Extended search: effectiveness of various methods

The main types of items identified by the extended systematic search were narrative reviews (36.4%), followed by RCTs (26.5%) and other (background) material (12.3%) (Table 5). Within the projects, most material took the form of narrative reviews and other primary data such as opinion pieces. Case studies were the most frequent type of material uncovered in the acupuncture project, whereas narrative reviews and RCTs were most frequently identified for the lipid-lowering project. In fact, a larger proportion of the extended systematic search citations referred to RCTs in the lipid-lowering than the acupuncture project (33.6% versus 8.2%; P < 0.05). However, a statistically significant larger proportion of the acupuncture literature uncovered through the extended systematic search was coded as other primary research (20.0% versus 3.2%; P < 0.05). There were no statistically significant differences in the proportions of systematic reviews and narrative reviews identified for each topic.

Table thumbnail
Table 5 Extended search: type of documents uncovered

DISCUSSION

This study shows that a significant amount of literature can be obtained for systematic reviews from the use of extended systematic search methods beyond the searching of major databases. The higher effectiveness of the extended systematic search method for the acupuncture as compared with the lipid-lowering project is perhaps not surprising. It may reflect the number of journals publishing alternative medicine studies, the number of studies evaluating alternative medicines, or the difficulty of publishing alternative medicine studies in medical journals. These factors make alternative medicine studies less likely to be indexed in databases such as MEDLINE and Embase.

The results of this study were based on the number of citations retrieved. Difficulties in acquiring the literature might have led to an underestimation of the effectiveness of the extended systematic search methods. The citations produced from MEDLINE and Embase database searches were retrieved with relative ease. However, some of the items identified through the extended systematic search were costly to acquire. In addition, these methods relied on the willingness of organizations and researchers to respond to the call for trials and other relevant information. Despite intensive and sustained efforts, not all personal communications produced responses.

This study did not examine overlap between extended methods. All extended methods were applied concurrently and at random. Items were coded in the database according to the method by which they were first found. Even though there was no prescribed order to the extended searches, the percentages of the extended methods used to uncover material for the systematic reviews might be affected.

Narrative reviews were the main type of documents unearthed by extended systematic search methods. This might result from the indexing methods used in the specialized databases, which could make it difficult to distinguish between RCTs and reviews. In addition, some sources such as reference lists did not provide abstracts, making it difficult to determine if a citation or topic would offer primary data or reviews of primary data and meet the inclusion criteria.

Material uncovered by the extended systematic search methods was more likely to be ordered and retrieved based on title information. Once retrieved, these items would be counted as part of this study, possibly affecting results in favor of the systematic search methods. It should be borne in mind, however, that many MEDLINE and all Current Contents records are indexed without abstracts and, therefore, likely to be ordered by title information alone.

The relative effectiveness of the extended systematic search methods also depended on the quality of the major database searches. While the project-specific search strategies applied to the major database searches in this study have not been formally evaluated, they were developed by librarians with extensive knowledge and understanding of the databases and Medical Subject Headings (MeSH). The use of MeSH has been shown to improve the effectiveness of MEDLINE searches [24].

In point of overlap, 46% and 52% of articles found by the extended systematic search methods for the acupuncture and lipid-lowering projects, respectively, were subsequently identified in MEDLINE, Embase, or HealthStar one year after the study was completed. Various factors explain these results. The National Library of Medicine (NLM) makes a concerted effort to get the major clinical journals indexed in MEDLINE as soon as possible. Foreign journal content, journals only in print form, and journal distribution patterns may all increase the lag time for indexing [25]. Therefore, journals containing appropriate and relevant material may not have been indexed at the time of the search. While a search of Current Contents corrects for part of this problem, a lag time effect still persists. In addition, items identified through conference abstracts or proceedings and not published or indexed at the time of the initial search of major databases may have been published in the year following the completion of this study.

The search of subject-specific and specialized databases (including trial registries) was the most effective search method. This might be attributable to the search of the Cochrane Library, which contained a well-established trial registry. The Cochrane Library was a key source of items for the lipid-lowering project.

Hand searching was the least effective method. This might reflect recent improvements in indexing. Since the early 1990s, international collaborative efforts have hand searched journals and identified more than 200,000 randomized or controlled trials not indexed as such in MEDLINE [26]. These references are submitted to NLM for proper indexing. However, records are submitted to NLM annually and, consequently, there may be an appreciable delay before RCTs are properly identified in MEDLINE. The Embase database is also currently seeking to improve the quality of indexing of RCTs. The effectiveness of hand searching may further diminish as indexing quality and the number of journals indexed improves.

It should be noted that the effectiveness of trial registries partially rests on hand searching efforts. The Cochrane Library Controlled Trials Register, for example, partly relies on hand searching of MEDLINE and non-MEDLINE journals. Hand searching may therefore be best used for the development trial registries rather than for individual systematic review projects.

Personal communication identified a greater number of RCTs for the acupuncture than for the lipid-lowering project. This might reflect the difficulty of publishing alternative medicine studies in mainstream medical journals. It should also be noted that while personal communication did not rank as the most effective method of finding additional studies, it identified unique items not found in the major databases.

Only a small number of items were found through the Internet. These took the form of narrative reviews, conference proceedings, and abstracts. This might reflect the practice of licensing agreements and copyright issues that often limit what researchers can make available through the Internet. New initiatives such as PubMed Central might reduce these barriers [27].

CONCLUSIONS

This study shows that the extended systematic search methods are effective tools for uncovering material for systematic reviews. This analysis suggests that, when available, trial registries and review groups may be the most effective systematic search methods. Where trial registries and review groups do not exist, communicating personally with researchers working in the field of interest and scanning of reference lists are the best alternatives.

It should be noted that the quality of the items uncovered has not been evaluated. Whether the quality of the items uncovered through the systematic search is such that they affect the conclusions of systematic reviews is unknown. This has yet to be assessed and will ultimately determine the value of the extended systematic search methods.

Table thumbnail
Table 2 Extended search: acupuncture in addiction treatment

REFERENCES

  • Sackett DL, Rosenberg WMC, Gray JA, Haynes RB, and Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996.  Jan 13; 312(7023):71–2. [PMC free article] [PubMed]
  • Davidoff F, Haynes B, Sackett D, and Smith R. Evidence based medicine. BMJ. 1995.  Apr 29; 310(6987):1085–6. [PMC free article] [PubMed]
  • Berry E, Kelly S, Hutton J, Harris KM, and Smith MA. Identifying studies for systematic reviews. an example from medical imaging. Int J Technol Assess. 2000.  Spring; 16(2):668–72. [PubMed]
  • Dickersin K, Scherre R, and Lefebvre C. Identifying relevant studies for systematic reviews. BMJ. 1994.  Nov 12; 309(6964):1286–91. [PMC free article] [PubMed]
  • Mulrow CD, Oxman AD. eds. Cochrane Collaboration handbook. In: The Cochrane Library. Issue 4. [database on disk and CDROM]. Oxford, U.K.: Update Software, 1997.
  • Goodman C. Literature searching and evidence interpretation for assessing health care practices. Stockholm, Sweden: Swedish Council on Technology Assessment in Health Care, 1993:16–32. [PubMed]
  • Goodman C. TA101: introduction to health care technology assessment. [Web document]. Bethesda, MD: The Lewin Group, 1998. <http://www.nlm.nih.gov/nichsr/ta101/ta101_c1.htm>.
  • Kirstensen FB, Horder M, and Poulsen PB. eds. Health technology assessment handbook. Copenhagen, Denmark: Danish Institute for Health Technology Assessment, 2001.
  • NHS Centre for Reviews and Dissemination (CRD). Undertaking systematic reviews of research on effectiveness: CRD's guidance for carrying out or commissioning reviews. 2d ed. York, U.K.: NHS Centre for Reviews and Dissemination, 2001.
  • Agences Nationale d'Accreditation et d'Evaluation en Santé. Guide d'analyse de la litterature et gradation des recommendations. Paris, France: l'Agence, 2000.
  • Dickersin K, Scherre R, and Lefebvre C. Identifying relevant studies for systematic reviews. BMJ. 1994.  Nov 12; 309(6964):1286–91. [PMC free article] [PubMed]
  • Adams CE, Power A, Frederick K, and Lefebvre C. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med. 1994.  Aug; 24(3):741–8. [PubMed]
  • Spoor P, Airey M, Bennett C, Greensill J, and Williams R. Use of the capture-recapture technique to evaluate the completeness of systematic literature searches. BMJ. 1996.  Aug 10; 313(7053):342–3. [PMC free article] [PubMed]
  • Jadad AR, McQuay HJ. Searching the literature. be systematic in your searching. BMJ. 1993.  Jul 3; 307(6895):66. [PMC free article] [PubMed]
  • Dickersin K, Hewitt P, Mutch L, Chalmers I, and Chalmers TC. Perusing the literature: comparison of MEDLINE searching with a perinatal trials database. Controlled Clin Trials. 1985.  Dec; 6(4):306–17. [PubMed]
  • Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986.  Oct; 4(10):1529–411. [PubMed]
  • Tonks A. Registering clinical trials. BMJ. 1999.  Dec 11; 319(7224):1565–8. [PMC free article] [PubMed]
  • McCray AT. Better access to information about clinical trials. Ann Intern Med. 2000.  Oct; 133(8):609–14. [PubMed]
  • Hetherington J, Dickersin K, Chalmers I, and Meinert CL. Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics. 1989.  Aug; 84(2):374–80. [PubMed]
  • McGrath J, Davies G, and Soares K. Writing to authors of systematic reviews elicited further data in 17% of cases. BMJ. 1998.  Feb 21; 316(7131):631. [PMC free article] [PubMed]
  • McGrath J, Davies G, and Soares K. Writing to authors of systematic reviews elicited further data in 17% of cases. BMJ. 1998.  Feb 21; 316(7131):631. [PMC free article] [PubMed]
  • Roberts I. Personal views. the private life of systematic reviews. BMJ. 1997.  Sep 13; 315(7109):686–7.
  • Centers for Disease Control and Prevention. Epi Info. [Web document]. Atlanta, GA: The Centers [cited 14 Sep 1999]. <http://www.cdc.gov/epo/epi/epiinfo.htm>.
  • Lowe HJ, Barnett GO. Understanding and using the Medical Subject Headings (MeSH) vocabulary to perform literature searches. JAMA. 1994.  Apr 13; 271(14):1103–8. [PubMed]
  • Auston I. NLM indexing [email]. Washington, DC: National Information Centre for Health Services Research, 1999. Personal communication.
  • Dickersin K, Hewitt P, Mutch L, Chalmers I, and Chalmers TC. Perusing the literature: comparison of MEDLINE searching with a perinatal trials database. Controlled Clin Trials. 1985.  Dec; 6(4):306–17. [PubMed]
  • Markovitz BP. Biomedicine's electronic publishing paradigm shift: copyright policy and PubMed Central. J Am Med Inform Assoc. 2000.  May–Jun; 7(3):222–9. [PMC free article] [PubMed]

Articles from Bulletin of the Medical Library Association are provided here courtesy of Medical Library Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...