NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Cover of Peer Review of Search Strategies

Peer Review of Search Strategies

Methods Research Reports

Investigators: , MLIS, MS and , MLIS.

Author Information
Rockville (MD): Agency for Healthcare Research and Quality (US); .
Report No.: 12-EHC068-EF

Structured Abstract


Many steps in the preparation of effectiveness and comparative effectiveness reviews (CERs) by the Agency for Healthcare Research and Quality’s Effective Health Care (EHC) Program involve outside review and input (e.g., by Key Informants, the Technical Expert Panel, Draft Report Peer Reviewers). However, development of bibliographic database search strategies is currently not consistently peer reviewed. An opportunity exists for the EHC Program to leverage its expert searchers across the 14 Evidence-based Practice Centers (EPCs) in the Program by implementing a process of peer review of search strategies.


Due to the number and frequency of observed errors in published search strategies uncovered in the research literature, an evaluation of the feasibility of instituting a peer review process is warranted.

Specific Aims:

  1. Evaluate whether the PRESS instrument or no-instrument (‘free-form’ evaluations) is preferred by Technical Expert Peer Reviewers (TEPRs) of search strategies.
  2. Evaluate the usefulness of a peer review process for database search strategies, that is, do peer reviews change search strategies.
  3. Evaluate the costs of implementing a formal peer review of search strategies program as a part of the review process.


We identified current research protocol phase CER search strategies to review, TEPRs from across the EHC Program, and the original expert searchers who worked on developing each of the search strategies. Each TEPR was assigned to either the control group who only wrote “free-form” reviews or to the group who initially wrote a “free-form” review, then trained using the PRESS Instrument, and finally completed the last review using the PRESS instrument. Original expert searchers were asked to comment on the reviews of their searches. One study researcher administered the peer review process and log time required to perform it, while the other, blinded study researcher analyzed qualitative and quantitative data derived from the reviews, as well as demographic information about the TEPR and original expert searchers.


As a whole, the group of people available to conduct and review search strategies in the EHC Program is professionally educated and very experienced. Of the 24 respondents to the demographic survey, 20 (83%) have master’s degrees in library science. Most have more than 10 years’ experience as a librarian or other information professional. Sixty-seven percent of respondents have more than 5 years experience contributing to systematic reviews, with 63 percent having contributed to more than 10 systematic reviews. These experienced searchers have a variety of relationships with the EPCs for whom they do searches. Of the 25 peer reviewers invited to participate, 24 completed the initial free-form round of reviews, and 15 completed either the PRESS review or a second free-form review as part of our control group. For the most part peer reviewers were positive about the review process, although many hesitated to incorporate the review process into their current workflow. All of the reviewers found the background material (systematic review protocol) helpful to the review. Of those who used the PRESS instrument, 82 percent (9) indicated that the instrument was helpful, 18 percent (2) reported that it was neither helpful nor limiting, and none of the reviewers indicated that the PRESS instrument was limiting. The PRESS instrument reviews contained more recommendations on the whole and in particular had more comments that could be termed error detection—specific comments about spelling or syntax indicating that a mistake had been made. In 97 percent of cases, the original searcher indicated that the comments did not cause them to alter their search strategies.


While the results of this study suggest that if a formal peer review process is to be valuable then it would need to be both timely and timed for a window of opportunity immediately prior to the finalization of the protocol. Even if a formal peer review process is not implemented, the PRESS instrument could be useful in informal peer review or even self review. If review of search strategies is to take place, then these results suggest that the use of the PRESS instrument would cut down the time taken, increase the likelihood of response and be more effective in identifying actual errors in search strategies. Additionally, the content of the reviews indicates that there are several search tactics for which there is no consensus, and further research could help us to understand variation in practice around such issues as limits, searching for observational studies, and searching for outcomes and comparators. The process of reviewing other searchers’ work can bring these issues to light, and a peer review-like process could be used to start investigations and discussions of what techniques work and why. Finally, many of the reviewers commented on the difficulty of reading the search strategies as currently presented. The EHC Program currently has no standards for reporting search strategies, and there is no recognized standard for reporting search strategies. Adopting standard of reporting designed to facilitate review may make it easier to review search strategies both internally and when reported to the public.

Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No. 290-2007-100572, Prepared by: Oregon Evidence-based Practice Center, Portland, OR

Suggested citation:

Relevo R, Paynter R. Peer Review of Search Strategies. Methods Research Report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-100572.) AHRQ Publication No. 12-EHC068-EF. Rockville, MD: Agency for Healthcare Research and Quality. June 2012.

This report is based on research conducted by the Oregon Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. HHSA-290-2007-100572). The findings and conclusions in this document are those of the authors, who are responsible for its content; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

None of the investigators has any affiliations or financial involvement that conflicts with the material presented in this report.


540 Gaither Road, Rockville, MD 20850; www​

Bookshelf ID: NBK98353PMID: 22787681


Related information

Similar articles in PubMed

See reviews...See all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...