Although this project gives us insight into the potential of peer review of search strategies for systematic reviews, it is important to remember some limitations of the current study. What started as a small sample suffered significant dropout. About a third of our peer reviewers failed to complete all reviews. As such, our conclusions are based on a small sample of respondents.

Similarly, because we limited the number of search strategies reviewed, the sample of original searchers answering our questionnaire was severely limited. It is also clear from the text comments that the question “did this change your search” was ambiguous. Additional research focusing on the original authors of search strategies could give different results with a larger sample size and more clearly worded questionnaire.

While the results of this study suggest that if a formal peer review process is to be valuable then it would need to be both timely and timed for a window of opportunity immediately prior to the finalization of the protocol. Even if a formal peer review process is not implemented, the PRESS instrument could be useful in informal peer review or even self review. If review of search strategies is to take place then these results suggest that the use of the PRESS instrument would cut down the time taken, increase the likelihood of response and be more effective in identifying actual errors in search strategies. The most recent version of the PRESS instrument (published after the initiation of this project)21,22, while not significantly differing from the version used in this study, contains additional information to guide the review and assist in the detection of errors. Any subsequent peer review research or implementation should use the most current version of the PRESS instrument.

Additionally, the content of the reviews indicates that there are several search tactics for which there is no consensus, and further research could help us to understand variation in practice around such issues as limits, searching for observational studies, and searching for outcomes and comparators. The process of reviewing other searchers’ work can bring these issues to light, and a peer review-like process could be used to start investigations and discussions of what techniques work and why. While this project focused on pharmacological treatment topics, as they are the more common type of effectiveness review within the AHRQ, these also represent the searches that are the simplest to conduct, since studies of this type tend to be well indexed. While we do not know if these results would be generalizable to more diffuse topics such as psychosocial interventions or health systems research, peer review could prove to be especially helpful for topics where there are no standardized techniques for translating topics into searches.

Finally, many of the reviewers commented on the difficulty of reading the search strategies as currently presented. The EHC Program currently has no standards for reporting search strategies, and there is no recognized standard for reporting search strategies23. As with the issue of timing of peer review of search strategies, the convenience of using the publically posted search protocol for this project, may have worked against the utility of peer review. The search strategies were not reported with the idea of peer review in mind. Recently, Craven24 and Niederstadt25 have both suggested standards for the reporting of search strategies that facilitate review of the strategies. Adopting standard of reporting designed to facilitate review may make it easier to review search strategies both internally and when reported to the public.