NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Identifying Signals for Updating Systematic Reviews
Methods Research Reports
Paul G Shekelle, MD, PhD, Sydne J Newberry, PhD, Helen Wu, MPhil, MS, Marika Suttorp, MSPH, Aneesa Motala, BA, and Yee-Wei Lim, MD, PhD. Ethan M Balk, MD, MPH, Mei Chung, PhD, MPH, Winifred W Yu, MS, RD, Jounghee Lee, PhD, and James M Gaylor, BA. David Moher, PhD, Mohammed T Ansari, MBBS, MMedSc, MPhil, Rebecca Skidmore, BA(H), MLS, and Chantelle Garritty, MSc.
Author Information and AffiliationsStructured Abstract
Background:
Methods of assessing the need for systematic reviews to be updated have been published, but agreement among them is unclear.
Objectives:
To compare two methods for assessing the need to update an evidence review, using three evidence reports on the effects of omega-3 fatty acids on cancer, cognition and aging, and cardiovascular diseases (with separate analyses for fish oil and alpha-linolenic acid). The RAND method combines a targeted literature search with the assessments of content experts. The Ottawa method relies on a quantitative and qualitative assessment of the study results from a similar targeted search.
Data Sources:
A MEDLINE search was conducted on a limited set of journals, including five pivotal general medical journals and a small number of specialty journals, from 1 year prior to release of the original reports using their search strategies.
Methods:
The search results were screened using the original eligibility criteria. Study-level data and findings of existing systematic reviews, randomized controlled trials, and large observational studies addressing the original key questions were abstracted. Using the RAND method, we contacted experts—including members of the original technical expert panels and the original peer reviewers—and sought their opinions regarding the status of the original reports and any new references. The results of the literature reviews and expert opinions were combined to determine the need for updating based on predetermined criteria. Using a modification of the Ottawa method, new trial data were meta-analyzed with the original meta-analysis results. A quantitative signal for the need to update was based on statistical differences with the original meta-analyses. Qualitative signals, such as differences in characterizations of effectiveness, new information about harm, and caveats about the previously reported findings, were sought for outcomes without existing meta-analyses. Agreement between the RAND and Ottawa methods was assessed for each report with the kappa statistic.
Results:
Overall agreement between the two methods ranged from “nonexistent” (kappa = 0.19, for fish oil and cardiovascular disease) to “almost perfect” (kappa = 1.0 for cognitive function). Many of the disagreements between the methods were due to a situation where the original review had a Key Question with no evidence and some evidence was identified in the update. In these situations, the RAND method produced a positive signal for updating and Ottawa's method produced a negative signal. A sensitivity analysis that reclassified these situations as agreement between the two methods yielded much better estimates of agreement: for three of the four conditions, agreement was “substantial” to “almost perfect” and overall agreement was “substantial.”
Conclusions:
The RAND method and the modified Ottawa method agree reasonably well in their assessment of the need to update reviews. Both methods alone or in combination may be considered as appropriate tools. Future research would confirm these conclusions for a larger cohort of reviews and assess the predictive validity of the methods with actual updates.
Contents
Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No.: 290-2007-10062I (Southern California EPC), 290-2007-10055I (Tufts EPC), 290-2007-10059I (Ottawa EPC). Prepared by: The RAND Corporation, Southern California Evidence-based Practice Center, Santa Monica, CA; Tufts Evidence-based Practice Center, Tufts Medical Center, Boston, MA; University of Ottawa Evidence-based Practice Center, Ottawa, Canada.
Suggested citation:
Shekelle PG, Newberry SJ, Wu H, Suttorp M, Motala A, Lim Y-W, Balk EM, Chung M, Yu WW, Lee J, Gaylor JM, Moher D, Ansari MT, Skidmore R, Garritty C. Identifying Signals for Updating Systematic Reviews: A Comparison of Two Methods. Rockville (MD): Agency for Healthcare Research and Quality. June 2011. Methods Research Report. AHRQ Publication No. 11-EHC042-EF. Available at http://effectivehealthcare.ahrq.gov/.
This report is based on research conducted by the Southern California, Tufts, and University of Ottawa Evidence-based Practice Centers (EPCs) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10062-I [Southern California EPC]; 290-2007-10055I [Tufts EPC]; and 290-2007-10059I [University of Ottawa EPC]). The findings and conclusions in this document are those of the author(s), who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.
This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.
No investigators have any affiliations or financial involvement (e.g., employment, consultancies, honoraria, stock options, expert testimony, grants or patents received or pending, or royalties) that conflict with material presented in this report.
- 1
540 Gaither Road, Rockville, MD 20850; www
.ahrq.gov
- NLM CatalogRelated NLM Catalog Entries
- Review Updating Systematic Reviews[ 2007]Review Updating Systematic ReviewsShojania KG, Sampson M, Ansari MT, Ji J, Garritty C, Rader T, Moher D. 2007 Sep
- Review Low-Dose Aspirin for the Prevention of Morbidity and Mortality From Preeclampsia: A Systematic Evidence Review for the U.S. Preventive Services Task Force[ 2014]Review Low-Dose Aspirin for the Prevention of Morbidity and Mortality From Preeclampsia: A Systematic Evidence Review for the U.S. Preventive Services Task ForceHenderson JT, Whitlock EP, O'Conner E, Senger CA, Thompson JH, Rowland MG. 2014 Apr
- Review Screening for High Blood Pressure in Adults: A Systematic Evidence Review for the U.S. Preventive Services Task Force[ 2014]Review Screening for High Blood Pressure in Adults: A Systematic Evidence Review for the U.S. Preventive Services Task ForcePiper MA, Evans CV, Burda BU, Margolis KL, O’Connor E, Smith N, Webber E, Perdue LA, Bigler KD, Whitlock EP. 2014 Dec
- Two methods provide similar signals for the need to update systematic reviews.[J Clin Epidemiol. 2012]Two methods provide similar signals for the need to update systematic reviews.Chung M, Newberry SJ, Ansari MT, Yu WW, Wu H, Lee J, Suttorp M, Gaylor JM, Motala A, Moher D, et al. J Clin Epidemiol. 2012 Jun; 65(6):660-8. Epub 2012 Mar 29.
- Effectiveness of interventions that assist caregivers to support people with dementia living in the community: a systematic review.[Int J Evid Based Healthc. 2008]Effectiveness of interventions that assist caregivers to support people with dementia living in the community: a systematic review.Parker D, Mills S, Abbey J. Int J Evid Based Healthc. 2008 Jun; 6(2):137-72.
- Identifying Signals for Updating Systematic ReviewsIdentifying Signals for Updating Systematic Reviews
- Assessment of a Method to Detect Signals for Updating Systematic ReviewsAssessment of a Method to Detect Signals for Updating Systematic Reviews
- GeneReviews®GeneReviews®
- Noninvasive Diagnostic Tests for Breast AbnormalitiesNoninvasive Diagnostic Tests for Breast Abnormalities
- Systematic Review of the Literature Regarding the Diagnosis of Sleep ApneaSystematic Review of the Literature Regarding the Diagnosis of Sleep Apnea
Your browsing activity is empty.
Activity recording is turned off.
See more...