Comparative Effectiveness Reviews are systematic reviews of existing research on the effectiveness, comparative effectiveness, and harms of different health care interventions. They provide syntheses of relevant evidence to inform real-world health care decisions for patients, providers, and policymakers. Strong methodologic approaches to systematic review improve the transparency, consistency, and scientific rigor of these reports. Through a collaborative effort of the Effective Health Care (EHC) Program, the Agency for Healthcare Research and Quality (AHRQ), the EHC Program Scientific Resource Center, and the AHRQ Evidence-based Practice Centers have developed a Methods Guide for Comparative Effectiveness Reviews. This Guide presents issues key to the development of Comparative Effectiveness Reviews and describes recommended approaches for addressing difficult, frequently encountered methodological issues.

The Methods Guide for Comparative Effectiveness Reviews is a living document, and will be updated as further empiric evidence develops and our understanding of better methods improves. Comments and suggestions on the Methods Guide for Comparative Effectiveness Reviews and the Effective Health Care Program can be made at www.effectivehealthcare.ahrq.gov.

Contents

• Comparing Medical Interventions: AHRQ and the Effective Health Care Program
  Jean Slutsky, David Atkins, Stephanie Chang, and Beth A Collins Sharp.
  Published: November 18, 2008.
  • References
• Principles in Developing and Applying Guidance
  Mark Helfand and Howard Balshem.
  Published: August 8, 2009.
  • Key Points
  • Introduction
  • Background and History
  • Principles—Developing Guidance
  • Principles—Conducting Comparative Effectiveness Reviews
  • Future Development of the Methods Guide
  • References
• Identifying, Selecting, and Refining Topics
  Evelyn P Whitlock, Sarah A Lopez, Stephanie Chang, Mark Helfand, Michelle Eder, and Nicole Floyd.
  Published: April 27, 2009.
  • Key Points
  • Introduction
  • Guiding Principles for Identifying and Selecting Topics
  • Processes for Identifying and Selecting Systematic Reviews
  • Principles and Processes for Refining Selected Topics
  • Challenges
  • References
• Developing and Selecting Topic Nominations for Systematic Reviews
  Michelle Eder, Alisha Feightner, Elizabeth Webber, Janelle Guirguis-Blake, and Evelyn P. Whitlock.
  Published: November 19, 2012.
  • Introduction
  • Topic Nomination Development
  • Topic Selection
  • Future Directions
  • Conclusion
  • References
  • Glossary
• The Refinement of Topics for Systematic Reviews: Lessons and Recommendations From the Effective Health Care Program
  Investigators: David I Buckley, Mohammed Ansari, Mary Butler, Clara Williams, and Christine Chang.
  Published: January 24, 2013.
  • Introduction
  • Methods
  • Results
  • Conclusion
  • References
  • Abbreviations
  • Glossary
  • Appendix A. EPC Topic Refinement Document
  • Appendix B. Example of Selected Aspects of a Topic Refinement
• Finding Evidence for Comparing Medical Interventions
  Rose Relevo and Howard Balshem.
  Published: January 5, 2011.
  • Key Points
  • Introduction
  • Regulatory and Clinical Trials Searching
  • Scientific Information Packets: Requests to Industry
  • Developing the Published Literature Search
  • Strategies for Finding Observational Studies
  • Specialized Database Searching
  • Using Key Articles
  • Hand Searching Journals
  • Corresponding With Researchers
  • Updating and Reporting the Search Strategy
  • Concluding Remarks
  • References
  • Appendix A. Techniques for Observational Studies and/or Harms
  • Appendix B. Specialized Databases
  • Appendix C. CONSORT-Style Flow Diagram of Literature Search, Annotated
• Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program
  Howard Balshem, Adrienne Stevens, Mohammed Ansari, Susan Norris, Devan Kansagara, Tatyana Shamliyan, Roger Chou, Mei Chung, David Moher, and Kay Dickersin.
  Published: November 18, 2013.
  • Key Points
  • Introduction
  • Methods
  • Background
  • Sources of Evidence
  • Guidance on the Process of Assessing for Selective Reporting of Outcomes and Analyses
  • Reporting the Search Strategy and Results
  • References
  • Abbreviations
  • Peer Reviewers
  • Appendix A. Definitions of the Data Elements From ClinicalTrials.gov (www.clinicaltrials.gov)
  • Appendix B. Definitions of the Data Elements From the World Health Organization International Clinical Trials Registry Platform (www.who.int/ictrp/network/trds/en/index.html)
  • Appendix C. FDA Web Site – Drugs@FDA
  • Appendix D. Grant Databases
• Avoiding Bias in Selecting Studies
  Marian McDonagh, Kim Peterson, Parminder Raina, Stephanie Chang, and Paul Shekelle.
  Published: February 20, 2013.
  • Key Points
  • Background
  • Types of Potential Biases in Selecting Studies
  • Guidance for Setting Inclusion Criteria To Avoid Bias in Selecting Studies
  • Discussion
  • Conclusion
  • References
• Selecting Observational Studies for Comparing Medical Interventions
  Susan Norris, David Atkins, Wendy Bruening, Steven Fox, Eric Johnson, Robert Kane, Sally C Morton, Mark Oremus, Maria Ospina, Gurvaneet Randhawa, Karen Schoelles, Paul Shekelle, and Meera Viswanathan.
  Published: June 14, 2010.
  • Key Points
  • Introduction
  • Decision Framework
  • Discussion
  • Conclusions
  • References
• Assessing the Risk of Bias in Systematic Reviews of Health Care Interventions
  Investigators: Meera Viswanathan, Carrie D. Patnode, Nancy D. Berkman, Eric B. Bass, Stephanie Chang, Lisa Hartling, M. Hassan Murad, Jonathan R. Treadwell, and Robert L. Kane.
  Published: December 13, 2017.
  • Acknowledgments
  • Key Recommendations
  • Introduction
  • Terminology
  • Constructs To Include and Exclude From Risk-of-Bias Assessment
  • Stages in Assessing the Risk of Bias of Studies
  • Identifying, Selecting, and Assessing Categories of Risk of Bias
  • Tools for Assessing Risk of Bias
  • Direction and Magnitude of Bias
  • Assessing the Credibility of Subgroup Analyses
  • Assessing the Risk of Bias for Harms
  • Assessing the Credibility of Existing Systematic Reviews
  • Reporting the Risk of Bias
  • Conclusion
  • References
• Assessing the Applicability of Studies When Comparing Medical Interventions
  David Atkins, Stephanie Chang, Gerald Gartlehner, David I Buckley, Evelyn P Whitlock, Elise Berliner, and David Matchar.
  Published: December 30, 2010.
  • Key Points
  • Introduction
  • General Guidance
  • Four Specific Steps
  • Limitations of This Approach
  • Conclusions
  • References
  • Appendix A. Example Adapted from Comparative Effectiveness Review of Therapies for Clinically Localized Prostate Cancer
• Prioritization and Selection of Harms for Inclusion in Systematic Reviews
  Investigators: Roger Chou, William L. Baker, Lionel L. Bañez, Suchitra Iyer, Evan R. Myers, Sydne Newberry, Laura Pincock, Karen A. Robinson, Lyndzie Sardenga, Nila Sathe, Stacey Springs, and Timothy J. Wilt.
  Published: February 2, 2018.
  • Introduction
  • Methods
  • Results
  • Recommendations
  • Discussion
  • References
  • Appendix A. Guidance From Prior Chapter
  • Appendix B. Key Informant Invite
  • Appendix C. Key Informant Interview Guide
  • Appendix D. Key Informant Interview Themes
  • Appendix E. Key Informants
  • Appendix F. Review of EPC Reports
• Assessing Harms When Comparing Medical Interventions
  Roger Chou, Naomi Aronson, David Atkins, Afisi S Ismaila, Pasqualina Santaguida, David H Smith, Evelyn Whitlock, Timothy J Wilt, and David Moher.
  Published: November 18, 2008.
  • Key Points
  • Introduction
  • Identifying Harms to Be Evaluated
  • Terminology
  • Sources of Evidence on Harms
  • Assessing Risk of Bias (Quality) of Harms Reporting
  • Synthesizing Evidence on Harms
  • Reporting Evidence on Harms
  • Summary
  • References
• Quantitative Synthesis—An Update
  Investigators: Sally C. Morton, M. Hassan Murad, Elizabeth O’Connor, Christopher S. Lee, Marika Booth, Benjamin W. Vandermeer, Jonathan M. Snowden, Kristen E. D’Anci, Rongwei Fu, Gerald Gartlehner, Zhen Wang, and Dale W. Steele.
  Published: February 23, 2018.
  • Introduction
  • Chapter 1. Decision to Combine Trials
    Elizabeth O’Connor, Kristen E. D’Anci, and Jonathan M. Snowden.
  • Chapter 2. Optimizing Use of Effect Size Data
    Christopher S. Lee and Benjamin W. Vandermeer.
  • Chapter 3. Choice of Statistical Model for Combining Studies
    Rongwei Fu, Marika Booth, and Dale W. Steele.
  • Chapter 4. Quantifying, Testing, and Exploring Statistical Heterogeneity
    Christopher S. Lee.
  • Chapter 5. Network Meta-Analysis (Mixed Treatment Comparisons/Indirect Comparisons)
    M. Hassan Murad, Gerald Gartlehner, Rongwei Fu, and Zhen Wang.
  • Future Research Suggestions
  • References
• Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update
  Nancy D Berkman, Kathleen N Lohr, Mohammed Ansari, Marian McDonagh, Ethan Balk, Evelyn Whitlock, James Reston, Eric Bass, Mary Butler, Gerald Gartlehner, Lisa Hartling, Robert Kane, Melissa McPheeters, Laura Morgan, Sally C Morton, Meera Viswanathan, Priyanka Sista, and Stephanie Chang.
  Published: November 18, 2013.
  • Introduction
  • Major Steps in Grading Strength of Evidence
  • Discussion
  • Conclusions
  • References
  • Acknowledgments
  • Peer Reviewers
  • Appendix A. A Tool for Evaluating the Risk of Reporting Bias
  • Appendix B. Grading Strength of Evidence: Decisionmaking Examples
• Using Existing Systematic Reviews To Replace De Novo Processes in Conducting Comparative Effectiveness Reviews
  C Michael White, Stanley Ip, Melissa McPheeters, Tim S Carey, Roger Chou, Kathleen N Lohr, Karen Robinson, Kathryn McDonald, and Evelyn Whitlock.
  Published: October 5, 2009.
  • Key Points
  • Introduction and Rationale
  • Locating Existing Systematic Reviews
  • Assessing the Relevance of Existing Systematic Reviews
  • Assessing the Quality of Relevant Systematic Reviews
  • Reporting Methods and Results
  • Discussion: Reiterate Justification for Using Existing Systematic Reviews
  • References
• Integrating Bodies of Evidence: Existing Systematic Reviews and Primary Studies
  Karen A Robinson, Roger Chou, Nancy D Berkman, Sydne J Newberry, Rongwei Fu, Lisa Hartling, Donna Dryden, Mary Butler, Michelle Foisy, Johanna Anderson, Makalapua Motu'apuaka, Rose Relevo, Jeanne-Marie Guise, and Stephanie Chang.
  Published: February 26, 2015.
  • Introduction
  • Methods
  • Results
  • Recommendations
  • Discussion
  • References
  • Appendix A. Interview Guide
  • Appendix B. Interview Themes
• Updating Comparative Effectiveness Reviews: Current Efforts in AHRQ’s Effective Health Care Program
  Alexander Tsertsvadze, Margaret Maglione, Roger Chou, Chantelle Garritty, Craig Coleman, Linda Lux, Eric Bass, Howard Balshem, and David Moher.
  Published: July 27, 2011.
  • Key Points
  • Background
  • When To Update CERs
  • How To Update CERs
  • Current and Future Research Efforts
  • References
• Guidance for the Conduct and Reporting of Modeling and Simulation Studies in the Context of Health Technology Assessment
  Issa J Dahabreh, Thomas A Trikalinos, Ethan M Balk, and John B Wong.
  Published: October 18, 2016.
  • Editors' Foreword
  • Introduction
  • Guidance Development Process
  • Principles for Good Practice in Modeling and Simulation
  • Good-Practice Recommendations for Modeling and Simulation in the Context of Health Technology Assessment
  • Explanation and Elaboration of Recommendations for Modeling and Simulation
  • Concluding Remarks
  • Bibliographic Note
  • Acknowledgments
  • References