Abstract

We propose a framework for organizing and describing the rationale behind methodological recommendations, and for communicating one's confidence in them. We start by defining the background context in which the recommendations are made. We distinguish recommendations that are testable (in that their likelihood to hold can be informed by theoretical arguments or empirical data) from nontestable ones, which represent beliefs or assumptions that are not verifiable. Nontestable statements can be justified, but their validity cannot be demonstrated. Testable statements can be assessed in terms of the adequacy of their evidentiary basis.

Both testable and nontestable statements can be evaluated regarding their feasibility of implementation, the expected impact of following them versus not, and their congruence with the desired characteristics of the background context. Considering these four dimensions, one can indicate one's confidence (along a continuum) about how closely a methods recommendation should be followed: some recommendations may be perceived and communicated as mandatory items (minimum standards), while others as highly desirable but not mandatory items. Finally, giving specific methods guidance for addressing difficult or ill-defined problems can be premature pending more research or clearer problem specification. In such cases, describing the problem and laying out attributes of a satisfactory resolution can serve until actionable guidance can be offered.

We view the proposed framework strictly as a communication tool to describe rationale for the recommendations to the intended audience and not as a device to deduce the “correctness” of a recommendation. Nonetheless, application of the framework can facilitate the latter, because methodologists can most effectively and honestly critique recommendations whose rationale is transparent. We anticipate that this initial instantiation of the framework for making methods recommendations will evolve.

Contents

• Preface
• Acknowledgments
• Technical Expert Panel
• Peer and Public Reviewers
• Background
• Developing the Framework
• Description, Explanation, and Elaboration
  • Define the Background Context (Step 1)
  • Decompose the Recommendation Into Testable and Nontestable Parts (Step 2)
  • Describe Statements in Four Dimensions (Step 3)
  • Opine on Whether a Recommendation Constitutes a Mandatory Item or a Desirable (but not Mandatory) Item (Step 4)
• Discussion
• References

Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services¹, Contract No. 290-2007-10055-I. Prepared by: Tufts Evidence-based Practice Center, Boston, MA

Suggested citation:

Trikalinos TA, Dahabreh IJ, Wallace BC, Schmid CH, Lau J. Towards a Framework for Communicating Confidence in Methodological Recommendations for Systematic Reviews and Meta-Analyses. Methods Research Report. (Prepared by Tufts Evidence-based Practice Center under Contract No. 290-2007-10055-I). AHRQ Publication No. 13-EHC119-EF. Rockville, MD: Agency for Healthcare Research and Quality; September 2013. www.effectivehealthcare.ahrq.gov/reports/final.cfm.

This report is based on research conducted by the Tufts Evidence-based Practice Center, (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10055-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

None of the investigators have any affiliations or financial involvement that conflicts with material presented in this report.

1

540 Gaither Road, Rockville, MD 20850; www​.ahrq.gov