Table 1Required domains: definitions and scores

DomainDefinition and ElementsScore and Application
Study LimitationsStudy limitations is the degree to which the included studies for a given outcome have a high likelihood of adequate protection against bias (i.e., good internal validity), assessed through two main elements:
  • Study design: Whether RCTs or other designs such as nonexperimental or observational studies.
  • Study conduct. Aggregation of ratings of risk of bias of the individual studies under consideration.
Score as one of three levels, separately by type of study design:
  • Low level of study limitations
  • Medium level of study limitations
  • High level of study limitations
DirectnessDirectness relates to (a) whether evidence links interventions directly to a health outcome of specific importance for the review, and (b) for comparative studies, whether the comparisons are based on head-to-head studies. The EPC should specify the comparison and outcome for which the SOE grade applies.
Evidence may be indirect in several situations such as:
  • The outcome being graded is considered intermediate (such as laboratory tests) in a review that is focused on clinical health outcomes (such as morbidity, mortality).
  • Data do not come from head-to-head comparisons but rather from two or more bodies of evidence to compare interventions A and B—e.g., studies of A vs. placebo and B vs. placebo, or studies of A vs. C and B vs. C but not direct comparisons of A vs. B.
  • Data are available only for proxy respondents (e.g., obtained from family members or nurses) instead of directly from patients for situations in which patients are capable of self-reporting and self-report is more reliable.
Indirectness always implies that more than one body of evidence is required to link interventions to the most important health outcome.
Score as one of two levels:
  • Direct
  • Indirect
If the domain score is indirect, EPCs should specify what type of indirectness accounts for the rating.
ConsistencyConsistency is the degree to which included studies find either the same direction or similar magnitude of effect. EPCs can assess this through two main elements:
  • Direction of effect: Effect sizes have the same sign (that is, are on the same side of no effect or a minimally important difference [MID])
  • Magnitude of effect: The range of effect sizes is similar. EPCs may consider the overlap of CIs when making this evaluation.
The importance of direction vs. magnitude of effect will depend on the key question and EPC judgments.
Score as one of three levels:
  • Consistent
  • Inconsistent
  • Unknown (e.g., single study)
Single-study evidence bases (including mega-trials) cannot be judged with respect to consistency. In that instance, use “Consistency unknown (single study).”
PrecisionPrecision is the degree of certainty surrounding an effect estimate with respect to a given outcome, based on the sufficiency of sample size and number of events.
  • A body of evidence will generally be imprecise if the optimal information size (OIS) is not met. OIS refers to the minimum number of patients (and events when assessing dichotomous outcomes) needed for an evidence base to be considered adequately powered.
  • If EPCs performed a meta-analysis, then EPCs may also consider whether the CI crossed a threshold for an MID.
  • If a meta-analysis is infeasible or inappropriate, EPCs may consider the narrowness of the range of CIs or the significance level of p-values in the individual studies in the evidence base.
Score as one of two levels:
  • Precise
  • Imprecise
A precise estimate is one that would allow users to reach a clinically useful conclusion (e.g., treatment A is more effective than treatment B).
Reporting BiasReporting bias results from selectively publishing or reporting research findings based on the favorability of direction or magnitude of effect. It includes:
  • Study publication bias, i.e., nonreporting of the full study.
  • Selective outcome reporting bias, i.e., nonreporting (or incomplete reporting) of planned outcomes or reporting of unplanned outcomes.
  • Selective analysis reporting bias, i.e., reporting of one or more favorable analyses for a given outcome while not reporting other, less favorable analyses.
Assessment of reporting bias for individual studies depends on many factors–e.g. availability of study protocols, unpublished study documents, and patient-level data. Detecting such bias is likely with access to all relevant documentation and data pertaining to a journal publication, but such access is rarely available.

Because methods to detect reporting bias in observational studies are less certain, this guidance does not require EPCs to assess it for such studies.
Score as one of two levels:
  • Suspected
  • Undetected
Reporting bias is suspected when:
  • Testing for funnel plot asymmetry demonstrates a substantial likelihood of bias, And/or
  • A qualitative assessment suggests the likelihood of missing studies, analyses, or outcomes data that may alter the conclusions from the reported evidence.
Undetected reporting bias includes all alternative scenarios.

CI = confidence internal; EPC = Evidence-based Practice Center; MID = minimally important difference; OIS = optimal information size

From: Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update

Cover of Methods Guide for Effectiveness and Comparative Effectiveness Reviews
Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet].

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.