Table 1Grading the strength of a body of evidence: required domains and their definitions

DomainDefinition and ElementsScore and Application
Risk of BiasRisk of bias is the degree to which the included studies for a given outcome or comparison have a high likelihood of adequate protection against bias (i.e., good internal validity), assessed through two main elements:
  • Study design (e.g., RCTs or observational studies)
  • Aggregate quality of the studies under consideration.
Information for this determination comes from the rating of quality (good/fair/poor) done for individual studies
Use one of three levels of aggregate risk of bias:
  • Low risk of bias
  • Medium risk of bias
  • High risk of bias
ConsistencyThe principal definition of consistency is the degree to which reported effect sizes from included studies appear to have the same direction of effect. This can be assessed through two main elements:
  • Effect sizes have the same sign (that is, are on the same side of “no effect”)
  • The range of effect sizes is narrow.
Use one of three levels of consistency:
  • Consistent (i.e., no inconsistency)
  • Inconsistent
  • Unknown or not applicable (e.g., single study)
As noted in the text, single-study evidence bases (even megatrials) cannot be judged with respect to consistency. In that instance, use “Consistency unknown (single study).”
DirectnessThe rating of directness relates to whether the evidence links the interventions directly to health outcomes. For a comparison of two treatments, directness implies that head-to-head trials measure the most important health or ultimate outcomes. Two types of directness, which can coexist , may be of concern: Evidence is indirect if:
  • It uses intermediate or surrogate outcomes instead of health outcomes. In this case, one body of evidence links the intervention to intermediate outcomes and another body of evidence links the intermediate to most important (health or ultimate) outcomes.
  • It uses two or more bodies of evidence to compare interventions A and B – that is, studies of A versus placebo and B versus placebo, or studies of A versus C and B versus C but not A versus B.
Indirectness always implies that more than one body of evidence is required to link interventions to the most important health outcomes.

Directness may be contingent on the outcomes of interest. EPC authors are expected to make clear the outcomes involved when assessing this domain.
Score dichotomously as one of two levels directness
  • Direct
  • Indirect
If indirect, specify which of the two types of indirectness account for the rating (or both, if that is the case) --namely, use of intermediate/ surrogate outcomes rather than health outcomes, and use of indirect comparisons. Comment on the potential weaknesses caused by, or inherent in, the indirect analysis. The EPC should note if both direct and indirect evidence was available, particularly when indirect evidence supports a small body of direct evidence.
PrecisionPrecision is the degree of certainty surrounding an effect estimate with respect to a given outcome (i.e., for each outcome separately)

If a meta-analysis was performed, this will be the confidence interval around the summary effect size.
Score dichotomously as one of two levels of precision:
  • Precise
  • Imprecise
A precise estimate is an estimate that would allow a clinically useful conclusion. An imprecise estimate is one for which the confidence interval is wide enough to include clinically distinct conclusions. For example, results may be statistically compatible with both clinically important superiority and inferiority (i.e., the direction of effect is unknown), a circumstance that will preclude a valid conclusion.

EPC = Evidence-based Practice Center

From: Methods

Cover of Off-Label Use of Atypical Antipsychotics: An Update
Off-Label Use of Atypical Antipsychotics: An Update [Internet].
Comparative Effectiveness Reviews, No. 43.
Maglione M, Maher AR, Hu J, et al.

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.