What is statistical heterogeneity, what is its effect on meta-analysis, and how should it be evaluated?

Statistical heterogeneity is “variation between trials in the underlying treatment effects being evaluated” (Higgins, Thompson, Deeks, et al., 2002) and is a consequence of clinical heterogeneity (e.g., differences among patients, interventions, outcomes) and methodological heterogeneity (e.g., differences in study designs, sources of bias).

Statistical heterogeneity among studies combined in meta-analysis may be detected if “variation in the results of the studies is above that compatible with chance alone” (Higgins, Thompson, Deeks, et al., 2002). The traditional test statistic (Cochran’s Q) for evaluating heterogeneity has low power when studies are few, and may have excessive power when studies are many and large (Higgins, Thompson, Deeks, et al., 2003). A more recently-introduced test statistic, called I^2, “describes the percentage of total variation across studies that is due to heterogeneity” (Higgins, Thompson, Deeks, et al., 2003). An I² value of 0% indicates no observed heterogeneity; values of 25%, 50%, and 75% are suggested to correspond with “low,” “moderate,” and “high” levels of heterogeneity, respectively (Higgins, Thompson, Deeks, et al., 2003).

Some degree of heterogeneity is expected since meta-analyses combine results of studies that differ to at least some degree both clinically and methodologically. “What matters is the extent to which it affects the conclusions of the meta-analysis” (Higgins, Thompson, Deeks, et al., 2003). Thus, it is important to investigate potential sources of heterogeneity for any effect on the interpretation of meta-analysis results.

In subgroup analysis, subgroup category point estimates are compared to see if they are significantly different from each other, thus identifying a potential source of heterogeneity. When more than one type of subgroup may be important, separate subgroup analyses give an incomplete and potentially misleading picture. Meta-regression can be used to test the effects of multiple subgroups at the same time (multivariate analysis) (Thompson and Higgins, 2002). Meta-regression describes an observational association across trials and should not be interpreted as derived from randomized comparisons (even though the individual trials may have been randomized). As such, meta-regression is considered an exploratory or hypothesis-generating analysis.

What information is provided by fixed-effect meta-analysis vs. random-effects meta-analysis?

Fixed-effect meta-analysis assumes that there is a common treatment effect and that variation in individual study results (described by the confidence interval around the point estimate of treatment effect) is due to chance. When there is heterogeneity that cannot be readily explained, causes of heterogeneity should be explored. Thus, a common meta-analysis protocol begins with a fixed effect analysis, followed by an exploration of heterogeneity, whether detected statistically or logically directed by known sources of potentially significant heterogeneity.

When heterogeneity is present but cannot be explained by subgroup analysis or meta-regression, a random effects meta-analysis may be conducted. This model assumes that there are different treatment effects that follow a normal distribution. Here, the point estimate is the average of the disparate treatment effects, while its confidence interval describes the uncertainty in the location of the mean of the different treatment effects (Cochrane Reviewers’ Handbook 4.2.1, http://www.cochrane.org/resources/handbook/). Thus, the result of a random-effects meta-analysis cannot be reported as an alternative estimate and variance of a fixed-effect analysis (Cochrane Reviewers’ Handbook 4.2.1). Nor does a random-effects analysis discount the issue of heterogeneity; “it is always advisable to explore possible causes of heterogeneity” (Cochrane Reviewers’ Handbook 4.2.1).

The use of fixed-effects versus random-effects meta-analysis is controversial. When there is no statistical heterogeneity, the results of both analyses are the same. However, the degree of heterogeneity beyond which fixed-effect results are likely to be misleading is unclear. Random-effects analyses are commonly represented as more “conservative” i.e., less-extreme point estimates and wider confidence intervals. But the random-effects assumption of a normal distribution of treatment effects may be inaccurate, with unknown effects on the result (Cochrane Reviewers’ Handbook 4.2.1); random-effects analysis may also generate a result more extreme than a fixed-effect estimate, with greater statistical significance (Poole and Greenland, 1999; Engels, Schmid, Terrin, et al., 2000). Finally, a disadvantage of the random effects model is that it gives more weight to small, less precise trials (Poole and Greenland, 1999).

A review of guidelines and practice regarding statistical methods in systematic reviews reported that, “Advice was generally consistent, advocating a cautious examination of potential causes of heterogeneity and the use of random effects meta-analyses to account for variation that cannot be explained (either instead of or in addition to fixed effect analyses). Specific guidance on choosing between fixed effect and random effects meta-analyses was not [generally] available” (Higgins, Thompson, Deeks, et al., 2002).

What method of analysis was chosen for this systematic review?

The original protocol called for a fixed-effect meta-analysis followed by subgroup analysis to explore potential causes of heterogeneity. Where statistical heterogeneity was high for important patient outcomes, subgroup analysis was to be followed by meta-regression.