NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Cover of Empirical and Simulation-Based Comparison of Univariate and Multivariate Meta-Analysis for Binary Outcomes

Empirical and Simulation-Based Comparison of Univariate and Multivariate Meta-Analysis for Binary Outcomes

Methods Research Reports

Investigators: , MD, , PhD, and , PhD.

Tufts Evidence-based Practice Center
Rockville (MD): Agency for Healthcare Research and Quality (US); .
Report No.: 13-EHC066-EF

Structured Abstract


Many questions in evidence-based medicine involve multiple outcomes. They can be approached with separate, independent meta-analyses, or they can be analyzed jointly, in a single model. We aimed to compare separate (univariate) with joint (multivariate) meta-analysis in real examples and in an illustrative simulation study.


We screened the whole Cochrane Library of Systematic Reviews (2012, first quarter) to identify sets of univariate meta-analyses of categorical outcomes that can also be analyzed jointly. Eligible were pairs or triplets of meta-analyses comparing the same interventions; having at least seven randomized controlled trials (RCTs) reporting all outcomes; and in which the numbers in the cross-classification of outcomes were exactly recoverable. Examples of outcomes with completely recoverable cross-classification include mutually exclusive outcomes, or sets of outcomes where the one is a subset of the other. We analyzed these data with univariate and multivariate meta-analysis. In an accompanying simulation study, we compared summary estimates and their standard errors with univariate and multivariate meta-analysis.


We identified 45 pairs or triplets of binary meta-analyses corresponding to 1473 RCTs and 258,675 randomized patients. In 38 (of 45) topics the first outcome was a subset of the second outcome; in 5 topics pairs of outcomes were mutually exclusive, and in 2 topics triplets of outcomes had an is-subset-of relationship. The 45 topics pertained to various medical areas (e.g., cardiology, surgery, mental health). Overall, the summary effects for each outcome and the accompanying confidence/credible intervals were very similar with univariate and multivariate meta-analysis (both using the approximate and the discrete likelihood). However, univariate and multivariate approaches yield different confidence/credible intervals for the difference between the summary effects of distinct outcomes (e.g., the difference in the log odds ratio for the first outcome minus the log odds ratio for the second outcome). Depending on the estimated covariance between the compared effects, the multivariate methods can yield tighter or wider confidence intervals than univariate methods. Most likely, systematic review conclusions from the meta-analyses in the empirical sample would remain qualitatively the same with either method of analysis. The simulation analyses were congruent with the aforementioned observations from the empirical analyses.


In the empirical sample and the simulation study, the numerical difference in the summary effects and their confidence intervals between univariate and multivariate meta-analysis was almost always small. In practice, in many (if not most) cases, conclusions based on the main effects of each outcome are likely to remain similar with either method. In principle, multivariate meta-analysis utilizes more information through the correlations; therefore, when possible, it is commendable to use both univariate and multivariate approaches in a sensitivity analysis. Multivariate meta-analysis should be preferred over univariate meta-analysis for estimating differences between outcome-specific summary treatment effects.

Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No. 290-2007-10055-I, Prepared by: Tufts Evidence-based Practice Center, Boston, MA

Suggested citation:

Trikalinos TA, Hoaglin DC, Schmid CH. Empirical and Simulation-Based Comparison of Univariate and Multivariate Meta-Analysis for Binary Outcomes. Methods Research Report. (Prepared by the Tufts Evidence-based Practice Center under Contract No. 290-2007-10055-I.) AHRQ Publication No. 13-EHC066-EF. Rockville, MD: Agency for Healthcare Research and Quality; March 2013.

This report is based on research conducted by the Tufts Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No.290-2007-10055-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

None of the investigators has any affiliations or financial involvement that conflicts with the material presented in this report. Authors TAT and CHS are involved in developing open source software for meta-analysis, but this software is available at no charge and the authors receive no financial benefit. All authors have developed multivariate meta-analysis methods.


540 Gaither Road, Rockville, MD20850; www‚Äč

Bookshelf ID: NBK132562PMID: 23617018


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (7.8M)

Related information

Similar articles in PubMed

See reviews...See all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...