NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Cover of Case Study Comparing Bayesian and Frequentist Approaches for Multiple Treatment Comparisons

Case Study Comparing Bayesian and Frequentist Approaches for Multiple Treatment Comparisons

Methods Research Reports

Investigators: , PhD, , PhD candidate, , MD, MS, , PhD, and , MD.

Minnesota Evidence-based Practice Center
Rockville (MD): Agency for Healthcare Research and Quality (US); .
Report No.: 12(13)-EHC103-EF

Structured Abstract


Bayesian statistical methods are increasingly popular as a tool for meta-analysis of clinical trial data involving both direct and indirect treatment comparisons. However, appropriate selection of prior distributions for unknown model parameters and checking of consistency assumptions required for feasible modeling remain particularly challenging. We compared Bayesian and traditional frequentist statistical methods for multiple treatment comparisons in the context of pharmacological treatments for female urinary incontinence (UI).

Data sources:

We searched major electronic bibliographic databases, U.S. Food and Drug Administration reviews, trial registries, and research grant databases up to November 2011 to find randomized studies published in English that examined drugs for urgency UI on continence, improvements in UI, and treatment discontinuation due to harms.

Review methods:

We fitted fixed and random effects models in frequentist and Bayesian frameworks. In a hierarchical model of eight treatments, we separately analyzed one safety and two efficacy outcomes. We produced Bayesian and frequentist treatment ranks and odds ratios (and associated measures of uncertainty) across all bivariate treatment comparisons. We also calculated the number needed to treat (NNT) to achieve continence or avoid harms from pooled absolute risk differences.


While frequentist and Bayesian analyses produced broadly comparable odds ratios of safety and efficacy, the Bayesian method's ability to deliver the probability that any treatment is best, or among the top two such treatments, offered a more meaningful clinical interpretation. In our study, two drugs emerged as attractive because while neither had any significant chance of being among the least safe drugs, both had greater than 50 percent chances of being among the top three drugs in terms of Best12 probability for one of the efficacy endpoints.


Bayesian methods are more flexible and their results more clinically interpretable but require more careful development and specialized software.

Key Messages

  • Both Bayesian and frequentist hierarchical models can be effective in multiple treatment comparisons.
  • Bayesian models sensibly shrink estimates towards each other, encouraging more borrowing of statistical strength from the entire collection of studies. Bayesian methods also lead to more clinically interpretable results (through their ability to assign probabilities to events), as well as more sensible rankings of the pharmacological treatments as compared to traditional NNT-based methods.
  • Further development of hierarchical Bayesian multiple treatment comparison methods is warranted, especially for nonbinary data models, simultaneous decisionmaking across multiple endpoints, assessing consistency, and incorporating data sources of varying quality (e.g., clinical vs. observational data).

Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No. 290-2007-10064-I2 Prepared by: Minnesota Evidence-based Practice Center, Minneapolis, MN

Suggested citation:

Carlin BP, Hong H, Shamliyan TA, Sainfort F, Kane RL. Case Study Comparing Bayesian and Frequentist Approaches for Multiple Treatment Comparisons. (Prepared by the Minnesota Evidence-based Practice Center under Contract No. 290-2007-10064-I2.) AHRQ Publication No. 12(13)-EHC103-EF. Rockville, MD: Agency for Healthcare Research and Quality. March 2013.

This report is based on research conducted by the Minnesota Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10064-I2). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

None of the investigators has any affiliations or financial involvement that conflicts with the material presented in this report.


540 Gaither Road, Rockville, MD 20850; www​

Bookshelf ID: NBK132729PMID: 23638487


Related information

Similar articles in PubMed

See reviews...See all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...