Context:

Although evidence of quality problems has been available for years, purchaser interest in quality-based purchasing (QBP) is a recent phenomenon. Furthermore, employers who support quality-based purchasing have expressed uncertainty about how to measure quality, especially outcomes, and what incentives to offer to stimulate performance improvement.

Objectives:

The objectives of this project were to develop a conceptual model of how incentives influence provider behavior, to summarize what is known from randomized controlled trials about the effectiveness of different QBP strategies, to describe ongoing QBP research, and to perform simulations to determine whether outcomes reports are too influenced by chance events to be used in QBP.

Data Sources:

We used online databases (e.g., MEDLINE^®) and bibliographies of retrieved articles for the literature search and government and foundation listings to identify ongoing research. For the simulations, we used data from public reports of myocardial infarction outcomes in California.

Study Selection:

For the literature review, we sought studies in which providers had been randomized to an incentive group or a control group. We included only projects involving interventions purchasers could plausibly adopt (payment strategies or public reporting of performance). Studies of interventions that were beyond purchaser purview (e.g., implementing clinical guidelines) were excluded.

Data Extraction:

We extracted information about the type of incentive used and the clinical and economic context in which it was applied.

Data Synthesis:

We evaluated 5,045 publications. Nine were randomized controlled trials, and many of these did not report key characteristics of the incentive or the context in which incentives were applied. Incentives used included additional fee-for-service, quality bonuses, and public release of performance data. The results were mixed: among the 11 performance indicators evaluated, 7 showed a statistically significant response to QBP strategies while 4 did not. We also found 18 ongoing research projects, none randomized. These will yield data about the approaches to QBP currently in use, provider awareness of and concerns about QBP, and some preliminary estimates of the potential impact of QBP.

Regarding assessments of outcomes reports, we found that, under reasonable assumptions and applications, outcomes reports generate meaningful information about provider performance. Providers with good (expected) performance are unlikely to be labeled as poor quality in any given period, and very unlikely to be mislabeled more than once in a 3-year period, even if one allowed approximately 10% of hospitals to be labeled poor performers annually. In addition, hospitals with superior performance were quite likely to be identified as such at least once in 3 years.

Conclusions:

Little is known about the impact of QBP on clinical performance. However, it does appear that basing incentives on measurements of outcomes is feasible without undue risk to the reputation or financial status of good hospitals. Ongoing research will only address some of the gaps in our knowledge about QBP, suggesting that much more additional research is needed. This should include comparisons of alternative QBP approaches and qualitative assessment of the barriers to and facilitators of quality improvement in response to QBP incentives.

Contents

• Preface
• Acknowledgments
• 1. Introduction
  • Background
  • Purpose of This Report
  • Rationale for Focus on Randomized Controlled Trials
  • Types of Incentives
  • Incentive Theory
  • Characteristics of Incentives
  • Conceptual Models of Individual Provider and Organizational Responses to Incentives
• 2. Methods for Literature Search
  • Technical Expert Advisory Panel
  • Target Audiences and Population
  • Key Questions
  • Literature Review Methods
  • Identifying Ongoing Research
• 3. Results for Literature Search
  • Synthesis of Literature About Quality-based Purchasing
  • Ongoing Research Into Quality-based Purchasing
• 4. Methods for Assessing the Usefulness of Outcome Reports
  • General Approach to Simulation
  • Enhancements to the Thomas and Hofer Model
• 5. Results of Simulations To Assess the Usefulness of Outcomes Reports
  • Scenario 1: Reproducing Thomas and Hofer
  • Scenario 2: Adding Another Hospital Category
  • Scenario 3: Updating Assumptions About the Hypothetical Distribution of Hospital Quality
  • Scenario 4: Fewer Patients per Hospital (N = 100)
  • Scenario 5: Identifying a Higher Proportion of Outliers
  • Scenario 6: More Patients per Hospital
• 6. Discussion
  • Analysis of Published and Ongoing Research
  • Evaluating Outcomes Reports
  • Future Research
  • Conclusion
• Acronyms and Abbreviations
• Appendixes
  • Appendix A Quality-based Purchasing Technical Expert Panel and Peer Reviewers
  • Appendix B General Approach to Simulations
  • Appendix C Assessing the Usefulness of Outcome Reports
• References and Included Studies

540 Gaither Road, Rockville, MD 20850. www​.ahrq.gov

Other Contributor: Kathryn McDonald, MM, EPC Associate Director.

Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services.¹ Contract No. 290-02-0017. Prepared by: Stanford-University of California San Francisco Evidence-based Practice Center.

Suggested citation:

Dudley RA, Frolich A, Robinowitz DL, Talavera JA, Broadhead P, Luft HS. Strategies To Support Quality-based Purchasing: A Review of the Evidence. Technical Review 10. (Prepared by the Stanford-University of California San Francisco Evidence-based Practice Center under Contract No. 290-02-0017). AHRQ Publication No. 04-0057. Rockville, MD: Agency for Healthcare Research and Quality. July 2004.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. Endorsement by the Agency for Healthcare Research and Quality (AHRQ) or the U.S. Department of Health and Human Services (DHHS) of such derivative products may not be stated or implied.

AHRQ is the lead Federal agency charged with supporting research designed to improve the quality of health care, reduce its cost, address patient safety and medical errors, and broaden access to essential services. AHRQ sponsors and conducts research that provides evidence-based information on health care outcomes; quality; cost, use, and access. The information helps health care decisionmakers - patients and clinicians, health system leaders, and policymakers - make more informed decisions and improve the quality of health care services.

The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services of a particular drug, device, test, treatment, or other clinical service.

1

540 Gaither Road, Rockville, MD 20850. www​.ahrq.gov