NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Henriksen K, Battles JB, Keyes MA, et al., editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign). Rockville (MD): Agency for Healthcare Research and Quality (US); 2008 Aug.

Cover of Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign)

Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign).

Show details

Confidential Performance Feedback and Organizational Capacity Building to Improve Hospital Patient Safety: Results of a Randomized Trial

, MD, MSc, , MD, MPH, , MS, , MS, , PhD, , MD, MBA, , RN, MBA, PhD, , MD, MPH, and , MD, MPH.

Author Information

Objective: The objective of this study was to evaluate the effect of two intervention strategies, performance feedback reporting and organizational capacity building, both of which aim to improve patient safety in hospitals but for which there is a paucity of empirical data on effectiveness. Methods: We randomly assigned the 127 non-Federal, acute care hospitals in Wisconsin to one of three groups: (1) performance feedback reporting, (2) performance feedback reporting and organizational capacity building, and (3) control (no interventions). Reported medical injury rates were based on the Wisconsin Medical Injury Prevention Program (WMIPP) surveillance criteria. We compared adjusted pre- and postintervention injury rates overall, in four broad categories, and for five priority areas targeted in the organizational capacity building. Results: The groups of hospitals were similar with respect to location, structure, inpatient utilization, facilities, and services offered. Overall medical injury rates for drug-associated injuries increased significantly during the study period in all groups. No statistically significant differences among the intervention groups or between either of the intervention groups and the control group were detected for overall injury or any of the five major category injury rates. Conclusion: The inability to demonstrate a reduction in medical injury rates in relation to either confidential performance feedback reporting or organizational capacity building may be due to either methodologic limitations of the study or ineffectiveness of the interventions.

Introduction

The Institute of Medicine (IOM) report, To Err is Human1 led to a recent increased interest in patient safety. That report emphasized the need to develop and evaluate alternative methods of patient safety reporting. Numerous groups have developed patient safety indices; a recent assessment by the RAND Corporation identified 14 different groups of patient safety indicators that had been prepared by various organizations.2 While the IOM report emphasized systems that monitor the occurrence of medical errors, an approach that focuses on medical injuries or adverse events that occur from medical care may be more reliable, engender less defensiveness, and promote a greater emphasis on patient outcomes than approaches that focus on negligence.3

One approach to identifying medical injuries based on hospital discharge data, the Wisconsin Medical Injury Prevention Program (WMIPP) Screening Criteria, appears to have adequate sensitivity and specificity for monitoring patterns and trends in medical injuries.4 The WMIPP criteria are based on the World Health Organization’s International Classification of Diseases discharge diagnosis codes and thus are subject to the well-known limitations of administrative discharge data.5 Because diagnosis codes are universally collected on all patients discharged from hospitals in the United States, the WMIPP screening criteria avoid the limitations of voluntary reporting systems that require proactive reporting by health care providers and are likely to have very low sensitivity for patient safety problems.

One potential use of patient safety reporting systems is to provide performance feedback to institutions and health care providers to inform and target their efforts to improve patient safety. Numerous theoretical approaches to improving patient safety have been proposed by researchers from health services research, public health, clinical medicine, and ergonomics. Karsh and colleagues identified three paradigms for directing improvement efforts focused on reducing injuries, reducing errors, or improving evidence-based practice. They also proposed a fourth paradigm, a human factors engineering paradigm.6 While these paradigms have different theoretical foundations, they do share the perspective that comprehensive, systems approaches might be needed to improve patient safety. In some respects, these approaches can be viewed as complementary, rather than competitive.6

Implementing the systemic changes needed to increase patient safety poses daunting challenges to complex organizations like hospitals. Most patient safety research to date has emanated from academic medical centers, particularly tertiary care hospitals. General, acute care hospitals face many of the same challenges in improving patient safety but may lack the specialized resources and personnel that are available in academic tertiary care centers. To address these challenges, we developed a program of technical assistance to build necessary organizational capacity. The organizational capacity-building component included guidance on interpreting the medical injury data, provision of evidence-based information on effective interventions to address patient safety priorities, and consultations on organizational strategies to implement the necessary systemic changes.7

We conducted a statewide, randomized, controlled trial evaluating the impact of confidential performance feedback of patient safety experience and organizational capacity building on the occurrence of medical injuries in acute care hospitals in Wisconsin. We hypothesized that hospitals receiving regular, confidential, hospital-specific reports of medical injury rates would increase patient safety efforts and reduce medical injuries. However, hospitals receiving technical assistance in building organizational capacity in addition to the same patient safety reports would achieve a greater increase in patient safety efforts and a greater reduction in medical injuries.

Methods

To evaluate the impact of two different intervention strategies for improving patient safety, we randomly assigned all non-Federal, acute care, general hospitals in Wisconsin to one of three groups: (1) performance feedback reports only, (2) performance feedback reports plus organizational capacity building, and (3) control.

Random Allocation

To reduce the potential of residual confounding differences among the three groups of hospitals, we employed a multistep process analogous to prognostic stratification8, 9, 10 to obtain treatment groups from hospitals of comparable size and baseline injury rates based on data from 1999. Using the ability of cluster analysis to create natural groupings, hospitals were first grouped together based on a hierarchic clustering of number of discharges, overall medical injury rate, drug injury rate, biologic injury rate, device and implant injury rate, procedure injury rate, radiation injury rate, urban/rural location, and children’s or tertiary care hospital. Some clusters were very similar and were combined; two hospitals were moved to clusters with similar averages, to create six clusters divisible by four and one cluster of six hospitals. Hospitals within each cluster were then assigned a random number and sorted according to that random number. The hospitals within each cluster were sequentially assigned to treatment groups 1, 1, 2, 3 in the larger clusters and 1, 2, 3 in the small cluster. The hospitals were allocated as follows:

  • Performance feedback group: 64 hospitals.
  • Performance feedback plus organizational capacity-building group: 30 hospitals.
  • Control group: 33 hospitals in which we monitored the occurrence of medical injuries throughout the demonstration project, but to which we provided no feedback or intervention.

Medical Injury Surveillance Criteria

The methods we used for identifying medical injuries using hospital discharge data have been described in detail elsewhere.4 Here, we briefly summarize the patient safety surveillance methods directly relevant to this study. To develop comprehensive surveillance or screening criteria for medical injuries, we reviewed the ICD-9-CM codes to identify N-codes or E-codes indicative of a medical injury. Medical injuries were classified into four broad categories: (1) drugs; (2) procedures; (3) devices, implants, or grafts; and (4) radiation. These categories were further divided into 40 subcategories to more precisely indicate the cause of injury. These surveillance criteria were validated by comparison to results obtained from a blinded review of medical records.4

Case Definition

In this study, a discharge was considered to fulfill the criteria for medical injury if any of the nine diagnosis fields or the special E-code field contained any one of the codes listed in the surveillance criteria. A given patient discharge could be associated with more than one type of medical injury.

Study Population

The study population included all patients discharged from acute care, non-Federal hospitals in Wisconsin, with the exception of newborn delivery discharges. Also excluded were alcohol and other drug abuse (AODA) hospitals and psychiatric hospitals. This study used Form UB-92 hospital discharge data collected by the Bureau of Health Information and Policy through September 30, 2003, and thereafter by the Wisconsin Hospital Association Information Center.

Interventions

Performance feedback. For each general, acute care, non-Federal hospital in Wisconsin, a report was developed containing the following elements:

  • Number of occurrences, risk-adjusted injury rates per 1000 discharges, percentile ranking among Wisconsin hospitals, and statewide increase in length of stay for all medical injuries and for four broad categories of medical injuries related to drug, device, procedure, or radiation.
  • Similar details for the 40 more specific subcategories of medical injuries.
  • Unadjusted rates for the 10 most frequent medical injuries at the particular institution with the greatest increase in length of stay.
  • Unadjusted rates of 16 subcategories of medical injury relevant to surgery, using only surgical discharges in the calculation of rates (see Appendix).

Risk adjustment factors included patient age, sex, severity of illness, risk of mortality, primary payer, hospital ownership, presence of a residency program, percentage of board-certified staff, trauma center level, provision of oncology services, percentage of facility discharges that were surgical operations, provision of transplant services, number of medical record personnel per 1,000 discharges, average number of diagnostic codes across all hospital discharges, proportion of diagnostic codes that were nonspecific, and proportion of diagnostic codes that were unspecified. The severity of illness and risk of mortality scores were based on the APR-DRG (All Patient Refined-Diagnosis Related Groups) indices calculated by the APR-DRG medical information system.11 Because these indices were intended to adjust for patients’ underlying illnesses, they were calculated after excluding all medical injury-related diagnostic codes.4

Reports utilized the most recent 6 months of available hospital discharge data; they were sent semi-annually to each performance feedback hospital and each performance feedback + organizational capacity-building hospital. At each hospital, reports were sent confidentially to the CEO, head of quality assurance, head of nursing, and head of the medical staff. The first baseline report was sent in February 2002 and covered the period from October 1, 2000 through March 31, 2001. The sixth and final report, which was sent in October 2004, included a table showing trends in injury rates related to the five intervention manuals (described below) for the performance feedback + organizational capacity-building hospitals.

Organizational capacity building. In addition to the semi-annual reports, hospitals in this group received additional support and materials. An intervention specialist with considerable education and experience in clinical nursing, nursing administration, and organizational management visited hospitals assigned to theorganizational capacity-building group for the purpose of answering questions on the reports; assuring the hospitals that all hospital-specific injury information in the reports was confidential; identifying obstacles to disseminating or using the reports; gathering information on mechanisms currently in place to address patient safety issues; and conducting a needs assessment to identify which specific medical injuries should be the focus of intervention efforts.

Using extensive literature review and examples of hospital best practices, WMIPP physician experts developed a set of five condition-specific manuals that summarized research evidence and suggested practical strategies to prevent medical injuries within a quality improvement framework. The selection of targets for intervention were based on AHRQ’s “Priority Areas for National Action,”12 the frequency and severity of medical injury occurrences based on WMIPP surveillance reports, input from organizational capacity-building hospitals, and expert panel assessments of the relative feasibility of potential interventions. (See Appendix for intervention targets and priority indicators.)

These hospitals also received followup newsletters designed to reinforce the educational manuals and provide additional practical, evidence-based resources for patient safety interventions. The key elements and contents of manuals and newsletters are described elsewhere.7 The timeline of performance feedback reports and capacity building are shown in Table 1.

Table 1. Timeline of performance feedback reports and capacity building.

Table 1

Timeline of performance feedback reports and capacity building.

Impact on Medical Injury Rates

Statistical analysis. A discharge was considered to involve a medical injury if any of the nine diagnosis fields or the special E-code field contained one of the WMIPP criteria. A given patient discharge could be associated with more than one type of medical injury. Rates of medical injuries were calculated as the number of discharges with the particular type of medical injury divided by the total number of discharges. All statistical analyses were performed using SAS® and Stata® software. The effect of the interventions on medical injury after accounting for differences in relevant covariates was assessed using linear mixed-model analysis. However, due to lack of normality in residuals, nonparametric tests, such as the Kruskal-Wallis and Friedman’s tests, were conducted. All analyses were repeated after excluding patients under 18 years of age and women admitted for childbirth, with similar results (data not reported).

As was the case in the performance feedback reports, to adjust for possible differences in patient mix among hospitals, we adjusted for severity of illness and risk of mortality levels, based on the All Patient Refined-Diagnosis Related Groups, Ver. 15, software system for inpatient care.11 Severity of underlying illness was a potential confounder because it varies among hospitals and is independently associated with the likelihood of injury.13 The assignment of a patient discharge to an illness severity or risk of mortality class considers not only the individual diagnoses but also the interaction among diagnoses and age and the presence of certain operating room and non-operating room procedures.

Results

Hospital characteristics within the three study groups are summarized in Table 2. The hospitals in the three groups did not significantly differ in their location, structure, inpatient utilization, facilities, and services offered. There was a significant difference in the number of medical record personnel per 1,000 discharges (P = 0.01). Characteristics of patients in the hospitals in each of the study groups are shown in Table 3. There were no significant differences in any of the patient characteristics measured among the three groups.

Table 2. Baseline hospital characteristics by study group.

Table 2

Baseline hospital characteristics by study group.

Table 3. Characteristics of patients in hospitals in each study group.

Table 3

Characteristics of patients in hospitals in each study group.

Overall injury rates per 1,000 discharges for all hospitals are shown in Figure 1 across the study years 2001–2005 via box plots at the different time points. Overall medical injury rates in the three study groups ranged from 10.0 to 10.4 per 1,000 discharges in the pre-intervention period and 11.0 to 12.0 per 1,000 discharges in the post-intervention period (Table 4). The randomized treatment design with longitudinal measurements called for analysis using a mixed normal linear model to detect possible treatment effect. This analysis showed a time trend but not a significant treatment effect nor a time-by-treatment interaction. Moreover, residual analysis indicated lack of normality, even after applying the logit and arc-sine transformations to the injury rates. Therefore, we report here the results of subsequently used nonparametric analytic methods.

Figure 1. Overall medical injury rates in Wisconsin hospitals, 2001–2005.

Figure 1

Overall medical injury rates in Wisconsin hospitals, 2001–2005.

Table 4. Pre- and post-intervention injury rates.

Table 4

Pre- and post-intervention injury rates.

Time trends were tested by Friedman’s test for dependent (here, longitudinal) observations. Overall and drug-injury rates showed statistically significant increases over time (P <0.0005). None of the other four major categories, nor any of the 40 subcategories, demonstrated monotonically changing injury rates over time.

To evaluate the possible effect of treatment, the change in rate from 2001 to 2005 was computed for each hospital. The Kruskal-Wallis test (nonparametric ANOVA) was then applied to these data. No statistically significant differences among the intervention groups or between either of the intervention groups and the control group were detected for overall injury or for any of the five major category injury rates.

The same nonparametric test was used to compare differences in rates of the five priority indicators from 2002 (latest baseline year prior to distribution of the manuals) to 2005. The central venous catheter complications injury indicator showed a significant difference between groups (P = 0.02), with the performance feedback group having the greatest decrease followed by the control group in a Poisson regression model. The performance feedback + organizational capacity-building group actually showed an increase in reported central venous catheter complications during the study period. No significant differences in rate change were seen for the other priority indicators.

Discussion

We found no evidence of a reduction in identified medical injuries—either overall or in specific categories—associated with confidential performance feedback to hospital administration or with confidential performance feedback coupled with organizational capacity building. Acute care hospitals in Wisconsin were randomly assigned to one of the intervention groups or to the control group, and there were no major baseline differences in hospital or patient characteristics among the three study groups that might have limited our ability to detect an impact of the interventions.

The framework for patient safety reporting and organizational capacity building efforts in this study7 was based on factors shown to influence line managers’ perceptions of hospital performance data.14 The key medical injury topics selected for intervention were based on the Agency for Healthcare Research and Quality’s (AHRQ) Priority Areas for National Action,12 which in turn, were based in part on the evidence of effective interventions being available for implementation.

The inability to detect a measurable impact on medical injury occurrence of confidential performance feedback to hospital administration, with or without organizational capacity building, could reflect several possibilities. Unfortunately, a detailed process evaluation originally proposed as part of this study could not be undertaken because of funding limitations. Hospitals in this study, unlike in most patient safety research, included all non-Federal, general, acute care hospitals in Wisconsin and were not restricted to hospitals with a particular interest in, or commitment to, patient safety. By analogy with the transtheoretical model of personal behavioral change,15 such hospitals may have been in a pre-contemplation stage with respect to the changes necessary to improve patient safety.

Another possible reason for our inability to detect an impact of our interventions might involve limitations of hospital discharge data collected for administrative purposes.5 In this study, it is possible that the interventions could have increased attention to patient safety throughout the hospital, including in medical records, which might have resulted in more complete coding of medical injuries and may have obscured any potential reduction in their actual occurrence. This might, for example, have accounted for the increase in reported central venous catheter complications in the performance feedback + organizational capacity-building group.

In light of the complexity, culture, and incentives of hospitals, it is also possible that the confidential performance feedback and organizational capacity-building interventions used in this study were insufficient to produce an appreciable or sustained increase in patient safety activity in hospitals. Despite the emphasis on patient safety reporting,1 there was no clear evidence of the effectiveness of different approaches to reporting on injury or error occurrence. A rigorous comparison of the impact of public vs. confidential patient safety reporting is clearly timely.

The model for organizational capacity building may need to expand to include focused instruction on how to respond to feedback provided at the individual and organizational levels. The recent decision by Medicare to no longer pay for the extra costs of care attributable to preventable errors16 could heighten receptivity to feedback and other interventions designed to reduce medical injuries.

Although this rigorously conducted, randomized trial of confidential performance feedback and organizational capacity building to improve hospital patient safety did not find a statistically significant impact, it might ultimately help in the search for effective patient safety interventions. While progress can be made by identifying what does work, it is also important to know what is insufficient. This study suggests opportunities for improvement in intensity and targeting of our capacity-building intervention and the need for comprehensive patient safety indicators that could be used to measure accurately the effect of patient safety interventions in future studies.

Acknowledgments

Funding for this work came from grant # U18 HS11893 from the Agency for Healthcare Research and Quality and from grant R49/CCR519614 from the Centers for Disease Control and Prevention. Support for this project was provided by specialists Anne Marbella, MS, Janice B. Babcock, MA, Chris McLaughlin, Michele Leininger, and Jenny Her.

APPENDIX. Definitions of intervention targets and priority indicators by ICD-9 codes

Priority indicator (applicable ICD-9 codes)Denominator
1. Surgical site infections:
  • Dehiscence (998.3) or
  • Persistent post-operative fistula (998.6) or
  • Non-healing surgical wound (998.83) or
  • Infected post-operative seroma (998.51) or
  • Other post-operative infection (998.59)
All patients with a surgical procedure
2. Perioperative cardiac events:
  • Cardiac event during a procedure (997.1) or
  • Myocardial infarction (410.00–411.89) or
  • Cardiac arrest (427.5)
All patients with a noncardiac surgical procedure
3. Central venous catheter (CVC) complications:
  • Infection/inflammation due to vascular device, implant, or graft (996.62) or
  • Iatrogenic pneumothorax (512.1) or
  • Emphysema resulting from a procedure (998.81) or
  • Complication due to other vascular device, implant, graft (996.74), or
  • Mechanical complication of other vascular device, implant, graft (996.1)
All patients with a code for placement or replacement of a CVC
4. Anticoagulation complications:
  • Poisoning by anticoagulants (964.2) or,
  • Accidental poisoning agents affecting blood (E858.2) or
  • Adverse effect of correct drug anticoagulants (E934.2)
All discharges – excluding under age 18 or OB DRG (due to low prevalence of anticoagulant use)
5. Catheter-related urinary tract infections (UTI):
  • Infection/inflammation reaction to indwelling urinary catheter (996.64) or,
  • As a cause of later complication (E8 79.6), plus any one of the following:
    • Urinary tract infection, site not specified (599.0) or
    • Acute cystitis (595.0) or
    • Subacute and chronic cystitis NOS (595.2) or
    • Cystitis unspecified (595.9) or
    • Infections of kidney (590.0–590.9) AND
    • Foley placement or replacement (procedure 57.94 or 57.95)
All discharges

References

1.
Institute of Medicine. Committee on Quality of Health Care in America: To err is human: Building a safer health system. Washington, DC: National Academies Press; 2000.
2.
Farley DO, Greenberg MD, Haviland A, et al. Prioritizing patient safety outcomes measures: Results of an expert consensus process. Santa Monica, CA: RAND Corporation; 2007.
3.
Layde PM, Cortes LM, Teret SP, et al. Patient safety efforts should focus on medical injuries. JAMA. 2002;287:1993–1997. [PubMed: 11960544]
4.
Layde PM, Meurer LN, Guse C, et al. Advances in patient safety: From research to implementation. Vol. 2: Concepts and methodology. Rockville, MD: Agency for Healthcare Research and Quality; 2005. [Accessed March 14, 2008]. Medical injury identification using hospital discharge data. AHRQ Pub. 05-0021-2. Available at: www​.ahrq.gov/downloads​/pub/advances/vol2/Layde.pdf. [PubMed: 21249827]
5.
Boufous S, Williamson A. Reporting of the incidence of hospitalized injuries: Numerator issues. Inj Prev. 2003;9:370–375. [PMC free article: PMC1731026] [PubMed: 14693903]
6.
Karsh BT, Holden RJ, Alper SJ, et al. A human factors engineering paradigm for patient safety: Designing to support the performance of the healthcare professional. Qual Saf Health Care. 2006;1:i59–i65. [PMC free article: PMC2464866] [PubMed: 17142611]
7.
Meurer JR, Meurer LN, Grube J, et al. Advances in patient safety: From research to implementation. Vol. 4: Program, tools, and products. Rockville, MD: Agency for Healthcare Research and Quality; 2005. [Accessed March 14, 2008]. Combining performance feedback and evidence-based educational resources: A model to promote medical injury prevention. AHRQ Pub. 05-0021-4. Available at: www​.ahrq.gov/downloads​/pub/advances/vol4/Meurer.pdf.
8.
Feinstein AR. Clinical biostatistics. XV. The purposes of prognostic stratification. Clin Pharmacol Ther. 1972;13:285–297. [PubMed: 5017376]
9.
Feinstein AR. Clinical biostatistics. XV. The process of prognostic stratification. Part 1. Clin Pharmacol Ther. 1972;13:442–457. [PubMed: 5026382]
10.
Feinstein AR. Clinical biostatistics. XVI. The process of prognostic stratification. Part 2. Clin Pharmacol Ther. 1972;13:609–624. [PubMed: 5042374]
11.
Averill RF, Goldfield N, Hughes JS, et al. All patient refined-diagnosis related groups (APR-DRGs): Ver. 20.0. Wallingford, CT: 3M Health Information Systems; 2003. [Accessed March 14, 2008]. Available at: www​.hcup-us.ahrq.gov​/db/nation/nis/APR-DRGsV20MethodologyOverviewandBibliography.pdf.
12.
Shojania KG, Duncan BW, McDonald KM, et al., editors. Making health care safer: A critical analysis of patient safety practices. Rockville, MD: Agency for Healthcare Research and Quality; 2001. [Accessed March 14, 2008]. Evidence Report/Technology Assessment No. 43 (Prepared by the University of California at San Francisco–Stanford University Evidence-based Practice Center under Contract No. 290-97-0013). AHRQ Pub. 01-E058. Available at: www​.ahrq.gov/clinic/ptsafety​/pdf/ptsafety.pdf. [PMC free article: PMC4781305] [PubMed: 11510252]
13.
Leape L, Brennan T, Laird N, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991;324:377–384. [PubMed: 1824793]
14.
Ginsberg LS. Factors that influence line managers’ perceptions of hospital performance data. Health Serv Res. 2003;38:261–286. [PMC free article: PMC1360884] [PubMed: 12650391]
15.
Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot. 1997;12:38–48. [PubMed: 10170434]
16.
Pear R. Medicare says it won’t cover hospital errors. The New York Times. Aug 19, 2007. [Accessed March 14, 2008]. Accessed August 30, 2007. Available at: www​.nytimes.com/2007​/08/19/washington/19hospital​.html?ex=1345176000&en​=7fc31f25dd9f629f&ei​=5088&partner=rssny.

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this page (154K)

Other titles in this collection

Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to PubMed

Similar articles in PubMed

See reviews...See all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...