NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

McDonald KM, Romano PS, Geppert J, et al. Measures of Patient Safety Based on Hospital Administrative Data - The Patient Safety Indicators. Rockville (MD): Agency for Healthcare Research and Quality (US); 2002 Aug. (Technical Reviews, No. 5.)

Cover of Measures of Patient Safety Based on Hospital Administrative Data - The Patient Safety Indicators

Measures of Patient Safety Based on Hospital Administrative Data - The Patient Safety Indicators.

Show details

Appendix FDetailed Results for Rejected Indicators

This appendix presents the literature review and clinician panel review results for all indicators rejected either pre- or post-panel review. It is organized into three sections.

Section 1 presents the literature review results for indicators rejected pre-panel review.

Section 2 presents the literature review results for indicators rejected post-panel review.

Section 3 presents the clinician panel review results for indicators rejected post-panel review.

Section 1. Literature Review Results for Indicators Rejected Pre-panel Review

Complications of Anesthesia - Shock

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 8, “post or intraoperative shock due to anesthesia”). Shock due to anesthesia (995.4) is the sole ICD-9-CM code in their original definition. It was also included as one component of a broader indicator (“adverse events and iatrogenic complications”) in AHRQ's original HCUP Quality Indicators.2

Evidence

We were unable to find evidence on validity from prior studies, because this complication is quite rare.

Complications Relating to Drugs

Source. This indicator (precise definition not available) was originally proposed by Hannan et al. as a criterion for targeting “cases that would have a higher percentage of quality of care problems than cases without the criterion, as judged by medical record review.”3 It was redefined and endorsed by Iezzoni et al.1 in the CSP (CSP 28, “complications related to drugs”), based on major drug classes: antibiotics, antifungals, antivirals, non-narcotic and narcotic analgesics, antipyretics, anesthetics, anticoagulants, fibrinolytics, blood products, anticonvulsant and anti-Parkinsonian agents, sedatives/hypnotics, psychotropics, stimulants, antineoplastics, immunosuppressants and antirheumatics, hormones, antiasthmatics, antiarrhythmics and other cardiovascular agents. Needleman and Buerhaus 4 considered adverse drug events as an “Outcome Potentially Sensitive to Nursing,” based on input from their Technical Expert Panel, but discarded it because the “event rate was too low to be useful.”

Evidence

Coding validity. This indicator, as defined in CSP, is highly problematic among medical cases (10% confirmation by coders, 20% by physicians), apparently because most drug-related complications are present at admission.5, 6

Construct validity. Explicit process of care failures in the CSP validation study were very unusual among medical cases with CSP 28 (2%), and no more frequent than among unflagged controls (5%). Physician reviewers identified potential quality problems in 16% of medical patients with CSP 28 (versus 2% of unflagged controls).6 Based on two-stage implicit review of 8,109 randomly selected deaths from 104 New York hospitals in 1985-86, Hannan et al.3 found that cases with a secondary diagnosis of “selected drug poisonings” were no more likely to have received care that departed from professionally recognized standards than cases without such codes (2.5% versus 1.7%, OR=1.09), after adjusting for patient demographic, geographic, and hospital characteristics. We were unable to find other evidence on the validity of this indicator.

Death Within One (or Two) Days of Any Surgical Procedure

Source. This indicator (with alternative time windows) was originally proposed by Hannan et al. as a criterion for targeting “cases that would have a higher percentage of quality of care problems than cases without the criterion, as judged by medical record review.”3 The University HealthSystem Consortium adopted this indicator for procedures involving anesthesia (2836).

Evidence

Construct validity. Based on two-stage review of 8,109 randomly selected deaths from 104 New York hospitals in 1985-86, Hannan et al.3 reported that patients who died within one day of a significant surgical procedure (except for cancer or trauma) were 2.8 times more likely to have received care that departed from professionally recognized standards than other patients who died (4.8% versus 1.7%), after adjusting for patient demographic, geographic, and hospital characteristics. In 46 of these 59 cases (78%) of substandard care, the patient's death was attributed at least partially to that care. A two-day window detected 35 additional cases of substandard care, but the association between second-day deaths and substandard care was weaker (4.4% versus 1.7%, OR=2.0). We were unable to find other evidence on the validity of this indicator.

In-hospital Burns

Source. This indicator (940.0–949.5) was originally proposed by Hannan et al. as a criterion for targeting “cases that would have a higher percentage of quality of care problems than cases without the criterion, as judged by medical record review.”3

Evidence

Construct validity. Based on two-stage review of 8,109 randomly selected deaths from 104 New York hospitals in 1985-86, Hannan et al.3 reported that cases with a secondary diagnosis of burn were not significantly more likely to have received care that departed from professionally recognized standards than cases without that code (7.4% versus 1.7%, OR=3.4), after adjusting for patient demographic, geographic, and hospital characteristics. We were unable to find other evidence on the validity of this indicator.

Mechanical Complications

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 10, “mechanical complication due to device, implant or graft, except organ transplant”). Their definition excludes mechanical complications due to prosthetic heart valves, coronary bypass grafts, other vascular devices or grafts, and nervous system devices, implants, or graft. The University HealthSystem Consortium and AHRQ's original HCUP Quality Indicators adopted this CSP indicator for major surgery patients (2932); Version 1.3 of the QIs included several additional (new) ICD-9-CM updates.2

Evidence

Coding validity. CSP 10 had a borderline confirmation rate among major surgical cases (61% by coders' review, 56% by physicians' review, 73% by nurse-abstracted clinical documentation).57 In comparison with the VA's National Surgical Quality Improvement Program database from 123 hospitals in 1994-95, in which “graft/prosthetic failure within 30 days after surgery” is the only mechanical complication qualifying for documentation, ICD-9-CM diagnoses (996.0x–996.5x) had a sensitivity of 14% and a predictive value of 2%.8

Construct validity. Explicit process of care failures in the CSP validation study were only moderately frequent among major surgical cases with CSP 10 (33%), after excluding a few patients who had mechanical complications at admission, but unflagged controls were not evaluated on the same criteria. Physician reviewers identified potential quality problems in 31% of major surgery patients with CSP 10 (versus 2% of unflagged controls). 6 Kovner and Gergen reported that among 506 community hospitals in the 1993 Nationwide Inpatient Sample, having more registered nurse hours per adjusted patient day was not associated with rates of mechanical complications due to a device, implant, or graft.9

Other Complications of Surgery

Source. This indicator (996–999) was originally proposed by Hannan et al. as a criterion for targeting “cases that would have a higher percentage of quality of care problems than cases without the criterion, as judged by medical record review.”3 However, subsequent authors found this list of ICD-9-CM codes to be overly broad, and created more specific indicators from the same list of codes.

Evidence

Construct validity. Based on two-stage review of 8,109 randomly selected deaths from 104 New York hospitals in 1985-86, Hannan et al.3 reported that cases with a secondary diagnosis of 996–999 were 2.5 times more likely to have received care that departed from professionally recognized standards than cases without that code (3.7% versus 1.7%), after adjusting for patient demographic, geographic, and hospital characteristics. In 24 of these 35 cases (69%) of substandard care, the patient's death was attributed at least partially to that care.

Postoperative Cardiac Abnormalities Except AMI

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 15, “postoperative cardiac abnormalities except AMI”). Their definition includes complete atrioventricular block, ventricular tachycardia, ventricular fibrillation, and functional abnormalities following cardiac surgery among persons less than 65 years of age.

Evidence

Coding validity. No evidence on validity is available from CSP studies. Geraci et al.10 confirmed only 3 of 20 episodes of ventricular tachycardia, fibrillation, or flutter (427.1, 427.4x) reported on discharge abstracts of VA patients hospitalized in 1987-89 for CHF, COPD, or diabetes; the sensitivity for ventricular tachycardia was 43% (3/7). We were unable to find other evidence on the validity of this indicator.

Postoperative Cerebral Infarction

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 1, “postoperative cerebral infarction”). Their definition is limited to infarctions secondary to occlusion or stenosis of precerebral or cerebral arteries, and excludes nonspecific strokes. The University HealthSystem Consortium adopted this CSP indicator for major surgery patients (2919).

Evidence

Coding validity. CSP 1 had a high confirmation rate among major surgical cases (83% by coders' review, 86% by physicians' review).5, 6 Nurse reviews were not performed. An earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York in FY1993 revealed a similarly high confirmation rate of 78% (43/55) among major surgical cases, although 28% of those patients (12/43) lacked clear documentation of a new or worsening neurologic deficit.11

Geraci et al.12 confirmed 0 of 26 episodes of cerebrovascular disease (436, 437) reported on discharge abstracts of VA patients hospitalized in 1987-89 for CHF, COPD, or diabetes; the sensitivity for stroke was 0% (0/2). However, the clinical definition of this complication (stroke) was much different from the ICD-9-CM definition (“acute, but ill-defined” and “other and ill-defined” cerebrovascular disease). Romano et al. identified 2 of 6 episodes of cerebrovascular disease (433.x–435.1, 435.8, 436) using discharge abstracts of diskectomy patients at 30 California hospitals in 1990-91; there was one false positive. In comparison with the VA's National Surgical Quality Improvement Program database from 123 hospitals in 1994-95, the ICD-9-CM diagnosis of stroke (431–434.xx, 436) had a sensitivity of 70% and a predictive value of 6% for acute stroke within 30 days after surgery.8 The 1985 National DRG Validation Study also suggested that the sensitivity of Medicare hospital claims data exceeds 75% for stroke (431, 432.9, 434.x, 436), even when it is coded as a secondary diagnosis (n=36) rather than as the reason for admission. 13

Hartz and Kuhn identified only 59 of 125 (47%) strokes by applying a related indicator (997.0x) to Medicare patients who underwent coronary artery bypass surgery in Wisconsin in 1990-91; the predictive value was 54% (59/117).14 Unfortunately, we found no evidence on the validity of the specific ICD-9-CM code for postoperative cerebral infarction (997.02), which was introduced in 1995.

Construct validity. Explicit process of care failures in the CSP validation study were no more frequent among cases with CSP 1 (43%) than among unflagged controls (46%), after excluding one patient who had stroke at admission. Indeed, cases flagged on this indicator were no more likely than unflagged controls (49% versus 52%) to have at least one of five specific process-of-care problems in the earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York.11 Physician reviewers identified potential quality problems in 31% of medical patients with CSP 1 (versus 2% of unflagged controls).6

Postoperative Coma or Stupor

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 18, “postoperative coma or stupor”). Their original definition was limited to coma, stupor, and persistent vegetative state. Needleman and Buerhaus4 identified postoperative central nervous system (CNS) complications as an “Outcome Potentially Sensitive to Nursing,” but their broader definition also includes acute delirium (293.0), reactive confusion (298.2), and reactive depression (309).

Evidence

Coding validity. In comparison with the VA's National Surgical Quality Improvement Program database from 123 hospitals in 1994-95, in which only coma “persisting >24 hours postoperatively” qualifies for documentation, the ICD-9-CM diagnosis of coma (780–780.01) had a sensitivity of 16% and an uninterpretable predictive value. 8

Construct validity. Needleman and Buerhaus4 found that nurse staffing was inconsistently associated with the occurrence of CNS complications among major surgery patients from 799 hospitals in 11 states in 1997, and was independent of CNS complications among medical patients.

Postoperative Complications Related to Urinary Tract Anatomy

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 5, “postoperative complications related to urinary tract anatomy”). Their definition includes stricture or kinking or ureter and other ureteric obstruction.

Evidence

We were unable to find evidence on validity from prior studies, because this complication is quite rare.

Postoperative Gastrointestinal Hemorrhage or Ulceration

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 4, “postoperative gastrointestinal hemorrhage or ulceration following non-GI surgery”). Their definition includes hemorrhage or acute nontraumatic perforation involving the esophagus, stomach, duodenum, jejunum, or unspecified gastrointestinal tract. The University HealthSystem Consortium (2928) and AHRQ's original HCUP Quality Indicators adopted this CSP indicator for major surgery patients.2 Needleman and Buerhaus4 identified postoperative gastrointestinal hemorrhage as an “Outcome Potentially Sensitive to Nursing,” but their definition excludes alcoholic, atrophic, and hypertrophic gastritis (535.11, 535.21, 535.31, 535.51, 535.61), excludes hemorrhage due to chronic ulcer, and includes acute and unspecified ulcers without hemorrhage or perforation.

Evidence

Coding validity. CSP 4 had a moderately high confirmation rate among major surgical cases (66% by coders' review, 73% by physicians' review, 68% by nurse-abstracted clinical documentation, and 75% if nurses also accepted physicians' notes as adequate documentation).57 An earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York in FY1993 revealed a similarly high confirmation rate of 83% (68/82) among major surgical cases, although 26% (18/68) of those patients lacked laboratory or clinical evidence of significant blood loss.11

By contrast, Geraci et al.12 confirmed 1 of 10 episodes of gastrointestinal hemorrhage (531.0, 531.2, 531.4, 531.6, 532.0, 532.2, 532.4, 532.6, 533.0, 533.2, 533.4, 533.6, 534.0, 534.2, 534.4, 534.6, 535.1, 537.83, 562.02–562.03, 562.12–562.13, 569.3, 569.85, 596.7) reported on discharge abstracts of VA patients hospitalized in 1987-89 for CHF, COPD, or diabetes; the sensitivity for hemorrhage requiring transfusion was 11% (1/9).

Construct validity. Explicit process of care failures in the CSP validation study were only moderately frequent among major surgical cases with CSP 4 (28%), after excluding one patient who had gastrointestinal hemorrhage at admission.15 Cases flagged on this indicator and unflagged controls did not differ significantly on a composite of 17 generic process criteria. Similarly, cases flagged on this indicator were no more likely than unflagged controls (26% versus 22%) to have at least one of four specific process-of-care problems in the earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York.11 Physician reviewers identified potential quality problems in 38% of major surgery patients with CSP 4 (versus 2% of unflagged controls).6

Needleman and Buerhaus4 found that higher registered nurse staffing (RN hours/adjusted patient day) and better nursing skill mix (RN hours/licensed nurse hours) were consistently associated with the occurrence of upper gastrointestinal hemorrhage among medical patients from 799 hospitals in 11 states in 1997, but were independent of gastrointestinal hemorrhage among major surgery patients. An increase from the 25th to the 75th percentile on these two measures of staffing was associated with 5.2% (95% CI, 1.4% to 8.9%) and 5.1% (95% CI, 0.5% to 9.7%) decreases, respectively, in the rate of upper gastrointestinal hemorrhage among medical patients.16 Kovner and Gergen reported that among 506 community hospitals in the 1993 Nationwide Inpatient Sample, having more registered nurse hours per adjusted patient day was not associated with rates of upper gastrointestinal hemorrhage after major surgery.9

Postoperative Infection

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 23, “wound infection”). Their definition, which includes both posttraumatic wound infection and unspecified postoperative infection, was included in AHRQ's original HCUP Quality Indicators.2 Needleman and Buerhaus4 identified postoperative infection as an “Outcome Potentially Sensitive to Nursing,” using the same CSP definition. It was endorsed by Miller et al. 17 in the original “AHRQ PSI Algorithms and Groupings,” although their definition excluded posttraumatic wound infection (958.3).

Evidence

Coding validity. CSP 23 (including both 998.5x and 958.3) had a high confirmation rate among major surgical cases (91% by coders' review, 61% by physicians' review, 60% by nurse-abstracted clinical documentation), but a poor confirmation rate among medical cases (28% by coders' review, 24% by physicians' review).57 Nurse reviews were not performed on medical cases, most of which were apparently present at admission. An earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York in FY1993 revealed even poorer confirmation rates of 43% (40/93) among major surgical cases (of whom 20 or 50% lacked physical examination evidence of the diagnosis) and 8% (7/86) among medical cases (of whom 2 or 29% lacked physical examination evidence of the diagnosis).11

Keeler et al.18 reported a confirmation rate of 75% (6/8) but a sensitivity of only 27% (6/22) for postoperative infection (998.5x) among Medicare hip fracture patients from 297 hospitals in 1985-86. Massanari et al. 19 identified 45% of cases of “nosocomial wound infection” using 1984 hospital discharge data from the University of Iowa, but no definitions were provided. Faciszewski et al.20 confirmed 71% (5/7) of reported cases of postoperative infection (998.5x) among 310 patients who underwent spinal fusion at the Marshfield Clinic in 1991-92. The sensitivity of coding for this complication was 28% (5/18). Among 185 total knee replacement patients from 5 Ontario hospitals in 1984-90, Hawker et al.21 found that the sensitivity and predictive value of unspecified postoperative infection codes were both 50% (2/4). Romano et al.22 identified 5 of 8 episodes of postoperative infection (998.5x, 999.3, 996.62) using discharge abstracts of diskectomy patients at 30 California hospitals in 1990-91; there were two false positives. Hartz and Kuhn identified only 46 of 385 (12%) infections by applying this indicator (998.5, 999.3, 996.6x) to Medicare patients who underwent coronary artery bypass surgery in Wisconsin in 1990-91; the predictive value was 84% (46/55).14 Belio-Blasco et al.23 reported that “discharge forms” had a sensitivity of 57% (132/230) and a specificity of 99.9% for identifying nosocomial surgical wound infection among surgical patients in a Spanish teaching hospital. In comparison with the VA's National Surgical Quality Improvement Program database from 123 hospitals in 1994-95, the ICD-9-CM diagnosis of wound infection (998.5x) had a sensitivity of 21% and a predictive value of 35% for wound infection within 30 days after surgery.8

Construct validity. Explicit process of care failures in the CSP validation study were only moderately frequent among major surgical cases with CSP 23 (24%), after excluding two patients who had wound infections at admission, and no more frequent among medical cases with CSP 23 than among unflagged controls (2% versus 5%, respectively). Major surgical cases flagged on this indicator and unflagged controls did not differ significantly on a composite of 17 generic process criteria. Similarly, cases flagged on this indicator did not differ significantly from unflagged controls (among either major surgical or medical cases) on one specific process-of-care problem in the earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York.11 Physician reviewers identified potential quality problems in 26% of major surgery patients and 3% of medical patients with CSP 23 (versus 2% of unflagged controls for each risk group).6 Needleman and Buerhaus4 found that nurse staffing was independent of the occurrence of wound infection among major surgery patients from 799 hospitals in 11 states in 1997.

Postoperative Infections Except Pneumonia and Wound

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 16, “postoperative infections except pneumonia and wound”). Their original definition included Clostridium difficile infection (which we also considered as a separate indicator, rejected #3), bacterial meningitis, empyema with or without fistula, mediastinal abscess, mediastinitis, acute or unspecified pyelonephritis, acute lymphadenitis. The University HealthSystem Consortium adopted this CSP indicator for major surgery patients (2937). Needleman and Buerhaus 4 considered “miscellaneous nosocomial infections” as an “Outcome Potentially Sensitive to Nursing,” based on input from their Technical Expert Panel, but discarded it after concluding that it was “not codable on the basis of discharge abstracts.”

Evidence

Coding validity. CSP 16 had a relatively high confirmation rate among major surgical cases (72% by coders' review, 73% by physicians' review, 73% by nurse-abstracted clinical documentation, and 77% if nurses also accepted physicians' notes as adequate documentation).57

Construct validity. Explicit process of care failures in the CSP validation study were only moderately frequent among major surgical cases with CSP 16 (44%), after excluding a few patients who had infections at admission, but unflagged controls were not evaluated on the same criteria. Physician reviewers identified potential quality problems in 40% of major surgery patients with CSP 16 (versus 2% of unflagged controls).6 Nursing skill mix was significantly associated (in the expected direction) with the aggregate rate of postoperative infections among 352 and 295 California hospitals in 1992 and 1994, respectively, but not among 126 and 131 New York hospitals in the same years.24 However, these authors used an entirely different definition of postoperative infections, which only partially overlapped the CSP 16 definition.

Shock or Cardiopulmonary Arrest In-hospital

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 12, “shock or cardiopulmonary arrest in hospital”). Their definition includes cardiac arrest, respiratory arrest, shock, and cardiogenic shock. Needleman and Buerhaus4 identified shock or cardiac arrest as an “Outcome Potentially Sensitive to Nursing,” but their definition also includes various resuscitative procedures (93.93, 99.60, 99.63).

Evidence

Coding validity. CSP 12 had a borderline confirmation rate among major surgical cases (53% by coders' review, 74% by physicians' review).5, 6 Nurse reviews were not performed. An earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York in FY1993 revealed a similar confirmation rate of 72% (58/81) among major surgical cases, although 2% (1/58) of those patients lacked clear documentation of cardiac arrest, respiratory arrest, hypotension, or poor perfusion.11

Geraci et al.10 confirmed only 4 of 16 episodes of cardiac arrest (427.5), hypotension, or shock (458, 785.5x) reported on discharge abstracts of VA patients hospitalized in 1987-89 for CHF, COPD, or diabetes; the sensitivity for cardiac arrest or shock was 19% (4/21). Romano et al. identified 3 of 16 episodes of hypotension, shock, or cardiac arrest (785.5x, 427.5, 458.9, 998.0, 37.91) using discharge abstracts of diskectomy patients at 30 California hospitals in 1990-91; there were no false positives (but these findings are driven mostly by hypotension, a far milder diagnosis than shock). Although postoperative shock is properly assigned a different code (998.0) than other causes of shock, Keeler et al.18 reported a sensitivity of only 2% (1/55), with no false positives, for this diagnosis among Medicare hip fracture patients from 297 hospitals in 1985-86. In comparison with the VA's National Surgical Quality Improvement Program database from 123 hospitals in 1994-95, in which “cardiac arrest” is defined as involving cardiopulmonary resuscitation within 30 days after surgery, the ICD-9-CM diagnosis (427.5) had a sensitivity of 27% and a predictive value of 56%.8

Construct validity. Explicit process of care failures in the CSP validation study were no more frequent among cases with CSP 12 (44%) than among unflagged controls (46%), after excluding one patient who had shock at admission. Physician reviewers identified potential quality problems in 18% of major surgery patients with CSP 12 (versus 2% of unflagged controls).6

Needleman and Buerhaus4 found that higher registered nurse staffing (RN hours/adjusted patient day) and better nursing skill mix (RN hours/licensed nurse hours) were consistently associated with the occurrence of shock or cardiorespiratory arrest among medical patients from 799 hospitals in 11 states in 1997, but were independent of these outcomes among major surgery patients. An increase from the 25th to the 75th percentile on these two measures of staffing was associated with 4.1% (95% CI, -2.5% to 10.8%) and 9.4% (95% CI, 2.6% to 16.3%) decreases, respectively, in the rate of shock or cardiorespiratory arrest among medical patients.16

Urinary Tract Infection

Source. This indicator (599.0) was originally developed under the auspices of the Healthcare Cost and Utilization Project. Needleman and Buerhaus4 identified urinary tract infection (599.0, 996.64) as an “Outcome Potentially Sensitive to Nursing.”

Evidence

Coding validity. Massanari et al.19 identified 62% of cases of “nosocomial urinary tract infection” (UTI) using 1984 hospital discharge data from the University of Iowa, but no definitions were provided. Geraci et al.10 confirmed only 7 of 86 (8%) episodes of UTI (599.x) reported on discharge abstracts of Veterans Affairs (VA) patients hospitalized in 1987-89 for congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), or diabetes; the sensitivity for a urinary tract infection was 64% (7/11). Romano et al.22 identified 17 of 36 episodes of UTI (590.1x, 590.2, 590.8x, 590.9, 595.0, 595.9, 599.0, 996.64) using discharge abstracts of diskectomy patients at 30 California hospitals in 1990-91; there were five false positives. Belio-Blasco et al.23 reported that “discharge forms” had a sensitivity of 38% (33/87) and a specificity of 99.9% for identifying nosocomial UTIs among surgical patients in a Spanish teaching hospital. In comparison with the VA's National Surgical Quality Improvement Program database from 123 hospitals in 1994-95, an ICD-9-CM diagnosis of kidney, bladder, or urinary tract infection (590.x, 595.x, 599.0) had a sensitivity of 45% and a predictive value of 24% for UTIs within 30 days after surgery (excluding catheter-related infections, 996.64).8

Construct validity. Needleman and Buerhaus4 found that higher registered nurse staffing (RN hours/adjusted patient day) and better nursing skill mix (RN hours/licensed nurse hours) were consistently associated with the occurrence of UTI among medical patients from 799 hospitals in 11 states in 1997. An increase from the 25th to the 75th percentile on these two measures of staffing was associated with 3.6% (95% CI, 1.2% to 6.0%) and 9.0% (95% CI, 6.1% to 11.9%) decreases, respectively, in the rate of UTI among medical patients.16 Nursing skill mix was associated with the UTI rate among major surgery patients (rate ratio 0.48, 95% CI 0.38–0.61), but aggregate registered nurse staffing was not (rate ratio 0.99, 95% CI 0.98–1.00). An increase from the 25th to the 75th percentile on nursing skill mix was associated with a 4.9% (95% CI, 0.3% to 9.5%) decrease in the rate of UTI among major surgery patients. These findings are consistent with Kovner and Gergen, who reported that among 506 community hospitals in the 1993 Nationwide Inpatient Sample, having more registered nurse hours per adjusted patient day was associated with a lower rate of UTI after major surgery.9 Nursing skill mix was significantly associated (in the expected direction) with the UTI rate among 352 and 295 California hospitals in 1992 and 1994, respectively, and among 131 New York hospitals in 1994.24 Total licensed nurses were not associated with the UTI rate in either state or either time period.

Section 2. Literature Review Results for Indicators Rejected Post-panel Review

Dosage Complications

Source. This diagnosis code was originally proposed by Iezzoni et al.1 as one component of a much broader indicator (CSP 28, “complications related to drugs”), which was part of the CSP. It was endorsed by Miller et al. 17 as one component of a broader indicator (“E codes”) in the original “AHRQ PSI Algorithms and Groupings.”

Evidence

Coding validity. This indicator, as defined in CSP, is highly problematic among medical cases (10% confirmation by coders, 20% by physicians), apparently because most drug-related complications are present at admission.5, 6 The AHRQ definition, and the present PSI definition, differ by excluding all of the poisoning codes. No evidence on the validity of the E code subset, by itself, is available from prior studies.

Construct validity. Explicit process of care failures in the CSP validation study were very unusual among medical cases with CSP 28 (2%), and no more frequent than among unflagged controls (5%). Physician reviewers identified potential quality problems in 16% of medical patients with CSP 28 (versus 2% of unflagged controls).6 Based on two-stage implicit review of 8,109 randomly selected deaths from 104 New York hospitals in 1985-86, Hannan et al. found that cases with a secondary diagnosis of “selected drug poisonings” were no more likely to have received “care that departed from professionally recognized standards” than cases without such codes (2.5% versus 1.7%, OR=1.09), after adjusting for patient demographic, geographic, and hospital characteristics.3

Iatrogenic Hypotension

Source. This diagnosis code was proposed by Miller et al.17 as one component of a broader indicator (“iatrogenic conditions”), which was part of the original “AHRQ PSI Algorithms and Groupings.” It was also included as one component of a broader indicator (“adverse events and iatrogenic complications”) in AHRQ's Version 1.3 HCUP Quality Indicators.2

Evidence

We were unable to find evidence on validity from prior studies, because this diagnosis code was introduced in 1995.

Intestinal Infection Due to Clostridium difficile

Source. This diagnosis code was originally proposed by Iezzoni et al.1 as one component of a much broader indicator (CSP 16, “postoperative infections except pneumonia and wound”), which was part of the CSP.

Evidence

Coding validity. No evidence on validity is available from CSP studies, because this code was grouped with other postoperative infections. Geraci et al.12 identified 0 of 6 episodes of antibiotic-associated diarrhea using the discharge abstracts of VA patients hospitalized in 1987-89 for CHF, COPD, or diabetes. However, the clinical definition of this complication (antibiotic-associated diarrhea) was much broader than the ICD-9-CM definition (Clostridium difficile colitis).

Postoperative Iatrogenic Complications - Digestive

Source. This diagnosis code was originally proposed by Iezzoni et al.1 as one component of a much broader indicator (CSP 26, “iatrogenic complications”), which was part of the CSP. Their definition includes central nervous system, cardiac, peripheral vascular, respiratory, gastrointestinal, urinary, and unspecified amputation stump complications, as well as complications affecting other body systems. It was also included as one component of a broader indicator (“adverse events and iatrogenic complications”) in AHRQ's original HCUP Quality Indicators.2 The University HealthSystem Consortium adopted this CSP indicator for cardiac procedure patients (2913).

Evidence

Coding validity. CSP 26 had a very high confirmation rate among major surgical cases (92% by coders' review) and a borderline confirmation rate among medical cases (59% by coders' review).5 Physician reviews were not performed. Faciszewski et al. 20 confirmed 48% (10/21) of reported cases of gastrointestinal complications (997.4) among 310 patients who underwent spinal fusion at the Marshfield Clinic in 1991-92. The sensitivity of coding for this complication was 40% (10/25). Romano et al.22 identified 7 of 15 episodes of gastrointestinal complications (with 3 false positives) using discharge abstracts of diskectomy patients at 30 California hospitals in 1990-91.

Construct validity. Explicit process of care failures in the CSP validation study were slightly but not significantly more frequent among cases with CSP 26 (58% surgical, 9% medical) than among unflagged controls (46% surgical, 5% medical).

Postoperative Iatrogenic Complications - Respiratory

Source. This diagnosis code was originally proposed by Iezzoni et al.1 as one component of a much broader indicator (CSP 26, “iatrogenic complications”), which was part of the CSP. Their definition includes central nervous system, cardiac, peripheral vascular, respiratory, gastrointestinal, urinary, and unspecified amputation stump complications, as well as complications affecting other body systems. It was also included as one component of a broader indicator (“adverse events and iatrogenic complications”) in AHRQ's original HCUP Quality Indicators.2 The University HealthSystem Consortium adopted this CSP indicator for cardiac procedure patients (2913).

Evidence

Coding validity. CSP 26 had a very high confirmation rate among major surgical cases (92% by coders' review) and a borderline confirmation rate among medical cases (59% by coders' review).5 Physician reviews were not performed. Faciszewski et al.20 confirmed 48% (11/23) of reported cases of respiratory complications (997.3) among 310 patients who underwent spinal fusion at the Marshfield Clinic in 1991-92. The sensitivity of coding for this complication was 55% (11/20). Romano et al.22 identified 2 of 10 episodes of respiratory complications (with 7 false positives) using discharge abstracts of diskectomy patients at 30 California hospitals in 1990-91.

Construct validity. Explicit process of care failures in the CSP validation study were slightly but not significantly more frequent among cases with CSP 26 (58% surgical, 9% medical) than among unflagged controls (46% surgical, 5% medical). We were unable to find other evidence on the validity of this indicator.

Postoperative Iatrogenic Complications - Urinary

Source. This indicator was originally proposed by Hannan et al. as a criterion for targeting “cases that would have a higher percentage of quality of care problems than cases without the criterion, as judged by medical record review.”3 It was endorsed by Iezzoni et al.1 as one component of a much broader indicator (CSP 26, “iatrogenic complications”) in the CSP. The definition of that indicator includes central nervous system, cardiac, peripheral vascular, respiratory, gastrointestinal, urinary, and unspecified amputation stump complications, as well as complications affecting other body systems. It was also included as one component of a broader indicator (“adverse events and iatrogenic complications”) in AHRQ's original HCUP Quality Indicators.2 The University HealthSystem Consortium adopted this CSP indicator for cardiac procedure patients (2913).

Evidence

Coding validity. CSP 26 had a very high confirmation rate among major surgical cases (92% by coders' review) and a borderline confirmation rate among medical cases (59% by coders' review).5 Physician reviews were not performed. Faciszewski et al. 20 confirmed 56% (5/9) of reported cases of genitourinary complications (997.5) among 310 patients who underwent spinal fusion at the Marshfield Clinic in 1991-92. The sensitivity of coding for this complication was 19% (5/26). Among 185 total knee replacement patients from 5 Ontario hospitals in 1984-90, Hawker et al.21 found that the sensitivity and predictive value of urinary tract complications (definition not given) were 38% (6/16) and 50% (6/12), respectively. Romano et al. identified 5 of 17 episodes of urinary complications (996.76, 997.5), with 8 false positives, using discharge abstracts of diskectomy patients at 30 California hospitals in 1990-91. Hartz and Kuhn identified only 18 of 113 (16%) episodes of acute renal failure (defined as an increase in serum creatinine of more than 1.0 mg/dL, resulting in a final value greater than 2.5 mg/dL) by applying this indicator to Medicare patients who underwent coronary artery bypass surgery in Wisconsin in 1990-91; the predictive value was 27% (18/66).14

Construct validity. Explicit process of care failures in the CSP validation study were slightly but not significantly more frequent among cases with CSP 26 (58% surgical, 9% medical) than among unflagged controls (46% surgical, 5% medical). Based on two-stage review of 8,109 randomly selected deaths from 104 New York hospitals in 1985-86, Hannan et al.3 reported that cases with a secondary diagnosis of 997.5 (urinary) were 3.2 times more likely to have received care that departed from professionally recognized standards than cases without that code (6.0% versus 1.7%), after adjusting for patient demographic, geographic, and hospital characteristics. In 4 of these 9 cases (44%) of substandard care, the patient's death was attributed at least partially to that care.

Postoperative Iatrogenic Complications - Vascular

Source. This diagnosis code was originally proposed by Iezzoni et al.1 as one component of a much broader indicator (CSP 26, “iatrogenic complications”), which was part of the CSP. Their definition includes central nervous system, cardiac, peripheral vascular, respiratory, gastrointestinal, urinary, and unspecified amputation stump complications, as well as complications affecting other body systems. It was also included as one component of a broader indicator (“adverse events and iatrogenic complications”) in AHRQ's original HCUP Quality Indicators.2 The University HealthSystem Consortium adopted this CSP indicator for cardiac procedure patients (2913).

Evidence

Coding validity. CSP 26 had a very high confirmation rate among major surgical cases (92% by coders' review) and a borderline confirmation rate among medical cases (59% by coders' review).5 Physician reviews were not performed.

Construct validity. Explicit process of care failures in the CSP validation study were slightly but not significantly more frequent among cases with CSP 26 (58% surgical, 9% medical) than among unflagged controls (46% surgical, 5% medical). We were unable to find other evidence on the validity of this indicator.

Postoperative Pneumonia

Source. This indicator was originally proposed by Iezzoni et al.1 as part of the CSP (CSP 19, “postoperative pneumonia”). Their definition includes virtually all bacterial causes of pneumonia (481–483, 485–486). Needleman and Buerhaus 4 identified postoperative pneumonia as an “Outcome Potentially Sensitive to Nursing,” but their definition aggregates bacterial, aspiration (507.0), and “hypostatic” (514) pneumonia, includes nonspecific respiratory complications (997.3), and excludes pneumococcal (481) and atypical (483) pneumonias. The University HealthSystem Consortium (2943) and AHRQ's original HCUP Quality Indicators adopted this CSP indicator for major surgery patients.2

Evidence

Coding validity. CSP 19 had a moderate confirmation rate among major surgical cases (unreported by coders' review, 64% by physicians' review, 48% by nurse-abstracted clinical documentation, and 76% if nurses also accepted physicians' notes as adequate documentation). 6, 7 An earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York in FY1993 revealed a similar confirmation rate of 76% (75/99) among major surgical cases, although 17% of those patients (13/75) lacked radiographic or laboratory evidence supporting the diagnosis.11

Keeler et al.18 reported a confirmation rate of 75% (30/40) but a sensitivity of only 26% (30/116) for pneumonia (482.x, 485, 486, 997.3, 998.5, 999.3) among Medicare hip fracture patients from 297 hospitals in 1985-86. All of the false positives in that study were due to 900-series codes. Massanari et al.19 identified 61% of cases of “nosocomial lower respiratory tract infection” using 1984 hospital discharge data from the University of Iowa, but no definitions were provided. Geraci et al.12 confirmed (by chest radiography) 0 of 7 episodes of pneumonia (482.9, 507.0) reported on discharge abstracts of VA patients hospitalized in 1987-89 for CHF, COPD, or diabetes; the sensitivity for a new alveolar infiltrate was 0% (0/5). Romano et al. 22 identified 1 of 1 episode of pneumonia (480.0–487.0, 507.0, 510.x, 513.x), with 3 false positives, using discharge abstracts of diskectomy patients at 30 California hospitals in 1990-91. Belio-Blasco et al. 23 reported that “discharge forms” had a sensitivity of 44% (29/66) and a specificity of 99.9% for identifying nosocomial pneumonia among surgical patients in a Spanish teaching hospital. In comparison with the VA's National Surgical Quality Improvement Program database from 123 hospitals in 1994-95, in which pneumonia is defined as a radiographic infiltrate associated with purulent sputum, positive culture/viral isolation, or seroconversion within 30 days after surgery, ICD-9-CM diagnoses (480–487.0) had a sensitivity of 38% and a predictive value of 41%.8? Adding “respiratory complications” (997.3) to the definition increased the sensitivity for pneumonia to 50%, but decreased the positive predictive value to 34%.

Construct validity. Explicit process of care failures in the CSP validation study were very frequent among major surgical cases with CSP 19 (83%), after excluding two patients who had pneumonia at admission.15 Cases flagged on this indicator and unflagged controls did not differ significantly on a composite of 17 generic process criteria. Indeed, cases flagged on this indicator were significantly less likely than unflagged controls (20% versus 64%) to have at least one of four specific process-of-care problems in the earlier study of elderly Medicare beneficiaries from Massachusetts, Alabama, Iowa, and New York.11 Physician reviewers identified potential quality problems in only 5% of major surgery patients with CSP 19 (versus 2% of unflagged controls).6 The striking discrepancy between the results of explicit nurse review and implicit physician review is not explained.

Needleman and Buerhaus4 found that higher registered nurse staffing (RN hours/adjusted patient day) and better nursing skill mix (RN hours/licensed nurse hours) were consistently associated with the occurrence of pneumonia (including aspiration and “hypostatic” pneumonia) among medical patients from 799 hospitals in 11 states in 1997. An increase from the 25th to the 75th percentile on these two measures of staffing was associated with 2.7% (95% CI, -0.4% to 5.8%) and 6.4% (95% CI, 2.8% to 10.0%) decreases, respectively, in the rate of pneumonia.16 Skill mix was “weakly” associated with the rate of pneumonia among major surgical patients. These findings are consistent with Kovner and Gergen, who reported that among 506 community hospitals in the 1993 Nationwide Inpatient Sample, having more registered nurse hours per adjusted patient day was associated with a lower rate of pneumonia after major surgery.9 Nurse staffing was not associated with the rate of pneumonia after invasive vascular procedures. Nursing skill mix was significantly associated (in the expected direction) with the pneumonia rate among 352 and 295 California hospitals in 1992 and 1994, respectively, but not among 126 and 131 New York hospitals in the same years.24

Unexpected Length of Stay (LOS)/Conditional LOS

Source. This indicator was originally proposed by Kuykendall et al.25 as a relatively unbiased tool to identify potential quality of care problems. The underlying premise was that significant complications increase LOS, and therefore unexpectedly long LOS may be a marker for inpatient complications. Poor provider adherence to normative practices may lead to either unexpectedly short or unexpectedly long LOS.

Evidence

Kuykendall et al's original analysis was based on linked medical records and administrative data for 1,477 patients who were discharged from 9 VA hospitals in 1987-89 with a primary diagnosis of diabetes, (COPD), or CHF. They used administrative data with or without additional clinical data (e.g., APACHE Acute Physiology Score) to derive expected LOS through multiple linear regression. Outliers were defined as patients whose deviation from expected LOS (expressed as a proportion of expected LOS) was either below the first quartile or above the third quartile. When this method was used to identify possible complications, and then compared with detailed chart abstraction, it had a sensitivity of 40%, 62%, and 54% for complications of diabetes, COPD, and CHF, respectively. By contrast, the sensitivity of the corresponding ICD-9-CM complication codes was 26%, 39%, and 33%, respectively. The confirmation rate, or predictive value, of unexpectedly high LOS was 20%, 29%, and 27% for diabetes, COPD, and CHF, respectively. These estimates were quite similar to the predictive values of ICD-9-CM codes (21%, 32%, and 33%, respectively). We were unable to find any independent validation of these findings.

More recently, Silber et al. proposed a more complex method for using LOS to identify adverse patient outcomes.26 Their method is based on the observation that with each passing day, patients are increasingly likely to be discharged until a transition point is reached, at which patients become less likely to be discharged the longer they have stayed. Silber et al. focus on the minority of patients whose hospital stay is prolonged beyond the transition point, and estimate the length of additional stay (LAS) beyond this point. Cox proportional hazards models were used to estimate LAS among prolonged-stay patients admitted for appendectomy and pneumonia, adjusting for demographic and clinical characteristics (e.g., MedisGroups severity score). We were unable to find any independent validation of these findings.

Obstetric Thrombosis or Embolism

Source. This indicator was created after review of ICD-9-CM codes.

Evidence

Coding validity. In a stratified probability sample of 1,611 vaginal and cesarean deliveries from 51 California hospitals in 1992-93, the weighted sensitivity and predictive value of coding for thromobembolic complications of delivery, using a broader definition that included all peripheral vascular complications (997.2) and nonthrombotic pulmonary emboli (673.1x, 673.3x, 673.8x), were 0% (0/6) and 100% (6/6), respectively.27 We were unable to find evidence on validity from prior studies, because this complication is quite rare.

Puerperal Infection

Source. This indicator (670.0x) was created after review of ICD-9-CM codes. It was also included as one component of a broader indicator (“obstetrical complications”) in AHRQ's original HCUP Quality Indicators.2

Evidence

In a stratified probability sample of 1,611 vaginal and cesarean deliveries from 51 California hospitals in 1992-93, the weighted sensitivity and predictive value of coding for puerperal infection and acute or unspecified endometritis (615.0, 615.9) were 45% (45/124) and 98% (45/53), respectively.27 We were unable to find other evidence on validity from prior studies.

Section 3. Clinician Panel Review Detailed Results for Rejected Indicators

Dosage Complications

This indicator is intended to flag cases of complications due to dosage errors that can be identified using administrative data. It is intended to capture all cases of dosage complications, not only those occurring in-hospital.

Definition

Quality Measure Number of events per 100 discharges of population at risk
Numerator Discharges with ICD-9-CM code denoting a dosage complication [Excessive amount of blood or other fluid during transfusion or infusion (E873.0), Incorrect dilution of fluid during infusion. (E873.1), Overdose of radiation in therapy (E873.2) Inadvertent exposure of patient to radiation during medical care (E873.3) Failure in dosage in electroshock or insulin-shock therapy (E873.4), Inappropriate too hot or too cold temperature in local application and packing (E873.5), Non-administration of necessary drug or medicinal substance (E873.6), Other specific failure in dosage excludes accidental overdose of drug (E873.8) Unspecified failure in dosage (E873.9), Wrong fluid in infusion (E876.1)] in any diagnosis field per 100 discharges.
Denominator Exclude all obstetric admissions (MDC 14 and 15).

Post-conference call panel ratingsa

Question Median Agreement status
Overall rating 4Disagreement
Not present on admission 7Indeterminate agreement
Preventability 8Agreement
Due to medical error 8Agreeement
Charting by physicians 3Indeterminate agreement
Bias (lower rating is favorable) 4Indeterminate agreement
a

Medical Complications 2 Multispecialty Panel

Changes to the indicator

Panelists did not suggest any changes to this indicator.

Concerns not addressable through changes

Panelists expressed a multitude of concerns regarding this indicator. The definition of this indicator included a variety of dosage complications, coded as E873.x. These complications do not include failure in dosage of a medicinal substance, or accidental poisoning. Adverse drug events are difficult to ascertain from administrative data. Panelists felt that the included dosage complications were often of dubious clinical importance, and in some cases very rare. Panelists also noted that a better denominator, but one that cannot be operationalized using administrative data, would be number of doses, rather than all patients most of whom would never have been exposed to the treatments measured in this indicator.

Panelists also expressed great concern regarding the documentation of these events. According to panelists, most of these events would not result in significant clinical sequelae, and therefore would be unreliably reported. Panelists noted that this indicator would have very poor sensitivity, and thus would not be useful. In addition, using an indicator with such poor sensitivity may unfairly punish those hospitals with the most detailed reporting systems for quality improvement. It may even discourage reporting of these events in some facilities. Due to the difficulties with this indicator, panelists felt that if this indicator were to be implemented, it would have to be used to identify cases for further internal review.

Summary

Because of the serious concerns surrounding this indicator, and since most of these could not be addressed using administrative data, panelists rated this indicator as poor and suggested that it not be used. Although panelists agreed that when the events did occur they were due to error, and expressed interest in following some of these complications, as well as other types of dosage complications, potential problems with this indicator were considered too great for use.

Iatrogenic Hypotension

This indicator is intended to flag cases of hypotension caused by medical care. The area level indicator is intended to capture all cases of iatrogenic hypotension, not only those occurring in-hospital. The hospital level indicator is restricted to secondary diagnoses, and is intended to capture cases occurring during the same hospitalization. Trauma patients are excluded as they may be more susceptible to non-preventable iatrogenic hypotension.

Definition

Quality Measure Number of events per 100 discharges of population at risk
Numerator Discharges with ICD-9-CM code of 458.2 in any diagnosis field per 100 discharges.
Denominator Exclude all obstetric admissions (MDC 14 and 15).
Exclude patients with any diagnosis of [trauma]

Post-conference call panel ratingsa

Question Median Agreement status
Overall rating 5Disagreement
Not present on admission 8Agreement
Preventability 4Indeterminate agreement
Due to medical error 5Indeterminate agreement
Charting by physicians 3Disagreement
Bias (lower rating is favorable) 6Indeterminate agreement
a

Procedural Complications Multispecialty Panel

Changes to the indicator

No changes were made to this indicator, as panelists felt that no changes would rectify concerns.

Concerns not addressable through changes

Panelists had many concerns regarding this indicator, especially related to the preventability and charting of this complication. First, panelists commented frequently on the unclear preventability of many cases of hypotension. While some cases may result from poor management of fluids and medication, hypotension in general often has multifactorial etiologies. Comorbidities, such as diabetes or congestive heart failure, or even the psychological state of the patient, may contribute to the development of hypotension. Panelists expressed concern that the cause of the hypotension is often difficult to identify.

Panelists also expressed great concern over the documentation of hypotension. The term ‘hypotension’ is not intrinsically connected to an objective physiological state. What one physician calls ‘hypotension’ another physician may not, depending on the severity and duration of the hypotension. This ambiguity leads to variable documentation and potentially systematic bias from variability in reporting. One panelist noted that blood pressures recorded by anesthesiologists may be rounded, effecting reporting as well. Finally, documentation is subject to the vigilance of monitoring of blood pressure. Panelists also expressed concern that hypotension may not be labeled often as iatrogenic, and thus will be coded elsewhere.

Summary

This indicator was rated as poor by panelists, primarily due to concern about the reliability of reporting and coding. In addition, many panelists felt that this complication may be less preventable than others reviewed. Panelists suggested that this indicator be dropped from further consideration.

Intestinal Infection Due to Clostridium Difficile

This indicator is intended to identify patients that may have acquired an intestinal infection (due to C. difficile) in-hospital. In order to eliminate infections present on admission, this indicator includes only secondary diagnoses (meaning the infection was not designated as the principal diagnosis).

Definition

Methods:
Quality Measure Number of events per 100 discharges of population at risk
Numerator Discharges with ICD-9-CM code of 008.45 in any secondary diagnosis field per 100 discharges.
Denominator Exclude all obstetric admissions (MDC 14 and 15).
Benchmark State, regional, or peer group average.

Post-conference call panel ratingsa

Question Median Agreement status
Overall rating 3Disagreement
Not present on admission 7Indeterminate agreement
Preventability 3Disagreement
Due to medical error 3Indeterminate agreement
Charting by physicians 7Disagreement
Bias (lower rating is favorable) 6Indeterminate agreement
a

Medical Complications 1 Multispecialty Panel

Changes to the indicator

None of the concerns raised by panelists were addressed by changing the specification of this indicator.

Concerns not addressable through changes

Most of the concerns surrounding this indicator were not addressable using administrative data. Concerns focused primarily on the potential for bias due to varying diagnostic practices, and differences in the number of patients with the infection present on admission. Panelists expressed that particularly for patients admitted from long term care facilities, some patients might have the disorder present on admission. At times, this infection may not be fully symptomatic at admission, but may develop into a fully symptomatic condition during the hospitalization. Similarly, the diagnosis of infection due to C. difficile is often missed, or not charted as such. A stool culture is required for a definitive diagnosis. Often physicians may treat “diarrhea” without actually obtaining a culture; in this case “diarrhea not otherwise specified” would be reported, and would include cases of C. difficile. The differences in charting may be a significant source of bias for this indicator. Specifically, some hospitals may routinely screen for this common complication, while others may not. The rate as detected by the indicator may be particularly high in facilities that screen. Panelists cautioned that implementation of an administrative data indicator for C. difficile has the potential to reduce screening for such infections.

Panelists also expressed that preventability of this complication varies, depending on the cause of the complication. Infections that result from cross-contamination between patients may be prevented through hand washing, isolation procedures, or other precautions. On the other hand, infections may also occur secondary to appropriate antibiotic use.

Summary

Panelists rated this indicator as poor due to concerns that this operationalization did not exclusively pick up nosocomial infections, and that this complication may not be reliably charted or may be screened for in some facilities. Although panelists expressed interest in tracking nosocomial C. difficile infections given better data, they suggested that this indicator not be considered further due to the multiplicity of concerns.

Postoperative Iatrogenic Complications - Digestive

Postoperative Iatrogenic Complications - Respiratory

Postoperative Iatrogenic Complications - Vascular

Postoperative Iatrogenic Complications - Urinary

These indicators were rated in one indicator, reported in the “Experimental” indicator results section in the main body of the report.

Postoperative Pneumonia

This indicator is intended to flag cases of postoperative pneumonia. It is identical to an indicator developed as part of the Complications Screening Program. This indicator limits pneumonia codes to secondary diagnosis codes in order to eliminate pneumonia that was present on admission. It further excludes patients who have major respiratory disorders, as these patients may have pneumonia present on admission, or may be more likely to develop pneumonia after surgical procedures. Finally, it excludes patients with immunosupression, including cancer and AIDS patients, as these patients are particularly susceptible to developing pneumonia.

Defintion

Quality Measure Number of events per 100 discharges of population at risk
Numerator Discharges with ICD-9-CM codes for pneumonia [pneumococcal pneumonia (481), other bacterial pneumonia {Klebsiella pneumoniae, pseudomoniae, pseudomonas, Hemophilis pneumoniae, streptococcus, stapnylococcus, anaerobes, e. coli, other gram negative, Legionnaires disease} (482.0–482.99)] in any secondary diagnosis field per 100 surgical discharges.
Denominator All [surgical] discharges
Exclude patients in MDC 4.
Exclude patients with any diagnosis of [AIDS], [immunocompromised] state or [cancer]

Post-conference call panel ratingsa

Question Median (MS) Agreement status (MS) Median (S) Agreement status (S)
Overall rating 5Indeterminate6Indeterminate
Not present on admission 7Indeterminate8Indeterminate
Preventability 4Indeterminate6Indeterminate
Due to medical error 2Agreement6Indeterminate
Charting by physicians 6Indeterminate7Indeterminate
Bias (lower rating favorable) 7Agreement7Indeterminate
a

Multispecialty Panel - Surgical Complications 1

Surgical Panel - Surgical Complications 1

Multi-specialty Panel Results

Changes to the indicator

There were no changes suggested to this indicator that would address the specific concerns of the panel.

Concerns not addressable through changes

Panelists were most concerned about the definition of pneumonia. Different physicians utilize different thresholds in diagnosing pneumonia. What some physicians may call atelactasis, other physicians may define as pneumonia. In addition, different methods are used to diagnose pneumonia. Some physicians may use clinical criteria such as examining x-rays for infiltrate, or requiring fever, yellow sputum, or elevated white blood cell count. Others may require a positive bronchoscopy culture. Because these different thresholds will yield different rates, panelists were concerned about the consistency of charting of this complication. They were also concerned that short length of stay would result in missing postoperative pneumonia that develops after discharge. Similarly, outpatient surgeries also involve risk for post operative pneumonia, but this indicator would not capture these cases either.

Panelists did express that despite the problems with this indicator, they remain interested in tracking the pneumonia rate, but believed that current administrative data is not the appropriate data source. It would be important and useful to track ventilator pneumonia, and other nosocomial pneumonias. They believed that many of these pneumonias are preventable, with current interventions, such as bed elevation, cross contamination prevention, and when appropriate, prophylactic antibiotics. Panelists were concerned about some bias with ventilator pneumonia, specifically the development of ventilator pneumonia depends on length of time on the ventilator, and comorbidities in the patient, such as serious illness, or immunocompromised state.

Surgical Panel Results

Changes to the indicator

The surgical panel suggested that trauma to the head and chest should be excluded. Chest trauma patients may appear to have pneumonia upon x-ray evaluation because of pulmonary contusion and or hemorrhage, or may be at higher risk for developing non-preventable pneumonia. Head trauma patients may have aspirated at the time of trauma leading to pneumonia. Although the diagnosis code for aspiration pneumonia is not included in this indicator, pneumonia without specified organisms is included and thus, some aspiration pneumonia may appear in this indicator.

Concerns not addressable through changes

The surgical panel expressed concern regarding potential bias for this indicator, given the potential effects of different patient case mix, particularly for some pre-existing disease (e.g., pulmonary diseases, diabetes) or behavioral risk factors (e.g., smoking). Panelists also indicated that the type of surgery would influence postoperative pneumonia rates (e.g., likely elevated rates for chest surgery or abdominal surgery). They suggested that this indicator be risk adjusted or stratified according to the type of procedure performed.

Summary across Panels

Both panels rated this indicator relatively poorly. Great concern was expressed regarding variation in diagnosis of pneumonia. Internist, intensivists and nurses directly treating postoperative pneumonia particularly expressed this concern. Although this indicator was not included in the final Accepted or Experimental indicator sets due to the concerns raised, panelists were hopeful that clinical measures to track postoperative pneumonia rate would be developed.

Obstetric Thrombosis or Embolism

This indicator is intended to flag cases of potentially preventable obstetric thrombosis or embolism in women delivering during the index hospitalization.

Definition

Quality Measure Number of events per 100 discharges of population at risk
Numerator Discharges with ICD-9-CM codes for obstetric thrombosis or embolism [DVT -postpartum unspecified (671.40), DVT- delivered with mention of postpartum complication (671.42), DVT - postpartum condition or complication (671.44), Obstetric pulmonary embolism (673.20)] in any diagnosis field per 100 deliveries.
Denominator All deliveries ([vaginal delivery],[cesarean delivery]).

Post-conference call panel ratingsa

Question Median Agreement status
Overall rating 3.5Disagreement
Not present on admission 6Indeterminate agreement
Preventability 2.5Indeterminate agreement
Due to medical error 2Indeterminate agreement
Charting by physicians 8Agreement
Bias (lower rating is favorable) 6.5Indeterminate Agreement
a

Obstetric Complications 2 Panel

Changes to the indicator

Panelists suggested no changes to this indicator.

Concerns not addressable through changes

Panelists expressed strong concern about this indicator. First, panelists questioned the preventability of post-partum vascular complications because of their unpredictable nature, and primary relationship to patient factors such as substance use and comorbidities. Some panelists did note that antepartum vascular complications might be preventable; however, it is not possible to track these events using the available administrative data.

Summary

Panelists rated this indicator as poor, and suggested that this is not a complication that was of interest to track and that this indicator should not be considered further.

Puerperal Infection

This indicator is intended to flag cases of potentially preventable puerperal infections in women delivering during the index hospitalization. This indicator excludes patients with infection of the amniotic cavity, as infection in these patients is more likely to be present on admission or non-preventable.

Definition

Quality Measure Number of events per 100 discharges of population at risk
Numerator Discharges with ICD-9-CM codes for major puerperal infection [Major puerperal infection, unspecified as to episode of care (670.00), Major puerperal infection, delivered with mention of post-partum complication (670.02), Major puerperal infection, post-partum condition or complication (670.04)] in any diagnosis field per 100 deliveries.
Denominator All deliveries ([vaginal delivery],[cesarean delivery]).
Exclude patients with a diagnosis code of antepartum infection of amniotic cavity [65840, 1, 3].

Post-conference call panel ratingsa

Question Median Agreement status
Overall rating 5Agreement
Not present on admission 6.4Indeterminate agreement
Preventability 4.5Indeterminate agreement
Due to medical error 3Indeterminate agreement
Charting by physicians 7Agreement
Bias (lower rating is favorable) 4.5Indeterminate agreement
a

Obstetric Complications 2 Panel

Changes to the indicator

No changes were suggested for this indicator.

Concerns not addressable through changes

Several concerns about this indicator were raised as reasons for the poor overall rating. Panelists felt that some hospitals may have a higher rate of these complications due to patient case mix. Specifically, they noted that patients with sexually transmitted diseases or overall poor health are more likely to develop these complications. They noted that these factors vary systematically with socioeconomic status. Further, many of these complications develop after discharge. Thus, there may be significant underreporting resulting from the exclusive use of inpatient data. Finally, panelists expressed concern that the use of this indicator would lead to the inappropriate overuse of antibiotics.

Summary

This indicator was rated less favorably than most other indicators, and panelists had no suggestions to improve the indicator. This indicator was not considered further.

Unexpected LOS/ Conditional LOS

This indicator is intended to identify patients who have unusually long lengths of stay. It is hypothesized that these patients have unusually long stays because they have developed major complications. Therefore, this measure is intended as a proxy for complications, compensating for problems of undercoding or bias in complications measures. This definition of unexpected length of stay was proposed by David Kuykendall (1995), although the original definition included demographic and longitudinal variables not available using administrative data.

Definition

Quality Measure Number of events per 100 discharges of population at risk
Numerator Unexpected: For each patient a predicted length of stay is calculated using a multiple linear regression model. The predicted length of stay depends on the principal diagnosis, age, and comorbidities of the patient. Then, an unexpected length of stay percentage is calculated:
(actual LOS - predicted LOS)/predicted LOS. Patients whose percentage is in the upper quartile (top 25%) are considered to have unusually long lengths of stay. (Kuykendall, 1995)
Conditional: Patients with an extended length of stay have a hospital stay that is longer than the “extended length of stay point” defined as the point in the distribution (days stayed) where, for any particular DRG, the rate of discharge changes from increasing to decreasing. In other words, at some point, for a group of patients within a DRG, fewer patients are discharged than were discharged on the previous day, and more patients are held in the hospital for longer stays (Silber, 1999).
Denominator All [Surgical] and [Medical] patients.

Post-conference call panel ratings

Question Median Agreement status
Overall rating 6Indeterminate
Not present on admission Not applicableNot applicable
Preventability 6Indeterminate agreement
Due to medical error 4.5Indeterminate agreement
Charting by physicians 8Agreement
Bias (lower rating is favorable) 7Agreement

Changes to the indicator

Panelists did not suggest any changes to this indicator.

Concerns not addressable through changes

Panelists had many concerns and mixed feelings about this indicator. Some panelists felt that length of stay was influenced by many factors besides quality of care. For instance, some providers extend length of stay for social reasons. Patients with little outside social support or resources may be unable to obtain home care, may not have follow-up medical care, or may have other health conditions that affect their ability to heal. For these reasons a patient may be hospitalized longer than other patients with the same condition. Panelists felt that if this indicator were to be used, it would be best used in comparing hospitals with similar case-mixes of underserved populations. Other factors that may influence length of stay that are unrelated to quality of care include age of the patient and certain comorbidities that may not be charted.

Panelists expressed mixed feeling regarding the validity of this indicator as a whole. Some noted that the validity of the concept of unusual length of stay being a proxy for complications may be more valid for surgical patients rather than medical patients, for whom many additional factors besides the development of complications may affect length of stay. Some panelists noted that this indicator is best used internally, as it could be misconstrued by the public, and that length of stay may better measure resource use rather than clinical quality of care.

Summary

Panelists were ambivalent about this indicator. Some felt that this indicator was of interest to track, but more felt that this indicator did not have sufficient face validity as a complications indicator. Panelists felt that this indicator should not be considered further.

References for Appendix F

1.
Iezzoni LI, Daley J, Heeren T, Foley SM, Fisher ES, Duncan C. et al. Identifying complications of care using administrative data. Med Care. 1994;32(7):700–15. [PubMed: 8028405]
2.
Johantgen M, Elixhauser A, Bali JK, Goldfarb M, Harris DR. Quality indicators using hospital discharge data: state and national applications. Jt Comm J Qual Improv 1998;24(2):88–105. Published erratum appears in Jt Comm J Qual Improv 1998;24(6):341. [PubMed: 9547683]
3.
Hannan EL, Bernard HR, O'Donnell JF, Kilburn H Jr. A methodology for targeting hospital cases for quality of care record reviews. Am J Public Health. 1989;79(4):430–6. [PMC free article: PMC1349969] [PubMed: 2494893]
4.
Needleman J, Buerhaus PI, Mattke S, Stewart M, Zelevinsky K. Nurse Staffing and Patient Outcomes in Hospitals. Boston, MA: Health Resources Services Administration; 2001 February 28. Report No.: 230-99-0021.
5.
Lawthers A, McCarthy E, Davis R, Peterson L, Palmer R, Iezzoni L. Identification of in-hospital complications from claims data: is it valid? Medical Care. 2000;38(8):785–795. [PubMed: 10929991]
6.
Weingart SN, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB. et al. Use of administrative data to find substandard care: validation of the complications screening program. Med Care. 2000;38(8):796–806. [PubMed: 10929992]
7.
McCarthy EP, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB. et al. Does clinical evidence support ICD-9-CM diagnosis coding of complications? Med Care. 2000;38(8):868–876. [PubMed: 10929998]
8.
Best W, Khuri S, Phelan M, Hur K, Henderson W, Demakis J. et al. Identifying Patient Preoperative Risk Factors and Postoperative Adverse Events in Administrative Databases: Results from the Department of Veterans Affairs National Surgical Quality Improvement Program. J Am Coll Surg. 2002;194(3):257–266. [PubMed: 11893128]
9.
Kovner C, Gergen PJ. Nurse staffing levels and adverse events following surgery in U.S. hospitals. Image J Nurs Sch. 1998;30(4):315–21. [PubMed: 9866290]
10.
Geraci JM, Ashton CM, Kuykendall DH, Johnson ML, Wu L. International Classification of Diseases, 9th Revision, Clinical Modification codes in discharge abstracts are poor measures of complication occurrence in medical inpatients. Med Care. 1997;35(6):589–602. [PubMed: 9191704]
11.
Iezzoni L, Lawthers A, Petersen L, McCarthy E, Palmer R, Cahalane M, et al. Project to validate the Complications Screening Program: Health Care Financing Administration; 1998 March 31. Report No.: HCFA Contract 500-94-0055.
12.
Geraci JM, Ashton CM, Kuykendall DH, Johnson ML, Wu L. In-hospital complications among survivors of admission for congestive heart failure, chronic obstructive pulmonary disease, or diabetes mellitus. J Gen Intern Med. 1995;10(6):307–14. [PubMed: 7562121]
13.
Fisher ES, Whaley FS, Krushat WM, Malenka DJ, Fleming C, Baron JA. et al. The accuracy of Medicare's hospital claims data: progress has been made, but problems remain. Am J Public Health. 1992;82(2):243–8. [PMC free article: PMC1694279] [PubMed: 1739155]
14.
Hartz AJ, Kuhn EM. Comparing hospitals that perform coronary artery bypass surgery: the effect of outcome measures and data sources. Am J Public Health. 1994;84(10):1609–14. [PMC free article: PMC1615096] [PubMed: 7943479]
15.
Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB, Mukamal K. et al. Does the Complications Screening Program flag cases with process of care problems? Using explicit criteria to judge processes. Int J Qual Health Care. 1999;11(2):107–18. [PubMed: 10442841]
16.
Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K. Nurse-staffing levels and the quality of care in hospitals. N Engl J Med. 2002;346(22):1715–22. [PubMed: 12037152]
17.
Miller M, Elixhauser A, Zhan C, Meyer G. Patient Safety Indicators: Using Administrative Data to Identify Potential Patient Safety Concerns. Health Services Research. 2001;36(6 Part II):110–132. [PMC free article: PMC1383610] [PubMed: 16148964]
18.
Keeler E, Kahn K, Bentow S. Assessing quality of care for hospitalized Medicare patients with hip fracture using coded diagnoses from the Medicare Provider Analysis and Review File. Springfield, VA: NTIS; 1991.
19.
Massanari RM, Wilkerson K, Streed SA, Hierholzer WJ Jr. Reliability of reporting nosocomial infections in the discharge abstract and implications for receipt of revenues under prospective reimbursement. Am J Public Health. 1987;77(5):561–4. [PMC free article: PMC1647025] [PubMed: 3105338]
20.
Faciszewski T, Johnson L, Noren C, Smith MD. Administrative databases' complication coding in anterior spinal fusion procedures. What does it mean? Spine. 1995;20(16):1783–8. [PubMed: 7502134]
21.
Hawker GA, Coyte PC, Wright JG, Paul JE, Bombardier C. Accuracy of administrative data for assessing outcomes after knee replacement surgery. J Clin Epidemiol. 1997;50(3):265–73. [PubMed: 9120525]
22.
Romano P. Can Administrative Data be Used to Ascertain Clinically Significant Postoperative Complications. American Journal of Medical Quality Press. [PubMed: 12153067]
23.
Belio-Blasco C, Torres-Fernandez-Gil MA, Echeverria-Echarri JL, Gomez-Lopez LI. Evaluation of two retrospective active surveillance methods for the detection of nosocomial infection in surgical patients. Infect Control Hosp Epidemiol. 2000;21(1):24–7. [PubMed: 10656350]
24.
Lichtig LK, Knauf RA, Milholland DK. Some impacts of nursing on acute care hospital outcomes. J Nurs Adm. 1999;29(2):25–33. [PubMed: 10029799]
25.
Kuykendall DH, Ashton CM, Johnson ML, Geraci JM. Identifying complications and low provider adherence to normative practices using administrative data. Health Serv Res. 1995;30(4):531–54. [PMC free article: PMC1070074] [PubMed: 7591780]
26.
Silber JH, Rosenbaum PR, Koziol LF, Sutaria N, Marsh RR, Even-Shoshan O. Conditional Length of Stay. Health Serv Res. 1999;34(1 Pt 2):349–63. [PMC free article: PMC1089006] [PubMed: 10199680]
27.
Romano P. Unpublished report to the California Office of Statewide Health Planning and Development; 2001 March.

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (1.5M)

Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to PubMed

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...