NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Henriksen K, Battles JB, Keyes MA, et al., editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 1: Assessment). Rockville (MD): Agency for Healthcare Research and Quality; 2008 Aug.

Cover of Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 1: Assessment)

Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 1: Assessment).

Show details

Adaptation of AHRQ Patient Safety Indicators for Use in ICD-10 Administrative Data by an International Consortium

, MD, PhD, , MD, , MD, MPH, FACP, , MD, PhD, , MD, MPH, , PhD, , MD, , RN, MPH, , PhD, , MD, PhD, , MD, PhD, , CHIM, , MD, , MD, MPH, , BS, , MA, , MBBS, MPH, , MHS, CHIM, , MD, PhD, , MSc, and , MD, MPH; for the IMECCHI Investigators.

Author Information

, MD, PhD, , MD, , MD, MPH, FACP, , MD, PhD, , MD, MPH, , PhD, , MD, , RN, MPH, , PhD, , MD, PhD, , MD, PhD, , CHIM, , MD, , MD, MPH, , BS, , MA, , MBBS, MPH, , MHS, CHIM, , MD, PhD, , MSc, and , MD, MPH; for the IMECCHI Investigators.*

Department of Community Health Sciences and Centre for Health and Policy Studies, University of Calgary, Canada (HQ, LS); Niederrhein University of Applied Sciences, Krefeld, Germany (SD); Victorian Department of Human Services, Melbourne, Australia (VS., JS, LS); Canadian Institute for Health Information, Canada (EW, LM, LTL, GW); Institut Universitaire de Médecine Sociale et Préventive, Centre, Hospitalier Universitaire Vaudois and University of Lausanne, Switzerland (BB, PH, J-MJ); Hospices Civils de Lyon, Pole Information Medicale Evaluation recherché, Lyon, F-69003, France (CMC, J-MJ, EP, BT-P); Agency for Healthcare Research and Quality, United States (EK); Organisation for Economic Co-operation and Development (NK); Departments of Internal Medicine and Pediatrics, and Center for Healthcare Policy and Research, University of California Davis, United States (PSR); Departments of Community Health Sciences and Medicine, University of Calgary, Canada (WAG)
*Address correspondence to: Dr Hude Quan, Department of Community Health Sciences, University of Calgary, 3330 Hospital Dr. NW, Calgary, Alberta, Canada T2N 4N1; e-mail: ac.yraglacu@nauqh.


Objective: The Agency for Healthcare Research and Quality (AHRQ) developed Patient Safety Indicators (PSIs) for use with ICD-9-CM data. Many countries have adopted ICD-10 for coding hospital diagnoses. We conducted this study to develop an internationally harmonized ICD-10 coding algorithm for the AHRQ PSIs. Methods: The AHRQ PSI Version 2.1 has been translated into ICD-10-AM (Australian Modification), and PSI Version 3.0a has been independently translated into ICD-10-GM (German Modification). We converted these two country-specific coding algorithms into ICD-10-WHO (World Health Organization version) and combined them to form one master list. Members of an international expert panel—including physicians, professional medical coders, disease classification specialists, health services researchers, epidemiologists, and users of the PSI—independently evaluated this master list and rated each code as either “include,” “exclude,” or “uncertain,” following the AHRQ PSI definitions. After summarizing the independent rating results, we held a face-to-face meeting to discuss codes for which there was no unanimous consensus and newly proposed codes. A modified Delphi method was employed to generate a final ICD-10 WHO coding list. Results: Of 20 PSIs, 15 that were based mainly on diagnosis codes were selected for translation. At the meeting, panelists discussed 794 codes for which consensus had not been achieved and 2,541 additional codes that were proposed by individual panelists for consideration prior to the meeting. Three documents were generated: a PSI ICD-10-WHO version-coding list, a list of issues for consideration on certain AHRQ PSIs and ICD-9-CM codes, and a recommendation to WHO to improve specification of some disease classifications. Conclusion: An ICD-10-WHO PSI coding list has been developed and structured in a manner similar to the AHRQ manual. Although face validity of the list has been ensured through a rigorous expert panel assessment, its true validity and applicability should be assessed internationally.


Patient safety is a critical component of health care quality that has been widely studied.1, 2, 3, 4, 5 Assessments of patient safety are traditionally done through chart reviews, surveys, and voluntary hospital reporting of adverse events and medical errors. These data collection methods focus on specific types of events, often collect data that may not be generalizable to any population of interest, cover limited geographic areas, and may be too labor-intensive for widespread use. Therefore, researchers have become interested in using routinely collected administrative data for population-based studies of adverse events.

In response, the Agency for Healthcare Research and Quality (AHRQ) and the University of California-Stanford Evidence-based Practice Center developed patient safety indicators (PSIs) for use with hospital administrative data coded using the International Classification of Disease, 9th Revision, Clinical Modification (ICD-9-CM), which are readily available and relatively inexpensive to use.6 The AHRQ PSIs were developed through a literature search, review of ICD-9-CM manuals, consultation with physician panels, and empirical data analyses. Over 200 ICD-9-CM codes representing potential patient safety problems were identified, and 48 indicators were labeled as the most promising PSIs by the AHRQ research team. Of these, 20 hospital-level and seven area-level PSIs were recommended by one or more multispecialty panels as a set of “accepted” indicators.6 The first set of AHRQ PSIs was released in 2003 and has been updated periodically since then.

The 20 hospital-level indicators are used to identify potential inpatient complications that might represent events related to patient safety. The seven area-level indicators are designed to detect patient safety events on a regional level such as “Foreign body left during procedure.” Although the seven area-level indicators are closely related to the 20 hospital-level measures, the method of defining these seven area-level indicators is different. Area level indicators are designed to estimate the prevalence of each PSI in a jurisdiction or region. Therefore, the denominator includes the entire eligible population of a region, rather than just cases treated in a particular hospital. The numerator is based on both principal and secondary diagnoses, whereas for hospital-level indicators, the numerator is based only on secondary diagnoses.6 Inclusion of principal diagnoses in the area-level numerators captures patients who were admitted due to complications that occurred in previous hospitalizations or outpatient care episodes.

To facilitate utilization of the PSIs, AHRQ developed and distributes (at no cost) SAS® and Windows®-based software tools. These tools can be used to help hospitals identify potential adverse events that might need further study and also to enable users to assess the occurrence of in-hospital adverse events using routinely collected ICD-9-CM hospital discharge abstract data.

The PSI tools for ICD-9-CM cannot be applied to ICD-107 data because ICD-10 uses an alphanumeric system, and many codes are not directly convertible. The ICD-10 classification has been developed and is maintained by the World Health Organization (WHO). Updates to the ICD-10 are based on recommendations of the Update and Revision Committee that meets annually to discuss and ratify proposals. Major updates (e.g., new codes) are implemented every 3 years, while minor updates (e.g., corrections) are implemented annually. Implementation of ICD-10 began in 1994, but it has not yet been adopted for morbidity coding in the United States. The major differences between the ICD-10 and ICD-9-CM coding systems are:

  • The tabular list of diseases in ICD-10 has 22 chapters, compared to 19 chapters in ICD-9-CM. The chapter on diseases of the nervous system and sense organs in ICD-9-CM is expanded to three chapters in ICD-10, including diseases of the nervous system, diseases of the eye and adnexa, and diseases of the ear and mastoid process. ICD-10 specifies certain conditions in more detail than ICD-9-CM by adding anatomical sites and type of injury (open or closed).
  • The codes in ICD-10 are alphanumeric, whereas codes in ICD-9-CM are numeric. Each code in ICD-10 starts with a letter (i.e., A - Z), followed by two numeric digits, a decimal point, and a digit (e.g., acute bronchiolitis due to respiratory syncytial virus is J21.0). In contrast, codes in ICD-9-CM begin with three-digit numbers (i.e., 001–999) followed by a decimal and up to two digits (e.g., acute bronchiolitis due to respiratory syncytial virus is 466.11).

Canada, Australia, New Zealand, and many European and Asian countries have used the ICD-10 to code hospital discharge diagnoses since the system was introduced, but the development of quality indicators based on ICD-10-coded data has lagged behind. Starting in 2004, the Canadian Institute for Health Information began evaluating the AHRQ PSIs and selected a subset for public reporting on health system performance.8 Concurrently, Drösler and colleagues in Germany mapped the AHRQ PSIs from ICD-9-CM to ICD-10 and licensed this mapping to the German subsidiary of 3M, so that German hospitals could monitor their rates of potential safety-related events.9 Demand from potential users of the AHRQ PSIs in these and other countries prompted us to conduct this study to develop an internationally harmonized ICD-10 coding algorithm for the PSIs.

This study was spearheaded by the International Methodology Consortium for Coded Health Information (IMECCHI), an international group of experts dedicated to the development and validation of health research methodologies for coded health data.10 At its meeting in 2005, IMECCHI members identified the development of ICD-10 WHO coding algorithms for PSIs as a high priority initiative. Coincidentally, the Organization for Economic Cooperation and Development (OECD) launched its Health Care Quality Indicator (HCQI) Project in 2001, and identified five priority areas for initial development of indicators that could be used to explore quality differences across 23 participating countries: (1) cardiac care, (2) diabetes mellitus, (3) mental health, (4) patient safety, and (5) prevention/health promotion combined with primary care.

To identify and evaluate potential indicators of patient safety, the OECD convened a Patient Safety Panel, which then solicited indicators covering “five core domains of patient safety”: (1) hospital-acquired infections, (2) sentinel events, (3) operative and postoperative complications, (4) obstetrics, and (5) other care-related adverse events. Fifty-nine indicators from seven different sources were evaluated through a nominal group process; the Panel agreed on a final list of 21 indicators (including 12 AHRQ PSIs) that were deemed suitable, based on both importance and scientific soundness.11 In followup, the OECD convened health ministerial representatives from its member countries and experts to collaborate around patient safety data systems on June 29 and 30, 2006, in Dublin, Ireland.12 At the meeting, international harmonization of ICD-10 PSI definitions was identified as an urgent task, and the OECD Secretariat agreed to facilitate this undertaking.


Selection of PSIs for Translation

Defining PSI events (numerators) requires searching diagnosis and procedure code fields in hospital discharge abstract data, but defining denominators often requires the use of diagnosis-related groups (DRGs) to identify eligible hospitalizations. For example, the events coded as “foreign body left during procedure” (PSI 5) are found by searching for the ICD-9-CM codes 998.4 and 998.7 in secondary diagnosis fields. The denominator for this indicator includes all surgical and medical discharges, which are determined by specific surgical and medical DRGs.

To develop ICD-10-WHO (World Health Organization version) definitions for each PSI, we therefore needed to consider the following realities:

Reality 1. Various country-specific ICD-10 versions are available. Canada, Australia, Germany, and other countries have enhanced ICD-10 by adding more specific codes and released country-specific versions, such as ICD-10-CA (Canadian modification13), ICD-10-AM (Australian modification14) and ICD-10-GM (German modification15). The National Center for Health Statistics has developed ICD-10-CM for eventual use in the United States, but these codes “are not currently valid for any purpose or use.”16 The basic ICD-10-WHO structure, scope, and code definitions are not altered in these country-specific modifications, which mainly extend code character levels, from the third and fourth levels of ICD-10 to fourth-, fifth-, or sixth-character levels (e.g., “O10.0 pre-existing essential hypertension complicating pregnancy, childbirth, and the puerperium” in ICD-10-WHO subsumes “O10.001 pre-existing essential hypertension complicating pregnancy, childbirth, and the puerperium - delivered, with or without mention of antepartum condition” in ICD-10-CA). ICD-10 country-specific modifications also include a few additional third- and fourth-level codes, consistent with the existing classification structure. Some countries do not adopt all codes from the chapter “External causes of morbidity and mortality” (e.g., 22 codes in ICD-10-GM vs. 1,366 codes in ICD-10-WHO).

Reality 2. Each country uses its own distinct procedure coding system, limiting data comparability because ICD-10-WHO classifies medical conditions only, not procedures. For example, Switzerland uses a procedure coding system derived directly from ICD-9-CM.17 Canada developed its own procedure classification (the Canadian Classification of Health Interventions [CCI]13). Australia developed the Australian Classification of Health Interventions (ACHI),18 and Germany developed the German procedure classification (OPS).19

Reality 3. Various patient classification systems have been utilized across countries. Australia has developed ICD-10 DRGs based on ICD-10-AM; Germany has introduced the G-DRG system based on ICD-10-GM. The Canadian Institute for Health Information (CIHI) developed ICD-10-CA/CCI case-mix groups (CMG +) to predict resource utilization; this new methodology was implemented on April 1, 2007.20 Switzerland currently uses All Patient-Diagnosis Related Groups (AP-DRGs) but will adopt the German DRG system soon.

Considering these three realities, we focused on developing ICD-10-WHO coding algorithms for PSIs that mainly rely on diagnosis codes for defining PSI inclusion and exclusion criteria. We finally selected 15 PSIs (Table 1). However, country-specific procedure codes are required for adapting the coding algorithms to define some of these PSIs, such as “Postoperative physiologic and metabolic derangement.”

Table 1. Selected PSIs and denominator definitions.

Table 1

Selected PSIs and denominator definitions. Adopted from AHRQ Technical Specifications, Version 3.0a, May 1, 2006. See AHRQ manual for further exclusions.)

Process of Translation

The following three major steps were taken to develop ICD-10 coding algorithms.

Step 1: Searching ICD-10 Diagnosis Codes

The original ICD-9-CM codes embedded in the AHRQ PSIs were converted to ICD-10 codes using currently available Australian and German ICD-9 and ICD-10 crosstable mapping algorithms. Our Australian investigators used a conversion table between ICD-9-CM and ICD-10-AM, while our German investigator used a conversion table to ICD-10-GM and manually reviewed the results for each code.21, 22 All ICD-10 codes identified in both translations were combined additively to generate a diagnosis code master list. All country-specific codes that extended original ICD-10-WHO codes were truncated, and country-specific additional codes were excluded to maintain ICD-10-WHO formatting. All codes were described using their titles as listed in the ICD-10-WHO manual.

Step 2: Panel Review and Assessment

Twenty-one members of the PSI working group—including physicians, health services researchers—and coding professionals, independently (or as a geographic research team) reviewed the comprehensive code list. Each reviewer compared ICD-10-WHO codes with AHRQ’s ICD-9-CM codes, using the AHRQ PSI Technical Manual (Version 3.0a)6 to clarify AHRQ’s intent with respect to each code. The reviewers were asked to rate each diagnosis code as “Include,” “Exclude,” or “Uncertain.” For those who reviewed as a team, a final rating was made after internal discussions.

The German results have been tested on large national databases; substantial concordance was shown between German PSI rates calculated in ICD-10 and the U.S. rates published by AHRQ.23 However, we could not be certain that the coding list generated using Australian and German conversion tables would capture every relevant ICD-10-WHO code. Therefore, we asked reviewers to propose diagnosis codes that might have been omitted from the master list of codes.

Step 3: Face-to-Face Meeting

The purpose of the face-to-face meeting was to discuss codes for which consensus was not reached at the second step. Because of inconsistencies in coding practices across countries and code descriptions between ICD-9-CM and ICD-10, the following guidelines were established before voting:

Rule 1. Codes were selected for detailed discussion when consensus was not reached due to differences in country-specific coding standards. If coders in one country were advised to use code A for a specific condition, but in another country they were advised to use code B for the same condition, then both codes A and B were retained.

Rule 2: When additional codes relevant to the AHRQ indicator concept—or codes included in the AHRQ indicator concept that did not seem relevant—were identified, modifications or improvements of AHRQ ICD-9-CM definitions were recommended to AHRQ.

Rule 3: Additional ICD-10 codes with incomplete matching with ICD-9-CM were included if these codes matched the clinical concept of the PSI.

Rule 4: When there was no ICD-10-WHO code corresponding to an ICD-9-CM code, recommendations to improve or modify ICD-10-WHO were made to the WHO Update and Revision Committee.

We also developed guidelines for decisionmaking on additional codes proposed by the reviewers (Figure 1). Before rating for inclusion or exclusion, participants in the face-to-face meeting discussed and accepted these guidelines. Six reviewers who had the greatest experience with and understanding of the PSIs rated each code after discussions and consultations with other experts, including a member of the WHO Update and Revision Committee.

Figure 1. Assessment process of proposed additional diagnosis codes to PSIs.

Figure 1

Assessment process of proposed additional diagnosis codes to PSIs.

A modified Delphi method24, 25 was employed to achieve consensus. After discussion, reviewers rated each code as “Include,” “Exclude,” or “Uncertain.” When there were discrepancies among raters, reviewers explained their decisions, and further discussion among all members ensued. After a second round of discussion, the reviewers voted again. This 2-day process involved frequent reference to the ICD-9-CM and ICD-10 manuals, published crosswalk tables, and technical documents from AHRQ.


Three documents were generated from this work: (1) ICD-10-WHO coding algorithms for PSIs, (2) recommendations to AHRQ for adjustments to the ICD-9-CM PSIs, and (3) recommendations to the WHO Update and Revision Committee for refinement of ICD-10-WHO.

During the time between the Consortium meeting and this publication, all of these recommendations to AHRQ and WHO have been discussed, and some are being implemented.

ICD-10-WHO Coding Algorithms

A list of 2,569 ICD-10 codes invoked by the PSI algorithms was generated using the ICD-9-CM-to-ICD-10-AM and ICD-10-GM crosstables. After the first individual or team rating, we reached consensus for inclusion or exclusion of 1,775 codes, leaving 794 codes for which there was disagreement among the six raters. Another 2,541 codes (not identified from crosswalk tables) were then proposed by raters as potential codes for inclusion in the PSI algorithms. At the face-to-face meeting, panelists therefore discussed and rated 3,335 (i.e., 794 + 2,541) codes and generated the list of ICD-10-WHO codes required to define inclusion and exclusion criteria of 15 PSIs (available at

In the translation process, we found that some ICD-10-WHO codes did not match ICD-9-CM codes but met PSI clinical definitions. Table 2 shows these additional codes for the numerator inclusion of PSI 1 and the denominator exclusion of PSI 8. These codes were proposed to enhance the sensitivity of PSIs in the ICD-10-WHO coding algorithm.

Table 2. Additional ICD-10-WHO codes to enhance PSIs in ICD-10 data.

Table 2

Additional ICD-10-WHO codes to enhance PSIs in ICD-10 data.

Recommendations to AHRQ

The American representatives to our Consortium described some validity concerns about the indicator complications of anesthesia (PSI 1) because it was heavily (>99 percent) dependent on External Cause of Injury (E) codes. Only 16 of the 36 States that contributed to AHRQ’s State Inpatient Databases (SID) in 2002 required reporting of E-codes, plus 3 States (CA, SC, WA) did not require reporting of E870–E879 (including E876.3 for “endotracheal tube wrongly placed during anesthetic procedure,” which is in the PSI definition). Empirical analyses have confirmed that the apparent prevalence of this PSI highly depends on the number of diagnosis fields used in the analysis because E-codes are often appended after a long list of other diagnosis codes. Others have criticized this indicator because of its weak association with inpatient mortality and length-of-stay26 and its high “nonpreventable” rate based on chart review.27

We recommend that AHRQ consider moving PSI 1 to its “experimental” list. Specific codes for intubation difficulties during pregnancy, childbirth, and the puerperium were not included in the AHRQ PSI 1 (complications of anesthesia) definition. Consequently, maternal hospitalizations should be excluded from the population at risk.

For decubitus ulcer (PSI 3), a major exclusion group covers hemiplegias and other neurologic problems that limit mobility. AHRQ may have missed one rare ICD-9-CM diagnosis code for hemiplegia: 334.1(hereditary spastic paraplegia). This code (which matches G11.4 in ICD-10) should be added to AHRQ’s denominator exclusion list.

The title for PSI 7 (selected infections due to medical care) suggests a broader set of infections than are actually captured, despite use of the adjective “selected.” In fact, only catheter and infusion-related infections are included, so AHRQ should consider a more specific indicator name.

Regarding postoperative hip fracture (PSI 8), based on Canadian chart review experience, it was suggested that 996.44 (periprosthetic fracture around prosthetic joint, corresponding to ICD-10 code M96.6, fracture of bone following insertion of orthopedic implant, joint prosthesis, or bone plate) should be added with a denominator exclusion.

Regarding the existing denominator exclusions for this indicator, which are intended to exclude patients at high risk of falling in the hospital, even with appropriate care, the codes for stroke should include vertebrobasilar insufficiency (ICD-9-CM 435.0, 435.1, 435.3); and the codes for alteration of consciousness should include 070.0 (viral hepatitis A with hepatic coma), 070.2 (viral hepatitis B with hepatic coma), 070.4x (other specified viral hepatitis with hepatic coma), 070.6 (unspecified viral hepatitis with hepatic coma), 070.71 (unspecified viral hepatitis C with hepatic coma), and 780.09 (other drowsiness, semicoma, unconsciousness, somnolence, stupor).

The group of experts also raised the global conceptual concern that the denominator exclusion for PSI 8 was somewhat counterintuitive. The patients excluded are precisely those for whom a “safe” hospital could institute safeguards to prevent falls and hip fractures in the hospital. AHRQ should revisit the large block of denominator exclusions identifying patients at risk for falls, since many of these diagnoses could be removed from the denominator exclusion list to produce an indicator that would capture a larger percentage of all in-hospital hip fractures. For example, the ICD-9-CM codes for poisoning in PSI 8 are very broad; patients with poisonings due to agents that would not affect alertness and awareness could be excluded. These poisonings—such as 960 (poisoning by antibiotics), 961 (poisoning by anti-infectives), and 962 (poisoning by hormones and synthetic substitutes)—should not affect patients’ risk of iatrogenic hip fracture. ICD-9-CM codes for delirium and other psychoses also capture many diagnoses with no discernible effect on, or association with, alertness and presumably no effect on the risk of falling. For example, 296 codes (episodic mood disorders) with a subtitle indicating “in remission” (5th digit of 5 or 6) should not be on the exclusion list.

Conversely, ICD-9-CM codes for acute alcohol and drug intoxications were omitted from the PSI 8 exclusion criteria, even though such intoxications are likely to be associated with increased risk of falling. Of course, such intoxications are also unlikely to affect postoperative patients because any alcohol or drugs taken at home would wear off before surgery, and these agents are not administered in hospital. For example, the current ICD-9-CM exclusion list includes 291.4 (idiosyncratic alcohol intoxication – pathologic: alcohol intoxication, drunkenness), but it omits 303.00–303.02 (acute alcoholic intoxication – acute drunkenness in alcoholism) and 305.00–305.02 (alcohol abuse – drunkenness NOS). The current exclusion list includes 292.1x (drug-induced psychotic disorders) and 292.2 (pathologic drug intoxication), but it omits 305.30–305.02 (hallucinogen abuse – acute intoxication from hallucinogens)

In the numerator definition for postoperative sepsis (PSI 13), ICD-9-CM 785.59 (other shock without mention of trauma) should be removed because it no longer refers to an infectious disorder (i.e., effective October 2003, “septic shock” was assigned to a separate code 785.52). Denominator exclusion criteria of several PSIs include infection diagnosis codes. Our comparison of ICD-9-CM with ICD-10 revealed that two types of infection had been moved from nonbacterial sections in ICD-9-CM to bacterial sections in ICD-10-WHO, suggesting that they should now be added to the list of denominator exclusions (based on a presumption of pre-existing bacterial infection).

This change may reflect new thinking about the nature of the pathogens, including Bartonella henselae, which causes cat-scratch disease (078.3), and Leptospira species, which cause leptospirosis (100.xx). We also identified the following conditions that are currently on AHRQ’s denominator exclusion list (Appendix P) but do not actually represent bacterial infections (although they are grouped with other bacterial diagnoses): 376.00 (acute inflammation of orbit, unspecified; cellulitis is coded separately), 386.30 (labyrinthitis, unspecified, a viral infection), 386.31 (serous labyrinthitis, diffuse labyrinthitis), 386.32 (circumscribed labyrinthitis, focal labyrinthitis), 598.0x (urethral stricture due to infection), and 686.01 (pyoderma gangrenosum; commonly associated with Crohn’s disease and leukemias).

Recommendations to WHO Update and Revision Committee

ICD-9-CM was specifically designed to code clinical conditions in morbidity databases. Therefore, classification of certain conditions in ICD-10-WHO is not as clinically precise as in ICD-9-CM. Although there are specific codes for decubitus ulcer (PSI 3) in both ICD-9-CM and ICD-10-WHO, the ICD-10-WHO code L89 (decubitus ulcer) does not specify the site of the ulcer, while M70 codes describe unspecified “soft tissue disorders related to use, overuse, and pressure.” To prevent misleading users of the classification, an exclusion notice for decubitus ulcer should be added to the M70 group; additional subcodes under L89 should be considered to specify ulcer location and/or stage.

For defining iatrogenic pneumothrax (PSI 6), a new specific code for iatrogenic pneumothorax should be added. This is a concept that exists in ICD-9-CM, but there is no exact match in ICD-10-WHO. The suggested placement of this code is J95.6 (under J95, “postprocedural respiratory disorders not elsewhere classified”). Currently, some country versions of ICD-10 include a specific code for this diagnosis (e.g., J95.80 for “iatrogenic pneumothorax” in Germany and “postprocedural pneumothorax” in Canada). In Switzerland and France, coders are instructed to use a combination of S27.0 (traumatic pneumothorax) and Y60.x (unintentional cut, puncture, perforation, or hemorrhage during surgical and medical care) instead of J95.8.

The inclusion term for “endotoxic shock” under R57.8 (other shock) should be deleted. It might mislead, since the code A41.9 (septicemia, unspecified) contains an inclusion of “septic shock,” which includes endotoxin-mediated shock. Furthermore, the code A41.9 is listed as an exclusion under R57. There should be an instruction within the ICD-10 at code T81.4 to use an additional code to identify septicemia. A wide range of postprocedural infections are classified to this code, making its use as a patient safety indicator questionable.


We developed ICD-10-WHO diagnosis coding algorithms for defining 15 AHRQ PSIs through an explicit and diligent review process of the ICD-9-CM and ICD-10-WHO codes by an international expert panel. Before applying the coding algorithms, users should consider several challenges and issues, which are presented below.

Procedure Codes and Major Diagnostic Category

The ICD-10-WHO coding algorithms are insufficient by themselves because defining the population at risk often requires procedure codes and/or DRGs. Defining surgical discharges requires identification of major therapeutic operations.

Some procedures are required in the definition of PSI events—e.g., postoperative hemodialysis to identify end-stage renal failure. Furthermore, the screening of a few PSI events (numerators) relies on coding of procedures that reflect a complication or a reopening (e.g., perineal laceration repair, vena cava filters).

Major Diagnostic Categories (MDCs) are sometimes used to delineate the at-risk population (Table 1). The MDC is assigned to each case during the DRG grouping process, based primarily on the major or principal diagnosis. Some countries (e.g., Germany) provide public domain tables unfolding which ICD-10 codes trigger certain MDCs.28 Similar tables may be available in other countries that use DRG-based systems for hospital payment.

Diagnosis Codes

Several AHRQ PSIs cannot be translated directly from ICD-9-CM to ICD-10-WHO because of differences in the architecture of these coding systems. For example, the ICD-10-WHO coding algorithm for PSI 1 (complications of anesthesia) does not include T88.4 (failed or difficult intubation) in the numerator definition, since there is no corresponding ICD-9-CM code. Instead, we included the ICD-10 code Y65.3 because it matches the ICD-9-CM code E876.3 (endotracheal tube wrongly placed during anesthetic procedure). However, external cause codes, which are labeled E codes in ICD-9-CM and Y codes in ICD-10-WHO, are not mandatory in several countries and are underused in other countries. The term “failed” in the description of T88.4 matches with “wrongly placed” in E876.3, but the term “difficult” in T88.4 has neither a corresponding description in ICD-9-CM nor an association with medical error. When users analyze ICD-10 hospital discharge data, they should make a decision on inclusion or exclusion of T88.4, based on whether the data contain codes for external causes of morbidity and mortality.

For the numerator definition of iatrogenic pneumothorax (PSI 6), no ICD-10-WHO code matches the ICD-9-CM code 512.1 (iatrogenic pneumothorax). Two proximate ICD-10-WHO codes of J93.8 (other pneumothorax) and J95.8 (other postprocedural respiratory disorders) do not explicitly refer to iatrogenic pneumothorax. Some country-specific versions of ICD-10 use their own codes for pneumothorax related to medical care, as described above. In Switzerland and France, coders are told to use a combination of S27.0 (traumatic pneumothorax) and Y60 (unintentional cut, puncture, perforation, or hemorrhage during surgical and medical care) to indicate that the event is related to medical care. In countries using S27.0 to identify iatrogenic pneumothorax, this code should be deleted from the denominator exclusion list (except if used as the principal diagnosis).

Several ICD-10 codes are not sufficiently precise to identify some exclusion conditions, such as gastrointestinal hemorrhage. For example, I98.2 (esophageal varices in diseases classified elsewhere) does not indicate whether bleeding was present, which is integral to the definition of gastrointestinal hemorrhage. Some countries use a supplementary subdivision to improve the specificity of this code. The ICD-10 codes for diverticular disease (K57.x) have similar limitations.

Timing of Occurrence

The AHRQ PSIs are used to detect complications or adverse events resulting from medical care. When applying this definition to U.S. hospital discharge abstract data, the hospital-level PSI numerators are based only on “secondary” diagnosis codes,6 because the principal diagnosis is defined in the Uniform Hospital Discharge Data Set (UHDDS)29 as “that condition established after study to be chiefly responsible for occasioning the admission of the patient to the hospital for care.” However, some countries do not follow the UHDDS definition. For example, Canada classifies diagnoses using diagnosis-type definitions: type (M) for the most responsible, type (1) for pre-admit comorbidity; type (2) for post-admit comorbidity; and type (3) for secondary diagnoses.13 The most responsible diagnosis is defined as “the one diagnosis or condition that can be described as being most responsible for the patient’s stay in hospital. If there is more than one such condition, the one held most responsible for the greatest portion of the length of stay or greatest use of resources (e.g., operating room time, investigative technology) is selected.” A secondary diagnosis refers in Canada to a condition for which a patient may or may not have received treatment that does not satisfy the requirements for determining comorbidity. France also defines the principal diagnosis as the diagnosis that contributed the most to the care provided to the patient during hospitalization.

Employing a diagnosis-type definition to identify conditions that develop after admission improves the validity of the PSIs in any data system. CIHI has adopted PSI 8 (“postoperative hip fracture,” renamed as “In-Hospital Hip Fracture Rate”) and publicly reported it in its annual Health Indicators Report since 2004. In CIHI’s definition, the indicator only includes events that are coded as postadmission (diagnosis type 2).30 Naessens, et al.,31 analyzed 2005 hospital discharge data from Mayo Clinic Rochester to determine the frequency of PSIs after distinguishing conditions diagnosed before and after admission. They reported that 63.1 percent of the cases identified by 20 PSI numerators occurred during hospitalization, but this percentage was only 22 percent for PSI 8. Similar results were obtained from statewide studies in New York and California, the two States that pioneered mandatory diagnosis-type reporting.32 For this reason, the 2007 version of the AHRQ manual encourages users who have data with “present-at-admission” indicators to exclude secondary diagnoses that preceded the admission of interest.6

Data Quality

While acknowledging the potential usefulness of the PSIs in clinical quality improvement, it is necessary to call attention to the issue of data quality. Data quality is a common concern in all indicator-related analysis and reporting. Given the nature of the PSIs—which rely on ICD-coded diagnoses and/or procedures for both outcome ascertainment and risk-adjustment—they are even more vulnerable to this challenge than indicators of mortality.33 Sedman, et al.,27 applied AHRQ PSI ICD-9-CM coding algorithms to children’s hospital administrative data in the United States and reviewed the medical records of a sample of flagged cases. Of the 11 PSIs studied, only two (failure to rescue and death in low-mortality DRGs) did not represent preventable errors in the majority of pediatric cases. Romano, et al.,34 assessed the validity of ICD-9-CM administrative data in recording obstetric complications with 1,611 clinical delivery records (postpartum and antepartum). They reported that third- and fourth-degree perineal lacerations were recorded accurately in the administrative data with sensitivity exceeding 90 percent and positive predictive value exceeding 85 percent.

Special caution and further validation efforts are needed if these indicators are going to be used for reporting and comparison across national boundaries, given that nations vary widely in the number of allowable diagnosis fields, the use of DRG-based payment systems for resource allocation, and related coding practices. Even within the United States, States with greater E-code usage and more filled diagnosis fields tend to have significantly higher PSI rates than other States.33 Although validation studies are mandatory for each PSI, they are sensitive to time and location of the available routinely collected data in various jurisdictions.


First, validation and comparison of the performance of the AHRQ ICD-9-CM and ICD-10-WHO coding algorithms have not yet been done. In the United States, a validation pilot project is now underway to assess the validity of 10 PSIs through detailed chart review at 48 collaborating hospitals. A separate validation study is also underway in the Veterans Affairs hospital system, following up on a smaller scale linkage study involving data from the National Surgical Quality Improvement Program. The University HealthSystem Consortium has partnered with AHRQ to validate failure-to-rescue,35 postoperative pulmonary embolism or deep vein thrombosis, and postoperative respiratory failure, while the National Association of Children’s Hospitals and Related Institutions has partnered with AHRQ to validate the pediatric versions of the PSIs.27 A research team is extracting information from inpatient charts in Calgary (Canada) for validating PSIs recorded in the ICD-10-CA administrative data. In addition, a collaborative project involving sites in Lyon (France), Lausanne (Switzerland), and Calgary will focus on validating postoperative pulmonary embolism or deep vein thrombosis in ICD-10 administrative data. Other validation studies at multiple international sites are being planned.

Secondly, we translated diagnosis codes for only 15 of the 20 PSIs. To strengthen international comparisons using the AHRQ PSIs, a uniform, detailed procedure classification system should be developed. Third, we employed AHRQ manuals as the “gold standard” to clarify clinical intent when developing the ICD-10-WHO coding algorithms, but the “true validity” of the PSIs is generally unknown. Finally, PSIs are focused on quality of care but by no means are intended to be comprehensive assessments of quality of care.


A set of algorithms for the AHRQ PSIs using ICD-10-WHO diagnoses has been developed and structured in close relation to the AHRQ PSI documentation. This work should support international applications in ICD-9-CM and ICD-10 data. Although the face/content validity of the list has been ensured through a rigorous expert panel assessment, its “true” validity needs to be assessed internationally. Hospital abstract administrative data have been analyzed for measuring PSIs in Australia, Belgium, Canada, France, Germany, Great Britain, Italy, Spain, Sweden, and Switzerland. It is anticipated that more countries will employ the PSIs for quality of care assessment. We welcome feedback from users regarding their experiences in applying these coding algorithms in order to make them more robust and useful internationally.


The face-to-face meeting on April 28–29, 2007, in Toronto, for translating ICD-9-CM PSIs to ICD-10 had in-kind support from the Canadian Institute for Health Information, AHRQ, OECD, institutes, and government agencies of team members. IMECCHI investigators include Bernard Burnand, University of Lausanne, Switzerland; Cyrille Colin, University of Lyon, France; Chantal Couris, University of Lyon, France; Carolyn De Coster, University of Manitoba, Canada; Saskia Drösler, Niederrhein University of Applied Sciences, Germany; Alan Finlayson, the National Health Service in Scotland, United Kingdom; Kiyohide Fushimi, Tokyo Medical and Dental University Graduate School, Japan; Min Gao, British Columbia Provincial Public Health Services Authority, Canada; William Ghali, University of Calgary, Canada; Patricia Halfon, University of Lausanne, Switzerland; Brenda Hemmelgarn, University of Calgary, Canada; Karin Humphries, University of British Columbia, Canada; Jean-Marie Januel, University of Lyon, France and University of Lausanne, Switzerland; Helen Johansen, Statistics Canada; Lisa Lix, Universality of Manitoba, Canada; Jean-Christophe Luthi, University of Lausanne, Switzerland; Jin Ma, Jiaotong University, China; Hude Quan, University of Calgary, Canada; Patrick Romano, University of California, Davis, United States; Leslie Roos, University of Manitoba, Canada; Fiona Shrive, University of Calgary, Canada; Vijaya Sundararajan, Victorian Department of Human Services, Australia; Sandrine Touzet, University of Lyon, France; Jack Tu, University of Toronto, Canada; and Greg Webster, Canadian Institute of Health Information, Canada.


Thomas EJ, Brennan TA. Incidence and types of preventable adverse events in elderly patients: Population-based review of medical records. Br Med J. 2000;320:741–744. [PMC free article: PMC27315] [PubMed: 10720355]
Forster AJ, Clark HD, Menard A, et al. Adverse events among medical patients after discharge from hospital. CMAJ. 2004;170:345–349. [PMC free article: PMC331384] [PubMed: 14757670]
Baker GR, Norton PG, Flintoft V, et al. The Canadian Adverse Events Study: The incidence of adverse events among hospital patients in Canada. CMAJ. 2004;170:1678–1686. [PMC free article: PMC408508] [PubMed: 15159366]
Romano PS, Geppert JJ, Davies S, et al. A national profile of patient safety in U.S. hospitals. Health Aff (Millwood) 2003;22:154–166. [PubMed: 12674418]
Baker GR, Norton P, Flintoft V. Knowledge translation and patient safety: The Canadian adverse events study. Healthcare Policy. 2006;1:45–50. [PMC free article: PMC2585339] [PubMed: 19305669]
Patient Safety Indicators: Technical specifications. Rockville, MD: Agency for Healthcare Research and Quality; 2006. [Accessed March 10, 2008]. Available at: www​.qualityindicators​​/psi_technical_specs_v30a.pdf.
World Health Organization. International statistical classification of disease and related health problems, 10th Revision (ICD-10) Geneva: World Health Organization; 1992. [Accessed March 10, 2008]. Available at: www​​/apps/icd/icd10online/
Patient safety in Canada: An update. Ottawa, ON: Canadian Institute for Health Information; 2007. [Accessed March 10, 2008]. Available at: secure​​/en/downloads/Patient​_Safety_AIB_EN_070814.pdf.
Drösler SE, Cools A, Köpfer T, et al. [Are quality indicators derived from routine data suitable for evaluating hospital performance? First results using the AHRQ patient safety indicators in Germany] Z Arztl Fortbild Qualitatssich. 2007;101:35–42. [PubMed: 17458363]
De Coster C, Quan H, Finlayson A, et al. Identifying priorities in methodological research using ICD-9-CM and ICD-10 administrative data: Report from an international consortium. BMC Health Serv Res. 2006;6:77. [PMC free article: PMC1513221] [PubMed: 16776836]
Millar J, Mattke S. Selecting indicators for patient safety at the health systems level in OECD countries. Int J Qual Health Care. 2006;18( Suppl 1):14–20. [PubMed: 16954511]
Organisation for Economic Co-operation and Development. First OECD Health care quality indicators seminar on improving patient safety data systems: Agenda and presentations; 29–30 June 2006; 2007. [Accessed March 10, 2008]. Available at: www​​/5/0,2340,en_2649_201185​_37159877_1_1_1_1,00.html.
Canadian coding standards for ICD-10-CA and CCI for 2007. Ottawa, ON: Canadian Institute for Health Information; 2007. [Accessed March 10, 2008]. Available at: secure​​/products/Canadian​_Coding_Standards_2007_e.pdf.
ICD-10-AM. Lidcombe, NSW Australia: National Centre for Classification in Health. 2007. [Accessed March 10, 2008]. Available at: nis-web​​.au/ncch_new/2.6.aspx.
ICD-10-GM. Cologne, Germany: German Institute of Medical Documentation and Information; 2008. [Accessed March 10, 2008]. Available at: www​​/klassi/diagnosen/icd10/index.htm.
About the International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) Hyattsville, MD: National Center for Health Statistics; [Accessed March 10, 2008]. Available at: www​​/otheract/icd9/abticd10.htm.
Classification Suisse des Interventions Chirurgicales (CHOP) Index systématique (ICD-9-CM, Vol. 3), Version 9.0. Neuchâtel, Switzerland: Office Féderal de la Statistique; 2008. [Accessed March 10, 2008]. Available at: www​​/portal/fr/index/news/publikationen​.Document.102347.pdf.
Bruce ACT.Australia: Australia Institute of Health and Welfare. Australian Classification of Health Interventions (ACHI); 2007. [Accessed March 10, 2008]. Available at: meteor​​.phtml/itemId/335419.
German Procedure Classification. Cologne. Germany: German Institute of Medical Documentation and Information; 2007. [March 10, 2008]. Available at: www​​/klassi/prozeduren/ops301/index.htm.
Gilardi F, Füglister K, Luyet S. Learning from others: The diffusion of hospital financing reforms in OECD countries. Presented at the workshop “The diffusion of policies and institutions”; Nicosia, Cyprus. 25–30 April 2006; [Accessed March 10, 2008]. Available at: www​​/iepi/users/anicolay​/public/gilardi_etal_ECPR.pdf.
The International Statistical Classification of Diseases and Related Health Problems. 10th Revision, Australian Modification (ICD-10-AM), Vol 5 ICD-10-AM Australian Coding Standards. 2nd Edition. Sydney, NSW, Australia: National Centre for Classification in Health; 2000.
Crosswalks for ICD-10. Cologne, Germany: German Institute of Medical Documentation and Information; 2007. [Accessed March 10, 2008]. Available at: www​​/klassi/diagnosen/ueberl/index.htm.
Drösler SE, Cools A, Köpfer T, et al. Are quality indicators derived from routine data suitable for evaluating hospital performance? First results using the AHRQ Patient Safety Indicators in Germany. ZaeFQ. 2007;101:35–42. [PubMed: 17458363]
Goodman CM. The Delphi technique: A critique. J Adv Nurs. 1987;12:729–734. [PubMed: 3320139]
Brook RH, Chassin MR, Fink A, et al. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2:53–63. [PubMed: 10300718]
Zhan C, Miller MR. Excess length of stay, charges, and mortality attributable to medical injuries during hospitalization. JAMA. 2003;290:1868–1874. [PubMed: 14532315]
Sedman A, Harris JM, Schulz K, et al. Relevance of the Agency for Healthcare Research and Quality Patient Safety Indicators for children’s hospitals. Pediatrics. 2005;115:135–145. [PubMed: 15579669]
Institut für das Entgeltsystem im Krankenhaus gGmbH (InEK) German Diagnosis Related Groups, Ver. 2006. Definitionshandbuch. [Accessed March 10, 2008]. pp. 1–5. Available at: www​
Official ICD-9-CM guidelines for coding and reporting. [Accessed March 10, 2008]. Available at: www​
Health system performance. Ottawa, ON: Canadian Institute for Health Information; 2006. [Accessed March 10, 2008]. Available at: www​​.jsp?cw_page=prtwg_2006_e.
Naessens JM, Campbell CR, Berg B, et al. Impact of diagnosis-timing indicators on measures of safety, comorbidity, and case mix groupings from administrative data sources. Med Care. 2007;45:781–788. [PubMed: 17667313]
Houchens RL, Elixhauser A, Romano PS. How often are potential patient safety events present on admission? Jt Comm J Qual Saf. 2008;34:154–163. [PubMed: 18419045]
Raetzman S, Stranges E, Coffey RM. Patient safety in hospitals in 2004: Toward understanding variation across States. Rockville, MD: Agency for Healthcare Research and Quality; 2007.
Romano PS, Yasmeen S, Schembri ME, et al. Coding of perineal lacerations and other complications of obstetric care in hospital discharge data. Obstet Gynecol. 2005;106:717–725. [PubMed: 16199627]
Horwitz LI, Cuny JF, Cerese J, et al. Failure to rescue: validation of an algorithm using administrative data. Med Care. 2007;45:283–287. [PubMed: 17496710]
PubReader format: click here to try


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this page (570K)

Other titles in this collection

Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to pubmed

Related citations in PubMed

See reviews...See all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...