Chapter  58:  The Autopsy as an Outcome and Performance Measure

Overview

An extensive literature documents a high prevalence of errors in clinical diagnosis discovered at autopsy. Multiple studies have suggested no significant decrease in these errors over time. Despite these findings, autopsies have dramatically decreased in frequency in the United States and many other countries. In 1994, the last year for which national U.S. data exist, the autopsy rate for all non-forensic deaths fell below 6%. The marked decline in autopsy rates from previous rates of 40–50% undoubtedly reflects various factors, including reimbursement issues, the attitudes of clinicians regarding the utility of autopsies in the setting of other diagnostic advances, and general unfamiliarity with the autopsy and techniques for requesting it, especially among physicians-in-training.

The autopsy is valuable for its role in undergraduate and graduate medical education, the identification and characterization of new diseases, and contributions to the understanding of disease pathogenesis. Although extensive, these benefits are difficult to quantify. This systematic review studied the more easily quantifiable benefits of the autopsy as a tool in performance measurement and improvement. Such benefits largely relate to the role of the autopsy in detecting errors in clinical diagnosis and unsuspected complications of treatment. It is hoped that characterizing the extent to which the autopsy provides data relevant to clinical performance measurement and improvement will help inform strategies for preserving the benefits of routinely obtained autopsies and for considering its wider use as an instrument for quality improvement.

This report does not attempt to address the roles of the autopsy in medical education; furthering medical research; quality control within pathology; verification, second-opinion consultations, and legal documentation of findings; the bereavement process for surviving family members; or other benefits that are described in many of the sources listed in the bibliography (Appendix F). In addition to being difficult to quantify, these benefits apply primarily to teaching hospitals. To address the role of the autopsy as an outcome measure and tool for quality improvement, the report focuses on benefits likely to apply to all hospitals, such as the detection of important diagnostic errors and related quality problems.

Reporting the Evidence

This report synthesizes the autopsy literature as it relates to the following four key questions:

1. To what extent does the autopsy reveal important diagnoses that were clinically unsuspected prior to death?

2. To what extent does the autopsy provide a useful performance measure or audit of clinical diagnosis in general?

3. What impact do autopsy findings have on clinical performance improvement?

4. To what extent are vital statistics compromised by low autopsy rates?

To address the above questions adequately, we also sought evidence pertaining to the properties of the autopsy as a diagnostic test. Specifically, we looked for any information describing autopsy quality, accuracy, and precision or reproducibility.

It is important to note that, though the phrase “diagnostic error” appears throughout this report, the discrepancies between clinical and autopsy diagnoses to which we refer do not necessarily represent errors in the sense of mistakes, “slips,” or other such terms. Some of these discrepancies do undoubtedly result from failures to consider an appropriately broad differential diagnosis, misinterpretation of test results, and other quality problems, so that resulting discrepant diagnoses detected at autopsy do warrant the label “diagnostic errors.” However, other such discrepancies clearly represent acceptable limits to clinical diagnosis, based on the performance of current technologies or the occurrence of atypical clinical presentations. (In fact, one of the areas of future research identified by this report involves characterizing the relative distribution of these two types of clinical-autopsy diagnostic discrepancies.) Despite these considerations, we use the term “diagnostic errors” because it appears so commonly in the autopsy literature.

Findings

To address the first key question pertaining to the extent to which autopsies reveal clinically unsuspected important diagnoses, we reviewed studies assessing the performance of the autopsy as a diagnostic test. Given the generally accepted role of the autopsy as the ultimate diagnostic standard for many aspects of clinical care, the test characteristics of the autopsy have received surprisingly little attention.

• The quality of the autopsy has received little systematic study, with the only evidence pertaining to perinatal autopsies, where two studies show that deficiencies relative to reporting standards (i.e., a proxy measure for potentially inadequate quality) appear to be common.

• The potential for error or disagreement in autopsy interpretations has been assessed in only one small study. In relation to the determination of principal diagnoses relating to the cause of death in technically adequate autopsy, diagnostic uncertainty persists in 1–5% of cases, although rates of up to 40% have been reported, depending on the type of autopsy cases, e.g., perinatal. Importantly, errors in classification of autopsy diagnoses involving even a few percent of cases substantially distort estimates of the performance of clinical diagnosis when autopsy is used as the gold standard.

• The reproducibility of judgments about errors in clinical diagnosis as indicated by autopsy findings has only been mentioned in passing in the autopsy literature. Studies from the health care quality and medical error literature suggest that reproducibility of similar types of judgments is likely fair to moderate at best.

There is insufficient literature to address: a) the quality of the autopsy, b) the technical adequacy in interpreting autopsy findings, and c) the reliability of judgments made regarding autopsy detected discrepancies. There is also no literature that addresses the quality of training in autopsy pathology or the ability of physicians to utilize autopsy findings.

In terms of the four main study questions:

1. To what extent does the autopsy reveal important diagnoses that were clinically unsuspected prior to death?

   • The chance that autopsy will reveal a misdiagnosis that may have affected outcome (i.e., a Class I error) was 10.2% (95% CI: 6.7–15.3%) using data from all studies and the base values of time (1980), autopsy rate (overall mean rate of 44.3%), country (U.S.) and case mix (general autopsies). Restricting the analysis to data from U.S. institutions only yielded a slightly higher point estimate but almost entirely overlapping confidence interval, 11.2% (95% CI: 6.9–17.5%). Adjusting for changes in autopsy rates, and the effects of case mix and the country, the probability of a Class I error showed a relative decrease of 26.2% per decade (p=0.10).

   • The base probability of the autopsy detecting a major error in a given case was 25.6% (95% CI: 20.8–31.2%) when data from all institutions were included. Using data from U.S. institutions only, the probability of the autopsy detecting a major error in a given case was slightly lower at 24.0%, but with an almost entirely overlapping 95% CI of 17.6–31.5%. Major error rates also showed a similar decrease over time, but, in contrast to the results for Class I errors, this relationship was statistically significant. Relative to the base rate in 1980, the prevalence of major errors exhibited a relative decrease of 28.0% (95% CI: 9.8–42.6%) per decade.

   • The regression analysis supported the expected inverse correlation between error rate and autopsy rate (i.e., that lower autopsy rates produce higher error rates due to selection of diagnostically challenging cases), but this effect is relatively modest. Specifically, every 10% increase in the autopsy rate is associated with a relative decrease in Class I errors of 7.8% (p=0.18). For major errors, this relationship was more substantial and statistically significant, with every 10% increase in autopsies associated with a relative decrease in major errors of 12% (p=0.0003).

   • Using the regression model to compute rates of autopsy-detected diagnostic errors over a range of autopsy rates and as a function of time, contemporary (year 2000) autopsies detect Class I errors in 3.8–7.9% of cases and major errors in 8.0–22.8%, of cases. These ranges reflect variations in autopsy rates from 5–100%.

   • The weak relationship between autopsy rates and error rates in the general analysis was supplemented by review of studies specifically addressing the issue of clinical selection of diagnostically challenging or uncertain cases. These studies indicated that clinicians cannot reliably predict which autopsies will be of high diagnostic yield, reinforcing the conclusion that the relatively unchanged diagnostic error rates do not simply reflect competing effects of medical progress (leading to fewer errors) and fewer autopsies (leading to selection for cases likely to have errors).

   • Because of the recent interest in medical error and patient safety, we specifically looked for studies that reported the proportion of autopsies that detected clinically unsuspected complications of care. These data were usually mentioned in passing in these studies, with no study specifically focusing on this issue. Thus, the extent to which these complications contributed to death (and even the extent to which they were truly unsuspected) was often unclear. For this reason, and because of the heterogeneity of the case mix in the relatively small sample of studies reporting the relevant data, we did not pool estimates for rates of autopsy-detection of unsuspected complications of care. Nonetheless, the 11 studies that did provide data on this point indicated that approximately 1–5% of autopsies disclose unsuspected complications of care.

2. To what extent does the autopsy provide a useful performance measure or audit of clinical diagnosis in general?

   • Autopsy studies commonly report diagnostic “error rates,” but these error rates involve autopsied cases only. It is commonly assumed that the true denominator of interest is all deaths; hence the interest in increased autopsy rates. However, the denominator of interest for clinical performance measurement is, in fact, all patients receiving care during the autopsy observation period. Only one autopsy study provides any data on clinical diagnoses for patients discharged alive from the hospital during the same observation period as for the autopsy series. Because of the importance of this question, we searched extensively for studies outside the autopsy literature per se for potentially relevant studies.

   • Specifically, we looked for studies reporting clinical diagnoses and other follow-up data on cohorts of patients (e.g., all patients admitted to a given hospital during a defined observation period), not just the diagnoses obtained for patients who died and went to autopsy. Supplementing autopsy findings with the results of ante mortem diagnostic testing and/or clinical follow-up for patients who did not die permits determination of the numerator and denominator required to assess the sensitivity of clinical diagnosis. Despite an extensive search, we found appropriate studies for only five target conditions: pulmonary embolism (PE), acute myocardial infarction (MI), acute appendicitis, aortic dissection, and active tuberculosis.

   • Among these five conditions, the performance of clinical diagnosis exhibited substantial variation, with excellent performance only for acute MI and to a lesser extent PE. Even for these two conditions, the high sensitivities obtained likely overstate clinical performance, as focusing on the dichotomous outcome of correct or incorrect identification of one target condition (PE or MI) obscures the extent to which other important conditions are missed once these target diagnoses are ruled out. A patient who is correctly identified as not having an MI counts as a success, regardless of whether or not the underlying cause of the patient's presenting complaint is ever diagnosed.

3. What impact do autopsy findings have on clinical performance improvement?

   • No intervention study has directly addressed the impact of autopsy findings on clinical practice or performance improvement. Consequently, the study objectives in this regard were not met, including not being able to perform a cost effectiveness analysis, as the effectiveness of the autopsy in reducing errors and other quality problems remains unknown. This does not invalidate the potential role of the autopsy in relation to clinical practice or performance improvement, but does reveal an important gap in the literature.

4. To what extent are vital statistics compromised by low autopsy rates?

   • Major error rates detected by autopsy indicate substantial inaccuracies in death certificates and hospital discharge data, both of which play important roles in epidemiologic research and health care policy decisions. Previous studies have suggested that these errors roughly cancel each other out (i.e., for a given condition, false positive and false negative diagnoses are roughly equal). However, this finding has not been consistent across studies. Even when present, this balancing effect applies only when considering the most general of diagnostic categories (i.e., cardiovascular, neoplastic, infectious, metabolic, and so on). Thus, the current evidence is adequate to suggest that the epidemiologic data for important diseases such as myocardial infarction, breast cancer, pneumonia, stroke, and so on, all contain substantial inaccuracies—in the 20–30% range reported for major errors.

Conclusions

The findings of this review have different implications depending on the level of analysis—individual clinicians, hospitals, or the health care system as a whole. From the point of view of the individual clinician, the chance that autopsy will reveal important unsuspected diagnoses in a given case remains significant. Moreover, clinicians do not seem able to predict reliably cases in which such findings are more likely to occur. Thus, clinicians have compelling reasons to request autopsies far more often than currently occurs.

At the institutional level, the role of the autopsy is less clear. The prevalence of missed diagnoses among autopsied patients (or even all deaths) provides a numerator, but not a denominator with which to assess the rate at which patients with a given condition remain undiagnosed until death. Using autopsy results to track hospital quality requires not only explicitly defined error rates, but also data on the number of patients discharged alive with diagnoses that appear among the list of conditions first detected at autopsy. Clearly, though, the unexpected findings at autopsy in specific cases are of interest to institutions as a whole and not just the individual treating clinicians. However, no study has ever examined the impact of performing autopsies (and communicating autopsy findings back to clinicians) on institutional performance improvement. This represents a major area for future research, but should not detract from the finding that many institutions perform too few autopsies to allow any meaningful assessment of local diagnostic performance and other quality problems, no matter how communication and feedback to clinicians occurs.

At the level of the entire health care system, existing literature provides two compelling reasons to pursue autopsies. First, results for the five conditions examined in this report suggest that clinical diagnosis in routine practice may not perform as well as is generally believed by clinicians or as suggested by the literature assessing specific aspects of clinical diagnosis (e.g., new tests) in research settings. Better characterizing the performance of clinical diagnosis for common conditions would clearly benefit the entire health system and identify important targets for quality improvement that could be pursued in a concerted manner.

The second benefit to the health care system as a whole relates to vital statistics and other epidemiologic data. Vital statistics impact important decisions about allocation of funding for research and other aspects of health care policy. The existing literature demonstrates that clinical diagnoses, whether obtained from death certificates or hospital discharge data, contain major inaccuracies compared with diagnoses generated from postmortem findings. The use of autopsy data to correct inaccuracies in epidemiologic data would likely confer multiple benefits on the health care system as a whole.

Future Research

1. Various aspects of the performance of the autopsy as a diagnostic test (e.g., the reproducibility of findings between pathologists) remain undefined and represent areas for further research. More specifically relevant to the present review is the inter-rater reliability for error classifications in specific cases, i.e., establishing the extent to which pathologists, clinicians or other peer reviewers agree that a particular case does or does not involve a clinically important diagnostic error.

2. The causes of important diagnostic discrepancies remain uncharacterized. This represents a very important area of investigation. Discrepancies between efficacy and effectiveness (i.e., differences between the performance of a diagnostic or therapeutic procedure in routine practice compared to the result in the research literature) have diverse causes. Broadly speaking, though, discrepancies are caused by a) quality problems related to underuse, overuse and misuse of diagnostic or therapeutic procedures, and b) patient factors, including atypical presentations and complex interactions between comorbid conditions and patient demographic factors. Neither of these categories are captured in the “efficacy literature” (i.e., clinical trials), as the nature of research settings make underuse, overuse or misuse unlikely, and stringent patient selection reduces the complexities of comorbid conditions and multiple competing diagnostic considerations.

   Autopsy data provide a window into discrepancies between efficacy and effectiveness both for therapeutics (by detecting clinically unsuspected complications of care) and diagnostics (by detecting the diagnostic discrepancies discussed in this report). In both cases, but perhaps especially the latter, the autopsy can play a pivotal role in spearheading investigations into the causes of these discrepancies. Where discrepancies prove to present quality problems, the institution benefits and, where they reflect differences between the types of patients receiving care in routine practice and clinical trials, the whole health system may benefit from awareness of these findings.

3. Future research should establish strategies for optimizing the utility of the autopsy at the institutional level. No study has ever directly assessed the impact of detecting errors in clinical diagnosis on subsequent clinical performance. Thus, future research should establish optimal methods of involving clinicians in the autopsy process (or communicating its results to them) and effective ways of stimulating change based on autopsy findings. Until such research is performed it is not clear to what extent autopsy rates need to be increased as opposed to achieving improvements in communication and utilization of information generated from autopsies performed at current rates.

4. Future research should establish the optimal means of using autopsy data to provide more accurate vital statistics and other important epidemiologic data. The first step might be to validate the findings suggested in this review, namely that current vital statistics contain substantial inaccuracies. Such an undertaking might involve funding a small number of demographically diverse institutions to achieve high institutional autopsy rates, with prospectively determined protocols for autopsy performance and error classification. Even one year's worth of data from such a project would likely document substantial inaccuracies in vital statistics. Continuing such a project could also provide ongoing epidemiologic data, as well as more meaningful error rates that could be used to fuel quality improvement efforts throughout the health system. Such a program would not replace autopsies as routinely performed elsewhere, that is, this suggested research program would not be equivalent to a system of regional autopsy centers performing autopsies on behalf of other institutions. Rather, these centers would act as surveillance centers for basic causes of death and detection of quality problems and present numerous opportunities for basic research into the pathogenesis of acute and chronic illnesses.

Background

The nomination for this report came from the College of American Pathologists (CAP), which sought a formal assessment of the extensive literature on the role of the autopsy as an outcome and performance measure. This literature, beginning in 1912¹ and continuing to the present,^{2, 3} documents the prevalence of significant diagnostic errors discovered at autopsy, and forms the basis for regarding the autopsy as a potential tool in clinical audit and quality assessment. The autopsy has numerous other known and suggested benefits, ^4–10 with specific examples including: more accurate vital statistics^11–16; provision of accurate prevalence data for specific target conditions^17–22; pathologic descriptions of new diseases^23–26; teaching gross anatomy, disease progression and pathology in both undergraduate and graduate medical education^{27, 28}; and comfort to family members in knowing the cause of death or allaying fears regarding heritable conditions.^29–31

Despite the continued fulfillment of these roles, the frequency of autopsies has steadily declined in this country and elsewhere (Appendix Figures 1&2). In 1994, the last year for which there are national data, the autopsy rate for all non-forensic deaths fell below 6%.³² A recent survey found that over half of all hospitals in one state reported performing no autopsies during a one year period.³³ The marked decline in autopsy rates from previous rates as high as 40–50% undoubtedly reflects various factors, including the lack of direct or indirect economic incentives and absence of regulatory requirements for minimum autopsy rates.^32–34 Other factors include the attitudes of clinicians regarding the utility of autopsies in the setting of other diagnostic advances, and general unfamiliarity with the autopsy and techniques for requesting it, especially among physicians-in-training.^{28, 35–41}

Although the autopsy plays a number of potentially valuable roles, quantification of this value is difficult except in relation to the impact of autopsy performance on clinical diagnostics or therapeutics. In this way, assessments of the value of postmortem examination are handicapped in a fashion similar to assessments of the physical examination performed by clinicians. The act of examining patients has a number of benefits—focusing the clinician on the patient as an individual, reassuring the patient, and various roles in medical education.^{42, 43} Nevertheless, assessments of the value of the physical examination inevitably focus on its impact on medical decision-making related to diagnosis or treatment.^44–50

Recognizing at the outset, then, that our review would not take into account less quantifiable attributes of the autopsy (e.g., its value in medical education), we set out to assess the extent to which the autopsy provides a valid measure of clinical performance, especially as regards diagnosis.

An extensive literature has focused on the prevalence of clinically important diagnoses that remained undetected until autopsy. Several well-known studies from this literature have compared rates of “diagnostic errors” detected by autopsy from different time periods and found surprisingly little change.^51–55 It is important to note that the discrepancies between clinical and autopsy diagnoses discussed here and in these studies do not necessarily represent errors per se. In some cases, misinterpretation of test results, failure to respond appropriately to abnormal clinical findings and other such mistakes or 'slips" in the diagnostic process do produce diagnostic discrepancies warranting the term “error.” In other cases, however, atypical clinical presentations or the limits of current diagnostic techniques result in misdiagnoses without any true “errors” on the parts of clinicians. Despite this important distinction, we refer to “diagnostic errors” throughout this report because of the ubiquitous use of this phrase in the autopsy literature.

Selection of Patients for Autopsy

Discussions of the significance of autopsy-detected diagnostic errors have generally focused on the degree to which they reflect increased selection by clinicians.^{56, 57} Given that autopsy rates have declined, it is quite plausible to suggest that cases for which clinicians request autopsy represent precisely those that have presented diagnostic difficulties. This question is obviously important (and one which we address in this report), but it is also important to clarify how autopsy-detected diagnostic error rates should be interpreted.

Figure 1

Selection of Patients for Autopsy

Figure 1

.

   Selection of Patients for Autopsy

Reproduced from Saracci⁵⁸ with permission

Figure 1

Autopsied patients represent a nonrandom sample of all deaths, which in turn constitute a non-random sample of all patients. This non-random sampling is further complicated by the fact that most autopsy series involve hospital deaths only, adding a further level of selection. Thus, the interpretation of autopsy-detected diagnostic error rates is complicated by secular changes in the proportion and demographics of patients passing through these successive selections. Fewer and sicker patients are admitted to the hospital for shorter periods of time.^59–67 For many conditions, overall mortality has decreased, and, as already noted, fewer deceased patients undergo autopsy. Before attempting to unravel these complicating factors, it is worth considering the extent to which one ought to expect the rate of autopsy detected errors in clinical diagnosis to decrease over time (as commonly assumed in studies reporting these rates).

Patients who die generally fall into one of two categories - treatment failures (e.g., complications from treatments, ineffective treatment available, etc.) or diagnostic failures (e.g., missed or incorrect diagnoses).¹While improvements in care reduce overall mortality for an increasing number of conditions, the proportion of deaths in these two categories may remain relatively unchanged. In fact, if the number of patients in the first category (treatment failures) decreases over time for many conditions, the relative proportion of deaths due to diagnostic failures might increase. Alternatively, if diagnostic advances outpace improvements in therapy, more patients will be diagnosed with a condition, but then not respond to treatment and therefore count as treatment failures. This issue is discussed in greater detail in the section on autopsy studies of the sensitivity of clinical diagnosis for pulmonary embolism.

The main point is that the relationship between diagnostic errors detected at autopsy and the overall performance of clinical diagnosis is quite complex. For a given condition it may be possible to predict trends in diagnostic error rates, but for clinical diagnosis taken as a whole, there is no reason to expect a decrease in error rates over time. The specific study questions below attempt to clarify the significance of autopsy-detected errors in clinical diagnosis in the context of this complexity.

Objectives

This report primarily addresses the following questions:

1. To what extent does the autopsy reveal important diagnoses that were clinically unsuspected prior to death?

2. To what extent does the autopsy provide a useful performance measure or audit of clinical diagnosis in general?

3. What impact do autopsy findings have on clinical performance improvement?

4. Do autopsy findings have a role to play in generating accurate epidemiologic data (e.g., vital statistics)?

Although the first two questions are often considered the same, they are in fact quite distinct. The first question addresses the degree to which clinicians have correctly identified important potentially treatable diagnoses (e.g., pulmonary embolism, dissecting aortic aneurysm, bowel perforation) prior to death. Using the autopsy literature to answer this question requires assessing the extent to which the rate of diagnostic errors detected at autopsy simply reflects selection by clinicians of diagnostically challenging cases. Studies directly relevant to this assessment include autopsy series in which close to 100% of deaths were autopsied and studies in which clinicians explicitly state their diagnostic confidence prior to autopsy results becoming known.

The second question asks how common is it for important treatable diagnoses to escape detection during life and therefore first come to attention at autopsy. Answering this question requires taking into account performance of clinical diagnosis among all patients with conditions of interest, not just those who died. Precise evaluations of the overall performance of clinical diagnosis are complicated by the lack of a uniform diagnostic standard for (presumptive) clinical diagnoses among patients who survive. Nevertheless, long-term clinical follow-up of patients who survive combined with autopsy results for patients who die provides an estimate of the true sensitivity of clinical diagnosis for a given condition of interest.

The third question asks what impact the autopsy has on quality improvement. In other words, to what extent does knowledge of clinically missed diagnoses or unrecognized complications of care reduce the subsequent occurrence of similar quality problems. Many commentators have responded to studies of diagnostic error rates detected at autopsy by calling for an increase in the rate at which autopsies are performed. But, increasing autopsy rates without appropriate utilization of the information derived from autopsies may achieve little benefit. If hospitals are to invest in increasing current autopsy rates, there would also need to be simultaneous improvements in the utilization of the information derived from autopsies, in order to have a chance of realizing a clinical benefit for future patients.

The above questions focus on clinical applications of autopsy-detected errors, but the autopsy undoubtedly has multiple other values. One of the more quantifiable or tangible potential applications of autopsy-detected diagnostic errors is in generating accurate vital statistics, which is the target of the fourth question.

Search Strategy

We conducted an extensive electronic literature of the MEDLINE® database, supplemented by hand searches of article bibliographies and information from experts in the field. The MEDLINE® search strategy involved the Medical Subject Heading (MeSH) terms autopsy and postmortem changes, as well as the following title words: autopsy, post-mortem, postmortem, necropsy, and posthumous. We searched the more than 35,000 citations retrieved with this initial broad search using terms that captured aspects of study design (e.g., epidemiologic studies, clinical trials, comparative study) or topics relating to diagnosis (e.g., diagnostic errors, diagnostic techniques and procedures, diagnosis, differential) or error (e.g., medical error, iatrogenic disease, sentinel surveillance, safety).

The Cochrane Library was searched using similar key terms and title words. Preliminary searches of the CINAHL® database revealed extensive overlap with articles already retrieved from MEDLINE®, so this database was not searched further.

Reference lists from all relevant articles were reviewed to identify additional studies. The extensive search was completed in September 2000, but periodic electronic searches were conducted throughout the project until November 1, 2001 as relevant studies continued to appear during this time. During the final preparation of the report, four additional studies were identified after a repeat search was conducted in April 2002.^319–322

Unlike systematic reviews of treatment efficacy, reviews related to health services research must take into account differences in practice patterns between countries, as well as different economic structures to health care systems. We were concerned that U.S. clinicians might regard patients in other countries as less likely to undergo extensive sophisticated imaging procedures (e.g., magnetic resonance imaging) and thus question the relevance of these studies to U.S. practice. On the other hand, we did not want to miss the opportunity to review as much of the world literature as possible, especially as this view likely represents a misconception. As a result, we did not rule out the possibility of including non-English studies, especially when English language abstracts were available and indicated relevant data addressing a research question not answered by data from English-language articles. However, we did not search other databases known to capture more of the non-English published literature (e.g., EMBASE).

Study Selection

The autopsy literature consists entirely of observational studies, rendering problematic the development of appropriate inclusion and exclusion criteria as most systematic reviews usually involve at least some randomized controlled trials. In the absence of quality scoring systems or other well-established criteria, we adopted fairly minimal inclusion and exclusion criteria.

Data Collection and Analysis

Articles identified from the literature search were stored in a reference database and categorized according to the study questions addressed (Appendix C). Structured abstraction forms (Appendix D) were then used to collect demographic data (pertaining to patients and institutions), salient methodologic features and results. Each article was abstracted by at least two of four reviewers, including three physicians and one non-physician research assistant. One of the physicians reviewed all of the articles.

Mean error rates were calculated using logistic regression methods. Specifically, the probability of autopsy detected errors served as the dependent variable in a regression model that included as predictors: study period, autopsy rate, case mix and country (US or non-US). Appendix B provides further details regarding the statistical methods.

The Autopsy as a Diagnostic Test

Accuracy of the Autopsy

Like most complex procedures involving multiple observational and cognitive elements, the autopsy almost certainly has an error rate of its own,⁶⁸ despite its role as a diagnostic standard. All diagnostic tests attempt to capture the “true diagnosis,” and test characteristics such as accuracy are defined in terms of this true diagnosis. For a “gold standard,” assessing the extent to which the test falls short of capturing the true diagnosis, presents practical difficulties because confirmatory tests generally rely on the gold standard for their own accuracy.

Cases in which the autopsy fails to provide a definite diagnosis convey some sense of the limitations or potential for inaccuracy on the part of the autopsy. A number of studies mention in passing the frequency with which adequately performed autopsy fails to establish a definitive diagnosis related to the cause of death (immediate, intervening or underlying). As shown in Appendix Table 1, technically adequate autopsies generally establish causes of death in all but 1–5% of cases, although some studies have reported slightly higher proportions. One of the outlier results indicating a substantially higher rate of persistent uncertainty after autopsy derives from data gathered in 1958.⁹⁰ Three other reports with rates of persistent uncertainty after autopsy that exceed 20% focused on perinatal deaths, ^91–93 but one study of adult deaths after cardiac surgery also reported persistent uncertainty after autopsy in 25% of cases.⁹⁴

Table 1

Sensitivity of Clinical Diagnosis Comparing Theoretic (more...)

Table 1

Sensitivity of Clinical Diagnosis Comparing Theoretic True Diagnosis with Autopsy Results as the Diagnostic Standard

		“True Diagnosis” (Observed Autopsy Diagnosis)
		Lung Cancer	Other Major Diagnosis	Total
Clinical Diagnosis	Lung Cancer	180 (182)	90 (88)	270 (270)
	Other Major Diagnosis	20 (54)	1710 (1676)	1730 (1730)
	Total	200 (236)	1800 (1764)	2000 (2000)

Adapted from Saracci ⁵⁸

It is important to realize that even fairly small error rates in any presumed diagnostic standard can significantly distort estimates of the sensitivity of clinical performance.⁹⁵ Table 1 (adapted from Saracci⁶⁸) illustrates this effect using a hypothetical series of 2000 autopsies in which the investigators report the agreement between clinical and autopsy diagnosis of lung cancer related to the cause of death.

Acknowledging that no test provides the “true diagnosis”, Table 1 compares the sensitivity of clinical diagnosis measured against the theoretic “true diagnosis” with that obtained when measured against the observed autopsy diagnosis. The values in parentheses correspond to the observed autopsy findings (rather than the “true diagnosis”), with the specific values in parentheses generated by assuming that 36 (2%) of the 1800 non-lung cancer cases were misclassified by autopsy as deaths due to lung cancer.

Using the true diagnosis as the diagnostic standard, the sensitivity of clinical diagnosis for detecting lung cancer as the cause of death is 180/(180+20) = 0.90. When the observed autopsy results are taken as the diagnostic standard, the same sensitivity is 182/(182+54) = 0.77. Thus, even a relatively small autopsy misclassification rate of 2% would reduce the observed sensitivity of clinical diagnosis for lung cancer from its “true” value of 90% to an apparent value of 77%.

This illustration assumes that all of the misclassified cases have the same distribution as those cases already in the left-hand column. In fact, it could be that none of these cases would be diagnosed clinically as lung cancer, so that the observed sensitivity would correspond to 180/236=76%. At the other extreme, all of these misclassified cases might be diagnosed clinically as lung cancer, resulting in an observed sensitivity of 216/236=91%. Given this range of values, the conclusion remains that small rates of misclassification at autopsy can produce significant changes in the apparent sensitivity of clinical diagnosis.

Precision of the Autopsy

In the absence of data on the accuracy of the autopsy, one can still ask about its reproducibility - i.e., the extent to which pathologists agree in their identification of the main autopsy diagnoses. Multiple studies within the pathology literature address the reproducibility for diagnoses in surgical pathology and cytology. These studies suggest a wide range of values for measures of agreement from excellent to poor, but the majority falls in the range conventionally regarded as indicating moderate to substantial agreement. ^96–107 The range of values varies depending on the specific test evaluated (e.g., cervical cytology, ^{96, 97} prostate ^{98, 99} or liver biopsies¹⁰¹) and the evaluation setting (e.g., general pathologists versus specialists^{98, 99}).

The above studies do not assess the extent to which pathologists confronted with the results of the diverse investigations that make up a given autopsy independently arrive at the same conclusions. Despite an extensive search, we found only three studies that address the issue of reproducibility among pathologists in interpreting autopsy findings.^108–110 The oldest of these studies¹¹⁰ compared the judgments of three reviewers in assigning a principal cause of death based on the findings presented in 50 autopsy reports. Because the three reviewers were third-year medical students (trained to abstract autopsy reports and code death certificates for the purposes of this study), we did not regard the briefly summarized data on inter-rater agreement, presented in passing in this study of the accuracy of death certificates, as indicative of agreement among pathologists in ordinary clinical practice.

A second study¹⁰⁹ involved the analysis of a series of 39 patients who died in an intensive care unit and underwent autopsy specifically to address pulmonary pathology. The study excluded patients infected with HIV, bilateral chest tubes or bilateral pleural infection. Four pathologists (who were blinded to clinical and microbiological data) independently examined postmortem lung biopsies from these patients with the purpose of judging the presence or absence of pneumonia. The authors reported these independent judgments as well as the diagnoses derived from the application of pre-selected criteria for the histologic diagnosis of pneumonia.¹¹¹

Table 2

Inter-rater Agreement for Pathologists Assessing the (more...)

Table 2

Inter-rater Agreement for Pathologists Assessing the Presence or Absence of Pneumonia at Autopsy

Observer pair	Observed Kappa (95% CI)*	Maximum possible Kappa†
A,B	0.72 (0.49–0.95)	0.83
A,C	0.65 (0.39–0.91)	0.65
A,D	0.52 (0.22–0.82)	0.52
B,C	0.81 (0.59–1.0)	0.81
B,D	0.66 (0.38–0.94)	0.66
C,D	0.84 (0.63–1.0)	0.84
D,D (6 months later)	0.82 (0.59–1.0)	1.0

*

Cohen's kappa, ^{112, 113} with 95% confidence-interval limits calculated according to the Wilson efficient-score method, corrected for continuity

†

Maximum possible kappa, given the observed marginal frequencies

Using their own judgments, the four pathologists exhibited unanimity in the diagnosis of 30 (77%) patients—7 with pneumonia and 23 without. The authors reported a kappa score for the overall group, but also provided enough data to allow calculation of the more meaningful kappa scores for the different possible pairs of observers (Table 2).

As seen in Table 2, the different possible pairs of pathologists exhibited “substantial” (kappa=0.6–0.8) to “near perfect”(kappa=0.8–1.) agreement, with only 1 pair falling in the “moderate” range (kappa=0.4–0.6). Six months later, one pathologist re-reviewed the same biopsies and reclassified only 2 patients (one from “pneumonia” to “no pneumonia” and the other vice versa), corresponding to a kappa score of 0.82. However, the overall prevalence of pneumonia as determined by each of the four pathologists varied from 18% to 38%. Moreover, this study assessed agreement with respect to a specific question with a dichotomous answer (pneumonia present or absent). For the performance of an autopsy, pathologists must choose from among multiple possible diagnoses, rather than determining the presence or absence of a single condition.

The third study¹⁰⁸ is the only one to address agreement among pathologists with respect to the autopsy as a whole - i.e. in determining principal underlying diseases and causes of death. This study involved four pathologists examining 35 autopsies. The authors reported excellent (near perfect) agreement for all pathologist pairs independently determining the principal disease (i.e., underlying cause of death), with kappa values between 0.83 and 0.97. Specifically, there were four cases that generated disagreement in the principal underlying disease. In three of these cases, only one pathologist disagreed with the other three, while in the fourth case two pathologists shared one diagnosis, while two shared a different diagnosis.

For assignments of the immediate cause of death, the authors reported kappa values ranging from 0.43–0.75. Importantly, this level of agreement discounts the roughly 30% of cases for which two of the pathologists could not decide on an immediate cause of death, since in these cases agreement was not measured. Listings of minor diseases showed much less agreement.

Overall, it is likely that pathologists exhibit substantial agreement in generating the principal autopsy diagnoses, although the literature contains little data addressing this issue.¹

Reproducibility of Judgments of Autopsy-detected Errors in Clinical Diagnosis

Numerous studies report group consensus for important discrepancies between clinical and autopsy diagnoses related to the cause of death.^{2, 51, 54, 55, 116–133} Unfortunately, only five studies^{55, 117, 124, 130, 131} mention the issue of reproducibility for these judgments of diagnostic discrepancies. In one of these studies,⁵⁵ three investigators independently classified each case, with a fourth reviewer for cases with discordant classifications. Reclassification occurred for 20% of Class I and II errors, but the authors provided no further details on the consensus process nor did they calculate a kappa score or other measure of inter-rater agreement. In another study,¹²⁴ the authors report that in 25 of 338 autopsies (7.4%, 95% CI: 4.9–10.9%) two reviewers could not reach consensus, so classification of clinical-autopsy discordances required a panel of peer reviewers. The other three studies^{117, 130, 131} provided too little detail regarding the assessment of agreement between reviewers to permit meaningful assessments of the reproducibility of judgments pertaining to diagnostic errors detected at autopsy.

Because these five studies provide so few data on the frequency of differences in opinion between reviewers assessing clinical-autopsy discrepancies, we reviewed studies from the medical error literature that provide data on comparable assessments. Several well-known studies have examined the reproducibility of peer review classifications of hospital deaths as “preventable” or “not preventable.” ^134–137 These studies are relevant to the present question because the definition of preventability lies at the heart of the two most commonly used classifications of discrepancies between clinical and autopsy diagnoses.^{51, 70}

Assessments of reviewer agreement from the medical error literature indicate that achieving anything beyond fair or modest inter-rater reliability for judgments about error or preventability for major adverse outcomes requires reviewer training and, in some cases, dropping reviewers that consistently yield divergent judgments.¹³⁶ In the most recent of these studies (and the one which most specifically focused on the issue of reproducibility), the authors noted that “if one reviewer rated a death as definitely or probably preventable, the probability that the next reviewer would rate that case as definitely not preventable (18%) was actually slightly higher than the probability that the second reviewer would agree with the first (16%).” ¹³⁷

Summary of the autopsy as a diagnostic test

• The quality of the autopsy has received little systematic study except in the case of perinatal autopsies, where deficiencies in the quality of reporting appear to be common.

• In at least 1–5% of cases, diagnostic uncertainty persists despite technically adequate autopsy. Classification errors affecting autopsy diagnoses at even this relatively small rate can substantially distort estimates of the performance of clinical diagnosis.

• Only one small study has specifically assessed the reproducibility of autopsy diagnoses among pathologists. This study showed excellent inter-rater agreement for the principal diagnosis, but data on agreement for other major diagnoses were not presented.

• The reproducibility of judgments about errors in clinical diagnosis as indicated by autopsy findings has received almost no attention. Extrapolation from studies of inter-rater reliability for peer review assessments of case notes in the healthcare quality and medical error literature suggests that reproducibility of such judgments is likely to be fair to moderate at best.

Autopsy Detection of Clinically Unsuspected Diagnoses

Quantitative Analysis of Diagnostic Error Rates

Notwithstanding the existing deficiencies in standardizing a systematic means of using the autopsy as a measure of clinical diagnostic performance, the extensive literature on clinically missed diagnoses detected at autopsy nevertheless bears attention from clinicians and others interested in measures of clinical quality.

We identified 50 studies reporting diagnostic error rates that met our inclusion criteria. These 50 studies reported a total of 61 distinct autopsy series, as many of them reported more than one observation period or presented data from more than one institution. (Thus, if a study reported autopsy data for two hospitals, we presented these separately whenever the original paper presented sufficient detail to allow this.) Appendix Tables 2–4 present the salient features of these autopsy series using the three classification systems adopted for this study. Studies in each table are listed chronologically (with respect to the period of study, not date of publication) within patient population categories, with studies of more general patient populations listed first, followed by studies of more specific patient populations (adult medical, surgical, pediatric, etc.). Appendix Figures 3 & 4 displays the error rates and 95% confidence intervals for the studies listed in Appendix Tables 2 & 3. (A comparable figure was not created for the third definition of error, major ICD disease discrepancies, because of the small number of studies involved.)

One would expect diagnostic error rates to decrease over time due to medical advances, and thus would expect an inverse correlation between error rates and study period. Appendix Figures 5 & 6 present time trends for Class I and major error rates, respectively. Inspection of these figures indicates no clear changes in error rates over time, but it is possible that trends are obscured by differences in case mix, country effects and changes in autopsy rates. The latter possibility is of considerable interest, as clinicians generally believe that autopsy cases represent precisely those cases most likely to have unexpected findings detected at autopsy. Thus, true improvements in clinical diagnosis might be offset by increased selection of autopsy cases by clinicians. We addressed this possibility as well as the effects of case mix and study country by performing logistic regression using a model in which study period, autopsy rate, case mix (using 7 different categories) and country (U.S. vs. non-U.S.) were included as predictors of error rates.

The results of the quantitative analysis are presented in detail in Appendix B and summarized below. Briefly, we performed logistic regression for each of the three definitions of error (Class I, major and discrepancies in major ICD disease categories). In each case, the analysis began with a model that included all predictors—study period, autopsy rate, case mix (general adult inpatients, medical patients, surgical parents, pediatric, etc.) and country (U.S., non-U.S.). Hospital teaching status was not included as a predictor in any of the models, because too few studies involved non-teaching hospitals and because the nature of the teaching status (academic medical center, community teaching vs. tertiary referral center, presence of residents across a broad range of specialties) was usually unclear.

Table 3

Mean Error Rates and Correlations with Time, Autopsy (more...)

Table 3

Mean Error Rates and Correlations with Time, Autopsy Rates

Error Classification	Error rate* (95% CI)	Change in error rate with time**	Change in error rate with increased autopsy rate†
Class I	10.2% (6.7–15.3%)	26.2% relative decrease per decade (p=0.1)	7.8% relative decrease for each 10% increase in autopsy rate (p=0.2)
Major errors	25.6% (95% CI: 20.8–31.2%)	28% relative decrease per decade (p=0.0006)	12% relative decrease for each 10% increase in autopsy rate (p=0.0003)
Discrepant ICD Disease Categories	11.7% (9.7–13.9%)	28% increase per year (p<0.0001)	1.4% decrease for each 5% increase in autopsy rate (p=0.1)

*

The error rates listed here represent the probability of a diagnostic error at time zero (1980 for Class I and major errors, 1975 for ICD discrepancies), with the referent case mix category (general inpatients or adult inpatients), reference country (U.S.) and unweighted mean autopsy rate (44.3%) for included studies.

†

Changes are all relative to the value for the base year and mean autopsy rate.

The results generated by the complete model were compared with models in which one variable at a time was dropped. For Class I errors, the differences between the results for the different models did not achieve statistical significance, so the mean error rate could be calculated from the model that used time as the only predictor for the probability of a Class I error. However, because the contributions of these other factors (autopsy rate, country, case mix) were still noticeable (even if not statistically significant) and were clearly plausible, we used the more complete model to compute the mean error rate and the range of error rates shown in Table 3. This decision seemed especially appropriate given that, in the analysis for “major errors,” time and autopsy rate both proved to have a statistically significant effect on the observed error rate.

Mean Diagnostic Error Rates and Relationships to Time and Autopsy Rate

Table 4

Class I Error Rates Computed for Varying Autopsy Rates (more...)

Table 4

Class I Error Rates Computed for Varying Autopsy Rates in 4 Different Years

Autopsy Rate	1970	1980	1990	2000
5%	17.6%	13.6%	10.4%	7.9%
10%	17.0%	13.1%	10.0%	7.6%
20%	15.9%	12.2%	9.3%	7.1%
40%	13.9%	10.6%	8.0%	6.1%
80%	10.4%	7.9%	5.9%	4.5%
100%	9.0%	6.8%	5.1%	3.8%

Table 5

Major Error Rates Computed for Varying Autopsy Rates (more...)

Table 5

Major Error Rates Computed for Varying Autopsy Rates in 4 Different Years

Autopsy Rate	1970	1980	1990	2000
5%	44.3%	36.4%	29.2%	22.9%
10%	42.7%	34.9%	27.9%	21.8%
20%	39.6%	32.1%	25.4%	19.7%
40%	33.7%	26.8%	20.8%	15.9%
80%	23.3%	17.9%	13.6%	10.2%
100%	19.0%	14.5%	10.8%	8.1%

The relationships between error rates and both time and autopsy rate were statistically significant for major errors, but not Class I error rates. Nonetheless, as shown below (Tables 4 and 5), the effects of these factors are noticeable enough for both error definitions so that a mean error rate per se is not meaningful in the absence of stipulated values for time and autopsy rate. Thus, the “means” listed in Table 3 are not true means. Rather, they represent the probability of an error for the base values of time, autopsy rate, case mix and country used in the regression model. Time was centered on 1980 for Class I and major errors and 1975 for major ICD discrepancies. For case mix, the base category consisted of general inpatients (or general adult inpatients), and the reference value for country was defined as the U.S. Autopsy rates were centered on the mean autopsy rate for the studies included in the model.

Using the base values for case mix, country, autopsy rate (the overall mean autopsy rate of 44.3%), and time (1980, the midpoint of the period of analysis), the rate of autopsy-detected diagnostic errors was 10.2% (95% CI: 6.7–15.3%) and 25.6% (95% CI: 20.8–31.2%) for Class I and major errors, respectively. Restricting the analysis to U.S. institutions yielded similar point estimates and almost entirely overlapping confidence intervals—11.2% (95% CI: 6.9–17.5%) and 24.0% (95% CI: 17.3–32.3%) for Class I and major errors, respectively.

Error Rates as a function of Time and Autopsy Rates

The expected inverse correlation between error rate and study period (i.e., the more recent the study the lower the error rate) was observed for Class I and major errors, though this relationship was statistically significant only for the latter. Specifically, Class I errors decreased at a rate of 26.2% per decade relative to the base prevalence in 1980 (p=0.10). The probability of a major error exhibited a relative decrease of 28.0% (95% CI: 9.8–42.6%) per decade.

One would expect the rate of decrease for Class I errors and major errors to be similar. As this was, in fact, observed (with relative decreases of 26.2% and 28.0% per year for Class I and major error rates, respectively), it is likely that the decrease in Class I errors is real, and that the statistically non-significant result reflects inadequate power. In fact, the 95% confidence interval for this relationship between Class I errors and time extends from a 48.8% decrease to a 6.3% increase (i.e., the 95% CI lies predominantly on the side of a relative decrease), reinforcing the interpretation that the time trend for Class I errors is real.

The expected inverse correlation between error rate and autopsy rate (i.e., the higher the autopsy rate, the lower the error rate) was also observed, though this relationship was again statistically significant only for major errors. Specifically, for every 10% increase in autopsies, the Class I error rates exhibited a relative decrease of 7.8% (p=0.2) and major errors decreased by 12.0% (p=0.0003) relative to the base error rate.

In contrast with the other two definitions of errors, discrepancies in major ICD disease categories (between clinical and autopsy diagnoses) showed an increase over time, and this relationship was statistically significant. Specifically, the error rate increased by roughly 28% per year (p<0.0001). We had included this definition because, in principle, it represents a relatively objective type of error and because several of the studies using this definition were generally well designed and had high autopsy rates. Unfortunately, these positive features were likely overwhelmed by the heterogeneous effects introduced by variations in coding practices in different countries over different time periods.

Although the impacts of time and autopsy rate were statistically significant only for major errors, the cumulative effects were noticeable for both error types. Table 4 shows Class I error rates for varying autopsy rates in 4 different years. These results were calculated using the regression model, as shown in Appendix B. Table 5 shows the same trends for major errors. These trends are displayed graphically (with a greater range of values) in Appendix Figures 7 & 8.

The results of table 4 and 5 strongly suggest a trend toward decreased error rates over time, though the magnitude of the decrease is modest. These results corroborate the findings of the single study⁵⁵ assessing trends in autopsy-detected error rates over time in which all time periods had high and approximately equal autopsy rates.

Using the regression model and standard error to calculate 95% confidence intervals (as shown in Appendix B) we can estimate the error rates expected in an academic medical center in the U.S., where one might find an autopsy rate of 20%. (In fact, many academic centers have even lower rates than this). The specific error rates to be expected with an autopsy rate of 20% are 6.7% (95% CI: 3.9–9.2%) and 20.3% (95% CI: 16.7–24.5%) for Class I and major errors respectively.

Clinical Selection of Cases for Autopsy

If the above analysis had shown a decrease in error rates over time by adjusting for decreases in autopsy rates, there would have been indirect support for the hypothesis that unchanged error rates reflect increased selection of cases by clinicians. In other words, as fewer autopsies are performed, the proportion of cases that presented diagnostic challenges to clinicians may increase, offsetting any improvements in overall diagnostic performance. Such an effect was not demonstrable in the above analysis, but is not ruled out by these results. Thus, we looked for studies specifically addressing the issue of clinical selection of cases for autopsy.

One study commonly cited as evidence that clinical selection does not occur simply demonstrated that major diagnostic groups did not differ between autopsied and non-autopsied cases.⁵⁷ Thus, what this study really shows is that patients with neoplastic diseases are no more or less likely to undergo autopsy than patients with cardiovascular disease, infectious diseases, and so on. Even in this limited sense of selection, other studies have shown a bias towards certain diagnostic categories (e.g., trauma and “ill-defined conditions”) and a bias against other diagnostic categories (e.g., infectious and neoplastic) in referral for autopsy.¹⁴

More important than the diagnostic categories for principal causes of death is the level of diagnostic certainty, and whether or not clinicians refer cases for autopsy primarily in uncertain cases. Addressing this question requires assessing diagnostic certainty using a chart-based tool or prospectively eliciting from clinicians their level of diagnostic certainty prior to autopsy results becoming available.

Table 6

Autopsy Rate Stratified by Formally Assessed Levels (more...)

Table 6

Autopsy Rate Stratified by Formally Assessed Levels of Diagnostic Certainty for Deaths in a Neonatal Intensive Care Unit (as reported by Dhar et al 124)

Diagnostic Certainty	Autopsy Rate (95% CI)
Completely Certain (Level 1)	53.1% (41.7–64.1%)
Almost Certain (Level 2)	61.3% (56.2–66.2%)
Probable or suspected (Level 3)	73% (62.4–81.6%)

In the only example we identified involving the first of these two approaches, Dhar et al formally assessed “diagnostic certainty” among deaths in a neonatal intensive care unit (ICU).¹²⁴ The authors defined three levels of diagnostic certainty ranging from “completely certain” (Level 1), in which the diagnosis was based on the definitive diagnostic standard attainable during life and on genetic, microbiologic or biochemical testing, to “probable, possible or suspected” (Level 3). As shown in the table 6, clinicians obtained autopsies for a greater percentage of patients with Level III than Level II, which in turn had a greater percentage of autopsies than Level I cases. Thus, the decision to obtain autopsies was clearly impacted by the level of certainty, even in this group of neonatal deaths with a higher overall autopsy rate than seen in most other patient populations.

Stronger evidence for the effect of clinical selection on the autopsy decision comes from a small number of studies conducted prospectively with the goal of assessing clinical certainty prior to autopsy. These studies are discussed in detail below; their results are summarized in Appendix Table 5.

Prospective Studies Addressing Autopsy Selection by Clinicians

Table 7

Autopsy Rate and Diagnostic Disagreements Stratified (more...)

Table 7

Autopsy Rate and Diagnostic Disagreements Stratified by Clinical Confidence (from Heasman and Lipworth127)

Clinical Confidence	Total Number of deaths	Autopsy rate	Disagreement rate
Fairly certain	9,248	57%	16%
Probable	3,694	76%	33%
Uncertain	1,282	88%	50%
Not stated	393	67%	28%
Total	14,617	65%	25%

Table 8

Number of Autopsies and Percent Diagnostic Errors Stratified (more...)

Table 8

Number of Autopsies and Percent Diagnostic Errors Stratified by Clinical Confidence and Patient Age (from Britton121, 122)

Clinical Diagnosis	Age < 70	Age≥ 70	Total
Fairly certain - # (% erroneous)	91 (15%)	91 (34%)	182 (25%)
Probable - # (% erroneous)	60 (33%)	91 (53%)	151 (45%)
Total - # (% erroneous)	151 (23%)	182 (43%)	333 (34%)

Table 9

Autopsy Rates and Diagnostic Confirmation Rates Stratified (more...)

Table 9

Autopsy Rates and Diagnostic Confirmation Rates Stratified by Clinical Confidence (from Cameron and McGoogan155, 156)

Clinicians' Assessments of Principal Diagnosis	Percent of Cases	Confirmation of Principal Diagnosis at Autopsy
Fairly certain	47%	75%
Probable	35%	55%
Uncertain	16%	30%

Table 10

Diagnostic Confirmation Rates Stratified by Clinical (more...)

Table 10

Diagnostic Confirmation Rates Stratified by Clinical Confidence Before and After Intervention to Increase Autopsy Rate (from Cameron et al123)

	Number of cases (Percent of cases with autopsy confirmation of principal clinical diagnosis %)
	5/75–6/77 (Autopsy rate 30%)	6 months in 1978 (Autopsy Rate 63%)
Total # of autopsies	326	154
Main diagnosis “certain” or “fairly certain”	168 (52%)	144 (94%)
Autopsy confirmation of main diagnosis	182 (56%)	131 (85%)
Autopsy confirmation of causes of death	60 (18%)	90 (58%)
Autopsy would normally NOT be requested	-----	44 (86%)
Autopsy would normally be requested	-----	110 (85%)

Table 11

Clinicians Assessment of Necessity in Cases Sent for (more...)

Table 11

Clinicians Assessment of Necessity in Cases Sent for Autopsy and Stratified by Clinical Confidence (from Hartveit293)

Confidence in clinical diagnosis	Clinicians' Assessments of Autopsy Necessity
	Essential	Desirable	Little Interest
Certain	22%	70%	8%
Uncertain	45%	54%	1%

Table 12

Diagnostic Errors Detected by Autopsy in the Single (more...)

Table 12

Diagnostic Errors Detected by Autopsy in the Single Study Including Discharge Diagnoses for Patients who did not Die (from Cameron & McGoogan155, 156)

	Principal diagnoses and causes of death as determined by autopsy in 1975-7			Principal discharge diagnoses from same hospitals in 1976
	Confirmed	Overturned	Missed	Alive	Died
Pulmonary TB	7	8	7	138	5
PE	44	35	99	140	18
Acute appendicitis	3	1	1	441	4
Bowel obstruction	6	3	0	159	13
Acute MI	198	58	51	1452	258
Cirrhosis	22	5	13	125	12

TB-tuberculosis; PE-pulmonary embolism; MI-myocardial infarction

1. Heasman MA, Lipworth L: Accuracy of certification of cause of death. In: General Register Office, ed. Studies on medical and populations subjects, No. 20. London: HMSO, 1966.

   This prospective study conducted by government departments of health and vital statistics in the United Kingdom involved 75 British hospitals, including 23 in London and 52 scattered throughout England and Wales. The 9,501 autopsies included in this study consisted of adult inpatients who died during a 6-month period in 1959.

   For each autopsy request, a clinician filled out a “dummy death certificate,” designed specifically for this study and independent of the usual death certificate. This form asked the clinician requesting autopsy to state the differential diagnosis at the time of death (prior to autopsy results) and indicate the cause of death as “fairly certain,” probable" or “uncertain.” Table 7 summarizes the autopsy rates and rates of clinical-autopsy diagnostic disagreement in terms of clinicians' stated confidence in their diagnoses.

   As expected, the less certain clinicians were regarding their diagnoses, the more likely they were to obtain an autopsy and the higher the chance of detecting an error in clinical diagnosis.

2. i) Britton M. Diagnostic errors discovered at autopsy. Acta Med Scand 1974;196:203–10. ii) Britton M. Clinical diagnostics: experience from 383 autopsied cases. Acta Med Scand 1974;196:211–9.

   This study of 383 adult inpatient autopsies conducted at Serafimerlasarettet University Hospital in Sweden from 1970-71 is notable for its prospective design and 96% autopsy rate. The author defined errors in clinical diagnosis in terms of discrepancies in ICD major disease categories. Error rates were then analyzed in terms of patient age and the certainty of clinical diagnosis as prospectively stated by clinicians prior to autopsy. Table 8 displays results for 333 of the 383 cases; the 50 cases identified by clinicians as “unknown” were not included, since (by definition for this study) no clinical diagnoses had been assigned to these cases prior to autopsy.

   The study author reported comparisons of each italicized cell to the adjacent cell (vertically or horizontally) as statistically significant (p<0.05). The relevant comparisons for the totals were also significant (i.e., the total error rate for patents < 70 years of age was significantly lower than for older patients, and the total error rate for “fairly certain” cases was significantly lower than the error rate for the “probable” cases). Overall, the prevalence of errors in the cases in which the clinical diagnosis was only “probable” was almost double that seen in the “fairly certain” group, even when age is taken into account.

3. i) Cameron HM, McGoogan E: A prospective study of 1152 hospital autopsies: I. Inaccuracies in death certification. J Pathol 1981; 133: 273–83.

   ii) Cameron HM, McGoogan E: A prospective study of 1152 hospital autopsies: II. Analysis of inaccuracies in clinical diagnoses and their significance. J Pathol 1981; 133: 285–300.

   This study of inpatient autopsies from hospitals in the South Lothian District in Scotland from 1975-77 also asked clinicians to record their assessment of the likelihood of their clinical diagnoses as “fairly certain,” “probable,” or “uncertain.” As shown in table 9, confirmation of clinical diagnoses correlated with clinical confidence.

4. Source: Cameron HM, McGoogan E, Watson H: Necropsy: a yardstick for clinical diagnoses. Br Med J 1980; 281: 985–8.

   The authors conducted a follow-up to the previous study in which they worked with clinicians to try to increase the autopsy rate at one of the institutions from the previous study. As before clinicians were asked to record their confidence in the clinical diagnoses, but were also asked to state whether or not an autopsy would normally have been obtained (i.e., in the absence of the initiative to increase autopsy frequency).

   Table 10 illustrates the impact of clinical selection on autopsy performance and diagnostic error rates. As shown in table 9, the confirmation of clinical diagnosis occurred more commonly in cases identified by clinicians as fairly certain.” The following year, the autopsy rate doubled, with the result that more “certain” or “fairly certain” cases underwent autopsies, and as a result the confirmation rate increased (and corresponding error rate decreased). Interestingly, though, confirmation rates did not differ between the two groups defined by clinicians' assessment of the need for autopsy. Cases identified as unlikely to undergo autopsy in the absence of the study had the same error rate as the cases for which clinicians' stated they would pursue autopsy even without the study. A similar result was obtained in the study summarized next.

5. Hartveit F. Clinical and Post-Mortem Assessment of the Cause of Death. J Pathol 1977;4:193–210.

   In this Norwegian study from The University of Bergen, clinicians were asked to identify cases as clinically “certain” or “uncertain,” but were also asked prospectively to record their impression of the autopsy as “essential,” “desirable,” or “of little interest.” The results of this study are presented in a very complicated manner, but table 11 conveys one of the main findings of the study, namely that clinicians' opinion of the necessity of an autopsy does not derive solely from their confidence in the clinical diagnosis. Even when clinicians regarded the diagnosis as clinically uncertain, autopsy was regarded as essential in only 45%. Also, though not indicated in the table below, autopsies in children were more likely to be regarded as essential despite the fact that clinical certainty about the diagnosis was also higher in children.

6. Landefeld CS, Chren MM, Myers A, Geller R, Robbins S, Goldman L. Diagnostic yield of the autopsy in a university hospital and a community hospital. N Engl J Med 1988; 318: 1249–54.

   Investigators in this prospective case-control study conducted daily reviews of in-hospital deaths in order to contact clinicians prior to receipt of autopsy results. Patients undergoing autopsy were paired with the next adult who died in the same hospital without undergoing autopsy.

   For each death, regardless of whether or not autopsy was eventually performed, clinicians were asked to record their assessment of the probability that the autopsy would reveal a major undiagnosed cause of death (i.e., a Class I or II finding). The questionnaire asked for an absolute probability (percentage from 0–100) and a qualitative estimate (5 point scale: “much more likely than usual,” more likely than usual,” “as likely as usual,” “less likely than usual,” much less likely than usual.”) The chi-square test for linear trend revealed no significant relationship between the physicians' estimated probability of Class I or II errors and the observed prevalence of such findings. The relationship between qualitative estimates and unexpected findings was not presented, but was reported as non-significant as well.

   The studies discussed above generally bear out the intuitively plausible claim that clinicians' diagnostic error rates occur more commonly in cases that clinicians identify as diagnostically uncertain. However, important diagnostic errors were still found in cases clinicians rated as diagnostically certain. Moreover, the relationship between clinicians' reported confidence in antemortem diagnoses and autopsy-detected error rate is complex. On the one hand, some studies show that clinicians can identify levels of diagnostic certainty that correlate with the rate of unexpected findings at autopsy. On the other hand, clinicians show little ability to predict the utility of the autopsy. One possible explanation is that the decision to request autopsy is not completely determined by clinicians' clinical certainty. Another possibility is that, in the context of a study explicitly targeting the accuracy of clinical diagnosis, clinicians reflected more carefully on the confidence of the clinical diagnosis (or even downplayed it) than they would routinely or when considering the decision to pursue autopsy. Probably both explanations play a role.

Summary of the autopsy as a diagnostic test

• The relationship between clinicians' reported confidence in antemortem diagnoses and autopsy-detected error rate is complex.

• To some extent, clinicians can identify levels of diagnostic certainty.

• Cases with greater uncertainty are more likely to go for autopsy and, in some studies, are more likely to have diagnostic errors.

• However, even in cases clinicians identify as “certain” with respect to confidence in diagnosis, the autopsy reveals major errors in roughly 5–15% of cases.

• Moreover, when asked specifically to predict unexpected findings or to state if the autopsy would be important/useful in a particular case, clinicians do not appear able to predict the findings (or utility) of postmortem findings.

The Autopsy as a Performance Measure for Clinical Diagnosis

The findings reviewed in the previous section indicate that, for a given patient who has died, there is a roughly 10% chance that autopsy will detect a clinically significant misdiagnosis. Moreover, clinicians' ability to predict cases likely to have unexpected findings at autopsy is at best weak to moderate. On an individual level, these data are quite compelling - i.e., from the point of view of an individual clinician deciding whether or not to pursue autopsy in the case of an individual patient death. But what about at the hospital or population level? To what extent do clinicians miss certain diagnoses in all patients, not just patients who die? As outlined in the Objectives section, this question can be understood as asking what proportion of clinically important diagnoses are first detected at autopsy.

Despite the extensive literature on diagnostic errors detected at autopsy, this question remains largely unanswerable with the existing literature, especially the autopsy literature per se. This gap in the literature does not result from the problem of selection bias - i.e., that clinicians may request autopsies precisely because they are uncertain about clinical diagnoses, so that autopsied cases are not representative of all deaths. Rather, this basic question remains largely unanswerable because the prevalence of clinically missed diagnoses at autopsy is not the same as the prevalence of diagnostic errors among all clinical cases, which is the true error rate of interest.

A typical study of diagnostic errors found at autopsy reports the number of clinically missed diagnoses and often lists the particularly common missed diagnoses (e.g., myocardial infarction, pulmonary embolism, systematic fungal infection, perforated bowel, and so on.) With only one exception,^{155, 156} these studies include no information relevant to estimating the relevant denominator, namely all cases of myocardial infarction, pulmonary embolism, fungal infection, and so on. If patient deaths were a random sample of all patients, error rates from autopsies alone would provide reasonable estimates of error rates for clinical diagnosis. However, precisely because some deaths represent diagnostic failures (not just treatment failures), autopsies do not provide a random sample with which to gauge the performance of clinical diagnosis.

Because of the importance of this issue, the one study from the autopsy literature that addresses the overall performance of clinical diagnosis is discussed in detail below. Before doing so, we address a point that will arise repeatedly in the discussion of estimates for clinical diagnostic performance, namely how to handle the non-autopsied deaths.

Impact of the Autopsy on Clinical Performance Improvement

Two examples from the medical error and patient safety literature illustrate the type of study required to demonstrate a benefit for performance improvement. In a prospective study aimed at reducing errors in radiographic interpretation by emergency physicians, example errors were used to create a teaching file.²²⁰ Moreover, a formal protocol for interpreting radiographs was adopted, and attending radiologists read all films within 12 hours to provide feedback and quality control. Subsequent outcome measures showed a significant decrease in the rate of significant missed diagnoses from 3% (95% CI: 2.8–3.2%) to 1.2% (95% CI: 1.03% to 1.37%). Further process redesign achieved a further reduction to 0.3% (0.26% to 0.34%). On a larger scale (comparable to what would be involved in an autopsy performance improvement project), a program at an Australian hospital ²²¹ showed a reduction in adverse events over an 8-year period, attributable to an intensive system for incident reporting and conducting root cause analyses of serious events.

No intervention study has examined the impact of autopsy-detected errors in clinical diagnosis on subsequent clinical performance. Given the absence of any studies of the impact of postmortem findings, the current literature provides no direct evidence for or against an impact of autopsy-detected errors on performance improvement at the level of individual practitioners or institutions. This is not to say that there is no valid role for the autopsy in relation to clinical practice or performance improvement, but instead reveals that the current literature is insufficient to address these issues.

Dissemination of findings from studies of autopsy-detected errors may, however, exert an impact on clinical practice at a broader level. While such an impact is plausible, the existing literature does not lend itself to analyzing trends in specific misdiagnoses, and certainly does not allow for inferences about causality. For instance, cirrhosis, bacterial pneumonia and tumors were commonly missed by clinicians and thus detected as major errors in autopsy studies prior to 1965. ^{51, 145} Subsequent studies were more likely to reveal previously uncommon diagnoses such as systemic fungal infections and pulmonary emboli,^{51, 145} suggesting some improvements in clinical practice. Similarly, even though specific tumor misdiagnoses continue to occur,²²²the recognition of the presence of a malignancy has almost certainly improved over time.^{51, 145, 223} These improvements, though difficult to document with existing data, plausibly result from general dissemination of the results of autopsies in the medical literature, even if local impacts remain unclear.

In the event that some connection could be shown between the identification of diagnostic errors at autopsy and subsequent performance, it is unclear how best to utilize the autopsy as a performance improvement tool at the institutional level. Discussions in the literature have generally assumed that the persistence of significant rates of diagnostic errors at autopsy implies that autopsy rates must be increased. However, the ability to measure the benefit of investing resources in such efforts is currently not feasible. Studies would be needed to establish optimal ways of utilizing the information generated by autopsies as currently performed or, at the least, to document the effectiveness of current methods.

Autopsy Detected Diagnostic Errors at the Health System Level

The previous sections have outlined the weak connection between autopsy-detected errors and overall clinical performance and the lack of any evidence for an impact of autopsy-detected errors on performance improvement. The prevalence of autopsy-detected errors in clinical diagnosis do, however, have major implications for vital statistics and other epidemiologic data, and these are discussed below.

Benefit of the autopsy for individual clinicians

We take as a given that many (ideally all) clinicians will have an intrinsic interest in the autopsy given the possibility of learning of important misdiagnoses in roughly 25% of cases, with roughly one third of these potentially having affected patient outcome. Importantly, though, as with the physical examination, quantitatively justifying this interest is difficult. (Interestingly, the quantifiable benefit of the physical examination is similar to the autopsy, with roughly 10% of diagnoses directly suggested by findings from patient examination.^44–46) While the autopsy has interest for individual clinicians (not to mention students and trainees) beyond what can easily be measured, the most quantifiable benefit for the autopsy at the individual practitioner's level is the potential impact on subsequent diagnostic performance.

Because physicians often fail to recognize diagnostic errors in the first place and thus miss the opportunity to change practice, the autopsy represents a potentially invaluable quality improvement tool, and demonstrating this value constitutes a crucial area of future research. Such research will face several important obstacles, though, including the need for substantially increasing autopsy rates in the first place, the relatively small number of cases (and even smaller number of errors) individual clinicians encounter, and substantial evidence that physicians tend not to change or improve their practice in response to interventions that consist only of the provision of new information.^282–289 On the other hand, by implementing strategies other than traditional conferences, pathologists and clinicians may achieve demonstrable effects on performance improvement, as observed with other more interactive ways of stimulating change, as educational outreach programs²⁹⁰ and involvement of local opinion leaders.²⁹¹

Benefit of the autopsy at the institutional level

The extensive literature search retrieved no reports of evaluations of interventions to improve clinical diagnostic performance based on autopsy-detected errors. Even assuming that such a benefit from autopsy findings is possible, increasing autopsy rates will not necessarily achieve this benefit without an established and effective mechanism for feeding back autopsy findings to clinicians and stimulating performance improvement.

Using institutional error rates for performance measurement is possible in principle, but meaningful comparisons are unlikely to occur even with modest increases in autopsy rates. For most hospitals, the error rates will have confidence intervals too wide to detect significant deviations from benchmark values in all but extreme cases.

Benefit of the autopsy at the level of the healthcare system

The existing literature provides two compelling reasons to pursue autopsies in order to benefit the healthcare system as a whole. First, results for the 5 conditions examined in this report suggest that clinical diagnosis in routine practice may not perform as well as is generally believed by clinicians or as suggested by the literature assessing specific aspects of clinical diagnosis (e.g., new tests) in research settings. Better characterizations of the performance of clinical diagnosis for common conditions would clearly benefit the entire health system and identify important targets for quality improvement that could be pursued in a concerted manner.

The second benefit to the entire health care system relates to vital statistics and other epidemiologic data. Vital statistics impact important decisions about allocation of funding for research and other aspects of healthcare policy. The existing literature demonstrates that clinical diagnoses, whether obtained from death certificates or hospital discharge data, contain major inaccuracies compared with diagnoses generated from postmortem findings. Because the accuracy of vital statistics is independent of consideration of impacts on prognosis, the error rate of interest is that found for major errors. Consequently, the existing evidence strongly suggests that substantial inaccuracies in 8–23% of diagnoses listed as causing or contributing to death. Given the importance of vital statistics and other epidemiologic data in conducting outcomes research, allocating research funding, and making other important policy decisions, using autopsy data to rectify this problem has the potential to have multiple benefits for the health care system as a whole.

Appendix Figure 1. Autopsy Rates Over Time Based on Autopsy Studies of Diagnostic Errors

Legend: these autopsy rates are those reported from all included studies of autopsy-detected diagnostic errors and thus reflect rates from institutions around the world. Appendix Figure 2 shows US data only and is limited to institutions with more than one time point, so that the general trend toward fewer autopsies is more apparent

NB: This figure does not include the 3 most recent studies,^319–321 though these studies were used in the regression analyses and have been included in several of the other figures and tables.

Appendix Figure 2. Trends in U.S. Autopsy Rates

Legend: Trends derived from the following sources: 54 US Hospitals in Professional Activity Study (PAS)³¹³; aggregate US autopsy rate³²; 79 university hospitals listed in the American Medical Association directory of approved internships and residencies³¹⁴; Yale-New Haven Hospital (New Haven, CT)^{28, 315}; Brigham & Women's Hospital (Boston, MA)^{51, 129}

The table on the following page shows additional data on US autopsy rates.

Addendum to Appendix Figures 2: Trends in Autopsy Rates - Studies from a single time period, but involving more than 2 US institutions

Appendix Figure 3. Class I Error Rates with 95% Confidence Intervals

Legend for Appendix Figures 3 & 4

The numbering of the autopsy series here corresponds to the vertical axis in Appendix Figure 3 and in Appendix Figure 4

Appendix Figure 4 - Major Error Rates with 95% confidence Intervals

Legend to Appendix Figure 4

The numbering of the autopsy series here corresponds to the vertical axis in the Appendix Figure 4

Appendix Figure 5. Class I Error Rates Over Time

Legend: Class I error rates reported in autopsy series included in review without adjustment from variations in autopsy rate or case mix.

NB: This figure does not include the 3 most recent studies,^319–321 though these studies were used in the regression analyses and have been included in several of the other figures and tables.

Appendix Figure 6. Major Error Rates Over Time

Legend: Major error rates reported in autopsy series included in review without adjustment from variations in autopsy rate or case mix.

NB: This figure does not include the 3 most recent studies,^319–321 though these studies were used in the regression analyses and have been included in several of the other figures and tables.

Appendix Figure 7. Class I Error as a Function of Autopsy Rate for 4 Different Time Periods

Legend: Class I error rates derived from regression model over a range of autopsy rates in 4 different time periods with a case mix consisting of general inpatients.

Appendix Figure 8. Major Errors as a Function of Autopsy Rate for 4 Different Time Periods

Appendix Figure 9a. Assessment of Publication Bias: Funnel Plot of All Studies Reporting Class I Errors

Legend: The above scatter plot graphs the observed error rate against sample size for studies reporting class I errors. For studies of therapeutic interventions, an inverted funnel distribution is expected if there is no publication bias, as the reported effect sizes are expected to be symmetrically distributed about the true effect. In this case, the inclusion of studies with different patient populations introduces additional asymmetry (e.g., the data point furthest to the right represents the only study involving nursing home patients). Therefore, we constructed a funnel plot limited to studies reporting autopsies among general inpatients, Appendix Figure 9b).

NB: Figures 9a & 9b and Figures 10a &10b do not include the 3 most recent studies,^319–321 though these studies were used in the regression analyses and have been included in several of the other figures and tables.

Appendix Figure 9b. Assessment of Publication Bias: Funnel Plot Restricted to Studies of Major Errors Among General Inpatients

Appendix Figure 10a. Assessment of Publication Bias: Funnel Plot For All Autopsy Studies Reporting “Major Errors”

Legend: This scatter plot is more symmetric appearing than that in Appendix Figure 9a. However, because the issue of asymmetry due to inclusion of different patient populations still applies, a separate plot involving general inpatient studies only was again created (Appendix Figure 10b).

NB: Figures 9a & 9b & Figures 10a & 10b do not include the 3 most recent studies,^319–321 though these studies were used in the regression analyses and have been included in several of the other figures and tables.

Appendix Figure 10b. Assessment of Publication Bias: Funnel Plot Restricted to Studies of Major Errors Among General Inpatients

Legend: Restricting the funnel plot to studies reporting general inpatients (as shown above), indicates a relative absence of large studies with low error rates and small studies with high error rates. As explained in the text, the present review differs from systematic reviews of therapeutic studies in that publication bias might be operating in opposing directions (as suggested here). On the one hand, studies reporting very low error rates might be considered uninteresting and subject to publication bias in a manner similar to “negative” therapeutic trials. On the other hand, studies reporting high error rates might be self-censored by the institutions themselves. It is even possible that institutions with high error rates are less likely to conduct these types of studies in the first place (e.g., because they have less interest in performance measurement and improvement).

A. Class I Error Rates

I. Time Trend

Coefficient for Year = -0.03041 → -0.3041 to use decade as unit of analysis instead of single year

Odds ratio = exp (–0.3041)= 0.7378

<Subtract above from 1.0 to obtain relative decreases>

→ 0.2622

Class I error rates showed relative decrease of 26.2% per decade (p=0.1044)

95% CI = exp (-0.3041 ± 2*0.1825)

= exp (-0.3041 ± 0.3650)

= exp (0.0609, -0.6691)

= (1.0628,0.5122)

<Subtract above from 1.0 to obtain relative decreases>

→ (-0.0628,0.4878)

→ Class I error rates showed relative decrease of 28.0% (95% CI: 48.8% decrease to 6.3% increase) per decade

II. Relationship to Autopsy Rate

Coefficient Autopsy Rate = -0.8122

(-0.8122/10 = -0.08122 to calculate relationship as per 10% change in Autopsy Rate

Odds ratio = exp (-0.08122) = 0.9220

<Subtract above from 1.0 to obtain relative decrease>

(0.0780)

For every 10% increase in autopsy rate, Class I error rate decreased by 7.8% (p=0.1848)

95% CI = exp (-0.08122 (2*0.6007/10)

= exp (-0.08122 (0.12014)

= exp (-0.20136, 0.03892)

= 0.8176, 1.04

<Subtract above from 1.0 to obtain relative decreases>

(-0.0400, 0.1824)

(Class I error rate exhibited 7.8% relative decrease (95% CI: 18.2% decrease to 4.0% increase) for each 10% increase in autopsies.

III. Calculation of “mean” Class I error rate

Because error rates varied with time and autopsy rate (as well as case mix, though to a lesser extent), a true “mean” error rate does not exist. However, we can estimate a “base error rate,” if the predictor variables are all set to their base values (i.e., time=1980, autopsy rate = mean rate of 44.3%, country=U.S., and case mix=general autopsies).

A point estimate for the base error rate can then be obtained from the regression equation,

Prob (Class I error) = 1 / (1 + e^-λ) where λ = β₀ + β₁ X ₁ + β₂ X ₂ + … + u

Intercept= -2.2622 and, for the base probability, the terms for time, autopsy rate and case mix all equal zero, because the equation is centered on these values.

Therefore,

λ= -2.2622 + (1980-1980)(–0.03041) + (-0.8122)(0.443–0.443) + (-0.09338)(-1), where the –1 in the country term reflects the value assigned to U.S. (non-U.S.=+1).

Thus, λ= -2.16882, so that base prob (Class I error) = 1 / (1 + e^{‐(-2.16882)})

→ Base prob (Class I error) = 0.10258→ 10.2%

Base error and 95% CI obtained from software were: 0.1023 and (0.06701, 0.1532)

Therefore, base rate of Class I errors was: 10.2% (95% CI: 6.7–15.3%)

Because the trends over time and the relationship to autopsy rate were not statistically significant for Class I errors, this base error rate provides a reasonable overall mean rate for Class I errors. It is clear, though (as shown in Table below), that the effects of study period (time) and autopsy rate are noticeable enough (even if not statistically significant for Class I errors). Therefore, a true “mean error rate” is not meaningful in the absence of stipulated values for time and autopsy rate. The table below shows how the Class I error rate varies with time and autopsy rate using the regression model above, with country equal to U.S. and case mix equal to general autopsies.

B. Class I Errors: U.S. only

Coefficient for Time is –0.05501

→ -0.5501 to use decade as unit of analysis instead of single year

odds ratio = exp (–0.5501)= 0.57689

<Subtract above from 1.0 to obtain relative decrease>

→ 0.42311

→ Error rate showed relative decrease of 42.3% per decade, but this relationship was not statistically significant (p=0.07)

Coefficient Autopsy Rate = -1.6629

→ -1.6629/10 = -0.16629 to calculate relationship as per 10% change in Autopsy Rate

odds ratio = exp (-0.166291) = 0.8468

<Subtract above from 1.0 to obtain relative decrease>

→ For every 10% increase in autopsy rate, error rate decreases by approximately 15.3%, but this relationship was not statistically significant (p=0.2).

Prob (Class I error) = 1 / (1 + e^-λ) where λ = β₀ + β₁ X ₁ + β₂ X ₂ + … + u

Intercept= -2.0678

For base probability, time, autopsy and case mix terms all equal zero and there is no country terms in the analysis restricted to U.S. only, so

λ = -2.0678

Therefore,

→ Base prob (Class I error) = 1 / (1 + e^‐(-2.0678))

→ Base prob (Class I error) = 0.1117→ 11.2%

The value calculated using the statistical software corroborated this estimate and provided the corresponding confidence interval.

Thus, the mean Class I error rate using data from U.S. only is 11.2% (95% CI: 6.9–17.5%).

C. Major Errors

I. Time Trend

Coefficient for Time is –0.03288 → -0.3288 to use decade as unit of analysis instead of single year

Odds ratio = exp (–0.3288 ) = 0.719787

<Subtract above from 1.0 to obtain relative decrease>

→ 0.2802

→ Major error rates showed decrease of 28.0% per decade (p=0.0058) relative to the base time period of 1980.

Coefficient Autopsy Rate = -1.2846

10% change → -1.2846/10= -0.12846

Odds ratio = exp (-0.12846) = 0.879449

<Subtract above from 1.0 to obtain relative decrease>

→ 0.120551

For every 10% increase in autopsy rate, major error rate decreases by 12.0% (p=0.0003)

The models dropping case mix, country and autopsy rate (one at a time) produced results with statistically significant differences from the above. Consequently, we used Model 1 (including autopsy rate, country, case mix as predictors of major errors) to compute the mean error rate and the range of error rates shown in the table below and the mean major error rate of 25.6% (95% CI: 20.8–31.2%). The point estimate can be obtained manually using the calculation shown below and was compared with the value generated by the statistical software. (The software was required to calculate the confidence interval.)

Prob (major error) = 1 / (1 + e^-λ) where λ = β₀ + β₁ X ₁ + β₂ X ₂ + … + u

Substituting in Intercept (β₀)= -0.9773

and base values of time (1980), autopsy rate (overall mean rate of 44.3%), country (U.S.=-1) and case mix (general autopsies, so that case mix terms CM2,…,CM7 all equal zero), then

λ = -0.9773 + (–0.02223)(1980-1980) + (-0.9603)(0.443–0.443) + (0.08833)(-1)

λ = -1.06563

→ Base prob (major error) = 1 / (1 + e^{‐(-0.88867)})

→ Base prob (major error) = 0.256235 → 25.6%

Calculation performed with statistical software confirmed this estimate and provided the corresponding confidence interval of (0.2077, 0.3116).

Thus, base probability of major error was: 25.6% (95% CI: 20.8–31.2%)

D. Major Errors: U.S. Studies Only

Coefficient for Time is –0.0246 → -0.246 to use decade as unit of analysis instead of single year

Odds ratio = exp (–0.246)= 0.78192

→ Error rate exhibited relative decrease of 21.8% per decade, but this relationship was not statistically significant (p=0.2).

Coefficient Autopsy Rate = -1.0411

10% change → -1.0411/10 = -0.10411

Odds ratio = exp (-0.10411) = 0.90113

For every 10% increase in autopsy rate, the major error rate decreased by 9.9% (p=0.06)

E. ICD Disease Category Discrepancies

Nb: for the analysis below, time was centered at 1975, rather than 1980, as the study periods ranged from 1970-1980. Also, all of the studies using this error classification involved general inpatients or adult inpatients (Case Mix = 1), so case mix was not included in the models below.

Intercept= -2.0252; Standard error = 0.1024

95% CI= -2.0252 ± 2*0.1024 → -2.230, -1.8204

probability of error = 0.1166 (95% CI: 0.0971 -0.1394)

Coefficient for Time is 0.2468

Odds ratio = exp (0.2468)= 1.2800

→ Error rate increased by roughly 28% per year (p<0.0001).

Coefficient Autopsy Rate = -0.2642

5% change → -0.2644/20 = -0.01321

Odds ratio = exp(-0.01321) = 0.9869

For every 5% increase in autopsy rate, error rate decreases by approximately 1.4%, but this relationship was not statistically significant (p=0.1)

Topics

1. Diagnostic Errors(“Dx errors”)-studies of clinical-autopsy diagnostic discrepancies

   Because of the complexity of this topic, articles tended to be indexed with one or more of the following sub-headings.

   Spec Dz's - studies reporting diagnostic discrepancies for specific diseases or diagnoses.

   Examples

   Bobrowitz ID. Active tuberculosis undiagnosed until autopsy. Am J Med. 1982;72:650–8.

   Zarling EJ, et al. Failure to diagnose acute myocardial infarction: the clinicopathologic experience at a large community hospital. JAMA. 1983;250:1177–81.

   Goldhaber SZ, et al. Factors associated with correct antemortem diagnosis of major pulmonary embolism. Am J Med. 1982;73:822–6.

   Population - studies providing data relevant to clinical diagnostic performance in a general population, not just at among deaths or autopsies (e.g., by providing discharge diagnoses for the same time period, or de facto, by achieving an autopsy rate close to 100%.)

   Examples

   DeRiemer K, et al. The epidemiology of tuberculosis diagnosed after death in San Francisco, 1986-1995. Int J Tuberc Lung Dis. 1999;3(6):488–93.)

   McCarthy BD, et al. Missed diagnoses of acute myocardial infarction in the emergency department: results from a multicenter study. Ann Emerg Med. 1993;22:579–82.

   Flum DR, et al. Has misdiagnosis of appendicitis decreased over time? A population-based analysis. JAMA. 2001;286:1748–53.

   Selection - studies assessing clinical selection bias of cases for autopsy (e.g., by prospectively asking clinicians about their expectation of new information at autopsy

   Example

   Cameron HM, McGoogan E. A prospective study of 1152 hospital autopsies: I. Inaccuracies in death certification. J Pathol 1981; 133: 273–83).

   Predictors - studies reporting factors other than clinical selection (above) and time of study (below) that predict the occurrence of diagnostic errors. Examples include demographics (age, gender, race, socioeconomic status, religion), hospital length of stay (LOS), clinical service (service), clinical diagnosis (Dx), and presence of a DNR order.

   Examples

   Landefeld CS, et al. Diagnostic yield of the autopsy in a university hospital and a community hospital. N Engl J Med 1988; 318: 1249–54.

   McFarlane MJ. Clinical diagnosis is not a source of bias in selection for necropsy. Arch Pathol Lab Med. 1989;113:64–7.

   Setting indicates that diagnostic error rates were compared in different settings (e.g., teaching vs. non-teaching hospital, nursing home vs. hospital)

   Examples

   Landefeld CS, et al. Diagnostic yield of the autopsy in a university hospital and a community hospital. N Engl J Med. 1988;318:1249–54.

   Time indicates that diagnostic error rates were assessed compared in different time periods

   Examples

   Goldman L, et al. The value of the autopsy in three medical eras. N Engl J Med. 1983;308:1000–5.

   Kirch W, Schafii C. Misdiagnosis at a university hospital in 4 medical eras. Medicine (Baltimore). 1996;75:29–40.

   Sonderegger-Iseli K, et al. Diagnostic errors in three medical eras: a necropsy study. Lancet. 2000;355:2027–31.

   Error analysis - studies including some sort of analysis of why misdiagnoses were made (anecdotal discussion of selected cases did not count as sufficient)

   Example

   Middleton K, et al. An autopsy-based study of diagnostic errors in geriatric and nongeriatric adult patients. Arch Intern Med. 1989;149(8):1809–12.

   Dx tests - studies examining the impact of antemortem diagnostic testing on autopsy-detected errors

   Goldman L, et al. The value of the autopsy in three medical eras. N Engl J Med. 1983;308(17):1000–5.

   Kirch W, Schafii C: Misdiagnosis at a university hospital in 4 medical eras. Medicine (Baltimore) 1996; 75: 29–40.

   Complications - studies of the role of the autopsy in detecting complications of care, including many of the general autopsy series that happen to mention complications as a specific type of missed diagnosis, but also other studies focused specifically on this issue.

   Examples

   Ebbesen J, et al. Drug-related deaths in a department of internal medicine. Arch Intern Med. 2001;161:2317–23.

   Gotti EW. Adverse drug reactions and the autopsy. Prevalence and perspective. Arch Pathol. 1974;97:201–4.

   Medicolegal - studies including any information on the impact of autopsy-detected errors on legal actions

   Examples

   Juvin P, et al. Postoperative death and malpractice suits: is autopsy useful? Anesth Analg. 2000;91:344–6.

   Nichols L, et al Are autopsies obsolete? Am J Clin Pathol. 1998;110:210–8.

   Performance - studies containing information on the “test characteristics” of the autopsy itself (e.g., quality of the autopsy, number of cases in which a diagnosis could not be established despite adequate autopsy)

   Examples

   Veress B, et al. The reliability of autopsy diagnostics: inter-observer variation between pathologists, a preliminary report. Qual Assur Health Care. 1993;5:333–7.

   Schned AR, et al A comprehensive quality assessment program on the autopsy service. Am J Clin Pathol. 1986;86:133–8.

   Fowler EF, et al. Evaluation of a teaching hospital necropsy service. J Clin Pathol. 1977;30:575–8.

2. Attitudes - articles describing attitudes towards the autopsy on the parts of patients, their family members, or the general public clinicians (including students, post-graduate trainees and practicing physicians), and pathologists.

   Examples

   McPhee SJ, et al. To redeem them from death. Reactions of family members to autopsy. Am J Med. 1986;80(4):665–71.

   Wilke A, French F. Attitudes toward autopsy refusal by young adults. Psychological Reports. 1990;67(1):81–2.

   Sanner M. A comparison of public attitudes toward autopsy, organ donation, and anatomic dissection. A Swedish survey. JAMA. 1994;271(4):284–8.

   Rosenbaum GE, et al. Autopsy consent practice at US teaching hospitals: results of a national survey. Arch Intern Med. 2000;160(3):374–80.

   Stolman CJ, et al. Attitudes of pediatricians and pediatric residents toward obtaining permission for autopsy. Arch Pediatr Adolesc Med. 1994;148(8):843–7.

   Trelstad RL, et al. The role for regional autopsy centers in the evaluation of covered deaths. Survey of opinions of U.S. and Canadian chairs of pathology and major health insurers in the United States. Arch Pathol Lab Med. 1996;120(8):753–8.

3. Autopsy rates - observational studies of trends in autopsy rates over time or intervention studies attempting to increase the autopsy rate

   Examples

   Cameron HM, McGoogan E, Clarke J, Wilson BA. Trends in hospital necropsy rates: Scotland 1961-74. BMJ. 1977;1:1577–80.

   Sinard JH. Factors affecting autopsy rates, autopsy request rates, and autopsy findings at a large academic medical center. Exp Mol Pathol. 2001;70:333–43.

4. Epidemiology - studies describing the role of the autopsy in tracking the epidemiology of target conditions

   Examples

   Welch HG, Black WC. Using autopsy series to estimate the disease “reservoir” for ductal carcinoma in situ of the breast: how much more breast cancer can we find? Ann Intern Med; 1997. p. 1023–8.

   Rakar S, Sinagra G, Di LA, Poletti A, Bussani R, Silvestri F, et al. Epidemiology of dilated cardiomyopathy. A prospective post-mortem study of 5252 necropsies. The Heart Muscle Disease Study Group. Eur Heart J. 1997;18:117–23.

   McFarlane MJ, Feinstein AR, Wells CK, Chan CK. The ‘epidemiologic necropsy’. Unexpected detections, demographic selections, and changing rates of lung cancer. JAMA. 1987;258:331–8.

5. Death Certificates - studies addressing the correlation between diagnoses on death certificates and autopsies and/or formal chart review

   Examples

   Kircher T, Nelson J, Burdo H. The autopsy as a measure of accuracy of the death certificate. N Engl J Med. 1985;313:1263–9.

   Jordan JM, Bass MJ. Errors in death certificate completion in a teaching hospital. Clin Invest Med. 1993;16:249–55.