 |
|
|
In To Err Is Human, the Institute of Medicine (IOM) recognized that error-reporting systems had dual, but simultaneously difficult functions of increasing accountability and providing information that may lead to safer health care.1 In IOM's estimation, mandatory reporting systems lend themselves to the former, and voluntary reporting systems to the latter. The IOM recommended funding and support of voluntary reporting efforts, specifically recognizing the need for confidentiality provisions, reporter feedback mechanisms, and improved understanding of how reported information is used. Leaders of the patient safety movement agree that voluntary reporting systems can measurably improve safety if reporting is protected against discovery and provides reporters with useful information from expert analysis.2 They also suggest that the complexity and expense of running a national voluntary error-reporting system would be overwhelming, and that specialty-based or system-specific reporting programs are more feasible.
Researchers at the Robert Graham Center, a research center and division of the American Academy of Family Physicians (AAFP), seized upon To Err Is Human and its recommendations for voluntary reporting. The Graham Center staff felt compelled to turn its research focus to patient safety, due to the scarcity of information about threats to patient safety in outpatient settings, where most Americans receive most health care.3 It was decided that an early pilot study was needed to see what threats to patient safety family and general physicians would recognize and report, and whether it made a difference if they reported by a Web- or paper-based system. This initial study, launched in 2000, led to three subsequent studies within the AAFP National Network for Family Practice and Primary Care Research (National Network) and with collaborators in six other countries. These studies have helped refine a specialty-based, Web-enabled error-reporting tool and have demonstrated its utility for identifying threats to the safety of patients who enter the primary care system.4 This paper is about this error-reporting tool—its evolution, the data it yields, reporters' experience with it, ways that it could contribute to other reporting programs, the legal and practical barriers to its full utilization, and the potential for its role in a broader patient safety agenda if barriers are lifted.
Family practice is complex, characterized by customized care that responds to individual patients' needs, values, and preferences across a broad spectrum of health and illness concerns.5–7 Its diversity, scope, and variation in structure and infrastructure may offer more opportunity for error than hospital-based care, which tends to be more highly regulated and procedure oriented. Making primary care safer for patients is complicated by a lack of understanding of the nature and distribution of errors that occur in this setting and that are amenable to systematic change. As well, many primary care physicians are only tenuously connected to other providers and may not be recognized as part of some health care delivery systems.8 Malpractice data reveal that most error-related claims for primary care physicians come from care delivered outside of hospitals, and suggest that seemingly trivial errors can severely injure and kill people.9
Prior to our own research, only two studies had explored the epidemiology of errors in primary care. One, an Australian study, used incident reports to describe events that resulted, or could have resulted, in harm to patients.10 These events were not necessarily errors. The other study reviewed a risk management database, applying a taxonomy developed from review of hospital records.11 The growing body of evidence, and the IOM patient safety report series, revealed that the health system could cause substantial harm, alerting the U.S. public and politicians of the need to protect people from safety threats. Primary care providers, including family physicians, had so far largely been excused from engaging in discussions about medical errors. While many important initiatives in hospitals offered hope for greater patient safety, other equally important opportunities were being overlooked because they existed outside of hospitals.
In developing the AAFP Patient Safety Reports, we attempted to design in the key characteristics of voluntary error-reporting systems proposed by safety experts to the degree possible.2, 12 These key characteristics are that the reporting systems be nonpunitive and undiscoverable, confidential, timely and responsive, and easy to use. As recommended by the IOM, we designed a reporting system that could support causal analysis, including free-text descriptions of error events, guided by a standard set of questions.13 Also in keeping with recommendations, we developed a core group of independent peer experts to analyze the error reports.
The AAFP error-reporting system and the International Taxonomy of Medical Errors in Primary Care have been developed and revised based on four research studies carried out in family physicians' offices between 2000 and 2004.
Our first study, completed in January 2001, taught us that physicians recognized many errors and were willing to report them, even errors with serious adverse patient outcomes.3 Fifty physicians of the AAFP National Research Network were randomly assigned to either arm of a crossover trial to test whether there were differences in reporting by paper or computer. This study was an early foray into describing the spectrum of errors recognized by physicians in primary care. The paper system included a pocket-sized paper card with prompts to report data items, and the computer reporting system had the same data prompts. Errors were defined as events in physicians' practices that should not have happened and were not anticipated. Physicians were encouraged to identify an incident “that should not happen in my practice and I don't want it to happen again.” These could be administrative or clinical errors. They could be adverse events (events resulting in unintended harm) or near misses (events that had the potential to cause harm). The main outcome measures were error category, error rate, and error consequence.
| Error types | AAFP 1st study (n = 330) % error reports | International (n = 429) % error reports | AAFP 2nd study* (n = 838) % error codes |
|---|---|---|---|
| Process errors | 86 | 79 | 96 |
| Office administration | 31 | 19 | 51 |
| Filing system | 12 | 9 | |
| Chart complete/available | 8 | 19 | |
| Patient flow | 2 | 6 | |
| Message handling | 4 | 3 | |
| Appointments | 3 | 12 | |
| Investigations | 25 | 17 | 13 |
| Treatments | 23 | 26 | 16 |
| Medications | 17 | 14 | |
| Communications | 6 | 14 | 13 |
| Payment | 1 | 1 | 3 |
| Workforce management | 2 | 1 | |
| Knowledge and skills | 14 | 21 | 4 |
More than one error could be coded for each report in this study.
There were few legal precedents regarding discoverability of physician error reports in 1999. We chose to use ethical review and approval by an institutional review committee as the standard for protecting participating physicians. In the research design, we used many safeguards to protect the identity of the reporting physicians. We destroyed any documentation that could link error reports to individual physician reporters. Documents naming physician participants were meticulously shredded and computer files destroyed, leaving only data related to the reported error. These protections made it impossible to do root-cause analysis or to provide individualized feedback. Individual reports were never made publicly available, and publications from this study include only aggregate analyses of reports.3, 13 On occasion, for training purposes, example cases combining the characteristics of two or more reports were discussed outside of the research team. Despite efforts to guarantee anonymity of the reporting physicians, however, some participants in this study are identifiable. Before publishing the paper reporting the study results, we asked participants if they wished to be named in the “Acknowledgments” section of the paper, and many elected to be named—choosing to make their participation in studying patient safety a matter of public record.
The first AAFP study led to a Commonwealth Fund-supported study of errors reported by physicians in seven countries: the United States, Canada, New Zealand, England, the Netherlands, Australia, and Germany.14 The computerized error-reporting system was tweaked to improve its function and no paper reports were made, because the first study provided reassurance that the types of errors reported by paper and computer would probably be similar. Reports remained anonymous because discoverability was not protected in every country.
This Primary Care International Study of Medical Errors (PCISME) spawned the LINNAEUS (Learning in an International Network About Errors and Understanding Safety) Collaboration, which continues to work together and its members have become recognized patient safety research experts in their own countries.15 With the PCISME study, the computerized error-reporting system was tested around the world in English and German and yielded important information about medical errors in primary care.14
This study raised new legal and practical issues. Data were reported in an electronic system developed in conjunction with a British software developer, who leased server space from a British Government agency for the central data repository. During report processing, however, the physical server holding study data at any particular time could be in any of a number of locations throughout the world. It was not the data themselves, but access to the data that was at issue. This is an important issue that may deserve further consideration by experts in international law: What access might third parties (such as malpractice lawyers) have to error report data that are not held in their own country—including data from reporters in other countries, operating under different legal codes? Some of the countries involved in our study have bilateral national agreements, but others do not. There is a body of legal precedent, for example, of American legal codes failing to have effect in New Zealand. And what is to be done about accessing data that have no known country of “residence” but exist only “virtually,” as electronic data might? International studies and even international error-reporting systems are likely to be characterized by a unique set of legal issues that have so far not been addressed. Currently, most efforts are aimed at establishing legal frameworks for reporting systems within nations, and not among them.
After the first two studies, the AAFP received support for a Developmental Center of Excellence for Research in Patient Safety (DCERPS) from the Agency for Healthcare Research and Quality (AHRQ) (Grant Number 5 P20 HSO1 1584) and a supplemental patient safety research grant (AHRQ Grant Number 5 R21 HSO 13554). These grants were an important next step for AAFP's patient safety agenda and competencies. Among other things, they permitted formalization of the error-reporting system as an in-house, Web-based process.
Federal funding carries legal protections from discoverability. However, because these legal protections had never been tested, we maintained the anonymity of reporters. We did so over protests from our reporting physicians, who would have preferred receiving specific feedback. We assured anonymity by using reporter codes specific only to reporter types, not individuals, and by removing “cookies” and other report evidence from reporters' computers with each submission. To enhance data security and the protection of reporters, the AAFP invested in more robust firewalls within the error-report server, and separated report data from the reporting process server the moment reports were submitted. The AAFP funded a formal, external test of the system's security before implementing the studies.
Several of the most severe outcomes resulted from seemingly innocuous and common communication errors. Most notable, the mishandling of messages from patients was associated with two deaths and a fetal demise.2
The supplemental AHRQ grant to the DCERPS supported a study of testing processes (laboratory, radiology) that included incident reporting, as well as focus groups and human systems engineering clinic evaluations. The study focus included all steps involved in ordering, receiving, and responding to patient investigations. The aim was to describe testing process errors recognized by family physicians and their office staffs, to investigate causes and consequences of these errors, and to generate hypotheses about interventions that might avert testing process errors. Eight practices were to report these errors over a 48-week period in 2003. This was to be followed by qualitative focus groups in each practice, and human systems engineer evaluations in two of the practices.
Participants submitted 661 testing process errors, but the study ended early, in week 32, because reporting significantly tapered off. The error reports have not yet been coded and analyzed.
Focus groups conducted as part of this research revealed that participants found it easy to make reports, usually taking only 1–3 minutes. Major barriers to reporting were remembering to report and other time commitments, such as paperwork and clinical responsibilities. Many of the focus group participants reported that participating in the study led to increased group awareness of safety. While participants identified the need for major system changes—typically electronic health records, more staff, and staff training—they also felt that reporting led to more immediate beneficial changes than paper-based tracking systems. The latter included increased awareness of errors and increased diligence. One participant commented, “The study has made me painfully aware of the fact that we don't have a system and secondly, how difficult it is to set up a system.” Many of the practices described practicewide changes they made, both small and large, as a result of participating in the study. One practice described how it changed from “writing everything down on little pieces of paper and sticking them on charts, to e-mailing back and forth on lab results.”
These studies have produced a Web-based error-reporting tool that contributes to our understanding of threats to patient safety in primary care. It is not the only method we have employed for this purpose, but it remains an important and evolving part of the AAFP patient safety research program. Future versions of the reporting system will emphasize data collected by pulldown menus and checked boxes, but some continued free-text component is essential.
The latter two DCERPS studies have developed the error-coding capacity of the research team and made it clear that timely analysis will require some level of automation, which is another essential focus of research. Our coding of report data has most recently turned from granularity—that is, using reports to finitely describe the taxonomy of errors—to functional aggregations of error types and causes. The goal is a taxonomy that can guide interventions and make sense of the cascades of errors that produce adverse outcomes and near misses. We are also leading an international conference on primary care patient safety taxonomies to better understand their functional differences, the knowledge they produce, and how they may be cross-walked to each other and to higher-level classifications. Revision of our primary care error taxonomy will likely lead to reporting-tool modifications. For a variety of reasons explained in the next section, the Web-based reporting tool will remain a research tool. Our next studies will focus on expanding the Web-based process to add a feedback function for reporters and embedding these in a Web-based quality-improvement collaborative.
| Reporting system characteristics | Barriers | Results | Future |
|---|---|---|---|
| Nonpunitive and undiscoverable |
|
|
|
| Confidential |
|
|
|
| Independent report collection and analysis by peer experts |
|
|
|
| Timely and responsive |
|
|
|
| Ease of reporting |
|
|
|
| 1. Creating an environment for safety. There should be a nonpunitive culture for reporting health care errors that focuses on preventing and correcting systems failures and not on individual or organization culpability. |
| 2. Data analysis. Information submitted to reporting systems must be comprehensively analyzed to identify actions that would minimize the risk that reported events recur. |
| 3. Confidentiality. Confidentiality protections for patients, health care professionals, and health care organizations are essential to the ability of any reporting system to learn about errors and effect their reduction. |
| 4. Information sharing. Reporting systems should facilitate the sharing of patient safety information among health care organizations and foster confidential collaboration with other health care reporting systems. |
5. Legal status of reporting system information. The absence of Federal
protection for information submitted to patient safety reporting systems discourages the
use of such systems, which reduces the opportunity to identify trends and implement
corrective measures. Information developed in connection with reporting systems should be
privileged for purposes of Federal and State judicial proceedings in civil matters, and
for purposes of Federal and State administrative proceedings, including discovery,
subpoenas, testimony, or any other form of disclosure.
|
The AAFP has invested significant time and resources in obtaining expert advice about the limitations of Federal protections, about other discoverability risks, and in maintaining state-of-the-art data protection. However, the AAFP cannot expose family physicians and their offices to the risks of nonanonymous error reporting until Federal protections are in place. The IOM recognized the tension between the public's demand for transparency and the health system's insistence on confidentiality and legal protection for reported errors.13 The IOM came down on the side of protecting reports out of concern that discoverability and fear of retaliation would be major impediments to effective reporting. It called for enhanced protection in order to establish voluntary reporting systems and the development of a national patient safety database of de-identified data. It considered this protection essential for the integrity and effectiveness of patient safety learning systems.
Even if Federal legislation clears these hurdles, the practical barriers to a national reporting system could be enormous. Leape estimates that such a system could attract reports of 0.5 to 5 million near misses annually.2 This would be 15 to 50 times the number of reports to the ASRS; and if per-case report review costs approach that of the ASRS (about $70 per case in 2002), the total cost would be staggering. The United States has a few national reporting systems that are limited in their scope,18 and Australia, England, and the Canadian province of Saskatchewan have or have announced plans to introduce national or provincial error reporting systems.19, 20 If the legal and practical barriers to national voluntary error-reporting systems in the United States prove insurmountable, system-specific or specialty-based reporting systems are potential solutions.2 These could feed information into a national database.
The AAFP has identified some other practical barriers to sponsoring a national patient safety reporting system. Our team has become adept and increasingly reliable in its coding of reports, but coding remains sufficiently tedious that opening the Web-based reporting system to the general membership of the AAFP would be impossible while using the current manual coding mechanisms. Most coding to date has focused on the primary error being reported. But some of the work done by us and others suggests that more attention to cascades of errors and the most proximal of identifiable errors may be more important for identifying potential remedies.21, 22 Cascade and root-cause analyses will require considerably more time, a bigger coding team, and the ability to query reporters. Our desire, shared by many of our study participants, is to have a feedback mechanism that offers evidence-based solutions to commonly reported errors.
A final practical issue for the near future is the need for specialized expertise in the storage and retrieval of large datasets.23 Medicine has appropriately reached outside of itself to draw on the skills of human systems engineers and aviation safety experts to improve reporting systems and how we respond to the errors they identify. As future users of primary care medical error-reporting data seek to mine them for patterns, or to analyze them with data from other settings or systems, complex data must be organized to be easily retrievable. Expertise in data management and organization may reduce these hurdles for the future, and may actually ensure that reports to one system (or specialty) can flow to other databases appropriately.23
Nearly 75 percent of family physicians belong to the AAFP and look to it as a trusted entity for improving the quality of the care they provide. Our studies suggest that members would make error reports to the AAFP, and when they did, they would want specific feedback to improve their performance. The AAFP could link reports and their analyses to a wider array of its existing programs to improve quality and safety. As a formal PSO, the AAFP could analyze reports, provide feedback to reporting practices, and provide tools for preventing similar errors, allowing participating practices to monitor their progress relative to peer-practices and to share relevant reports with other safety organizations. The AAFP could tailor its existing educational arm to reflect report findings, produce quality improvement publications and “toolboxes,” e-mail “patient safety cases of the week” to its membership, and provide learning collaboratives. Physician participation would be linked to professional development, licensing requirements, and maintenance of certification. The AAFP's investments and vision expose a desire to combine its trusted relationship with members, its well-established educational infrastructure, and its new capacities to position itself as a catalyst for improving quality in primary care. Other medical professional organizations could similarly support their members.
The AAFP error-reporting system would become a part of routine research and learning. If the legal and practical barriers can be overcome, the AAFP could join Federal and specialty partners, health systems, and other countries in doing patient safety research. The international LINNAEUS collaboration and the AAFP's leadership in exploring how its error taxonomy links to other patient safety classifications hint at this future. Outside of research, sharing report data across systems and borders could facilitate more robust learning. Medication error reports from primary care, for example, could be repackaged and shared with the Food and Drug Administration and other medication error-reporting processes.
Over the past 5 years, the American Academy of Family Physicians has developed and tested a Web-based, voluntary error-reporting system through a series of domestic and international studies. This research effort has produced an error-reporting process that is accepted and used by physicians and clinic staff. The reporting system feeds into a robust classification process that is revealing where things go wrong in primary care, how errors differ from inpatient settings, and potential ways to make care safer. Patient safety data are a critical input to the efforts of providers to redesign care processes in ways that will make health care safer for all patients.1
Our qualitative data reveal that participation in reporting errors can induce physicians and their staffs to modify error-prone systems in their practices. Currently, a chasm of legal and practical problems exists between the collection of error reports and providing constructive feedback to reporting providers to assist them in improving the quality and safety of their offices. These gaps require more study. Our research will move from anonymous reports to confidential reports under the legal protections afforded by Federal grant funding, so that we may begin testing direct feedback to reporters, doing root-cause analysis, and engaging the other education and dissemination programs of the AAFP. Whether this Web-based error-reporting system evolves beyond research, and whether the AAFP becomes a full-fledged change agent for primary care practice, depends on bridging the legal and practical chasms identified by the IOM and others.
We sincerely appreciate the help of Tom Robinett, J.D., general counsel of the American Academy of Family Physicians, for his help with the legal issues around discovery; and the considerable assistance of Lisa Klein in preparing this document.
Disclaimer: The information and opinions contained in research from the Robert Graham Center do not necessarily reflect the views or policy of the American Academy of Family Physicians.
Robert Graham Center: Policy Studies in Family Medicine and Primary Care, Washington, DC (RLP). American Academy of Family Physicians (DG, MJ). University of Otago, New Zealand (SMD). University of Chicago (JH).
Address correspondence to: Robert L. Phillips, Jr., M.D., M.S.P.H., The Robert Graham Center, 1350 Connecticut Avenue, NW, Suite 201, Washington, DC 20036; phone: 202-331-3361; fax: 202-331-3374; e-mail: bphillips@aafp.org.
Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Two patients committed suicide after calling to say they were very depressed and considering suicide; both times their messages were not directly delivered to the physicians but simply placed on the doctors' desks, where they were found too late. The fetal demise occurred when a term-pregnant patient who called about not feeling her baby move was not appropriately triaged by a junior nurse covering the phones over lunch, and the message of the patient's phone call was placed in a stack of charts. The patient then sought help at an urgent care center, where she was forced to wait, only to be referred back to her own physician without evaluation. By the time the patient finally reached her physician's office, it was closed. She went to the emergency room, where she received the tragic news.
