Abstract

Background: Human errors that result in rare critical events during the course of routine medical care are especially difficult to study. Although the incidence of severe respiratory depression with routine sedation is small, millions of patients receive this care annually, putting hundreds or thousands of persons at risk. Elimination of sedation errors is particularly important since associated deaths and neurological injuries are virtually 100 percent avoidable.Objectives: The researchers tested the feasibility and validity of using a commercial human simulator to probe care systems for system vulnerabilities (“accidents waiting to happen”).Methods: The study required (1) development and validation of a simulated rare event, (2) use of the standardized event to test care systems in context, and (3) video analysis for deviations in observed care relative to gold-standard care. Scenario: The standard event was reproducible, with physiology that degraded over time if no interventions occurred, and improved when treated appropriately. “Crash-testing” actual care domains: Team performance in managing the simulated event was videotaped and data files of the simulator's physiology variables were captured.Analysis: The quality of each team's performance was assessed using the simulator data files to calculate “time-out-of-range” measures for the critical variables, and team behaviors were analyzed for deviations from idealized care.Results: Available technology supports the creation of a sedation critical event that is realistic, reproducible, and portable. The simulator-based provocative test readily allowed comparison of rescue performance in different sedation care settings (e.g., emergency room, radiology department) to be contrasted with a gold standard.Conclusion: This research supports the feasibility of using available human simulation as a “crash-test dummy,” capable of measuring the rescue systems used in a variety of actual sedation care settings. The findings demonstrate that personnel deemed competent and safe, on the basis of meeting hospital training requirements for airway management, had profound performance deviations when compared to gold-standard practice.

Introduction

Human error and a lack of patient safety are recognized as leading causes of preventable death in America. ¹ As we have noted in previous studies, ² managing the pain and anxiety associated with medical procedures is complex and presents typical patient safety challenges. The pressure to satisfy patient desires for painless, stress-free care for all types of procedures has led to an explosion in the use of more potent sedative medications by nonanesthesiologists. For example, the use of ketamine and propofol by gastroenterologists, emergency medicine physicians, and intensivists has grown significantly in the past several years. ^{3 – 5} The safety of these practice changes have proven difficult, if not impossible, to assess with prospective studies due to the number of patients that would be required.

Although severe respiratory depression is relatively rare, millions of patients are exposed to this risk, resulting in an aggregate morbidity and mortality that is noteworthy. Large, sufficiently powered, multicenter trials to evaluate safety in light of these changing practices simply do not exist. Instead, the literature is replete with descriptions of how sedative medications can be used in a variety of settings on a series of patients (usually fewer than 200 in a cohort) without a fatality. ^{6 – 8} Given the expected incidence of an induced crisis with routine sedation, it is not surprising that these studies rarely uncover a critical event. The authors invariably conclude that the described sedation practice is safe. ⁹ In the absence of critical incidents however, safety cannot be presumed since no data are available to assess the efficacy of rescue processes.

Sedation care delivery systems that lack the ability to manage respiratory depression pose a serious threat to patients. In the mid-1980s, 86 deaths in the United States were attributed to the just-released anxiolytic, midazolam. ¹⁰ Despite the introduction of guidelines for safe sedation care and Joint Commission on Accreditation of Healthcare Organizations (JCAHO) standards, “conscious sedation” for minor procedures remains associated with an alarming number of fatalities. Cote et al. ¹¹ published a critical incident review of 90 pediatric sedation-related critical incidents resulting in death or severe neurological injury. The critical incident analysis determined that the overwhelming majority of these deaths were avoidable and associated with a “failure to rescue” and a failure to provide airway and ventilatory support in a timely fashion. This report has served as a warning that pediatric patients are at especially high risk for injury during sedation.

Our research investigated the feasibility and potential impact of a new methodological approach for measuring patient safety in health care. Pediatric sedation (i.e., the acute treatment of procedure-related pain and stress with medications in children) served as the test domain for this research. We explored the potential to use available human simulation to provocatively test current sedation care settings for safety and rescue capability in an objective manner. A pediatric simulator was programmed to create a realistic sedation event that posed multiple challenges and required rescue interventions. Rescue performance that defined a gold standard was generated by a pediatric anesthesiologist who performs a high volume of pediatric sedations, using standard tools and techniques. The same simulated event was used to “crash test” actual care settings where sedation care is provided. Finally, we used quantitative and qualitative methods of analysis to review videotapes and simulator physiologic data files collected during the exercises to identify care management problems. In summary, this research seeks to test the feasibility and validity of using a human simulator to probe care systems for latent system failures (system vulnerabilities or “accidents waiting to happen”).

Methods

Development and validation of the simulated critical sedation event

The simulator platform used was that of the METI™ (Medical Education Technologies, Inc., Sarasota, FL) pediatric human simulator. The design and validation strategy consisted of (1) defining the expected behavior for the natural evolution of apnea in a child associated with sedative medication overdose, (2) testing of observed simulator behavior, and (3) making software and hardware modifications to the simulation until the observed behavior and the modeled behaviors matched.

Expected behaviors. Three physicians who practice pediatric anesthesia, pediatric intensive care, and pediatric cardiac anesthesia participated in a structured interview to define the onset and cessation of hypoxia, and hypoxic cardiac response associated with drug-induced apnea. Empiric data on typical rates of desaturation (based on lung volumes and functional residual capacity in a normal 5-year-old child), along with expected respiratory mechanics were derived from the available literature. ⁷

Observed simulated event behavior. While an apneic event was initiated on room air and 100 percent O₂, the PaO₂, SaO₂, heart rate (HR), and blood pressure (BP) were captured in a data file. Respiratory mechanics were directly measured using an intensive care unit-(ICU-) grade ventilator (Siemens 2000; Siemens Medical Solutions USA, Inc., Malvern, PA).

Hardware and software modifications. An iterative process was used to align expected behavior with that of the observed behavior. Three levels of complexity for interventions were created, such that success with level 1 would trigger the next level of patient complexity and require more advanced rescue interventions. This approach would allow sedation system performance to be challenged at both low and high levels of task complexity. ⁸

Gold-standard performance. To provide a reference against which to compare the performance of sedation care in the various settings in which it is provided, we assessed rescue performance in managing the simulated scenario and embedded events under “ideal” conditions. An experienced pediatric anesthesiologist (JC), who was not given knowledge of the scenario features in advance, provided sedation and managed the scenario and apneic events using a standard complement of resources that represented the standard-of-care provided by anesthesiologists working in the pediatric procedural sedation unit of our children's hospital. Such resources have been in place for more than 2 years, governing approximately 1,300 sedations per year. This served to define a demonstrably attainable standard of performance, both in terms of the process followed and in terms of how well the critical variables were maintained within their desired ranges.

Test settings. Several settings were selected to test the method. Pilot studies were conducted in the interventional radiology department and the emergency medicine department. Both settings provide pediatric sedation care on a regular basis. When a “code blue” is initiated at Dartmouth Hitchcock Medical Center (DHMC), responders include RNs; flight nurses; critical care nurses from adult, pediatric, and neonatal ICUs; respiratory therapists; anesthesiology residents; physician supervisors of the sedation provider; secretarial support; and administrative support. Code responders were not alerted to the fact the response was a simulation exercise until arrival and direct observation of the simulated patient. The validated scenario was used to test the rescue performance of the Pain-free Pediatric Sedation Unit at the Children's Hospital at Dartmouth (CHaD).

Data collection. Data were obtained from video recordings of each team's performance, and from the simulator output log describing the physiological state of the simulated patient. The video recordings, done with a Sony Digital 8 HandyCam, captured a single wide-angle view of rescue performance by the care team and code responders. The video was recorded from the foot of the patient bed, looking down from a height of 8 feet. Monitors, equipment, clinicians, interventions, and the simulated patient were all visible on the tape. The camera was panned to cover the entire range of activities within the confines of the room in which care was being provided. Audio was captured using the single microphone integrated in the digital video camera.

Quantitative analysis. The simulator's output was used to perform quantitative analyses of the quality of control exercised by the teams over the patient's physiology. Video markers had been developed that represented treatment milestones. Phase I care was based on first responder performance of monitoring and detecting the event, mobilizing rescue resources, and performing basic airway maneuvers of verbal stimulation, jaw lift, and use of an oral airway. Phase II care consisted of advanced airway management with the arrival of providers who are experts in the provision of positive pressure ventilation, the placement of a definitive airway, and the provision of Pediatric Advanced Life Support. Hypoxia and hypotension were defined as SpO₂<60 percent and systolic BP<50 mm Hg, respectively, as these parameters would be associated with negative patient outcomes over time. Time out-of-range for each of these parameters was calculated for the gold-standard exercise, and the exercises within actual sedation care settings. This method of analysis has been previously described for comparison of simulator performance between novices and experts. ¹²

Qualitative analysis. The video recordings were analyzed to produce process traces that described the behavior of each team during the case, and to identify specific care management problems. ¹³ These problems were further analyzed to identify contributing factors using a published taxonomy. ^{14, 15} The goals of this analysis were to identify deviations in performance from the standard-of-care that was proven possible in the gold-standard performance exercise. For each care management problem observed by experts, contributory factors were listed that represented the latent error in the system of care (i.e., the accidents that were waiting to happen under the right triggering conditions). This methodology is currently used by the Dartmouth Hitchcock Department of Anesthesia Quality Assurance Committee to review over 200 critical incidents and/or close call reports. ¹⁶ All three expert reviewers (pediatric anesthesiologists practicing at CHaD) had experience performing critical incident review using the methodology outlined by Vincent et al. ¹⁵ In addition, all three experts reviewed the video-record of the gold-standard case prior to reviewing the video-record of care in the actual sedation care setting. Contrasting observed care against “ideal care” was intended to reduce reviewer variability. ¹⁷ Individual review data were compiled to fill a data table template (Table 1A). The aggregate data were reviewed by the group to confirm face validity. If two of three reviewers felt an identified performance deviation was trivial and unlikely to have the potential for patient harm, it was deleted.

Table 1A

Video markers of gold standard and test sedation Monitors and rescue equipment.

The experts developed a tool for itemizing the equipment and behaviors associated with best practice and time ranges that they agreed were reasonable for categorizing performance as good, intermediate, or poor (Table 1B).

Table 1B

Obstructive and central apnea “behaviors”.

Results

Development and validation of the simulated critical sedation event

Observed simulated apneic event behavior. The METI pediatric simulator was programmed to be a 5-year-old child of the height and weight specified and, after a period of stability, apnea was initiated with an instantaneous 100 percent neuromuscular blockade. The simulator needed to be programmed to have abnormally high oxygen consumption to develop hypoxia, and it was therefore set at the maximum of 2,000 ml per minute. Internal physiological data files captured the development of hypoxia and hemodynamic behavior over time. The same experiment was conducted after having the simulator intubated and on 100 percent oxygen for 5 minutes prior to initiation of the apneic event. In both instances the simulator did not behave in a realistic manner, taking much longer to desaturate than expected.

Hardware and software changes. The current model of the pediatric simulator only used CO₂ insufflation to create hypoxia at the mechanical sensor. Adding a T-connection to the CO₂ inlet line for a nitrogen line that included a calibrated nitrogen flow regulator allowed creation of more realistic behavior. In addition, a scenario was created in the software to better model the expected hemodynamic response to hypoxia described by our experts. ⁹

The measured behavior of the simulator after hardware and software modifications approximated the expected behavior in terms of the onset of hypoxia as a function of the FiO₂ at the time of apnea, and the hemodynamic response to both untreated hypoxia and the response to rescue (Figure 1). The standard event was reproducible, with physiology that degraded over time if no interventions occurred, and improved when treated appropriately (Figure 2). Performance in managing the simulated event was videotaped and data files of the simulator's physiology were captured. The data recordings were graphed and annotated (Figure 3).

Figure 1

Behavior of the modified pediatric simulator. Legend: Experiment in which pediatric simulation (with hardware and software modifications) was allowed to progress after initiation of the apneic event with no therapeutic interventions, followed by full (more...)

Figure 2

Physiological data of simulated sedation scenario managed by experts (ideal care setting).

Figure 3

Simulated sedation emergency (nonideal conditions).

Video and physiological data analysis. Rescue equipment, interventions, and critical task execution were captured on video (Table 1A), which was scored for the initiation and completion times for a subset of these tasks from the onset of obstruction/apnea. Time-out-of-range was then calculated for the critical variables (Table 1B).

Compared to ideal, the sedation unit tested had six deviations in monitoring and three deviations in using rescue equipment. Treatment milestones failed to be met in two of the Phase I tasks (the diagnosis of apnea and use of an oral airway) and four of the Phase II tasks (arrival of the expert airway provider, the expert performing positive pressure ventilation or attempting two-person ventilation, and the time to achieve intubation). Total times to complete tasks were 85 seconds longer (for Phase I) and 281 seconds longer (for Phase II) in the actual-care setting when compared to the gold standard. Both settings avoided the terminal bradycardia and hypotension that will develop if hypoxia is severe and prolonged. Total time with severe hypoxia (SpO₂<60 percent) in the actual care setting was 90 seconds. The scenario did not progress to Level 2 and Level 3 difficulty, due to intubation being performed during the Level 1 event.

Qualitative analysis. Video review by pediatric anesthesia experts identified care management problems and the contributory factors associated with each (Table 2).

Table 2

Qualitative review to identify rescue system problems in an actual sedation unit.

Discussion

Identifying hazards and vulnerabilities in the performance of complex health care delivery processes is a methodologically challenging problem. Error reporting systems provide limited detail and are subject to various reporting biases. ¹⁸ Direct observation has proved useful for gaining detailed insight regarding performance of complex systems. However, techniques that rely on opportunistic observation of actual cases are ill-suited for the investigation of relatively rare events (e.g., respiratory arrests due to oversedation in pediatric patients). ¹⁹

We sought to develop a reproducible method that could be used to assess the systems that provide sedation care in a typical hospital. We used an interactive patient simulator to model a classic oversedation response for a pediatric patient. Instead of bringing care teams into a simulator center as is often done, we designed the scenario as an event running on a portable system that could be brought into actual sedation care settings to observe how the teams in these settings responded to manage the event. This corresponds to what human factors researchers refer to as a “field experiment,” in which certain aspects of a naturally occurring work situation are deliberately manipulated by investigators to permit targeted observations of a specific type of problem-solving scenario. ¹³ This class of investigations is intended to allow observation of behavior under conditions that are highly representative of actual work conditions ^{10 20} Specifically, we wanted to create conditions that would allow us to observe performance using the actual resources (i.e., the personnel, equipment, and procedures) that would be brought to bear on a real patient experiencing the same type of event. In this study, we extended previous work that used a gold-standard comparison with both quantitative and qualitative forms of analysis. ³

Our primary conclusion is that this methodological approach is technically feasible and generates powerful data regarding system performance during the management of rare events. Modifying an available pediatric human simulator allowed scenarios to be produced by the simulator when calibration protocols were followed. The simulator could be moved significant distances within our facility and could be interfaced with the actual devices used by clinicians in the settings in which it was evaluated. Physiologic data files and event logs were easy to synchronize to video, using the time that the sedation provider administered sedative medication as the scenario start time. A single wide-angle-view videotape proved adequate as a data source and allowed for post-hoc analysis by multiple experts. The audio provided by the digital video camera was also adequate for analysis, but the amount of noise made it difficult to follow individual commentary.

This methodological study investigates the use of provocative testing in the sedation care domain as a means for objectively measuring safety. Specifically, we were interested in the measurement of pediatric rescue capability. The method we describe has great potential for achieving this aim. Based on a single sedation care system test, we were able to show that the sedation system currently in use had multiple latent system failures. The rescue efforts deviated significantly from gold-standard care. The actual rescue was associated with more than 3 minutes of hypoxia and 90 seconds of severe hypoxia. The first responders, despite having met training requirements for sedation, did not stimulate the patient when respiratory effort was initially still present. They also were delayed in selecting the correct size of oral airway and inserting it. The STAT paging system failed to mobilize the attending anesthesiologist covering this sedation location. The code team members responsible for advanced airway management assumed that the initial airway management was adequate and failed to assume care to get a positive outcome.

Our observations have been that the skill set associated with positive pressure ventilation is quite complex and that anesthesiologists and respiratory therapists are far more successful at creating a seal and ventilating than are novices, who rarely perform this task. In this case, the experts never tried to provide positive pressure ventilation (a strategic error). Many other latent system failures were identified (Table 2) and these findings proved to be robust and easy to corroborate. For example, subsequent tests of our paging system were associated with the same failure in six of eight tests. This latent system failure has since been corrected.

The qualitative analysis performed used the approach described by Vincent ¹⁵ to identify care management problems and associated contributing factors. These latent failures existed in an actual clinical unit that was providing sedation care. This provocative test uncovered failure opportunities across the full spectrum of resources supporting sedation care—from the blunt-end (system resources) to the sharp-end (provider-interface factors) of the system. ²¹

To date, the METI human simulators have been used primarily as a tool for training. Training applications range from teaching basic skills to high-order crisis resource allocation and team collaboration. ²² This study supports the finding that, even in training exercises, latent system failures can be uncovered while managing a simulated event. DeAnda and Gaba ²³ identified 132 unplanned incidents during 19 simulations using a modified critical incident methodology (range = 3–14 incidents, mean = 6.9 incidents per simulation exercise). In addition, the classes of incidents were similar to those identified previously by Cooper et al. in field studies. ^{24, 25}

In summary, this study demonstrates that using simulation as a safety probe is feasible and of enormous potential impact. In addition to its practical aims, this work seeks to exemplify the type of research that increasingly will be needed in efforts to improve patient safety. The field experiment methodology used here represents one such approach that human factors researchers have employed successfully in a number of domains. These researchers have pointed out ^{13, 26 – 28} that, to make research relevant to practical problems in complex work systems, naturalistic studies are a necessary complement to traditional, controlled experimental research. Future investigations using this methodology will benefit from a more sophisticated scenario and must be validated against empiric data. Additionally, video coding schemes that delineate ideal performance and allow for the easy identification of performance deviations need to be refined. While these investigations are being pursued to advance the efficacy and safety of pediatric procedural sedation, our ultimate goal is to provide a fundamental method for assessing safety in domains in which events are rare, but doomed to happen. Only by designing our health care systems not only to use preventive strategies, but strategies that capture error and support recovery, will safety be maximized.

Conclusion

This research supports the feasibility of using available human simulation as a crash-test dummy, capable of measuring the rescue systems used in actual sedation-care settings. This provocative test readily allowed rescue performance in different sedation-care settings (such as the emergency room and in radiology departments) to be contrasted with the relevant gold-standard performance. We also demonstrated that personnel deemed competent and safe on the basis of meeting our hospital training requirements for airway management had profound performance deviations when compared to gold-standard practice.

Acknowledgments

Funded in part by NIH grant 1 RO3 HD041229-01, National Institute for Child Health and Human Development, and the Burnap Fund for the PainFree Pediatric Simulator.

References

1.

Kohn LT, Corrigan, JM, Donaldson MS, editors. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press; 2000. [PubMed: 25077248]

2.

Blike G, Cravero J, Nelson E. Same patients, same critical events—different systems of care, different outcomes: description of a human factors approach aimed at improving the efficacy and safety of sedation/analgesia care. Qual Manag Health Care. 2001;10(1):17–36. [PubMed: 11702468]

3.

Holloway VJ, Husain HM, Saetta JP, et al. Accident and emergency department led implementation of ketamine sedation in pediatric practice and parental response. J Accid Emerg Med 2000 Jan;17(1):25–8. [PMC free article: PMC1756278] [PubMed: 10658987]

4.

Petrack EM, Christopher NC, Kriwinsky J. Pain management in the emergency department: patterns of analgesic utilization. Pediatrics. 1997;99:711–4. [PubMed: 9113948]

5.

Lowrie L. Weiss AH, Lacombe C. The pediatric sedation unit: a mechanism for pediatric sedation. Pediatrics 1998 Sept;102(3):E30. [PubMed: 9724678]

6.

Havel CJ Jr., Strait RT, Hennes H. A clinical trial of propofol vs midazolam for procedural sedation in a pediatric emergency department. Acad Emerg Med 1999 Oct; 6(10):989–97. [PubMed: 10530656]

7.

McCarty EC, Mencio GA, Walker LA, et al. Ketamine sedation for the reduction of children's fractures in the emergency department. J Bone Joint Surg AM 2000 Jul;82(A):912–8. [PubMed: 10901305]

8.

Egelhoff JC, Ball WS, Koch BL, et al. Safety and efficacy of sedation in children using a structured sedation program. Am J Roentgenol 1997 May;168 (5):1259–62. [PubMed: 9129423]

9.

Heuss LT, Schnieper P, Drewe J. et al. Risk stratification and safe administration of propofol by registered nurses supervised by the gastroenterologist: a prospective observational study of more than 2000 cases. Gastrointest Endosc. 2003;57(6):664–71. [PubMed: 12709694]

10.

Epstein B. Department of Health and Human Services, Office of Epidemiology and Biostatistics, Center for Drug Evaluation and Research. Data Retrieval Unit HFD-737; June 27, 1989.

11.

Cote CJ, Notterman DA, Karl HW. Adverse sedation events in pediatrics: a critical incident analysis of contributing factors [comment] Pediatrics. 2000;105(4 Pt 1):805–14. [PubMed: 10742324]

12.

King PH, Pierce D, Higgins M. et al. A proposed method for the measurement of anesthetist care variability. J Clin Monitor Comp. 2000;16(2):121–5. [PubMed: 12578069]

13.

Woods DD. Process-tracing methods for the study of cognition outside of the experimental psychology laboratory. In: Klein G, Orasanu J, Calderwood R, et al, editors. Decisionmaking in action: models and methods. Norwood, NJ: Ablex; 1993. pp. 228–54.

14.

Vincent C, Taylor-Adams S, Chapman EJ. et al. How to investigate and analyze clinical incidents: clinical risk unit and association of litigation and risk management protocol. BMJ. 2000;320:777–81. [PMC free article: PMC1117773] [PubMed: 10720366]

15.

Vincent C. Patient safety: understanding and responding to adverse events. NEJM 2003 Mar; 348(11):1051–6. [PubMed: 12637617]

16.

Blike GT, Unpublished data regarding quality assurance incident review in the Department of Anesthesia at Dartmouth-Hitchcock Medical Center; 2003 Oct 3.

17.

Roth EM, Bennett K, Woods DD. Human interaction with an “intelligent” machine. Int J Man-Machine Studies. 1987;27:479–525.

18.

Cook RI, Woods DD, Miller C. A tale of two stories: contrasting views of patient safety. Chicago: National Patient Safety Foundation; 1998.

19.

Mackenzie CF, Jeffries NJ, Hunter A. et al. Comparison of self reporting deficiencies in airway management with video analyses of actual performance. Hum Factors. 1996;38(4):623–35. [PubMed: 8976625]

20.

Brunswik E. Perception and the representative design of psychological experiments. 2^nd edition. Berkeley, CA: University of California Press; 1956.

21.

Cook R, Woods D. Operating at the sharp end: the complexity of human error. Chapter 13, In: Bogner S, editor. Human error in medicine. Mahweh, NJ: Lawrence Erlbaum Associates; 1994.

22.

Gaba D, Fish K, Howard S. Crisis management in anesthesiology. Philadelphia: Churchill Livingstone; 1994.

23.

DeAnda A, Gaba D. Unplanned incidents during comprehensive anesthesia simulation. Anesthesia & Analgesia. 1990 Jul;71(1):77–82. [PubMed: 2363534]

24.

Cooper JB, Newbower RS, Long CD. et al. Preventable anesthetic mishaps: a study of human factors. Anesthesiology. 1978;49:399–406. [PubMed: 727541]

25.

Cooper JB, Newbower RS, Kitz RJ. An analysis of major errors and equipment failures in anesthesia management: considerations for prevention and detection. Anesthesiology. 1984;60:34–42. [PubMed: 6691595]

26.

Perrow C. Normal accidents: living with high-risk technologies. New York: Basic Books, 1984.

27.

Reason J. Human error. New York: Cambridge University Press; 1990.

28.

Hoffman RR, Woods DD. Studying cognitive systems in context: preface to the special issue. Hum Factors. 2000;42(1):1. [PubMed: 10917142]

Footnotes

⁷ Data from pediatric anesthesia experts and parameters used to depict pulmonary function for a 5-year-old, 20 kg child, 112 cm tall, is available at:http://an​.hitchcock.org/ahrq/Blike2005​.pdf.

⁸ The scenario code is available at: http://an​.hitchcock.org/ahrq/Blike2005​.pdf.

⁹ A detailed description of these modifications is available at: http://an​.hitchcock.org/ahrq/Blike2005​.pdf.

¹⁰ We prefer this more precise term to the commonly used concept of “ecological validity” (cf. Hammond, 1998).