Purpose

The ESP Coordinating Center (ESP CC) is responding to a request from the Department of Veterans Affairs (VA) National Center for Patient Safety for a rapid evidence review on implementing High Reliability Organization (HRO) principles into practice. The purpose of this review is to evaluate the literature on frameworks, metrics, and evidence of effects of HRO implementation. Findings from this review will be used to inform the implementation of the VA’s HRO Initiative.

Background

In their 2000 report “To Err is Human,” the Institute of Medicine’s (IOM) Committee on Quality of Health Care in America cited deaths due to medical errors as more common than those due to motor vehicle accidents, breast cancer, or AIDS.¹ Despite continued widespread, discrete process improvement initiatives such as handwashing protocols, patient identification to reduce ‘wrong person’ procedures, protocols for clear communications between care teams and visual indicators for high risks such as fall injury or allergies, a 2016 British Medical Journal report estimated that medical errors continue to be the third leading cause of death in the US.² Additionally, the IOM Committee identified care fragmentation as a root cause of medical errors.¹ In response, they called for a comprehensive, system-level approach to improve patient safety, that shifts the focus away from a culture of blame to one of error analysis and process improvement. Therefore, health care organizations have begun to explore system-level approaches to cultivating a culture of safety, with a focus on collaboration, communication, and coordination.

AN HRO IS:

“An organization that experiences fewer than anticipated accidents or events of harm, despite operating in highly complex, high-risk environments.”

HRO is one such organizational approach to achieving safety, quality, and efficiency goals.^3,4 At the core of HRO is a culture of “‘collective mindfulness’, in which all workers look for, and report, small problems or unsafe conditions before they pose a substantial risk to the organization and when they are easy to fix.”^3,5 Use of HRO is designed to change the thinking about patient safety through the following 5 principles: (1) sensitivity to operations (ie, heightened awareness of the state of relevant systems and processes); (2) reluctance to simplify (ie, the acceptance that work is complex, with the potential to fail in new and unexpected ways); (3) preoccupation with failure (ie, to view near misses as opportunities to improve rather than proof of success); (4) deference to expertise (ie, to value insights from staff with the most pertinent safety knowledge over those with greater seniority); (5) resilience (ie, to prioritize emergency training for many unlikely but possible system failures).⁴ See Figure 1 below.

Figure 1

Five HRO principles.

HRO was originally pioneered in extremely hazardous industries, such as nuclear power and commercial aviation, where even the smallest of errors can lead to tragic results. These industries have achieved and sustained extraordinary safety levels, thereby generating much interest in how to adapt HRO principles to health care and replicate this success. In their 2007 book “Managing the Unexpected,” Weick and Sutcliffe define the 5 principles of HROs and describe how these principles can be applied to improve reliability across diverse industries.⁵ In their 2008 seminal Agency for Healthcare Research and Quality (AHRQ) white paper, Hines et al apply these 5 principles to health care settings and describe the specific challenges threatening health care reliability, such as higher workforce mobility and care of patients rather than machines.⁴ Implementation of HRO initiatives into health care settings is an inherently complex and costly process that involves organizing people, processes, and resource activities across often large organizations. For example, the Nationwide Children’s Hospital’s HRO journey involved increasing their quality improvement (QI) personnel from 8 in 2007 to 33 in 2012, with a budget increase from $690K to $3.3M.⁶ External consultants, such as Healthcare Performance Improvement, LLC, can provide support to organizations undertaking an HRO journey. HRO interventions commonly include activities like basic error prevention education; leadership training in reinforcement approaches; enhanced root cause analysis processes using an electronic tracking system; promotion of a ‘just culture’ – a culture in which providers and staff are fairly penalized for mistakes – that supports routine reporting errors; sharing good catches and lessons learned; and training in error prevention technique by provider peer safety coaches.

Examples of health systems’ successful adoption of HRO principles are already emerging. Providence St. Joseph Health – a national, not-for-profit Catholic health system comprised of more than 50 hospitals, 800 clinics and 5 million patients across 7 states – has had success implementing their HRO program, Caring Reliably. Two years after implementation of the program, which included partnering with an outside consulting firm to coach them through a leader toolkit, which focused on culture, and a toolkit for everyone, which reduced errors, Providence St. Joseph Health experienced a 5% improvement in the safety climate domain of the Safety Attitudes Questionnaire and a 52% decrease in serious safety events (G. Battey, oral communication, February 2019).⁷ The VA has also experienced HRO implementation successes. The Harry S. Truman Memorial Veterans Hospital began a 3-year HRO project in March 2015 by partnering with the VA National Center for Patient Safety to deliver Clinical Team Training to every inpatient and outpatient clinical service. This included formal interactive classroom training, application of the principles in a project that was unique for each clinical area, and refresher classroom and simulation training after one year. In May 2016, Truman VA augmented their HRO program using a 23-module HRO Toolkit provided by VISN 15, as part of its HRO initiative rolled out across all 7 of its medical centers. According to Truman VA Associate Director Robert Ritter (R. Ritter, oral communication, February 2019), their HRO program has already resulted in remarkable improvements in staff attitudes and perceptions and significant increased participation in morning multidisciplinary huddles. However, despite the promise of increased excellence as described in the 2013 Joint Commission’s HRO report,³ major barriers to widespread implementation readiness of HRO at the VA and elsewhere include the complexity of organization-wide incorporation of safety culture principles and practices and prioritizing the adoption of process improvement tools and methods, among other competing priorities.

To reaffirm their commitment to high reliability and zero harm (working to “reduce errors and to ensure that any errors that may occur do not reach our patients and cause harm”), in February 2019, the VA rolled out a new initiative outlining the definitive steps for becoming an HRO.⁸ The first step is for HRO activities to begin at 18 lead facilities selected based on greater readiness as demonstrated by higher levels of safety performance, leadership commitment, and staff engagement. Initial HRO activities include the establishment of work groups, performance readiness assessments, and conducting training events and programs. Following analysis of lessons learned from these lead sites, the VA plans a national roll-out to achieve the goal of an VA-wide HRO transformation. To ensure success of HRO-related activities and consistent outcomes across the enterprise, VA is using resources from the Joint Commission Center for Transforming Healthcare resource library, including the Oro 2.0 High Reliability Assessment tool. Additionally, VA is working on developing a standard set of HRO tools, including training, implementation models, and measures.

Emerging literature can guide health systems in implementing and evaluating their HRO journey.^9,10 However, an understanding of available frameworks, metrics, and initiatives and their use are currently limited by their complexity and wide variability of their key characteristics, their target participants (eg, leadership, medical staff), their foundation, their structure, which of the 5 HRO principles they address, and health system setting type. Understanding the quality and applicability of existing HRO resources is important to developing best practices, identifying barriers and facilitators to implementation, spreading implementation initiatives to other systems, measuring progress, and identifying knowledge gaps.

Scope

This rapid evidence review will address the following key questions and eligibility criteria:

Key Questions

View in own window

	Key Question 1. What are the frameworks for guiding HRO implementation? Key Question 1A. What are the main implementation strategies of these frameworks? Key Question 1B. What were the processes for developing these frameworks (eg, consensus, literature review, etc)? Key Question 1C. What are the intended settings of these frameworks? Key Question 1D. Who participates in implementing these frameworks? Key Question 1E. What are the processes for implementing these frameworks?
	Key Question 2. What are the metrics for measuring a health system’s progress towards becoming an HRO? Key Question 2A. What are the main characteristics (ie, domains, scales) of these metrics? Key Question 2B. What were the processes for developing these metrics (eg, consensus, literature review, etc)? Key Question 2C. To what extent have these metrics been validated or used to inform health system decision-making?
	Key Question 3. What is the evidence on HRO implementation effects? Key Question 3A. On patient safety/organizational change goals (eg, number of sites that met goal of 50% reduction in serious safety events)? Key Question 3B. On patient safety/organizational change measures (eg, mean change in number of serious safety events)? Key Question 3C. On process measures (eg, mean change in inter-departmental communication, provider or patient satisfaction)?

Eligibility Criteria

The ESP included articles published from January 2010 to January 2019 that describe implementation frameworks, metrics for measuring progress towards becoming an HRO, and its effects. The timeframe of 2010 and onward was selected because it is 2 years after the publication of AHRQ’s 2008 white paper, when one could reasonably expect publication of new research on implementing HRO principles in health care settings. To be included, articles needed to be explicitly grounded in HRO theory and specifically seek to advance organizational or cultural change. We operationalized this by only including articles that evaluated HRO principles at the organization level or higher (ie, we excluded articles of HRO implementation in individual departments). Outcomes for KQ3 include any that are linked to the pathway between the 5 principles of HROs (ie, sensitivity to operations, reluctance to simplify, preoccupation with failure, deference to experience, and resilience) and the ultimate goal of health care organizations: exceptionally safe, consistently high-quality care, as outlined in the AHRQ white paper.⁴ See Figure 2 below for the logic model linking the 5 HRO principles to the end goal of improved patient safety outcomes, based on the model described in Hines 2008.⁴

Figure 2

HRO logic model.

We prioritized articles using a best-evidence approach to accommodate the timeline (ie, we considered meeting safety goals [KQ3A] to be a higher priority than intermediate outcomes [KQ3B and KQ3C]). We also prioritized evidence from systematic reviews and multisite comparative studies that adequately controlled for potential patient-, provider-, and system-level confounding factors. We only accepted inferior study designs (eg, single-site, inadequate control for confounding, noncomparative) to fill gaps in higher-level evidence.

Literature Flow

The literature flow diagram (Figure 3) summarizes the results of search and study selection (see Supplemental Materials Appendix B for full list of excluded studies). Our search identified 525 unique, potentially relevant articles. Of these, we included 20 articles that addressed one or more of our key questions. Eight articles addressed Key Question 1,^3,15–21 8 articles addressed Key Question 2,^3,16,22–27 and 7 articles addressed Key Question 3.^6,17,28–32

Figure 3

Literature flowchart.

KEY QUESTION 1. What are the frameworks for guiding HRO implementation?

We identified 8 frameworks that guide implementation of HRO principles into a health care system: the Joint Commission’s High Reliability Health Care Maturity Model (HRHCM)³; the Institute for Healthcare Improvement’s (IHI) Framework for Safe, Reliable and Effective Care¹⁸; the American College of Healthcare Executives’ (ACHE) Culture of Safety framework¹⁶; 2 frameworks developed at Johns Hopkins’ (JH) Armstrong Institute for Patient Safety and Quality including an Operating Management System¹⁷ and a Safety and Quality framework¹⁵; the Office of the Air Force Surgeon General’s Trusted Care framework¹⁹; Advancing Research and Clinical Practice through close Collaboration (ARCC) Model²⁰; and a framework focused on developing high reliability teams.²¹ The Joint Commission’s HRHCM and IHI Framework for Safe, Reliable and Effective Care emerged as the most comprehensive, as they both covered all 5 strategies commonly reported across frameworks (Figure 4); were the most rigorously developed; were broadly applicable; and were sufficiently detailed to inform implementation.

Figure 4

5 Common HRO implementation strategies.

Appendix C of the supplementary materials contains full details on these frameworks’ implementation strategies, development process, and intended settings, as well as staff and processes required for implementation. Highlighted findings appear below.

KQ1A. What are the main implementation strategies of these frameworks?

Table 2 summarizes the 5 commonly reported key HRO implementation strategies from these 8 frameworks.

Table 2

Common HRO implementation strategies across 8 identified frameworks.

The first key strategy is developing leadership. The Joint Commission discussed the need for leadership (eg, board members, CEO/management, and lead physicians) to commit to the goal of zero patient harm.³ IHI described the need for leaders to facilitate and mentor teamwork, improvement, respect, and psychological safety.¹⁸ ACHE incorporated elements from both of these frameworks, including selecting, developing, and engaging a board; prioritizing safety in the selection and development of leaders; and establishing a compelling vision for safety. The JH Operating Management System framework and the Air Force emphasized the importance of leadership accountability.^17,19 The JH Safety and Quality framework encouraged QI leaders to pursue formal degrees to support their work.¹⁵ The ARCC and high reliability team models did not explicitly discuss leadership as a key strategy, although the ARCC model did discuss the importance of developing and using mentors to guide evidence-based decision-making.²⁰

The second key strategy is supporting a culture of safety. The Joint Commission described building trust, accountability, identifying unsafe conditions, strengthening systems, and assessment as key activities within this strategy.³ The IHI listed culture, including psychological safety, accountability, teamwork and communication, and negotiation, as one of their major 2 domains.¹⁸ The ACHE named their framework “culture of safety” and emphasized the need to both lead and reward a just culture and establish organizational behavior expectations.¹⁶ The Air Force described the importance of trust between leaders and staff, respectful communication, and willingness to admit errors within their culture of safety domain.¹⁹ The ARCC model incorporated an assessment of culture as a key aspect of implementation,²⁰ and the high reliability team model emphasized that responses to poor outcomes should be based on behavioral choices and not severity of outcome.²¹ Neither JH frameworks explicitly discussed culture of safety.

The third key strategy is building and using data systems to measure progress. The Joint Commission discussed the need to track and display quality measures and to involve IT support in the development of solutions to quality problems.³ IHI described the need for open sharing of data and other information concerning safe, respectful, and reliable care and to continually improve work processes and measure progress over time.¹⁸ The JH Operating Management System discussed the need to share and synthesize data to gain insights to make new discoveries and improve processes,¹⁷ and their Safety and Quality framework included a plan to evaluate processes.¹⁵ The Air Force described standardizing processes to gather and share information about patient care episodes, knowledge data, and processes to improve care delivery.¹⁹ The ARCC model described data management and outcomes monitoring as one of their implementation workshops.²⁰ The high reliability team model did not include a strategy related to measurement of progress.

The fourth key strategy is providing training and learning opportunities for providers and staff. The Joint Commission discussed the importance of training all staff on robust process improvement (eg, a blended performance improvement model aimed at improving patient safety in health care settings by integrating Lean Six Sigma and formal change management principles) as appropriate to their jobs.^3,33 IHI and the Air Force discussed developing learning systems, although the learning has more to do with implementing QI initiatives and learning from results, rather than learning how to implement HRO principles.^18,19 The JH Safety and Quality framework listed examples of training that each type of staff member should receive.¹⁵ The ARCC model described a workshop dedicated to evidence-based practice skills-building,²⁰ and the high reliability team model discussed implementation of TeamSTEPPS, a teamwork curriculum for health care staff.²¹ ACHE and the JH Operating Management System did not specifically discuss training or learning opportunities.

The fifth key strategy is implementing quality improvement interventions to address specific patient safety issues. This strategy is discussed in broad strokes as robust process improvement by the Joint Commission and Air Force,^3,19 and as improvement and measurement by the IHI.¹⁸ In the ARCC model, participants complete a 12-month evidence-based practice implementation project focused on improving quality of care, safety, and patient outcomes.²⁰ The JH Safety and Quality framework discussed the role of safety and quality experts in designing and directing system improvement efforts and provided examples of potential initiatives.¹⁵ The high reliability team framework described simulation training where teams can practice briefing, huddles, and debriefing strategies.²¹ Neither the ACHE nor the JH Operating Management System explicitly discussed QI initiatives.

In addition, we identified several complementary practices for strengthening implementation. We identified these by looking across the 8 frameworks to see what complementary practices were commonly recommended. These complementary practices are meant to be applied across implementation strategies to strengthen the overall delivery of HRO.

• Incorporation of justice, equity and patient-centeredness: The ACHE describes building trust, respect and inclusion as a key domain of building a safety of culture.¹⁶ The framework encourages leaders to value diversity and inclusion when selecting leaders and staff and to work towards evaluating and eliminating disparities in patient care. The Air Force selected patient-centeredness as a key domain of its framework.¹⁹ This practice could be integrated into HRO delivery through activities such as hiring a diverse workforce or prioritizing QI initiatives that address safety issues that disproportionately affect patients from racial/ethnic minority groups.
• Involvement of a variety of stakeholders involved in health care delivery, including patients and families: The JH Operating Management System described establishing patient and family advisory councils as an implementation activity that could be undertaken to advance one of their key implementation strategies.¹⁷ Other possible activities include assessing patient perspectives of culture of safety or inviting patients to serve on HRO leadership committees.
• Assembling transdisciplinary teams: Several frameworks – including the JH Operating Management System,¹⁷ ARCC model,²⁰ and high reliability team framework²¹ – discuss forming transdisciplinary teams as an important activity towards advancing HRO. This practice could be integrated into HRO delivery through activities like inviting providers from different specialties to attend daily safety huddles; or having nurses, physicians, and staff all attend the same HRO training sessions together.
• Utilizing change management strategies such as Lean Six Sigma to promote change: Most frameworks recommended health systems use complementary change management strategies – such as Lean Six Sigma,^15–17 IHI’s Model for Improvement,¹⁸ or a combination of strategies such as the Joint Commission’s robust process improvement^3,19 – to implement HRO principles into practice. This complementary practice could be integrated into several aspects of HRO delivery, such as training staff on Lean Six Sigma, or applying Lean thinking to root cause analysis to identify what is contributing to patient safety events and identifying and implementing solutions.

KQ1B. What were the processes for developing these frameworks (eg, consensus, literature review, etc)?

The Joint Commission’s HRHCM stood out as being the most rigorously developed framework, as the process involved a literature review, consensus among subject experts, pilot testing among an expert panel, and pilot testing with leadership at 7 US hospitals. However, the latter pilot testing effort was primarily focused on evaluating the tool to measure a health system’s progress on the framework (KQ2). The Air Force¹⁹ and JH Safety and Quality framework¹⁵ were developed through both a literature review and consultation with health care leaders and content experts. The IHI¹⁸ framework was developed specifically for the IHI Patient Safety Executive Development Program curriculum and was informed by an analysis of high-performing, proactive, and generative work settings. The ACHE framework was developed through partnership between the ACHE, the IHI, and the National Patient Safety Foundation (NPSF) Lucian Leape Institute (LLI). It involved consensus-building with industry leaders and experts who have had success in transforming their organizations into system-wide cultures of safety.¹⁶ The ARCC model was initially developed through a strategic planning process on how to rapidly integrate research findings into clinical processes.²⁰ The 2 remaining articles did not discuss the process of how frameworks were developed (JH Operating Management System,¹⁷ high reliability teams²¹).

KQ1C. What are the intended settings of these frameworks?

All frameworks were intended to be delivered in any health care delivery setting, except for the Air Force’s framework, which was designed specifically for the Air Force Medical Service.¹⁹ IHI’s framework was initially developed for use in acute care settings, although it has since evolved to be applicable to other settings.¹⁸

KQ1D. Who participates in implementing these frameworks?

Most frameworks were intended to be implemented by a variety of health care leaders, providers, and staff, including frontline providers, local and middle managers, and high-level managers and executives, as well as safety and quality leaders, across a variety of service areas.^3,15,18–20 IHI’s framework also included components to be implemented by patients and families.¹⁸ Exceptions are the ACHE¹⁶ and the JH Operating Management System¹⁷ frameworks, which were specifically designed for health care leadership,¹⁶ and the high reliability team framework which was designed for nursing professionals.²¹

KQ1E. What are the processes for implementing these frameworks?

Articles varied in the depth of information provided on how to operationalize the implementation of these frameworks, with the ARCC, Joint Commission, and IHI models emerging as the most comprehensive.

• The ARCC model provided details on providing learning and training opportunities (ie, 6 educational workshops, 8 days of educational and skills-building sessions over 1 year), as well as on implementing an intervention to address a specific patient safety issue (ie, 12-month project focused on improving quality of care, safety, and/or patient outcomes).²⁰
• The Joint Commission³ and IHI¹⁸ provided high-level recommendations for operationalizing HRO implementation, including building and using tools to measure progress (ie, assess the current state of HRO maturity; develop tools to advance maturity), as well as specific examples of activities that could advance these strategies.

Other frameworks provided some guidance on how to operationalize implementation, although they were less comprehensive.

• ACHE described 2 levels of implementation practices: foundational practices which focus on laying the groundwork for HRO implementation and sustaining practices which focus on spreading and embedding HRO concepts, specifically a culture of safety.¹⁶
• The JH Operating Management System suggests approaches to implementing the core concepts of the model, including developing and using data systems (ie, providing leaders with a standardized reporting format to assist in reporting on department progress), using systems engineering methodology, and convening stakeholder groups.¹⁷
• The JH Safety and Quality initiative provided recommendations based on the role of a specific health care provider or staff member. For example, they have specific suggestions on training and learning opportunities (ie, provide front line providers and staff with basic medical school education on safety and quality; provide managers with patient safety certificate programs and workshops on Lean Six Sigma and other change management processes).¹⁵
• The Air Force’s suggestions for operationalization include standardizing and stabilizing processes, engaging staff in behaviors to continuously improve these processes, mentoring staff, and leadership goal-setting, as well as a description of the desired future state of HRO integration into practice.¹⁹
• The high reliability team framework described specific approaches that touch on several implementation strategies including learning and training opportunities (ie, simulation training and provision of a structured HRO curriculum) and supporting a culture of safety (ie, development of a just culture system for penalizing staff when patient harm occurs).²¹

KEY QUESTION 2. What are the metrics for measuring a health system’s progress towards becoming an HRO?

We identified 8 articles^3,16,22–27 on 6 tools for measuring the progress toward becoming an HRO (Table 3). The Joint Commission’s HRHCM/Oro^TM 2.0 emerged as the most rigorously developed, validated, and applicable tool for VA settings. However, other tools such as the ACHE’s Culture of Safety Organizational Self-Assessment Tool¹⁶ may be useful in developing specific items missing from the Oro^TM 2.0 framework, such as teamwork culture and system-focused tools for learning and improvement.²⁷ Four additional tools have unclear applicability to the VA, as they were developed in countries outside the US,^22–25 did not report measurement items,^23,24 or require qualitative expertise to analyze results.²² Full details on these studies appear in Supplementary Materials, Appendix C, and selected findings appear below.

Oro^TM 2.0

The tool that most comprehensively addressed all 5 HRO implementation strategies identified in KQ1 was the HRHCM/Oro^TM 2.0.^3,34 As discussed in KQ1, the HRHCM is the Joint Commission’s framework for implementing HRO principles. This framework includes 4 levels (beginning, developing, advancing, approaching) for each of the 14 components (56 total) to guide health care leaders in assessing their systems’ level of maturity on becoming an HRO. The Oro^TM 2.0 is a web-based application that uses branching logic to guide health care leaders through the HRHCM assessment and produces a visual report that synthesizes data from multiple respondents within a single hospital.³⁴ Of note, the Oro^TM 2.0 was designed to be used at the individual hospital, rather than at a system level, and is only available to Joint Commission-accredited organizations. The tool outputs data into reports that could theoretically be shared between hospitals but it is not an automatic feature.

To develop the metrics used in by the HRHCM/Oro^TM 2.0, a team at the Joint Commission spent over 2 years engaging with high reliability experts from academia and industry, leading safety scholars outside of health care, and the published literature.³ Iterative testing with hospital leaders – first among 5 individuals in executive leadership positions, then among leadership teams from 7 US hospitals – was conducted to finalize the framework and included metrics. The resultant tool has since been validated in peer-reviewed research studies, including 1 study that tested the content validity of the tool at 6 VA sites.²⁷ Another study tested the internal reliability and discriminative ability in detecting different levels of HRO maturity in 46 hospitals from the Children’s Hospitals’ Solutions for Patient Safety network.²⁶

The VA study was a secondary analysis of qualitative data from 138 VA employees with patient safety expertise at various levels of leadership (eg, patient safety managers, executive leadership and service chiefs, infection control nurses) from 6 VA sites. The original study validated the AHRQ-developed patient safety indicator tool; the secondary analysis looked at how well responses mapped onto the Joint Commission’s HRHCM model. Researchers found that 12 of the 14 HRHCM components were represented, indicating good content validity. Two additional HRO components were identified through interviews that were not represented in the HRHCM model: teamwork culture and systems-focused tools for learning and improvement. While less applicable to the VA, the study that tested the HRHCM in 46 children’s hospitals found that the HRHCM had good internal reliability (Cronbach’s alpha = 0.72 to 0.87, depending on the domain), good discriminative ability (ie, health system average scores on beginning, developing, advancing, and approaching levels of maturity resembled a bell curve), and was responsive to change (ie, safety culture decreased after major organizational changes), indicating it may perform well at detecting progress on becoming an HRO.

Table 3

Metrics for measuring progress on becoming an HRO.

KEY QUESTION 3. What is the evidence on HRO implementation effects?

We identified articles from 7 health care organizations, primarily children’s hospitals, on the effects of HRO initiative implementation on safety culture, HRO process, and patient safety measures.^6,17,28–32 Full details on these articles are available in Supplementary Materials Appendix C, and selected findings appear below.

The most notable finding is that organizations experienced significant reductions in serious safety events (SSEs) (range, 55% to 100%) following the implementation of the 4 most comprehensive, multicomponent HRO initiatives.^6,29–31 Moreover, time since initiation and safety improvements appear to have a dose-response relationship, and the improvements were maintained for upwards of 9 years (Table 4).^6,29–31 Of note, only one of these studies explicitly discussed using one of the frameworks discussed in KQ1 (ie, the IHI framework).⁶ Two years after implementation, SSE reductions were 55% and 83%, respectively, in hospitals with a 12-month average of 0.9 (Ohio Children’s Hospital Association)³⁰ and 1.15 (Nationwide Children’s Hospital)⁶ SSEs per 10,000 adjusted patient days. At 4 years, Cincinnati Children’s Hospital Medical Center reported a 67% reduction in SSE rates and a baseline 12-month average of 0.9 events per 10,000 adjusted patient days.³¹ After 9 years, Genesis Health System reported achieving its goal of zero SSE (100% reduction).²⁹ In these studies, SSE was typically defined as “the most serious harm events that occur in hospitals and are defined by serious patient harm events that directly results from a deviation in best practice or standard of care.”³⁰ Improvements in safety culture were also reported, including improvement in safety attitudes⁶ and an increase in safety success story reporting,²⁹ but changes across various other safety culture dimensions had mixed results.³¹ At Cincinnati Children’s Hospital Medical Center,³¹ responses to the AHRQ Hospital Survey on Patient Safety Culture indicated improvements in organizational learning and continuous improvement, feedback and communication about error, and staffing. However, they reported no change in supervisor/manager expectations and actions promoting safety, teamwork within hospital units, nonpunitive responses to error, and a decline in communication openness.

A commonality across the 4 hospitals that reported SSE reductions is that they implemented their HRO initiative with the help of the same external consultant, Healthcare Performance Improvement (HPI), LLC.^6,29–31 Although the components varied somewhat across these 4 hospitals, they generally aligned with the 5 strategies discussed in KQ1: (1) developing leadership (eg, leadership training); (2) supporting a culture of safety (eg, increased communication through safety huddles; routine sharing of good catches and lessons learned); (3) providing training and learning opportunities for providers and staff (eg, error prevention training for staff; provider peer safety coaches coached their peers in use of the error prevention techniques); (4) building and using data systems to track progress (eg, enhanced root cause analysis processes using an electronic tracking system); and (5) implementing interventions to address specific patient safety issues (eg, embedding “time outs” and “debriefs” into standard surgical processes, using standardized checklists). Despite these similarities, initiatives conceptualized their goals of zero patient harm in different ways: one initiative’s board encouraged management to “aspire to eliminate preventable harm” by reducing the preventable harm index to zero⁶; one aimed to reduce SSEs to zero²⁹; and 2 others aimed to reduce SSEs by 75%-80%.^30,31 In addition, the structure of the Ohio Children’s Hospital Association was unique in that it is a state-wide collaboration of 8 tertiary pediatric referral centers that specifically refuse to compete on matters related to patient safety.³⁰ To promote transparent sharing of critical safety data among the collaborative to facilitate lessons learned without fear of undue liability, Ohio House Bill 153 was passed in 2010 to provide a legal framework expressly providing peer review protection for the 8 participating hospitals.

In addition to the 4 HPI-assisted initiatives, we also identified a similarly comprehensive initiative independently implemented by JH Hospital and Health System: the Operating Management System.¹⁷ Although the study did not report on SSEs, the authors reported improved compliance in Joint Commission process measures and a 79% reduction in potential preventable harms.

Finally, we found that process improvements are possible even with less intensive HRO initiatives that are more focused in scope.^28,32 When the Riley Hospital for Children at Indiana University Health implemented a Daily Safety Brief, they found improvement in communication, awareness, and working relationships, but not in comfortability in sharing errors.³² The Children’s National Medical Center experienced an increase in Apparent Cause Analysis (ACA) reliability scores following implementation of 13 interventions across education, process, and culture categories. They also reported an increase in efficiency (4 fewer days to turn around ACA) and increased satisfaction with the process.²⁸

While the results of these studies are promising, the overall strength of this evidence is low. Each initiative was only evaluated in a single study (consistency unknown), and each study was fair quality (common methodological weaknesses included lack of reporting on implementation fidelity and no concurrent control groups), with generally indirect outcomes and populations (few reported whether they met their goal of zero harm; none were conducted in Veterans). The main strengths of these studies were that they generally provided sufficient detail on how the intervention is conceptually linked to HRO, their main intervention components, and how they evaluated effects. Their main limitation was that a cause-effect relationship could not be established between these HRO initiatives and outcomes, because no study used a concurrent control group that would have ruled out the possibility that the effect was due to concurrent interventions (eg, implementation of an Electronic Medical Record [EMR]) or improved specialty-specific disease management).⁶

Table 4

Key findings from studies assessing effects of HRO implementation.

Limitations

Gaps and Future Research

The biggest gaps in knowledge on HRO implementation are (1) whether the improvements in process and safety outcomes are truly caused by HRO interventions or due to concurrent interventions or secular trends; (2) if HRO does indeed lead to improved outcomes, which components of HRO interventions are causing the effects; (3) whether certain implementation frameworks lead to better outcomes; and (4) what are the contextual factors (such as barriers and facilitators) affecting successful HRO implementation. Randomized controlled trial study designs are not a practical option for evaluating HRO interventions due to both the complexity of intervention as well as the delivery; therefore, other study designs such as quasi-experimental or natural experiments should be utilized instead. The VA HRO initiative is in a unique position to conduct these types of experiments. Implementing HRO principles at a select number of VA sites while other sites serve as a “wait-list” control would create a natural experiment to see if HRO implementation leads to improved outcomes. If this approach is taken, consideration should be given to how much wait-list control sites have begun implementing HRO concepts on their own or whether they’re implementing similar initiatives such as Lean Six Sigma. In addition, the widescale implementation of HRO across different sites likely means that each site will deliver slightly different interventions based on their individual contexts. Careful recording of the intervention components, when they were delivered, where they were delivered (eg, medical or surgical service areas), and whether they continued to be delivered may help to elucidate the effects of some of these individual intervention components on outcomes. This can inform where to invest future resources, and to tailor HRO delivery to specific contexts.

In addition, we were unable to determine what the mechanism of change was between HRO implementation and improvement in outcomes. While HRO delivery is theorized to lead to change in thinking about patient safety, resulting in improved processes and outcomes, this was not empirically examined in any of our included studies. Instead, some studies suggested that the impact of HRO on other process measures, such as safety culture, is mixed.³¹ This indicates that the mechanism of action driving changes in outcomes is more complex. Future studies should evaluate what is the mechanism of change, such as improved mindfulness or safety culture, to help answer both the how and why HRO implementation may lead to improved patient safety outcomes. Future studies may also want to consider the extent to which HRO implementation overlaps – or doesn’t – with system redesign strategies, as these are complementary approaches to improving quality of care.

Conclusions

A variety of frameworks and evaluation tools are available for HRO implementation and evaluation, with the Joint Commission’s High Reliability Health Care Maturity (HRHCM)/ORO 2.0 among the most rigorously developed and validated. Multicomponent HRO interventions that include several of the 5 common implementation strategies and that are delivered for at least 2 years are associated with improved process outcomes, such as staff perceptions of safety culture, and important patient safety outcomes, such as reduced SSEs. Future research studies should incorporate concurrent control groups through quasi-experimental designs to rule out the possibility that the effects are due to other interventions or secular trends. Future research should also focus on identifying whether certain frameworks, metrics, or components of interventions lead to greater improvements.

Operational Partners

Operational partners are system-level stakeholders who have requested the report to inform decision-making. They recommend Technical Expert Panel participants; assure VA relevance; help develop and approve final project scope and timeframe for completion; provide feedback on draft report; and provide consultation on strategies for dissemination of the report to field and relevant groups.

• William Gunnar, MD
  Executive Director
  National Center for Patient Safety
• Amy Kilbourne, PhD, MPH
  Director
  Quality Enhancement Research Initiative

Technical Expert Panel

To ensure robust, scientifically relevant work, the TEP guides topic refinement; provides input on key questions and eligibility criteria, advising on substantive issues or possibly overlooked areas of research; assures VA relevance; and provides feedback on work in progress. TEP members are listed below:

• Laura Damschroder, MPH, MS
  Center for Clinical Management Research
  Ann Arbor, MI

Key Informants

The ESP sought input from 2 Key Informants with diverse experiences and perspectives in implementing HRO interventions into large, integrated health care systems.

• Glenda J. L. Battey, PhD
  Providence St Joseph Health
  Renton, WA
• Robert G. Ritter, FACHE
  Harry S. Truman Memorial Veterans’ Hospital
  Columbia, MO

Peer Reviewers

The Coordinating Center sought input from external peer reviewers to review the draft report and provide feedback on the objectives, scope, methods used, perception of bias, and omitted evidence. Peer reviewers must disclose any relevant financial or non-financial conflicts of interest. Because of their unique clinical or content expertise, individuals with potential conflicts may be retained. The Coordinating Center and the ESP Center work to balance, manage, or mitigate any potential nonfinancial conflicts of interest identified.

APPENDIX A. Search Strategies

APPENDIX B. List of Excluded Studies

Exclude reasons: 1=Ineligible population, 2=Ineligible intervention, 3=Ineligible comparator, 4=Ineligible outcome, 5=Ineligible timing, 6=Ineligible study design, 7=Ineligible publication type 8=Outdated or ineligible systematic review

APPENDIX C. Evidence Tables

Data Abstraction: Key Question 1 Studies (PDF, 111K)

Data Abstraction: Key Question 2 Studies (PDF, 77K)

Data Abstraction: Key Question 3 Studies (Part 1) (PDF, 49K)

Data Abstraction: Key Question 3 Studies (Part 2) (PDF, 80K)

Quality Assessment Of Included Primary Studies (PDF, 61K)

APPENDIX D. Peer Review Comments

References

1.

Aboumatar HJ, Weaver SJ, Rees D, Rosen MA, Sawyer MD, Pronovost PJ. Towards high-reliability organising in healthcare: a strategy for building organisational capacity. BMJ Qual Saf. 2017;26(8):663–670. [PubMed: 28546510]

2.

American College of Healthcare Executives. Leading a Culture of Safety: A Blueprint for Success. 2017.

3.

Chassin MR, Loeb JM. High-reliability health care: getting there from here. Milbank Q. 2013;91(3):459–490. [PMC free article: PMC3790522] [PubMed: 24028696]

4.

Day RM, Demski RJ, Pronovost PJ, et al. Operating management system for high reliability: Leadership, accountability, learning and innovation in healthcare. J Patient Saf Risk Manag. 2018;23(4):155–166.

5.

Frankel A, Haraden C, Federico F, Lenoci-Edwards J. A Framework for Safe, Reliable, and Effective Care. White Paper. Cambridge, MA: Institute for Healthcare Improvement and Safe & Reliable Healthcare; 2017.

6.

Melnyk BM. Achieving a high-reliability organization through implementation of the ARCC model for systemwide sustainability of evidence-based practice. Nurs Adm Q. 2012;36(2):127–135. [PubMed: 22407205]

7.

Office of the Air Force Surgeon General. Trusted Care Concept of Operations (CONOPS). 2015.

8.

Riley W, Davis SE, Miller KK, McCullough M. A model for developing high-reliability teams. J Nurs Manag. 2010;18:556–563. [PubMed: 20636504]

9.

Ikkersheim DE, Berg M. How reliable is your hospital? A qualitative framework for analysing reliability levels. BMJ Qual Saf. 2011;20(9):785–790. [PubMed: 21441603]

10.

Kenneth MJ, Bendaly N, Bendaly L, Worsley J, FitzGerald J, Nisker J. A measurement tool to assess culture change regarding patient safety in hospital obstetrical units. J Obstet Gynaecol Can. 2010;32(6):590–597. [PubMed: 20569541]

11.

Mousavi SM, Dargahi H, Mohammadi S. A study of the readiness of hospitals for implementation of high reliability organizations model in Tehran University of Medical Sciences. Acta Med Iran. 2016;54(10):667–677. [PubMed: 27888596]

12.

Mousavi SMH, Jabbarvand Behrouz M, Zerati H, et al. Assessment of high reliability organizations model in Farabi Eye Hospital, Tehran, Iran. Iran J Public Health. 2018;47(1):77–85. [PMC free article: PMC5756604] [PubMed: 29318121]

13.

Randall KH, Slovensky D, Weech-Maldonado R, Patrician PA, Sharek PJ. Self-reported adherence to high reliability practices among participants in the children’s hospitals’ solutions for patient safety collaborative. Jt Comm J Qual Patient Saf. 2019;45(3):164–169. [PubMed: 30471989]

14.

Sullivan JL, Rivard PE, Shin MH, Rosen AK. Applying the high reliability health care maturity model to assess hospital performance: a VA case study. Jt Comm J Qual Patient Saf. 2016;42(9):389–411. [PubMed: 27535456]

15.

Brilli RJ, McClead RE, Jr., Crandall WV, et al. A comprehensive patient safety program can significantly reduce preventable harm, associated costs, and hospital mortality. J Pediatr. 2013;163(6):1638–1645. [PubMed: 23910978]

16.

Crandall KM, Sten MB, Almuhanna A, Fahey L, Shah RK. Improving apparent cause analysis reliability: a quality improvement initiative. Pediatr Qual Saf. 2017;2(3):e025. [PMC free article: PMC6132456] [PubMed: 30229162]

17.

Cropper DP, Harb NH, Said PA, Lemke JH, Shammas NW. Implementation of a patient safety program at a tertiary health system: A longitudinal analysis of interventions and serious safety events. J Healthc Risk Manag. 2018;37(4):17–24. [PubMed: 29604147]

18.

Lyren A, Brilli R, Bird M, Lashutka N, Muething S. Ohio children’s hospitals’ solutions for patient safety: a framework for pediatric patient safety improvement. J Healthc Qual. 2016;38(4):213–222. [PubMed: 26042749]

19.

Saysana M, McCaskey M, Cox E, Thompson R, Tuttle LK, Haut PR. A step toward high reliability: implementation of a daily safety brief in a children’s hospital. J Patient Saf. 2017;13(3):149–152. [PubMed: 25119785]

20.

Muething SE, Goudie A, Schoettker PJ, et al. Quality improvement initiative to reduce serious safety events and improve patient safety culture. Pediatr. 2012;130(2):e423–e431. [PMC free article: PMC3408689] [PubMed: 22802607]