Design

The Oregon Long-term Care Medication Risk Modeling project is a developmental study; it is the first health care study using ST-PRA to create medication risk models for multiple LTC facilities.

Sample

A convenience sample of 6 LTC chains and l8 facilities participated in the study. One organization operated a single large campus and participated in both the nursing and CBC modeling. There were 10 nursing facilities from 4 nursing chains, ranging in size from 88 to 214 licensed beds. Three CBC chains provided eight facilities: five assisted living and three residential care facilities. The size of the assisted living facilities (ALFs) ranged from 60 to 87 licensed beds, and the three residential care facilities ranged in size from 64 to 122 beds. The modeling teams were composed of trained ST-PRA facilitators, 1 or 2 research staff, and medication staff (nurses, certified medication aides, and caregivers) recruited from the 18 facilities. Over the course of 3 months, four chain-specific nursing facility teams and three CBC teams met to build medication risk models using the ST-PRA process.

Overview of ST-PRA Modeling

The central tool of ST-PRA is the risk tree, also known as a fault tree. An ST-PRA risk tree starts with a top-level event, which is the undesirable outcome (e.g., a wrong dose delivered to a patient). Risk modeling teams then identify the failures that link together, leading to the top-level event. The risk tree in Figure 1 illustrates the interrelationships between human errors, cultural or behavioral norms, systems, and equipment failures that together may lead to a top-level event.

Figure 1

Example of risk or fault tree—dose transcribing error.

This risk tree illustrates an initiating error (order taker writes down the wrong dose) combining with a read-back of the order that fails to catch the error, and three other capture opportunities that do not catch the initiating error. In the process of building the ST-PRA models, the modeling teams used Relex software (Version 7.6; Relex Software Corp.; Greensburg, PA) to identify 50–500 different combinations of failures leading to a particular top-level event risk being analyzed. What follows is a description of the major steps of ST-PRA as applied in the Oregon project.

Major steps in the Oregon risk assessment process

1.

Organizing risk modeling teams. Facility managers nominated staff based on each person's knowledge of medication systems and their ability to communicate candidly about the processes and behaviors that could lead to system failures within their facilities. A total of 16 nurses and certified medication assistants (CMAs) participated in the nursing facility modeling groups. A total of 14 nurses and caregivers participated in the CBC modeling groups. Staff members from the same chain were grouped together, and each chain-specific group met four times for approximately 3 hours. Groups typically included one or two staff from two or three different facilities. Pharmacists and physicians associated with chains participated in separate focus groups, reviewing the preliminary models and providing feedback about their accuracy and completeness.

2.

Build initial process maps. Beginning with an undesirable outcome or top-level event, each modeling team developed preliminary process maps that represented medication delivery processes under the direct control of the facility (processing of prescriber orders, transcription, receipt/storage of delivered drugs, pharmacy dispensing errors, medication administration) for each of the four classes of patient events being studied. Process maps helped establish the scope of the model by putting bounds on the processes and types of medications to be analyzed by the teams.

3.

Build control systems maps. Control system mapping is a tool for identifying both active and passive controls in the process being modeled. Active controls are features of the system that are in place to specifically help manage the risks of the undesirable outcome under analysis. For example, a unit dose or blister-pack cards are active controls that reduce the chance of wrong-dose errors. Passive controls are those features of the system that help control the specific risk under analysis, but that exist in the system for some purpose other than controlling the risk being modeled (e.g., the shape and color of pills). The purpose of identifying these controls is to be able to evaluate them through ST-PRA modeling.

4.

Identify process and control system failure modes. In the Oregon model, system components are the physician, pharmacist, nurse, records clerk, medication aide, and resident. While we rarely refer to humans as system components, they are components in the ST-PRA risk tree. Once the process and control system maps were complete, the modeling teams used a modified FMEA analysis to identify individual failures (errors, at-risk behaviors, systems, and equipment failures) that could lead to the undesirable outcome under analysis. The FMEA provided the source data for building the initial risk trees.

Table 1Examples of summary failure modes and effects analyses

View in own window

Process	Failure mode	Downstream controls	Failure effect
In-room patient identification	Nurse enters wrong room	Visual identification of patient	Medication delivered to wrong patient (note: ST-PRA will model failures of downstream controls that might capture nurse entering wrong room before medication reaches the wrong patient)
		Arm band check
		Verbal name check
		Verbal discussion of medication with patient
Physician ordering	Medication order written in wrong chart	Nurse review of order	Medication administered to wrong patient (note: ST-PRA will model failures of downstream controls that might capture the physician error before reaching the patient)
		Pharmacy review of order
		Verbal discussion of medication with patient

5.

Build initial risk trees. With failure modes in hand, the risk modeling teams began building the initial risk trees, creating “branches” or “splits” whenever they identified a significant variation, for example, whether medication staff pull their blister-packed drug cards by taking them from bins presorted by administration time, or use the medication administration record (MAR) to pull individual cards sorted by patient name or room number. Risk trees went through multiple iterations until the modeling groups were satisfied that they have captured the most important elements of the process.

6.

Collect quantitative denominator data. Denominator data for creating rates for the events that are represented in the risk trees were collected from participating nursing and CBC facilities. The research team analyzed data from random, stratified samples of de-identified MARs to establish nursing and CBC per-order and per-dose denominators for error rates. Standardized rates for the average number of prescriber orders and oral doses of medications delivered per staff per year were developed from these MAR analyses. These rates were used during the modeling meetings; for example, if a nurse said she saw a particular error (such as an omitted dose) twice a week, the team used the denominator data to turn the time frequency estimates into an estimate of errors per dose. Throughout the model building, the teams continued collecting data to support the analysis.

7.

Estimate human error, at-risk behavior, control systems, and equipment failure rates. Once the risk models were well developed, the modeling team members began to identify and estimate data—based upon their experience, the experience of others where applicable, published rates, and human-reliability expert input—for rates of control systems failures, at-risk behaviors, errors per medication order, and errors per oral medication dose. During the process-mapping and risk tree-building stages, modeling teams found many variations in process. These included how medication bins or drawers were organized (by shift, by time of administration, or in alphabetical order by either patient names or drug names), how medications were delivered (pop/pass/sign versus sign/pop/pass), as well as the use of medication carts versus preparation in a medication room. The modeling teams were required to explore the relative risks of each variation—some being assessed directly, other variations requiring risk tree modeling to determine the relative merits of one variation versus another.

The teams also had to develop rates for at-risk behaviors, such as failure to check the MAR when administering medications. This is a commonplace at-risk behavior not easily identified in postevent investigations, particularly in terms of estimating a normative rate for a particular institution. Nurses and caregivers spend long shifts getting to know the patients, their diagnoses, and their medications. Despite policies and procedures that require checking the MAR during medication administration, medication staff admit that in practice, for a variety of reasons, they fail to universally accomplish this safety check. ¹²

The teams noted that even if pharmacies or medication aides entered the time on blister-packed oral medications to operationally fulfill the “five rights” (right patient, right drug, right dose, right time, right route), skipping the MAR check could lead to a wrong-drug, wrong-dose or omitted-dose error if the MAR contained new orders to add, change, or discontinue drugs. The group discussed rates of failure to check the MAR based on different ratios: 1 in 100 doses, 5 in 100 doses, or 50 in 100 doses. The models were calculated based on the team's rate estimates. The focus group then reviewed the probability estimates and either agreed with what the model identified as the risks, or went through the estimation process again until they reached a consensus. Through this repetitive process, the interdisciplinary team arrived at an estimate for the local cultural norm for checking the MAR. Experience indicates that these team estimates are more accurate than rates derived from event data, and are often more accurate than rates predicted by senior management within the facility. ^{13, 14}

8.

Run the quantitative model and “cut set” reports. After the rate estimates were put into the model, the software calculated intermediate- (gate) and top-level event probabilities (i.e., for wrong drug, dose, resident, or omission). PRA software will generally produce a report listing the probabilities of each unique combination of failures, a “cut set,” that could produce the top-level event. The software ranks the combinations of failures according to their probabilities, allowing the modeling teams to focus on those combinations of errors that have the highest probability of producing the medication error being modeled.

9.

Compare the model against data and group expectations. Team members continued iterative development of the model until the model represented the best available representation of the risk. Once completed, the chain-specific models were shared with chain management, pharmacy representatives, and nursing facility medical directors to collect their observations and input.

10.

Consolidating risk models. This project was designed first to build chain-specific models, and then merge the models into consolidated models of nursing and CBC facilities. Very early in the project, the research team found that medication processes and rates described by each team were similar and decided instead to develop only one model for each facility type, nursing and CBC. Across chains and facilities, the critical elements of taking prescriber orders, transcribing them, ordering and receiving drugs from pharmacies, storing, and administering medications were similar, with occasionally important variations requiring new branches in the risk trees.

Modeling medication delivery systems risks in multiple facilities mirrors the challenges of building single-hospital models, where input from many different units or departments must be accommodated during the model-building process. Also, the scale and politics are different; for example, direct competitors had to agree to share their models during consolidation. However, the processes for building these models are essentially the same as would be followed for individual institutions. Single institutions represent far fewer logistical problems; modeling is usually completed within a few days rather than weeks. The processes followed within the modeling groups, however, reflected the typical ST-PRA group experience.

Variations identified by the groups required judgment calls by the modeling team (group members and consultants) to assure a reasonable scope for the project. Many low-risk failure paths and many inconsequential variations were not built out in the model, but were modeled at a high level. Practice differences by shift or individuals were common within single facilities, as well as procedural differences between facilities within the same chain. Policies and procedures varied by chain, for example, whether a chain initially purchased a 3- to 14-day supply of medications to bridge the gap created by delayed preauthorizations by insurers. Group members believed there was less likelihood of “borrowing” medications to assure continuous therapy if this policy was in place. This variation is included in the models because it could raise or lower the cumulative risk of an error occurring. If an event was identified and might be worthy of future study, it was included in the risk trees as an undeveloped event. An example of this is the variety exhibited in borrowing practices when drugs are not available. Hundreds of variations in medication delivery practice were seen across the sites, but most had little or no impact on serious medication errors that reach residents. Variation and subsequent judgment calls regarding what should be modeled are common in aviation and nuclear power PRAs, as well. ^{15, 16}

Probably the most difficult task associated with creating these models was keeping all the modeling group and quantitative data organized and accurately recorded. In every group meeting, at least one and often two research team members recorded data. With seven groups running more or less simultaneously, the research team had to devise methods to record and synthesize large amounts of information over relatively short periods of time by using spreadsheets, the risk analysis software, and copious note taking.

Modeling the real versus the ideal medication delivery system

Given the prospective nature of ST-PRA, modeling teams are willing to describe and discuss common deviations from policies and procedures (P&Ps) that they would otherwise not divulge, for fear of criticism or punishment. ¹² For example, rates of noncompliance for checking the MAR before giving medications are not generally reported in the literature, nor are there any descriptions of the behaviors that lead up to borrowing medications from one patient's supply to sustain treatment for another patient. These practices are generally forbidden in P&Ps, but this study found that they occur with some frequency to cope with circumstances that are often outside the staff's control. Given that the ST-PRA modeling teams were making assessments of future risk, they were much more open about what really happens in the administration of medications.

Modeling teams explained that most borrowing occurs when a clinically important drug (such as an antibiotic, anticonvulsive, anticoagulant, cardiovascular, or analgesic) is not immediately available in the form or dose needed by a particular patient. Concerned about the impact on the resident of missing one or more doses of these drugs, a staff member borrows the drugs to assure therapeutic continuity. Borrowing also occurs when insurer preauthorization policies delay acquisition of clinically important drugs. Sometimes borrowing occurs to avoid surveyor citations for “drug not available” entries in the MAR. Borrowing results because emergency boxes are not typically stocked with every dose and form of a drug; preauthorization approval can take 2 weeks or more; someone can forget to order a refill; or the pharmacy is unable to deliver the drug on time. The model estimates the risks associated with these types of practices. Managers and policymakers can decide if the size of the risk requires action on their part and, if so, have some clues about where to intervene.

Understanding the context of errors

In addition to providing access to largely unobtainable behavioral data, risk modeling provides context unavailable in single-event investigations, such as root-cause analysis. Single-event analyses cannot capture the range of combinations of elements that might operate to produce the same outcome—they can only describe what occurred in the event under study. The example above, which describes borrowing practices, illustrates that there are multiple reasons why a particular behavior or event occurs. ST-PRA offers the advantage of modeling several variations of systems and their human components, including behaviors associated with specific work processes.

During this study, the modeling teams were able to visualize the “system” of medication delivery through defining the combinations of failures that lead to delivery of incorrect medications to a resident. They learned the value of redundant control systems and capture opportunities from the model. For example, repeated checking of transcribed orders from one month's MAR to the next substantially reduced the probability of errors involving incorrectly transcribed doses or drugs reaching the resident. Modeling teams and managers can use the models to see the interrelationships between component reliability and system reliability—an interrelationship that, if ignored, can lead to blaming individuals for failures outside of their control. Too often, when adverse events occur, health care professionals lack the time and/or resources to thoroughly understand the elements and risk factors that led up to the incident, resulting in sometimes unfocused, punitive, or piecemeal solutions. ¹⁸ Health care institutions tend to focus only on the breach—the error that contributed to the patient event—and ignore the important contributions of the system and component failures that preceded the individual's lapse. ¹⁹ This is both an unfair and unrealistic approach to error management, particularly if the error arose from systems failures outside of the control of the individual.

The Oregon risk models force us to see serious medication errors in the context of the systems that produce them, leading to a better understanding of both systemic and human component safety risks. Having these models provides Oregon policy, public health, and LTC leaders with a common mental model to establish priorities for planning and evaluating alternative statewide medication risk-reduction strategies. ²⁰ At the facility level, managers can use the rankings of relative risks for the modeled errors to determine where to concentrate limited resources for maximum risk reduction.

Role of risk models and regulation

Troyan Brennan ²¹ has argued that regulation cannot improve quality if it depends primarily on licensing and disciplinary actions to achieve what he calls “culling,” or removing defective performers. Brennan offers five recommendations for improving quality through regulation. ST-PRA risk models can be used to achieve or inform each of Brennan's recommendations:

• Help regulators integrate new knowledge into public policy
• Support the development of shared regulatory/provider aims
• Encourage provider self-improvement
• Support innovation to improve quality
• Improve outcomes (risk) measurements

Unfortunately, space does not permit an in-depth discussion of each of these recommendations; however, the following example helps to illustrate the unique contributions of ST-PRA risk models.

Oregon's public health and regulatory leaders, in partnership with the LTC provider community, intend to use the models to develop shared goals or aims for quality and risk reduction in the State. These models provide State regulators and policymakers with a view of risk that can inform both future legislation and current enforcement practices. For example, regulators were unaware that the so called “2-hour rule” ¹⁷ —currently interpreted by Oregon surveyors as meaning that all medications (including vitamins and nutritional supplements) must be administered in the “window” of 1 hour before or after the time specified in the prescriber's orders—would play a prominent role in the risk modeling. This rule encourages undesirable, at-risk behavioral norms or short cuts, as medication staff delivering large volumes of medications at certain times of the day become pressed for time. Regulators and providers now have an opportunity to discuss these findings objectively and to evaluate how to reach their shared goal of timely medication administration to achieve therapeutic efficacy. Through the model, Oregon regulators and providers can begin a dialogue to analyze the alternative, associated risks using an evidence-based approach to narrow the scope of the rule by asking: Which risks are increased? Which are decreased?

ST-PRA as a tool for State-level change

Even though it has been used for decades to make very important policy and operational decisions in other high-risk industries, probabilistic risk modeling is a new tool in health care. ¹⁶ As noted earlier, while the ST-PRA methodology has limitations, it is uniquely suited to prospective risk assessment of combinations of events leading up to the top-level event or undesirable outcome. A skilled and knowledgeable leader or team is required to guide a modeling group through the ST-PRA process, gaining group member trust and encouraging the candor necessary to discuss deviations from policy that are often ignored or underestimated by outsiders. Model-building teams require enough time to carry out the development of accurate and complete risk trees, which are critical to the success of the ST-PRA. In this particular study, the underlying common medication processes (ordering, transcribing, receipt/storage, and administration) were similar across facilities, and, while important variations were identified, they did not prevent consolidation of each chain's models into one “master model” for nursing facilities and one for CBC facilities. How interfacility and interchain variances are addressed during model building requires good process and control-system modeling, judgment, and PRA skills on the part of the team leaders. Consolidated models would have been large and relatively unmanageable if each facility had unique processes in place.

This type of State-level assessment requires trust between all parties, a significant commitment of resources, protection of this very sensitive data to maintain confidentiality, consideration of potential policy or legal consequences to participants, and an agreement that the model(s) will be used to improve quality rather than to punish the provider community. The public-private partnering demonstrated in this study requires trust and cooperation between the State agency and provider leadership. ²²

Finally, through this research, the OHCA and State of Oregon partnership will have a representative model of the active controls LTC organizations can put into place to manage the risks of medication error. The control-system model that accompanies the ST-PRA will provide an overall mapping of the strategies in place to manage medication risk. A new piece of these controls is the ST-PRA itself, which provides guidance to system managers on the risks inherent in medication delivery systems and the relative contribution of each active control to reducing the risk of medication error. Each medication adverse event investigated by State agencies or participating facility will become a learning opportunity to inform the risk model, by either validating the model or adding risk elements not discovered in the initial ST-PRA development. The result of this research is a “living risk management system” that can and should be used by system managers to manage the risks of medication error. ³