• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jamiaJAMIA - The Journal of the American Medical Informatics AssociationInstructions for authorsCurrent TOC
J Am Med Inform Assoc. 2005 Jul-Aug; 12(4): 383–389.
PMCID: PMC1174882

Implementing a Commercial Rule Base as a Medication Order Safety Net

Abstract

A commercial rule base (Cerner Multum) was used to identify medication orders exceeding recommended dosage limits at five hospitals within BJC HealthCare, an integrated health care system. During initial testing, clinical pharmacists determined that there was an excessive number of nuisance and clinically insignificant alerts, with an overall alert rate of 9.2%. A method for customizing the commercial rule base was implemented to increase rule specificity for problematic rules. The system was subsequently deployed at two facilities and achieved alert rates of less than 1%. Pharmacists screened these alerts and contacted ordering physicians in 21% of cases. Physicians made therapeutic changes in response to 38% of alerts presented to them. By applying simple techniques to customize rules, commercial rule bases can be used to rapidly deploy a safety net to screen drug orders for excessive dosages, while preserving the rule architecture for later implementations of more finely tuned clinical decision support.

Medication errors are a national concern and have received substantial attention since the 1999 Institute of Medicine (IOM) report suggested that 44,000 to 98,000 deaths may occur annually in the United States from medical errors and that more than 7,000 of these deaths were medication related.1 Furthermore, it has been reported that more than half of all preventable medication errors are the consequence of improper physician orders.2,3 Subsequent IOM reports in 2001 and 2003 strongly recommended the development of automated information systems that provide immediate access to clinical decision support (CDS) tools for clinicians.4,5 Because problem orders may slip through these initial synchronous lines of defense6 and because patient parameters such as renal function may change rendering an initially appropriate order inappropriate,7 asynchronous safety nets are needed in addition to synchronous point-of-care CDS. Although commercial vendors often have a robust set of drug dosage rules, the effectiveness of this rule-based CDS is frequently diminished by poor positive predictive value of these rule sets.8 Judicious use of alerts is necessary to avoid decision support overload, that can result in oversight of clinically significant alerts. Unfortunately, alert overload is a common feature of decision support systems using unmodified commercial rule bases.9,10 Yet, we are unaware of studies that have evaluated the impact of modifying commercial rule bases to address this issue. The purpose of this paper is to describe the process, challenges, and outcomes of implementing a commercial drug dosage rule base to improve medication safety in a multihospital system. This process resulted in rapid deployment of asynchronous CDS as a safety net for drug dosage with acceptable positive predictive value of the rules.

Background

BJC HealthCare is a large, nonprofit health care organization affiliated with Washington University School of Medicine that delivers services to residents in the greater St. Louis metropolitan, southern Illinois, and mid-Missouri regions. It includes 12 hospitals ranging from a large, urban, university teaching hospital to four very small, rural, community hospitals. In 1994, we implemented an expert system, DoseChecker™, to identify possible drug dosing errors at one facility and have now expanded this system to include six facilities using an internally developed rule base.7 Existing pharmacy computer systems either lacked drug dosage rules or did not have them activated due to the poor positive predictive value of these rules. Because the development of a rule base is a labor-intensive process, we initially screened only 55 problematic drugs with our in-house–developed system.7 We hypothesized that use of a commercial rule set with frequent updates incorporating newly marketed drugs would be a more efficient method for deploying a more comprehensive set of medication rules. We evaluated the major vendors of drug dosage rules and selected one based on acceptability of rules and future compatibility with a proposed computerized physician order entry (CPOE) system. Because CPOE is not currently available at BJC facilities, an asynchronous method of CDS has been employed. Pharmacy orders are integrated with patient-specific demographics and laboratory data via real-time HL7 messages and are stored in a clinical data repository. This new pharmacy expert system, dubbed DoseRanger™, uses a commercial knowledge base and accesses data from the clinical repository, issuing alerts to pharmacists when rule violations are detected. Pharmacists evaluate each alert and contact prescribers if a dosage adjustment is deemed necessary, recording a response for each alert on a Web page for database storage and subsequent analysis. High-priority alerts are also escalated to pagers if no response occurs within a specified period of time.

Methods

A commercial rule base (Cerner Multum, Kansas City, MO) with 48,262 total rules for 1,537 distinct drugs as of January 2003 was tested at five facilities within BJC HealthCare. The rule base contains rule types for maximum single dose, maximum daily dose, minimum dose, maximum frequency, continuous infusion, lifetime dose, and contraindications. In a preliminary review, we found an overwhelming alert volume when all rule types were implemented simultaneously. Therefore, to avoid CDS overload, we chose to test only maximum single-dose rules in this initial implementation (N = 21,744). We selected maximum single-dose rules over other rule types because these rules are relatively straightforward and not only allow screening of one-time doses, but also enable checking of multidose regimens in which individual doses are too high.

Phase I: Preimplementation

Data from all drug orders during the month of January 2003 were integrated with serum creatinine, height, weight, age, and gender to test maximum single-dose rule performance at each of the five facilities having data feeds into our database. Both alert volume and rate identified potentially problematic rules. Using these data, pharmacy domain experts at each facility established new single-dose limits for drugs with unacceptably high false-positive alert rates, while problematic rules not amenable to dose modification were inactivated. Rules from the vendor had three parameters comprising the primary database key and defining a unique rule:

  1. Drug code: a code identifying a unique drug, strength, dose form, and product route.
  2. Rule type: a code identifying rule type (e.g., single dose, daily dose, lifetime dose, contraindication).
  3. Case ID: a code identifying all pertinent conditions and patient parameters for the rule (e.g., age, weight, creatinine clearance [CLcr], disease states).

We created derived tables containing these additional concepts:

  1. Site or facility: identifier for each hospital.
  2. Dose-limit multiplier: a numeric value by which the vendor's maximum or minimum dose will be multiplied to establish a new dose threshold. This value was determined by clinical pharmacists at each site using standard pharmaceutical compendia as references, weighing potential alert volume against risk of toxicity. In addition, facility B required Pharmacy and Therapeutics Committee approval of all rule changes.
  3. Priority: a code allowing immediate delivery of high-priority alerts while saving lower priority alerts for delayed delivery at predetermined times.
  4. Status: a flag that indicates whether a rule is active or inactive.

These concepts create a rule environment with a high degree of flexibility in customizing facility-specific rules. Previous experience with DoseChecker™ had revealed that obtaining consensus from medical staffs and pharmacists to standardize rules across the entire health care enterprise was impractical. [triangle] shows some example rules in a denormalized database view. Our derived tables are created automatically by structured query language (SQL)-stored procedures during each monthly update process. New drugs are given a status of pending until reviewed by a pharmacy domain expert who determines status, priority, and whether a dose-limit multiplier is appropriate. Because each facility has diverse physicians, patient populations, and pharmacy computer systems, we recognized that the commercial rules for some drugs would need to be customizable at the institutional level. Although this strategy results in more complex initial development, it also results in lower long-term maintenance costs and more rapid implementation than developing an internally created rule base.

Table 1.
Denormalized Database View of Example Rule Customizations Showing Architecture That Allows for Activation/Inactivation, Dose Multipliers, Priority, and Facility-Specific Rules

Phase II: Implementation

After customizing the rules for each facility, the system was deployed at facility A in June 2003 and at facility B in September 2003. The system delivered the alerts for each dosage rule violation via a secure Intranet Web page, fax, or network printer. Alerts included patient demographics, pertinent laboratory results, and basic information about the drug order triggering the alert (e.g., dose, route, and frequency). Alert disposition and validity were assessed using a Web-based response form on which pharmacists recorded their actions and consequences. A sample alert is shown in Appendix 1.

Phase III: System Refinement

After implementation, continued data analysis resulted in some additional rule modifications by pharmacy domain experts at each facility. However, a substantial number of alerts were still considered clinically insignificant, and enhancements to DoseRanger™ were implemented to achieve appropriate rule sensitivity. Although somewhat arbitrary, these modifications were made to reduce the volume of marginal alerts given the imprecision of measurements and laboratory tests. These had also been included in our previous system, DoseChecker™. These included:

  1. CLcr: Because of the inherent imprecision of CLcr estimations, the patient's CLcr value was increased by 20% before applying the maximum dose rules, reducing the number of alerts that resulted from the CLcr being slightly below the threshold for satisfying a rule. For example, a rule with a CLcr threshold of 30 mL/min would not be satisfied until the patient's estimated CLcr was ≥25 mL/min.
  2. Weight: The patient's actual weight was increased by 20% before applying the maximum dose rules to reduce alerts resulting from the patient weight being marginally below a rule threshold. For example, a rule with a weight threshold of 45 kg would not be satisfied until a patient's weight was ≥37.5 kg.
  3. Dosage calculation: Dosages based on body weight or surface area (mg/kg or mg/m2) were given a ± 5% margin of error to allow for rounding of height, weight, and drug dosage to practical values.
  4. Duplicate alert suppression: Alerts were not generated if a previous alert had been issued within the past seven days for the same patient, drug, dose, route, and frequency; it is not uncommon for drug orders to be re-entered into pharmacy systems to change nondose-related information such as schedule times or comments. Many pharmacy systems generate a new “order” for these changes that would trigger a duplicate alert from the clinician's perspective.

Results

Phase I: Preimplementation

At five BJC Healthcare facilities, a total of 192,668 drug orders from January 1 to February 1, 2003, were retrospectively screened with unaltered maximum single-dose limits as supplied by the vendor, resulting in 17,667 rule violations and an alert rate of 9.2%. Of these violations, 13,366 (76%) were from the teaching hospital despite its accounting for only 56% of the orders screened. Further, only 58 drugs caused 90% of the alerts. Results of this screening for each of the five facilities are shown in [triangle].

Table 2.
Phase I Results of Maximum Single-Dose Screening

Phase II: Implementation

In collaboration with clinical pharmacists, rule modifications were made and the system was subsequently deployed in June 2003 and September 2003 at facilities A and B, respectively. [triangle] shows the number of drugs for which modifications to the commercial rule base were made and compares the global alert rate before and after implementation. It was necessary to inactivate one or more rules for certain routes or dose forms in 114 drugs at facility A and 95 drugs at facility B due to idiosyncrasies of the pharmacy system order entry process. These peculiarities are common to many pharmacy systems and include continuous intravenous infusions (e.g., 100 mg morphine drip), bulk containers, and items entered as entire package sizes (e.g., 20 mg vecuronium) rather than the actual clinical dose received by the patient. Local domain experts modified the maximum allowable dose from the vendor's default for 119 drugs at facility A and 60 drugs at facility B.

Table 3.
Impact of Rule Modifications

[triangle] depicts the pre- and postimplementation alert rates for the top ten alerting drugs at each facility before any rule modifications. Categories of modification include: dose-limit multiplier of the commercial rule, inactivation of the rule, or a combination of both. For example, certain drugs may have the intravenous route inactivated due to entry in the pharmacy system as a continuous infusion but may have modified the oral dosage form rule with a dose-limit multiplier.

Table 4.
Results of Rule Modification on High-Volume Alert Drugs

The preimplementation alert rates of 12.3% at facility A and 6.0% at facility B were reduced to less than 1% postimplementation by rule modifications. This reduced the absolute monthly alert volume from 13,366 to 462 at facility A and from 2,413 to 270 at facility B.

For the 12 month period from July 2003 through June 2004, pharmacists received 7,190 alerts and contacted the physician to suggest a dosage change in 698 (9.7%) at facility A ([triangle]). Physicians changed the dose in 408 instances, or 58.5% of alerts presented to them. Likewise at facility B, from September 2003 through August 2004, pharmacists received 3,160 alerts and contacted physicians in 1,526 (42.3%). Facility B physicians changed the dose in 443 (29.0%) of the alerts presented to them.

Table 5.
Screening Results for First 12 Months Post-Deployment

Phase III: System Refinement

The system refinements installed after deployment at facility A in June 2003 resulted in a 50% relative reduction in alert volume (26 to 13 alerts per day). Dates for implementation and impact of these enhancements are shown in [triangle], which displays the weekly alert volume for facility A. Corresponding data for facility B are not as impressive since some refinements were already in place at the time of deployment in September 2003.

Figure 1.
Impact of refinements on weekly alert volume. Calc = calcium; CLcr = creatinine clearance.

Discussion

The decision to build a home-grown system rather than buy a commercial system using CDS rules is always a challenge. We had previous experience with building our own system, DoseChecker™, which allowed for ultimate customization but required substantial resources to develop rules for a comprehensive set of drugs. Indeed, in more than ten years of DoseChecker™ deployment, the list of drugs screened had grown to only 55. The advantages of a commercial rule base are (1) comprehensiveness, (2) regular updates including new drugs, and (3) larger pool of experts to draw on in authoring rules. The drawbacks of commercial rules are their conservative nature and high sensitivity, which results in frequent alerts. When commercial rules are implemented through a third-party vendor, there is usually little or no provision for customization. One could argue that standardization rather than customization is more desirable, but in reality, there may be no clear consensus for dosage in the medical literature and there is variation in commercial rules depending on the vendor chosen. While we did allow for customization by facility, 85% of drugs at facility A and 89% at facility B had no modifications. Therefore, this relatively small number of modifications allowed us to take advantage of the comprehensive nature of the commercial rule base while ameliorating its limitations for practical use.

To rapidly deploy a drug dosage safety net, we implemented maximum single-dose rules for more than 1,400 drugs from a commercially available rule base. A retrospective analysis of one month of orders was performed to identify rules with unacceptably high rates of alerts deemed clinically insignificant by clinical pharmacy experts. Two hospitals were selected as intervention sites based on adequate pharmacy staffing to handle alert volumes, funding for site licenses, and overall desire to deploy the new system. As can be seen in [triangle], only 222 drugs at facility A and 148 drugs at facility B required some form of modification, representing fewer than 15% of all drugs with rules.

Multiple factors contributed to the high alert rate obtained with direct implementation of an unaltered commercial rule base:

  1. Commercial rules tend to be conservative for medical/legal reasons and generally adhere to U.S. Food and Drug Administration–approved labeling. Hospitalized patients are more severely ill, and larger dosages than usually employed may be warranted to treat their conditions. Common examples are narcotics, antipsychotics, steroids, and oncology protocols.
  2. Pharmacy computer systems vary in the process by which drug orders are entered, and some entries do not accurately reflect the clinical dose administered to the patient. Examples include drugs for which entire packages are entered into the system rather than the actual clinical dose (e.g., large-volume parenterals, metered-dose inhalers, and other items entered for charge capture purposes).
  3. Clinically insignificant alerts may be caused by parameters that are marginally outside the threshold for triggering a warning. Examples are weight, CLcr, and body surface area.
  4. Duplicate alerts are generated when changes are made to a nondosage parameter such as schedule times or comments. Most systems generate a new sequential transaction number when any change to an order is made, resulting in a new order being screened and thus a duplicate alert from the clinician's perspective.

While vendors sometimes have rules for specific clinical conditions, these often cannot be reliably inferred from available electronic data. Examples include meningitis, hemodialysis, and bone marrow transplantation. In these cases, a more generalized or default rule must be used in an asynchronous environment.

There were large differences in the rates at which pharmacists contacted physicians at facilities A and B. Pharmacists at facility A, a university teaching hospital, only contacted housestaff physicians on rounds or via pagers. Pharmacists at facility B, a community hospital, used informal communication notes placed in the medical record to notify community physicians about suggested dosage changes for lower priority alerts. Facility A pharmacists, using the more interruptive method of notification, were less likely than facility B pharmacists to contact physicians about alerts, 9.7% (698/7,190) versus 42.3% (1,526/3,610). Of alerts for which physicians were contacted, 408 of 698 (58%) at facility A and 443 of 1,526 (29%) at facility B resulted in a dosage change. However, the rate of dosage changes based on number of alerts was greater at facility B (12.3%, 443/1,526) versus 5.7% (408/7,190) at facility A. More study is needed to assess physician attitudes toward acceptance of alerts and their delivery methods.

While we did not have CPOE in place to test our system directly with physicians, we believe that the lessons learned in deploying this system to clinical pharmacists have provided valuable information for the future implementation of CPOE. Commercial rule bases can provide an excellent source of knowledge; however, the conservative nature of their rules can frequently result in decision support overload.11 Our experience strongly suggests that developers of CDS applications must provide mechanisms to customize vendor-supplied rules to achieve an appropriate level of sensitivity and specificity. Speed has been repeatedly reported as the single most important parameter in CDS.12,13 Many failures in CPOE implementation have been reported because of added time required for physicians to enter orders, making CDS overload an even more significant issue.14

Conclusion

Automated screening of drug dosage has the potential to prevent adverse medication events and improve patient safety. However, implementing a comprehensive commercial rule set results in an unacceptably high volume of clinically insignificant alerts. A method for customizing vendor rules was devised while preserving their architecture for finer tuning of CDS in later implementation phases, without impacting the automated monthly rule update process. While the response to alerts was different at two institutions, this rule modification architecture was usable in two very different clinical environments and enabled those facilities to take advantage of a large rule set without requiring cross-institutional consensus, suggesting that this approach may be generalizable across many facilities and institutions. Although this strategy results in some additional development costs, it is offset by lower maintenance costs and more rapid implementation and significantly lower alert volumes for busy clinicians. Once a single-dose safety net is established, a more deliberate approach can be taken for expansion of CDS to include maximum daily dose, maximum frequency, minimum daily dose, maximum course of therapy, and maximum lifetime dose.

Appendix 1. Sample alert form

[triangle]

Notes

Presented in part as a poster at the 2003 Fall Meeting of the American Medical Informatics Association, Washington, DC.

Supported by AHRQ grant U18 HS11898-01

References

1. Kohn LT, Corrigan JM, Donaldson MS (eds). To err is human: building a safer health system. Washington, DC: National Academy Press; 1999.
2. Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6:313–21. [PMC free article] [PubMed]
3. Bobb A, Gleason K, Husch M, Feinglass J, Yarnold PR, Noskin GA. The epidemiology of prescribing errors. Arch Intern Med. 2004;164:785–92. [PubMed]
4. Institute of Medicine Committee on Quality of Health Care in America. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press; 2001.
5. Institute of Medicine Committee on Data Standards for Patient Safety. Patient safety: achieving a new standard of care. Washington, DC: National Academy Press; 2003.
6. McMullin ST, Reichley RM, Watson LA, Steib SA, Frisse ME, Bailey TC. Impact of a web-based clinical information system on cisapride drug interactions and patient safety. Arch Intern Med. 1999;159:2077–82. [PubMed]
7. McMullin ST, Reichley RM, Kahn MG, Dunagan WC, Bailey TC. Use of an expert system to identify potential dosing errors at a large university hospital. Am J Health Syst Pharm. 1997;54:545–9. [PubMed]
8. Gardner RM, Evans RS. Using computer technology to detect, measure, and prevent adverse drug events. J Am Med Inform Assoc. 2004;11:535–6. [PMC free article] [PubMed]
9. Peterson JF, Bates DW. Preventable medication errors: identifying and eliminating serious drug interactions. J Am Pharm Assoc. 2001;41:156–60. [PubMed]
10. Glassman PA, Simon B, Belperio P, Lanto A. Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care. 2002;40:1161–71. [PubMed]
11. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc. 2004;11:104–12. [PMC free article] [PubMed]
12. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective decision support: Making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10:523–30. [PMC free article] [PubMed]
13. Lee F, Teich JM, Spurr CD, Bates DW. Implementation of physician order entry: user satisfaction and usage patterns. J Am Med Inform Assoc. 1996;3:42–55. [PMC free article] [PubMed]
14. Chan W. Increasing the success of physician order entry through human factors engineering. J Health Info Manag. 2002;16:71–9. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • MedGen
    MedGen
    Related information in MedGen
  • PubMed
    PubMed
    PubMed citations for these articles
  • Substance
    Substance
    PubChem Substance links

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...