• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jamiaJAMIA - The Journal of the American Medical Informatics AssociationVisit this articleSubmit a manuscriptReceive email alertsContact usBMJ
J Am Med Inform Assoc. 2005 May-Jun; 12(3): 269–274.
PMCID: PMC1090457

A Trial of Automated Decision Support Alerts for Contraindicated Medications Using Computerized Physician Order Entry

Abstract

Background: Automated clinical decision support has shown promise in reducing medication errors; however, clinicians often do not comply with alerts. Because renal insufficiency is a common source of medication errors, the authors studied a trial of alerts designed to reduce inpatient administration of medications contraindicated due to renal insufficiency.

Methods: A minimum safe creatinine clearance was established for each inpatient formulary medication. Alerts recommending cancellation appeared when a medication order was initiated for a patient whose estimated creatinine clearance was less than the minimum safe creatinine clearance for the medication. Administration of medications in patients with creatinine clearances less than the medication's minimum safe clearance were studied for 14 months after, and four months before, alert implementation. In addition, the impact of patient age, gender, degree of renal dysfunction, time of day, and duration of housestaff training on the likelihood of housestaff compliance with the alerts was examined.

Results: The likelihood of a patient receiving at least one dose of contraindicated drug after the order was initiated decreased from 89% to 47% (p < 0.0001) after alert implementation. Analysis of the alerts seen by housestaff showed that alert compliance was higher in male patients (57% vs. 38%, p = 0.02), increased with the duration of housestaff training (p = 0.04), and increased in patients with worsening renal function (p = 0.007).

Conclusion: Alerts were effective in decreasing the ordering and administration of drugs contraindicated due to renal insuffiency. Compliance with the alerts was higher in male patients, increased with the duration of housestaff training, and increased in patients with more severe renal dysfunction.

Adverse drug events (ADEs) have been shown to contribute to the morbidity and mortality associated with the treatment of disease as well as the cost of care.1,2,3 A substantial portion of these ADEs are preventable, with estimates in the literature ranging from 20% to 69%.4,5,6,7 Preventable ADEs are often the result of medication errors.8 Thus, efforts to reduce medication errors have the ability to lower the rate of ADEs substantially and improve the overall delivery of health care.

Information and knowledge offered to the clinician to facilitate the best decision and thereby reduce medication errors is termed clinical decision support (CDS). Automated CDS systems transform clinical data gathered in an electronic medical record (EMR) as well as expert-based or evidence-based practice guidelines into useful patient-specific knowledge to assist clinical decision making. Recent findings show that computerized physician order entry (CPOE) in conjunction with basic CDS such as Drug–Allergy and Drug–Drug interaction checking decreases the likelihood of serious medication errors,9,10 although limited by clinician noncompliance with alert recommendations.11,12,13

Clinical decision support designed to reduce medication errors associated with renal dysfunction should be an efficacious intervention because the risk for error is high, and the CDS systems should be able to accurately assess a patient's renal function. One of the earliest reports of such a system was published by Rind et al.14 This study was not performed with CPOE and real-time alerts but demonstrated the utility of automated alerts to improve dosing in renal insufficiency as well as decrease the use of contraindicated medications. More recently, Chertow et al.15 published a study on real-time decision support delivered during CPOE and designed to improve dosing of renally excreted medications. The clinicians were provided a highlighted dosing suggestion as the default order when attempting to order. The intervention was shown to be beneficial but did not completely eliminate poor dosing due to clinician noncompliance with the dosing recommendations.

In a recent study by Oppenheim et al.,16 the intervention was designed to identify completed medication orders with inappropriate dosing and to suggest to clinicians that the order may require modification. This intervention was also found to be beneficial but, again, did not reduce all dosing errors because clinicians often did not comply with the alerts. These examples both show that CDS has great potential to reduce dosing errors in renal dysfunction; however, clinician noncompliance can limit the efficacy of these efforts.

We implemented and analyzed automated alerts designed to reduce the use of contraindicated drugs in patients with renal insufficiency. Our goals were twofold: evaluate the utility of these alerts as well as analyze the factors that may play a role in noncompliance with alert recommendations among housestaff. Understanding and overcoming alert noncompliance issues will be important in designing improved CDS systems and safer drug administration processes.

Methods

CPOE and CDS Environment

The University of Illinois Hospital and Medical Center utilizes a commercially available EMR (Millennium®, Cerner Corporation) which is used as the primary source of presentation of all results and orders to clinicians. All medication and laboratory orders are placed using CPOE, predominantly by housestaff. Nurses and pharmacists sometimes place medication orders based on a physician's verbal order. Although infrequent, attending physicians can directly place a medication order using CPOE. The commercially available automated CDS (Discern Expert®, Cerner Corporation) has been previously described.17,18,19

Development of Alerts

At each instance of a newly reported serum creatinine level, an estimated creatinine clearance (CrClest) was determined using the Cockcroft-Gault equation.20 Although usually available, there were instances in which entry of height was neglected or occurred after the serum creatinine level was reported. In these situations a normalized CrClest was determined based on only gender, serum creatinine level, and age.21 Determinations of CrClest were only made at the time of a newly reported serum creatinine level. The CrClest was used because it is a better estimate of renal function than the serum creatinine level,22 whereas the gold standard, a 24-hour creatinine clearance, is rarely available.

A group of pharmacists reviewed the entire inpatient hospital formulary and chose a safe CrClest for all drugs deemed to be potentially contraindicated for patients with renal insufficiency. The drugs along with their safe CrClest are shown in [triangle].

Table 1.
Potentially Contraindicated Drugs with Safe Renal Function Cutoffs

The alerts were designed to trigger when a clinician attempted to order one of the potentially contraindicated drugs for a patient whose most recent CrClest was less than the corresponding safe level for the drug. The clinician received a “pop-up” alert suggesting that they not proceed with the order due to the patient's renal insufficiency. In the alert message, the clinician is provided the most recent CrClest, the established “safe” CrClest value for each drug, and a pager number for the on-call pharmacist if the clinician had any questions regarding the alert. A picture of this type of alert is shown in [triangle].

Figure 1.
Example of a contraindication alert. This alert is displayed to the ordering clinician upon initiation of an order for a contraindicated medication only when the most recent CrClest is below a previously determined safe value. Clicking “OK” ...

Determination of Likelihood of Drug Administration before and after Alerts Implementation

To examine changes in the administration of contraindicated drugs in patients with renal insufficiency, a historical cohort was established for the four-month period before alert implementation. During this four-month control period, CrClest were available in the EMR for clinician review; however, no alerts were generated. During the subsequent 14-month study period not only were CrClest available for review, but alerts were also generated when a medication that was contraindicated due to renal insufficiency was ordered.

The control cohort was electronically mined for instances in which the alerts would have been generated if active, i.e., attempts to order a drug when the patient's most recent CrClest is less than the safe CrClest for the given drug. For all the instances in both the control and study periods, the electronic medical record was reviewed for record of administration of at least one dose of the medication associated with the order in question.

Determination of Alert Compliance and Characteristics of the Patient and Clinician

A log of all alerts is automatically recorded by the software. For each contraindication alert over the 14-month study period, the EMR was reviewed to determine if the alerting order was completed or cancelled before completion. Completion of the order was considered noncompliance with the alert recommendation. The time of day, patient age, gender, creatinine level, and CrClest were recorded as well as the type of clinician: housestaff, pharmacist, nurse, medical student or attending physician. Only the alerts received by housestaff were included in the analysis of compliance. The start date of the individual residents' training was recorded and compared with the date of the alert to calculate the duration of training.

Statistical Analysis

For the administration of contraindicated medications, the proportions of patients receiving at least one dose in the control and study periods were compared using a χ2 test. Characteristics of the control and study groups were compared using a Student's t-test for continuous variables and the χ2 statistic for proportions. A p-value of 0.05 was chosen for statistical significance of all comparisons.

Analysis of the housestaff compliance with the alert recommendations was performed using multiple logistic regression. The dependent variable was the compliance with the alert. The independent variables were the time of day, patient gender, patient age, patient's renal function, and length of training of housestaff. The CrClest was used as the measure of renal function. Based on initial trends using bivariate analysis, the time of day was described by a dichotomous variable, either 7 am to 1 am or 1 am to 7 am. The training was categorized as either ≤365 days, or >365 days.

Because some clinicians received more than one alert, a clustering effect based on clinician was possible. This was accounted for by using the generalized estimating equations modification of logistic regression analysis.23

Results

Over the 14-month study period, 323 alerts were generated. In the four-month control period there were 87 situations that would have generated an alert if the alerts had been implemented at that time. In both time periods the most commonly ordered drug was metformin. Features of the control and study groups are presented in [triangle]. The control and study periods did not differ with respect to patient gender, age, or degree of renal dysfunction. The likelihood of a patient receiving at least one dose of the contraindicated medication decreased from 89% to 47% after alert implementation (p < 0.0001), a 42% absolute reduction in administration. This reduction was almost entirely due to cancellation of the order after viewing the alert, which occurred in 41% of instances in which an alert was given.

Table 2.
Group Characteristics and Administration Likelihood

The distribution of clinicians receiving the alerts was 70% medical housestaff, 13% nurses, 11% pharmacists, 4% medical students, and 2% attending physicians. There were a total of 233 unique patients. For the 226 alerts received by housestaff, the alert compliance rate was 42%; for the remaining clinicians the compliance rate was 38% (p = 0.54).

For the remainder of the analysis of compliance, only the alerts seen by housestaff were analyzed further, investigating some of the factors affecting compliance in housestaff using multivariate analysis. For this subset of 226 alerts, the average age of the patient was 66 years, and 77% were women. The analysis looked at both patient-specific as well as non–patient-specific factors. Among the patient-specific factors evaluated, age was not found to be associated with alert compliance (p = 0.32). Patient gender, however, was found to be associated with the alert compliance rate; with compliance in female patients lower than that in male patients, 38% vs. 57% (p = 0.02). Alert compliance also decreased with improving renal function or increasing CrClest (p = 0.007).

The time of day was not found to be significantly associated with the likelihood of alert compliance. However, the duration of housestaff training was found to be associated with the likelihood of alert compliance. Housestaff having more than one year of training had a compliance rate of 50%, whereas those with less than one year of training had a compliance rate of 37% (p = 0.04). The relationship is shown in [triangle].

Figure 2.
Compliance versus duration of housestaff training. The y-axis represents the percentage of housestaff compliant with the alert recommendation to discontinue an order for a contraindicated drug. The x-axis represents the duration of housestaff training ...

Discussion

This study found that real time “pop-up” CDS alerts generated during CPOE were able to reduce the number of medication errors by decreasing both the completion of medication orders and administration of medications that were considered to be contraindicated due to renal insufficiency. This intervention, however, was limited by noncompliance with the alert recommendations as has been shown in many other trials of CDS alerts with CPOE.11,12,13,14,15,16,19,24,25,26

It is interesting to note that noncompliance with alerts for renal dysfunction has been a problem regardless of the type or strategy of decision support implementation. Chertow et al.15 did not generate alerts at the time of order but rather modified the default suggested dose and frequency of renally cleared medications. This intervention was successful but limited by significant noncompliance. In the intervention reported by Oppenheim et al.,16 no warnings or dosing assistance was initially given; however, upon completion of the order, alerts were generated if a suspected dosing error occurred. This intervention was also efficacious but again limited by noncompliance. Our intervention, which generated warnings at the time of order initiation, was efficacious but was also limited by noncompliance. Compliance is, therefore, an important element to consider in CDS implementation strategies because it clearly limits efficacy. An understanding of the factors that promote or diminish compliance needs to occur to fully realize the benefits of CDS.

The problem of noncompliance with alerts for medication orders has been studied in the past.11,12,13,27 Some of the causes determined in the earlier studies are likely related to a portion of the noncompliance observed in our study. In the earlier studies, re-ordering or renewal of orders for previously given medications has been shown to increase noncompliance.11,12,13,27 In our study this also may have been the case as many of the potentially contraindicated medications may have been previously given to the patient either earlier in the admission or in the ambulatory setting. This seems particularly likely for the most commonly alerted medication, metformin.

Noncompliance can also be associated with a compelling indication for the medication despite the risk.12 Such careful consideration of risk–benefit likely was not a large factor in our study as the most commonly alerted medications, metformin and nonsteroidal anti-inflammatory medications, are not life-saving inpatient medications and have safer alternatives in patients with renal dysfunction.

Noncompliance can also result when the ordering clinician has knowledge of clinical information that the CDS system does not. This relevant clinical information may not be present in the EMR or may be stored in a format not accessible to the CDS. The CDS alerts in our study were based on only the assessment of renal function and the drug being ordered. Because serum creatinine is frequently ordered during the course of inpatient care, the CDS system and clinician were both likely considering the same information at the time of medication order entry.

Our analysis looked at some additional factors that are associated with noncompliance for our contraindication alerts. An important finding in our study was that in the case of physician housestaff, increased duration of training improved the compliance to the alert recommendations. This finding was also reported by Oppenheim et al.16; however, the exact methodology of the comparison was not specified. Although our study was not designed to determine the reason for this finding, it is interesting to postulate a cause. The alert in this study suggests to clinicians that they are making a medication error based on a patient's renal insufficiency.

Our alerts provided two elements of information: the first is patient related (i.e., the presence of renal insufficiency based on a calculated CrClest) and the second is drug related (i.e., the drug being ordered is not safe to use in renal insufficiency). It is reasonable to assume that a clinician would comply with the alert recommendations if they understand and accept as true these two elements of information being provided.

As they advance through training, housestaff members are more likely to have been exposed to the use of the calculated CrClest as an estimate of renal function and have acquired knowledge of drugs that are not safe to use in patients with renal insufficiency. For the knowledgeable housestaff, the alerts may simply remind the physician of the presence of renal insufficiency in a given patient, a piece of data that may have been forgotten or never reviewed before ordering. Housestaff members with less experience may not fully understand estimates of renal function and are less likely to know which drugs are safe in patients with renal insufficiency. A mistrust or misunderstanding of the alert information, particularly the non–patient-specific element, may cause the clinician to ignore the alert recommendation. This hypothesis can be tested in future studies by surveying clinicians receiving alerts and measuring their reasons for either complying with or ignoring alerts.

An unexpected finding in our study was the relationship between patient gender and noncompliance. Housestaff was more likely to listen to the suggestion to cancel an order when contraindicated in a male patient than in a female patient. The degree of renal dysfunction was the same in the male and female patients, 37 mL/min, as was the age, 66 years (p > 0.5 in both). The average creatinine level of the female patients was, however, lower than that of the male patients; 1.7 ± 1.4 mg/dL vs. 2.8 ± 2.7 mg/dL (p = 0.0002). One possible explanation for this gender-based difference is that some clinicians may be more trusting of the serum creatinine level than the CrClest as determined by the Cockroft-Gault equation.20 This would produce the tendency to disregard a low CrClest when the creatinine level is listed as normal. This situation is more likely to occur in female patients due to smaller average ideal body weight and a gender-based factor of 0.85 in the Cockroft-Gault equation20; thus, the tendency to trust the creatinine level over the CrClest would produce a gender-based difference in compliance. This finding suggests an increased need for educating housestaff that the use of CrClest as an assessment of renal function is preferred over serum creatinine level.22

The positive association between compliance and the degree of renal dysfunction is not surprising. In a study by Dexter et al.,28 compliance with reminders for preventive care in inpatients showed an association with the strength of the indication. Similarly, compliance with our contraindication alerts improved with the increased risk associated with decreasing renal function.

Although the alerts were statistically beneficial, they did not completely eradicate the ordering of medications contraindicated due to renal insufficiency. Our institution chose to institute additional alerts that are triggered by evidence of noncompliance or override. We have reported on such alerts in the past, implemented for digoxin use.17 These additional alerts are triggered when a clinician completes an order for a contraindicated medication when a given patient's CrClest is < 67% of the threshold CrClest that is considered safe for the given medication. This alert is sent to both the pharmacy and the nursing unit suggesting that the drug not be administered without an explanation from the clinician as to why the alert was disregarded. We hope that these additional alerts will further decrease the use of contraindicated medications and plan on studying their efficacy.

Limitations

The institution studied is a public teaching hospital with the majority of orders place by housestaff physicians. It is not clear to what extent these findings are applicable to facilities in which attending physicians perform the majority of the medication ordering. It would not be surprising to find that attending physicians have similar behavior as related to patient gender and estimates of renal function, but this of course needs to be verified experimentally.

The study examined only one type of alert: contraindication due to renal insufficiency. This was done as actions related to this type of alert are simple to interpret: compliance (medication not administered) or noncompliance (medication administered). Outcomes associated with alerts for dosing, relative contraindications, drug side effects, or interactions are much more ambiguous to interpret. There is a continuing need for studies with a broad variety of alerts to further enhance the knowledge in this field.

Conclusions

We developed and implemented CDS alerts to decrease the use of contraindicated medications in patients with renal insufficiency. The alerts were successful in decreasing the likelihood of clinicians completing contraindicated orders and decreased the administration of these medications. The efficacy of the alerts was limited by clinician noncompliance. Housestaff was less compliant when alerted for female patients than for male patients, whereas alert compliance improved with worsening patient renal function and increasing duration of housestaff training. Due to the problem of noncompliance, it is likely that as CDS evolves, a variety of alert mechanisms and forms of decision support will be used together to ensure appropriate patient safety while allowing for clinician education and autonomy.

Notes

Portions of the work presented at the Society of General Internal Medicine Annual Meeting, Vancouver, May 2003.

Dr. Polikaitis was an employee of Cerner Corporation at the time the trial was performed and Cerner supported his efforts. Drs. Didomenico and Galanter have no financial conflicts.

The authors thank Amy Looi, RN, for technical assistance in the CDS alert development and Dr. Ahsan Arozullah for assistance with the statistical analysis.

References

1. Bates DW, Spell N, Cullen DJ, et al. The costs of adverse drug events in hospitalized patients. JAMA. 1997;277:307–11. [PubMed]
2. Classen DC, Pestotnik SL, Evans S, et al. Adverse drug events in hospitalized patients. JAMA. 1997;277:301–6. [PubMed]
3. Johnson JA, Bootman JL. Drug-related morbidity and mortality: a cost-of-illness model. Arch Intern Med. 1995;155:1949–56. [PubMed]
4. Bates DW, Boyle DL, Vander Vliet MD, et al. Relationship between medication errors and adverse drug events. J Gen Intern Med. 1995;10:199–205. [PubMed]
5. Bates DW, Cullen D, Laird N, et al. Incidence of adverse drug events and potential adverse drug events; implications for prevention. JAMA. 1995;274:29–34. [PubMed]
6. Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drug events in hospitalized adults. J Gen Intern Med. 1993;8:289–94. [PubMed]
7. Leape LL, Lawthers AG, Brennan TA, Johnson WG. Preventing medical injury. Qual Rev Bull. 1993;19:144–9. [PubMed]
8. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric patients. JAMA. 2001;285:2114–20. [PubMed]
9. Bond CA, Raehl CL, Franke T. Medication errors in United States hospitals. Pharmacotherapy. 2001;21:1023–36. [PubMed]
10. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:1311–6. [PubMed]
11. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163:2625–31. [PubMed]
12. Hsieh TC, Kuperman GJ, Jaggi T, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc. 2004;11:482–91. [PMC free article] [PubMed]
13. Payne TH, Nichol WP, Hoey P, Savarino J. Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp. 2002:602–6. [PMC free article] [PubMed]
14. Rind DM, Safran C, Phillips RS, et al. Effect of computer-based alerts on the treatment and outcomes of hospitalized patients. Arch Intern Med. 1994;154:1511–7. [PubMed]
15. Chertow GM, Lee J, Kuperman GJ, et al. Guided medication dosing for inpatients with renal insufficiency. JAMA. 2001;286:2839–44. [PubMed]
16. Oppenheim MI, Vidal C, Velasco FT, et al. Impact of a computerized alert during physician order entry on medication dosing in patients with renal impairment. Proc AMIA Symp. 2002:577–81. [PMC free article] [PubMed]
17. Galanter WL, DiDomenico J, Polikaitis A. Use of an automated decision support system to prevent exacerbation of an adverse drug event. J Healthc Inf Manag. 2002;16(4):44–9. [PubMed]
18. Raschke RA, Gollihare B, Wunderlich TA, et al. A computer alert system to prevent injury from adverse drug events. JAMA. 1998;280:1317–20. [PubMed]
19. Galanter WL, Didomenico R, Polikaitis A. Effectiveness of automated clinical decision support alerts for inpatient digoxin use with computerized physician order entry. J Am Med Inform Assoc. 2004;11:270–7. [PMC free article] [PubMed]
20. Cockcroft DW, Gault MH. Prediction of creatinine clearance from serum creatinine. Nephron. 1976;16:31–41. [PubMed]
21. Hull JH, Hak LJ, Koch GG, et al. Influence of range of renal function and liver disease on predictability of creatinine clearance. Clin Pharmacol Ther. 1981;29:516–21. [PubMed]
22. Levey AS, Coresh J, Balk E, et al. National Kidney Foundation practice guidelines for chronic kidney disease: evaluation, classification, and stratification. Ann Intern Med. 2003;139:137–47. [PubMed]
23. Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes. Biometrics. 1986;42:121–30. [PubMed]
24. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other anti-infective agents. N Engl J Med. 1998;338:232–8. [PubMed]
25. Nightingale PG, Adu D, Richards NT, Peters M. Implementation of rules based computerised bedside prescribing and administration: intervention study. BMJ. 2000;320:750–3. [PMC free article] [PubMed]
26. Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of “corollary orders” to prevent errors of omission. J Am Med Inform Assoc. 1997;4:364–75. [PMC free article] [PubMed]
27. Abookire SA, Teich JM, Sandige H, et al. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp. 2000:2–6. [PMC free article] [PubMed]
28. Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med. 2001;345:965–70. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • Compound
    Compound
    PubChem Compound links
  • MedGen
    MedGen
    Related information in MedGen
  • PubMed
    PubMed
    PubMed citations for these articles
  • Substance
    Substance
    PubChem Substance links

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...