• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jamiaJAMIA - The Journal of the American Medical Informatics AssociationInstructions for authorsCurrent TOC
J Am Med Inform Assoc. 2006 Jul-Aug; 13(4): 385–390.
PMCID: PMC1513672

Prescribers' Responses to Alerts During Medication Ordering in the Long Term Care Setting

James Judge, MD, b Terry S. Field, DSc, a Martin DeFlorio, RPh, b Jane Laprino, b Jill Auger, RPh, a Paula Rochon, MD, MPH, c David W. Bates, MD, MSc, d and Jerry H. Gurwitz, MD a , *

Abstract

Objective

Computerized physician order entry with clinical decision support has been shown to improve medication safety in adult inpatients, but few data are available regarding its usefulness in the long-term care setting. The objective of this study was to examine opportunities for improving medication safety in that clinical setting by determining the proportion of medication orders that would generate a warning message to the prescriber via a computerized clinical decision support system and assessing the extent to which these alerts would affect prescribers' actions.

Design

The study was set within a randomized controlled trial of computerized clinical decision support conducted in the long-stay units of a large, academically-affiliated long-term care facility. In March 2002, a computer-based clinical decision support system (CDSS) was added to an existing computerized physician order entry (CPOE) system. Over a subsequent one-year study period, prescribers ordering drugs for residents on three resident-care units of the facility were presented with alerts; these alerts were not displayed to prescribers in the four control units.

Measurements

We assessed the frequency of drug orders associated with various categories of alerts across all participating units of the facility. To assess the impact of actually receiving an alert on prescriber behavior during drug ordering, we calculated separately for the intervention and control units the proportion of the alerts, within each category, that were followed by an appropriate action and estimated the relative risk of an appropriate action in the intervention units compared to the control units.

Results

During the 12 months of the study, there were 445 residents on the participating units of the facility, contributing 3,726 resident-months of observation time. During this period, 47,997 medication orders were entered through the CPOE system—approximately 9 medication orders per resident per month. 9,414 alerts were triggered (2.5 alerts per resident-month). The alert categories most often triggered were related to risks of central nervous system side-effects such as over-sedation (20%). Alerts for risk of drug-associated constipation (13%) or renal insufficiency/electrolyte imbalance (12%) were also common. Twelve percent of the alerts were related to orders for warfarin. Overall, prescribers who received alerts were only slightly more likely to take an appropriate action (relative risk 1.11, 95% confidence interval 1.00, 1.22). Alerts related to orders for warfarin or central nervous system side effects were most likely to engender an appropriate action, such as ordering a recommended laboratory test or canceling an ordered drug.

Conclusion

Long-term care facilities must implement new system-level approaches with the potential to improve medication safety for their residents. The number of medication orders that triggered a warning message in this study suggests that CPOE with a clinical decision support system may represent one such tool. However, the relatively low rate of response to these alerts suggests that further refinements to such systems are required, and that their impact on medication errors and adverse drug events must be carefully assessed.

Introduction

Adverse drug events occur commonly in the long-term care setting and as many as half of these may be preventable if errors in prescribing and monitoring drug therapy could be averted. 1,2 Errors that have been found to be responsible for substantial numbers of adverse drug events include suboptimal choices of medications in frail older adults, excessive dosing of psychoactive medications, failure to monitor nursing home residents to avoid the development of renal insufficiency, and errors in the management of residents on warfarin therapy. Residents of long-term care facilities taking medications in several categories including antipsychotics, anticoagulants, diuretics, and antiepileptics have been identified as being at particularly high risk for experiencing preventable adverse drug events (i.e., related to medication errors). 2 In the hospital setting, computer-based clinical decision support accompanying computerized physician order-entry systems has been demonstrated to reduce the rate of serious errors in prescribing and monitoring medications. 3,4

Computerized physician order entry refers to various computer-based systems that share the common features of automating the medication ordering process. 5 Clinical decision support is built into many computerized physician order entry systems. Clinical decision support can provide computerized advice regarding drug doses, routes, and frequencies. Additionally, clinical decision support systems can perform drug allergy checks, drug-laboratory value checks, and drug-drug interaction checks. Clinical decision support systems can also prompt clinicians regarding corollary orders, such as the performance of an international normalized ratio (INR) in a patient on warfarin who is newly prescribed an interacting antibiotic. 5

The impact of computerized clinical decision support on improved medication prescribing and monitoring depends on the extent to which physicians act on the alerts presented. A number of studies in the in-patient and ambulatory settings have assessed the response of physicians to alerts related to a variety of issues including medication prescribing, 6–12 corollary orders, 11,13 and adherence to prevention guidelines 14 and healthcare quality standards. 15 Most of these studies have found low response rates or high “override” rates as summarized in a recent review. 16 Few of these studies have compared prescribers who received alerts to those not receiving them and most have limited their analysis to a single recommended action, such as cancellation of a medication order.

We developed a computer-based clinical decision support system for the long-term care setting based on evidence derived from observational studies of preventable adverse drug events, consensus recommendations for the appropriate use of medications in geriatric patients, and known high-risk drug-drug interactions. 17 Within a randomized trial of this system, we examined opportunities for improving the safety of medication use in the long-term care setting by determining the proportion of medication orders that would generate an alert to the prescriber via a computerized clinical decision support system and assessed the likelihood that the alerts would have an impact on medication prescribing and orders for corollary laboratory tests.

Methods

Setting

This study was conducted in the long-stay units of a large, academically-affiliated long-term care facility with four years of experience using computerized physician order entry (CPOE) without a computer-based clinical decision support system (CDSS). At the time of the study, approximately 90% of new medication orders were entered using CPOE. All medication prescribing was performed by a staff of contracted physicians, nurse practitioners, and physicians' assistants. This group of providers averaged 27 during the period of the study, including nine physicians providing on-site care during the day (six internists and three psychiatrists), seven internists who covered evenings, and 11 nurse practitioners and physicians' assistants. Telephone orders were entered by nurses; these orders were excluded from the study. In March 2002, a CDSS was added to the CPOE system. Over a subsequent one-year study period, prescribers ordering drugs for residents on three resident-care units of the facility were presented with alerts in the form of warning messages; however, these alerts were not displayed to prescribers on the four control units.

Design of the Clinical Decision Support System (CDSS)

The CDSS was designed by a team of geriatricians, pharmacists, health services researchers, and information systems professionals; the process of developing the system has been described previously. 17,18 The design principles were: 1) messages should be evidence-based; 2) messages should be perceived as useful and informative by practitioners; and 3) the system should have a minimal impact on the time required for the practitioner to complete an order.

The team reviewed the types of preventable adverse drug events based on previous research, 1,2 as well as widely accepted published criteria for suboptimal prescribing in the elderly available at the time of this study. 19 We also reviewed all serious drug-drug interactions from standard pharmaceutical drug interaction databases and included alerts for a limited number of more than 600 potential serious interactions that were reviewed. Reasons for exclusion of alerts for specific drug interactions included that the medications were not on formulary at the facility or the medications were never used in elderly patients or the long-term care setting.

The CDSS system was designed to provide alerts in response to specific drug orders whenever the order: 1) involved potential high-severity drug interactions; 2) was for a patient with a recent abnormal lab test result that suggested a possible danger related to use of the ordered medication; 3) could lead to adverse effects that require special monitoring in order to identify them early; 4) should within certain dose ranges to reduce the risk of adverse effects in elderly patients; or 5) should be accompanied by prophylactic measures to proactively address situations where there was a high likelihood of adverse drug effects (e.g., constipation with opioid use). Alerts included specific instructions for laboratory monitoring, as well as less explicit recommendations for reconsidering drug orders and monitoring for possible side effects.

The CPOE system in place at the time of the CDSS implementation was capable of linking laboratory test orders and results and current drug orders in real-time. However, the system had several limitations. It was not capable of combining dose and strength information to determine the total daily dose associated with a drug order; therefore, some alerts may not have been necessary (e.g., the medication order was already within the recommended dose range). The underlying software was not capable of distinguishing multiple orders for the same drug in different forms or strengths, or orders that had been cancelled and re-ordered within the same prescribing session. These orders were interpreted as multiple orders for drugs in the same category and triggered a number of inaccurate alerts about drug interactions. For example, an order for erythromycin that the prescriber initiated, cancelled, and then re-ordered within the same session of CPOE would be interpreted as an interaction signaling the need for an alert for increased risk of QT prolongation. Despite the fact that some triggers were likely to produce a substantial number of these unnecessary alerts, we opted to include them in the CDSS if the potential impact of the type of drug interaction in question was considered clinically important.

The facility's long-stay resident units were randomly assigned to intervention (n=3) or control (n=4) status. For residents on the intervention units, the alerts were displayed in a pop-up box to prescribers in real-time when they entered a drug order. The pop-up boxes were informational; they did not require specific actions from the prescribers and did not produce or revise orders automatically. For residents on the control units, the alerts were recorded in an audit trail file, but were not displayed to the prescribers. [triangle] [triangle] provide examples of pop-up box message displays.

Figure 1
Republished with permission by Blackwell Publishing, and previously published in the Journal of the American Geriatrics Society 2005; 53:1780–89 and by CMA Media Inc., previously published in “Clinical application of a computerized system ...
Figure 2
Republished with permission by Blackwell Publishing, and previously published in the Journal of the American Geriatrics Society 2005; 53:1780–89.

Tracking Alerts

Every medication order entered using the CPOE system during the 12-month study period from March 1, 2002 to March 1, 2003 was processed through the CDSS system, including orders for residents in both the intervention and control units. The CDSS did not process routine medication renewals (e.g., 30- or 60-day scheduled renewals), medication adjustments made by physicians on the monthly medication renewal order sheets, or telephone orders for medications.

On a bi-weekly basis, electronic audit trails of all alerts triggered by the CDSS were output, together with electronic files of all medication-related actions, and orders for laboratory tests. These files were linked by encrypted patient identifiers. A complete history of medication orders, changes to orders, and laboratory test orders around the time of each warning message was prepared and output for review by the study pharmacist (JA). These audit trails were related to both alerts that were actually shown to prescribers in the intervention units and alerts that were triggered within the system but not displayed to prescribers on the control units. For each warning message, the pharmacist identified all actions during the drug ordering session that could have been a response to a recommendation in the message. Potentially appropriate actions by the prescriber included: 1) cancellation of a drug order; 2) replacement of an ordered drug with another choice; 3) change in dose; 4) order for a recommended laboratory test; or 5) order for a recommended additional drug. Responses to certain alerts could not be evaluated because the recommended tests were not ordered electronically, such as fingerstick tests for blood sugar levels and bladder scans recommended in association with the initiation of certain drug therapies for management of urinary incontinence. The pharmacist was blinded to the intervention status of the units.

Analysis

There were forty-one different alerts in the CDSS. The full list of alerts in the CDSS was previously published in the Journal of the American Geriatrics Society. 17 We collapsed these into 13 categories (see [triangle]). Prior to further analysis, the study pharmacist manually excluded from further analysis alerts regarding dose recommendations when the dose ordered was already within the recommended range, alerts related to drug interactions when the two drugs that triggered the warning were actually the same drug, and alerts recommending the ordering of a laxative when that treatment had already been included in a standing order for the resident. We included the remaining alerts in the analyses. We first assessed the frequency of drug orders associated with the various categories of alerts across all long-stay units of the facility. To assess the impact of actually viewing a warning message on prescriber behavior during drug ordering, we calculated separately for the intervention and control units the proportion of the alerts, within the respective categories, that were followed by an appropriate action and estimated the relative risk of an appropriate action in the intervention units compared to the control units, together with the 95% confidence interval for each category.

Table 1
Table 1. Distribution of Alerts (N = 4,282)

Results

During the 12 months of the study, there were 445 residents on the long-stay units of the facility, who contributed 3,726 resident-months of observation time. During this period, 47,997 medication orders were entered through the CPOE system—approximately nine medication orders per resident per month. During the processing of the 47,997 orders, 9,414 alerts were triggered (2.5 alerts per resident-month).

The study pharmacist manually excluded from further analysis 5,132 (54%) alerts: 4,790 alerts relating to dose recommendations when the dose ordered was already within the recommended range; 238 alerts relating to drug interactions when the two drugs that triggered the warning were actually the same drug; and 104 alerts recommending orders for a laxative in the circumstance of an order for an opioid, when a laxative was already included as a standing order. [triangle] describes the distribution of the remaining 4,282 alerts by category. The most frequent types of alerts were related to risks of central nervous system side-effects such as over-sedation (20% of all alerts). Alerts for risk of drug-associated constipation (13%) or renal insufficiency/electrolyte imbalance (12%) were also common. Twelve percent of alerts were related to orders for warfarin.

[triangle] presents the number of alerts for the intervention and control units in each category with the percent of alerts that were associated with appropriate follow-up actions by the prescribers. Overall, prescribers who received alerts were significantly more likely to take an appropriate action (relative risk 1.11, 95% confidence interval 1.00, 1.22). Alerts related to orders for warfarin and those related to potential central nervous system side effects were most likely to engender an appropriate action, such as ordering a recommended laboratory test or canceling an ordered drug. For alerts related to dose recommendations and drug interactions, prescribers who saw the alerts were slightly, but not significantly more likely to take an appropriate action. Viewing the alerts in other categories was not associated with an increased likelihood of taking a recommended action.

Table 2
Table 2. Actions by Ordering Physicians on Intervention and Control Units

Discussion

We examined the rates of alerts generated by a computerized clinical decision support system in the long-term care setting and the responses of prescribers to these alerts. An average of 2.5 alerts were generated per resident-month, of which a little under half were directly relevant to the medication order. Our findings indicate that the average nursing home with 100 residents would have 115 opportunities per month to improve the safe use of medications using this clinical decision support system. The impact of the alerts was modest; overall, 31% of alerts were followed by an appropriate action compared to 28% in units where the alerts were not displayed to prescribers. The alerts significantly affected therapeutic management decisions in only two domains: orders related to warfarin and those for drugs with potential central nervous system side effects.

These two categories of alerts were commonly triggered by the system. Problematic prescribing of psychoactive medications in nursing homes has long been an issue of concern, 20 leading to federal regulations about their use in the setting. 21 Various interventions have been developed and tested to reduce the risks of excessive psychoactive drug use in nursing homes. 22 Academic detailing to educate physicians and nursing home staff about the use of psychoactive medications in geriatric patients has had some success in improving the quality of drug prescribing; 23,24 however, academic detailing efforts are expensive and often difficult to sustain over the long-term. Although the CDSS resulted in modification of only 17% of orders for drugs with central nervous system side effects, this was a substantial improvement over the control units. Warfarin requires careful monitoring and specific attention to the use of the large number of drugs with which it may interact. Alerts from the CDSS were associated with substantially higher rates of appropriate monitoring and changes in drug choices and doses. These two impacts highlight the potential value of CDSS in the long-term care setting.

The computerized clinical decision support system utilized in this study was not technologically sophisticated. It did not offer the capacity to add several features recommended for CDSS 25 such as offering alternative orders within the alerts that could be directly accepted by prescribers. It is probably most appropriate to consider this a first-generation system, as over half of the alerts were determined to be unnecessary based on retrospective review by a clinical pharmacist. This was primarily related to the inability of the system to assess the total 24 hour dose of a drug that was already in use and relate it to the recommended dose range, and to recognize prior medication orders leading to unnecessary warnings about drug interactions and recommendations for therapies (e.g., laxatives in the setting of opioid use) when they had already been ordered. Healthcare providers using CPOE should not be required to see displayed alerts that are not directly relevant to their orders. Time is a major factor impacting on receptivity to use of these systems by the busy practitioner. If these systems substantially increase the time required to carry out a task, their potential benefits for improving patient safety may be subverted by their inefficiencies. Moreover, high signal to noise ratios may produce alert fatigue and lead prescribers to flip past alerts without considering or even reading them. 16 Despite these limitations, clinicians in this facility voted to retain the alerts at the end of the study and expanded their use to the control units.

Previous studies of alerts in other settings have found a range of responses to computerized alerting systems, with most finding low rates. 6,7,8,9,11,12 Our study differs in several ways. Very few studies have assessed a range of potentially appropriate activities following display of an alert, such as consequent orders or changes in related medication orders. 10 Few have been set within randomized trials that allow direct comparison of responses to alerts in intervention and control units, 15,13,26,5 and many of these focused on specific drug dosing issues. To our knowledge, none have been conducted in the long-term care setting.

This study has several limitations. There was potential for contamination by cross-over between intervention and control units since clinicians exchanged duties and covered for each other extensively. To assess the possibility that this may have led to changes in prescribing and orders for corollary laboratory tests in the control units, we assessed the rate of responses to “unseen” alerts in the control units during the first vs. the last quarter of the year. The rate of response was actually lower in the last quarter, suggesting that prescribers did not adopt new habits due to seeing alerts in the intervention units. This is consistent with a previous research study that found physicians who had received alerts had no better knowledge of the issues highlighted in the alerts at the end of a one-year period than they had at the beginning. 27 In addition, the present study was carried out in a single nursing home with substantial information technology infrastructure, which makes it quite different from most long-term care facilities across the United States at the present time. 28

Long-term care residents are at high risk for experiencing adverse drug events. Successful implementation of CPOE adapted for the long-term care setting may have the potential to reduce this risk. The experience described in this study suggests that there are many opportunities for improving medication safety in the long-term care setting. Alerts did result in modest improvement in two areas (relating to warfarin and central nervous system medications), but not for many others. Further refinement of computerized clinical decision support systems for use in the long-term care setting is essential in order to increase its potential impact on medication safety.

Footnotes

This research was supported by Grants from the Agency for Healthcare Research and Quality (HS010481 and HS15430).

References

1. Gurwitz JH, Field TS, Avorn J, et al. Incidence and preventability of adverse drug events in nursing homes Am J Med 2000;109:87-94. [PubMed]
2. Gurwitz JH, Field TS, Judge J, et al. The incidence of adverse drug events in two large academic long-term care facilities Am J Med 2005;118:251-258. [PubMed]
3. Bates DW, Gawande AA. Improving safety with information technology N Engl J Med 2003;348:2526-2534. [PubMed]
4. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors JAMA 1998;280:1311-1316. [PubMed]
5. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safetyA systematic review. Arch Intern Med 2003;163:1409-1416. [PubMed]
6. Abookire SA, Teich JM, Sandige H, et al. Improving allergy alerting in a computerized physician order entry system Proc AMIA Symp 2000:2-6. [PMC free article] [PubMed]
7. Hsieh TC, Kuperman GJ, Jaggi T, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system J Am Med Inform Assoc 2004;11:482-491. [PMC free article] [PubMed]
8. Nightingale PG, Adu D, Richards NT, Peters M. Implementation of rules based computerised bedside prescribing and administrationintervention study. BMJ 2000;320:750-753. [PMC free article] [PubMed]
9. Payne TH, Nichol WP, Hoey P, Savarino J. Characteristics and override rates of order checks in a practitioner order entry system Proc AMIA Symp 2002:602-606. [PMC free article] [PubMed]
10. Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care J Am Med Inform Assoc 2006;13(1):5-11Jan–Feb. [PMC free article] [PubMed]
11. Steele AW, Eisert S, Witter J, et al. The effect of automated alerts on provider ordering behavior in an outpatient setting PLoS Med 2005;2(9):e255Sep. [PMC free article] [PubMed]
12. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care Arch Intern Med 2003;163:2625-2631. [PubMed]
13. Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of “corollary orders” to prevent errors of omission J Am Med Inform Assoc 1997;4:364-375. [PMC free article] [PubMed]
14. Litzelman DK, Tierney WM. Physicians' reasons for failing to comply with computerized preventive care guidelines J Gen Intern Med 1996;11:497-499. [PubMed]
15. Apkon M, Mattera JA, Lin Z, et al. A randomized outpatient trial of a decision-support information technology tool Arch Intern Med 2005;165:2388-2394. [PubMed]
16. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry J Am Med Inform Assoc 2005. Dec 22 epub. [PMC free article] [PubMed]
17. Rochon PA, Field TS, Bates DW, et al. Computerized physician order entry with clinical decision support in the long-term care settinginsights from the Baycrest Centre for Geriatric Care. J Am Geriatr Soc 2005;53:1780-1789. [PubMed]
18. Rochon PA, Field TS, Bates DW, et al. Clinical application of a computerized system for physician order entry with clinical decision support to prevent adverse drug events in long-term care CMAJ 2006;174:52-54. [PMC free article] [PubMed]
19. Beers MH. Explicit criteria for determining potentially inappropriate medication use by the elderlyAn update. Arch Intern Med 1997;157:1531-1536. [PubMed]
20. Rango N. Nursing home care in the United States N Engl J Med 1982;207:883-889. [PubMed]
21. Elon R, Paulson LG. The impact of OBRA on medical practice within nursing facilities J Am Geriatr Soc 1992;40:958-963. [PubMed]
22. Ray WA, Taylor JA, Meador KG, et al. A randomized trial of a consultation service to reduce falls in nursing homes JAMA 1997;278:557-562. [PubMed]
23. Avorn J, Soumerai SB, Everitt DE, et al. A randomized trial of a program to reduce the use of psychoactive drugs in nursing homes N Engl J Med 1992;327:168-173. [PubMed]
24. Ray WA, Taylor JA, Meador KG, et al. Reducing antipsychotic drug use in nursing homes. A controlled trial of provider education Arch Intern Med 1993;153:713-721. [PubMed]
25. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision supportmaking the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003;10:523-530. [PMC free article] [PubMed]
26. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomesa systematic review. JAMA 1998;280:1339-1346. [PubMed]
27. Glassman PA, Belperio P, Simon B, Lanto A, Lee M. Exposure to automated drug alerts over timeeffects on clinicians' knowledge and perceptions. Med Care 2006;44:250-256. [PubMed]
28. Kaushal R, Blumenthal D, Poon EG, et al. The costs of a national health information network Ann Intern Med 2005;143:165-173. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • MedGen
    MedGen
    Related information in MedGen
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...