|
|
AMIA Annu Symp Proc. 2008; 2008: 278–282. Published online 2008. | PMCID: PMC2656060 |
Assessing the Performance Characteristics of Signals Used by a Clinical Event Monitor to Detect Adverse Drug Reactions in the Nursing Home Steven M. Handler, MD, MS,1,2 Joseph T. Hanlon, PharmD, MS,1,3,5 Subashan Perera, PhD,1,6 Melissa I. Saul, MS,2 Douglas B. Fridsma, MD, PhD,7 Shyam Visweswaran, MD, PhD,2 Stephanie A. Studenski, MD, MPH,1,4 Yazan F. Roumani, MS, MBA,1 Nicholas G. Castle, PhD,8 David A. Nace, MD, MPH,1 and Michael J. Becich, MD, PhD2,9 1Division of Geriatric Medicine, Department of Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, PA; 2Department of Biomedical Informatics, School of Medicine, University of Pittsburgh; 3Department of Pharmacy and Therapeutics, School of Pharmacy, University of Pittsburgh; 4Geriatric Research Education and Clinical Center (GRECC), Veterans Affairs Pittsburgh Healthcare System (VAPHS), Pittsburgh, PA; 5Center for Health Equity Research and Promotion (CHERP), VAPHS, Pittsburgh, PA; 6Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh; 7School of Computing and Informatics, Department of Biomedical Informatics, Arizona State University, Phoenix, AZ; 8Department of Health Policy and Management, Graduate School of Public Health, University of Pittsburgh and; 9Department of Pathology, School of Medicine, University of Pittsburgh. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose Adverse drug reactions (ADRs) are a common cause of morbidity and mortality in the nursing home (NH) setting. Traditional non-automated mechanisms for ADR detection are time-consuming, costly, and fail to detect the majority of ADRs. We describe the implementation and pharmacist evaluation of a clinical event monitor using signals previously developed by our research team to detect potential ADRs in the NH. The overall positive predictive value (PPV) for all signals combined was 81% (54/67), with individual signal PPVs ranging from 0-100%. The PPVs were 53% (10/19) for the antidote signals category and 96% (44/46) for the laboratory/medication combination signals category. The majority 75% (12/16) of the preventable ADRs were laboratory/medication combination signals. The results suggest that ADRs can be detected in the NH setting with a high degree of accuracy using a clinical event monitor that employs a set of signals derived by expert consensus. The most frequent medication-related adverse events in nursing homes in the United States are adverse drug reactions (ADRs) [ 1]. ADRs are defined by the World Health Organization as unintended or noxious responses to a drug given in a dosage intended for prophylaxis, diagnosis, or therapy [ 2]. In the nursing home (NH) setting, the incidence of ADRs ranges from 1.19 to 7.26 per 100 patient-months. Although comprehensive chart review is the primary ADR case-finding technique for research, and is considered by some to be the “gold standard,” it is time-consuming, costly, and impractical for routine clinical use [ 3]. Therefore, a critical need exists for the development of alternative strategies to detect ADRs in NHs. ADRs can be detected by computerized clinical event monitors through the processing of pharmacy order signals and laboratory test result signals. Hospital studies indicate that these automated clinical decision support systems, which provide feedback to healthcare professionals based on information available in electronic format, are less expensive and much faster to use than manual chart reviews, and can identify events not often detected by clinicians during the course of routine care [ 3]. However, there is increasing concern about the false-positive alerts (i.e., alert fatigue) generated by hospital-based clinical event monitors which have been in use for over two decades, and have been described in more detail in a recent systematic review [ 4]. More recently, computerized ADR detection has been examined in the ambulatory and NH settings using largely the same pharmacy and laboratory signals used by hospital-based systems [ 5, 6]. The objective of this study was to determine the incidence and positive predictive values of signals specifically designed for use by a clinical event monitor to detect ADRs in the NH setting. Setting and Subjects The project was conducted in a single, independently owned, non-profit NH affiliated with the University of Pittsburgh Medical Center. The facility is located in a suburban setting, with 178 licensed beds. Prescribers were primarily community physicians and advanced practitioners (i.e., nurse practitioners and physician assistants). All patients, except those enrolled in hospice, were included in the 15-week study period (October 1, 2007 to January 13, 2008). The study protocol was reviewed and determined to be a quality improvement project by the University of Pittsburgh Institutional Review Board (IRB). The University of Pittsburgh Medical Center Total Quality Council approved this study as a quality improvement project. No additional IRB approval was required prior to publication. Source of Patient Data The Medical Archival System (MARS) data repository has been collecting and archiving clinical and financial records from all University of Pittsburgh Medical Center hospitals since 1986 [ 7]. In order to collect relevant nursing-home specific patient data in electronic format, we developed a new data repository, called MARS-LTC. MARS-LTC contains long-term care (e.g., NH) specific patient data that are stored as they are generated. Current real-time data feeds exist for laboratory data (Quest Diagnostics), pharmacy data (Rx Partners-LTC), and census and Minimum Data Set data (Achieve Healthcare Technologies). Clinical Event Monitor Description MARS-AiDE is a rule-based expert system that consists of a knowledgebase of signals that are applied by a rule engine to patient data contained within MARS-LTC. An overview of the MARS-AiDE clinical event monitor is shown in the below. We identified all active NH patients in the census database in MARS-LTC. Using the medical record number from the census, a list of active medications was generated. For certain rules, this information was then compared against current laboratory information. This process generated a list of alerts that fulfills the conditions of one or more of the signals, suggesting a potential ADR. Signals Used by the Clinical Event Monitor A detailed description of the development and selection of the knowledgebase of signals to detect potential ADRs in the NH is reported elsewhere [ 8]. Briefly, a multidisciplinary expert panel of NH physicians, pharmacists, and advanced practitioners reached consensus agreement on a list of 40 signals that a clinical event monitor can use to detect potential ADRs in the NH setting. In this manuscript, we present the findings associated with 37 of the 40 signals categorized into one of the following three groups: 1) 15 laboratory/medication combination signals (triggered by abnormal laboratory values when certain medications are present); 2) 12 medication concentration signals (triggered by elevated, or supratherapeutic medication concentrations); and, 3) 10 antidote signals (triggered by administration of medications given to counteract the effects of a medication with toxic effects). The 3 Resident Assessment Protocol signals (triggered by responses to certain Minimum Data Set items, and taking of certain medications) will be presented in a subsequent publication. For each of the 37 signals, we created additional rules to try and improve specificity. This resulted in a total of 24 laboratory/medication combination rules, 32 medication concentration rules, and 20 antidote rules. For example, the elevated creatinine or blood urea nitrogen concentration signal was operationalized into the following 3 rules: absolute increase > 0.25 mg/dL in baseline serum creatinine, relative increase ≥ 25% in baseline serum creatinine, and absolute increase ≥ 10 mg/dL in baseline blood urea nitrogen. Knowledge Engineering/Development For each of the 15 laboratory/medication combination signals, the first author (SMH) used standard pharmacy reference textbooks to create an initial list of medications that were reported to be associated with a particular laboratory abnormality. A drug information specialist (AHK) expanded the initial list of medications by using additional online references. The drug information specialist then conducted a comprehensive primary literature search using OVID, MEDLINE, and PUBMED for articles published in the English language between January 1, 1975, and July 1, 2007, using various MeSH terms and limiting the search to adults. Based on the results of the primary literature search, additional medications were added if the evidence supporting the association between the medication and laboratory test of interest was derived from case reports or study designs with stronger empirical evidence (e.g., case series, cohort studies, case-control, cohort studies with controls, etc.). Similarly, medications were removed from the initial list if evidence was not sufficient to support the association between the medication and laboratory test of interest. Any discrepancies in the data regarding the association between the medication and laboratory test of interest were resolved by discussion between two study investigators (SMH and JTH). Clinician Notification Similar to other clinical event monitors, the rules within MARS-AiDE are used to define computer-detectable events that potentially indicate an ADR. The MARS-AiDE system is designed as a screening tool. Its signals and associated rules have been chosen to be sensitive, but not specific. When the conditions specified by a rule are met, the MARS-AiDE system issues an alert to be acted upon by a consultant pharmacist (a pharmacist who is mandated by United States Federal law to review and manage the medication regimens of NH patients). On a weekly basis during the study period, a consultant pharmacist received a list of patients by email that contained alerts detailing the possible ADRs during the previous 7 days. Each alert showed the patient’s location, medical record number, attending physician, the rule that was triggered, and the date and time of the firing. Each alert also included extensive medication information, including the name, strength, frequency, route of administration, start and stop date, and if the medication was a standing or an as-needed (PRN) medication. If the alert was triggered by laboratory data, then each record also included the normal laboratory reference range, the most recent value, and the baseline value and corresponding dates (when available) (). Adverse Drug Reaction Assessments For each potential ADR alert, the consultant pharmacist used a structured implicit review process according to the following criteria: whether an ADR was present, if the ADR was preventable, and the seriousness of the ADR. We used the Naranjo causality algorithm [ 9] to determine the likelihood of whether an ADR was actually due to the drug identified by the clinical event monitor, rather than the result of other factors. The Naranjo algorithm is used to compute a weighted score based on answers to a short standardized questionnaire that correlates with causality probability. Similar to other clinical event monitor studies, computer alert signals with a score of ≥1 on the Naranjo scale, indicating a possible ADR, were classified as a true positives [ 10]. ADRs were considered preventable if they were associated with a medication error. We used the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) definition of medication errors [ 11], defined as any event that may cause or lead to inappropriate medication use or patient harm while the medication is in the control of the health care professional, patient, or consumer. Medication errors were further characterized by the step(s) in the medication use process where the error may have occurred, including: prescribing, order communication, dispensing, administering, and monitoring. ADRs were considered serious if they resulted in a transfer to a higher level of care (e.g., emergency department evaluation or hospitalization) or death. After an extensive training period, between-pharmacist-investigator reliability for identifying and classifying ADRs was assessed through independent review of the same 10 medical records by the consultant pharmacist (SBS) and two study investigators (SMH and JTH). All three identified and classified the incidents the same way. To assure that the ADR assessments were applied consistently, a senior investigator (JTH) verified the accuracy for every tenth patient evaluated. Calculation of Performance Characteristics For counting purposes, multiple firings on the same day, for the same patient, for the same rule were counted as a single alert. Specifically, multiple firings triggered by a single drug administration (e.g., multiple administrations of sodium polystyrene for hyperkalemia) were treated as a single alert. Similarly, if multiple drugs generated multiple alerts (e.g., furosemide and lisinopril are being administered in the presence of increasing creatinine), they were treated as a single alert. When more than one drug was associated with a single potential ADR at a given time, the drug-event pair with the highest Naranjo was retained and used for all calculations. To calculate a positive predictive value (PPV) for each signal, we divided the number of times that an alert was issued with respect to a particular rule and an ADR was confirmed (i.e., the number of true-positives), by the number of times the alert issued with or without an ADR being confirmed (i.e., the sum of true-positives and false-positives). Simple descriptive statistics were used to summarize ADR preventability and seriousness. During the 15-week study period, there were a total of 274 unique patients that met inclusion criteria. The clinical event monitor processed 5,729 medication orders, generating 67 alerts, an average of 4.8 per week. The overall PPV for all signals combined was 81% (54/67). Individual signal PPVs ranged from 0–100% (See Table below). The PPVs were 53% (10/19) for the antidote signals category and 96% (44/46) for the laboratory/medication combination signals category. | TablePositive Predictive Values (PPV) of Signals to Detect Potential Adverse Drug Reactions. |
Alerts were generated for 50% (5/10) of the antidote signals and 73% (11/15) of the laboratory/medication combination signals. There were no medication concentration signal firings. Of the true positive firings, 30% (16/54) were considered preventable ADRs. The majority (75% or 12/16) of the preventable ADRs were laboratory/medication combination signals. Of the preventable ADRs, 88% (14/16) occurred at the monitoring and 69% (11/16) at the prescribing stage of the medication use process. Overall, 6% (3/54) of the confirmed ADRs were considered serious, requiring emergency department evaluation, or hospitalization. All ADRs that were rated as serious were also considered preventable. Our results suggest that ADRs can be detected using a clinical event monitor in the NH setting with a high degree of accuracy. The overall PPV of 81% is substantially higher than PPVs previously reported in the literature, which range from 3–50% [ 4]. It is possible that our clinical event monitor performed better than previously described systems because: 1) we developed a list of consensus signals to detect potential ADRs by experts in geriatrics, rather than using existing hospital-based signals; 2) we simultaneously combined multiple data sources in order to enhance ADR detection; and, 3) we employed a standardized knowledge engineering process for the laboratory/medication combination signals category. Similar to previous studies, signals associated with laboratory test results performed better than antidote signals [ 4]. The antidote signals category may not have performed as well because these medications can be used to treat multiple medical conditions, only a fraction of which are related to the presence of an ADR. Having a better understanding of the context of the data as they relate to patients’ underlying medical conditions may help to improve this signal category’s performance. It is also important to note that during the course of our study, no signals from the medication concentration category fired. This is not entirely surprising since previous research suggests that a substantial proportion of older adults do not receive appropriate laboratory monitoring while being prescribed chronic medications [ 12]. Our results suggest that about one-third of the ADRs were preventable, and that of the preventable ADRs, 88% were associated with errors in the monitoring stage of the medication use process. This differs slightly from the only other study that used a clinical event monitor to detect ADRs in the NH setting, where 42% were judged preventable and 80% occurred at the monitoring stage [ 6]. Implications Developing a clinical decision support system that has a relatively low false-positive rate is particularly important in order to reduce alert fatigue. Furthermore, having a system that produces an average of less than 5 alerts per week could allow for the routine inclusion of ADR assessments as part of the monthly medication regimen review process conducted by consultant pharmacists on all NH patients in the United States. Future Direction Further research needs to be conducted to do the following: 1) determine the incidence and PPVs of the 3 Resident Assessment Protocol signals selected by expert consensus; 2) validate the findings of this study for a longer period of time in NHs with differing resident or facility characteristics; and, 3) describe the epidemiology and patient characteristics associated with ADRs detected by our clinical event monitor. The results of this study are being used by our health-system to select appropriate signals to develop a clinical event monitor system that can maximize the detection and possible prevention of potential ADRs, while minimizing the number of false-positive alerts. Conclusion The results suggest that ADRs can be detected in the NH setting with a high degree of accuracy using a clinical event monitor that employs a set of signals derived by expert consensus. This study was supported in part by NIH grants KL2RR024154-01, R01AG027017 and a Merck/AFAR Junior Investigator Award in Geriatric Clinical Pharmacology. The authors thank Andrea Ho-Kean (AHK), RPh, PharmD for her help in knowledge development, Scott B. Stephens (SBS), RPh from RxPartners LTC for his assistance with adverse drug reaction assessments, the staff at University of Pittsburgh Medical Center, Seneca Place nursing home, and University of Pittsburgh Medical Center MARS-LTC and DSI software developers. 1. Handler SM, Wright RM, Ruby CM, et al. Epidemiology of medication-related adverse events in nursing homes. Am J Geriatr Pharmacother. 2006;4(3):264–72. [PubMed] 2. Edwards IR, Aronson JK. Adverse drug reactions: Definitions, diagnosis, and management. Lancet. 2000;356(9237):1255–9. [PubMed] 3. Bates DW, Evans RS, Murff H, et al. Detecting adverse events using information technology. J Am Med Inform Assoc. 2003;10(2):115–28. [PubMed] 4. Handler SM, Altman RL, Perera S, et al. A systematic review of the performance characteristics of clinical event monitor signals used to detect adverse drug events in the hospital setting. J Am Med Inform Assoc. 2007;14(4):451–8. [PubMed] 5. Honigman B, Lee J, Rothschild J, et al. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc. 2001;8(3):254–66. [PubMed] 6. Gurwitz JH, Field TS, Judge J, et al. The incidence of adverse drug events in two large academic long-term care facilities. Am J Med. 2005;118(3):251–8. [PubMed] 7. Yount R, Vries JK, Council C. The medical archival system: Information retrieval system based on distributed parallel processing. Inf Proc Manag. 1991;27(4) 8. Handler SM, Hanlon JT, Perera S, et al. Consensus list of signals to detect potential adverse drug reactions in nursing homes. J Am Geriatr Soc. 2008;56(5):808–15. [PubMed] 9. Naranjo CA, Busto U, Sellers EM, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981;30(2):239–45. [PubMed] 10. Evans RS, Pestotnik SL, Classen DC, et al. Development of a computerized adverse drug event monitor. Proc Annu Symp Comput Appl Med Care. 1991:23–7. [PubMed] 11. Forrey RA, Pedersen CA, Schneider PJ. Interrater agreement with a standard scheme for classifying medication errors. Am J Health Syst Pharm. 2007;64(2):175–81. [PubMed] 12. Raebel MA, Carroll NM, Andrade SE, et al. Monitoring of drugs with a narrow therapeutic range in ambulatory care. American Journal of Managed Care. 2006;12(5):268–74. [PubMed]
|
