Summary of Key Findings

We have presented the results of a systematic review of the literature regarding the use of health IT to enable all phases of medication management as well as reconciliation and education. We have focused on MMIT systems that were integrated with other health IT systems. Our review identified a total of 789 studies dealing with health IT and medication management. Three hundred and sixty-one of these articles were only listed in the bibliography of this report and were not synthesized because they did not include comparative data, statistical methods, or qualitative methods. The remaining 428 articles were synthesized after being identified from an initial retrieval of 40,582 articles. We used these 428 articles to address the seven key questions (KQs). Overall, we found that the literature on MMIT applications was heterogeneous. The majority were based on observational methods, often with identifiable opportunity for bias (e.g., descriptive before and after studies without statistical adjustment for time trends or group differences). Research methods were not uniform across MMIT applications, with 77 of 88 RCTs studying CDSS.

KQ1. Effectiveness

Process and other outcomes related to use and satisfaction with MMIT were often improved, especially for prescribing and ordering and the monitoring phases. Improvements in the appropriateness of prescribing and decreased errors (e.g., correct doses and timing, better choices of antibiotics, fewer drug-drug interaction potentials, and corrected doses related to body weight or liver function) seem to be consistently shown. Changes in workflow, improvements in communication, and improved efficiencies such as time reductions are also positive, although fewer studies addressed these types of outcomes. Clinical endpoints were sometimes found to be improved with the use of MMIT, more often in the observational studies than in controlled clinical trials. CDSS applications and, to a lesser extent, CPOE systems have been shown to be useful, especially when studying prescribing and monitoring in hospitals and clinics. Notable was the identification of strong emotions expressed by users of CPOE (clinicians), both positive and negative, which were reported in the qualitative studies. A number of unintended consequences of the technologies were found, some of which were unfortunate and some of which were beneficial. Few cost studies and full economic evaluations were identified. Those articles that were included found that health IT interventions may offer cost advantages despite their increased acquisition costs. Proof of clinical improvements and economic effectiveness through the use of MMIT is lacking. However, given the uncertainty that surrounds the cost and outcomes data, and limited study designs available in the literature, it is difficult to reach any definitive conclusion as to whether the additional costs and benefits represent value for money.

KQ2. Gaps in Evidence and Knowledge

The major gap in the research is true full economic evaluations, weighing all costs and benefits of the various MMIT technologies across all settings and participants. For the effectiveness research, we found gaps were related to setting (few studies were carried out in pharmacies, long-term care facilities, homes, or communities), people (few studies assessing outcomes for pharmacists, nurses, nurse practitioners, physician assistants, other health professionals including dentists and psychologists, or patients and families), and MMIT technologies (rigorous studies of all but CDSS and CPOE, and especially those related to dispensing and administering, were sparse). Prescribing and monitoring were relatively well-studied while order communication, dispensing, administering, reconciliation, and education were understudied.

Gaps were also found in the sophistication and complexity of the quantitative research methods. Many of the studies initially identified were descriptive in nature. These are listed in the bibliography of this report. Qualitative studies and the quantitative studies that were hypothesis-based and comparative were analyzed. A good number of the studies, including those that were more strongly controlled (e.g., RCTs and cohort studies), often had methodology or reporting flaws or both including inconsistent use of standard methods for identifying and describing their methods, poorly justified or incorrect choices, or poor application of statistical tests and failure to adjust for group differences or cluster randomization. We also often found underpowered studies and situation-specific studies that were difficult to generalize or transfer to other settings or situations.

In addition, we found substantial deficiencies in reporting data important to the understanding of published studies. Although we identified data deficiencies in many aspects of studies, most serious were in descriptions of baseline data related to what was in place with respect to medication management in the health care setting before implementation of the MMIT system and descriptions of the MMIT implementation itself. Context is important for understanding studies and assessing their potential for application; detailed information on the setting and participants was also not often provided in studies.

KQ3. Value Proposition for Implementers and Users

Value propositions are determined by the balance of financial, clinical and organizational benefits. Limited data were available to address these issues comprehensively. Of note, we found that the various stakeholders had very different needs, perceptions, and access to MMIT systems and this must be addressed in valuing systems.

KQ4. System Characteristics

Different features of MMIT are important to various groups and settings. Very few studies (n = 21) reported on the specific feature sets of the systems being used and their links to purchase, implementation, and use. Few head-to-head comparisons using comparative effectiveness analysis methods, for example, were found. The evidence identified uses both qualitative and quantitative methods to gain an understanding of which features are important to users and stakeholders. Of note, we found that desired feature sets differed between the planning phase (perceived to be of value) and after implementation (based on actual use).

KQ5. Sustainability

Sustainability is vital to health IT. Before it can be fully understood and studied, it must be defined. For this document we chose to use a definition of sustainability that suggests sustainable systems are cost effective and clinically-effective. Because the evidence on economics data are lacking, we can add only a small amount of information on the sustainability of MMIT applications. Some data exist on effectiveness and use. We have included some data on patterns and characteristics that are important to use, including data on barriers and facilitators of successful implementations and ongoing system use. Use is higher in physicians, larger and better funded organizations, hospital settings, some larger primary care groups, and in academic medical centers.

KQ6. Two-Way EDI

Very little evidence exists on bidirectional communication between pharmacists and physicians to enhance the order communication process. Extrapolation of data on one-way communication, factors that work to increase electronic communication on medications between prescribers and pharmacists are external incentives, a supportive regulatory environment, and existence of standards for prescribing EDI. Three factors work against effective EDI: incomplete consideration of the workflow and financial effects of e-Prescribing on pharmacists and pharmacies, regulatory and legal uncertainties, and low adoption rates of EDI capable EMR systems. Further development and evaluation of two-way EDI technologies for outpatient order communication regarding drugs is required to facilitate adoption. Pharmacies and pharmacists should take a more active role in the EDI development and evaluation process.

KQ7. RCTs in CDSS

CDSS applications are well-studied although problems with methods and reporting exist. CDSS is probably the best studied type of MMIT in terms of studies with strong methods and a sufficient number of studies to provide reliable answers to research questions of any of the MMIT applications. The first RCT was published in 1976, over 35 years ago. Of the 88 RCTs in this document, 77 are on CDSSs. The quality and sophistication of study methods, analysis, and reporting of the RCTs has improved over time, and there tends to be more measurement of clinical outcomes. However, evaluations of health care delivery, such as comparisons of effectiveness of treatment or prevention methods (e.g., drugs, services, and medical devices) are held to a higher standard than the types of reported research projects included in this evidence report. The studies in this report, while they met their own research objectives, collectively do not contain the research designs and associated clarity of findings to be able to definitively inform patients, clinicians, and policymakers regarding the effectiveness and overall impact of CDSS applications. Furthermore, the more rigorous and transferable research conducted tends to show no or limited effect on patient-important clinical outcomes.

This report was not designed to evaluate specific MMIT applications. In addition, MMIT interventions were not catalogued and characterized in great detail. Therefore we found no obvious themes that would suggest that a certain type of MMIT intervention with a certain type of implementation for a certain type of user in a particular setting would be successful. However, the following areas of commonality emerged in our analysis.

General

  1. Research to date has concentrated on measurement of process changes and descriptive and pilot studies. In addition, some studies based on stronger methods have failed on issues such as adequate concealment of allocation and blinding, poor understanding of some methods, lack of adjustment of groups, and statistical challenges. Processes in health care are poor surrogates for clinical- or patient-important clinical outcomes, therefore it is important that new studies address clinical outcomes and use the most appropriate methods, and use them correctly, to adequately study MMIT applications. Researchers should also be encouraged to consider the generalizability or transferability of their results for all of their projects. Researchers in health IT could strengthen their studies by using interdisciplinary teams with representation of multiple stakeholders, learning from other domains such as health technology assessment and economics, and with better reporting of their studies and results.
  2. Standard and accepted definitions are lacking for MMIT applications, as well as standards for presenting the results of studies of health IT applications. Definitions are inconsistent for MMIT applications (e.g., e-Prescribing, CPOE, EMR, or EHR hospital information systems), study designs (e.g., observational or before-after), and outcomes (e.g., adverse drug events, adverse drug effects, prescribing errors, or errors per patient, 100 orders, day, hospital day, or physician). This has made identification of studies, data abstraction, synthesis of evidence, and presentation of findings challenging. Many study reports did not include important information that would have made this report stronger. Noticeable deficiencies centered on the MMIT application, its setting within the institution, training and implementation details, and maintenance and updating information. Professional associations interested in MMIT are pushing for standardization of definitions. AHRQ can join this movement for more standardization of terminology and definitions.

Effectiveness

  1. Interventions most frequently targeted prescribing and monitoring stages of the medication use process.
  2. Physicians who provided care in the hospital and ambulatory care settings were most likely to be the target of the intervention.
  3. CDSS and CPOE applications were the most frequently studied type of health IT application studied. Seventy-seven of the 88 RCTs in this report study CDSSs.
  4. Improvements in prescribing accuracy and decreased errors such as appropriate scheduling and choice of medications, prescribing taking into account weight-based dosing and dosing based on liver function, avoidance of drug-drug interactions and potential allergies and in being in accordance with guideline recommendations were consistently identified as improvements with the use of MMIT. Workflow, communication, interaction with peers and time considerations were found to be improved less often.
  5. Studies that used health IT to identify and intervene on patients with actual problems (e.g., elevated blood pressure) or needed care (e.g., hemoglobin A1c monitoring) appear to be more effective than CDSS approaches that identify potential problems (e.g., potential adverse drug events). This was particularly true when patient-centered principles were employed, such as providing patients with reminders and decision support recommendations about their current health status. However, this may be alternatively explained by the greater difficulty in measuring outcomes, such as potential for ADEs.
  6. Studies that have been successful in improving a patient’s clinical outcomes target high risk and vulnerable populations who have poor disease control,515,610,624,693 lack sufficient access to health care providers to manage their condition,408 or subpopulations with sufficient economic resources to respond to the CDSS intervention.694
  7. The effect of similar CPOE systems on mortality can vary substantially as a function of the extent to which implementation strategies disrupt or delay critical activities in the clinical setting, or demand additional time for order-entry from clinical staff.
  8. Highly targeted interventions, focused on specific medical problems appear to be demonstrated as more effective than more diffusely focused CDSS and CPOE. Again this may be due to the greater difficulty in measuring the outcomes of diffusely focused CDSS and CPOE in the generally smaller sample size (and inadequate power) studies that were identified.

Qualitative

  1. No qualitative studies were identified that directly addressed the effect of an MMIT system on intermediate health care outcomes for any phase of the medication management continuum (prescribing, order communication, dispensing, administering and monitoring). Patient safety was the main health outcome mentioned in qualitative studies. Before MMIT implementation, most studies found that clinicians expected that the MMIT system would improve patient safety. Once implemented, most clinicians felt that MMIT did improve patient safety.
  2. Differences in study outcomes for similar qualitative studies across settings were not apparent, suggesting that findings from qualitative studies could be transferrable across settings.
  3. Despite the willingness of many of the participants to use a new MMIT system designed to improve prescribing and ordering of medications including CPOE, reservations were expressed by some implementers that the MMIT system and the resulting change in workflow would impair existing interactions and relationships among health care providers and between physicians and patients.
  4. MMIT systems often substantially facilitated clinicians’ monitoring of patients’ adherence with their prescribed medication regimen.737 However, barriers were reported to using health IT systems for medication monitoring in some situations.747 For example, clinicians caring for patients with HIV/AIDS using a CPOE and CDSS system integrated with the hospital, pharmacy, and laboratory systems identified six barriers to using reminders, including workload, time to document, reminders that did not apply, inapplicability of reminders to the situation, lack of training to teach the users how best to use the new or modified system, quality of provider-patient interaction, and use of paper forms.747
  5. From qualitative studies, system design including workflow changes, challenges with the system interface and new communication processes demonstrated that without adequate attention to system changes, the new kinds of medical errors with potential detrimental impact to patient safety could occur. Unintended negative consequences including the need to develop workarounds (one-off or nonstandardized changes) to workflow and the frustration generated in some studies with MMIT implementation are important to recognize and deal with to improve the success of implementation.
  6. MMIT implementation did not just mean that a clinician needed to learn a new IT system but it also affected most of the other parts of the delivery of care processes, including how the interdisciplinary care team worked together.

Economics and Costs

  1. Cost analyses can provide useful information on ‘upfront’ costs compared with ‘downstream’ cost avoidance if they explicitly measure all direct health care costs (e.g., capital costs, health professionals’ time), direct nonhealth care costs (e.g., home care services, transportation) as well as indirect costs (e.g., productivity gains or losses) that could be affected by the intervention of interest.
  2. It is important to be aware that the greatest reported costs associated with these health IT are associated with the purchase of new software to add to preexisting EMR systems, as well as implementation costs (e.g., management, clinical team involvement, training costs) and maintenance costs. This assumes a large investment has already been made to purchase, implement, and maintain an MMIT system.
  3. The full enumeration of the total costs needs to be synthesized with the consequences or outcomes of the intervention (i.e., cost-effectiveness analysis, cost-utility analysis, and cost-benefit analysis). Full economic evaluations linked to clinical outcomes need to be done.
  4. Adoption of newer technologies needs to be based on formal evaluation of whether the additional health benefit (effectiveness) is worth the additional cost. Given the tension between the clinical benefits of CPOE and CDSS and the high up-front costs, decisionmakers deciding whether to implement CPOE and CDSS need to better understand how and when financial benefits of such systems accrue (e.g., short-term compared with long-term benefits). These types of analyses are important for well-informed decisionmaking.

Unintended Consequences

  1. Unintended consequences, both positive and negative, were found across many of the studies as main endpoints, or were alluded to in others. Some were minor and some much more serious. A tracking system of major and clinically important unintended consequences would be useful for many audiences and should be considered by system developers and funding agencies.

CDSS

  1. Many studies have evaluated CDSS tools for improving the effectiveness of anticoagulants (proportion of days in therapeutic range for anticoagulants) and improving the choice, route, duration of antibiotics, and reducing ADEs related to antibiotic use, and most are successful.
  2. Sophisticated CDSS and other advanced clinical support should be built to insure added clinical value without being burdensome to those who use them. These sophisticated systems are difficult to develop, implement, and maintain.

Values

  1. Values related to MMIT systems and implementations need to be determined from all stakeholders. Clinicians, administrators, and likely patients and their families have different values and place varying importance on each.

System Characteristics

  1. Most often authors spoke about barriers and concerns towards implementation and acceptance, rather than characteristics of MMIT that could facilitate implementation, purchase, and use of such systems.

Two-Way EDI

  1. Two-way EDI between prescribers and pharmacists is not common. Both facilitators and barriers exist that impact movement to implementation of e-Prescribing and two-way communication designed to enhance and streamline prescription optimization.

Limitations

Our review has a number of limitations. With the exception of PDA applications using patient-specific input, we focused on applications that enable medication management and that are integrated with other health IT systems. A number of technologies, such as smart intravenous pumps, bar-code scanners, and reporting systems for diabetes or asthma monitoring were not integrated with other health IT systems and were thus excluded. Indexing of individual articles in electronic databases is poor. Although we tried to be thorough in our search methods, we feel that we did not capture all potential articles—a very difficult task in new and multidisciplinary areas of study.

Further, we concentrated only on the main or major endpoints reported in studies with comparison groups and hypothesis testing. Given the heterogeneity in the literature, it was often difficult to discern main endpoints; where possible we determined main endpoints as those declared as such, or those that were the basis of power calculations (infrequently), or were stated to be main outcome measures in the abstract or objectives. We identified instances where the main endpoint was not clear. In these cases we gave priority to outcomes related to medication management and clinically important patient outcomes. We did not test the replicability of our abstraction of these outcomes.

Because of the lack of clear definitions on some of the technologies and issues associated with health IT, we were unable to address some key questions as thoroughly as we would have liked. This is especially noted in KQ5 relating to sustainability and KQ3 on value propositions. We feel that these are important issues for all health IT, that need to be addressed to effectively answer questions about ongoing use and effectiveness of these technologies.

It has proven difficult to synthesize the evidence on such a range of technologies, implemented in a number of settings and used by various stakeholders. Each intervention is so complex that it is often difficult to tell which studies are assessing the same processes. Also, outcome measures used by authors were variable. For example, similar outcomes such as prescribing changes were measured as changes in daily doses; prescribing rates per hospital, per physician, per 1,000 patient days, etc. The number of orders and compliance rates were difficult to extract and synthesize.

Our ability to draw conclusions is also reliant on the quality of the evidence we have found. In most cases, the research relies on observational studies, with RCTs and other methodologies with stronger controls only available on a select group of health ITs and phases of medication management. Even in the case of CDSSs, a lack of RCTs addressing electronic decision support integrated with other types of health IT still exists. Only a small minority of these studies focus on clinical outcomes—the endpoints that are most important to guide decisions by patients, providers and policymakers, about adopting these interventions. Furthermore, a very small number report improvement in these clinical outcomes.

We found great variation in the level of description of the health IT employed, with studies frequently lacking details on standards, hardware, integration, implementation dates and processes, and other similar factors. A large number of studies neglected to report the study dates (see Evidence Tables in Appendix C). We repeat Chaudhry’s call for a set of standards for reporting on health IT research.607

Although the absence of a contemporaneous comparable control group is a problem with all observational studies, the creation of control groups by comparing intervention patients to those that do not participate, or do not have a problem to those that do is fundamentally far more likely to introduce major bias in the comparison (e.g., comparing patients with alerts to no alerts,18 pharmacists volunteering to provide the intervention compared with those that do not volunteer,694 and other similar problems701). The direction of the bias will depend on the study.

Many observational studies suffered from selecting an outcome that was distantly or only marginally related to the intervention. Length of stay, all-cause ADEs were examples of this problem. Gurwitz and colleagues697 were able to show that only one-third of ADEs could have been prevented by the CDSS alerts provided. Moreover, in a substantial proportion of negative studies, minimal adoption was seen. The clinicians failed to adjust therapy or treatment based on recommendations, and thus it is not very surprising to find that the interventions had no effect on outcomes. Finally, the rate of some outcomes such as readmission, mortality, and nosocomial infections was too low to detect clinically meaningful differences if they had existed with the numbers involved in the study.

Implications

The strength of this document lies in the breadth of health IT applications used across the phases of medication management, and in the organization of those findings, both through synthesizing the body of evidence by key questions and a tabular presentation of those findings. A review of this scope for MMIT has not been completed previously. We searched for literature across many domains and reviewed a substantial number of studies. The implications of the report fall within the purview of future research, policy, and evaluation. We have detailed gaps in evidence in KQ2 and future research needs in Chapter 5.

Important implications of this evidence report exist for health care decisionmakers, especially AHRQ and the U.S. National Coordinator of Information Technology. A large amount of health care spending in the United States is currently being funneled into development and implementation of various health ITs. Certainly the burden of evidence is towards positive effects on process changes and measures of satisfaction and perceived benefits among users. These early indications are logical precursors to changes in demonstrated effects in benefits such as quality of care and clinical outcomes, economic benefits, or both as the technologies advance and mature. A lack of proven effectiveness in improving patient outcomes and a lack of studies on value and cost-effectiveness still exist. Currently, most systems are in their infancy and need to be continued to be scrutinized for effectiveness and safety.

Because MMIT systems specifically, and health IT in general, are expensive to develop, support, and update, it is essential that these burgeoning health ITs be rigorously assessed for cost-effectiveness and clinical-effectiveness. This effectiveness information is essential for policymakers who are allocating scarce health care resources which have multiple competing priorities. Computerization of health care will continue with the adoption of more and newly developed MMIT and other health IT applications. Clinicians, researchers, policy advisors, and health administrators should be prepared for a major investment of time and resources for implementation and use. They need to consider direct and indirect effects on health care processes such as altered work flows, adverse patient outcomes, and indirect costs. Because of the paucity of successful clinical outcome studies, and the heterogeneity of the systems, the specific interventions, and measures of effectiveness, this systematic review has been unable to clarify which factors of topic, design, or implementation may assist in the success of the MMIT.

Administrators will be able to plan for implementation better using the quantitative and qualitative findings and results. They will also be able to use this report to balance their expectations of MMIT installations and interact better with vendors and consultants.

Researchers and research funders will have a roadmap of the evidence that supports the effectiveness of MMIT applications, an outline of gaps, and lists of remaining challenges. Researchers should be aware of quality and reporting issues related to research methods as described in this review, as well as the need for research teams to include expertise or consultation from all clinician groups affected by the technology, informaticians, and those with research skills in a wide range of methodologies (research synthesis, complex interventions, pragmatic trials, usability studies, statistical planning and analysis, health technology assessment methods, and knowledge translation skills). Researchers and evaluators also need to adhere to established publication guidelines such as the STARE-HI guidelines783 for presenting results of their studies, to ensure that readers will have the information they need to plan for implementation of MMIT systems.

The meaningful use objectives should also be deployed in all projects and implementations. Research funders can direct their programs and reinforce use of standard definitions, reporting standards, and meaningful use objectives. They can also encourage multicenter trials and those that have potential for broad applicability. Adherence of the MMIT systems to local, regional, and national standards is also important to encourage and foster. At the same time, incremental studies which show the transferability and reproducibility of findings from one study to other health care settings, systems (vendors), and health care issues (type of disease or patient and setting) should also be encouraged.

Although the strength and breadth of the body of evidence supporting the usefulness of MMIT for improving health care is not uniform across people, places, and technology, it still is substantial. We can learn much from reviewing the original studies and systematic reviews on MMIT. We also feel that the content of this report can help us leverage our existing knowledge of MMIT to a broader audience and that this can improve the health and health IT effectiveness for many people in various health care settings.