Results of the Literature Search

The literature search retrieved 40,582 articles for screening for inclusion; this includes 93 from hand searches and 408 from the grey literature (Figure 3). We excluded 7,797 duplicates and screened 32,785 titles and abstracts. A total of 4,578 articles were screened at full-text. Reasons for exclusion at this stage included inaccessible copies of full-text, articles in foreign languages that were not translated, unretrievable theses, studies not using integrated technologies or technologies that did not impact on medication management (“not MMIT”), studies that were not a primary study with first hand observations (often review articles), or studies not measuring an outcome of interest to our key questions.

Figure 3 is a PRISMA flow diagram, summarizing the search and selection of articles at the various stages in the review. A total of 40582 studies were retrieved through searching the databases. 7796 were duplicate records. We screened 32786 titles and abstracts, excluding 28429 records as not relevant to the topic. 4357 articles were screened at full text; 389 were excluded as per the following categories: 150 unable to retrieve or foreign language; 32 thesis; 18 not MMIT (not dealing with medication management or not integrated technologies); 2181 not a primary study and 260 not measuring an outcome of interest to our key questions. The final review includes 789 articles; 428 in the results synthesis for key questions 1–7; a further 361 studies are included in the bibliography as meeting our content criteria but not meeting methods criteria of comparison groups and hypothesis testing.

Figure 3

Literature flow of medication management studies. To address the various outcomes measures or interest in the seven key questions, 428 articles were synthesized. An additional 361 articles met content criteria for integrated technology enabling medication (more...)

We found 361 articles which met our content criteria for Key Question (KQ) 1: Effectiveness but did not use formal qualitative methods or have comparison groups with hypothesis testing or appropriate statistical analyses (quantitative studies). These articles are not included in the synthesis but they are integrated into the report bibliography with the other articles that were synthesized. This left 428 articles that are synthesized in the evidence report.

A total of 377 articles are quantitatively synthesized in KQ1: Effectiveness. One article was included as evidence for KQ3: Value Proposition although 30 articles were cited directly and many more were used in some of the summaries described in this section; 21 articles were synthesized for KQ4: System Characteristics, 24 for KQ5: Sustainability, 33 for KQ6: Two-way Prescription EDI and 77 for KQ7: RCTs of CDSS. A number of articles were included in more than one key question response as they addressed more than one aspect of medication management.

KQ1. Within and across the phases of medication management continuum, what evidence exists that health IT applications are effective?

Effectiveness Studies Overall

KQ1: Effectiveness includes the largest number of articles for any of the seven key questions. Our searching for KQ1 concentrated on content with no limits on methods. The articles were divided into two groups: (1) all qualitative studies were included in the analysis that forms the basis of this report; and (2) all quantitative studies were included in the analysis section if they included a comparison group and data on each group and if they contained statistical methods defined by statistical testing, a statement of hypotheses-based research defined a priori, or both.

Articles that met these criteria for qualitative or quantitative studies were analyzed in this report (n = 377). An additional 51 studies that did not meet the above methods criteria were included in KQs 3 to 6; the literature on these topics was sparse. Articles that met content criteria but that did not meet these methodological criteria are included in our bibliography (n = 361) but not in any of the tables nor are they analyzed in the report.36–396

The final analysis for addressing KQ1 included 379 articles. Substantial variation exists in the concentration of evidence and content across issues related to MMIT. Table 1 shows the numbers of studies within each of the five phases plus reconciliation and education by study design. By far, more studies are done in the prescribing phase (n = 263) with a substantial number done in monitoring (n = 77). Dispensing is the phase that is least studied and little evidence exists on education and reconciliation. Figure 4 depicts the trends in publication frequency of articles included for analysis in the report. We saw a dramatic increase in publication of MMIT studies after 2000, most notably in studies dealing with prescribing.

Table 1. Research design for studies across the phases of medication management and education and reconciliation.

Table 1

Research design for studies across the phases of medication management and education and reconciliation.

Figure 4 is a line graph depicting the number of included articles published (on the y-axis) ranging from 0–50, by the year of publication (along the x-axis) from the mid-1970s to June 2010 for each of the five MM phases. Each phase is represented by a different line. Prior to 1990 each of the 5 lines are at about 1–2 articles; from 1990–2000 they tend to be below 5 articles/year. From 2000–2005, dispensing, order communication and administering are still maintained at fewer than 5 articles/year. In 2006 these three spike to 8–10 articles. The number of prescribing articles increased from under 10 in the early 2000’s to 50 in 2009. The number of monitoring articles also saw an increase to closer to 10 articles per year in the 2000’s. Overall, most publications across the medication management phases are published increasingly after 2000.

Figure 4

Trends in publication of articles relating to the phases of medication management across years until searching was completed in June 2010.

Strengths and Limitations of the Evidence

Table 1 illustrates that a variety of research methods were used in the studies, with the majority using observational methods. A substantial numbers of RCTs and qualitative studies were included. The large number of observational studies is reflective of the nature of the domain in that many of the articles retrieved were more often directed at the observational description or evaluation of existing systems rather than based on classical research methods of hypothesis-driven projects.

Settings. Settings where studies were performed also show variation. Table 2 includes the settings for studies across the medication management phases plus education and reconciliation. Studies often reported multiple settings. Most studies were set in hospitals and ambulatory care. Few studies were done in the community (n = 1), home (n = 5), or long-term care (n = 8). The lack of comparative, hypothesis-driven studies set in pharmacies is offset by a larger group of pharmacy studies that were descriptive in nature. Despite the lack of studies set in pharmacies, many studies relied on pharmacies and pharmacists.

Table 2. Settings for the phases of medication management and reconciliation and education.

Table 2

Settings for the phases of medication management and reconciliation and education.

Clinicians. Physicians were the most represented clinicians studied (Table 3). Many of the health professionals functioned in primary care and other ambulatory settings. Often studies did not differentiate among specialties or included many specialties in a single study. Nurses were most often studied in the administering phase and pharmacists were involved in order communication. We did not identify any studies that evaluated dentists and found few studies of mental health professionals or midlevel practitioners (e.g., nurse practitioners, midwives, and physician assistants).

Table 3. Clinicians evaluated in outcomes studies of medication management phases, education, and reconciliation.

Table 3

Clinicians evaluated in outcomes studies of medication management phases, education, and reconciliation.

Patient population studied. Patients studied represented those who were most likely to need medication: adults, middle aged people, and those over the age of 65 years. Infants, children, and adolescents were also studied but to a lesser extent (Table 4). Monitoring and reconciliation concentrated on older persons.

Table 4. Patients and caregivers studied by phase of medication management and education and reconciliation.

Table 4

Patients and caregivers studied by phase of medication management and education and reconciliation.

Technology. Tables 5 and 6 list the MMIT applications studied. Table 5 includes those MMIT applications that were the main focus of the study while Table 6 includes the MMIT that were integrated with the MMIT studied. The CDSS and reminder systems were most common in prescribing and monitoring. CPOE and e-Prescribing were also commonly used. Systems associated with pharmacy use were less commonly studied. Considering integration, the health IT in medication management is well-integrated with comprehensive systems such as EMRs and hospital information systems as well as other components of the broader health IT domain, remembering that integration with a health IT system was a requirement for inclusion in our review. Although prescribing again is the major phase studied, the other phases are represented.

Table 5. Main health IT studied by medication management phase and education and reconciliation.

Table 5

Main health IT studied by medication management phase and education and reconciliation.

Table 6. Health IT integrated with the health IT being studied.

Table 6

Health IT integrated with the health IT being studied.

Summary. In summary, prescribing is the major medication management phase studied. Studies were often evaluative rather than research centered in nature, as reflected in the number of observational studies. Substantial numbers of RCTs and qualitative studies exist. Most often studies were set in hospitals and in ambulatory care facilities. Few studies were set in pharmacies, although most of the articles showed interactions with pharmacists and pharmacies. Long-term care, community settings, and homes were not often studied. CDSS systems were the most common MMIT application studied. CPOEs were also studied substantially. The MMIT applications studied were very often embedded within a larger EMR, hospital, or pharmacy information system and integrated with other health IT applications. Many different MMIT systems were studied although again few were done outside the prescribing and monitoring phases and variation existed in the number of studies of each kind of health IT. Physicians were the health professionals most often studied. Few patients were evaluated.

Process Changes—Prescribing

Summary of the Findings for Process Changes

Of the 378 articles that have outcomes associated with MMIT, 174 (46 percent) are reports of the evaluation of processes in the prescribing phase of medication management (Appendix C, Evidence Table 1). Because prescribing and ordering are substantially different in the hospitals and ambulatory settings, the remainder of this section will provide analyses with the articles divided into hospital-based studies (n = 107) and ambulatory care-based studies (n = 67). The community- and home-based studies are included with ambulatory care. Only one study in this section was done in a long-term care facility.397

General Study Characteristics

Strengths and Limitation of the Evidence. The studies of process changes in MMIT based in hospital settings have a lower proportion of RCTs than ambulatory care studies. The 107 hospital-based studies are comprised of 19 RCTs (18 percent of hospital studies),398–416 84 observational studies,18,417–499 3 cohort studies,500–502 and 1 mixed methods study.503 (Table 7)

Table 7. Research methods of studies that evaluated process changes associated with the prescribing phase of MMIT.

Table 7

Research methods of studies that evaluated process changes associated with the prescribing phase of MMIT.

The 67 articles set in primary care, communities, and home (ambulatory care) were studied using 40 RCTs (61 percent of nonhospital studies),504–543 2 cohort studies,544,545 1 case control study,546 1 mixed methods study,547 and 22 observational studies.431,548–568 The long-term care study was an RCT.397

Patient population. Not all studies included descriptive data on patients. Of those that did, the patient populations reflected the pattern of medication use with more studies including participants who were, on average, over the age of 44 years. All age groups were studied in both hospital and ambulatory settings.

Thirty-seven studies set in hospitals provided descriptive data on participants. Four studies included infants (birth to 2 years),418,463,467,491 five studies included children (2 to 11 years),411,418,437,467,569 six studies included adolescents (12 to 18 years),411,418,437,446,467,490 eight studies evaluated adults (19 to 44 years),408,411,418,437,446,467,490,492 22 studies included middle age participants (45 to 64 years),399,401–403,406,414,425,428,430,431,433,445,448,449,454,467,475,481,485,489,490,493 and 27 studies included geriatric participants (65 years and up).399,401–403,406,407,413,414,416,425,428,430,431,433,440,445,448,449,453,467,481,483,485,486,489,490,493

Thirty-six of the 67 studies done in ambulatory settings included descriptive data on the patient population studied. Many studies included participants in a wide range of ages. Of the 36 studies that included patient information, one studied infants (up to 2 years of age),559 two evaluated medication management in children (2 to 11 years),539,559 three evaluated adolescents (12 to 18 years),504,537,539 15 were of adult population (19 to 44 years),504,505,508,512,515,522,527,528,530,531,537,541,543,545,565 24 studied patients in the middle age range (45–64 years),504,505,508,510,513,515,518–520,522,526–528,530,531,537,538,541–543,545,557,565,568 and 22 studies include geriatric patients (65 years and up).505,507,510,513,515,519,522,524,528,530,531,533,541–543,545,552,565,566,568 The single long-term care study did not describe its patients.397

Clinicians studied. Only 11 of the hospital studies included descriptive information on clinicians. Several of these included multiple groups of health professionals: hospitalists398,405,407,452,454,486 other physicians,407,415,487,488,502 other health professionals,452 and nurses.439 Many of the other studies evaluated clinicians but did not provide sufficient demographic information for analysis or discussion.

Few of the studies set in ambulatory care provided substantial information on clinicians. Those clinicians who were specifically described were primary care clinicians,506,510,514,529,532,534,535,552,563,566 other physicians,514,525,535,544,547,553,554 nurses and midlevel practitioners (physicians assistants, nurse practitioners, advanced practice nurses, and midwives),535,561 and pharmacists.518

Technology. Technology associated with studies set in hospital settings often evaluated several integrated MMIT systems, although some studies included only one MMIT. Individual MMIT applications included CDSS systems,18,397–399,401–405,407–418,420,422–428,430,431,433,435,448,449,451–454,458,460–462,464,466,468–478,480–483,485,486,489–502 CPOE systems,400,406,417,419,421,427,429,432,437,438,441–444,446–450,453,455,457,459,463,466–469,479–484,487,488,491,493,503 pharmacy information system,438 computerized unit dose drug dispensing system,465 e-prescribing,434,436,439 medication safety reporting system,456 an internet electronic diary for patients,408 and eMAR systems.439,456

Studies set in ambulatory care also studied a range of MMIT applications with the majority of the MMIT applications based on CDSS systems. The MMIT applications were CDSS,504–531,533–543,545,546,548,550–560,563–566,568,570 CPOE,507,527,532,540,551,558,567 e-Prescribing,540,544,547,549,561,562 and a pharmacy information system.507

These MMIT systems that were the focus of study were also integrated with a range of other MMIT applications, most of which were some form of EHR or EMR systems. The hospital-based studies included integration with bar-coding systems,493 billing or administration,400,406,446,474,485 CDSS,417,429,432,442,448,461,462,479 CPOE,398,400,402,405,407,410,411,413,415,416,423,426,427,430,433,435,440,445,448,454,458,460–462,464,471,472,480,486,490,492,494–497,500,501 EHR and EMR systems,18,398,405,407,409,411,415,416,418,422,423,430,431,434,436–438,442,444,445,447,448,453,455,460,462,466,473,475,479,480,483,489,492,498,501 hospital information systems,400,403,406,412,414,415,424,425,428,438,442,443,450,451,453,456,461,465,467–470,474,478,481,482,487,488,491,499 imaging systems,405,437,442,443,456,463,466–468,470,475,483 laboratory systems,18,401,405,407,409,415,418,420,437,442,444,446,452,453,456,461,466,470,473,475,477,483,485 formulary systems,406,442,443 and an integrated pharmacy system.401,404,415,417,420,434,436,439,441–443,463,473,477,479,485,493,502

Systems integrated within ambulatory studies were also mostly EMR and EHR systems. The ambulatory studies included billing and administrative systems,504,511,512,532,564,567 CPOE,511,512,518,522,529,532,548,550,564,570 EHR and EMR systems,505–507,510–513,515,517–524,526–528,531,533,535,536,538–540,542,543,545,546,548,550,551,554,556–561,564,566,568 formulary systems,544,549 hospital information systems,508,509,541,565 imaging systems,513,532,538 insurance systems,542,549 laboratory systems,504,509,513,516,538,541,548,550,555 and pharmacy systems.504,508,511,516,518,541,548,552,562


Prescribing changes. Sixty-one studies evaluated changes in prescribing in hospital settings. Fifty-three (87 percent) showed statistically significant improvements in at least half of its main endpoints. Categorizing these 53 articles into groups based on study methods, 11 were RCTs,398,399,402–405,408–410,413,416 two were cohort studies,501,502 and 40 were observational.417,423,424,426,428,430,431,440,443,446,447,451,452,456,458–460,462,464,466–473,475,478,482,486,489,490,492,496–499,547,571 Eight (13 percent) of the 61 did not show statistically significant changes. Two were RCTs411,414 and six were observational studies.418,420,445,449,477,485

Thirty-two studies set in ambulatory care found improvements in prescribing with the MMIT as defined by at least half of the main endpoints being positive. Twenty-three were RCTs.504,506–508,511–515,517,522,525,526,528–531,533,535–537,539,541 One was a case-control study.546 Seven were observational.549,553,554,562,565,566,568 Eleven studies set in ambulatory care settings sought to determine if prescribing was improved with the introduction of MMIT and they did not demonstrate differences. Of these, five were RCTs,510,520,523,524,538 two were cohort studies,544,545 and four were observational.557,560,564,572

Errors. Twenty-two articles studied prescribing errors in hospital settings. Fifteen showed statistically significant improvement in at least half of the main endpoints. Two were RCTs400,405 and 13 were observational.421,434,438,439,450,453,454,457,465,474,479,491,493 Seven did not show statistically significant improvements: a mixed methods study503 and six observational reports.419,432,436,444,455,495 Two studies reported errors related to ambulatory care studies and both were positive, one using CPOE,558 and one using CDSS,559 both in observational studies.

Time considerations. Seven studies evaluated time considerations in hospital settings. Two observational studies showed statistically significant improvements in considerations of time.439,488 One study found a statistically significant increase in time to prescribe.487 One study evaluated mean time on antimicrobial management but did not do statistical testing.401 Three observational studies did not show any differences in time.419,441,481 Five studies assessed time savings in the ambulatory care settings. Four were positive: an RCT of CPOE and CDSS that reduced time to respond to alert situations,527 a cohort study on time spent on asthma management,545 and two observational studies, one on e-Prescribing561 on time spent on computer and paper tasks and one on time spent on ordering laboratory testing for monitoring.550 One RCT showed that time spent on patient care did not decrease with the introduction of a CPOE system.532

Adherence to guidelines, reminders, and recommended practice. Twenty-four studies measured improvements in compliance with guidelines, reminders, and recommended practices in hospital based studies. Nineteen identified statistically significant improvements in compliance: four RCTs,407,412,521,573 one cohort study,500 and 14 observational studies.424,425,427,429,435,437,438,447,461,470,476,483,484,494 Four did not find any differences in compliance: one RCT415 and three observational studies.442,448,480 One observational study422 showed a decrease in adherence after the introduction of a CDSS system into a hospital EHR.

Thirteen studies that took place in nonhospital settings (primary care, community, and homes) considered compliance with guidelines, reminders, or recommended practice. Seven were RCTS of which five showed positive results for at least half of the main endpoints.505,509,534,542,543 Two RCTs518,519 did not identify a difference for measured compliance. One cohort study544 did not show a difference in compliance with the formulary using e-Prescribing and one mixed methods study547 reported no change in physician compliance with drug alert overrides. Four of four observational studies reported improvements in compliance with guidelines, reminders, or recommended practices.548,556,563,566

Workflow. No studies set in hospitals studied workflow as one of their main endpoints that were changes in process. Two studies set in ambulatory care studied workflow. One study using CDSS reminders showed a significant reduction in missed followup appointments that had been scheduled by nurses.555 An RCT of CDSS and e-Prescribing did not affect the rate of callbacks generated between physicians and pharmacists.540


Much research has been done to evaluate changes in process related to prescribing in hospital settings and ambulatory care situations. The research is varied in methods although many RCTs exist. A higher proportion of RCTs are done in the ambulatory care studies than in hospitals. Clinicians are the most studied. Pharmacists are often included in studies but are less frequently the major thrust of analyses. Many MMIT applications are studied in the prescribing phase. These prescribing MMIT applications are also frequently supported or integrated with EMR, EHR and hospital information systems. CDSS systems are often studied and frequently integrated with CPOE or e-Prescribing systems. Pharmacy-based MMIT applications generally lack evidence.

With respect to the process changes measured in the prescribing studies, changes in prescribing and compliance with reminders, guidelines, and standard practice are the most common outcomes for hospital- and ambulatory-based studies (Table 8). The RCTs were concentrated on evaluating changes in prescribing with some of them assessing compliance. The RCTs were positive 80 percent of the time while the observational studies were positive 77 percent of the time. Studies done in ambulatory care settings have not evaluated errors as outcome measures. Quantitative workflow studies are generally absent across all settings. MMIT in prescribing is associated strongly with improvements in prescribing and also associated, but to a lesser extent, with reducing errors and improving compliance with guidelines, reminders, and recommended practice. Time reductions or changes are not as often improved and workflow improvement assessments are lacking evidence.

Table 8. Summary of the number of studies reporting statistically significant process changes in studies of prescribing by process for hospital and ambulatory based studies.

Table 8

Summary of the number of studies reporting statistically significant process changes in studies of prescribing by process for hospital and ambulatory based studies.

Systems that provide information support, such as CDSS and CPOE systems, are combinations of technical capabilities and a knowledge base. The content of this knowledge base is probably more important than the technical aspects. Research findings and scientific evidence (i.e., evidence-based content) are difficult to compile and even more difficult to keep current. We did not find evaluations of the knowledge base of the systems or comments on updating, although some of the systems depended on clinical practice guidelines for their evidence base. Similarly, outcomes that were associated with correct knowledge such as adherence to best practice guidelines were also not often evaluated to show that they were accurate and current. Future research must address how this need for strong evidence to support the knowledge base of CDSSs, provide evidence backing for order sets for CPOE systems, and clinical practice guidelines on which to base best practice can be best met.

Order Communication

Summary of the Findings for Process Changes

Order communication is less well-studied than prescribing with only 16 studies: two RCTs,540,574 and 14 observational studies.438,552,567,575–585 (Appendix C, Evidence Table 2) Order communication involves clinicians and pharmacists. The oldest study in this group was published in 1999,574 reflecting the recent advances in communication applications related to MMIT.

Strengths and Limitations of the Evidence

The evidence in this section is predominantly observational with two RCTs. The studies were mainly based on large sample sizes, from 39 clinicians578 to almost one million prescriptions.577 The outcomes, most often measures of efficiency and changing work patterns, were usually reported as being positive.

General Study Characteristics

Participants. All studies included physicians, other prescribers, and pharmacists. The main unit of analysis in 12 of the 16 studies was prescriptions, orders, and medications. The main unit of analysis for the other four studies were patients,552,578 pharmacists,580 and clinicians.574 The patients were of geriatric age (65 years or greater) or adults (45 to 64 years),578 or geriatric alone.552 Except for these two articles, all others included undifferentiated patients.

Location. All studies included a pharmacy. Most studies were hospital-based but one study was of three mail order pharmacies,575 two were HMO pharmacies only,552,574 and three were in community pharmacies.579,585,586 Nine studies were set in hospitals,438,574,576,578,580–584 and four in primary care.540,577,585,587

Drugs and diseases. Thirteen studies did not concentrate on one disease or disorder. One study each evaluated venous thromboembolism,578 cancer,584 and HIV/AIDS.585

Technology. The MMIT in the order communication phase is varied: six CDSS,438,540,552,577,580,582 eight CPOE,438,567,576,578,580–582,584 two eMAR systems,581,583 four e-Prescribing,540,575,579,585 one e-transmission,575 and two pharmacy information systems.438,574 Several studies included more than one MMIT as the major focus of the study. The ordering systems (CPOE and e-Prescribing), however, predominate.

The following MMIT applications were integrated in 15 of the 16 studies: one CDSS,567 nine EMRs,438,540,574,576–578,580,581,585 three hospital information systems,438,576,582 two imaging systems,574,581 three laboratory systems,574,578,581 six pharmacy systems,552,577,579,583–585 and one eMAR system.580 The nine EMRs and three hospital information systems reflect maturing of the MMIT systems with respect to order communication.


No process changes were presented for adherence with guidelines, monitoring, or preventive care. One article described decreases in prescribing of contraindicated drug-drug combinations in ambulatory settings.577 Another looked at the agreement between pharmacists and family physicians (need for clarification of prescriptions) with and without e-Transmission of prescriptions, again in the ambulatory setting.575 All other process changes that were the main focus of the order communication articles dealt with errors and efficiencies.

Errors. Two hospital studies addressed errors (Table 9). Mahoney et al.438 showed a decrease in drug-allergy violations, excessive dose, incomplete or unclear orders, and therapeutic duplication with the introduction of a CPOE and CDSS system into a pediatric standalone and another general hospital. Varkey et al.567 found an increase in the frequency of intercepted prescription errors after the introduction of another CPOE and CDSS system into the Mayo Clinic ambulatory practices.

Table 9. Summary of the number of statistically significant process changes in studies of order communication by process for hospital and ambulatory based studies.

Table 9

Summary of the number of statistically significant process changes in studies of order communication by process for hospital and ambulatory based studies.

Prescribing. Two studies showed improvements in prescribing with increased interaction between pharmacists and physicians (Table 9).552,577

Efficiency and workflow. Five hospital-based studies sought to change response times (Table 9). Four showed decreased times for processing and validating orders.576,578,581,584 One found increased time with the introduction of a CPOE and a CDSS system.582 Another found an increased time to checking the prescription with an e-Prescribing system compared with a paper based system (11 vs. 6 minutes, p < 0.01).575 Some changes were substantial. For example, a decrease from 115 minutes to 5 minutes for verification of a prescription in a study by Wielthrolter and colleagues.584 Three hospital studies found changes in work flows and processes with the introduction of a pharmacy information system574 and a CPOE and CDSS system with eMAR integration.580 Fewer callbacks occurred with the introduction of a CPOE and CDSS system integrated into a hospital EMR.540

Pearce et al.585 completed an ambulatory care based setting (three HIV clinics and two private pharmacies) and found decreased time to respond to a refill request and changes in communication patterns with MMIT involved in order communication. Mitchell and colleagues583 found that an eMAR system was associated with more accurate and complete recording of information. Ekeldahl and colleagues showed that the rate of picking up prescriptions did not change with the introduction of an e-Prescribing system.579


The evidence for the effects of MMIT on order communication comes from a limited number of studies, many of which were observational. The studies often include large numbers of participants or prescribing events. Most of the process evaluations show improvements, often in efficiency related to times and changing work patterns (Table 9).


Summary of the Findings for Process Changes

Some overlap and duplication of studies occurs between the dispensing phase of medication management and the phases of order communication and administering. Nine studies were identified as evaluating dispensing (Appendix C, Evidence Table 3).438,507,552,574,585,586,588–590 Much diversity was seen across these studies. In addition to these articles, a health technology assessment (HTA) report from the Canadian Agency for Drugs and Technologies in Health (CADTH) produced a systematic review on errors in dispensing and administering in hospitals in 2009.11 It sought to assess the effectiveness and economic impact of MMIT applications designed to improve medication dispensing and administering in hospitals.

Strengths and Limitations of the Evidence

Study methods included three RCTs,507,574,588 one cohort study,586 and five observational studies.438,552,585,589,590 The HTA report found 30 studies on dispensing, administering, or both, most of which were done using observational methods. In addition, many of these studies evaluated technologies that were older, no longer available, or only available in Europe. Overall the authors of the report stated that the evidence on the effectiveness of MMIT for improving medication dispensing is lacking, of poor quality, and has limited applicability.11 The year of publication of the nine papers in this AHRQ document were more recent: 1997,589 1999,574 2001,552 2007,438,507,586 2008,588 and 2010.585,590

General Study Characteristics

Participants. Raebel and colleagues507 and Halkin and colleagues552 reported data based on patients as the unit of study. Both studies included patients older than 65 years. Two reports studied pharmacists.574,588 All others reported data on medications or prescribing events as their unit of analysis.

Location. Four studies were set in community pharmacies552,585,586,588 and one in an HMO pharmacy.507 The other study locations were pharmacies and clinics in hospitals.438,574,589,590

Drugs and diseases. Aspirin for patients with diabetes was studied,588 and two others targeted groups of medications with high potential for interactions.507,552 One study included e-Prescribing in three HIV clinics and two private pharmacies.585 One article concentrated on drugs that are used heavily by seniors.507 All others studied the range of prescriptions available in the pharmacies.

Technology. The technology described in these dispensing studies varied considerably. Four studies evaluated pharmacy information systems,438,507,574,588 three looked at eMAR systems,585,589,590 and CDSS,438,507,552 one at an e-Prescribing system,586 and one evaluated a CPOE application.438 These systems were integrated with an EHR or EMR system,438,507,574,585 hospital information system,438,589 a pharmacy system,552,585,590 a laboratory system,574 a formulary,589 or a CPOE system.590 The HTA report includes a good description of their evaluation of MMIT that provides additional background information on administering and dispensing MMIT applications.591


Each of the main endpoints for the trials was found to be positive. Efficiency, monitoring, and preventive care outcomes were not reported in the nine studies.

Errors. Three of the four hospital based studies assessed errors and all showed improvements with eMAR,589 CPOE, CDSS and a pharmacy information system,438 and a pharmacy information system with CPOE.590 None of the ambulatory care studies assessed errors. The HTA report provided some evidence that BCMA is associated with reduced errors for dispensing (pharmacists) and administering (nursing), with the BCMA either as a stand-alone system or integrated with other health IT applications. Evidence on other outcomes or technologies in dispensing was found to be lacking or inconclusive.11

Adherence to guidelines. For pharmacists who were prompted electronically to suggest aspirin to patients with diabetes when they were filling other prescriptions, the use of aspirin increased.588

Changes in medications to be administered. Four of the four ambulatory studies demonstrated statistically significant improvements in what drugs were dispensed. Alerts to pharmacists improved dispensing of medications with high potential for interactions in an HMO pharmacy,552 while the use of contraindicated medications decreased with most of the decrease associated with amitripyline in another study.507 Refill utilization was improved585 and aspirin use increased while pharmacists were being prompted to include aspirin use when dispensing medications for patients with diabetes.588

Other process changes. Murray and colleagues574 showed changes in workflow for pharmacists (more time interacting and problem solving) and who they interacted with (more time interacting with peers and physicians). Workflow was also changed in another study using a pharmacy information system.574 A commercial EMR system reduced the time to process and fill a refill request for HIV medication.585 Nilsson and colleagues586 showed that acute prescriptions were picked up more often for an e-Prescribing system compared with a paper-based system (91 percent vs. 85 percent, p < 0.01).


Few reports studied dispensing. Three of these nine studies were RCTs. All studies showed statistically significant improvements in process. External evidence suggests that the existing studies dealing with dispensing are weak and dated, with reports of currently used MMIT applications not being readily available.


Summary of the Findings for Process Changes

Nineteen studies measured changes in process associated with the administering phase of medication management (Appendix C, Evidence Table 4). All deal with nurses and either pharmacists or physicians. The technology is complex, integrated and often part of a complete package of a hospital information system or an EMR or EHR system. All studies but two12,592 were done using observational methods.

As noted in the dispensing section, CADTH produced a systematic review in 2009 on errors in dispensing and administering in hospitals.11 This HTA report assessed the effectiveness and economic impact of MMIT applications designed to improve medication dispensing and administering. They found that the evidence on medication administering with MMIT was based on observational studies and that many of the studies were done on systems that have been updated or are no longer available. Many studies and descriptive papers that reported on medication administering and health IT, including Bar Code Medication Administration (BCMA) were reviewed for this report and were rejected. Most of the rejection decisions were because the medication administration system was stand-alone and not integrated with other MMIT applications. This nonintegration was especially true for older studies—most of the more recent studies show medication administering systems that are integrated. One good example of this integration is by Helmons and colleagues.593 Nonintegrated systems are not included in this report, as integration with other MMIT applications was an inclusion criterion.

Strengths and Limitations of the Evidence

One document was an RCT592 and one was a cohort study.12 All of the others were observational studies.34,438,439,581,583,589,593–602 Two were published in the late 1990s and 12 of the 19 were published since 2004.

General Study Characteristics

Participants. All but one study included nurses. Three studies included pharmacists,439,589,598 and four discussed physicians.465,592,593,596 The main focus of the study was medications or prescriptions,34,438,439,465,581,589,592–595,598,599,601,602 nurses597,600 and patients: infants12 and those whose ages were unspecified.596 Medications were not limited to a specific drug or class of drugs except for one study of the need for antibiotics596 and one study of aspirin use.592

Location. All of the studies but one were set in hospitals: acute care or tertiary,439,465,581,596,598,600–602 critical care units,12,593–595 pediatric standalone hospitals,438,465 general hospitals,34,438 other specialty hospitals,465,581 and the emergency department.597 Pediatric hospitals or wards were often studied: neonatal ward and adult ICU,595 general pediatrics,438 and pediatric nephrology.465 One of the studies was done in an ambulatory setting,592 and none were done in long-term care, community, or home settings.

Technology. The MMIT applications that were the focus of the administering phase of medication management were varied: automated drug dispensing system,599 BCMA systems,12,34,593,594,598,600,601 eMAR systems,12,34,439,583,589,601,602 CDSS,438,592,596 computerized unit dose drug dispensing system,465 CPOE,438,581,595,597 e-Prescribing,439,602 and a pharmacy information system.438

The MMIT systems that are integrated with these systems above are most likely to be hospital wide or pharmacy systems: a CPOE system,593 EHR and EMR systems,438,581,592,593,596,601 hospital information system,34,438,465,589,598 imaging systems,581,597 laboratory systems,581,597 eMAR,594 a formulary,589 and pharmacy information systems.12,34,439,583,595,597,599,601


Errors. Thirteen studies evaluated administering errors. The issue of errors in administering drugs using MMIT is complex as many errors identified in MMIT systems are related to transcription and timing. These easily measured errors may be masking other more substantial errors. Eight studies had major endpoints that were found to be positive in reporting decreased errors.438,439,465,581,589,594,601,602 Another measured variances (differences between the order and administered medications) and found significant reductions after the introduction of a CPOE system integrated with the pharmacy and eMAR systems in a hospital.595 The relative risk reduction in many of the studies was high and often approximately 40 to 50 percent. Four studies had endpoints that were not found to show statistically significant improvements.34,583,593,598 Another hospital-based study showed increased errors, mostly related to a BCMA system,12 because the BCMA system recorded issues such as timing of medication administering more accurately. The HTA report from CADTH also provided evidence that BCMA reduced errors in administering medications in hospitals.11

Efficiency. Efficiency is also important in medication administering. Four of five studies that measured efficiencies showed improvements. One study showed that time from ordering to administering in a hospital setting decreased from 90 minutes before implementation of comprehensive MMIT systems to 11 minutes.438 Another article that measured time efficiencies had similar reductions (79 percent vs. 89 percent of medications were administered within 1 hour of ordering).439 An eMAR system reduced time from ordering to administering from 325 to 88 minutes.581 Shirley and colleagues599 did not find a change in time to administering after implementation of an eMAR. No changes in time allocation were seen for direct patient care and medication administration after a BCMA system integration for hospital nurses.600 In contrast, Banet and colleagues described a system that integrated CPOE and eMAR and showed that nurses spent less time on paper documentation and searching for charts and more time on working with computers and charting in patient rooms with no changes in documentation time overall or time spent on direct patient care.597

Adherence to guidelines. One study with an anesthesia medication system had improvements in adherence to administering antibiotics during surgery.596 Shirley and colleagues599 found improved adherence to scheduled dosing. Persell and colleagues592 identified no difference in self-reported aspirin use.

Other changes in process. Helmons and colleagues593 found no changes in error rates (they had few errors at baseline) but measured improved charting and labeling.

Summary. Although few studies evaluated administering with the use of MMIT, most of the 19 showed improvements, mostly in the realm of errors and efficiencies. Results were mixed with respect to whether the MMIT systems for drug administering altered the time nurses spent on various tasks.


Summary of the Findings for Process Changes

Medication monitoring can been defined as the process of assessing a patient’s response to a medication and documenting its outcomes.603 Suboptimal medication monitoring describes a common pathway of systems failures that underlie monitoring errors and can be categorized as over, under, or inappropriate monitoring. Medication monitoring errors generally refer to one of three situations: inadequate laboratory evaluation of drug therapies, or a delayed or failed response by the clinician to symptoms (patient reported aspects of their disease or disorder), or to clinician observed or measured signs of the condition or of drug toxicity, or laboratory evidence of toxicity.604 Therefore, for the purposes of this report, we divided the health IT interventions designed to improve medication monitoring into studies that enhance laboratory-, sign-, or symptom-based medication monitoring. In the clinician and patient encounter the patient reports symptoms they are experiencing (e.g., fatigue, sudden weight gain, or dizziness) and the clinician observes or measures signs of the disease or disorder (e.g., blood pressure, heart rate, fever). Clinicians integrate information gained from assessments of symptoms, signs, and results of laboratory tests to determine disease status, often putting varying weights on the three aspects.

Previous systematic reviews provided information on the impact of health IT on medication monitoring.605–607 However, these systematic reviews are limited to a specific type of medication monitoring system (e.g., clinical event monitors), a single practice setting (e.g., ambulatory or acute care), or are more than 5 years old. This evidence report yielded a total of 47 articles describing health IT intervention designed to improve one or more change in process related to the medication monitoring phase in the acute, ambulatory, or long-term care settings (Appendix C, Evidence Table 5).397,401,402,407,412,437,442,446,461,472,473,477,481,505,511,515,516,518–520,526–528,534,537,541,543,553–555,608–624

For consistency, author-reported changes in process were selected. By definition, a study which showed statistically significant changes in at least half of its main endpoints was considered a positive study. Overall, 70 percent (33 of 47 studies) of the articles were rated as positive studies.397,401,402,407,412,437,461,472,473,477,505,515,516,527,528,537,541,554,555,608,610,612–623

Study methods included 30 RCTs397,401,402,407,412,505,511,515,516,518–520,526–528,534,537,541,543,609–613,616–620,624 and 17 observational studies.437,442,446,461,472,473,477,481,553–555,608,614,615,621–623 Monitoring, along with CDSS, are the two areas that include the highest proportion of RCTs.

General Study Characteristics

Intervention targets. Nearly three-quarters (72 percent; 34 of 47) of the health IT medication monitoring interventions targeted physicians exclusively.397,401,402,407,412,437,442,446,461,472,481,505,511,515,520,527,528,534,543,553,554,609–611,613,615–620,622–624 Eight of these studies targeted physicians along with other health care professionals,518,519,526,537,541,555,612,621 four targeted pharmacists,473,477,516,614 and one targeted nurses.608

Location. The overwhelming majority of health IT medication monitoring interventions studies (70 percent; 33 of 47) were conducted in an academic setting.397,401,402,407,412,437,442,446,461,472,473,477,481,505,515,518–520,527,534,554,608–611,614–619,621,622 Of those that were conducted in an academic setting, 19 of these studies402,407,437,446,477,505,515,518,519,527,554,609–611,614–616,618,619 came from the following benchmark institutions: (1) Brigham and Women’s Hospital/Partners Health Care, (2) LDS Hospital/Intermountain Health Care, (3) the Department of Veterans Affairs, and (4) the Regenstrief Institute.

The preponderance of studies (59 percent; 28 of 47) took place in the ambulatory care setting.472,505,511,515,516,518–520,526–528,534,537,541,543,553–555,609–613,616,617,619,620,624 Eighteen of the studies took place in the acute care,401,402,407,412,437,442,446,461,473,477,481,608,614,615,618,621–623 and one in the nursing home setting.397

Patient populations studied. The vast majority (n = 36) of the health IT interventions targeted the adult population.397,401,402,407,412,461,472,481,505,511,515,516,518–520,526–528,534,537,541,543,554,609–612,614–617,619–622,624 Only four of the 44 articles were conducted in the pediatric population,437,442,446,553 and two targeted both adult and pediatric patients.477,613 Five studies did not explicitly mention the study patient population studied.473,555,608,618,623

Type of medication monitoring. The majority (n = 29) of the health IT interventions focused on laboratory-based medication monitoring.397,401,402,407,412,442,461,472,473,477,481,511,515,516,527,528,534,537,541,543,555,609,611,612,614,615,619,620,623 Five studies505,518,526,613,624 targeted sign-based (clinician observed or measured aspects of the disease process) medication monitoring. While three interventions focused on symptom-based monitoring (patient reported symptoms),520,608,621 ten studies437,446,519,553,554,610,616–618,622 provided a combination of laboratory-, sign-, or symptom-based medication monitoring.

A significant degree of overlap (n = 36) of health IT interventions that involved laboratory-, sign-, or symptom-based monitoring along with the prescribing phase of the medication use process existed.397,402,412,437,446,461,472,473,477,481,505,515,518,519,526–528,534,537,541,543,553–555,610,613–620,622–624 Prescribing was most commonly associated with laboratory-based medication monitoring (n = 30),397,402,412,446,461,472,473,477,481,515,519,527,528,534,537,541,543,553–555,610,614–620,622,623 followed by sign-based medication monitoring (n = 15),437,446,505,518,519,526,553,554,610,613,616–618,622,624 and symptom-based medication monitoring.437,553 This overlap was most often a result of the evaluation of clinical practice guidelines, order sets, or both that contain prescribing and monitoring elements.

Drugs and diseases. Twenty-four of the health IT medication monitoring interventions studies dealt with chronic disease management such as asthma,446,553 asthma and chronic obstructive pulmonary disease,518,613 congestive heart failure and coronary artery disease,519 deep venous thromboembolism,402 depression,520 diabetes,537,610,619 diabetes and coronary artery disease,554,616 HIV,527 hyperlipidemia,515,528,534,541,543 hypertension,505,526,624 and multiple common chronic conditions.617,620,623 Sixteen studies addressed potentially nephrotoxic,397,615 hepatotoxic,241 or cardiotoxic473 medications with a narrow therapeutic index,442,461,555,618 and certain laboratory and medication combinations.407,412,481,511,516,609,611,612 Four provided guidance about potentially inappropriate antibiotic management,401,477,614,622 and three provided information about pain management.437,608,621

Technology. Almost all of the included studies regarding MMIT interventions (91 percent; 43 of 47)397,401,402,407,412,442,461,472,473,477,481,505,511,515,516,518–520,526–528,534,537,541,543,553,555,608,609,611–624 used a CDSS with alerts or reminders. Three studies used a CPOE system without alerts437,442,446 and one study involved the use of a personal health record (PHR).610 Twelve of the studies used interruptive alerts to display and prompt the clinician for an immediate response while providing patient care.397,407,412,472,481,505,543,608,609,611,613,624


As noted above, more than two-thirds (33 of 47) of the interventions were associated with a positive process outcome. A number of themes emerged from the variety of interventions that were conducted in various health care settings, using varying degrees of technological sophistication, and providing information to a number of health care professionals, as well as directly to patients.

By type of medication monitoring. The majority of the health IT interventions focused on laboratory-based medication monitoring.397,401,402,407,412,442,461,472,473,477,481,511,515,516,527,528,534,537,541,543,555,609,611,612,614,615,619,620,623 Of these 29 studies, 22397,401,402,407,412,461,472,473,477,515,516,527,528,537,541,555,612,614,615,619,620,623 or 76 percent of these interventions showed statistically significant changes in at least half of these main endpoints. Two505,613 of the five505,518,526,613,624 studies (40 percent) that targeted sign-based medication monitoring showed that greater than 50 percent of the process endpoints improved. Of the three interventions that focused on symptom-based monitoring,520,608,621 two608,621 resulted in statistically significant changes in at least half of their main process endpoints. Ten studies437,446,519,553,554,610,616–618,622 provided a combination of laboratory-, sign-, or symptom-based monitoring, and seven437,554,610,616–618,622 or 70 percent showed statistically significant changes in at least half of their main process endpoints.

By type of intervention. One of the most frequently reported types of intervention (n = 12) provided decision support to improve chronic disease management (i.e., prescribing, monitoring, and clinical endpoints).505,520,526,528,537,541,543,554,555,610,616,624 The type of chronic diseases varied based on patient population, but included the management of asthma, chronic obstructive pulmonary disease, depression, diabetes, hyperlipidemia, and hypertension. Overall, 67 percent of these interventions resulted in a statistically significant change in at least half of its major endpoints. Another common intervention (n = 10) assessed the adherence to guideline recommendations for a variety of acute and chronic medical conditions including asthma, atrial fibrillation, coronary artery disease, cardiovascular disease, congestive heart failure, chronic obstructive pulmonary disease, depression, diabetes, glucose regulation in the ICU, pain management, and peripheral vascular disease.402,412,446,518,519,553,617,620,621,623 Overall, 60 percent of these interventions resulted in statistically significant change in at least half of its main endpoints.

Other common interventions (n = 8) included providing alerts and reminders to obtain laboratory testing for newly prescribed or chronically used medications.407,442,472,511,516,609,611,612 Overall, 50 percent of these interventions showed a statistically significant change in at least half of their main endpoints.

Seven studies targeted changing prescribing behavior by providing laboratory-, sign-, or symptom-based monitoring information for potentially nephrotoxic medications, medications for asthma and COPD, and hyperlipidemia.473,505,515,534,613,614,622 Overall, 86 percent of these interventions resulted in improvements in at least half of the major process changes reported as endpoints. Another metric commonly assessed was the response time to a variety of alerts (n = 7) including the management of narrow therapeutic index and potentially nephrotoxic medications, initiation of primary and secondary prevention, and time to pain assessment and management.461,477,481,527,615,618,619 Overall, 71 percent of these interventions showed statistically significant improvements in at least half of its main endpoints.

Finally, two interventions assessed pain management including error reassessment rate and patient controlled analgesia order set use.437,608 Overall, both of these interventions showed statistically significant changes in at least half of its main endpoints.

In our analysis, 70 percent (33 of 47 studies) of the included studies showed statistically significant changes in at least half of their main endpoints. Of these studies, the majority targeted physicians exclusively (n = 34), were conducted in academic institutions (n = 33), were developed for use in the ambulatory care setting (n = 28), focused on the adult population (n = 36), and provided CDSS with alerts or reminders to support chronic disease management (n = 12). When compared with sign- or symptom-based medication monitoring, laboratory-based medication monitoring studies were most likely (76 percent of the time) to be associated with a statistically significant change in at least half of its main endpoints. Moreover, these laboratory-based medication monitoring studies were conducted in a variety of health care settings including ambulatory, acute, and long-term care. The most successful types of studies focused on changing prescriber behavior, improving response time to generated alerts, and improving the diagnosis and management of chronic diseases.

Reconciliation, Discharge Summaries, and Education

Summary of the Findings for Process Changes

Reconciliation. Reconciliation of medications using MMIT is a complex process. Some of this stems from the complexity of medication management itself. Another issue is the challenge of interoperability of health IT across health care systems. The problem of medication reconciliation is especially acute for patients who receive care across settings: from hospitals, specialists, and primary care—most often the elderly and those with multiple health challenges. Two review articles provide documentation of the difficulties of effective medication reconciliation using health IT and the lack of published evidence to support its value.625,626

Four studies on medication reconciliation are included (Appendix C, Evidence Table 6).13,14,627,628 One was a cohort study628 and the others are quantitative observational. All were set in hospitals with the reconciliation done at discharge or transfer to another facility. One hospital was a Statepsychiatric hospital13 and the others were general hospitals.

One study was PDA-based,13 one was based on an e-MAR system,14 and the others were based on integrated systems: CDSS and COPE within an EMR,627 and an e-Prescribing system integrated with a pharmacy information system.628

All studies showed substantial improvement in agreement among records of medications provided by various clinicians involved in the care of the patients (Appendix C, Evidence Table 6). For example, one Dutch study showed improvements in agreement on prescriptions between the pharmacists and general practitioners with e-Prescribing compared with paper systems at discharge (31 percent vs. 49 percent) and at 10 days after discharge (33 percent vs. 53 percent).628 Grasso and colleagues13 showed a decrease in errors in the psychiatric hospital with the use of PDAs for reconciliation compared with paper summaries (rate of errors before PDAs was 22 percent compared with 8 percent after). Poole and colleagues14 also showed improvements in prescribing (more therapeutic drug duplications were identified and resolved with an automated discharge medication worksheet for physicians).

In summary, although few studies exist on MMIT for medication reconciliation, the four included showed substantial improvements in the ability to electronically reconcile medication lists and make the necessary adjustments resulting in reduced errors and better prescribing.

Education. Only one article targeted the education associated with MMIT systems and measured change in processes as their main endpoint (see Appendix C, Evidence Table 6).537 This RCT showed that combining patient education with submission of blood glucose levels to ambulatory care clinicians showed improvements in prescribing as well as improved hemoglobin A1c levels. Most of the articles targeting educational aspects of medication management that measured changes in knowledge are covered in the section with intermediate outcomes.

Combined Phases of Medication Management

Summary of the Findings for Process Changes

Although some studies in this report assessed systems that covered the whole medication management process (five phases plus reconciliation and education), only one provided cross-phase study with changes in process. This observational study by Mahoney and colleagues438 took place in a U.S. pediatric hospital and an affiliated acute care hospital. The study started in 2002 and was completed in 2006 with publication in 2007. The hospitals included a full EMR system that incorporated CPOE, CDSS, and the pharmacy information system into one clinical information technology (hospital information system). All aspects of the medication management system were addressed electronically. An analysis of 1.4 million orders after implementation as compared with a similar number before implementation showed reductions in drug allergy violations, excessive doses, incomplete or unclear orders, and therapeutic duplication.


Summary of the Findings for PDAs

We included 13 studies using PDAs.13,514,534,553,563,593,629–635 The studies often covered multiple medication management phases, such as prescribing (n = 7), order communication (n = 1), administering (n = 3), and monitoring phases (n = 6), as well as reconciliation (n = 1). Outcome measures focused on process and other intermediate measures, only two measured patient outcomes (blood glucose levels in both cases).630,631 Eight of the studies included a CDSS component.514,534,553,563,593,630,631,634 Two applications were tied to handheld BCMA units,593,635 and two were used for e-Prescribing.639,632 Most interventions targeted specific diseases such as diabetes,630,631 asthma,553,634 cancer,633 high blood pressure,534 psychiatric patients,13 or the use of certain classes of medications such as nonsteroidal anti-inflammatory drugs514 and antibiotics.563 Two studies were qualitative,629,632 two mixed methods,633,635 five observational,13,553,563,593,631 and four were RCTs.514,630,633,634 Of the quantitative studies, five reported significant improvements as a result of the intervention13,563,630,631,634 and four reported no significant effects.514,534,553,593 An RCT of adherence to nonsteroidal anti-inflammatory drug prescribing guidelines in an ambulatory clinic showed stable levels of safe prescribing in the intervention group and a deterioration in the control group given PDAs without the guidelines.514 Similarly, a PDA which provided physicians with Framingham scores and recommendations for patients at risk of high blood pressure, found no difference in levels of screening of patients and no effect on lipid management.534 A PDA-based CDSS for international asthma guidelines improved quality-of-life scores for patients and cost reductions.634 A crossover RCT of diabetic patient use of an insulin regimen dosage optimizer showed improvement in blood glucose levels during the phase when the advice was switched on.630

Intermediate Outcomes

Summary of the Findings

Articles measuring intermediate outcomes as their main endpoint were selected. We focused on the intermediate outcomes of: use; measures which were correlated with use (such as ease of use of the system, perceptions of users of the system, computer experience, etc.); knowledge, skills, and attitudes of the users; satisfaction; and usability (Table 10). Few hypothesis-driven studies with comparison groups assessed such intermediate outcomes as their main measure; 42 studies published in 44 articles were retrieved (Appendix C, Evidence Table 7). Only six studies were RCTs with quality scores from two to seven out of nine. The study results tended to show positive levels of satisfaction and use and measured a number of correlates of both to determine driving factors barriers, or both. Some negative impacts of systems on work processes were found.

Table 10. Intermediate outcomes across the phases for medication management.

Table 10

Intermediate outcomes across the phases for medication management.

Strengths and Limitations of the Evidence

Of the 43 included studies, 25 were observational, nine mixed methods or qualitative, six RCTs, and three cohort (Table 11). The RCTs rated two,636,637 four,638 six,639 and seven540,640 out of eight on the methods quality scale. The cohort studies scored three,641 five,642 and six643 out of nine. Studies of complex interventions often covered more than one phase of medication management.540,644–647

Table 11. Study designs used in studies measuring intermediate outcomes across the phases for medication management.

Table 11

Study designs used in studies measuring intermediate outcomes across the phases for medication management.

General Study Characteristics

Study participants tended to be practicing clinicians (Table 12). Most of the studies were conducted in hospitals (n = 27) or primary care (n = 17), one in long-term care, and four in pharmacies, and assessed intermediate outcomes for health care staff. Twenty-two of the studies were performed in academic settings. Prescribing was the most commonly studied phase of medication management, but each other phase was represented. Three systems used hand-held devices. CDSS, e-Prescribing, and CPOE systems were most commonly studied. Most studies did not report on the proprietary nature of their systems, 17 studied commercial systems and seven were home grown. Many studies looked to correlate use of medication management systems with other factors. Only nine studies assessed intermediate outcomes for patients (Table 13).633,637,639,641,642,648–651

Table 12. Clinician study participants in studies assessing intermediate outcomes across the phases of medication management.

Table 12

Clinician study participants in studies assessing intermediate outcomes across the phases of medication management.

Table 13. Patient study participants in studies assessing intermediate outcomes across the phases of medication management.

Table 13

Patient study participants in studies assessing intermediate outcomes across the phases of medication management.

Prescribing and ordering. Twenty-six studies looked at intermediate outcomes for interventions aimed at the prescribing phase (see Appendix C, Evidence Table 22). CDSS (n = 12), CPOE (n = 11), and e-Prescribing systems (n = 6) formed the bulk of the primary systems studied. Three studies assessed usability issues related to CPOE or CDSS.638,647,652 One study focused on the use of standards for medical history, formulary, and benefits.653 Satisfaction and correlates of satisfaction were measured in ten studies;636,637,644,645,654–659 use and measures correlated with use were studied in 11 studies.534,643,649,650,653,660–665 Pirnejad and colleagues666 used mixed methods to determine the impact of CPOE on the collaboration of nurses and physicians in hospitals. Glassman and colleagues667 looked at the impact of drug-drug interaction alerts on physician knowledge over time. Two studies assessed perceptions of technology on work.656,658 Participants were generally health care providers, located in either hospitals (16 studies), primary care (ten studies), or both, and one pharmacy.540 The majority were performed in the United States.

Order communication. Four studies looked at the order communication phase;540,645,668,669 three focussed on e-transfer of prescriptions,540,645,668 and all studied the perceptions of pharmacy staff as well as other stakeholders. Rupp and Warholak645 administered a survey and followed up with interviews of American chain community pharmacy staff to assess their attitudes towards e-Prescribing and recruited a sample of 1094 pharmacists, technicians, and interns from 276 pharmacies. Porteous and colleagues668 surveyed 494 patients, 145 general practitioners, and 148 pharmacists, and held interviews and focus groups to assess peoples’ views regarding the upcoming implementation of e-Transfer of prescription information in the United Kingdom. On the other hand, Kirking and Thomas669 performed a survey looking at pharmacists’ attitudes towards computer technology used to detect and prevent adverse drug interactions, and correlated their findings with pharmacist computer use. Their sample included 218 pharmacists in Michigan using one of two pharmacy computer systems and a group of nonusers. Johnson and colleagues540 assessed the perceived usefulness of alerts and override comments appended to e-Prescriptions.

Dispensing. Two studies looked at dispensing. Chan646 looked at factors associated with the use of drug dispensing and eMAR systems in nursing homes by analyzing surveys of long-term care facilities. Rupp and Warholak645 assessed pharmacist personnel staff views of e-Prescribing.

Administering. Eight studies assessed technologies used at the administering phase. O’Morrow,670 Hurley,671 Holden,672,673 Topps,674 and their colleagues assessed American nurses’ attitudes and satisfaction toward bedside point-of-care BCMA technologies to verify drug administering. The usage patterns of BCMA verification in five medical departments of a Dutch hospital were tracked.675 The perceptions of the effects of a newly implemented CPOE in two groups of 211 Dutch nurses previously using different paper prescription systems were assessed by Niazkhani et al.644 Chan646 assessed factors associated with medication administration records use in nursing homes.

Monitoring. Four of the five monitoring studies focused on patient self-monitoring. Weingart and colleagues641 measured the use of PatientSite, a patient internet portal, by 416 patients in three primary care practices to facilitate communication between physicians and patients regarding medication adherence and adverse effect rates. In an RCT involving 117 patients, Ross and colleagues639 provided patient online records for heart failure patients and assessed self-efficacy. McCann and colleagues633 performed an RCT on 112 cancer patients with a mobile phone application to monitor their chemotherapy toxicity symptoms compared with standard care and measured perceived benefits. Schmidt and colleagues642 tested patient adherence using a telemonitoring intervention which included a beeping medication box integrated with the patient’s EHR data. The final study assessed the usability of a CDSS Antibiotic Wizard in an ICU using an ergonomic survey tool to detect deficiencies in the system as viewed by 40 physicians.647

Reconciliation. One study evaluated patient and physician satisfaction and perceptions of a discharge reconciliation application.651

Education. The study by Liu and colleagues648 is the only study focusing on the education aspect of medication management. Their study provides hospitalized patients in a Taiwanese hospital with a system that integrates their pharmacy, EMR, and CPOE information into an education tool to increase their knowledge of their medication regimens. Knowledge was assessed in pre-post surveys of 154 patients and they reported perceived knowledge gains.


Prescribing. Many studies measured use descriptively (e.g., presenting the percentage of time e-Prescribing was used for writing prescriptions), but did not meet our criteria for having comparison groups and being hypothesis-driven. From our included studies, use and measures that were correlated with use were frequently measured. Rogers and colleagues660 found that usage rates increased following iterative changes made to a CDSS system based on feedback from users. Shannon and colleagues643 reported a significantly higher rate of e-Prescribing after emergency department physicians were allocated hand-held PDAs with e-Prescribing software. Access to medication history was more frequently used for patients with low socioeconomic status and a greater number of medications.650

Some studies found that ease of use and perceived usefulness relating to improved care and care processes of the MMIT applications were positively correlated with level of use.653,661–664 Kralewski and colleagues665 found that use of e-Prescribing systems was correlated with such cultural factors in primary care practices as trust, adaptability, and business orientation. Wang and colleagues653 found that a positive performance measure based on ease, efficiency, and care was correlated with nonuse of an e-Prescribing system incorporating standards for medication history, benefits, and formulary. Use of a CDSS in an Australian hospital was positively correlated with computer sophistication and access to laboratory data and negatively with years of experience.661 Musser and Tcheng640 measured preference and use of a graphic interface compared with a text-based interface for an anesthetic CPOE; clinicians used the graphic interface more often, but both interfaces had their proponents. A study assessing the frequency of use of three common pediatric order sets found differential use rates, with asthma order sets used significantly more often than both appendectomy and community-acquired pneumonia order sets.649

Usability. Three studies looked at usability and also included data on comparison groups.638,647,652 Rosenbloom and colleagues638 found that highly visible hyperlinks significantly increased the use of educational material and patient information. Rohrig647 and Li and colleagues652 used usability testing to identify issues in a CDSS and CPOE system respectively. Their results were used to inform new iterations of their existing systems.

Satisfaction. Ten studies measured satisfaction as a main outcome. Satisfaction with various systems used by various health care providers tended to be positive.644,645,656–658 Satisfaction was lower in an intervention group of residents provided with CDSS within an e-Prescribing system in a small RCT, but they only used the system for 2.8 percent of their prescriptions.636 No difference in satisfaction levels were detected for patients or physicians using a discharge CPOE application compared with usual care.637,659

Differences in satisfaction and perceptions of the systems were found between nurses and physicians;656,657 medical and surgical staff;654,657 and residents compared with physicians.654 Perceptions of the system impact on work were also found to be different among health care providers.656,658 Other factors correlated with satisfaction included computer sophistication, experience, training, system characteristics, and perceived improvements in care.654–657

Knowledge. Glassman and colleagues667 found no change in physician knowledge of selected drug-drug interactions over 2 years of using a CDSS with alerts in 97 American primary care physicians, although most preferred having the system.

Attitudes. Pirnejad and colleagues666,676 studied nurses’ and physicians’ attitudes to the impact of CPOE on the nurse-physician collaboration in the medication process. They found that the original paper-based system and the new CPOE system supported their work and collaboration differently. The new system led to problems in the synchronization and feedback aspects of the joint medication care, leading to the recognition that new systems do not always directly replace the work entailed in old systems and that care processes can be negatively impacted.666

Attitudes toward MMIT often varied by groups of users. Junior students were more positive about CPOE than interns and residents.677 Similarly, using a diffusion of innovations model, Rahimi and colleagues678 found that a CPOE was perceived to work better for nurses than physicians (57 percent vs. 13 percent); further, more physicians felt that the system was not adapted to their practice and more would have liked a return to the old system, compared with the nurses.

Johnson and colleagues540 measured perceptions of pharmacists to appended alerts and override comments on e-prescriptions; they found some information ( e.g., allergy alerts) more useful than others (e.g., insurance status).

Order communication. Of the four studies assessing the communication phase, outcomes assessed included only satisfaction645 and attitudes.540,668,669

Rupp and Warholak645 found that chain community pharmacy personnel who dispensed e-Prescriptions were generally satisfied with e-Prescribing and rated e-Prescriptions more favorably than paper prescriptions on seven criteria related to safety, efficiency, effectiveness, communication, and relationships with patient and prescriber. They further produced 11 best practice recommendations to improve e-Prescribing in a community pharmacy setting. In the United Kingdom, before e-Transfer of prescription information being implemented, Porteous and colleagues668 found that various stakeholders viewed e-Transfer as a good idea (68 percent of patients [95 percent confidence interval (CI), 64 percent to 72 percent], 83 percent of general practitioners [95 percent CI, 77 percent to 89 percent], and 87 percent of community pharmacists [95 percent CI, 82 percent to 92 percent]). Concerns were expressed about security and sharing of confidential information. Benefits revolved around improved repeat prescription processes, convenience, and a greater role for pharmacists in medication management.

The potential for pharmacy systems to assist pharmacists in detecting adverse drug interactions by having greater access to patient information in the form of patient medication profiles was assessed by Kirking669 in a survey study asking pharmacists using two systems and a third group using no system how often they detected potential drug interactions and how often they contacted prescribers. Computer users reported an average of twice as many detected interactions per week (16.1 vs. 8.7, ns) and had significantly more contacts with prescribers per week (21.5 vs. 16, p <0.05). The majority of the differences were the result of users of one of the unnamed computer systems, while the other groups had use rates similar to the noncomputer group.

Dispensing. One study suggests that drug dispensing and eMAR technologies were used more in nursing homes with higher occupancy rates; fewer metropolitan than rural homes using systems.646 Rupp and Warholak645 presented best practice recommendations for community pharmacies using e-Prescribing based on surveys showing satisfaction with e-Prescribing in community chain pharmacies.

Administering. Administering phase articles focused on nurses using BCMA systems670–675 or eMAR systems.646 O’Morrow670 found no differences in the attitudes of 17 nurses regarding patient care, charting, computer benefits, computer capability, computer characteristics, legal issues, or management tools before and after implementation of a BCMA system. Hurley and colleagues,671 on the other hand, found significant improvements on a satisfaction scale of 1,087 nurses after implementation of a similar system for efficacy, safety, care, and access factors. Holden et al.672,673 assessed nurses’ perceptions and acceptance of BCMA; perceived ease of use and perceived usefulness; predicted satisfaction with the process before and after BCMA.673 In their second study, nurses’ perceptions of the medication administering process changed with the implementation of BCMA compared with a control group; while perceived safety, accuracy and consistency in checking patient identification improved, ease of use, usefulness, and efficiency were perceived to decrease.672 Topps and colleagues674 looked at nurse, pharmacist, and respiratory technicians’ perceptions of BCMA before and after implementation; surveys after implementation showed that the staff felt that fewer medication errors occurred with a smoother administering of medication; they did, however, perceive that more time was spent administering medications, which took time away for other patient care. Overall, satisfaction and perceived benefits were improved in the study, by Niazkhani and colleagues,644 of nurses who went from two paper-based prescribing systems to a CPOE system. Perception of effects did depend on which previous paper system they were used to, and workflow support was perceived as worse by both groups.

Van Onzenoort and colleagues675 measured usage of bar code point-of-care systems by nurses and found that only 55 percent of 23,492 drug administrations were verified using the system; use depended on department, drug route, nurses available, nurse age, and timing of administering.

Monitoring. Most monitoring phase interventions were geared toward patients and showed positive effects on the intermediate outcomes of use, knowledge (self-efficacy), and satisfaction. PatientSite patient internet portal had a 48 percent response rate to index messages and a higher rate of ADE reporting via site (13 percent vs. 3 percent nonresponders, p = 0.01).641 Ross and colleagues639 found that online records for heart failure patients improved self-efficacy (91 percent vs. 85 percent p = 0.08) and satisfaction. Chemotherapy patients using a mobile phone symptom system reported a number of benefits: better communication, better symptom management, and reassurance of physician access.633 Finally, patients who telemonitored their congestive heart failure issues consistently used a beeping medication box integrated with their EHR to increase adherence to their regimen.642

One study assessed usability; the Rohrig647 usability study of Antibiotic Wizard showed good usability. Physicians did report some weaknesses in the design of health IT which were to be used to inform future versions.

Reconciliation. A study of satisfaction with a reconciliation system found that patients reported satisfaction for self-reported perceptions of clear instructions on what medications to take, how much and how often the medications were to be taken, other instructions on taking the medication, potential side effects, and general understanding of the medications. Health care provider perceptions of satisfaction with reconciliation and instructions did not differ for five factors except for three factors reported by physician assistants and nurse practitioners. Physician assistants and nurse practitioners reported that patients had clearer instructions on discharge (p = 0.01); how much, how often, and when to take their medications at home (p = 0.05); and the medication discharge process was viewed as being sufficient for them as caregivers (p = 0.0003).651

Education. Use of an integrated pharmaceutical system to provide information to patients to understand the pharmacological properties of their medications resulted in significantly improved patient knowledge after use of the system.648

Economic Outcomes

The introduction of health IT in the medication management process holds the promise of increasing efficiencies, improving quality of care, and reducing costs. However, even if these technologies are effective, they are expensive to implement and maintain and thus a review of the economic literature to determine cost-effectiveness and value for money for such interventions is warranted.

All studies passing the inclusion criteria that were considered to be cost or economics studies were reviewed and categorized into two groups based on the type of economic evaluation used in the analysis: (1) full economic evaluations; and (2) partial economic evaluations. A full economic evaluation is the comparative analysis of alternative courses of action in terms of both costs and consequences. Therefore, the economic evaluations which identify, measure, value, and compare the costs and consequences of the alternative being considered were further classified into one of the three categories: (1) cost-effectiveness analysis; (2) cost-utility analysis; and 3) cost-benefit analysis.679 The label, partial economic evaluation, indicates that the studies do not entirely fulfill both of the necessary conditions for a full economic evaluation (i.e., costs and consequences). However, cost analyses can provide useful information on ‘upfront’ costs compared with ‘downstream’ cost avoidance.679 For this reason, both full economic evaluations and cost analyses were included in this review. In each of these classifications, articles were further categorized by setting (i.e., hospital or community).

Descriptive information on the populations, interventions evaluated, the study year, perspective, and country of study were abstracted for each study. Data specific to the costs and effectiveness of each comparison were also abstracted and summarized in Appendix C, Evidence Tables 8a and 8b.

Full Economic Evaluations

Only five of the 31 (16 percent) economic articles reviewed conducted economic evaluations that provided information on the incremental costs and the incremental effects of an MMIT application. The following section reports the findings of five economic evaluations dealing with the use of CPOE (n = 2) and CDSS systems (n = 3) for improving prescribing practices for various conditions (Appendix C, Evidence Tables 8a).

Hospital. The potential economic consequences of implementing an eMAR system were estimated in a study using data from various literature sources.680 In a tertiary care hospital setting, the projected incremental effectiveness of the eMAR was 261 ADEs averted over the 10-year time horizon compared with the standard paper ordering approach. Given that the incremental cost of the new electronic medication ordering system was USD$3.3 million during that same period, the incremental cost-effectiveness ratio was USD$12,700 per ADE averted.

A 1-year RCT in a hospital family medicine center evaluated the effect of three reminder systems on compliance with tetanus vaccination.530 A computer-generated physician reminder system was found to cost $0.43 per additional vaccination recorded compared with usual care. The telephone reminders to the patients cost $5.43 per additional vaccination, while the mailed letter reminder to the patients to recommend tetanus vaccination was $6.05 versus standard care.

Community. A group of Norwegian researchers510 conducted a cost-effectiveness analysis alongside an RCT involving 146 general practices from two separate geographical areas. The objective of the evaluation was to compare the costs and effects of a multifaceted intervention, including computerized reminders to physicians, aimed at improving prescribing of antihypertensive and cholesterol-lowering drugs compared with the passive dissemination of guidelines. The cost per additional patient started on a thiazide rather than another antihypertensive agent in the intervention group was compared with usual care. Over the 1-year study period, the authors calculated that the incremental cost-effectiveness ratio of the intervention was USD$454 per additional patient started on thiazides. It was found that reduced drug expenditures based on increased use of thiazides did not outweigh the costs of the intervention. The authors commented that if the effect was sustained for a second year, the intervention would have been expected to lead to savings.

A Spanish study published in 2005634 evaluated the cost-effectiveness of a CDSS designed to promote guidelines for the treatment of asthma. Over the 1-year study period, the authors found that from a societal perspective, the intervention dominated standard care (i.e., less costly and more effective). From the health care payer perspective, the incremental cost-effectiveness ratio was €61 per percentage point reduction in the St. Georges Respiratory Questionnaire.

Setting not stated. Using information obtained from a systematic review of the literature, Karnon et al.681 developed a decision tree model to estimate the net benefits of three interventions aimed at reducing medication errors (i.e., CPOE, ward pharmacists, and bar coding), either through prevention or detection. Based on estimated quality of life utility decrements associated with experiencing a preventable ADE, it was concluded that the CPOE had a mean net benefit of GBP £31.5 million, ward pharmacists of GBP £27.25 million, and bar-coding of GBP £13.1 million over a 5-year time horizon with the intervention and maintenance costs included in their model. It was noted that the monetary value of lost health needed to be included for the interventions to have a high probability of producing positive net benefits.

Partial Economic Evaluations

Most of the economic literature reported the results of partial evaluations (26 of 31 studies, 84 percent). All of these evaluations took the form of cost analyses. In other words, the costs of the alternatives were examined separately and the effectiveness, efficacy, or both measures were not used in the analyses, which results in an inability to answer efficiency questions about an intervention.

Hospital. A computerized ADE surveillance system was used to help identify and prevent specific types of ADEs in patients in hospitals.682 The authors compared the length of stay in hospital of patients incurring an ADE with a historical control group of inpatients who did not have ADEs, and showed that the average length of stay for patients with severe ADEs was 20 days, 13 days for patients with moderate ADEs, and five days for those with no ADEs. This translated into a cost of USD$38,007 for patients with severe ADEs compared with USD$22,474 for patients with moderate and USD$6,320 for patients with no ADEs. Given this significant difference in the length and cost of hospitalization between patients with severe and moderate ADEs, the authors concluded that this suggests that the prevention and reduction of ADEs could reduce the length and cost of hospitalization for certain patients. However, it is important to acknowledge that the cases were not matched for disease severity and that no direct cost analysis was made of the ADEs prevented by the system compared with before the implementation of the system.

The same author measured the effect of a CDSS aimed at improving the use of and reducing the cost of antibiotics in four separate studies. The first was conducted in an academic, tertiary, private hospital and the average cost for 24 hours of antibiotic therapy recommended by the CDSS was USD$10.85 less per patient than what was actually prescribed by physicians.409 The same CDSS was evaluated in two studies that took place in a 12-bed shock/trauma/respiratory ICU. The 7-month pilot study revealed a mean reduction in the cost of antibiotics of USD$87.03 per patient compared with the preintervention period.683 The other ICU study was 12 months in duration and the mean cost of antibiotics for the computer regimen followed, regimen overridden, and no CDSS, respectively was USD$102 compared with USD$340 and $427, while the cost of hospitalization was USD$26,315 compared with USD$35,283 and USD$44,865.475 Finally, an antibiotic-dose monitor was incorporated into the CDSS to check the renal function of patients to identify those who were potentially receiving excessive dosages of antibiotics.614 The patients in the intervention group received fewer mean doses of study antibiotics at a lower average cost (USD$80.62) than patients during the preintervention period (USD$92.96) of this 12-month study. If this reduction of USD$12.34 per patient is summed for all 4,483 patients in the intervention period, this would result in a total decrease in cost of more than USD$55,000 a year.

Another CDSS by Barrenfanger and colleagues,684 designed to improve antibiotic prescribing by electronically notifying the pharmacist of potential problems with a patient’s antimicrobial therapy, was introduced in a 450-bed community teaching hospital and evaluated over a 5-month time period. The study compared patients whose microbiologic data were processed in the normal manual manner in the pharmacy to patients whose microbiological data were processed using the computer software. The study patients were matched by diagnosis related groups to patients in the control group. Additionally, the control group patients were adjusted for severity to make the groups more comparable. The study group had an average total standard cost of USD$13,294 per patient; the severity adjusted control group had an average total standard cost of USD$16,106 per patient, a decrease of USD$2,812 per patient in the study group. By using these severity adjusted data, the estimated variable cost savings annually from the improvement of interventions is USD$2,932,000 (2,000 inpatients for whom susceptibility testing is done multiplied by $1,466). If the list price of the CDSS (USD$44,500) was subtracted from the expected annual cost savings from the use of the program to improve interventions (USD$2,932,000), the resulting savings (USD$2,887,500) was still substantial in the first year.

A 3-month RCT was designed to evaluate the effect of a CDSS for the management of antimicrobial utilization in a 648-bed tertiary care academic hospital.401 Antimicrobial utilization was managed by an existing antimicrobial management team using the system in the intervention arm and without the system in the control arm. The Web -based system was developed to alert the AMT of potentially inadequate antimicrobial therapy (a “back-end”’ or postprescription review). Expenditures for antimicrobial drugs were USD$285,812 for the intervention group and USD$370,006 in the control arm, for a savings of USD$84,194 (23 percent) overall or $37.64 per patient.

An antiinfective decision support tool, designed specifically for a pediatric population, was introduced in a 26-bed ICU in an academic hospital.469 During the 6-month period before CDSS installation, all patient care orders from the physicians were handwritten. The study found no difference in hospital costs in the period before CDSS installation (USD$28,257.67) compared with the time after CDSS installation (USD$25,032.11) or in antiinfective costs per patient (USD$274.79 in the control group compared with USD$289.60 in the intervention group).

An evaluation of a CDSS on appropriate antibiotic treatment used a cohort study followed by a multicentre, cluster RCT.399 The cohort study compared the advice of the CDSS with physician performance with respect to appropriate empirical antibiotic treatment and costs. The RCT compared hospital wards using the CDSS compared with antibiotic monitoring without the CDSS. In the cohort study, all cost components, except those related to expected adverse events, were significantly lower for the treatments suggested by the CDSS compared with those used by physicians. Total antibiotic costs were €289 lower per patient for CDSS, a relative decrease of 48 percent. In the RCT, the use of the CDSS resulted in significantly lower antibiotic costs in intervention versus control wards, the difference originating from lower ecological costs in intervention wards in Israel and Italy. Direct antibiotic costs, as well as costs incurred by observed adverse events, were similar.

A Canadian study in an orthopedic institution assessed the safety and potential cost savings of a computerized, laboratory-based program (i.e., CPOE and CDSS) to manage inpatient warfarin therapy after major joint arthroplasty.685 The authors estimated that the potential savings per patient of CAD$5.50 per day was due to a reduction in nursing time, for a total annual figure of CAD$55,836. It is important to note that the cost estimates and potential cost savings are speculative and are meant to be illustrative and not conclusive in nature.

A computerized order set within an CPOE was designed to manage pediatric inpatients with asthma.446 A before-after study of the system found no significant difference in the total inpatient costs among the groups before and after intervention. The hospital charges were USD$3,567 and USD$3,759, while the pharmacy charges were USD$373 and USD$429 in the groups before and after intervention, respectively.

The costs associated with the implementation of a CPOE and CDSS system over 10 years (1993 to 2002) were measured in a 720-adult bed, tertiary care academic hospital.686 Using data on the reductions in items such as ADEs, drug costs, and laboratory test usage, it was estimated that the system saved the hospital USD$28.5 million over the 10-year period, even after including the capital and operational costs of USD$11.8 million. The authors stated that it took over 5 years to realize a net benefit and over 7 years to realize an operating budget benefit.

Chertow et al.468 studied the effect of adding a CDSS to an existing CPOE for prescribing drugs to patients with renal insufficiency in a hospital setting. The authors measured the difference between the intervention and control groups in hospital and pharmacy costs and found no differences between the groups (USD$4,881 compared with USD$4,968 in total costs for the intervention and the control groups, respectively).

An evaluation of the introduction of a CPOE and eMAR system on the delivery of health care in an academic health system was done using a before-after design.581 Based on total costs per admission, no significant difference was seen in any of the U.S. hospitals in the system.

A cost analysis of the implementation of an expensive CPOE (i.e., total capital cost of implementation was USD$2.9 million and operating costs were USD$2.3 million) in the management of surgical patients in an academic, multispecialty hospital was done by Stone et al.419 Based on the data from 6 months before and 6 months after the intervention, a redistribution of workload was found. The personnel changes resulted in a savings of USD$445,500. The authors also noted that because of considerable gains in efficiencies (e.g., time necessary to have orders accessible to nursing, radiology, and laboratory), this implementation would likely result in long-term cost savings and improved quality of care.

An RCT done in 1993 assessed the effects of a network of microcomputer workstations for writing all inpatient orders (i.e., CPOE) on health care resource utilization.687 The overall aim of the CPOE was to encourage cost-effective ordering and to reduce costs. Using the costs associated with inpatient charges (i.e., bed, tests, and drugs), it was determined that total charges per admission were significantly less (USD$887) for the intervention teams than for the control teams, with similar differences in all types of charges. The authors claim that if these effects were extrapolated to all medicine service admissions at that hospital, the projected savings in charges per year would be $3 million in 1993 U.S. dollars. It was noted that the workstation network hardware costs were approximately USD$20,000 per ward, with additional costs for installation and maintenance.

In two separate RCTs, Tierney et al. evaluated the effect of a CDSS that provided guidelines for the treatment of patients with ischemic heart disease or chronic heart failure519 and patients with asthma or chronic obstructive pulmonary disease.518 In both studies, care recommendations were displayed electronically to either physicians, pharmacists, or both physicians and pharmacists, compared with no care recommendations. In the heart disease study, the patients in the group receiving only the physician intervention had significantly elevated total health care charges (physician only: USD$6,302, compared with pharmacist only: USD$7,387, compared with physician and pharmacist: USD$7,639, compared with control: USD$7,025). In the asthma and chronic obstructive lung disease study, the authors found no difference in total costs (i.e., total inpatient and outpatient charges) across groups (physician only: USD$8,006, compared with pharmacist only: USD$5,333, compared with physician and pharmacist: USD$5,652, compared with control: USD$5,800).

A recent publication by Pointek and colleagues688 measured the impact of an ADE alert system on cost and quality outcomes in seven community hospitals within a health network. The ADE alerts were triggered in real time, which enabled immediate pharmacy intervention. The results showed a statistically significant decrease in average pharmacy department costs per patient (USD$867 versus USD$826, p < 0.001) from before to after implementation. In contrast, the external control group had a significant increase in pharmacy department costs (USD$734 versus USD$797, p = 0.029). Drug costs decreased significantly from baseline (USD$360 versus USD$337, p < 0.001) in the study group. Conversely, there were significant increases in drug costs in the external control group (USD$401 versus USD$429, p = 0.029). The authors applied the observed percentage of cost decrease from baseline exhibited by the study group to both the internal and external control groups’ results and found that this yielded a combined pharmacy department cost savings estimate in excess of USD$11 million. It was noted that these savings coincided with only modest quality improvements in projected mortality rates and length of stay. An important limitation in this study is that it did not compare ADE rates before and after implementation of the system.

Community. McMullin and colleagues689,690 published two papers that evaluated the impact of a CDSS on prescription costs on a range of medications used in primary care. The first study was a retrospective cohort study using pharmacy claims data, which found that the average cost per new and refilled prescriptions was USD$4.99 lower in the intervention group, with the 6-month savings being USD$3,450 per clinician. A 6-month extension of this study showed a 12-month savings on new prescriptions of USD$109,897.

A cluster, unblinded, pragmatic (i.e., real world) RCT was conducted in a routine clinical setting, to assess the cost and effectiveness of a CDSS based on recommendations of the European Society of Cardiology and other societies for hypercholesterolemia management in comparison with usual care for patients with hypercholesterolaemia.528 The total direct costs of hypercholesterolaemia management (i.e., physician visits, laboratory analyses, and the lipid-lowering drugs prescribed) for the intervention and control groups were calculated. The impact on total costs was markedly different in the two groups: €264,658 in the usual care group and €170,061 in the intervention group.

Ornstein et al.691 set out to measure the impact of displaying prescription cost information in a computer-based patient record system at the time of prescribing on reducing drug costs by family physicians. When compared with a 6-month period where cost information was not displayed, it was concluded that no impact was found on overall drug costs to patients that could be related to the intervention. The mean cost per prescription in the control period was USD$21.83, and in the intervention period was USD$22.03.

Weingart et al.692 designed an empirical study to understand the potential benefits of medication safety alerts generated by an e-Prescribing system in ambulatory care. Using a modified Delphi technique and data on 1.8 million prescriptions, the authors estimated that e-Prescribing alerts possibly averted 133 to 846 ADEs. These alerts could have avoided health care resource utilization in a number of areas (e.g., hospitalizations, emergency room visits), for a total savings to the system of USD$141,012 to USD$1,012,386. An expert panel reviewed a sample of common drug interaction alerts, estimating the likelihood and severity of ADEs associated with each alert, the likely injury to the patient and the health care resource utilization required to address each ADE. The analysis estimated that the cost savings due to the e-Prescribing by using third-party-payer and publically available information was USD$402,619 (inter quartile range [IQR] $141,012-$1,012,386) with an average cost savings per clinician of USD$173 (IQR $61-$436).

Community and hospital. One group of researchers developed a CDSS that used the clinical information contained in administration claims data from physicians, hospitals, pharmacies, and laboratories to identify common errors in care and departures from widely accepted clinical guidelines.620 This differs from the other CDSSs discussed in this section, in that the CDSS was not deployed within a hospital setting or within an integrated delivery system in which EHR systems provided the backbone of clinical information. The authors conducted a 12-month RCT to test the hypothesis that the claims-driven CDSS could increase compliance with evidence-based practices and effect improvements in patient outcomes as measured by decreased hospitalization and attendant cost. The sentinel system was designed as a rule-based artificial intelligence engine combined with an automatic message generator that conveys clinical recommendations and supporting literature to treating physicians. Nine hundred and eight clinical recommendations were issued to the intervention group. Among those in both groups who triggered recommendations, there were 19 percent fewer hospital admissions in the intervention group compared with the control group (p < 0.001). Charges among those whose recommendations were communicated were USD$77.91 per member per month lower and paid claims were USD$68.08 per member per month lower than among controls compared with the baseline values (p = 0.003 for both). According to the paper, the intervention cost USD$1.00 per member per month to deploy and was associated with lower paid claims of USD$8.07 per member per month in the intervention group compared with controls, suggesting an eightfold return on investment from the payer perspective. However, it is important to note that this study was not intended as a formal cost-effectiveness analysis or cost savings analysis in that they did not directly measure costs at the patient or caregiver level, nor did they consider noneconomic costs or benefits.

An extension of this analysis was published 3 years later.504 This study used data from two additional years to analyze the effect of the intervention on resource utilization. This evaluation showed that the intervention reduced the average total charges (i.e., billing, pharmacy, and laboratory data) in the study group by USD$24.77 per member per month compared with the group without the CDSS.

Economics Summary

Most of the studies (84 percent) reviewed that evaluated the economics of MMIT would not be considered full economic evaluations. Full economic evaluation studies measure the cost per successful patient outcome over time, whereas cost analyses measure only the costs of the alternatives examined. Cost analyses can provide useful information on ‘upfront’ costs compared with ‘downstream’ cost avoidance but an ideal economic evaluation would explicitly measure all direct health care costs (e.g., capital costs, health professional’s time) and direct nonhealth care costs (e.g., home care services, transportation), as well as indirect costs (e.g., productivity gains or losses related to illness or death by the patients and caregivers) that could be affected by the intervention of interest. It is important to be aware that the greatest costs of these health ITs are associated with the purchase of new software (capital outlay) to add to preexisting EMR systems, as well as implementation costs (e.g., management, clinical team involvement, training costs, maintenance costs), which were not included in the cost side of the economic evaluation in most studies. Additionally, the full enumeration of the total costs needs to be synthesized with the consequences or outcomes of the intervention (i.e., cost-effectiveness analysis, cost-utility analysis, and cost-benefit analysis). The effectiveness of any given system is dependent on the system’s design, implementation, the users of the system, and the setting into which the system is being introduced. Adoption of newer technologies needs to be based on formal evaluation of whether the additional health benefit (effectiveness) is worth the additional cost. Given the tension between the clinical benefits of integrated CPOE and CDSS systems and the high upfront costs, decisionmakers deciding whether to implement them need to better understand how and when financial benefits of such systems accrue (e.g., short-term compared with long-term benefits). These types of analyses are important for well-informed decisionmaking.

In summary, a few of the studies reviewed found that health IT interventions may offer cost advantages despite their increased acquisition costs compared with care provided without the health IT. However, given the uncertainty that surrounds the cost and outcomes data, and limited study designs available in the literature, it is difficult to reach any definitive conclusion as to whether the additional costs and benefits represent value for money. It is necessary that sophisticated concurrent prospective economic evaluations be conducted in the real world to address whether health IT interventions in the medication management process are actually cost-effective.

Clinical Outcomes

Summary of the Findings

Among the clinical outcomes assessed in 76 articles (Table 14), 54 percent reported significant benefits (Table 15). Studies that used monitoring approaches to identify and intervene with patients with actual problems (e.g., excessive blood pressure, increase in creatinine after being placed on a nephrotoxic drug) or needed care (e.g., hemoglobin A1c monitoring) appear to be more effective than CDSS approaches that identify theoretical problems (potential for adverse drug events). The effectiveness of monitoring interventions in ambulatory care is enhanced (or only effective) if patients are also sent reminders and decision support recommendations.

Table 14. Research design for studies across the phases of medication management and education and reconciliation that address clinical outcomes as their main outcomes.

Table 14

Research design for studies across the phases of medication management and education and reconciliation that address clinical outcomes as their main outcomes.

Table 15. Summary of the number of studies reporting statistically significant differences in clinical primary endpoints between study groups for hospital and ambulatory based studies.

Table 15

Summary of the number of studies reporting statistically significant differences in clinical primary endpoints between study groups for hospital and ambulatory based studies.

Highly targeted interventions, focused on specific problems that provide problem-related specific interventions appear to be more effective than a more diffusely focused CDSS integrated with a CPOE system (e.g., nonpatient-specific guidelines for cardiovascular risk reduction).

Many studies have evaluated CDSS tools for improving the effectiveness of anticoagulants (proportion of days in therapeutic anticoagulant range) and improving the choice, route, duration of antibiotics, and reducing ADEs related to antibiotic use and most are successful.

Studies that have been successful in improving patient outcomes target high risk and vulnerable populations who have poor disease control,515,610,624,693 lack sufficient access to health care providers to manage their condition,408 or subpopulations with sufficient economic resources to respond to the CDSS intervention.694

While high risk groups have the potential to show the greatest benefits of IT, one study, which implemented a CPOE (prescribing, dispensing, and order communication system) in a children’s hospital, reported substantial harm—a 270 percent relative increase in mortality after CPOE was implemented (2.8 percent vs. 6.6 percent unadjusted, adjusted OR 3.81, 95 percent CI 1.94 to 5.55).15 This before-after study and its methods have been debated17 and its conclusions contested. However, the increase in mortality they found provides important lessons about CPOE implementation, particularly in settings which include high-risk patients. Critically ill patients are most likely to benefit from IT but also most likely to be affected by dysfunctional technology and implementation strategies because delays in definitive treatment can increase the risk of mortality. As other groups have shown that CPOE systems either have no effect or a nonsignificant reduction in mortality in children’s hospitals,16 the disparity in findings likely relates to the extent to which both the technologies and implementation strategies have disrupted or delayed critical activities in the clinical setting, and demanded additional time for order-entry from clinical staff.

Two studies that implemented computerized decision support CDSS drug use increased mortality,15 and length of stay.18 Both studies lacked sufficient power to conduct a valid assessment.

Strengths and Limitations of the Evidence

Overall, 28 of 76 (37 percent) studies assessing clinical endpoints were RCTs, and the mean quality rating was 4.4 out of 9 (range two to seven).401–403,407,408,515,518–520,524,526–528,537,541,543,610,620,624,630,634,637,695–700 Low ratings are because most RCTs of health IT cannot be blinded, and the majority are cluster RCTs, where equivalence in the distribution of measured and unmeasured confounders (clinician and patient characteristics) cannot be assured. Statistical adjustment for differences in the intervention and control groups has not been conventionally advocated even though it is likely required for unbiased comparisons.

The remaining studies were cohort, case control or observational; the majority were before-after studies or variants of this approach. Typically, in the before-after variant design, three time periods were assessed. Preintervention outcomes were compared with outcomes evaluated at two time periods of after implementation intervention. These comparisons sought to assess changes in care and the care processes associated with the interventions that were subsequently introduced. Only one study was a true time-series.482 In most of the before-after studies, no adjustment was done for differences in patient mix or cointerventions in the time periods with and without the intervention. Unless a systematic trend for changes in the patient population mix was shown, this problem may have minimal effect on the reported results. The only exception is with length of stay, where well-documented trends in reductions in length of stay due to many factors unrelated to IT interventions are shown. For these outcomes, the positive benefits in reductions of length of stay shown in nine of 15 studies that measured this outcome are likely overestimated.

While the absence of a contemporaneous comparable control group is a problem with all before-after studies, the creation of control groups by comparing intervention patients to those that do not participate, or do not have a problem, to those that do is fundamentally far more likely to introduce major bias in the comparison (e.g., comparing patients with alerts to no alerts,18 pharmacists volunteering to provide the intervention compared with those that do not volunteer,694 and other similar problems642,701). The direction of the bias will depend on the study. Volunteers in any study tend to have better outcomes than nonvolunteers, and selecting patients with problems compared with those that do not will ensure that at least both will regress to the mean—people with problems get better and those with no problems get worse, resulting in an overestimation of the effect of most interventions.

Many of the observational studies suffered from selecting an outcome that was distantly or only marginally related to the intervention. Almost all of the studies that measured quality-of-life, length of stay, and all cause ADEs were examples of this problem. Gurwitz and colleagues697 were able to show that only one-third of ADEs could have been prevented by the CDSS alerts that were provided. Moreover, in a substantial proportion of negative studies, minimal adoption was evident. The clinicians failed to adjust therapy or treatment to match the recommendations, and thus it was not surprising to find that the interventions had no effect on outcomes. Finally, the rate of some outcomes such as readmission, mortality, and nosocomial infections were too low to detect clinically meaningful differences if they had existed.

General Study Characteristics

A total of 76 studies assessed improvements in clinical endpoints or reduction in adverse events (Appendix C, Evidence Table 9).15,16,18,401–403,407,408,423,425,430,437,446,452,453,455,459,467,468,475,482,489,501,515,518–520,524,526–528,537,541,543,545,550,555,581,610,614,615,620,624,630,631,634,637,641,642,682–685,688,693–714 Prescribing and monitoring are the phases that were well-studied with respect to clinical outcomes (95 percent of all studies). Forty included the monitoring phase, only two evaluated clinical outcomes associated with order communication,15,581 three studied drug administering581,630,693 and one each looked at dispensing,15 reconciliation,695 and a cell phone-based diabetes management program for educational purposes.537 A total of 85 different endpoints were assessed for different aspects of MMIT (Table 14).


Prescribing. Clinical outcomes have always been the most important and the most difficult to measure and study in health IT applications. The studies must be done in clinical settings which are complex and nonstandard. The health IT must be held constant to fit the traditional model of clinical trials and this does not reflect the reality of clinical practice and technology use. It is also difficult to ascertain if a technology can affect clinical outcomes—drugs, surgeries, and other similar interventions are easier to tie to outcomes. The health IT is a “long distance” from actual clinical care with many steps and factors involved and the latency between exposure to the health IT and outcome.

Many studies have been done on the effect of health IT on prescribing. Consequently, many systematic reviews have addressed the effects of these applications on clinical outcomes. Two Cochrane reviews have been done. One addresses onscreen point-of-care computer reminders on outcomes of clinical importance. The review by Shojania and colleagues715 found some clinical improvements across studies with blood pressure (being reduced by a mean of 1.0 mmHg). Durieux and colleagues716 showed small improvements in time to therapeutic stabilization, risk of toxic drug levels and length of hospital stay (mean decrease of 0.4 days). Another eight reviews show similar findings for clinical outcomes: more changes in process and some limited and rather small improvements in clinical outcomes: alerts and prompts to improve prescribing behaviors (five studies, three showed statistically significant improvements),717 CDSSs to improve prescribing in older adults (two studies, mixed outcomes),718 CDSSs on medication safety (five CPOE and seven CDSS of which three did not show improvements),719 e-Prescribing in hospitals (23 of 25 studies showed medication error reduction and four of seven with reduced ADEs of 35 percent to 98 percent),720 CPOE in pediatric and ICUs (12 studies of proven error reduction but no effect on clinical outcomes),721 CPOE in neonatal ICUs (reductions in errors but little if any effect on clinical outcomes),722 outpatient CPOE (five studies of medication safety of which one showed reductions),723 and CPOE with CDSS to reduce ADEs in hospitals and ambulatory settings (10 studies: five showed improvements, four showed trends and one was not significant).724

Prescribing—Strengths and Limitations of the Evidence

The evidence in this section is weak although many RCTs exist (Table 14). Numbers of participants in the trials are often small, studies are short term, and are often done by those who have developed and implemented systems. To support the potential for bias in assessment of a health IT by its developers, Garg and colleagues725 completed a large and well-done systematic review of CDSSs. Their review evaluated RCTs of CDSSs for improving practitioner performance and patient outcomes. Using the data on practitioner performance, they found that if the trialists evaluated their own CDSS the trials were successful in 51 of 69 studies (74 percent). If the trialists were independent of the system being evaluated (i.e., not the developers), only five of 18 trials were positive (28 percent). The odds ratio (OR) adjusted for trial quality for a successful trial designed to improve practitioner performances if the evaluator was the developer was 6.6 (95 percent CI 1.7 to 26.7). The only other predictor of success besides the evaluator being the developer, in improving provider performance was if the users of the CDSSs were prompted to use the system automatically (adjusted OR 2.8, CI 1.2 to 7.1). It is difficult, however, to separate out developer bias from system effectiveness as they are confounded. Commercial systems often do not have the resources to show changes in clinical outcomes and therefore this proof of clinical effectiveness is not completed.

Twenty-one RCTs studied the prescribing phase.401–403,407,408,515,518–520,524,526–528,537,541,543,630,637,697–699 In addition, seven cohort or case-control studies,501,545,701,702,709,710,712 and 27 observational studies16,18,423,425,430,437,446,452,453,455,459,467,468,475,482,489,550,555,631,683,688,693,703,704,706,708,714 also looked at the prescribing phase and reported clinical outcomes. Only the RCTs will be discussed below because of their strength of evidence. Four studies were done in the late 1990s.407,527,630,699 All of the rest were done after 2000.

Prescribing—General Study Characteristics

Participants. Because these studies evaluated clinical outcomes, all assessed patients and their caregivers. All RCTs used cluster randomization (clinicians or care units) to avoid problems of contamination (where the same caregiver is asked to use decision-support for a random half of patients but not the remainder).

Location. One study was done in a long-term care center,697 one was set in homes,630 and five were set in hospitals.401–403,407,637 All of the others were done in ambulatory care settings.

Drugs and diseases. Most studies evaluated specific diseases or conditions: asthma,519 high cholesterol levels,515,528,699 hospitalized patients at risk of deep venous thrombosis or pulmonary embolism,519 depression,520 infections in hospitalized patients,403 high blood pressure,526,699 and HIV.527

Technology. All studies involved CDSS. Six also included CPOE.402,407,519,527,697,699 A PDA was also featured in the home-based article.630

Prescribing—Clinical Outcomes

Please also see the section on CDSS (KQ7: RCTs of CDSS) for additional description of clinical outcomes. As seen in the systematic reviews, fewer articles address clinical outcomes than address process or other outcomes such as satisfaction and attitudes. Many of the studies that did evaluate clinical outcomes also did not find the expected improvements.

Adverse drug events. Gurwitz and colleagues697 found that the rate of ADEs and preventable ADEs were not decreased with implementation of a CDSS and CPOE system in a long-term care setting.

Disease related outcomes. A number of studies looked at disease outcomes. Kucher and colleagues402 found that fewer patients at risk for venous thromboembolism were diagnosed with either deep venous thrombosis and pulmonary embolism at 90 days with the introduction of CDSS and CPOE in an academic hospital. Zanetti and colleagues403 studied prophylactic antibiotics in prolonged cardiac surgery. This RCT found similar rates of infection in both study groups. Of note, both control and intervention groups reduced their rates of infection. Rollman and colleagues520 addressed identification of depression in adult ambulatory care and found that the CDSS did not affect the rates of depression in the control or intervention groups in an RCT. Both groups improved their depression scores over time to the same extent.

Hospital stay. Three RCTs on prescribing looked at hospital length of stays. One RCT did not find differences in quality-of-life scores, hospitalizations, emergency department visits, or heart failure exacerbations.519 Safran and colleagues,527 in another RCT of CPOE and CDSS for clinic patients in an academic setting, did find a lower hospitalization rate for intervention (reminders) group (44 percent vs. 35 percent, RRR 26 percent, p = 0.04). Hospital length of stay was not different in the RCT by Overhage and colleagues407 of CDSS and CPOE (eight days for both groups), nor in the study by McGregor and colleagues.401

Physiological measures. Eleven RCT studies of the prescribing phase addressed physiological outcomes: hypertension in two articles and both showed no difference,526,699 high cholesterol with some positive findings in one541 but not the other four515,524,528,543 and two with reductions in blood glucose levels.537,630 A study on depression found no change in patient depression rating scores.520 One study with asthma patients found improved lung function and airway hyperresponsiveness.408

Order communication. Order communication issues seemed to be at the heart of this before-after study of a children’s hospital by Han and colleagues15 which showed increases in mortality after introduction of CPOE and CDSS integrated within a hospital information system. This is an important study and has garnered much discussion in the literature of its methods and findings with respect to the increase in mortality (2.8 percent vs. 6.6 percent unadjusted, adjusted OR 3.81, 95 percent CI 1.94 to 5.55). Length of stay showed improvement in one hospital, but not another in a study of CPOE implementation by Mekjjan and colleagues.581

Dispensing and administering. The study by Han and colleagues15 evaluated dispensing; while three studies addressed administering.581,630,693 One study was an integrated system in the Ohio State University Health System (James Cancer Center and three other tertiary care hospitals). The hospital information system included laboratory, imaging, dietary, eMAR, and CPOE as well as all EMR capabilities. They found a reduced length of stay for patients with heart disease (14 percent) and transplant patients (15 percent) but not for those with cardiothoracic surgery or those in the cancer center.581 Holdsworth et al.693 found significant reductions in ADEs following the implementation of a CPOE system in a pediatric population. The third administration study was a cross-over RCT of diabetic patients using a hand-held insulin regimen optimizer, which showed improvements in blood glucose levels when patients received advice through the device.630

Monitoring. Most of the prescribing interventions were integrated with hospital clinical information systems or EMR systems. This provided the opportunity to use existing structured electronic information to assist clinicians in identifying patients who needed a change in their treatment plan. The system made recommendations that suited the particular patient profile. Starting with monitoring of treatment choices for antimicrobial therapy in relation to antibiotic choice, a wide range of clinically useful monitoring and prescription and treatment recommendation options have been studied including those aimed at improving chronic disease management (e.g., Asthma-Critic), providing early detection of adverse events (e.g., creatinine monitoring for nephrotoxic effects), and glycemic and coagulation monitoring to predict and recommend optimal dose changes. Of the 21 RCTs that included the monitoring phases, 15 were set in the prescribing phase. The issues related to RCTs and observational studies have been addressed in the general overview of studies in this area.

Reconciliation. One RCT at two academic hospitals studied a computerized reconciliation program that was integrated into a CPOE system and that required a process redesign.695 They found a reduction in unintentional discrepancies between preadmission medication and admission or discharge medication that had potential for harm (1.44 vs. 1.05 potential ADEs per patient, absolute risk reduction 0.72, 95 percent CI 0.52 to 0.99).

Education. In another RCT, Grant and colleagues610 studied a PHR system for patients with diabetes that was integrated into a fully functioning EMR (laboratory, imaging, CDSS, and pharmacy). Patient education was a major, but not the only, component of the PHR. No change was noted for hemoglobin A1c levels, although it is important to note that the patients were fairly well-controlled at baseline (7.1 percent vs. 7.2 percent, p = 0.45).

Qualitative Studies

Summary of Findings

Fifty-three articles that were complete or partially qualitative studies were identified that dealt broadly with MMIT (Appendix C, Evidence Table 10).20,439,503,540,547,597,629,632,633,635,652,666,671,674,726–764 No qualitative studies were identified that directly addressed the effect of an MMIT system on intermediate health care outcomes for any phase of the medication management continuum (prescribing, order communication, dispensing, administering and monitoring, as well as reconciliation, education, and adherence).

Strengths and Limitations of Evidence

The primary limitation of synthesizing qualitative studies to gain a deeper insight into the effect of MMIT applications in improving other intermediate health care outcomes within and across the medication management continuum is that no qualitative studies are available that directly address this question. Most of the qualitative studies identified examine the expectations or experiences of implementing an MMIT system on the process (but not the outcomes) of medication prescribing. These studies identify a large number of benefits to the health care delivery processes as well as a large number of barriers to uptake and use of the various systems studied. The strengths of the amalgam of evidence are that similar themes were identified across studies, health care settings were assessed by more than one study, studies were carried out in settings across the care continuum, study participants included physicians, pharmacists, nurses, other health care providers as well as some administrative management personnel, and multiple different types of qualitative data collection approaches including interviews, focus groups, observations, and document reviews were used across the set of studies evaluated. A small number of qualitative studies were available that examined MMIT systems on the processes of care for other phases of the medication management continuum.

MMIT is tremendously complicated and at the same time undeniably valuable. Strong and varied evaluations are vital and we have many evaluations of MMIT already. These evaluations show important changes to process. Clinical outcomes are more often mixed or nonexistent. We also see, in our evaluation studies, unintended consequences of MMIT and surprising results such as the increased mortality in a children’s hospital after a poor implementation of a set of MMIT applications.15 Because of these challenges in our results of quantitative studies, we include a fuller discussion of some of the qualitative studies in MMIT. These qualitative studies hold the promise of understanding more richly how MMIT is and should be used. The following paragraphs provide descriptions of some of the more important qualitative studies and their findings.

Prescribing and ordering. No qualitative studies were identified that directly addressed the effect of an MMIT system on intermediate health care outcomes. However, many qualitative studies provided evidence and examined positive and negative expectations and experiences of how an MMIT system designed to improve prescribing of medications could affect medication errors and medication safety,20,439,503,547,629,632,652,666,676,726,727,729–731,733–736,738–740,742,745,746,748,749,751–753,755,759,761–767 which could be considered as precursors to intermediate health care outcomes.

Before system implementation. Positive and negative expectations of an MMIT system designed to improve prescribing and ordering of medications were identified by physicians and other health care providers or staff in hospital or ambulatory clinic staff prior to system implementation. Some positive expectations were that an MMIT would reduce medication errors,727,733 increase pharmacological knowledge available,740 provide educational benefits,726,727 improve patient confidentiality,727 be flexible (e.g., prescribing from any location),666,729 allow for customization or tailoring to the individual prescriber or the patient (e.g., patient reminders), allow switching the system on and off,729,733 be concise,729,733 provide access to other areas of a medical chart,740 save time,740 and incorporate valuable allergy, dosing, and interaction alerts.745,746 Pharmacists felt that MMIT would facilitate new collaborations among physicians, pharmacists, and nurses.746

One group of physician study participants had a positive attitude towards implementation of a CDSS, provided that they had some control over the system.729 Many groups studied could be described as hopeful but cautious727,729,733,740,748 while others, mainly physicians (although they were who was studied most often), were skeptical.726,735,761,765 Hospital pharmacists felt that MMIT would allow them to spend more time with patients and improve collaborative working relationships with physicians and nurses.746

Only one study was identified that used qualitative methods to solicit patient views before implementation of an MMIT system focused on improving prescribing.748 Patients on a general surgery ward were interviewed before implementation of an e-Prescribing and an eMAR system. Their attitudes about the current paper-based system were generally positive and many had a mistrust of computer systems in general. However, they anticipated advantages of the e-Prescribing and eMAR system in terms of time, improving accuracy and efficiency, and decreasing mistakes. Patients identified that an electronic system may be an advantage for staff when the first language is not English.

Despite the willingness of many of the participants to use a new MMIT system designed to improve prescribing of medications including CPOE, some negative expectations were that the MMIT system would impair existing interactions and relationships among health care providers and between physicians and patients (e.g., diminishing patient contact because they need to leave the consulting room to enter the prescriptions),726,727,729,740,762 the ability to cope with the new system,726 implementation would be onerous,726 the costs of the system,727 especially to the health profession including the time efficiency and workload redistribution,735 the technical challenges such as data entry time, software compatibility and updating,727,729 problems with availability or level of technical support,727,735 social and cultural barriers,727 deskilling of staff (people becoming dependent on the system for routine decisionmaking without understanding the background, reasons, or consequences of the decision made by the MMIT),726 need for more security,726 errors in prescribing such as decisionmaking errors,727 transcription errors,727 or overconfidence errors,727 that the system would not remove medication errors but could even create new errors,735 obscured responsibilities,729 loss of own reasoning and clinical autonomy,729 ensuring that the patient and not the computer would have the leading role in the encounter,729 difficulty with knowledge management (e.g., too much information or erroneous information—’garbage in–garbage out’),729,740 including prescribing alerts that were redundant or repetitive, of low priority, or difficult to interpret,739 resistance towards change,729 computer shortages,740 and altering workflow routines.740

Some underlying key needs for an MMIT system designed to improve prescribing of medications would be that the system would not diminish the patient provider relationship,726,727,729,740 be easy to use,733 flexible, concise, and customizable,729,733 clinically and technically trustworthy, reliable, and fast,729,733 integrated into other relevant systems,740 workflow needs to be maximized including development of new workflows726,727,739,735,740 and there be enough time and resources available to support implementation.726,727,735,740

A Delphi survey was done in the United Kingdom to identify and reach consensus on the key clinical issues involving patient safety for which general practitioners in primary care might benefit from MMIT support, particularly in relation to medicines management. The key themes that emerged were importance of computerized alerts, need to minimize spurious alerts making it difficult to override critically important alerts, having audit trails of such overrides, support for safe repeat prescribing, effective computer–user interface, importance of call and recall, and need for safety reports. User interface, repeat prescribing, need to be able to run safety reports, and other safety issues were also agreed upon.738

After system implementation. The reporting of how MMIT systems designed to improve prescribing of medications improved intermediate health outcomes were also sparse among the qualitative studies. Drug alerts, including drug interaction alerts, were stated to be beneficial to improve patient safety.547,632,666,767 E-Prescribing triggered a variety of clinician behaviors (other than terminating or changing a prescription) that may improve patient safety.632 One study identified 22 previously unexplored medication error sources users reported to be facilitated by CPOE which would likely have a detrimental effect on health outcomes.752 These were grouped as (1) information errors generated by fragmentation of data and failure to integrate the hospital’s several computer and information systems, and (2) human-machine interface flaws reflecting machine rules that do not correspond to work organization or usual behaviors such as selecting the wrong patient because a list is alphabetical versus by team or floor, or unclear log on and log off procedures or processes so that the next person does work using the previous person’s permissions.

During or after system implementation, physicians, nurses, and other health care providers or staff in hospital or ambulatory clinic found that health IT improved safety alerts,666 provided useful drug alerts including drug interaction alerts, which appeared even if a different prescriber had ordered some of the drugs,547 allowed physicians to prescribe electronically from everywhere in the hospital, improved on features of a previous paper-based system,666 were user friendly and allowed benefits when the system was integrated,751 and was designed to take into account diverse cultures.751

Physicians in one study felt that an electronic CPOE system improved the quality of care for patients because they got faster access to information and more up-to-date information, they received automatic reminders, and they could speed up care because knowing what had already been done would allow them to reduce the number of duplicate procedures carried out.761 The multiple checks within the system also could lead to improved patient safety.761 Better communication among physicians and structured reports for patients (e.g., discharge summaries) were also felt to improve quality of care.761

In another study, physicians, physician assistants, and nurse practitioners felt that a computerized patient record system-based pain CDSS played a very positive role in assisting them with patient care. They reported that this was because data were more legible, could be accessed remotely, and reminders provided helpful decision support. However, they also reported that at the same time as being helpful, the reminder system was considered time consuming, redundant, and the speed of the system slow.763 Medical trainees also reported that an MMIT system provided valuable educational content such as geriatrics pharmacology review and nonpharmacologic treatment options.731

MMIT systems designed to improve prescribing of medications also generated some challenges that could be categorized as challenges: (1) with the computer system and software, (2) of the interaction between the MMIT system and the health care provider working with the system, (3) of the effects of the system on the collaborative working relationships of the health care team, and (4) of the MMIT working within the local context and environment. Computer system and software challenges include difficulties with user rights, inflexibilities, and displacements with the use of CPOE,734,764 CPOE design failures, especially a faulty computer interface, lack of connection with other parallel systems, inadequacy of decision-support, and human errors occurring in interactions with the computer,503,727,759,761,767 difficulties with the text presentation (e.g., too much information presented, data density), too many decisions that needed to be made at one time, unappealing color scheme and lack of notation, caution or problems with a prescription,652,761,767 interface problems,750,755 content problems,755 and increased data entry time.727

Challenges generated from the interaction of the MMIT system and the health care provider working within the system included the need to take up new tasks and increased demands on the clinicians with the CPOE system),734,764 maintaining complete lists of patients and their medications,632,736 poor recording of data within the record such as allergy information,632 propagation of errors if information was cut and pasted compared with creating new information and other mistakes,755,761 transcription errors,727 getting patient-specific formulary data,736 encouragement to ignore interactions alerts as many were viewed as too trivial or unnecessary, which indicated that sensitivity and specificity required improvements,547,632,753 initial difficulties with the technical components of the system,439,767 awkward prescription writing leading to workarounds,632 unfamiliarity with the disease codes in the system,751 difficulties with finding information in the chart because of multiple places where the information was stored,761 reducing clinical situation awareness,759 overconfidence,727 and increased workload.750

Challenges generated by the MMIT system that affected collaborative working relationships of the health care team included damaging the workflow, synchronization and feedback mechanisms between nurses and physicians,666,676,750 altering the pace, sequencing, and dynamics of clinical activities,750,759 and providing only partial support for the work activities of all types of clinical personnel.759

Challenges related to the MMIT working within the local context and environment included external implementation challenges (e.g., communication with pharmacists and vendor support),736 lack of computer resources,751 the need to keep their EHR systems up to date,751 poorly reflecting organizational policy and procedure,759,766 doctors’ concerns that their views and opinions about the design and implementation of the new system had not been adequately addressed,764 high cost,727 social and cultural barriers,727 and problems with technical support.727

A number of studies related that participants felt it took longer to prepare a prescription using the MMIT system compared with conventional pen and paper.761,764,767 One study identified that physicians and nurses in an acute care setting found that CPOE did not meet naive and early expectations.734 Some adverse effects of the CPOE system were noted.

Attitudes towards MMIT systems in the early stages were mixed.768 Over time, and with experience of making the system work for them, attitudes changed to become more balanced and the potential benefits of the system become clearer to most.439,676,761 Some physician participants felt the MMIT was more efficient during consultation and led to better quality, while others were felt it took longer and took away from patient focus.751 Physician users tended to provide comments related to the culture of professional quality (feeling that the computer facilitated quality). Alternately, those physicians that chose not to use the system tended to provide comments that focused on human relations. For example, they reported on their relationships with their patients that they felt were detrimentally affected by computer use.751 Some physicians felt that MMIT helped physicians become more cost conscious by suggesting therapies that were less costly. This cost savings however, only directly benefited insurers and not the clinicians, patients, or health care facility.751 Some physicians felt the MMIT systems improved their personal performance by allowing them to log on to the system from anywhere including home, while others felt this was an intrusion into their home life.761,767 Basic formatting and organization of information such as information that was legible, could improve order accuracy, or all in one place was seen as a benefit to MMIT.761 MMIT applications were also felt to improve interdisciplinary work by improving communication with colleagues,761 and having everyone reading from the same page.767

Alerts and reminders are important components of MMIT for prescribing. Important themes of these alerts or reminders in EMR systems included themes of efficiency, usefulness, information content, user interface, workflow, and training.742,745 Effectiveness focused on the positive effect of alerts on allergy awareness and patient education.745 Efficiency related to ensuring that the alerts and reminders were efficient, useful, and did not waste time.742,745 Usefulness concerned whether the alerts were helpful and appropriate.742 Information content was concerned with accurate, comprehensive, timely, rich, and accessible information.742,745 The user interface was felt to be important for smooth and efficient work and provision of valuable information that was accurate and provided quickly.742,745

The value of e-Prescribing alerts was diminished by the quantity of irrelevant and inappropriate alerts.632 Workflow issues related to the information being available when and only when needed.742 The need for training to improve the use of alert was noted.745 Attitudes to evidence-based guidelines were also seen as an important factor as to how alerts would be taken up, with physicians preferring that alerts be severity-rated, that only substantial ones should appear, and that user interface design be enhanced.745 The biggest surprise from a set of focus groups (reported in 2002) with a group of clinicians (physicians, physician assistants, and nurse practitioners) was the considerable negative emotion associated with alerts and reminders (feelings of being criticized, embarrassment, guilt, frustration, annoyance, and anger).742 One study of a set of three successful and three unsuccessful CPOE implementations across six hospitals identified 14 facilitating factors and 14 barriers comparing successful and unsuccessful implementation.20 More people from the successful hospitals group reported supportive administering and heads of medical sections, direct involvement of physicians, mandatory implementation, adequate training, and sufficient hardware facilitated success. In terms of barriers, only inadequate hardware and lack of ability to easily complete patient transfer and advance admission orders (medical records package) differentiated the successful compared with unsuccessful groups. Changes involved in instituting a physician CPOE system are system wide and involve individual as well as organizational factors.20 One study was identified that determined how clinicians use information management strategies during adaptation to an established CPOE system.749 User created strategies identified that information overload must be carefully managed and communication is vital and is often negatively affected by new systems.

Only one study using qualitative methods to solicit patient views after implementation of an MMIT system that focused on improving prescribing was identified.748 Patients on a general survey ward were interviewed after implementation of an e-Prescribing and administering system. Concerns were identified including loss of personal touch, not understanding the system, and perceived extra time needed if nursing staff had to check the drugs prescribed on the computer.748 Despite the concerns raised, on balance the feedback provided by patients was that generally they did not have a strong opinion (assessment) either positively or negatively as to whether MMIT would impact the quality of medication prescribing compared with paper-based process.748

MMIT also impacted the professionalization of pharmacy. The effects of a health IT system that generated an e-Prescription on the professionalization of community pharmacists were improving the analytical capacity of the pharmacists and physicians, greater dissemination of therapeutics and professional knowledge, better integration of process tasks, increased process automation, elimination of intermediaries, facilitation of the interpretation of prescriptions, increased tracking capability, and greater informational capability improves relevance and meaningfulness of interaction and improves quality of information transmitted.730 E-Prescribing has tremendous capacity to change and improve pharmacists’ professional work and interactions.730 One study showed that overly ambitious expectations sometimes lead to failed implementation.629

Order communication. Seven qualitative studies specifically addressed the implementation of an MMIT system to affect the order communication and verification of prescriptions.540,597,671,732,736,746,752 None of these studies focused specifically on how MMIT affected intermediate health care outcomes. All of the studies addressed implementation issues. Nursing perspectives based on implementation of a BCMA system within the hospital setting found that an MMIT system was more time consuming but the nurses acknowledged that it produced a positive benefit because the extra time available was wisely spent to assure verification, generating an increased sense of safety for the patients,671 or made improvements in the clarify of orders, organization of time their tasks, improved efficiency and standardization of documentation provided by templates, general improvement in emergency department processes, and decreased number of verbal orders and time searching for charts.597 One study identified 22 previously unexplored medication error sources that users reported to be facilitated by CPOE including errors related to order communication and verification such as information errors generated by fragmentation of data and failure to integrate the hospital’s several computer and information systems.752

These findings were consistent with another study carried out in a long-term care setting where numerous workarounds associated with the implementation of an eMAR and medication safety practices in nursing homes, were identified related to the technology itself creating unintentional blocks including slow wireless speed and the need to print each order on a separate page.732 Organizational processes such as the limited resource of fax machines were also identified.732 In the ambulatory setting limited electronic connectivity of e-Prescribing systems to pharmacies or pharmaceutical benefits managers (who administrate pharmacy prescriptions) meant that despite one-way electronic (non-fax) communication of prescription information from the practice there was still conventional communication (e.g., fax) back from pharmacies for clarifications and renewals.736 Pharmacist perspectives about a commercial e-Prescribing system revealed barriers to that systems’ ability to maintain complete lists of patients and their medications, use of CDSS, and getting patient-specific formulary data.736 Factors associated with these issues related to product limitations, external implementation challenges (e.g., communication with pharmacists and vendor support), and physician preferences on specific product features.736 A system that appended alerts and comments to the bottom of e-Prescriptions and was designed to reduce pharmacy callbacks did not reduce the number of callbacks but did change the nature of the callbacks.540 Hospital pharmacy leaders with and without CPOE entry system experience all believed CPOE would improve patient safety through the allergy, dosing, and interaction alerts which they saw as valuable to medication management processes.746 Some expressed concern that poor design or implementation could lead to increased errors.746 Most believed the system would lead to improved efficiencies facilitating more time spent with patients.746 Most felt CPOE would improve working relationships with physicians and nurses by facilitating new collaborations.746

Medication dispensing and administering. Ten qualitative studies focused on evaluating health IT applications to improve medication dispensing and administering including studies of BCMA,635,671,674,728,743,754,756 PDA,769 eMAR,732,754 CPOE,597 and automated medication dispensing.744 All of these studies focused on evaluation of the process of care delivery before or after implementation of the systems.

Before implementation of a bar-code point-of-care eMAR system a group of pediatric nurses working in an American pediatric hospital provided qualitative responses to questions as part of a survey.674 Themes derived from the survey done before implementation indicated that the nurses felt that medications would be given in a timely manner with less error, but may result in an increase in time with this increase in safety, along with more reported errors, but fewer errors in administering actual meds (near misses). The surveys collected after implementation indicated that the staff felt there were fewer medication errors with a smoother administering of medication.674 Implementation of MMIT applications for medication dispensing and administering generated substantial number of nonIT workarounds.728,732 In one study done in a hospital setting, these workarounds were categorized into omission of process steps (seven workarounds), steps performed out of sequence (one workaround), and unauthorized process steps (seven workarounds).728 Probable causes for these workarounds included technology, task, organizational, patient, and environmental related causes.728 A further study examined how nurses integrated BCMA and an eMAR system into everyday clinical practice and found that the implementation of new IT in the clinical setting can be disruptive to existing patterns of articulation work, or work that coordinates the activities of people across time and space.754

Another study of a system put in place in a long term care institution identified workarounds related to the technology itself and organizational processes.732 The workarounds occurred at new medication order entry, communication with the pharmacy, and administering.732 The technology introduced intentional blocks (safety features such as excessive dose blocking, dual documentation, and ADE monitoring) that led to workarounds related to the technology itself and organizational processes.732 Organization process blocks leading to workarounds included the double checking of preparation and administration documents.732 Integrating BCMA systems within real-world clinical workflows requires critical attention to ensure that technology safety features are used as intended and that nonIT systems are designed to support this use.728,732

Nursing perspectives about a BCMA, eMAR system integrated with pharmacy, CPOE, and electronic charting in a hospital after implementation found that in terms of access, the nurses appreciated greater access to medications and information (e.g., policies, guidelines, drug resources, patient files), but identified some delays in getting medications from the pharmacy.671 Another study carried out in a hospital and long-term care setting found that nurses were surprised that BCMA generated unanticipated side effects such as confusion created by automated removal of medications by BCMA, degraded coordination between nurses and physicians, and dropping activities such as not scanning wristbands or medications to reduce workload during busy periods.743 One study conducted interviews with nurses before and after the implementation of a BCMA. Before implementation most nurses expected the system to improve patient safety and after BCMA implementation most of the nurses reported that they felt BCMA improved safety although a number of concerns remained about the cumbersome and technical aspects of the system itself.756

After an automated medication dispensing system was installed interviews with all workers and managers who were affected (nurses, pharmacy managers, pharmacists, pharmacy technicians, hospital administrators, and patient care managers) resulted in themes of distrust, resistance, miscommunication, unrealistic expectations (skepticism that it reduced medication errors), speed and scale of implementation, concurrent changes, inadequate support, and social factors.744 Nursing perspectives were mostly positive on the use of a mobile PDA with a bar-code reader used to obtain medication profiles of patients and then uses as a decision support to identify drug therapy problems (e.g., drug interactions) for elderly home care patients, despite some system usability issues with the machine.635 Furthermore, some patients showed an interest when they saw the results from the electronic assessment.635

One ethnographic case study identified that the physician–nurse communications, mechanisms to ensure cooperation, and the procedures for preparing and administering the medications are the key process areas to address before implementing a system to augment the nursing administering of medications.762

Monitoring. Four qualitative studies assessed the clinician737,747 and patient633,760 perspective on the use of MMIT for medication monitoring. None addressed the effect of the systems on intermediate health care outcomes. The use of MMIT systems both facilitated and generated barriers to the process of patient monitoring by clinicians.737,747 One mobile phone-based system study showed that the MMIT system was well-accepted by patients as a mechanism to monitor symptoms for chemotherapy related toxicity.633

Adherence. No qualitative studies examined the effect of MMIT systems on medication adherence. MMIT systems facilitated patient monitoring737 by clinicians, however, barriers were reported to using health IT systems for patient monitoring.747 EMR with e-Prescribing facilitated monitoring and communication between physicians and patients with respect to the process of care that included checking active and inactive prescriptions and new and refill prescriptions, names of medication, and other medication themes (ordering and refilling prescriptions, mail-order issues, adherence, self regulation, alternate over the counter medication use issues).737 Clinicians caring for patients with HIV/AIDS using a CPOE and CDSS system integrated with the hospital, pharmacy, and laboratory systems identified six barriers to using reminders, including workload, time to document, reminders not applying, inapplicability to the situation, training shortcomings, quality of provider-patient interaction, and use of paper forms.747

Patients’ perceptions and experiences were studied based on their use of a mobile phone-based advanced symptom management system (ASyMS©) for chemotherapy-related toxicity monitoring and management.633 Patients with lung, breast, or colorectal cancer who used the system generally felt that, with training, the handset was straightforward and easy to use, entering data twice a day for 14 days was acceptable, the system did not impact on patients’ daily routines, and the set of six symptoms that were recorded on the handset were adequate (although some patients did indicate that they would have liked the opportunity to report other symptoms). They were very happy with the alerting facility of the system often reporting that they felt ‘secure’ in the knowledge that someone was being alerted about their symptoms, the real time, quick response rate of the data collection and alerting facility was viewed positively.633 However, one patient viewed the alerting system negatively, as she felt this part of the system was not sufficiently individually tailored.633

Another study focused on the patient perceptions of MMIT by studying a home telemonitoring device for ulcerative colitis that included their list of medications and questions designed to gather medication side effects.760 Patients felt that the system improved safety, feeling that the program ‘would catch something I might not recognize’ or help them ‘respond quickly to a threat’ to their health.760

Other studies with qualitative findings were also found.757,758,770

Population Level Outcomes

Only one study met our inclusion criteria that assessed population level outcomes as a primary endpoint (Appendix C, Evidence Table 11). Yu and colleagues712 in 2009 conducted a case-control study using actual reportable ADEs from a relatively large number of pediatric hospitals, comparing the rates of ADEs between cases and controls in hospitals with various degrees of CPOE implementation. The study found that patients from hospitals without CPOE were 42 percent more likely to experience a reportable ADE after adjusting for comorbidities; thus a significant benefit is associated with CPOE implementation.

Composite Outcomes

Only one included study assessed a composite outcome as their primary endpoint (Appendix C, Evidence Table 11). Holbrook and colleagues771 performed an RCT of 511 adult patients with type 2 diabetes receiving either usual care or an intervention involving shared access by patient and primary care provider to a Web -based diabetes tracker. The tracker interfaced with the providers’ EMR and a phone reminder system, which sent monthly reminders for medications, laboratory reports, or physician visits. The main endpoint of process composite score for checks of glycated hemoglobin, blood pressure, low density lipoprotein cholesterol, albuminuria, body mass index, foot surveillance, exercise, and smoking improved significantly more in the intervention group than in the control group (1.33 vs. 0.06 composite score scale; difference 1.27, 95 percent CI 0.79 to 1.75, p <0.001).

Variation in Impact Depending on Medication Type or Form

Summary of the Findings

Although most studies looked at medication management in general, regardless of drug families, types or forms, 135 articles dealt with one or a few drugs or drug classes.18,399,401,403–405,409–411,414,416,418,420–428,430,431,433,437,440–442,444–449,451,452,454,458–464,466,469–473,475–478,481,482,486,489,491,492,494,496–499,501,502,505–512,514–517,520–525,530,534,535,538,542,545,546,548,553,555,557,559,562–566,568,570,577,578,582,588,592,596,608,611–615,618,621,622,624,630,631,633,647,661,683–685,698,701,702,705,713,731,772

Prescribing and monitoring phases were again most often studied, with few studies looking at order communication, dispensing, or administering, and none on education (Table 16). No included studies addressed the issue of sound-alike or look-alike drugs, and four dealt with altering prescribing of generic drugs over name brand.414,458,510,535

Table 16. Number of studies across the medication management phases using MMIT to assist in the management of specific drugs or drug classes.

Table 16

Number of studies across the medication management phases using MMIT to assist in the management of specific drugs or drug classes.

Specifically, 30 articles focused on antibiotics,18,399,401,403,405,409,418,423,426,428,451,452,458,460,464,469,475,477,482,506,523,525,562,563,596,614,647,661,683684 seven on vaccinations,404,410,411,424,478,530,566 two on respiratory medications,446,613 three on psychotropics,476,502,520 two on nonnarcotic pain relievers,514,773 three on lipid-lowering agents,515,517,706 two on corticosteroids,462,553 12 on cardiovascular drugs,414,448,449,505,509,510,521,522,534,588,592,624 and four on insulin.466,630,631,703 Narrow therapeutic index drugs were considered in 20 studies,421,425,427,447,461,463,470–472,481,507,512,555,577,612,618,633,685,701,702 and controlled substances in seven.437,445,486,501,535,564,731

The form of medications was rarely mentioned, and was detected in only 18 studies.405,433,456,460,464,470,496,530,538,545,548,559,578,630,675,701,713,772 Prescribing changes from one drug form to another was the focus of two of these.460,464

We focused here on narrow therapeutic index, controlled drugs, and the forms of drugs. The 20 studies reporting on narrow therapeutic index drugs overwhelmingly measured process (n = 15) and clinical outcomes (n = 5), only two measured costs,612,685 and one study was a qualitative assessment of patients on chemotherapy.633 The effect of the MMIT systems was generally positive on the main outcomes of process change measures of prescribing or laboratory monitoring changes. Clinical outcomes frequently were better with the use of the MMIT, but in some instances no change was observed.425,702 Systems used to assist monitoring or prescribing of narrow therapeutic index drugs were all either CDSS or CPOE systems.

Six of the seven studies on controlled substances measured changes in process, four of which showed a positive impact.437,486,501,535 Only two measured clinical outcomes with mixed results.437,501 The controlled substance interventions were some form of reminders, alerts, or CDSS in all cases but one, which dealt with order sets for opioids in CPOE.437

The evidence in this small selection of articles indicates that health IT interventions designed to influence the management of patients taking narrow therapeutic index or controlled drugs have positive impacts in terms of changes in process; results are less clear for clinical outcomes with a number of studies showing no change.

Only two studies targeted changing the form of a drug,460,464 both of which employed a CDSS and had positive results. Due to a lack of reporting of the form of medication being studied, we can make no conclusions about the variation in effectiveness of MMIT by drug form.

Strengths and Limitations of the Evidence

Narrow therapeutic index drugs. Of the 20 narrow therapeutic index drug studies, three are RCTs507,612,618 with quality scores eight, seven, and seven out of nine respectively. Three cohort studies are included685,701,702 with low quality scores of three, two, and three out of 10 respectively. The remaining studies were observational421,425,427,447,461,463,470–472,481,512,555,577 or mixed methods.633

Controlled substances. The evidence for managing controlled substances rests on seven studies. The quality scores for the one RCT,535 one nonrandomized controlled trial,445 and one cohort study501 were generally low. The other four studies included a qualitative study,731 and three observational studies.437,486,564

General Study Characteristics

Narrow therapeutic index drugs. The narrow therapeutic index drug studies took place in hospitals (n = 14), ambulatory care (n = 6), and one at home. The drugs included digoxin,461,618 chemotherapy,421,447,633 anticoagulants,425,427,470,471,481,512,555,685,701,702 and others463,472,612 (Table 16). Three studies included CPOE interventions to assist with inpatient dosing,421,447,701 one on side effect monitoring by patients,633 and the remainder were CDSS alerts or reminder systems. Studies on anticoagulents measured adherence to prescribing and monitoring guidelines facilitated by some form of computer decision support.425,463,470,471,481,685 Two studies were of alerts sent to pharmacists for prescriptions written in primary care; one for prescriptions of drugs determined to be inappropriate for elderly patients507 and one for drug-drug interactions.577 One study implemented order sets within a CPOE for dosing of gentamicin and caffeine in the neonatal ICU, and assessed errors and drug turn-around times.463 Niiranen555 studied a computer-based warfarin followup system used by nurses to ease the burden on clinic physicians. Otherwise, prescribing physicians were most often the target of the alerts, reminders, or dosing support.

Controlled substances. Two studies of controlled substances occurred in primary care settings. In an RCT, Fortuna and colleagues535 assessed the effect of computerized prescribing alerts on the prescription rates of heavily marketed hypnotics and benzodiazepines compared with their generic counterparts in 257 physicians. Smith and colleagues564 implemented a CDSS module to reduce prescribing on nonpreferred drugs in elderly patients in 15 primary care clinics. The other five studies were performed in hospital settings and used CDSS interventions445,486,501,731 and order sets in a CPOE437 geared towards prescribing physicians.


Narrow therapeutic index drugs. The interventions aimed at pharmacists both resulted in significant reductions in inappropriate prescribing. Raebel and colleagues507 reported a relative risk reduction of 16 percent inappropriate prescribing for elderly patients, and Humphries et al.577 reported a 31 percent relative risk reduction in drug-drug interaction prescribing. The studies of CDSS dealing with narrow therapeutic or narrow therapeutic index drugs frequently resulted in better laboratory monitoring of patients,461,472,612,618 prescribing adherence,427,470–472 dosing,447,555,618 or avoidance of errors.421,463,512 Cordero and colleagues463 found reduced errors and quicker medication turnaround times with the use of CPOE ordering and dosing in the neonatal ICU. Negative results were found by Riggio481 with longer times to stop heparin treatment in patients experiencing heparin induced thrombocytopenia following implementation of an alert for 100 patients. Time from alert to laboratory test and start of direct thrombin inhibitor treatment did not vary before and after the implementation.

Clinical outcomes were measured in six studies. We considered positive studies to have at least 50 percent of the outcomes as being significantly impacted by the technology. Under this measure, four of the studies did not show significant impact of the technologies on patient outcomes425,555,685,702 though they tended towards being positive. Balcezak and colleagues701 found better prescribing of heparin when a computerized nomogram was used by prescribers, but the nomogram was only used for 10 percent of prescriptions written. The highest quality evidence comes from Raebel,507 White,618 and Feldstein612 and their colleagues who all showed positive, significant impacts of the technologies on narrow therapeutic index drug management.

Controlled drugs. The primary care RCT by Fortuna and colleagues535 found a significant decrease in the prescribing rates of heavily marketed drugs with the implementation of an alert plus education intervention, with a relative risk reduction of 74 percent. The observational study by Smith and colleagues564 in 15 primary care clinics to reduce prescribing on nonpreferred drugs in elderly patients showed a significant decrease in exposure of elderly patients to nonpreferred drugs, but no change in nonelderly patients, and a nonsignificant positive trend of preferred drugs in elderly patients.

The hospital-based quantitative studies showed generally positive process measures,437,486,501 with improved adherence to dosing in two,486,501 and better monitoring of patient pain levels.437 Morrison and colleagues445 found no change in prescription rates for laxatives to patients on opioids. Clinical outcomes were only measured by Peterson501 and Wrona437 and their colleagues. Peterson and colleagues found no change in length of stay or rate of altered status, but a significant reduction in falls (p = 0.001). Wrona and colleagues found improved respiratory rate in patients on morphine and hydromorphone with order sets outlining monitoring and documentation requirements.

Unintended Consequences of MMIT Applications

Summary of the Findings

The unintended consequences of health IT are important and often not well-studied. (Note that this section is not about drug-related ADEs.) These unintended consequences associated with an MMIT are often identified after a system is implemented, despite careful planning and installation. Unintended consequences can be minor or major and they can be viewed as being helpful to the installation or detrimental. Eighteen studies were identified that reported unintended consequences of MMIT installations (Appendix C, Evidence Table 12).15,16,450,457,480,503,508,732,734,743,752,759,774–779 Because we report only those outcomes that the authors reported as the primary or main findings of the study, this listing of articles on unintended consequences is likely not comprehensive.

Strengths and Limitations of the Evidence

One study is a large observational study of medication errors reported to MEDMARX facilities that covers all the phases of medication management.774 As in previous sections of this report most of the studies evaluated prescribing. All of the remaining 17 studies (one RCT,508 eight quantitative observational studies,15,16,450,457,480,732,775,777 six qualitative studies,734,759,776,778–780 and two mixed methods studies503,752) evaluated prescribing. Several of these studies also evaluated other phases. The order communication phase was evaluated in two studies, one observational15 and one qualitative study.732 Dispensing was studied in one observational study.15 Administering has one observational study15 and two qualitative studies.732,743 No studies of unintended consequences evaluated the monitoring phase or education and reconciliation.

General Study Characteristics

Participants. Most of the studies were done at an institution level rather than a patient or provider level. Raebel and colleagues508 studied medications with potential for harm to pregnant women and Han and colleagues15 studied admissions to a children’s hospital after implementation of a CPOE system. Nurses were evaluated in two studies,732,743 and the rest of the studies included a range of clinicians.

Location. All studies were done in single hospitals or groups of hospitals. One study was done in a long-term care center.732

Drugs and diseases. Raebel and colleagues508 studied drugs with potential for harm to the fetus in pregnant women (category D and X medications). All other studies included all medications.

Technology. All of studies but two involved CDSS and CPOE systems integrated with EMR systems, dispensing systems or pharmacy information systems. The two studies that did not include CDSS and CPOE systems involved BCMA743 and eMAR systems.732 They were both integrated within a hospital-based information system.


Ash and colleagues list a number of unintended consequences of MMIT and other health IT systems.778 These unintended consequences were categorized into direct compared with indirect, desirable compared with undesirable, and anticipated compared with unanticipated occurrences. Ash and colleagues contend that most unintended consequences center on errors, security concerns, and issues related to alerts, workflow, ergonomics, interpersonal relations, and reimplementation (e.g., updates). They also assert that all health IT systems will have unintended consequences.

Mortality. The University of Pittsburgh study of increased mortality with the introduction of an inflexible CPOE system is an example of a very serious unintended consequence.15 Because of the seriousness of the implications of this study, many people reviewed this article. Much attention has been given to this article and its methods.17 Another similar study shows that with careful planning, another children’s hospital did not see the same increase in mortality in admitted children after careful implementation of health IT.16

Errors. New and different types of errors were identified as unintended consequences in three studies.450,457,503 Although most MMIT systems are associated with decreased errors, not all of the systems sought to determine new or different types of errors—they most often studied existing types and classes of medication errors. One study felt that problems with communication would probably lead to errors in medication management,775 and another study postulated the same increase in errors based on challenges to existing and changing roles.734 The study of use of inappropriate medications during pregnancy was stopped early because the system was not accurate enough, causing the system to “miss” notification of drugs that should have been alerted and to give alerts that were not needed.508

Prescribing. Prescribing was not addressed specifically, although alert fatigue was a common theme in the studies of unintended consequences of MMIT.480,752

Efficiency. Ash and colleagues776 list 47 types of unintended consequences and Kopppel and colleagues752 list 22. Ash and colleagues go on to verify that the types of unintended consequences they found were common in institutions outside those that she and her colleagues studied.777 Unintended consequences were related to roles,734,743,752,776,781 communication,775,779 workflow alterations or automation of poor existing workflows,752,759,779 inflexibility of the new system,743,752,759 poor content or poor display of content,752,759,776 alert fatigue,480,776,779 and overdependence on the system.779 Rather than fix the system, most often workarounds were instituted by clinical staff.732,743

Summary. Seventeen of the 18 studies listed above report serious unintended consequences of MMIT in multiple categories. From these studies we see that unintended consequences exist for many health IT projects regardless of the quality of the implementation or the amount of planning that went into the project. Although consequences were viewed as being positive or negative, both provided useful information for those interested in MMIT implementation.

KQ2. What knowledge or evidence deficits exist regarding needed information to support estimates of cost, benefit, impact, and net value with regard to enabling health IT applications in terms of prescribing, order transmission, dispensing, administering, and monitoring, as well as reconciliation, education, and adherence? Discuss gaps in research, including specific areas that should be addressed and suggest possible public and private organizational types to perform the research and/or analysis


We identified gaps in the report—some that we expected and some not. We address gaps by the key questions (Table 17). In this section, some overlap exists with Chapter 5 (Future Research). Most of the gaps cross multiple phases of medication management. Where an issue is more strongly associated with a phase we mention the phase or other aspect (e.g., reconciliation).

Table 17. Summary of gaps and needs across key questions.

Table 17

Summary of gaps and needs across key questions.

General Gaps

Medication management phases. The literature places a great emphasis on studying the prescribing phase of medication management, with 263 of our included studies falling in that phase (Table 18). We feel that more study should be done on the phases of order communication, dispensing, and administering. In addition, the educational requirements for effective use of MMIT applications by health professionals needs to be studied. The evidence on the need to train patients and their families on how best to use MMIT systems as well as incorporating disease-specific information and management education into patient-based MMIT applications is needed.

Table 18. Frequency of medication management phases studies plus reconciliation and education.

Table 18

Frequency of medication management phases studies plus reconciliation and education.

Reconciliation of medications is vital, especially at the time of transfer to another health care setting, including transfer to and from home and community. Little evidence is available that MMIT systems are capable of and effective at doing this medication reconciliation and making adjustments to regimens. Challenges with system interoperability and standardized representation of medication data make effective reconciliation using MMIT applications difficult.

Order communication is ripe for more research and development, especially in two-way communication to improve and speed up “perfection” of orders and prescriptions.

Research methods. This same pattern of disparity for the number of studies in the medication management phases exists for the distribution of study methods. Most included studies are quantitative observational studies (Table 19). Although these studies provide good evidence for understanding and evaluating MMIT applications, more studies with control groups are needed to provide stronger methods where appropriate. MMIT applications are “complex” interventions and can be considered to be programmatic and pragmatic in their evaluation. Future research using methods appropriate for these complex interventions are needed. Studies of full MMIT systems and components of MMIT systems are needed.

Table 19. Frequency of research designs for included studies.

Table 19

Frequency of research designs for included studies.

Many studies were not powered to find the differences sought. We also identified other issues in study methods including inappropriate analyses, labeling of methods, and adjusting data sets in some of the observational studies. For example, studies seeking to identify such factors as barriers or facilitators of use of health IT systems did not report adjustment for multiple comparisons (e.g., Bonferroni corrections, bootstrapping, or Monte Carlo simulations). Some studies addressing feature preferences tested for 40 or more associations without adjustment. The authors of sections of this report also have commented on incorrect choice of statistical analysis techniques in some studies that could have led to positive findings that are not justified. Studies need strong statistical and methodological advice. We also agree with Bernstam and other informatics researchers and educators who suggest that research into MMIT systems needs to include those with informatics and research training and experience.782

Another gap in the research realm is the absence of formal study of MMIT in relation to knowledge translation (translational research). Much evidence exists on many aspects of MMIT.


Health care providers. Physicians are well-studied. Nurses, pharmacists, midlevel practitioners (e.g., nurse practitioners, physician assistants, advance practice nurses, and midwives), and hospital administrators are not (Table 20). Studies that include mental health professionals are also lacking. Studies that include nonphysician clinicians are not focused on the unique needs of the participants. The important issue of nursing workarounds that have developed to deal with systems that match physician but not nursing needs is also inadequately studied. Use and usability studies need to include all health professionals who use MMIT systems and studies need to be done that will allow knowledge gained in usability studies to be transferred to other settings.

Table 20. Number of studies that evaluated the effects of MMIT on clinicians (the clinicians were the major focus of the outcomes of the articles).

Table 20

Number of studies that evaluated the effects of MMIT on clinicians (the clinicians were the major focus of the outcomes of the articles).

Patients. The age range of patients impacted by the MMIT were generally well-represented across age groups with notable concentration among those who require more prescription medications (e.g., middle age and geriatrics) (Table 21). However, the special needs of medication management for children such as age- and weight-based dosing were not adequately pursued. More study of pediatric patients would be beneficial. Many of these patient-specific studies used data from patients to evaluate MMIT systems and their functioning in hospitals and primary care settings. However the needs of the patients and their families to manage medications outside of hospitals and clinics were not studied. This lack of evaluation of MMIT systems that patients and families will use at home and the effects of these systems on patient care and outcomes is an important gap that will only grow because of the advent of new systems, improvements in existing ones, and the move of patient centered care, chronic disease management with the aid of health IT, and continued time and money pressures on health care providers. Qualitative studies that address pharmacists as well as patient needs and opportunities and important outcomes were also lacking.

Table 21. Frequency with which patients or caregivers across age groups were studied as the main focus of an article (how MMIT affects patients).

Table 21

Frequency with which patients or caregivers across age groups were studied as the main focus of an article (how MMIT affects patients).

Settings. Hospitals and ambulatory care, but not necessarily specialty clinics, are also well- represented in the studies of this report (Table 22). The gaps are in other settings. Very few pharmacies or long-term care facilities were studied. Many existing articles on pharmacies and pharmacists were excluded because of lack of comparative data or integration of MMIT. Long-term care facilities, community locations, and homes also need formal evaluation to determine the effectiveness and use of MMIT applications for their constituents. One study evaluated outcomes at the population level.712 MMIT applications tend to target individuals and few of them measure population level effects. Research into the effect of MMIT on populations is challenging and research will have to be carefully planned.

Table 22. Study settings in which the MMIT application was studied (studies could take place in more than one setting).

Table 22

Study settings in which the MMIT application was studied (studies could take place in more than one setting).

Health IT systems. CDSS and CPOE systems are well-studied, most often in the prescribing and monitoring phases (Table 23). All other MMIT applications lack evidence of their effectiveness, especially in terms of workflow, communication, and clinical outcomes. Many studies did not report important details of the MMIT application itself, making the studies in this report more difficult to synthesize. From the descriptions in the articles we felt that descriptions of the system, including components and implementation issues such as training could have been added but they were not.

Table 23. Technologies that were the main focus of the studies of MMIT.

Table 23

Technologies that were the main focus of the studies of MMIT.

Another substantial gap that we noted is that the content of the MMIT systems was not studied. Systems like CDSS and CPOE and functions like drug-drug interactions and the knowledge base that reminders, alerting systems, and order sets are based on need a strong, evidence-based foundation of knowledge that is based on health research and reliably updated and disseminated. Assessment of the need for and value of this clinical evidence base was absent.

Health information exchange. Health information exchange is defined as the movement of health information across organizations using nationally accepted standards was not studied in any of the documents retrieved. Medication management is complex and challenged by interoperability of systems, and like reconciliation, it has not been evaluated in MMIT studies that we identified in this document.

Reporting. Through the process of data abstraction, we found problems with standardization and expanded inclusion of data elements in terms of reporting health IT studies. We feel that authors should be encouraged to strive for publication in the peer-reviewed literature rather than trade publications and news magazines. We also feel that authors should include more data in their publications of MMIT interventions. Lacks appear in descriptions of what was in place for medication management before implementation of MMIT systems (baseline data), full explanations of the MMIT system and its implementation process, settings, including culture, and participants (both health professionals and patients and their families). A guideline for writing evaluation reports in health IT, the STARE-HI, was published in 2009.783 We recommend that this document be used for planning and reporting research studies of MMIT. A list of the STARE-HI elements follows:

  1. Title
  2. Abstract
  3. Keywords
  4. Introduction
    1. Scientific background
    2. Rationale for the study
    3. Objectives of study
  5. Study context
    1. Organizational setting
    2. System details and system in use
  6. Methods
    1. Study design
    2. Theoretical background
    3. Participants
    4. Study flow
    5. Outcome measures or evaluation criteria
    6. Methods for data acquisition and measurement
    7. Methods for data analysis
  7. Results
    1. Demographic and other study coverage data
    2. Unexpected events during the study
    3. Study findings and outcome data
    4. Unexpected observations
  8. Discussion
    1. Answers to study questions
    2. Strengths and weaknesses of the study
    3. Results in relation to other studies
    4. Meaning and generalizability/applicability of the study
    5. Unanswered and new questions
  9. Conclusion
  10. Authors’ contribution
  11. Competing interests
  12. Acknowledgements
  13. References
  14. Appendices

Another of the challenges in this report to do with retrieval of studies from the bibliographic databases and also for abstraction and combining data, were inconsistencies in the use of terminology. We observed differences in how authors categorized medication errors, ADEs, and therapeutic failures. Several authors are seeking consensus on terminology in health IT. These definitional aspects are also addressed in the STARE-HI reporting guidelines listed above.783 Most of the studies in this evidence report do not follow these guidelines.

Benefit and impact. Benefit and impact are similar but not identical. In the pharmaceutical world benefit can be thought of as being “can it work” often under ideal situations (i.e., efficiency research). Much of the evidence answering KQ1: Effectiveness is of this kind of research: evaluation of a project, often near its implementation and for a short period of time. Many of these studies attest to the fact that for process and other soft outcomes, many of the MMIT systems do work.

Impact, or pragmatic studies, refer to measuring the effect of an intervention in the real world. Very few studies in this report are in this category. Trials of this nature are complex, long-term, have large numbers of people/situations being studied, and are done on mature and well- functioning systems. These trials are costly to complete and require maturity in the systems. Their location is likely best at those centers in the United States that have established and mature health care systems that have solid support for technology, strong research teams, experience with qualitative and quantitative methods and expertise in collaborative projects that include clinicians, experienced informaticians, and patients and their families.

The gaps for completing benefit studies include the medication management phases of order communication, dispensing, and administering; people besides physicians (pharmacists, nurses, other health care professionals, patients and families, vulnerable populations); nonhospital settings (long-term care facilities, community, pharmacies, and home settings); generics, forms of medication, and controlled substances; MMIT applications beyond CDSSs; and dispensing, administering, adherence tools, and patient involved health IT.

Cost and economics. Cost and economics are complex issues and important to many people, groups, organizations, and governments. To complete a comprehensive economic evaluation (e.g., cost-effectiveness, cost-utility, or cost-benefit analysis) one needs to quantify all costs and benefits within a given perspective (e.g., societal). Strong economic evaluations can piggyback on an RCT, or an economic model may be developed with data from a number of sources. Well-designed studies with an economic evaluation component included, is the best way to move forward in this area.

Many studies have provided cost data, but useful economic data involves far more input. An example of a cost study with data that is limited in its use is by Chisolm and colleagues,446 who did a before-after study of children with asthma in a children’s hospital. Their pharmacy charges were $373 before CPOE with standardized order sets were put in place, and $429 after implementation.

Therefore, the gaps for estimates of costs in this report of MMIT are almost identical to those listed above. In addition, we identified gaps in research quality centering on research design and analysis. We need highly trained and experienced researchers and economists to complete useful and usable cost and economics studies in the complex and changing domain of MMIT.


This report identified broad based strengths and gaps in the MMIT literature. Many of the major endpoints sought were found to show positive and statistically significant improvements, especially those that dealt with process and issues related to use, usability, knowledge, skills, and attitudes. Clinical endpoints and full economic evaluations were lacking. We also identified gaps in the study of the phases of medication, people involved, locations of studies, and research methods. We also identified areas where these gaps are becoming more important such as patient and family needs and opportunities related to MMIT, complete MMIT systems, and interoperability. Much research has been done in MMIT, and moving forward needs directed and careful planning and vision to fill gaps in our evidence base, harness the best established and new research methods, and build on what we already know to embrace new and advancing abilities of MMIT.

KQ3. What critical information regarding the impact of health IT applications implemented to support the phases of medication management is needed to give clinicians (physicians, nurses, psychologists, dentists), pharmacists, health care administrators, patients, and their families a clear understanding of the value proposition particular to them?

The value propositions of health IT applications have been difficult to quantify with more of a focus in recent years on framing how best to consider and measure it.784 Menachemi and Brooks785 review the benefits and costs of EHRs and associated patient safety technologies. They have found that studies assessing the benefits of the technologies in process and clinical outcomes are far more frequent than those assessing the return on investment. This trend is supported by the considerable evidence presented in the current report; while we include numerous studies assessing process changes and clinical outcomes, the body of evidence on cost-effectiveness is sparse. A number of barriers to measuring return on investment in health technologies exist. Technologies do not result in a direct income stream and the benefits often accrue to organizations other than the ones making the investment as, for example, clinical benefit to patients and financial benefits to payers rather than the hospitals making the investments.785 Investments in health IT produce a fundamentally different kind of asset to health care providers, and the technologies and changes they bring are so complex that it is difficult to measure their benefits.784 Certainly the body of literature looking at return on investment for the various technologies covered in this report, across the various settings, is very limited.

We use the Center for Information Technology Leadership’s (CITL) value framework (Table 24), which defines value as the sum of a technology’s financial, clinical, and organizational benefits.786 This fits well with the definition used by AHRQ whereby “value” is defined as “clinical, organizational, financial, or other benefits derived from the adoption, utilization, and diffusion of health IT less the costs of achieving these benefits” ( The same considerations for stakeholder value propositions are elements outlined by Ash and colleagues787 as important themes to consider when implementing a system, specifically CPOE. We recognize that this framework does not include patients as an element, but we believe that the framework could be applied to the patient perspective and incorporate value propositions for patients where applicable.

Table 24. Summary of the evidence in relation to the CITL value framework.

Table 24

Summary of the evidence in relation to the CITL value framework.

The required information to make an assessment of benefits is different depending on the stakeholder. The costs incurred by primary care physicians in practice will be different and balanced against different organizational benefits than those incurred in hospitals, and influenced by factors such as practice size, the sophistication of the technology, and others.786 Similarly, what constitutes benefits to a patient will be different from that of other users. Subramanian and colleagues788 have looked at costs and benefits to a number of stakeholders using CPOE with CDSS in long-term care facilities. Their process sheds light on the facets that we need to understand and study before we can make sweeping generalizations about value of health IT application. They identified the various stakeholders, the potential costs and benefits of the health IT, and factors which could affect costs and benefits. Ideally, such an assessment would be available for each stakeholder using each technology in each setting. This is not often the case so realistically we will broadly look at factors taken into account in making a value assessment and determine what we know and where the gaps lie.

Summary of the Findings

Given that the value framework is the sum of financial, clinical, and organizational benefits, the current body of literature summarized in our systematic review in KQ1: Effectiveness would indicate that too many unanswered questions exist to make a true value assessment for the different stakeholders and technologies in the applicable settings.

Financial Benefits

Cost reductions. The data on cost savings from the use of health IT in medication management are sparse. The few studies included in our review suggested that some cost savings may exist, which could be substantial over time. The economic information looks more favorable after the technology has been in place for an extended period of time so that the large upfront investment gets spread over time and then do we start to see a return on investment. However, a full economic evaluation requires the comparative analysis of alternative courses of action in terms of both costs and consequences, which provides the best information for making a decision to adopt an intervention or not, and very few of these have been rigorously completed in this field. We don’t have good evidence of a positive return on investment. Also, the initial expenditure and ongoing costs were rarely reported and the included cost analyses were based on projections of savings given reported changes in care processes rather than improved clinical outcomes for patients.

Revenue enhancements. No studies that quantified revenue enhancements were captured for this review. Because of the nature of health IT assets, they do not per se bring about additional revenues to the investors.

Productivity gains. Evidence captured in KQ1: Effectiveness suggests that some productivity gains are achieved, often measured as improvements in efficiency in care processes.429,457,463 Gains achieved by reductions in outcomes such as lengths of stay or rehospitalizations have been less successful, though Durieux and colleagues716 do report a significant decline in hospital length of stay in a review of drug dosing decision support technologies. A number of studies reported positive improvements in efficiency outcomes such as drug turnaround times,584,586,628 and time to administering drugs.439,600 One study reported that nurses spent about the same time on computer documentation as paper documentation.561 In our review, efficiencies were rarely the main endpoints of any of the studies; they were frequently reported as secondary outcomes or additional measures analyzed, but without any assessment of the power of the analysis. Because of the quality of the studies, it is difficult to attribute true productivity gains except in the cases of some well-established systems as suggested by Chaudhry and colleagues.607 The qualitative evidence indicates that stakeholders believe that gains in productivity have occurred.439,547,632

Clinical Benefits

Care processes. Certainly this aspect of values is the most studied across the phases of medication management, with 379 studies included in our review in KQ1: Effectiveness. These studies included a number of settings and stakeholders, and most reported improvements in processes of prescribing changes, adherence to guidelines or quality measures, error reductions, preventive care procedures done, and monitoring initiated. However, the studies were often observational and often had small sample sizes. In more than 80 percent of the cases in which an improvement in process was sought, it was found to be positive. The findings of improvement were consistent across settings, levels of care, providers, and medication management phase. We report a positive effect in the use of MMIT in the prescribing and monitoring of controlled and toxic drugs as well. To balance this positive nature of the results, a growing body of evidence delineates unintended consequences of some technologies that will also contribute to the value proposition of stakeholders.632,734,752

Patient clinical outcomes. We reported on 78 studies that assessed clinical outcomes as their primary endpoints, the majority of which focused on prescribing and monitoring phases. About half of these studies reported positive effects of the MMIT on patient outcomes. However, when clinical measures were the primary endpoint, often no differences between the intervention and control groups in the higher quality studies were seen (see Table 15). The strongest evidentiary weight for clinical outcomes is found in the use of CDSSs for the prescribing and monitoring phases, and the overall benefit is somewhat positive but most often mixed.725 The measurement of clinical outcomes is often so far removed from the MMIT intervention that it becomes difficult to make general conclusions about their efficacy, and adoption rates are still quite low. We found that efficacy was greater in interventions targeting specific populations or applications. Thus, a value assessment on patient outcomes would warrant a look at specific technologies, populations, and settings beyond the scope of this report.

Organizational Benefits

Stakeholder satisfaction. For implementation, adoption, and ongoing use of any technology to be successful, the people using the system need to find it useful, usable, and nondisruptive. Many studies have looked at workflow issues, satisfaction, and perceptions of users with regard to health IT. Our review includes only those providing qualitative data or comparison groups. The literature on satisfaction indicates that generally the stakeholders studied were satisfied with the technologies of interest, namely CPOE, CDSS, and e-Prescribing.651,654–656,656–658 Some studies, however, found no differences in satisfaction.651,659 Levels of satisfaction and positive perceptions were shown to be positively correlated with measures such as ease of use, productivity, quality of care, and reliability.654–657,661,673 Our review of the qualitative research in the area shows that the implementation of MMIT generates emotion, both positive and negative. MMIT implementation did not just mean that a clinician needed to learn a new IT system but it also affected most of the other parts of the delivery of care processes including how the interdisciplinary care team worked together. When determining the proposition values, the type of technology and how well it meets expectations and workflow are important considerations for users, greatly impacting their perceptions and openness to adoption/use.

Some literature has focused on comparing perceptions and attitudes of different health care providers, such as nurses compared with physicians and trainees;656,678 and residents compared with physicians using the same technologies.654,657,677 The findings from these studies indicate that perceptions of the benefits of MMIT can depend on the role of the user. The type of system and how it affects health care providers’ work will impact how satisfied these stakeholders are with the technologies.

For any one technology or setting, insufficient data exist to determine levels of satisfaction among all stakeholders. From the literature we see that satisfaction and perceptions of the MMIT can vary according to provider role, setting, and technology, and no overall answer to the question of stakeholder satisfaction exists. We have a deficiency of comprehensive studies of patients as stakeholders.

Risk mitigation. No literature was captured on risk mitigation in relation to the use of MMIT. A focus of the greater body of research, especially commentaries and narrative reviews, is on the use of technologies to reduce medication errors. Such benefits could have repercussions on risk mitigation, but also needs to be balanced with the fact that some technologies have been shown to result in new kinds of errors.


Only one study attempted to look at the value propositions across stakeholders in the use of MMIT and they concluded that to facilitate adoption of CDSSs in the long term care setting, financial incentives to both the institutions and physicians should be considered.788 Certainly, from the literature, we see no clear understanding of what information is needed from the standpoint of each stakeholder. We can surmise from studies that physicians consider cost, usability, patient improvements, and easy integration into workflow as important factors to consider before they purchase MMIT technologies. Hospital administrators place emphasis on other aspects such as costs, return on investment, and organizational change. The relative importance of these factors will vary among physicians practicing in different settings, with cost being more important to physicians in private practice than in hospitals, and other related issues. Capitation rates will also be a factor for physicians and will vary across U.S. states.786 Similarly, the importance of these factors will vary among pharmacists depending on their practice setting and the type of technology. For patients, convenience, usability, portability, and patient-centered functionality have been reported as important factors in their value assessment of consumer health IT.4 For MMIT, patients will likely be concerned with reduced medication costs, avoidance of ADEs, and improved disease management, although no studies evaluated their value-based concerns. Work needs to be done to identify the needed critical information before we can truly assess what is missing.

From the information garnered in this report, a growing body of evidence supports the use of some technologies (e.g., CDSSs) in prescribing and monitoring, which show positive changes in process, while large gaps in knowledge of the impact of the use of MMIT for other applications still exist (see KQ2: Gaps in Knowledge).

KQ4. What evidence exists regarding the impact of the characteristics of medication management health IT applications, such as open source, proprietary, conformity with Federal and other interoperability standards, and being Certification Commission for Healthcare Information Technology (CCHIT) certified, impact, likelihood for purchase, implementation, and use of such IT applications

Summary of the Findings

Few studies (n = 21)45,48,632,653,661,663,667,789–802 demonstrated evidence of the impact of the characteristics of MMIT applications on likelihood to purchase, implement, and use such IT applications (Table 25). Little substantial evidence was found from studies that assessed open source health IT applications that met our inclusion criteria. Only two articles discussed conformity with standards653,801 and one the Certification Commission for Healthcare Information Technology (CCHIT) certified system.800 Such system characteristics as the use of proprietary IT systems was suggested by seven articles45,632,653,663,798–800 and homegrown IT application by one article.667 Two reported on a stand-alone e-Prescribing system.653,798 Most of the articles suggest that the decision to adopt health IT applications has been influenced by the feature sets of health IT applications. Each of the 21 articles included in this section established evidence on likelihood to use, one on purchase,800 and five on implementation.653,789,791,793,798 A sizeable number (n = 20) of articles were on the prescribing and ordering phases, with only one on the administering phase of medication management.45

Table 25. Number of articles addressing system type in relation to likelihood to purchase, implement, or use an MMIT system.

Table 25

Number of articles addressing system type in relation to likelihood to purchase, implement, or use an MMIT system.

The findings of the articles included in our study suggest that certain features of systems improve the likelihood of purchase, implementation, and use of MMIT. However, the literature is sparse and evidence from studies with stronger methods that can address this question is lacking. Most often authors spoke about barriers and concerns towards implementation and acceptance rather than characteristics of MMIT that could facilitate implementation, purchase, and use of such systems. Insufficient details were given about the technology they were studying. Head-to-head comparisons of systems differing in these features were not found.

A systematic review on CDSS revealed that widespread dissemination of appropriate CDSS might improve clinical practice, but providing information in electronic format alone does not ensure uptake.803 Fundamental issues related to system characteristics included the availability and accessibility of hardware, technical support and training, system integration into clinical workflow, timeliness of clinical messages, and acceptance of the system by various stakeholders.803 Another review involving descriptions of 112 information systems identified that for successful implementation, core components were order entry, guideline adherence, and decision support.804 Involving end users in the development process was also shown to be a key to success.804 However, these systematic reviews did not explore whether health IT system characteristics like proprietary or homegrown, system configuration, system characteristics, CCHIT certified, conformity with interoperability standard and standalone or integrated had any impact on purchase, implementation, or use.803,804

Strengths and Limitations of the Evidence

Most of the studies were surveys (n = 18), although two used qualitative research methods632,801 and one collected data from scientific literature, organizations, government, and professional reports.797 Therefore, the strength of the evidence is relatively weak. Nineteen articles were published in the original literature and one was from the grey literature.800

General Study Characteristics

Participants. More than half of the studies (n = 13)48,653,661,663,667,790,792,794–797,799,802 evaluated physicians as the user of the technology. One article each included pharmacists,661 nurses,667 directors and the leader of IT application users,45 chief information officer,791 pharmacy directors,45,793 and two administrative and other medical staff.661,667 Two reported combinations of different types of health care providers.632,798 One study convened a panel of technical experts representing organizations having direct experience in implementing e-Prescribing standards.801 The size of the studies ranged from 14 to 18,600 participants.

Study setting. In most of the studies, the participants were primarily from hospitals,661,663,789–791,793 and some were set in pharmacies,45 ambulatory care,632,653,667,798–800 and primary care.794,795,797 Four evaluated a combination of various settings.48,792,796,802

Technology. Primarily five groups of health IT systems, namely, CPOE, CPOE with CDSS, CDSS, e-Prescribing, EHRs, and five other systems were studied.

Research methods. Research methods were weak: eighteen articles were surveys, two used qualitative research,801,805 while one used data from scientific literature, organizations, government, and professional reports.797


Identification of feature sets. Bell and colleagues conducted an expert panel consensus that resulted in 60 specific functional recommendations for e-Prescribing to improve patients’ health outcomes and reduce costs.806 This list of features is useful for those considering an assessment in this area. We identified that one or more of these recommended features were the driving forces toward possible purchase, implementation, and use of health IT applications. Major features addressed in most of the articles were medication lists,208,632,653,663,789,792,794,798,801 dosing calculations,661,789,792,793,799 CDSSs (alerts and messages for allergies, drug-drug interaction, drug approval),45,632,653,661,667,789–794,796,798–801 e-Prescribing,48,208,632,653,667,790,792,794–798,802 and order communication of the prescription to pharmacies.45,632,792,794,798 Other factors were access to laboratory test results,45,789,790,792,794,797,799,800,802 implementation of guidelines,661,789,792,795,796,799 transcription services,791 formularies,653,793,795,801 tallman letters and change of color to differentiate between look-alike drug name pairs,45 integration with another system like BCMA, pharmacy information systems, etc.,45,791,793 and medication reconciliation (Table 26).792,793 (Tallman letters are the use of capitals to help guarantee recognition of differences between drugs with similar names, as for example, NovoLOG and NovoLIN, and HumaLOG and HumuLIN, helps differentiate these products.)

Table 26. List of articles addressing various features that were instrumental in the decision to purchase, implement, and use.

Table 26

List of articles addressing various features that were instrumental in the decision to purchase, implement, and use.

Standards and conformity. Wang and colleagues653 suggest that mandating the use of standards is necessary but not sufficient for achieving the desired effects of e-Prescribing. Bell and colleagues evaluated two standards (Medication History Standard and Formularies and Benefits Standards) from the U.S. National Council for Prescription Drug Programs (NCPDP) that were considered as initial standards for e-Prescribing under Medicare.801 Another study considered CCHIT certified IT applications to be the deciding factor for likelihood to purchase, implement, and use.800 The 2008 Healthcare Information and Management Systems Society Analytic Ambulatory health IT survey reported commercial, proprietary EMR systems were being used without any one vendor being the dominant leader.800 Apart from these two articles,653,800 four other articles45,632,663,799 reported the use of commercial proprietary systems with medication management health feature sets.

Use. All the studies addressing the decision to use were based primarily on one or more of the feature sets discussed above (Table 26). Three studies were on CPOE systems with CDSS capabilities45,667,793 and one on a CPOE system alone.663 The features that were more important were allergy checking, drug interactions, medication formulation, interface with the pharmacy information system and direct order communication, and integration with the BCMA and laboratory systems. Two studies were on CDSS,661,667 with one being integrated with CPOE.667 Their important features were e-Prescribing, drug-drug interactions, calculation of dosing, and access to additional information.

Seven studies were based on EHR, EMR, or clinical information systems789,790,792,794,797,799,800 with one study789 mentioning that a sizeable number of hospitals reported having implemented several key functionalities of CPOE and CDSS. The important features were CPOE with CDSS, electronically available laboratory test results, medication lists, e-Prescribing with electronic transmittal of prescriptions to pharmacies, access to reference materials, and dosing calculations.

Three studies were on general health IT systems.791,795,796 Grossman et al.,796 found the percentage of physicians reporting access to clinical activities such as obtaining guidelines, generating reminders, and writing e-Prescriptions increased from 2000–2001 to 2004–2005 (p < 0.05). Six studies45,48,632,653,798,801,802 were on e-Prescribing with one being integrated with another system and hand-held access.632 Some of the more important features addressed by these studies were e-Prescribing, medication lists, drug interaction and allergy alerts, receiving laboratory results electronically, changing doses, formularies, and order communication of prescription to pharmacies. According to the study by Bell and colleagues,801 implementation of medication history standard and formulary and benefit standards in e-Prescribing would likely enhance usability of such systems if standard implementation was improved.

The qualitative study by Weingart and colleagues632 found that the most valuable aspects of e-Prescribing in ambulatory care were the ease of changing doses, renewing prescriptions, ensuring legibility, and transmitting prescriptions to in- and out-of-state pharmacies. Participants were dissatisfied with the unreliability of transmitting prescriptions successfully to the pharmacy, creating medication lists, recording of allergy information, and quantity of irrelevant and inappropriate alerts. Despite their complaints about alerts, participants preferred to continue receiving alerts as a safeguard against missing a major interaction.

The studies of such health IT systems as pharmacy information systems found that important features were CDSS alerts, interface with the laboratory system, and tallman letters and change of color to differentiate between look-alike drug name pairs.45

Purchase. One article reported on likelihood to purchase in a group of which half of the respondents of that survey were planning to purchase a CCHIT certified EMR system.800 The important features were electronic connectivity for laboratory test results and orders, nursing and physician orders for medications, and prescription refills.

Implementation. Five articles reported on the likelihood to implement an MMIT system.653,789,791,793,798 Larger hospitals, those located in urban areas, and teaching hospitals are more likely to implement EHR systems.789 Collectively, the important features were allergy checking, drug interactions, medication history, dosing calculation, medication formulation, and availability of laboratory test results.

Wang and colleagues653 conducted a descriptive field study of ten commercially available ambulatory e-Prescribing systems, to compare the functional capabilities offered by commercial ambulatory electronic system with 60 expert panel recommendations suggested by Bell and colleagues.798,806 Data were collected from vendors by telephone interview and at sites where the systems were functioning, through direct observation of the systems and through personal interviews with prescribers and technical staff. Five of the systems were full EHR systems and five were nonEHR systems. Among the 60 e-Prescribing recommendations by Bell and colleagues,806 nine recommendations were not implemented by any of the ten systems.798 These included recommendations that would require e-Prescribing systems to handle prescription fulfillment data (their recommendations 10, 47, and 48), to use more complex drug benefit data (recommendation 22), and to use more advanced drug knowledge bases (recommendations 26 and 49).798,806 Prescribing systems that were part of EHR systems implemented more recommendations than did stand-alone nonEHR systems. Considering all 60 recommendations, the median EHR-based system fully implemented 60 percent, whereas the median nonEHR system fully implemented 35 percent (p = 0.09). Including partial and full support together, median implementation levels were 72 percent for EHR systems and 46 percent for nonEHR systems (p = 0.06). On average, the systems fully implemented 50 percent of the recommended capabilities, with individual systems ranging from 26 percent to 64 percent implementation. Only 15 percent of the recommended capabilities were not implemented by any system.

Level of care. Six studies evaluated systems for ambulatory care.632,653,667,798–800 Features of the six ambulatory care studies centered around clinician experience using commercial proprietary systems, with CDSS capabilities being the most common feature used.

KQ5. What factors influence sustainability (use and periodic updates) of health IT applications that support a phase of medication management continuum (prescribing, dispensing, administering, and patients’ taking of medications)?

Sustainability of Health IT and Medication Management Systems

AHRQ seeks to support activities that can demonstrate the effect of health IT on important outcomes relating to quality, safety, efficiency, and effectiveness. Moreover, AHRQ places priority on initiatives to identify and overcome barriers to health IT implementation and adoption and to foster long-term sustainability.808 Intuitively, a system’s sustainability refers to its capacity to continue providing value. Sustaining the benefit of health IT applications may require ongoing resources for maintenance and updating, training and support for those who use the systems, as well as institutional support that encompasses planning, implementation and maturing of the systems, and replacement as needed. Thus the concept of sustainability raises questions about the long-term viability of many health IT interventions, as well as important concerns about the potential health impact of migrating existing processes to less sustainable or costly forms.

We conducted an additional comprehensive review of the literature to find a suitable operational definition and set of metrics of sustainable health IT. We found this necessary because we did not have, and could not readily find a prespecified definition that was widely accepted or supported in the literature of health IT.

In search of a definition of sustainability relevant to health IT, we did additional searching in the core informatics journals using the key term “sustainability” to identify articles that have discussed the concept (Table 27). Some articles defined sustainability quite narrowly (e.g., the decline of prescribing improvements once experimental alerts were removed from a system that had integrated CPOE and CDSS systems). We believe that the most relevant available definition comes from Humphreys and colleagues,9 who defined sustainability as the ability of a health service to provide ongoing access to appropriate quality care in a cost effective and health-effective manner.

Table 27. Frequency of core informatics journal articles that mention sustainability to the end of 2009.

Table 27

Frequency of core informatics journal articles that mention sustainability to the end of 2009.

Our literature reviews revealed three important findings: although sustainability is mentioned frequently in the core informatics literature, it is poorly and infrequently defined, and none of the articles identified in the primary literature searching done to produce this evidence report explicitly studied sustainability. These findings were not entirely surprising. A previous AHRQ-sponsored Evidence Report that assessed the costs and benefits of health IT in pediatrics found only one article that explicitly discussed sustainability.21

Future Sustainability of Health IT and Medication Management Systems

In 2009, the United States passed the Health Information Technology for Economic and Clinical Health (HITECH) act to authorize incentive payments through Medicaid and Medicare to clinicians and hospitals when they use electronic health records (EHRs) for patient care. The legislation ties payments specifically to the achievement of advances in health care processes and outcomes. Starting in 2011, the HITECH act will make available incentive payments totaling up to $27 billion over 10 years. This legislation will require substantial collaboration between health IT workforce professionals including those from IT, health information management, and biomedical informatics to accomplish its goals.

According to Dr. David Blumenthal, the U.S. National Coordinator for Health Information Technology at the Department of Health and Human Services, “[this legislation] will lead us toward improvements and sustainability of our health care system that can only be attained with the help of a reliable and secure nationwide electronic health information system.”809 HITECH’s goal is not just based on the adoption, but also on the “meaningful use” of EHRs. Meaningful use is defined by a set of core health IT objectives that constitute an essential starting point, as well as an additional menu of activities which providers and hospitals will choose to implement during 2011 to 2012 (see Figure 5).809 Overall, these features should help clinicians make better medical decisions and potentially avoid preventable errors.

Figure 5 is a summary of meaningful use objectives for health information technology. Meaningful use is defined by a set of core health IT objectives that constitute an essential starting point for demonstration of health information technology which is being used in a meaningful way. For each aspect of meaningful use of health IT which is to be measured, there is a stated objective (i.e., a descriptive narrative) and a means to demonstrate that the objective as mean met, know as a measure. The two-column table with alternating blue and white shading depicts two groups (Core and Menu) of Meaningful Use Measures (column 1) and Objectives (column 2). There are 15 objectives and measures in the Core Set and 12 objectives and measures in the Menu Set. There are two general categories which (rows, Additional choices for hospitals and critical access hospitals and Additional choices for eligible professions) which do not have a measure associated with them. Rather these categories describe the two objectives which follow each of them.

Figure 5

Summary overview of meaningful use objectives. Source: New England Journal of Medicine, 2010. Used with permission.

This legislation may lead to improvements and sustainability of health IT applications that specifically support the medication management continuum. For example, to receive incentive payments, eligible professionals (e.g., physicians, optometrists, podiatrists, and chiropractors), and hospitals (e.g., acute care hospitals and critical care access hospitals) must implement and use the following core set of objectives that relate to medication management: CPOE, e-Prescribing, implementation of at least one decision support rule, and maintenance of active medication and allergy lists. Eligible professionals and hospitals may implement and use the additional menu set of objectives that relate to medication management: incorporation of clinical laboratory test results in the EHRs, performance of medication reconciliation across care settings, and sending reminders to patients for followup care.


We conducted an additional literature review of the core informatics journals to identify articles that have discussed sustainability related to MMIT systems and found that while sustainability is not infrequently mentioned in this informatics literature, it is often poorly defined, and none of the articles included in this evidence report explicitly discussed sustainability. Future research should develop an operational definition of sustainability that can be used to study its determinants. Moreover, it is likely that the HITECH Act of 2009 will lead to improvements and sustainability of health IT applications that specifically support the medication management continuum through meaningful use.

5a. To what extent does the evidence demonstrate that health care settings (inpatient, ambulatory, long-term care, etc.) influence implementation, use, and effectiveness of such health IT applications?


Reports of implementation tend to be opinion pieces or descriptive studies. A number of articles looked at some or all of implementation, adoption rates, and factors related to adoption. These focused mostly on CPOE in hospitals,789,810–814 e-Prescribing, or ambulatory CPOE in primary care.48,172,794,796,815–819 These articles did not meet our criteria for inclusion in KQ1: Effectiveness because of methods limitations. The general findings for hospitals show that implementation and adoption are generally greater in larger, academic, urban, public hospitals. Adoption in primary care practices tends to increase with younger, recent medical grads, larger practice size, and also with more specialized physicians. Yet, overall, actual usage of the systems is low with varying rates across MMIT systems, facilities, and groups of people.

Poon and colleagues820,821 discussed a number of barriers to CPOE implementation in U.S. hospitals, and provided recommendations to overcome the barriers based on experiences in successful hospitals. They categorized barriers into physician and organizational resistance, cost and lack of capital, and vendor or product immaturity. They further provided recommendations to overcome these barriers.

Ash et al.787 provided a consensus statement with a list of categories and considerations for a successful implementation of CPOE (see for more details). Their predominant themes to consider are: motivation for implementation; CPOE vision, leadership, and personnel; costs; integration; value to users; project management and implementation staging; the technology; ongoing staff training, support, and evaluation. These themes reflect the considerations for the values propositions of the various stakeholders as addressed in KQ3: Value Proposition.

The implementation of a new health IT can have unintended consequences, often as a result of the interaction between the technology and the sociotechnical system within which it is implemented. This would include the workflows, culture, social interactions, and technologies in place. Harrison et al.822 have modeled this interactive sociotechnical system to help users and implementers understand how and where unintended consequences could arise within a particular setting. Some examples of unintended consequences have been reported in this document, and our sections on intermediate and qualitative research in KQ1: Effectiveness describe some information on workflows, social interactions, communication, and interdisciplinary work challenges with MMIT implementation. Assessing the potential impact of a new MMIT system using the framework of Harrison et al.822 could easily help to avoid some of the unintended consequences reported to date.

Beyond CPOE and e-Prescribing systems, the useable literature on implementation and adoption of other technologies is negligible. Some surveys did look at the adoption of various health IT applications for patient safety. Menachemi and colleagues823 measured rates of adoption of CPOE, BCMA applications, pharmacy IT systems, pharmacy dispensing, EHRs, PDAs and CDSSs in Floridian acute care hospitals. Pharmacy systems were widely adopted (85 percent for IT systems and 64 percent for dispensing), and the rest ranged from 12 to 40 percent. They also studied health IT adoption in U.S. pediatric hospitals and found a fairly high level of adoption (almost 50 percent for EHRs, 40 percent CPOE, and 36 percent CDSS). Furukawa and colleagues824 used national survey data to measure adoption of technologies across the United States. They found a range of levels of adoption across technologies, from 62 percent for automated drug dispensing to five percent for BCMA. Their analysis supports the findings that hospital size, teaching status, hospital or clinical ownership, and system membership are associated with adoption. Robinson et al.790 found that adoption of 19 health IT capabilities, including MMIT, was higher when practices were evaluated for pay-for-performance and public reporting purposes, and in practices participating in quality improvement initiatives. From our review of the qualitative literature, we find that unintended negative consequences, the need to develop workarounds including changes to workflow, and the resultant negative emotions generated with MMIT implementation are important to recognize and deal with in order to improve the success of implementation.

The American Society of Health-System Pharmacists (ASHP) have published the results of an ongoing series of surveys assessing the adoption and use of pharmacy informatics applications in U.S. hospitals and trends in pharmacy practice.825 Again the studies are descriptive surveys and not included in our KQ1: Effectiveness. They do, however, show how hospitals are progressing rapidly in their adoption of health IT used in their pharmacies.826,827


Due to the observational nature of many of the studies assessing health IT across settings, this question is difficult to answer. Hospitals and primary care are well-studied, especially for the two phases of prescribing and ordering, and monitoring. Gaps are seen in the other phases of medication management, and education and reconciliation. Further, some MMIT applications are well-studied while others, such as those that pharmacists and nurses use, are less well-evaluated. A limited number of studies are carried out in long-term care settings, pharmacies, or with patients at home, or other community settings. Many of the hospital- and clinic-based studies tended to show improvements in process with some, but limited, evidence of clinical improvements. These research gaps in settings, MMIT applications, and health professionals, together with proof of effectiveness are similar to the deficiencies seen with cost-effectiveness or other types of economics studies, especially full economic analyses which are required to address and satisfy the definition of sustainability used here.


From KQ1: Effectiveness we see that the study of use of systems is rarely done in a completely rigorous way. Articles that measure use tend to frame it in the context of adoption and implementation, looking merely to ascertain if systems are used, not how they are used and if they are being used appropriately. Few measure levels of use. Again, the definition of sustainability is not met without the inclusion of economics studies.

5b. What is the impact (challenges, merits, costs, and benefits) of having electronic access to patients’ computerized medication records, formulary information, billing information, laboratory records in the quality and safety of care provided by health IT applications that support at least one phase of the continuum of medication management (prescribing, dispensing, administering, and patients taking of medications)?

Almost all of the MMIT applications we report were integrated with at least one other system. The systematic review in KQ1: Effectiveness addresses the effect of these integrated technologies on a range of outcomes, many related to patient safety and health care quality. Evidence is available to address prescribing and also monitoring. The other phases are not well- evaluated. The study of patient access to their medication records and integration of these data into clinic and hospital information systems (EMRs and EHRs) is exciting. Some evidence exists that the use of MMIT integrated into clinician-based systems holds much promise and will be an exciting area of research in the next decade. This is especially important with the efforts by the U.S. government to improve health care delivery and to implement health IT systems to enhance this expanded delivery.

KQ6. Two-Way Prescriber and Pharmacy Electronic Data Interchange (e-Prescribing)
(a) What evidences exists demonstrating the barriers and drivers of implementation of complete EDI that can support the prescription, transmittal and receipt, and perfection process of e-Prescriptions
(b) How do barriers, facilitators, and economic incentives vary across pharmacists, physicians and other relevant stakeholders with respect to adoption and use of complete EDI (e-Prescribing/ordering with e-transmission)?

All studies eligible for inclusion in this evidence report were reviewed to determine if they evaluated two-way, complete EDI between prescribers and pharmacies, commonly referred to as e-Prescribing. To be considered a true one-way e-Prescribing system the article had to describe a computer system used by a prescriber to generate a prescription (authorization to supply drug) that is transmitted electronically to a pharmacy information system. Further, for the system to be considered a two-way e-Prescribing system it had to be capable of transmitting a message from dispenser to prescriber by electronic means. This criterion is broadly consistent with the definition of e-Prescribing promulgated under the U.S. Medicare Modernization Act of 2003.828 We did not consider systems used for inpatients of a hospital to be an e-Prescribing system; these technologies are reviewed elsewhere in the report, often under the rubric of CPOE systems.

Summary of the Findings

Thirty-three reports434,549,561,575,579,584–586,645,668,724,730,736,797,800,801,806,829–844 were checked for eligibility and only one585 met the above criteria for inclusion for bidirectional e-Prescribing systems. Nearly all systems self-described by investigators as “e-Prescribing” allowed physicians or other prescribers to generate a prescription through a software application that were later reproduced in paper form prior to being dispensed by a pharmacist (incomplete one-way e-Prescribing). One report585 described an interrupted time-series study of a two-way e-Prescribing system intended to reduce the time required for prescribers to respond to pharmacist queries and refill requests. The authors did not describe any barriers or facilitators to uptake of the system used in the small pilot study.

We have extrapolated key themes from the data available on one-way or incomplete one-way e-Prescribing to describe potential barriers and drivers to implementation of complete two-way EDI. These data may be useful indicators of issues that would need to be addressed before widespread implementation of two-way EDI would be expected to yield benefits for stakeholders. The following facilitators and barriers are listed in order of high to low frequency of mention in the reviewed literature.


  1. External monetary or other incentives to prescribers. Nearly all reports of e-Prescribing implementations in the United States described some financial incentive that was offered to prescribers to adopt an e-Prescribing system.839 In most of those cases where no financial incentive was offered, the system was adopted by a health system that required its prescribers to adopt the system.
  2. Supportive regulatory environment. Formal endorsement by regulators such as the State Boards of Pharmacy or Medicine seemed necessary enablers for prescribers to adopt e-Prescribing systems.736,839
  3. Existence of some standard for prescription electronic data interchange. A set of messaging standards to enable the electronic flow of prescription information between diverse software platforms have been developed for use in the prescribing and order communication processes.834,836,845 While not all standards have been judged suitable for implementation,839 the core set of standards currently available should facilitate further development and testing of e-Prescribing solutions.


  1. Incomplete consideration of the effects of e-Prescribing on pharmacists and pharmacies. Most evaluations of one-way e-Prescribing systems conducted in the United States focused almost entirely on the e-Prescribing system from the perspective of the prescriber, the prescriber’s staff, or both.736,833–836,838,839 Several of these reports described a lack of awareness of the e-Prescribing process on the part of pharmacies and pharmacists and a subsequent need to educate pharmacists on the specific e-Prescribing process adopted by the prescriber.736,835 Pharmacists and pharmacy staff generally reported that e-Prescribing systems negatively impacted their workflow.645,833,834 The authors of an AHRQ commissioned report839 on e-Prescribing pilot projects conclude that the prescribing workflow advantages observed for prescribers using e-Prescribing may actually reflect a burden shift to pharmacists. While reduced pharmacy to prescriber callback rates are touted as a potential advantage to e-Prescribing, the highest quality evidence available did not support a reduced callback rate.575 A sample of e-Prescribing prescriptions sent to selected pharmacies in Denmark was prospectively compared with a sample of handwritten prescriptions sent to the same set of pharmacies. The investigators’ adjusted analysis indicated a significantly higher likelihood (relative risk, 1.7; 95 percent CI, 1.3 to 2.2) of pharmacy callbacks to prescribers for electronic compared with paper prescriptions.575 This finding is especially significant as nearly two-thirds of prescriptions are transmitted electronically in Denmark.575,797 All pharmacies in Denmark are government owned and therefore likely share the same IT infrastructure.
  2. Pharmacists are an essential part of the medication use process and better integration of e-Prescribing and pharmacy information systems through, at a minimum, one-way complete electronic data interchange should be a focus of further research.
  3. Regulatory and legal uncertainties. Some participants in U.S. studies were unsure whether complete one-way e-Prescribing was permitted under relevant State laws.736,839 e-Prescriptions for controlled substances were not evaluated as a component of the reviewed studies because of the perceived prohibition on the use of e-Prescribing for these drugs under relevant State and/or Federal laws. Prescribers were also concerned that notification by pharmacies of prescription fill status (filled or not filled) could increase their exposure to malpractice claims.839
  4. Low preexisting adoption rate of EMRs and EHRs. Nearly all of the systems evaluated in the United States described the use of prescription writing software limited to generating e-Prescriptions, but without any other clinical record keeping functionality.736,839 These systems generated prescriptions and retrieved pharmacy dispensing histories while requiring providers to concurrently maintain paper-based medical records. Prescribers report deferring adoption of e-Prescribing (prescription writing) systems in favor of complete EMR systems that include e-Prescribing functionality.839 Thus the low rate of EMR adoption in the U.S. likely decreases the rate of e-Prescribing adoption. Addressing barriers to EMR adoption800 may increase the rate of e-Prescribing amongst physicians and other providers.

Summary of Evidence

No reports documenting the use of complete two-way EDI (prescribing) systems were located in the literature search for this report. Evidence from the limited set of one-way e-Prescribing studies was extrapolated to identify possible key facilitators and barriers to completely electronic, two-way e-Prescribing systems. Possible facilitators include monetary or other incentives to providers, a permissive regulatory environment, and the existence of enabling technical standards necessary for e-Prescribing. Barriers identified included the low rate of EMR adoption in the United States, regulatory and legal uncertainties, and inadequate consideration of the effects of e-Prescriptions on pharmacists and pharmacies.

KQ7. What evidence exists regarding the extent of integration of electronic clinical decision support (CDS) in a health IT system for prescribing and dispensing of medications?
To what extent does the use of CDSS in a health IT system for prescribing and dispensing of medications impact the various outcomes of interest including health care process, intermediate and clinical?

Summary of the Findings: All Phases of Medication Management

Seventy-seven RCTs in total were designated as primarily studying CDSSs related to medication management and with integration with other health ITs.397–399,401–405,407,409–416,504–531,533–543,592,609,611–613,616–620,624,630,634,636–638,697–700,771 Full details of the studies are contained in Appendix C, Evidence Tables 1315. These studies involved 4,709 providers and approximately 828,441 patients in total (numbers were not specified in all articles). Patients included were primarily adults, with only two studies addressing issues specific to children. Seven studies addressed seniors exclusively. Currently, AHRQ has contracted with Duke University to prepare an evidence report focused on CDSS due for release in 2011.8

All studies assisted with at least the prescribing (71 percent) or monitoring (29 percent) phases of medication management. Notably, none concentrated solely on the order communication, dispensing, or administering phases of medication management. Reconciliation and education were also not addressed.

The studies were much more likely to focus on process changes than clinical (patient-important) outcomes. Furthermore, many studies did not report directly which outcome was their main endpoint—a fundamental flaw. A total of 36 articles measured changes in process as their main endpoints, 24 of which were deemed to have positive results—meaning that at least 50 percent of the changes in process measured showed that the MMIT improved medication management. Only five of 34 studies measuring clinical outcomes, whether a main endpoint or not, had a statistically significant impact on a clinical outcome. These five RCTs were all published recently (since January 2005), addressed primarily the prescribing and monitoring phases of MMIT, and a variety of disease and drug target groups, usually in an outpatient setting.402,537,541,620,634 Where clinical outcomes were thought to be designated main endpoints, 12 of 16 studies showed no differences in clinical outcomes between intervention and control groups.403,518–520,526,528,624,630,637,697,699,700 No study was able to demonstrate a positive impact on mortality.

Regarding integration of the CDSS, authors used various descriptions of other components of the integration. EMRs, EHRs, and hospital information systems were specified in 41 of the studies, and CPOE was integrated with CDSS in 10 studies, seven of which specified CPOE in addition to the EMR.

Strengths and Limitations

As per our inclusion criteria, all trials used randomization for allocation. However, by applying the Verhagen/AHRQ RCT quality scale,10 the overall quality of methods of the studies was generally only fair at best with a mean quality score of 4.4 out of total possible nine points. One of the most important features to avoid bias, allocation concealment was only described to a minimally acceptable degree by 25 studies. Twenty articles scored six or more and none of the studies scored the maximum nine points. Mean followup of the studies was 9.9 months. Twenty-four studies (31 percent) used a cluster design. This design is prone to bias. Cluster numbers are often small, and therefore, if clusters initially randomized to control group drop out, or participants within the clusters (who are known to be in the intervention or control group) are selected in a biased manner, trial results may not be valid.

Overall, high quality is lacking from RCTs that address CDSS integrated with other types of health IT. Only a small minority of these focus on clinical outcomes—those outcomes that are most important to guide decisions of patients’ providers and policymakers about these interventions. Furthermore, a very small number report improvement in these clinical outcomes.

General Study Characteristics

Of the 77 trials, 46 (60 percent) were rated as impacting primarily the prescribing phase of medication management, 12 (16 percent) aimed primarily at medication monitoring, 15 (19 percent) tried to impact both phases and one addressed administering. Three trials (4 percent) attempted to influence a mix of prescribing, monitoring, order communication, and administering phases of medication management.

The setting for the studies was judged to be ambulatory care in 53 (69 percent), or hospital-based in 19 (25 percent), with a small minority based in long term care (two (3 percent)), or other settings (three (4 percent)) such as community or home. Approximately half (36 or 47 percent) of these studies were identified as associated with academic institutions.

Health care providers were a target of the CDSS in 64 studies and included physicians in most cases where targets were specified. However, many studies did not address the specific type of provider targeted by the intervention. Three studies identified pharmacists as one of the intervention targets and one study targeted nurses specifically. Patients were named as targets of the intervention in 22 studies, 13 of which exclusively targeted patients.

A wide variety of diseases and drugs were studied as the topic of the CDSS. Of the 42 studies where disease targets were mentioned, 19 dealt with vascular disease including risk factors, eight with diabetes, six with asthma, and four with infections, including HIV. Drug topics were evaluated in 42 studies—19 were vascular medications, 13 antibiotics or vaccines, and five addressed multiple medications. The CDSS system was known to be ‘home grown’ in 26 studies, a commercially available product in 14, a hybrid of both in four, and unknown in 33.

Thirty-five CDSS were thought to be integrated with an EMR or EHR system. Fourteen were integrated with CPOE or prescription writing systems, another 17 with a laboratory or imaging system and ten other multiple systems.

Characteristics of each of the CDSS were beyond the scope of this review, so it is unclear whether any signals from these RCTs indicated how a system should be designed, installed, maintained and training supplied, to optimize the chance of success. Similarly, we were not able to critique the suitability of control groups in this systematic review, which were typically described as usual care.


Of the 77 studies, 54 indicated in some way that they had a primary or main outcome and only 16 appeared to have designated a clinical outcome as a main endpoint. Clinical outcomes were defined liberally as any clinical morbidity, mortality, quality of life, adverse event, or clinical surrogate such as improved LDL cholesterol levels. Only eight studies addressed mortality in any way; none had a significant effect.

Overall, only five studies noted a positive change in clinical outcomes.402,537,541,620,634 All were published since early 2005. Four of the five took place in an outpatient setting.537,541,620,634 The studies addressed venous thromboembolism prophylaxis,402 asthma control,634 cholesterol management,541 diabetes care,537 and recommended drugs.620 The mean quality score of these five studies was only 4.8 out of nine. Two studies with the highest methodologic quality (six out of nine) are further described. One evaluated a CDSS which calculated venous thromboembolism risk and recommended venous thromboembolism prophylaxis when the risk was high thus improving their main endpoint of venous thromboembolism rates in a group of inpatients primarily with cancer. The other used a university affiliated managed care plan data to identify gaps in recommended drug therapy and monitoring to recommend drugs to stop or add, or for monitoring to take place. However, this analysis was based on a post-hoc outcome applied to a subgroup of the original participants and the changes in hospitalization are very high given the small change in recommendation use. In summary, we found no consistent impact of CDSS on clinical outcomes, and the quality of the studies is generally inadequate.

In 38 studies, a process endpoint was determined to be a main endpoint. In 26 cases, the process was judged to be positively affected; with improvement in at least 50 percent of the process measures reported. The changes in process measured in these studies generally dealt with reminders about recommended medications or vaccines,403,404,407,410,509,525,530,535,536 dose adjustments,398,412 recommended laboratory monitoring for medications prescribed or chronic disease management,412,504,513,516,612,619,771 ‘inappropriate’ medications avoided,397,413,416,507,508,512,533 and other similar outcomes. Some of the alerts or reminders were based on established guidelines, while others were assessing more locally derived quality measures and standards of care.

Only one of the studies we reviewed scored at least eight out of nine for the AHRQ methods quality assessment. Terrell et al.416 randomized 63 emergency physicians to receive or not receive alerts to disrupt intended prescriptions of nonrecommended medications for seniors to be discharged from the emergency department. The CDSS resulted in a small decrease in the number of visits with a nonrecommended prescription from 3.9 percent to 2.6 percent (95 percent CI 0.34 to 0.89, p = 0.02). No clinical outcomes were measured in this study.

One article measured a composite score in which a shared CDSS to support the primary care of diabetes improved the process of care and some clinical markers of the quality of diabetes care.771 One other study evaluated whether actively or passively displaying context-sensitive links to infrequently accessed educational materials and patient information using an inpatient CPOE would affect access rates to the materials, and found that the active alerts were more effective.638

Notably, the negative effects of the CDSS intervention were virtually never reported. Specifically, only two studies referred to any harm incurred by the study intervention.508,630 This implicates a major publication bias, a result of not requiring studies to measure and report on harm.

In terms of costs, 11 studies reported that they had intended to measure costs or cost-effectiveness. However, no full cost-effectiveness analysis was found as part of the RCT. Separate publications on resource utilization are covered in the KQ1: Effectiveness section on economics outcomes.


In summary, despite it being 34 years since the first RCT619 in 1976, in this important area of health IT research, little high quality evidence shows a consistently positive effect of CDSS on clinical outcomes. Implementers, developers, and funders of MMIT applications need to continue to produce and rely on the best possible research evaluating outcomes important to people and institutions. The informatics world can strengthen their abilities to determine value for money in MMIT projects by obtaining input during planning for research projects from health technology appraisal methods and those who have expertise in clinical care, research methods, informatics, statistics, and stakeholders who will be affected by the MMIT system.846