Results

Literature Search Results

The flow of articles through the literature search and screening process is depicted in Figure 2. We identified 15,176 citations from all sources (after removing duplicates). After applying inclusion/exclusion criteria at the title-and-abstract level, 1407 full-text articles were retrieved and screened. Of these, 1084 articles were excluded at full-text review, with 323 articles remaining for data abstraction. Of these, 323 articles were abstracted for KQ 1 (representing 311 unique studies), and 160 articles (representing 148 unique studies) for KQs 24. Appendix E provides a complete listing of articles excluded at the full-text stage, with reasons for exclusion.

The flow of articles through the literature search and screening process is depicted in Figure 2. We identified 15,176 citations from all sources (after removing duplicates). After applying inclusion/exclusion criteria at the title-and-abstract level, 1407 full-text articles were retrieved and screened. Of these, 1084 articles were excluded at full-text review, with 323 articles remaining for data abstraction. Of these, 323 articles were abstracted for KQ 1 (representing 311 unique studies), and 160 articles (representing 148 unique studies) for KQs 2–4.

Figure 2

Literature search flow. Abbreviations: CDSS = clinical decision support system, KMS = knowledge management system, KQ = key question, RCT = randomized controlled trial

Key Question 1

KQ 1: What evidence-based study designs have been used to determine the clinical effectiveness of electronic knowledge management and CDSS?

Key Points

  • Clinical effectiveness of CDSSs/KMSs as characterized by demonstrating an impact on a clinical outcome was most frequently assessed using a quasi-experimental study design (43 out of the 311 included studies).
  • RCTs were the next most frequent study design used to assess clinical outcomes of CDSSs/KMSs (29 out of 311 included studies), followed by observational studies (17 out of 311 included studies).
  • Our analysis suggests that more RCTs measuring clinical outcomes are needed to evaluate the comparative effectiveness of CDSSs/KMSs.

Detailed Analysis

The objective assessment of the clinical effectiveness of a CDSS/KMS intervention is important in understanding the value of that intervention in a clinical setting. Selection of appropriate study design is critical for the proper evaluation of clinical performance in a system.18 KQ 1 examined past use of different study designs in the existing CDSS/KMS evidence base in the evaluation of clinical effectiveness. Clinical effectiveness is simply defined as how well a particular intervention produces optimum processes and outcomes for patients. New CDSS/KMS interventions are invariably evaluated through a variety of direct, process-oriented measures that describe compliance with and acceptance of the system, but the clinical effectiveness of a CDSS/KMS is best evaluated with the direct measurement of patient-centric clinical outcomes following implementation.

While the responses to KQs 24 considered only those CDSS/KMS implementation studies that employed an RCT design, KQ 1 examined the relative prevalence of significant outcome measures not only in RCT studies but also in studies employing other evaluation designs (quasi-experimental, observational). KQ 1 thus presents a horizon scan of the current state of CDSS/KMS implementation studies in order to provide context for KQs 24 and to inform discussion of CDSS/KMS evaluation moving forward.

Types of study designs. There were 311 studies that met basic inclusion criteria. We categorized these studies as 1 of 12 study designs falling into 3 basic study types: RCT, quasi-experimental, and observational. Table 4 describes the selected study designs and the number of included studies for each design.

Table 4. Types of evaluation studies included in this review.

Table 4

Types of evaluation studies included in this review.

Categories of outcomes. To evaluate the use of specific study designs on the evaluation of CDSS/KMS clinical effectiveness, we abstracted outcome data from all included studies, compiling the relative prevalence of six key outcome categories in each of the three study designs. We considered direct measurement of clinical outcomes the means of measuring clinical effectiveness while evaluating KQ 1. Table 5 summarizes the outcome categories abstracted from the included studies and gives specific examples. Further details on the relative prevalence of outcome categories by study design are in Table G-1 of Appendix G.

Table 5. Outcome categories abstracted.

Table 5

Outcome categories abstracted.

Impact of study type on outcomes examined. Table 6 shows the prevalence of different outcome categories as they relate to basic study design. The total number of studies containing a particular outcome measure is given, followed by the percentage of studies containing the outcome measure over the total number of studies within the given study design. All three study designs reported health care process measures most frequently, with 86 percent of all RCTs, 75 percent of all quasi-experimental studies, and 69 percent of all observational studies including at least one process-level measure in their evaluation. The most frequent process measures reported in all three categories were outcomes that demonstrated compliance with CDSS/KMS-provided recommendations (Table G-2 in Appendix G). Other direct measures, such as the use of and satisfaction with CDSS/KMS by health care providers, were also frequently reported, especially in RCTs, with 35 percent of all RCTs containing outcomes related to CDSS/KMS use and implementation. Other outcomes related to CDSS/KMS use, including patient satisfaction (relationship-centered outcomes), efficiency (economic and workload outcomes), and patient well-being (clinical outcomes), were reported less frequently overall.

Table 6. Number of studies containing outcome measures by study type.

Table 6

Number of studies containing outcome measures by study type.

Outcomes in RCTs. In RCT studies, health care process measures were reported most frequently (Table 6), with compliance with CDSS/KMS-recommended treatment the most commonly reported specific outcome (reported in 67 RCT studies). Health care provider use and implementation was the second most commonly reported outcome category for RCT studies, with health care provider acceptance the most frequently occurring specific outcome in that category (reported in 24 RCT studies). Clinical outcomes were reported moderately frequently in RCT studies, with morbidity the most commonly reported clinical outcome. A complete breakdown of outcomes by specific study type can be found in Table G-2 of Appendix G.

Outcomes in non-RCTs. In non-RCT studies (quasi-experimental and observational), health care process measures were also the most frequently reported outcome type. Clinical outcomes were the second most commonly reported outcome for non-RCT studies, with mortality and morbidity being the most commonly reported clinical outcomes (Table G-2 of Appendix G).

Clinical outcomes. In Table 7, we further categorized the proportion of studies that measured clinical effectiveness into specific study type. This analysis demonstrates that 20 percent of all RCTs included clinical outcomes as at least one of their reported outcome measures, compared with 36 percent of quasi-experimental and 40 percent of observational studies including clinical outcomes.

Table 7. Proportion of specific study design containing clinical outcomes.

Table 7

Proportion of specific study design containing clinical outcomes.

Outcomes related to successful CDSS/KMS implementation. According to Davis’ Technology Acceptance Model,19 users accept and use technology (such as a CDSS/KMS) based on two key factors: perceived usefulness and perceived ease-of-use. That is, a recommendation is likely to be successfully acted upon if health care providers perceive the CDSS/KMS intervention as useful in aiding critical decisionmaking at the point of care. Health care providers appear most comfortable considering recommendations when CDSS/KMS interventions provide adequate information toward decisive action in a timely manner. This finding seems to be consistent with studies reporting health care provider acceptance and satisfaction of using. Such studies are also likely to report health care process measures and/or clinical outcomes. In our studies, 19 articles (19%) reporting health care provider use and implementation outcomes also reported clinical outcomes. Similarly, 69 articles (70%) reporting health care provider use and implementation outcomes also reported health care process measures.

Discussion and Future Research

In the current body of literature, most CDSS/KMS implementation studies did not examine clinical outcomes—instead focusing on the more immediately measurable process-oriented measures. Of the included studies that examined clinical outcomes, very few were RCT studies. These trends can likely be attributed to the relative difficulty of implementing RCT studies in real clinical settings as well as the logistical issues involved in measuring the indirect clinical impact of CDSS/KMS interventions.

Challenges in conducting RCT studies in real clinical settings. One of the challenges in conducting RCT studies in real clinical settings is the enforcement of true randomization without allowing contamination. Clinicians frequently consult with one another about treatment options or medications, especially when they change their shift. Also, clinicians may be tempted to share their experiences of using CDSSs/KMSs with their colleagues and inadvertently influence their attitude toward the use of CDSSs/KMSs.18 Therefore, avoiding contamination among clinicians assigned to CDSS/KMS interventions within the same ward or hospital setting is usually difficult to achieve. We found 50 of the included RCTs (34%) conducted cluster RCTs, where groups of patients and clinicians, rather than individuals, are randomized in order to protect against contamination across trial groups.

Large randomized trials related to the use of CDSSs/KMSs tended to occur most often in well-established institutions that relied upon locally developed information systems such as Brigham and Women’s Hospital/Partners Health Care/Massachusetts General Hospital in Boston, Regenstrief Institute in Indianapolis, and LDS Hospital/Intermountain Healthcare in Utah. This trend may be related to factors common at these research-intensive institutions, such as the availability of well-defined electronic medical records system, infrastructure supporting the implementation of a CDSS/KMS to selected groups, and a clinician culture that supports the exploration of CDSS/KMS adoption as part of their clinical practice. These factors may well explain the higher adoption rate of CDSSs/KMSs among these institutions, which subsequently provided them with the opportunity to conduct more randomized trials to evaluate the clinical impact of CDSS/KMS interventions.

Challenges in measuring clinical outcomes. All three study types reported a much higher prevalence of health care process measures (outcomes directly related to the implementation of, and compliance to, the CDSS/KMS intervention being evaluated) than of clinical outcomes (patient-centric outcomes often separated from the actual CDSS/KMS temporally and practically). This difference is likely due to the fact that, regardless of design, process measures (e.g., compliance with CDSS/KMS-recommended drug dosage) are generally easier and faster to measure and evaluate than clinical outcomes (e.g., length of stay, morbidity). The impact of CDSSs/KMSs on clinical outcomes related to the CDSS/KMS implementation must often necessarily occur for several days to several months after the initial implementation, and measuring such impacts often requires costly and cumbersome followup, delaying evaluation of the CDSS/KMS. In situations where the health care process measures and clinical outcomes are closely aligned (e.g., a CDSS that provides drug-dosage calculations based on patient parameters), measuring the process may serve as an acceptable surrogate for a clinical outcome. In cases where the CDSS/KMS process is not closely related to clinical effectiveness (e.g., systems that recommend treatment plans from evidence-based standards), clinical outcomes will need to be measured directly to understand the true effects of CDSSs/KMSs.

Given the challenges inherent both in implementing RCTs and in measuring the clinical impact of interventions in real clinical settings, the relative lack of studies that reported on RCTs assessing a clinical outcome is not surprising. Although studies that both follow RCT design and directly measure patient-centered clinical outcomes would be ideal, such studies are clearly not always feasible—logistically or economically. Whether studies should dedicate presumably limited resources either to adhering to RCT design or to measuring clinical outcomes depends on the nature of the CDSS/KMS being evaluated. If the CDSS/KMS itself is closely related to clinical outcomes (as discussed above), then process-oriented measures are likely sufficient, and resources should be dedicated to the execution of RCT studies. If, however, the CDSS/KMS process is linked only indirectly to clinical effectiveness, then health care process measures will not be sufficient. In these cases, measuring clinical outcomes directly becomes necessary to evaluate clinical effectiveness. When limited resources will necessarily be devoted to the time and effort required to measure clinical outcomes, quasi-experimental and observational studies can be effective choices for study design, provided they are conducted as rigorously as possible.

Key Question 2

KQ 2: What contextual factors/features influence the effectiveness or success of electronic knowledge management and CDSSs?

Key Points

  • A meta-analysis of included studies confirmed the three key factors/features identified in the review by Kawamoto et al. (2005)9 that were associated with a successful CDSS that improved clinical practice, although we were unable to distinguish the impact of a specific factor/feature. These factors were significant across all three endpoints assessed: (1) adherence to performing preventive care, (2) adherence to performing a clinical study, and (3) adherence to prescribing a treatment. The three features are:
    • Automatic provision of decision support as part of clinician workflow
    • Provision of decision support at time and location of decisionmaking
    • Provision of a recommendation, not just an assessment
  • The meta-analysis also identified six additional factors/features that were correlated with the success of a CDSS across all three endpoints:
    • Integration with charting or order entry system to support workflow integration
    • Promotion of action rather than inaction
    • No need for additional clinician data entry
    • Justification of decision support via provision of research evidence
    • Local user involvement in the development process
    • Provision of decision support results to patients as well as providers
  • Additionally, one factor/feature was found to correlate with a successful CDSS across two of the three endpoints evaluated:
    • Justification of decision support via provision of reasoning
  • Four factors/features were significant for only one endpoint:
    • Request documentation of the reason for not following CDSS recommendations (adherence to prescribing a treatment, only four studies in the group)
    • Recommendations executed by noting agreement (adherence to performing a clinical study, only two studies in the group)
    • A CDSS accompanied by periodic performance feedback (adherence to performing a clinical study, only three studies in the group)
    • A CDSS accompanied by conventional education (adherence to performing a clinical study, nine studies in the group)

Detailed Analysis

This section of the evidence report examines the factors/features that influence the effectiveness or success of CDSSs/KMSs. We present findings from the literature search on the generalized factors/features of successful CDSSs and then the factors/features of CDSSs according to outcomes.

Within this body of evidence, we examined the inclusion of 14 factors/features in electronic CDSSs that were identified from a previous review9 and from suggestions from the TEP that were viewed as potentially important in determining a CDSS’s success in improving clinical practice. To further assess the impact of various factors/features on the success of a CDSS, we used meta-analysis to analyze the 14 most common features across the three outcomes for which we had the most studies—adherence to performing a preventive care service, adherence to performing a clinical study, and adherence to prescribing a treatment. The majority of the 148 included studies described CDSSs that included the following five factors/features:

  1. Provision of decision support at the time and location of decisionmaking (n = 125; 84.5%)
  2. Automatic provision of decision support as part of clinician workflow (n = 116; 78.4%)
  3. Provision of a recommendation, not just an assessment (n = 109; 73.6%)
  4. Integration with charting or order entry to support workflow integration (n = 96; 64.9%)
  5. No need for additional clinician data entry (n = 84; 56.8%)

Of the 14 electronic factors/features that we identified, three had been shown in a previous review to be strongly associated with improving clinical practice: (1) automatic provision of decision support as part of clinician workflow, (2) provision of decision support at time and location of decisionmaking, and (3) provision of a recommendation, not just an assessment. From the meta-analysis conducted for this review, we identified six additional factors/features that correlated with a successful CDSS implementation: (4) the incorporation with charting or order entry system to support workflow integration, (5) the promotion of action rather than inaction, (6) no need for additional clinician data entry, (7) justification of decision support via provision of research evidence, (8) local user involvement in the development process, and (9) provision of decision support results to patients as well as providers. The local user involvement factor/feature was found to be present in 50 locally developed systems and 4 commercially developed systems. Thus, while this feature was present across both types of systems, it was proportionally more frequently associated with locally developed systems.

We observed that two studies (1.4%) included all nine of those factors. One hundred forty-six (98.6%) of the 148 studies included some combination of the 9 factors—8 studies (5.4%) included 8 factors; 19 studies (12.8%) included 7 factors; 29 studies (19.6%) included 6 factors; 29 studies (19.6%) included 5 factors; 23 studies (15.5%) included 4 factors; 17 studies (11.5%) included 3 factors; 11 studies (7.4%) included 2 factors; and 10 studies (6.8%) included 1 factor.

The following section presents findings from the literature search on three key categories of outcomes (clinical, health care process measures, health care provider use) related to the effectiveness or success of CDSSs/KMSs. Within each category, we present general observations of the factors/features that the majority of systems possessed, followed by an examination of the factors/features of the CDSSs for each outcome.

Clinical Outcomes

General observations. The six studies that evaluated the effectiveness or success of CDSSs/KMSs on clinical outcomes and reported a significant reduction in length of stay, morbidity, mortality, and adverse events consistently had two of the previously identified key factors/features identified in the Kawamoto et al. (2005)9 review:

  1. Automatic provision of decision support as part of clinician workflow2024
  2. Provision of a recommendation, not just an assessment20,21,2327

Factors/features of the six studies that evaluated CDSSs on clinical outcomes across settings. Three studies (50%) evaluated in the academic setting consistently had two of the previously identified key factors/features (automatic provision of decision support as part of clinician workflow and provision of decision support at time and location of decisionmaking) and one newly identified factor/feature: local user involvement in the development process.2123 Four studies (66.7%) conducted in the ambulatory setting consistently had two of the previously identified key factors/features (automatic provision of decision support as part of clinician workflow and provision of decision support at time and location of decisionmaking).20,22,24,26,27 Two studies (33.3%) evaluated in the hospital setting consistently had the three previously identified key factors/features and three newly identified factors/features: integration with charting or order entry system, promotion of action rather than inaction, and local user involvement in development process.21,23 All CDSS interventions (100%) implemented in locally developed systems consistently had two of the previously identified key factors/features: automatic provision of decision support as part of clinician workflow and provision of decision support at time and location of decisionmaking.

Length of stay. We identified 6 of the 148 eligible studies (4.1%) that evaluated inpatient or emergency department length of stay as an outcome of CDSS/KMS effectiveness or success. These studies are summarized in Table H-1 of Appendix H.

We conducted a meta-analysis that focused on CDSS studies in which at least one outcome was related to length of stay. Length of stay was defined over a fixed time interval and was converted to a fraction (ratio) by dividing by the length of the time interval. This creates a unitless ratio as described by Kleinbaum, Kupper, and Morgenstern.28 The ratio of two such measures is similar to a relative risk, which is the ratio of two proportions (which are also unitless). Of the six studies, five (83.3%) included data with a common dichotomous endpoint and were included in the meta-analysis.23,26,27,2931 All of the studies had one of the previously identified key factors/features associated with CDSS success (provision of a recommendation, not just an assessment). Sixty-seven percent of the studies had two of the previously identified key factors/features (automatic provision of decision support and provision of a recommendation, not just an assessment) and one newly identified factor/feature: local user involvement in the development process.

One of the six studies found a significant reduction in length of stay.23 Paul et al. (2006) evaluated a standalone system that focused on decreasing inappropriate antimicrobial use by recommending the three “best” antibiotic regimens in 2326 patients over 7 months and reported that the intervention group had significantly lower length of stay than the control group (RR 0.9082; 95% CI 0.8392 to 0.9828). That system included the following factors/features: integration with charting or other entry system; automatic provision of decision support; provision of decision support at time and location of decisionmaking; provision of a recommendation, not just an assessment; promotion of action rather than inaction; and local user involvement in the development process. Another study that almost reached significance in the meta-analysis26,27 assessed an intervention that provided clinicians in 64 community clinics and 7412 patients with recommendations to improve appropriate guideline-based diabetes testing and found that intervention subjects had shorter inpatient durations than control subjects (0.99 days versus 1.1; P = 0.01). The intervention included three factors/features: provision of a recommendation, not just an assessment; local user involvement in the development process; and CDSS accompanied by periodic performance feedback.

Morbidity. We identified 22 of the 148 eligible studies (14.9%) that evaluated morbidity as an outcome of CDSS/KMS effectiveness or success. These studies are summarized in Table H-2 of Appendix H.

We conducted a meta-analysis that focused on CDSS studies in which at least one outcome was related to morbidity. Of the 22 studies, 16 (72.7%) included data with a common dichotomous endpoint and were included in the meta-analysis.2024,26,27,3243 The studies consistently had two of the previously identified key factors/features: automatic provision of decision support as part of clinician workflow and provision of a recommendation, not just an assessment.

Three of the 22 studies reported a significant reduction in morbidity.21,22,26,27 Kucher et al. (2005)21 evaluated alerts that identified patients at risk for developing deep vein thrombosis (DVT) among 2506 high-risk hospitalized patients over 40 months (RR 0.60; 95% CI 0.43 to 0.84). McDonald et al. (1984)22 investigated reminders regarding preventive care services to improve provider adherence in 12,467 patients for 2 years and found that intervention patients had significantly fewer hospitalizations and emergency department visits than control patients (RR 0.69; 95% CI 0.52 to 0.91). Khan et al. (2010)26,27 evaluated recommendations to improve appropriate guideline-based diabetes testing and reported that intervention patients had fewer hospitalizations (0.17 admissions versus 0.20; P = 0.01) and emergency department visits (0.27 visits versus 0.36; P < 0.0001) than control patients (RR 0.75; 95% CI 0.70 to 0.80). Those CDSSs included the following factors/features:

  • Two included automatic provision of decision support as part of clinician workflow21,22
  • Two included provision of decision support at time and location of decisionmaking21,22
  • Two included provision of a recommendation, not just an assessment21,26,27
  • One included integration with charting or order entry system21
  • One included no need for additional data entry21
  • One included promotion of action rather than inaction21
  • Two included justification of decision support via provision of reasoning21,22
  • Two included justification of decision support via research evidence21,22
  • Two included local user involvement in development process21,22
  • One included provision of decision support results to patients as well as providers26,27
  • One included a CDSS accompanied by periodic performance feedback26,27

Mortality. We identified 7 of the 148 eligible studies (4.7%) that evaluated mortality as an outcome of CDSS/KMS effectiveness or success. These studies are summarized in Table H-3 of Appendix H.

We conducted a meta-analysis that focused on CDSS studies in which at least one outcome was related to mortality. Of the 7 studies, 6 (85.7%) included data with a common dichotomous endpoint and were included in the meta-analysis.20,23,24,30,44 The studies consistently had two of the previously identified factors/features (automatic provision of decision support as part of clinician workflow and provision of a recommendation, not just an assessment) and three newly identified factors/features: integration with charting or order entry system; no need for additional data entry; and promotion of action rather than inaction.

Two of the six studies reported a significant reduction in mortality.20,24 Ansari et al. (2003)20 assessed treatment reminders to improve the appropriate use of beta blockers for patients with congestive heart failure in 169 patients for 1 year and found a significant reduction in mortality (RR 0.12; 95% CI 0.016 to 0.87). Roumie et al. (2006)24 evaluated guideline-based recommendations for patients with uncontrolled hypertension in 1341 patients for 6 months and reported that the intervention groups had a significantly lower mortality rate than the control group (RR 0.24; 95% CI 0.06 to 0.88). Those CDSSs included the following factors/features:

  • Two included automatic provision of decision support as part of clinician workflow20,24
  • One included provision of decision support at time and location of decisionmaking20
  • Two included provision of a recommendation, not just an assessment20,24
  • Two included integration with charting or order entry system20,24
  • Two included no need for additional data entry20,24
  • One included promotion of action rather than inaction20
  • One included justification of decision support via provision of research evidence24
  • One included a CDSS accompanied by conventional education20

Adverse events. We identified 5 of the 148 eligible studies (3.4%) that evaluated adverse events as an outcome of CDSS/KMS effectiveness or success. These studies are summarized in Table H-4 of Appendix H.

We conducted a meta-analysis that focused on CDSS studies in which at least one outcome was related to adverse events. All of the studies (100%) included data with a common dichotomous endpoint and were included in the meta-analysis.30,36,37,4446 The studies consistently had the three previously identified factors/features and two newly identified factors/features: integration with charting or order entry system and local user involvement in the development process.30,36,4446

None of the six studies found a significant reduction in adverse events. Of the studies that reported adverse events data, McGregor et al. (2006)30 evaluated alerts in a commercially developed system to detect potentially inappropriate antimicrobial therapy in 4507 patients for 12 weeks and reported that fewer intervention patients experienced diarrhea as a side effect of antimicrobial therapy (5.7% versus 6.6%; P = 0.21). That CDSS included the following factors/features: integration with charting or other entry system; automatic provision of decision support; no need for additional data entry; provision of decision support at time and location of decisionmaking; provision of a recommendation, not just an assessment; promotion of action rather than inaction; and local user involvement in the development process.

Health Care Process Measures

General observations. Fifty-two studies that evaluated the effectiveness or success of CDSSs on health care process measures and reported a significant improvement in the appropriate ordering/completion of preventive care services, clinical studies, and treatment consistently had the nine key factors/features correlated with a successful CDSS, three previously reported in 20059 and six identified through meta-analysis for the current report.

Previously identified factors/features and the relevant included studies were:

  1. Automatic provision of decision support as part of clinician workflow21,23,2932,35,38,4783
  2. Provision of decision support at time and location of decisionmaking21,23,2932,35,38,4760,6264,6774,7691
  3. Provision of a recommendation, not just an assessment21,23,2932,35,38,41,4754,56,57,59,6168,70,71,74,75,77,79,80,8284,86,87

Newly identified factors/features and the relevant included studies were:

  1. Integration with charting or order entry system21,23,29,30,35,4754,5659,6266,6870,7274,7683,88
  2. No need for additional data entry21,2931,35,38,48,5054,5660,6266,6870,7274,7679,8184
  3. Promotion of action rather than inaction 21,23,29,30,4749,51,52,56,57,59,60,62,6466,68,70,8286,88
  4. Justification of decision support via provision of research evidence 21,29,49,50,52,61,79,83,8587
  5. Local user involvement in the development process 21,23,2932,35,4953,59,60,65,66,74,81,8587,89
  6. Provision of decision support results to patients as well as providers 32,52,56,57,65,66,84

Factors/features of the 52 studies that evaluated CDSSs on health care process measures across settings. Twenty-four studies (46.2%) evaluated in the academic setting consistently had the three key factors/features previously associated with a successful CDSS and two newly identified key factors/features (integration with charting or order entry system to support workflow and no need for additional clinician data entry).21,23,29,30,35,48,51,53,6066,69,72,75,76,82,83,85,87,90,91 Sixteen studies (30.8%) evaluated in the community setting consistently had the three previously identified key factors/features.32,41,47,52,5457,59,67,68,74,78,80,81,84,86 Four studies (7.7%) evaluated in both academic and community settings consistently had the three previously identified key factors/features and two newly identified key factors/features (integration with charting or order entry system to support workflow and promotion of action rather than inaction).49,58,71,88 Four studies (7.7%) evaluated in the VA setting consistently had the three previously identified key factors/features and two newly identified key factors/features (integration with charting or order entry system to support workflow and no need for additional clinician data entry)38,50,70,79 Thirty-seven studies (71.2%) were conducted in the ambulatory setting and consistently had the three previously identified key factors/features.32,41,47,49,50,52,5463,65,66,6881,8489 Nine studies (17.3%) conducted in the hospital setting consistently had the three previously identified key factors/features and three newly identified key factors/features (integration with charting or order entry system, no need for additional data entry, and promotion of action rather than inaction).21,23,29,30,35,48,51,64,82 Four studies (7.7%) conducted in the emergency department consistently had the three previously identified key factors/features 31,67,83,90,91 Thirty-five CDSS interventions (67.3%) implemented in locally developed systems consistently had the three previously identified key factors/features.21,23,29,31,32,35,38,41,4852,6065,67,70,72,73,75,76,7988 Eleven CDSS interventions (21.2%) implemented in commercially developed systems consistently had two of the previously identified key factors/features (automatic provision of decision support as part of clinician workflow and provision of decision support at time and location of decisionmaking) and two newly identified key factors/features (integration with charting or order entry system and no need for additional data entry).30,47,53,5659,68,74,77,78,89

Preventive care adherence. We identified 43 of the 148 eligible studies (29.1%) that evaluated adherence to order/complete a preventive care service as an outcome of CDSS effectiveness or success. These studies are summarized in Table H-5 of Appendix H.

We conducted a meta-analysis that focused on CDSS studies in which at least one outcome was related to ordering or completing preventive care services. Of the 43 studies, 25 included data with a common dichotomous endpoint and were included in the meta-analysis.4,21,40,41,47,50,51,5557,60,63,68,71,75,76,84,85,89,9298 Across the studies, we examined the specific factors/features of each CDSS, and those odds ratios were combined using the DerSimonian and Laird random effects model.17 The odds ratios listed by factor indicate the summary odds ratios of studies that included the specific factor/feature as compared to those that did not. Findings from this analysis are listed in Table 8. (Note that we attempted to perform a meta-regression on several of the endpoints. However, studies tended to have similar factors present, creating a high degree of correlation. As a result, the random effects meta-regression models failed to converge.)

Table 8. Random effects estimates of the odds ratio for preventive care adherence.

Table 8

Random effects estimates of the odds ratio for preventive care adherence.

This analysis confirmed that the three previously identified key factors/features critical for CDSS success had a statistically significant impact on promoting adherence to preventive care outcomes: automatic provision of decision support as part of clinician workflow (OR 1.45; 95% CI 1.28 to 1.64), provision of decision support at time and location of decisionmaking (OR 1.35; 95% CI 1.20 to 1.52), and provision of a recommendation, not just an assessment (OR 1.50; 95% CI 1.30 to 1.74). The analysis also supported the six newly identified factors/features universally associated with CDSS success: integration with charting or order entry system to support workflow integration (OR 1.47; 95% CI 1.21 to 1.77), no need for additional clinician data entry (OR 1.43; 95% CI 1.22 to 1.69), promotion of action rather than inaction (OR 1.28; 95% CI 1.09 to 1.50, justification of decision support via provision of research evidence (OR 1.60; 95% CI 1.04 to 2.46), local user involvement in development process (OR 1.45; 95% CI 1.23 to 1.73), and provision of decision support results to patients as well as providers (OR 1.18; 95% CI 1.02 to 1.37).

Finally, this analysis discovered one new factor/feature that also was associated with a successful CDSS: justification of decision support via provision reasoning (OR 1.51; 95% CI 1.22 to 1.87). Unfortunately, because many of the studies included more than one factor/feature, and because the studies did not specifically evaluate whether the systems with and without an individual factor differed in terms of their impact on the outcome of interest, it is difficult to determine the importance of individual factors/features.

Fifteen studies reported a significant improvement in preventive care adherence, and those CDSSs included the following factors/features:

  • Eleven included automatic provision of decision support as part of clinician workflow21,47,50,51,55,60,63,68,71,75,76
  • Thirteen included provision of decision support at time and location of decisionmaking 21,47,50,51,55,60,63,68,71,76,84,85,89
  • Ten included provision of a recommendation, not just an assessment21,41,47,50,51,63,68,71,75,84
  • Seven included integration with charting or order entry system21,47,50,51,63,68,76
  • Eight included no need for additional data entry21,50,51,60,63,68,76,84
  • One included request documentation of the reason for not following the CDSS recommendations60
  • Three included recommendations executed by noting agreement51,60,68
  • Seven included promotion of action rather than inaction21,47,51,60,68,84,85
  • Six included justification of decision support via provision of reasoning21,50,51,60,68,85
  • Three included justification of decision support via provision of research evidence21,50,85
  • Six included local user involvement in development process21,50,51,60,85,89
  • One included provision of decision support results to patients as well as providers84
  • Two included a CDSS accompanied by conventional education50,71

Clinical study adherence. We identified 29 of the 148 eligible studies (19.6%) that evaluated adherence to order/complete a clinical study as an outcome of CDSS effectiveness or success. These studies are summarized in Table H-6 of Appendix H.

We conducted a meta-analysis that focused on CDSS studies in which at least one outcome was related to ordering or completing clinical studies. Of the 29 studies, 20 included data with a common dichotomous endpoint and were included in the meta-analysis.26,27,31,39,48,49,61,62,6567,69,70,77,78,87,99103 Across the studies, we examined the specific factors/features of each CDSS, and those odds ratios were combined using the DerSimonian and Laird random effects model.17 The odds ratios listed by factor indicate the summary odds ratios of studies that included the specific factor/feature as compared to those that did not. Findings from this analysis are listed in Table 9.

Table 9. Random effects estimates of the odds ratio for clinical study adherence.

Table 9

Random effects estimates of the odds ratio for clinical study adherence.

This analysis confirmed that CDSSs that included the three previously identified key factors/features that are critical for CDSS success had a statistically significant impact on clinical study adherence outcomes: automatic provision of decision support as part of clinician workflow (OR 1.85; 95% CI 1.52 to 2.25), provision of decision support at time and location of decisionmaking (OR 1.78; 95% CI 1.46 to 2.17), and provision of a recommendation, not just an assessment (OR 2.01; 95% CI 1.63 to 2.48). The analysis also supported the six newly identified factors/features universally associated with CDSS success: integration with charting or order entry system to support workflow integration (OR 1.56; 95% CI 1.29 to 1.87), no need for additional data entry (OR 1.58; 95% CI 1.31 to 1.89), promotion of action rather than inaction (OR 1.52; 95% CI 1.23 to 1.87), justification of decision support via provision of research evidence (OR 2.93; 95% CI 1.40 to 6.12), local user involvement in development process (OR 1.41; 95% CI 1.18 to 1.70), and provision of decision support results to patients as well as providers (OR 1.41; 95% CI 1.26 to 1.58).

Finally, this analysis discovered three new factors/features that were also associated with a successful CDSS: recommendations executed by noting agreement (OR 1.43; 95% CI 1.15 to 1.78), CDSSs accompanied by periodic performance feedback (OR 1.98; 95% CI 1.30 to 3.01), and CDSSs accompanied by conventional education (OR 1.39; 95% CI 1.13 to 1.71). Unfortunately, because many of the studies included more than one factor/feature and because the studies did not specifically evaluate whether the systems with and without an individual factor differed in terms of their impact on the outcome of interest, it is difficult to determine the importance of individual factors/features.

Thirteen studies reported a significant improvement in clinical study adherence, and those CDSS interventions included the following factors/features:

  • Twelve included automatic provision of decision support as part of clinician workflow31,48,49,61,62,6567,69,70,77,78
  • Ten included provision of decision support at time and location of decisionmaking31,48,49,62,67,69,70,77,78,87
  • Eleven included provision of a recommendation, not just an assessment31,48,49,61,62,6567,70,77,87
  • Nine included integration with charting or order entry system48,49,62,65,66,69,70,77,78
  • Nine included no need for additional data entry31,48,62,65,66,69,70,77,78
  • One included request documentation of the reason for not following the CDSS recommendations48
  • Two included recommendations executed by noting agreement65,66
  • Six included promotion of action rather than inaction48,49,62,65,66,70
  • Two included justification of decision support via provision of reasoning49,70
  • Three included justification of decision support via provision of research evidence49,61,87
  • Five included local user involvement in development process31,49,65,66,87
  • Two included provision of decision support results to patients as well as providers65,66
  • Two included CDSS accompanied by periodic performance feedback70,87
  • Five included CDSS accompanied by conventional education49,67,70,77,78

Treatment adherence. We identified 67 of the 148 eligible studies (45.3%) that evaluated treatment adherence as an outcome of CDSS effectiveness or success. These studies are summarized in Table H-7 of Appendix H.

We conducted a meta-analysis that focused on CDSS studies in which at least one outcome was related to ordering treatments or prescribing therapies. Of the 67 studies, 46 studies included data with a common dichotomous endpoint and were included in the meta-analysis.20,23,24,29,30,32,35,3841,47,49,5254,5659,61,64,7274,77,7983,86,88,9092,104114 Across the studies, we examined the specific factors/features of each CDSS, and those odds ratios were combined using the DerSimonian and Laird random effects model.17 The odds ratios listed by factor indicate the summary odds ratios of studies that included the specific factor/feature as compared to those that did not. Findings from this analysis are listed in Table 10.

Table 10. Random effects estimates of the odds ratio for treatment adherence.

Table 10

Random effects estimates of the odds ratio for treatment adherence.

This analysis confirmed that CDSSs that include the three previously identified key factors/features critical for CDSS success had a statistically significant impact on treatment adherence outcomes: automatic provision of decision support as part of clinician workflow (OR 1.60; 95% CI 1.34 to 1.90), provision of decision support at time and location of decisionmaking (OR 1.75; 95% CI 1.48 to 2.07), and provision of a recommendation, not just an assessment (OR 1.61; 95% CI 1.34 to 1.93). The analysis also supported the six newly identified factors/features universally associated with CDSS success: integration with charting or order entry system to support workflow integration (OR 1.67; 95% CI 1.40 to 2.00), no need for additional data entry (OR 1.78; 95% CI 1.45 to 2.18), promotion of action rather than inaction (OR 1.71; 95% CI 1.35 to 2.16), justification of decision support via provision of research evidence (1.50; 95% CI 1.06 to 2.14), local user involvement in development process (OR 1.95; 95% CI 1.42 to 2.66), and provision of decision support results to patients as well as providers (1.97; 95% CI 1.20 to 3.21).

Finally, this analysis also identified two factors/features that were significant, namely: request documentation of the reason for not following CDSS recommendations (2.05; 95% CI 1.08 to 3.89) and justification of decision support via provision reasoning (OR 1.69; 95% CI 1.21 to 2.35). Unfortunately, because many of the studies included more than one factor/feature and because the studies did not specifically evaluate whether the systems with and without an individual factor differed in terms of their impact on the outcome of interest, it is difficult to determine the importance of individual factors/features.

Twenty-six studies reported a significant improvement in treatment adherence, and those CDSSs included the following factors/features:

  • Twenty-two included automatic provision of decision support as part of clinician workflow23,29,30,32,35,38,49,5254,5659,64,7274,7983
  • Twenty-five included provision of decision support at time and location of decisionmaking23,29,30,32,35,38,49,5254,5659,64,7274,7983,86,88,90,91
  • Twenty included provision of a recommendation, not just an assessment23,29,30,32,35,38,41,49,5254,56,57,59,64,74,79,80,82,83,86
  • Twenty-one included integration with charting or order entry system23,29,30,35,49,5254,5659,64,7274,7983,88
  • Eighteen included no need for additional data entry29,30,35,38,5254,5659,64,7274,79,8183
  • Three included request documentation of the reason for not following the CDSS recommendations79,81,86
  • Two included recommendations executed by noting agreement59,72
  • Twelve included promotion of action rather than inaction23,29,30,49,52,56,57,59,64,82,83,86,88
  • Seven included justification of decision support via provision of reasoning29,49,82,83,86,88,90,91
  • Six included justification of decision support via provision of research evidence29,49,52,79,83,86
  • Twelve included local user involvement in the development process23,29,30,32,35,49,52,53,59,74,81,86
  • Three included provision of decision support results to patients as well as providers32,52,56,57
  • One included CDSSs accompanied by periodic performance feedback56,57
  • Four included CDSSs accompanied by conventional education49,54,56,57,80

Health Care Provider Use

We identified 17 of the 148 eligible studies (11.5%) that evaluated provider use as an outcome of CDSS/KMS effectiveness or success. Those studies are summarized in Table H-8 of Appendix H.

The studies consistently had two of the three previously identified key factors/features: provision of decision support at time and location of decisionmaking and provision of a recommendation, not just an assessment.4,54,80,82,88,115128 Nine studies (52.9%) evaluated in the community setting consistently had two of the three previously identified key factors/features: provision of decision support at time and location of decisionmaking and provision of a recommendation, not just an assessment.4,54,80,117,118,120122,125,128 Thirteen CDSS interventions (76.5%) implemented in locally developed systems consistently had two of the three previously identified key factors/features: provision of decision support at time and location of decisionmaking and provision of a recommendation, not just an assessment.80,82,88,115117,120128 Three CDSS interventions (17.6%) implemented in commercially developed systems consistently had one of the three previously identified key factors/features: the provision of decision support at time and location of decisionmaking.4,118,119

Key Question 3

KQ 3: What is the impact of introducing electronic knowledge management and CDSSs?

  1. Changes in the organization of health care delivery
  2. Changes in the workload and efficiency for the user
  3. Changes in health care process measures and clinical outcomes

Key Points

  • There is strong evidence from the ambulatory setting that electronic CDSSs used at the point of care can enhance a variety of health care process measures.
  • We found that 86.5 percent of the studies measured some type of health care process measures whereas only 19.6 percent of the studies assessed a clinical outcome, thus more emphasis on the impact of CDSSs on clinical outcomes such as mortality, morbidity, length of stay, and adverse events are needed.
  • The evidence is scarce that these systems increase the value of care while decreasing costs.
  • There is limited evidence examining the impact of decision support tools on provider attitudes, workload, and efficiency.
  • Longer evaluation periods and larger sample sizes are needed to better assess the impact of CDSSs on outcomes.
  • More emphasis on the impact of CDSSs on providers, efficiency, and workload is needed to better understand how provider interaction and attitudes impact the quality of care delivered.

Detailed Analysis

Highlighted papers. Given the size and complexity of the published evidence, we examined a set of 12 high-quality, recently published papers in which the interventions were thoroughly described to guide our analysis of the impact of CDSSs on 6 outcome categories. For each outcome of interest, we present a summary of the included studies, the meta-analysis results when applicable, and a discussion of the key papers to help orient the reader to the broader evidence base and to inform the observations about the larger group of studies that evaluated each outcome category.2527,49,53,70,78,101,105,115,116,119,129,130

Six key categories of outcomes. From our examination of the impact of CDSSs and KMSs on clinical effectiveness and improved quality of care and patient outcomes, we present findings from the literature on six key categories of outcomes. During the initial review of the literature and data abstraction phase, we observed that the evidence concerning the organization of health care delivery (KQ 3a) was limited, and though we attempted to address this key question, we did not find evidence to support the impact of CDSSs/KMSs. The key categories of outcomes related to KQs 3b and 3c are:

  1. Clinical outcomes (length of stay, morbidity, mortality, measure of health-related quality of life, adverse events)
  2. Health care process measures (the recommended preventive care, clinical study, or treatment was ordered, completed, and adhered to; user knowledge)
  3. User workload and efficiency outcomes (number of patients seen, clinician workload, efficiency)
  4. Relationship-centered outcomes (patient satisfaction)
  5. Economic outcomes (cost and cost-effectiveness)
  6. Use and implementation outcomes (acceptance, satisfaction, use, implementation)

Impact on Clinical Outcomes

Length of stay. We identified 6 of the 148 eligible studies (4.1%) that specifically examined the impact of CDSSs/KMSs on length of stay. These studies are summarized in Table I-1 of Appendix I.

Of these six studies, four (66.7%) were conducted in the U.S.,26,27,29,30,34 one (16.7%) in Europe,31 and one (16.7%) in multiple countries.23 Four of the studies (66.7%) were implemented in an academic setting,23,29,30,34 and one (16.7%) in the community setting, 26,27 with one (16.7%) setting not reported.31 Three studies (50%) evaluated the systems in the inpatient environment,23,29,30 one (16.7%) in the ambulatory environment,26,27 and two (33.3%) in the emergency department.31,34 Duration of the evaluation period across the studies ranged from 12 weeks30 to 2.3 years.31 Five interventions (83.3%) were implemented using a system developed within the specific health care organization,23,26,27,29,31,34 and one (16.7%) was implemented using a commercially available system.30 Three systems (50%) aided health care providers with tasks for diagnosis,23,31,34 three (50%) for pharmacotherapy,23,29,30 one (16.7%) for chronic disease management,26,27 and two (33.3%) for laboratory test ordering.29,31 All of the systems (100%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter.23,26,27,2931,34 Two (33.3%) of the systems did not have a response requirement,23,34 one (16.7%) required a noncommittal acknowledgement,29 and in three studies (50%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.26,27,30,31 In three studies (50%), the recommendations were integrated within a CPOE or EHR system;2931 two (33.3%) were delivered via fax or computer printout26,27,34 and one (16.7%) via a standalone system.23 The recommendations were automatically delivered to the health care provider in all of the studies.23,26,27,2931,34 All six studies (100%) received a “Good” quality score.23,26,27,2931,34

We conducted a meta-analysis of the effect of CDSSs on length of stay (Figure 3). Of the six studies, five (83.3%) provided the necessary endpoint data to be included in meta-analysis.23,26,27,2931 The interventions included recommendations for appropriate antibiotic therapy,30 guideline-based reminders for corollary orders,29 diagnostic management of children with fever,31 risk assessment calculators for infection and antibiotic treatment recommendations,23 and guideline-based diabetes testing recommendations26,27 The combined relative risk for all studies was 0.96 (95% CI 0.88 to 1.05). However, if the Roukema et al.31 study, which was conducted in the pediatric population in the emergency department setting rather than the hospital setting, was excluded from the analysis, the combined relative risk for all studies was 0.91 (95% CI 0.86 to 0.97).

We conducted a meta-analysis of the effect of CDSSs on length of stay (Figure 3). Of the six studies, five (83.3%) provided the necessary endpoint data to be included in meta-analysis. The interventions included recommendations for appropriate antibiotic therapy, guideline-based reminders for corollary orders, diagnostic management of children with fever, risk assessment calculators for infection and antibiotic treatment recommendations, and guideline-based diabetes testing recommendations. The combined relative risk for all studies was 0.96 (95% CI 0.88 to 1.05). However, if the Roukema et al.31 study, which was conducted in the pediatric population in the emergency department setting rather than the hospital setting, was excluded from the analysis, the combined relative risk for all studies was 0.91 (95% CI 0.86 to 0.97).

Figure 3

Meta-analysis of length of stay outcomes.

One high-quality, recently published paper26,27 was examined in detail to guide observations about this group of studies. Khan et al. (2010)26,27 investigated guideline-based diabetes testing recommendations for 7412 patients from 64 community clinics and found that overall inpatient length of stay was significantly lower in the intervention group (0.99 versus 1.1 days; P = 0.01) and for the following intervention subgroups: seniors (1.22 versus 1.44 days; P = 0.002) and men (0.94 versus 1.1 days; P = 0.03).

In addition to the Khan et al.26,27 study, which achieved statistically significant results in the community ambulatory setting with a locally developed CDSS that automatically delivered system-initiated (push) recommendations asynchronously to the provider, there is evidence from four studies conducted in the academic setting of locally developed CDSSs that automatically delivered system-initiated (push) recommendations synchronously at the point of care demonstrated a trend toward reducing length of stay.23,29,30,34 This finding was supported by evidence collected from three studies that included more than 2000 patients;23,29,30 however, only one study34 included an evaluation period longer than 1 year. Notably, two studies were published after 2008.26,27,34 As mentioned previously, the Roukema et al. (2008)31 study reported that an intervention designed to promote the appropriate ordering of laboratory tests for children in the emergency department increased the median (25th to 75th percentile) length of stay from 123 (83–179) to 138 (104–181) minutes.

From the research included in this section, we concluded that limited evidence suggests that CDSSs are effective at reducing length of stay or demonstrating a trend toward reducing length of stay.

Morbidity. We identified 22 of the 148 eligible studies (14.9%) that specifically examined the impact of CDSSs/KMSs on morbidity. These studies are summarized in Table I-2 of Appendix I.

Of these 22 studies, 16 (72.7%) were conducted in the U.S.,2022,24,26,27,3440,68,75,108,111,113 3 (13.6%) in Europe,32,4143 1 (4.5%) in Brazil,33 and 2 (8%) in multiple countries.23,131 Eleven of the studies (50%) were implemented in an academic setting,2123,3437,39,40,75,108,131 5 (22.7%) in a community setting,26,27,32,4143,68 2 (9.1%) in both academic and community settings,24,33 3 (13.6%) in a VA setting,20,38,111 and one (4.5%) with the setting not reported.113 Six studies (27.3%) evaluated the systems in the inpatient environment,21,23,33,3537,131 13 (59.1%) in the ambulatory environment,20,22,24,26,27,32,3943,68,75,108,111 1 (4.5%) in both inpatient and ambulatory,38 1 (4.5%) in the emergency department,34 and one did not have the setting reported.113 Duration of the evaluation period across the studies ranged from 3 months35 to 4.5 years.38 Twenty interventions (90.9%) were implemented using a system developed within the specific health care organization,2024,26,27,3241,75,108,111,113,131 and 2 (18.2%) were implemented using a commercially available system.42,43,68 Four systems (18.2%) aided health care providers with tasks for diagnosis,23,34,42,43,131 9 (40.9%) for pharmacotherapy,20,2224,33,35,38,41,113 10 (45.5%) for chronic disease management,20,22,24,26,27,32,3941,108,111 2 (9.1%)for laboratory test ordering,22,68 and 6 (27.3%) for additional clinical tasks.21,22,36,37,4143,68 Sixteen of the systems (72.7%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,2023,3240,68,108,111,131 3 (13.6%) delivered recommendations outside of the health care provider–patient encounter,26,27,75,113 1 delivered recommendations both during and outside of the health care provider–patient encounter,42,43 and 2 (9.1%) were not clearly described.24,41 Three of the interventions (13.6%) required a mandatory response,21,35,68 4 (18.2%) did not have a response requirement,23,34,111,131 4 (18.2%) required a noncommittal acknowledgement,22,40,42,43,108 and in 11 studies (50%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.20,24,26,27,32,33,3639,41,75,113 In 8 studies (36.4%), the recommendations were integrated within a CPOE or EHR system;20,21,24,35,39,42,43,68,108 7 (31.8%) were delivered via fax or computer printout,22,26,27,34,38,41,75,111 6 (27.3%) via a standalone system,23,32,33,36,37,113,131 and 1 (4.5%) was integrated within a CPOE or EHR and delivered via fax or computer printout.40 The recommendations were automatically delivered to the health care provider in 17 studies (77.3%),2024,26,27,34,35,3843,68,75,108,111 in 2 studies (9.1%), the health care provider had to initiate an action to receive the recommendation,32,131 and in 3 studies (13.6%) the mode was not clearly described.33,36,37,113 Thirteen studies (59.1%) received a “Good” quality score,2024,26,27,3438,40,75,108 7 (31.8%) had a “Fair” score,32,33,42,43,68,111,113,131 and 2 (9.1%) received a “Poor” score.39,41

We conducted a meta-analysis of the effect of CDSSs on morbidity (Figure 4). Of the 22 studies, 16 (72.7%) provided the necessary endpoint data to be included in the meta-analysis.2024,26,27,3243 The combined relative risk of morbidity outcomes was 0.88 (95% CI 0.80 to 0.96).

We conducted a meta-analysis of the effect of CDSSs on morbidity (Figure 4). Of the 22 studies, 16 (72.7%) provided the necessary endpoint data to be included in the meta-analysis. The combined relative risk of morbidity outcomes was 0.88 (95% CI 0.80 to 0.96).

Figure 4

Meta-analysis of morbidity outcomes.

One high-quality, recently published paper26,27 was examined in detail to guide observations about this group of studies. Khan et al. 201026,27 assessed guideline-based diabetes recommendations to improve cholesterol, creatinine, proteinuria, and A1C testing and found that the overall number of hospitalizations was significantly lower in the intervention group for all subjects (0.17 versus 0.20, P = 0.01) and for the following subgroups: seniors (0.21 versus 0.27, P = 0.001) and men (0.17 versus 0.21, P = 0.02). The number of emergency department visits was significantly lower for all intervention subjects (0.27 versus 0.36, P < 0.0001) and for the following intervention subgroups: seniors (0.21 versus 0.36, P < 0.0001), men (0.23 versus 0.36, P < 0.0001), and women (0.30 versus 0.37, P = 0.01) in the intervention group.

From the research included in this section, we concluded that there is modest evidence from 7 studies (31.8%) conducted in the academic community inpatient and ambulatory settings that locally developed CDSSs integrated in a CPOE or EHR system or nonintegrated (paper or standalone system) that automatically delivered system-initiated (push) recommendations or required user-initiated (pull) requests for recommendations synchronously at the point of care or asynchronously outside the point of care are effective at reducing the proportion of patients who are admitted or readmitted to the hospital or emergency department,22,26,27,34,41,75 or who experience a hypoglycemia episode,33 or who have deep-vein thrombosis or pulmonary embolism at 30 days.21 However, the majority of those interventions were conducted in the academic ambulatory setting and evaluated locally developed nonintegrated CDSSs that automatically delivered system-initiated (push) recommendations synchronously at the point of care. This finding was supported by evidence from six studies that included evaluation periods of at least 1 year21,22,33,34,41,75 and from five studies that were evaluated with more than 2000 patients.21,22,26,27,41,75 Notably, four studies were published after 2008.26,27,33,34,41 In addition to the seven studies (31.8%) that reported statistical significance, there is evidence from the academic, community, and VA inpatient, ambulatory, and emergency department settings that locally developed CDSSs demonstrated a trend toward a reduction in morbidity. These studies described interventions that were integrated in a CPOE or EHR system and nonintegrated (paper-based or standalone system), delivered recommendations automatically (system-initiated) and required user action to receive the recommendation (user-initiated), and provided recommendations synchronously at the point of care. Examples of improved morbidity included a reduction in the proportion of patients who are admitted or readmitted to the hospital or emergency department;20,24,32,36,37 a reduction in significant cardiovascular diagnosis34 and lower cardiovascular event rates;42,43 and a reduction in the number of patients who experienced surgical site infections,35 have a shorter duration of fever,23 or have a colorectal adenoma detected.68 However, the majority of the studies were conducted in the community ambulatory setting and evaluated CDSSs that were locally developed, integrated in a CPOE or EHR system, and automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care. This supporting evidence was determined from five studies that included evaluation periods of at least 1 year20,36,37,42,43,68 and three studies that were evaluated with more than 2000 patients.23,42,43,68 However, only four studies were published after 2008.34,36,37,42,43,68 While representing only a limited subset of studies, in these studies there was no significant effect of a mandatory clinician response on patient morbidity.

Mortality. We identified 7 of the 148 eligible studies (4.7%) that specifically examined the impact of CDSSs/KMSs on mortality. These studies are summarized in Table I-3 of Appendix I.

Of these seven studies, six (85.7%) were conducted in the U.S.20,21,24,30,44,113 and one (14.3%) in multiple countries.23 Four of the studies (57.1%) were implemented in an academic setting,21,23,30,44 one (14.3%) in both academic and community settings,24 one (14.3%) in a VA setting,20 and 1 (14.3%) had a setting that was unclear.113 Four studies (57.1%) evaluated the systems in the inpatient environment, 21,23,30,44 two (28.6%) in the ambulatory environment,20,24 and 1 (14.3%) had an environment that was unclear.113 Duration of the evaluation period across the studies ranged from 12 weeks30 to 3 years and 4 months.21 Six interventions (85.7%) were implemented using a system developed within the specific health care organization,20,21,23,24,44,113 and one (14.3%) was implemented using a commercially available system.30 One system (4.3%) aided health care providers with tasks for diagnosis,23 five (71.4%) for pharmacotherapy,20,23,24,30,113 two (28.6%) for chronic disease management,20,24 and two (28.6%) for additional clinical tasks.21,44 Five systems (71.4%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,20,21,23,30,44 one (14.3%)delivered recommendations outside of the health care provider–patient encounter,113 and one system (14.3%) was not clearly described.24 Two of the interventions (28.6%) required a mandatory response,21,44 one (14.3%) did not have a response requirement,23 and in three studies (42.9%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.20,24,30,113 In four studies (57.1%), the recommendations were integrated within a CPOE or EHR system,20,21,24,30 two (28.6%) via a standalone system,23,113 and one (14.3%) delivered via pager and integrated within a CPOE or EHR.44 The recommendations were automatically delivered to the health care provider in six studies (85.7%), 20,21,23,24,30,44 with one study (14.3%) having a mode that was unclear.113 Six studies (85.7%) received a “Good” quality score,20,21,23,24,30,44 and one received a “Fair” quality score.113

We conducted a meta-analysis of the effect of CDSSs on mortality (Figure 5). Of the seven studies, six (85.7%) provided the necessary endpoint data to be included in meta-analysis.20,23,24,30,44 The combined odds ratio was 0.79 (95% CI 0.54 to 1.15). Thus, patients in the intervention group with a CDSS had an odds of dying that was 79 percent as large as those in the control group, and this combined effect did not reach statistical significance.

We conducted a meta-analysis of the effect of CDSSs on mortality (Figure 5). Of the seven studies, six (85.7%) provided the necessary endpoint data to be included in meta-analysis. The combined odds ratio was 0.79 (95% CI 0.54 to 1.15). Thus, patients in the intervention group with a CDSS had an odds of dying that was 79 percent as large as those in the control group, and this combined effect did not reach statistical significance.

Figure 5

Meta-analysis of mortality outcomes.

None of the 10 key papers reported data describing the impact of CDSSs on mortality. Of the studies that reported mortality data, Ansari et al. (2003)20 was conducted in the ambulatory VA setting for 1 year with 169 patients and found that a locally developed CDSS integrated in a CPOE or EHR system that promoted the appropriate use of beta blockers for CHF patients was effective at reducing patient mortality by 12 percent (P = 0.05). Roumie et al. (2006)24 assessed a locally developed CDSS integrated in a CPOE or EHR that promoted guideline-based hypertension treatment in the academic and community ambulatory settings with 1341 patients, 182 residents, staff physicians, nurse practitioners, and physician assistants for 6 months and reported that 3 (0.6%) patients died in the provider education and electronic alert group; 4 (0.9%) patients died in the provider education, alert, and patient education group; and 8 (2.5%) patients died in the provider education group (P = 0.027).

In addition to the two studies that showed statistical significance, there is evidence from two studies conducted in the academic inpatient setting of locally developed CDSSs that automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care demonstrated a trend toward reducing patient mortality.23,44 Notably, none of these studies were published after 2008. While this represented only a limited subset of studies, there was no significant effect of a mandatory clinician response on mortality.

From the research included in this section, we concluded that limited evidence suggests that CDSSs are effective at reducing patient mortality or demonstrating a trend toward reducing patient mortality.

Health care-related quality of life (HRQOL). We identified 6 of the 148 eligible studies (4.1%) that specifically examined the impact of CDSSs/KMSs on HRQOL or functional status. These studies are summarized in Table I-4 of Appendix I.

Of these six studies, five (83.3%) were conducted in the U.S.26,27,39,40,108,111 and 1 (16.7%) in Europe.132 Three of the studies (50%) were implemented in an academic setting,39,40,108 two (33.3%) in a community setting,26,27,132 and one (16.7%) in a VA setting.111 Five studies (83.3%) evaluated the systems in the inpatient environment39,40,108,111,132 and one (16.7%) in the ambulatory setting.26,27 Duration of the evaluation period across the studies ranged from 6 months132 to 2 years and 4 months.39,40 All interventions (100%) were implemented using a system developed within the specific health care organization.26,27,39,40,108,111,132 Five systems (83.3%) aided health care providers with tasks for chronic disease management26,27,39,40,108,111 and one (16.7%) for additional clinical tasks.132 Four of the systems (66.7%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,39,40,108,111 and 2 (33.3%) delivered recommendations outside of the health care provider–patient encounter.26,27,132 Two of the interventions (33.3%) did not have a response requirement,111,132 two (33.3%) required a noncommittal acknowledgement,40,108 and in two studies (33.3%), it was unclear to the abstractor if such requirement was present.26,27,39 In two studies (33.3%), the recommendations were integrated within a CPOE or EHR system; 39,108 three (50%) were delivered via fax or computer printout,26,27,111,132 and one (16.7%) was both within a CPOE or EHR and delivered via fax or computer printout.40 The recommendations were automatically delivered to the health care provider in all six studies (100%).26,27,39,40,108,111,132 Three studies (50%) received a “Good” quality score,26,27,40,108 two (33.3%) had a “Fair” score,111,132 and one (16.7%) received a “Poor” score.39

One high-quality, recently published paper26,27 was examined in detail to guide observations about this group of studies. Khan et al. (2010)26,27 assessed diabetes guideline-based testing recommendations and reported a significant improvement in patient exercise habits (adjusted effect +5.0, 95% CI +0.9, +9.1, P = 0.017) and a modest trend toward improved quality of life in the physical component score and patient diet.

Of the studies that reported quality-of-life data, two other studies of locally developed CDSSs that automatically delivered system-initiated (push) recommendations to providers were effective at improving quality-of-life scores.111,132 One found that the intervention patients who were treated by providers who received evidence-based treatment recommendations for the management of chronic heart failure had significant improvements in the mental component score compared to patients in the control group at 6 and 12 months.111 Another study reported that patients who received depression and anxiety treatment advice by intervention providers who utilized computer-based guidelines had significantly lower scores (a low score indicated better mental health) at 6 weeks (P = 0.04), but the significant effect was not maintained and at 6 months compared to usual care.132 In addition to those studies that demonstrated a statistical improvement in HRQOL, there is evidence that locally developed CDSSs that automatically delivered system-initiated (push) recommendations to providers demonstrated a trend toward improving patient quality of life.39,108 Murray et al. (2004)108 found that patients who received care from intervention physicians who received evidence-based hypertension reminders had higher quality-of-life scores with the exception of the role of physician compared to those patients in the pharmacist intervention, dual-intervention, and control groups. Tierney et al. (2005)39 reported that patients who were treated by physicians who received evidence-based treatment suggestions for asthma and chronic obstructive pulmonary disease (COPD) had greater quality-of-life scores for pain, general health, social function, and emotional subscales compared with the pharmacist intervention and control groups.

From the research included in this section, we concluded that limited evidence suggests that CDSSs are effective at improving or demonstrating a trend toward higher quality-of-life scores.

Adverse events. We identified 5 of the 148 eligible studies (3.4%) that specifically examined the impact of CDSSs/KMSs on adverse events. These studies are summarized in Table I-5 of Appendix I.

Examples of these outcomes included bleeding risks and thromboembolic complications;45 diarrhea as a side effect of antimicrobial use as indicated by testing for Clostridium difficile;30 adverse drug events from medication errors or from adverse drug reactions;46 postdischarge adverse events related to medical management within 1 month after discharge;36,37 and adverse events including cardiopulmonary arrest, transfer to the intensive care unit (ICU), myocardial infarction, delirium, stroke, renal insufficiency, acute renal failure, dialysis, return to operating room, and death.44

Of these 5 studies, 4 (80%) were conducted in the U.S.,30,36,37,44,45 and one (20%) was conducted in multiple countries.46 Four of the studies (80%) were implemented in an academic setting,30,36,37,44,46 and one (20%) was in both academic and community settings.45 Three studies (60%) evaluated the systems in the inpatient environment,30,36,37,44 one (20%) in the ambulatory environment,45 and one (20%) in a long-term facility.46 Duration of the evaluation period across the studies ranged from 12 weeks30 to 26 months.36,37 Four interventions (80%) were implemented using a system developed within the specific health care organization,36,37,4446 and one (20%) was implemented using a commercially available system.30 Two systems (40%) aided health care providers with tasks for pharmacotherapy30,46 and three (60%) for additional clinical tasks.36,37,44,45 All five systems (100%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter.30,36,37,4446 One of the interventions (20%) required a mandatory response,44 one (20%) did not have a response requirement,46 and in three studies (60%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.30,36,37,45 In two studies (40%), the recommendations were integrated within a CPOE or EHR system;30,46 one (20%) was integrated within a CPOE or EHR and via pager,44 one (20%) via a standalone system,36,37 and one (20%) had a format that was not clearly described.45 The recommendations were automatically delivered to the health care provider in three studies (60%);30,44,45 in one study (20%), the health care provider had to initiate an action to receive the recommendation,46 and in one study(20%) the mode was not clearly described.36,37 Three studies (60%) received a “Good” quality score,30,36,37,44 one (20%) had a “Fair” score,46 and one (20%) received a “Poor” score.45

We conducted a meta-analysis of the effect of CDSSs on adverse events using the five studies (Figure 6). The combined relative risk was 1.01 (95% CI 0.90 to 1.14). Thus, patients in the intervention group with a CDSS were as likely to experience an adverse event as patients in the control group.

We conducted a meta-analysis of the effect of CDSSs on adverse events using the five studies (Figure 6). The combined relative risk was 1.01 (95% CI 0.90 to 1.14). Thus, patients in the intervention group with a CDSS were as likely to experience an adverse event as patients in the control group.

Figure 6

Meta-analysis of adverse events.

None of the 10 key papers reported data describing the impact of CDSSs on adverse events. Of the studies that reported adverse events data, one found that a commercially developed CDSS designed to detect potentially inappropriate antimicrobial therapy used the frequency of C. difficile testing as an indicator for the presence of diarrhea and adverse effect of antimicrobial use.30 This study included 4507 patients for 12 weeks and reported that fewer intervention patients experienced diarrhea as a side effect of antimicrobial therapy (5.7% versus 6.6%; P = 0.21). The intervention was terminated at 12 weeks in order to expand the intervention to the control group. Although that one study demonstrated that CDSSs reduced or prevented adverse events, four studies did not observe any effect on reducing or preventing adverse events.36,37,4446

From the included evidence, we concluded that limited evidence suggests that CDSSs are effective at reducing or preventing adverse events.

Impact on Health Care Process Measures

Recommendations to order/complete a preventive care service. We identified 43 of the 148 eligible studies (29.1%) that specifically examined the impact of CDSSs/KMSs on the rates of ordering or completing recommended preventive care services. These studies are summarized in Table I-6 of Appendix I.

Of these 43 studies, 29 (67.4%) were conducted in the U.S.,21,22,39,47,50,51,58,60,68,71,75,76,84,85,9294,9698,126,133142 5 (11.6%) in Europe,4,41,56,57,74,123 6 (14%) in Canada,63,89,95,143145 2 (4.7%) in Australia,55,146 and 1 (2.3%) in New Zealand.147 Twenty of the studies (46.5%) were implemented in an academic setting,21,22,39,51,60,63,75,76,85,9496,136140,142,144146 15 (34.9%) in a community setting,4,41,47,5557,68,74,84,92,93,97,133,134,141,143,147 5 (11.6%) in both academic and community settings,58,71,98,126,135 1 (2.3%) in a VA setting,50 and 2 (4.7%) did not specify the location.89,123 Five studies (11.6%) evaluated the systems in the inpatient environment21,51,94,96,98 and 38 (88.4%) in the ambulatory environment.4,22,39,41,47,50,5558,60,63,68,71,7476,84,85,89,92,93,95,97,123,126,133147 Duration of the evaluation period across the studies ranged from 6 weeks146 to 40 months.21 Twenty-seven interventions (62.8%) were implemented using a system developed within the specific health care organization,21,22,39,41,50,51,60,75,76,84,85,9298,123,126,133,134,136139,142,144,145 10 (23.3%) were implemented using a commercially available system,4,47,5658,68,74,89,135,146,147 and 6 (14%) had a source that was not clearly described.55,63,71,140,141,143 Five systems (11.6%) aided health care providers with tasks for diagnosis,47,74,85,98,123 7 (16.3%) for pharmacotherapy,22,41,51,5658,126,146 11 (25.6%) for chronic disease management,4,22,39,41,47,50,58,92,138,141,143 10 (23.3%) for laboratory test ordering,22,58,60,68,71,123,126,138,140,142 3 (7%) for initiating discussions with patients,71,97,143 and 35 (81.4%) for additional clinical tasks.21,22,41,47,50,51,5558,60,63,68,71,7476,84,89,9396,98,123,126,133140,142,144,145,147 Forty (93%) of the systems delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,4,21,22,39,47,50,51,5658,60,63,68,71,74,76,84,85,89,9298,123,126,133147 one (2.3%) delivered recommendations outside of the health care provider–patient encounter,75 and for two studies (4.7%), the delivery mechanism for the CDSS was not clearly described.41,55 Four (9.3%) of the interventions required a mandatory response,21,51,68,85 3 (7%) required the health care provider to justify the reason for not complying with the recommendation,133,134,137,139,140 11 (25.6%) did not have a response requirement,47,56,57,84,92,93,95,126,138,144,145,147 5 (11.6%) required a noncommittal acknowledgement,22,58,96,136,142 1 (2.3%) required both a mandatory response and justification for not complying with the recommendation;60 and in 19 studies (44.2%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.4,39,41,50,55,63,71,7476,89,94,97,98,123,135,141,143,146 In 13 studies (30.2%), the recommendations were integrated within a CPOE or EHR system,4,21,39,47,51,5558,68,74,94,135,147 20 (46.5%) delivered via fax or computer printout,22,41,60,63,71,75,76,84,93,9597,133,134,137142,144,145 5 (11.6%) via a standalone system,85,89,92,123,146 2 (4.7%) via online recommendations,136,143 2 (4.7%) were integrated both within a CPOE or EHR and delivered via fax or computer printout,50,126 and 1 (2.3%) was via online recommendations and computer printout.98 The recommendations were automatically delivered to the health care provider in 35 studies (81.4%);4,21,22,39,41,47,50,51,5658,60,63,68,71,7476,84,85,9397,126,133135,137142,144146 in 6 studies (14%), the health care provider had to initiate an action to receive the recommendation,89,92,98,123,136,147 and in 2 studies (4.7%) the mode for assessing the CDSS was not clearly described.55,143 Twenty studies (46.5%) received a “Good” quality score,21,22,47,50,51,71,7476,84,9294,96,133,134,138,141,142,146,147 16 (37.2%) had a “Fair” score,4,5557,60,63,68,85,95,97,98,135,137,139,140,143145 and 7 (16.3%) received a “Poor” score.39,41,58,89,123,126,136

We conducted a meta-analysis (Figure 7) that focused on CDSS studies in which at least one outcome was related to ordering or completing preventive care services. Of the 43 studies that assessed a response to recommendations for ordering treatment or prescribing therapies, 25 studies (58.1%) included data with a common dichotomous endpoint and were included in the meta-analysis.4,21,40,41,47,50,51,5557,60,63,68,71,75,76,84,85,89,9298 Clinical decision support systems were found to have a statistically significant impact on the ordering or completing of preventive care services, with the overall effect of clinical decision support having an odds ratio of 1.42 (95% CI 1.27 to 1.58).

We conducted a meta-analysis (Figure 7) that focused on CDSS studies in which at least one outcome was related to ordering or completing preventive care services. Of the 43 studies that assessed a response to recommendations for ordering treatment or prescribing therapies, 25 studies (58.1%) included data with a common dichotomous endpoint and were included in the meta-analysis. Clinical decision support systems were found to have a statistically significant impact on the ordering or completing of preventive care services, with the overall effect of clinical decision support having an odds ratio of 1.42 (95% CI 1.27 to 1.58).

Figure 7

Meta-analysis of recommended preventive care service ordered.

We examined one high-quality, recently published paper92 in which the CDSS intervention was thoroughly described to guide observations about this group of studies. Bertoni et al. (2009)92 evaluated a handheld CDSS that calculated the Framingham risk score for cardiac disease and delivered recommendations for lipid screening and management-based national guidelines at 66 community clinics. They found that the lipid level screening rate increased in both the intervention and control practices (43.6% to 49% [intervention]; 40.1% to 50.8% [control]; net difference −5.3% P = 0.22).

From the research studies cited above, we concluded that there is strong evidence from 21 studies (48.8%) conducted in the academic, community, and VA inpatient and ambulatory settings that locally and commercially developed CDSSs are effective at improving appropriate ordering of preventive care procedures. These interventions were integrated in a CPOE or EHR system and nonintegrated (paper-based, online system, or standalone system); delivered recommendations automatically (system-initiated) and required user action to receive the recommendation (user-initiated); and provided recommendations synchronously at the point of care and asynchronously outside the point of care.21,22,41,47,50,51,58,60,71,7476,84,85,94,136141,143 However, the majority of the studies were conducted in the academic ambulatory settings and evaluated CDSSs that were locally developed, nonintegrated, automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care, and did not require a mandatory clinician response. This conclusion is supported by evidence from 12 studies that included evaluation periods longer than 1 year21,22,41,50,51,58,71,75,84,94,140,141 and 12 studies that were evaluated with more than 2000 patients.21,22,41,50,51,58,60,75,84,94,140,141 However, only five studies were published after 2008.41,58,74,141,143

With regard to improving the quality of care, very few of the studies demonstrated effectiveness of CDSSs designed to promote the appropriate ordering of preventive care procedures on clinical outcomes21,22,41,75 or on economic outcomes.47 In addition to the 21 studies (48.8%) that achieved statistical significance, there is supportive evidence from the academic and community inpatient and ambulatory settings of locally and commercially developed CDSSs that demonstrated a trend toward improving the appropriate ordering of preventive care procedures. These interventions were integrated in a CPOE or EHR system and nonintegrated (paper-based, online system, or standalone system); delivered recommendations automatically (system-initiated) and required user action to receive the recommendation (user-initiated); and provided recommendations synchronously at the point of care and asynchronously outside the point of care.5557,63,68,89,93,95,97,98,123,133135,142,144147 However, the majority of these studies were conducted in the academic or community ambulatory settings and the interventions were locally developed, nonintegrated, automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care, and did not require a mandatory clinician response demonstrated a trend toward improving appropriate ordering of preventive care procedures. This observation showing a trend for effectiveness is supported by evidence from 7 studies that included evaluation periods longer than 1 year56,57,68,93,95,133,134,144,145 and 11 studies that were evaluated with more than 2000 patients.5557,68,93,98,133135,142,144146 However, only three of these studies were published after 2008.68,98,135 Notably, with regard to improving the quality of care, very few of the studies that demonstrated a trend toward effectiveness of CDSSs to promote the appropriate ordering of preventive care procedures on clinical outcomes56,57,68,98 or on economic outcomes.56,57,63,95,123,144

With regard to the future direction of the field of using mobile devices to enhance the delivery and quality of care, one study demonstrated that use of a handheld computer-based decision support program at the point of care led to higher rates of preventive care screening in the intervention group for cervical and colorectal cancer, hyperlipidemia, hypertension, and in promoting prophylaxis with acetylsalicylic acid.89 However, another study found no effect of the intervention on lipid screening between the intervention and control group as screening rates increased for both groups.92

Recommendations to order/complete a clinical study. We identified 29 of the 148 eligible studies (19.6%) that specifically examined the impact of CDSSs/KMSs on the ordering and completion of recommended clinical studies. Examples of these interventions included reminders to order blood tests when ordering a medication, alerts to update a laboratory test, recommendations to refer patients for genetic testing, notices for x-ray orders, and suggestions to diagnose dementia and obesity. These studies are summarized in Table I-7 of Appendix I.

Of these 29 studies, 17 (58.6%) were conducted in the U.S.,26,27,39,48,49,61,65,66,69,70,77,87,101,102,148151 7 (24.1%) in Europe,31,67,99,103,118,128,152 3 (10.3%) in Canada,62,100,153 1 (3.4%) in Australia,78 and 1 (3.4%) in an unspecified country.154 Nine of the studies (37.5%) were implemented in an academic setting,39,48,61,62,65,66,69,87,148 6 (25%) in a community setting,67,99,103,118,128,152 5 in both academic and community settings,49,100,101,149,153 1 (4.2%) in a VA setting,70 and 3 (12.5%) in settings not clearly described.31,102,154 Two studies (6.9%) evaluated the systems in the inpatient environment,48,148 24 (82.8%) in the ambulatory environment,26,27,39,49,61,62,65,66,69,70,77,78,87,99103,118,128,149152,154 and 3 (10.3%) in the emergency department.31,67,153 Duration of the evaluation period across the studies ranged from 14 weeks154 to 2.4 years.49 Twenty interventions (69%) were implemented using a system developed within the specific health care organization,26,27,31,39,48,49,61,62,65,67,70,87,100,101,103,128,148,149,151,153,154 7 (24.1%) were implemented using a commercially available system,77,78,99,102,118,150,152 and 2 (6.9%) were implemented in a site that was not clearly described.66,69 Nine systems (31%) aided health care providers with tasks for diagnosis,31,62,67,69,77,87,100,148,152 2 (6.9%) for pharmacotherapy,61,77 5 (17.2%) for chronic disease management,26,27,39,49,69,152 16 (55.2%) for laboratory test ordering,31,48,61,65,66,70,78,100102,128,149151,153,154 2 (6.9%) for initiating discussions with patients,78,103 and 4 (13.8%) for additional clinical tasks.99,103,118,148 Twenty-seven of the systems (93.1%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,31,39,48,49,62,6567,69,70,77,78,87,99103,118,128,148154 and two (6.9%) delivered recommendations outside of the health care provider–patient encounter.26,27,61 Six of the interventions (20.7%) required a mandatory response,65,66,78,148,151,153 2 (6.9%) required the health care provider to justify the reason for not complying with the recommendation,48,70 5 (17.2%) did not have a response requirement,62,101,128,149,154 1 (3.4%) required a noncommittal acknowledgement,102 and in 15 studies (51.7%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.26,27,31,39,49,61,67,69,77,87,99,100,103,118,150,152 In 20 studies (69%), the recommendations were integrated within a CPOE or EHR system,31,39,48,49,65,66,69,70,77,78,101,102,128,148154 3 (10.3%) were delivered via fax or computer printout,26,27,61,62 3 (10.3%) via a standalone system,67,87,100 and 3 (10.3%) via other delivery methods.99,103,118 The recommendations were automatically delivered to the health care provider in 21 studies (72.4%);26,27,31,39,48,49,61,62,65,66,69,70,77,78,101,102,149154 in 5 studies (17.2%), the health care provider had to initiate an action to receive the recommendation,67,100,103,128,148 and in 3 studies (10.3%) the mode of CDSS delivery was not clearly described.87,99,118 Sixteen studies (55.2%) received a “Good” quality score,26,27,31,49,61,65,66,69,70,78,101,102,128,149,152154 9 (31%) had a “Fair” score,48,62,67,77,87,118,148,150,151 and 4 (13.8%) received a “Poor” score.39,99,100,103

We conducted a meta-analysis (Figure 8) that focused on CDSS studies in which at least one outcome was related to ordering or completing of recommended clinical studies. Of the 29 studies that assessed a response to recommendations for ordering or completing clinical studies, 20 (69.0%) included data with a common dichotomous endpoint and were included in the meta-analysis.26,27,31,39,48,49,61,62,6567,69,70,77,78,87,99103 Clinical decision support systems were found to have a statistically significant impact on the ordering or completing of clinical studies with the overall effect of clinical decision support having an odds ratio of 1.72 (95% CI 1.47 to 2.00). Note that there was a strong suggestion of publication bias in these studies (see Appendix I), and therefore these results should be viewed with caution.

We conducted a meta-analysis (Figure 8) that focused on CDSS studies in which at least one outcome was related to ordering or completing of recommended clinical studies. Of the 29 studies that assessed a response to recommendations for ordering or completing clinical studies, 20 (69.0%) included data with a common dichotomous endpoint and were included in the meta-analysis. Clinical decision support systems were found to have a statistically significant impact on the ordering or completing of clinical studies with the overall effect of clinical decision support having an odds ratio of 1.72 (95% CI 1.47 to 2.00). Note that there was a strong suggestion of publication bias in these studies (see Appendix I), and therefore these results should be viewed with caution.

Figure 8

Meta-analysis of recommended clinical studies ordered.

Five high-quality, recently published papers26,27,49,70,78,101 in which the CDSS interventions were thoroughly described were examined in detail to guide observations about this group of studies. Bell et al. (2010)49 assessed treatment reminders to improve provider adherence to national asthma guidelines at 12 academic and community clinics for 2.4 years and found that rates of performing spirometry significantly improved in the suburban intervention practices (P = 0.003).26,27 evaluated recommendations to improve appropriate guideline-based diabetes testing and reported that intervention patients were significantly more likely to receive guideline-appropriate testing for cholesterol (OR 1.39, 95% CI 1.07 to 1.80, P = 0.012), creatinine (OR 1.40; 95% CI 1.06 to 1.84, P = 0.018), and proteinuria (OR 1.74; 95% CI 1.13 to 1.69, P = 0.012). Walker et al. (2010)78 assessed guideline-based reminders to discuss chlamydia testing for women 16 to 24 years of age in 68 community clinics for 12 months and found that the rate of chlamydia testing significantly increased across intervention and control groups but that the intervention clinics had a greater increase in testing (27%; OR 1.3, 95% CI 1.1 to 1.4). Lo et al. (2009)101 assessed reminders to order appropriate laboratory tests in 22 clinics for 6 months and reported that there was no difference between intervention and control provider with regard to appropriately ordering laboratory tests within 14 days of a medication prescription (41% versus 39%) (OR 1.048, 95% CI 0.753 to 1.457, P = 0.782). Sundaram, et al., (2009)70 evaluated reminders to assess HIV risk behaviors or to offer HIV testing on 32 providers for 9 months and reported no change in testing rates between the intervention and control providers (0.29% versus 0.52%) (P = 0.75).

From the research reported in this section, we concluded that there is modest evidence from 19 studies (65.5%) conducted in the academic and community inpatient and ambulatory settings that locally and commercially developed CDSSs are effective at improving appropriate ordering of clinical studies. These studies included interventions that: were integrated in a CPOE or EHR system and nonintegrated (paper-based or standalone system); delivered recommendations automatically (system-initiated) and required user action to receive the recommendation (user-initiated); and provided recommendations synchronously at the point of care and asynchronously outside the point of care.26,27,31,48,49,61,6567,69,77,78,87,118,128,150154 However, the majority of these studies were conducted in the academic and community ambulatory settings and evaluated locally developed CDSSs integrated in CPOE or EHR systems that automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care and did not require a mandatory clinician response. Of those studies that reported a statistically significant effect, 9 studies included evaluation periods longer than 1 year31,49,65,66,78,118,128,150,153 and 9 were evaluated with more than 2000 patients.26,27,48,49,65,66,77,151153 Additionally, 10 of these studies were published after 2008.26,31,49,67,69,77,78,87,150,153

With regard to improving the quality of care, very few of the studies that demonstrated effectiveness of CDSSs assessed the effect of appropriate ordering of clinical studies on clinical outcomes26,27,31 or on economic outcomes.26,27,48,151 In particular, while the Roukema et al.31 study evaluated a decision support intervention in the pediatric emergency department that successfully promoted appropriate ordering of laboratory tests, it was also associated with an increase in length of stay. In addition to the 19 studies (65.5%) that reported statistical significance, there is limited supporting evidence from the academic and community ambulatory settings that locally and commercially developed CDSSs demonstrated a trend toward improving appropriate ordering of clinical studies. These studies described interventions that were integrated in a CPOE or EHR system and nonintegrated (paper-based or standalone system); delivered recommendations automatically (system-initiated) and required user action to receive the recommendation (user-initiated); and provided recommendations synchronously at the point of care.62,99101,103,149 However, the majority of these studies were conducted in the community ambulatory setting and the CDSSs were locally developed, automatically delivered system-initiated (push) recommendations synchronously at the point of care, and did not require a mandatory clinician response. This observation showing a trend for effectiveness is supported by evidence from one study that included an evaluation period longer than 1 year95 and three studies that were evaluated with more than 2000 patients.62,99 However, only two of these studies were published after 2008.101,149 Notably with regard to improving the quality of care, none of the studies that demonstrated a trend toward effectiveness of CDSSs assessed the effect of appropriate ordering of clinical studies on clinical outcomes, and very few assessed the effect on economic outcomes.62,103

Recommendations to order/prescribe treatment. We identified 67 of the 148 eligible studies (45.3%) that specifically examined the impact of CDSSs/KMSs on the ordering and prescribing of therapy. These studies are summarized in Table I-8 of Appendix I.

Of these 67 studies, 42 (62.7%) were conducted in the U.S.,20,24,25,29,30,35,3840,44,45,47,49,52,58,59,61,77,7983,88,92,104,105,108111,113,114,119,125,126,155162 18 (26.9%) in Europe,32,41,54,56,57,64,74,86,90,91,99,106,107,163170 4 (6%) in Canada,53,72,73,127 1 (1.5%) in multiple countries,23 and 1 (1.5%) location was not reported.112 Twenty-four of the studies (35.8%) were implemented in an academic setting,23,25,29,30,35,39,40,44,53,61,64,72,82,83,90,91,108,109,114,155159,162,166 22 (32.8%) in a community setting,32,41,47,52,54,56,57,59,74,80,81,86,92,99,107,110,125,160,161,163,165,169,170 12 (17.9%) in both academic and community settings,24,49,58,88,104106,112,119,126,164,167,168 4 (6%) in a VA setting,20,38,79,111 1 (1.5%) in both academic and VA settings,45 and 4 (6%) did not have the setting clearly reported.73,77,113,127 Thirteen studies (19.4%) evaluated the systems in the inpatient environment,23,29,30,35,44,64,82,114,156,158,159,161,169 47 (70.1%) in the ambulatory environment,20,24,32,3941,45,47,49,52,54,5659,61,7274,77,7981,86,88,92,99,104111,119,125127,155,157,160,162168,170 2 (3%) in both inpatient and outpatient environments,38,112 3 (4.5%) in the emergency department,25,83,90,91, 1 (1.5%) in a long-term care facility,53 and 1 (1.5%) in an unreported setting.113 Duration of the evaluation period across the studies ranged from 10 weeks64 to 4.2 years.104 Forty-eight interventions (71.6%) were implemented using a system developed within the health care organization,20,2325,29,32,35,3841,44,45,49,52,61,64,72,73,7983,86,88,92,104,105,108,110,111,113,125127,155162,164,165,167170 14 (20.9%) were implemented using a commercially available system,30,47,53,5659,74,77,99,107,114,119,163,166 and 5 sources (7.5%) were not clearly described.54,90,91,106,109,112 Nine systems (13.4%) aided health care providers with tasks for diagnosis,23,47,74,77,81,88,90,91,107,125 45 (67.2%) for pharmacotherapy,20,2325,29,30,35,38,41,53,54,5659,61,72,73,77,79,80,82,83,88,104,107,109,110,112114,119,125127,155157,159162,165170 5 (7.5%) for laboratory test ordering,29,58,61,80,126 25 (37.3%) for chronic disease management,20,24,32,3941,47,49,52,58,64,80,81,86,88,92,105,106,108,110,111,157,162165 and 9 (13.4%) for additional clinical tasks.25,44,45,47,74,90,91,99,126,158 Fifty-eight of the systems (86.6%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,20,23,25,29,30,32,35,3840,44,45,47,49,5254,5659,64,7274,77,7983,86,88,92,99,104106,108111,114,119,125127,155160,162,163,165170 4 (6%) delivered recommendations outside of the health care provider–patient encounter,61,112,113,161 2 (3%) provided recommendations using both mechanisms,90,91,164 and 3 (4.5%) did not clearly describe how the CDSS was delivered.24,41,107 Thirteen (19.4%) of the interventions required a mandatory response,25,35,44,59,82,83,90,91,109,110,114,119,159,160 5 (7.5%) required the health care provider to justify the reason for not complying with the recommendation,79,81,86,127,158 6 (9%) required a noncommittal acknowledgement,29,40,53,58,108,156 and 43 (64.2%) did not have a response requirement.20,23,24,30,32,38,39,41,45,47,49,52,54,56,57,61,64,7274,77,80,88,92,99,104107,111113,125,126,155,157,161170 In 39 studies (58.2%), the recommendations were integrated within a CPOE or EHR system;20,24,25,29,30,35,39,47,49,5254,5659,64,7274,77,8083,88,104,105,107109,114,119,127,155,156,158160,167,168 1 (1.5%) provided recommendations via an online system, 8 (11.9%) delivered recommendations via fax or computer printout,38,41,61,79,111,112,157,161,162 12 (17.9%) via a standalone system,23,32,86,9092,106,113,125,165,166,169,170 5 (7.5%) had a combination of two of these formats,40,44,110,126,164 and 3 (4.5%) did not clearly describe the format.45,99,163 The recommendations were automatically delivered to the health care provider in 54 studies (80.6%).20,2325,29,30,35,3841,44,45,47,49,5254,5659,61,64,7274,77,79,8183,86,104,105,108112,114,119,126,127,155162,167170 in 9 studies (13.4%), the health care provider had to initiate an action to receive the recommendation,32,80,9092,107,125,164166 1 study (1.5%) delivered recommendations using both modes,88 and mode was not reported in 3 studies (4.5%).99,106,163 Thirty-five studies (52.2%) received a “Good” quality score,20,2325,29,30,35,38,40,44,47,49,52,53,59,61,64,7274,79,88,9092,105,108110,112,119,158,159,161,164,165 24 (35.8%) had a “Fair” score,32,54,56,57,77,8083,86,104,106,107,111,113,114,125,127,155,157,160,162,166170 and 8 (11.9%) received a “Poor” score.39,41,45,58,99,126,156,163

We conducted a meta-analysis (Figure 9) that focused on CDSS studies in which at least one outcome was related to ordering treatments or prescribing therapies. Of the 67 studies that assessed a response to recommendations for ordering treatment or prescribing therapies, 46 studies (68.7%) included data with a common dichotomous endpoint and were included in the meta-analysis.20,23,24,29,30,32,35,3841,47,49,5254,5659,61,64,7274,77,7983,86,88,9092,104114 The overall effect of clinical decision support on treatment or prescribing outcomes was statistically significant and estimated as an odds ratio of 1.57 (95% CI 1.35 to 1.82). Thus, intervention providers with decision support were 1.6 times more likely to order the appropriate treatment or prescribe the correct therapy than control providers.

We conducted a meta-analysis (Figure 9) that focused on CDSS studies in which at least one outcome was related to ordering treatments or prescribing therapies. Of the 67 studies that assessed a response to recommendations for ordering treatment or prescribing therapies, 46 studies (68.7%) included data with a common dichotomous endpoint and were included in the meta-analysis. The overall effect of clinical decision support on treatment or prescribing outcomes was statistically significant and estimated as an odds ratio of 1.57 (95% CI 1.35 to 1.82). Thus, intervention providers with decision support were 1.6 times more likely to order the appropriate treatment or prescribe the correct therapy than control providers.

Figure 9

Meta-analysis of recommended treatment studies ordered.

Six high-quality, recently published papers49,53,92,105,119 Terrell et al. (2009)25 in which the CDSS interventions were thoroughly described were examined in detail to guide observations about the larger group of studies that evaluated treatment and prescribing outcomes. Bell et al. (2010)49 evaluated treatment reminders to improve provider adherence to asthma guidelines in part through the appropriate ordering and completion of clinical studies and found that the number of prescriptions for controller medication significantly increased in the intervention urban practices (P = 0.006). Bertoni et al. (2009)92 assessed a PDA-based decision support system that calculated the Framingham risk score and provided recommendations for lipid screening and management-based national guidelines and related to the appropriate ordering and completion of preventive care services. They reported that the appropriate treatment of cholesterol levels decreased in both the intervention and control practices but that the net change favored the intervention practices (+9.7%, CI 2.8% to 16.6%, P < 0.01) and that overtreatment of dyslipidemia with inappropriate prescriptions decreased in the intervention practices (net change, −4.9%, P = 0.01). Field et al. (2009)53 evaluated medication dose adjustment recommendations for long-term care residents with renal insufficiency in 22 long-term care units for 12 months and reported that overall final medication orders were more often appropriate in the intervention units (RR 1.2 [1.0, 1.4]). Fortuna et al. (2009)119 evaluated prescribing alerts for hypnotic medications embedded in an EHR among 257 providers over 12 months and found that the relative risk of prescribing a medication was less in both the alert group (RR 0.74; 95% CI 0.57 to 0.96) and the alert-plus-provider-education group (RR 0.74; 95% CI 0.58 to 0.97). Hicks et al. (2008)105 investigated diabetes and coronary artery disease treatment reminders to improve provider adherence to national guidelines in 14 clinics for 18 months and found a significant improvement in the rates at which appropriate medications were prescribed (P < 0.001). Terrell et al. (2009)25 investigated prescribing alerts that targeted potentially inappropriately prescribed medications for elderly patients on 63 emergency department physicians for 2.5 years. They reported that there were significantly fewer inappropriate prescriptions in the intervention group compared to the control group (OR 0.59; 95% CI 0.41 to 0.85).

From the research studies cited above, we concluded that there is strong evidence from 40 studies (59.7%) conducted in the academic, community, and VA inpatient and ambulatory settings that locally and commercially developed CDSSs are effective at improving appropriate ordering of treatment. This statement is supported by studies describing interventions that were integrated in a CPOE or EHR system and nonintegrated (paper-based or standalone system); delivered recommendations automatically (system-initiated) and required user action to receive the recommendation (user-initiated); and provided recommendations synchronously at the point of care and asynchronously outside the point of care.23,25,29,35,38,41,44,45,49,5254,5659,61,73,74,77,7981,83,86,88,92,104,105,109,113,126,155,156,158,159,161,163,165,166,169 However, the majority of the studies were conducted in the academic or community ambulatory settings and evaluated CDSSs that were locally developed, integrated in a CPOE or EHR system, automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care, and did not require a mandatory clinician response.

Of the studies that achieved a statistically significant effect, 13 studies included evaluation periods longer than 1 year,25,38,41,49,53,5658,83,86,104,105,109,163 and 18 were evaluated with more than 2000 patients.23,25,29,41,49,54,5659,73,83,86,88,92,105,126,156 Additionally, 15 of these studies were published after 2008.25,41,49,53,58,73,74,77,80,81,83,88,92,105,113 Notably with regard to improving the quality of care, only a few of the studies that demonstrated effectiveness of CDSSs assessed the effect of appropriate ordering of treatment on clinical outcomes23,29,35,38,41,44,45,56,57,75,113 or on economic outcomes.23,29,56,57,86,163 In addition to the 40 studies (59.7%) that reported statistical significance, there is supportive evidence from the academic, community, and VA inpatient and ambulatory settings of locally and commercially developed CDSSs that demonstrated a trend toward improving appropriate ordering of treatment. These studies described interventions that were integrated in a CPOE or EHR system and nonintegrated (paper-based or standalone system); delivered recommendations automatically (system-initiated) and required user action to receive the recommendation (user-initiated); and provided recommendations synchronously at the point of care and asynchronously outside the point of care.30,32,39,40,64,72,82,90,91,99,108,111,112,125,127,157,160,162,164,170 However, the majority of the studies were conducted in the academic ambulatory settings and evaluated CDSSs that were locally developed, integrated in a CPOE or EHR system, automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care, and did not require a mandatory clinician response. This observation showing a trend for effectiveness is supported by evidence from 7 studies that included evaluation periods longer than 1 year,39,40,72,111,125,157,160,162 and 7 studies were evaluated with more than 2000 patients.30,72,99,127,157,160,162,164 However, only three of these studies were published after 2008.82,127,164 With regard to improving the quality of care, only a few of the studies that demonstrated a trend toward effectiveness of CDSSs assessed the effect of appropriate ordering of treatment on clinical outcomes30,32,39,40,108,111 or on economic outcomes.30,39,40,108

Impact on user knowledge. We identified 5 of the 148 eligible studies (3.4%) that specifically examined the impact of CDSSs/KMSs on user knowledge. These studies are summarized in Table I-9 of Appendix I.

Of these 5 studies, one (20%) was conducted in the U.S.,117 two (40%) in Europe,118,123 one (20%) in Canada,143 and one (20%) in multiple countries.171 Three of the studies (60%) were implemented in a community setting117,118,143 and two (40%) in an unreported setting.123,171 Four of the studies (80%) evaluated the systems in the in the ambulatory environment117,118,123,143 and one (20%) did not clearly report the setting.171 Duration of the evaluation period across the studies ranged from 3 months171 to 1 year.118 Two interventions (40%) were implemented using a system developed within the specific health care organization,117,123 two (40%) were implemented using a commercially available system,118,171 and one (20%) did not specify a source of the CDSS/KMS.143 One system (20%) aided health care providers with tasks for diagnosis,123 one (20%) for chronic disease management,143 one (20%) for laboratory test ordering,123,126 one (20%) for initiating discussions with patients,143 and three (60%) for additional clinical tasks.117,118,171 117,118,123,126 Four (80%) of the systems delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter117,118,123,143 and one (20%) did not report a relation.171 One (20%) of the interventions required a mandatory response,171 one of the interventions (20%) did not have a response requirement,117 and in three studies (60%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.118,123,143 In one study (20%), the recommendations were integrated within a CPOE or EHR system;117 1 (20%) via a standalone system,123 2 (40%) delivered online,143,171 and the format of one study (20%)118 was not clear. In three studies (60%) the health care provider had to initiate an action to receive the recommendation117,123,171 and two studies (40%) did not clearly describe how recommendations were delivered.118,143 No studies received a “Good” quality score, four (80%) had a “Fair” score,117,118,143,171 and one (20%) received a “Poor” score.123,126

None of the 10 key papers reported data describing the impact of CDSSs/KMSs on user knowledge. Of the studies that reported user knowledge data, Alper et al. (2005)171 reported that an electronic knowledge resource accessed by providers during and outside of the health care provider–patient encounter increased the number of questions answered (75.8% versus 71.2%) and the number of questions for which the answer changed decisionmaking (64.6% versus 23.4%); however, the number of questions for which the providers did not find an answer that could have changed decisionmaking did not improve with access to the resource (19.6% versus 23.4%). Del Fiol et al. (2008)117 found providers reported that in 62% of sessions, the use of an information retrieval tool embedded in an EHR system that provided access to topic or nonspecific links to clinical resources to aid in answering clinicians’ questions at the point of care enhanced their decisions or knowledge. Holbrook et al. (2009)143 found that 48% of providers who used a Web-based diabetes tracker that included diabetes care reminders reported that their knowledge of diabetes blood sugar control targets had improved. Emery et al. (2007)118 reported that a cancer risk assessment tool improved clinician confidence in managing the risk of familial cancer. Hobbs et al. (1996)123 found that providers reported their knowledge of lipid disorders improved; however, no distinction was made between those who received the intervention (a standalone decision support system for the management of hyperlipidemia) and those who did not.

From the research included in this section, we concluded that there is limited evidence regarding the effect of CDSSs/KMSs on user knowledge.

Impact on Workload and Efficiency

Number of patients seen/unit time. Of the eligible studies, none examined the impact of the CDSSs/KMSs on the number of patients seen/unit time.

Clinician workload. Of the eligible studies, none examined the impact of CDSSs/KMSs on clinician workload.

Efficiency. We identified 7 of the 148 eligible studies (4.7%) that specifically examined the impact of CDSSs/KMSs on efficiency. Examples of metrics used to assess efficiency included median search times or session times using the KMS, clinician response time, questionnaires assessing effort required to complete a process using the CDSS. These studies are summarized in Table I-10 of Appendix I.

Of these seven studies, five (71.4%) were conducted in the U.S.,30,36,37,110,117,151 one (14.3%) in Canada,172 and one (14.3%) was conducted in multiple countries.171 Four of the studies (57.1%) were implemented in an academic setting,30,36,37,151,172 two (28.6%) in a community setting,110,117 and one (14.3%) did not report a specific setting.171 Three studies (42.9%) evaluated the systems in the inpatient environment,30,36,37,172 three (42.9%) in the ambulatory environment,110,117,151 and one (14.3%) did not report a specific environment.171 Duration of the evaluation period across the studies ranged from 12 weeks30,171 to 30 months.110 Four interventions (57.1%) were implemented using a system developed within the specific health care organization,36,37,110,117,151 and three (42.9%) were implemented using a commercially available system.30,171,172 Two systems (28.6%) aided health care providers with tasks for pharmacotherapy,30,110 one (14.3%) for chronic disease management,110 two (28.6%) for lab test ordering,151,172 and three (42.9%) for additional clinical tasks.36,37,117,171 Five of the systems (71.4%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,30,36,37,110,117,151 one (14.3%) delivered recommendations outside of the health care provider–patient encounter,172 and one (14.3%) delivered recommendations both in real time and outside of the health care provider–patient encounter.171 Three of the interventions (42.9%) required a mandatory response,110,151,171 one (14.3%) did not have a response requirement,117 and in three studies (42.9%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.30,36,37,172 In three studies (42.9%), the recommendations were integrated within a CPOE or EHR;30,117,151 one (14.3%) was delivered online,171 one (14.3%) via a standalone system,36,37 one (14.3%) via pager,172 and one (14.3%) was delivered online and via email.110 The recommendations were automatically delivered to the health care provider in four studies (57.1%);30,110,151,172 in two studies (28.6%), the health care provider had to initiate an action to receive the recommendation,117,171 and in one study (14.3%) the mode was not clearly described.36,37 Three studies (42.9%) received a “Good” quality score,30,36,37,110 four (57.1%) had a “Fair” score,117,151,171,172 and 0 received a “Poor” score.

None of the 10 key papers reported data describing the impact of CDSSs on efficiency. Of the studies that reported efficiency data, one observed that use of the KMS that provided topic and nonspecific infobutton links to clinicians at the point of care significantly reduced the time that health care providers spent seeking information according to an evaluation of 90 providers and 3,729 session duration from 43 seconds to 35.5 seconds (P = 0.008)117 McGregor et al. (2006)30 found that clinicians who received CDSS alerts spent roughly one hour less each day resolving inappropriate antibiotic prescriptions in the intervention arm than the control arm of the trial. Alper et al. (2005)171 reported from 780 clinician queries with 52 physicians and nurse practitioners that the KMS had a positive trend on reducing the time searching and answering clinical questions using DynaMed when accessed during the health care provider–patient encounter as well as outside of the encounter; however, the study also reported that system use did not improve time searching for information or time for unsuccessful searches. Etchells et al. (2010)172 observed in a study with 165 critical laboratory values for 108 patients which were sent via an alphanumeric pager to the physician that median physician response time after receiving a critical value decreased from 39 minutes to 16 minutes (P = 0.33). However, two studies reported that use of the CDSS increased the time to complete a desired action. Graumlich et al. (2009)36,37 reported from a study with 70 physicians and 631 patients that clinicians found the effort to use the electronic discharge planning tool for discharge planning was more difficult than usual care (paper). Tierney et al. (1987)151 observed in a study of 111 physicians and 5946 patients that use of a CDSS that displayed past diagnostic test results to the clinician prior to ordering a new test increased the time to order by 4.5 seconds (8%) (P < 0.01).

From the research included in this section, we concluded that there is limited evidence of CDSSs/KMSs demonstrating improvement in efficiency.30,117,171,172 This finding is supported by evidence from studies that all included evaluation periods less than 6 months, although the McGregor article30 reported that the study was discontinued at 12 weeks to implement the CDSS throughout the entire hospital based. Of note, only one of these studies evaluated the CDSS/KMS with more than 2000 patients30 and only two were published after 2008.117,172

Impact on Relationship-centered Outcomes

Patient satisfaction. We identified 6 of the 148 eligible studies (4.1%) that specifically examined the impact of CDSSs/KMSs on patient satisfaction. Patient satisfaction was assessed qualitatively using either telephone interviews conducted by study personnel, mailed patient questionnaires, or occasionally on-site interviews during patient clinic visits. These studies are summarized in Table I-11 of Appendix I.

Of these six studies, four (66.7%) were conducted in the U.S.,34,36,37,39,47 one (16.7%) in Canada143, and one (16.7%) did not report location.154 Three of the studies (50%) were implemented in an academic setting,34,36,37,39 two (33.3%) in a community setting,47,143 and one (16.7%) did not report the setting.154 One study (16.7%) evaluated the systems in the inpatient environment,36,37 four (66.7%) in the ambulatory environment,39,47,143,154 and one (16.7%) in the emergency department.34 Duration of the evaluation period across the studies ranged from 14 weeks154 to 28 months.39 Four interventions (66.7%) were implemented using a system developed within the specific health care organization34,36,37,39,154 one (16.7%) was implemented using a commercially available system,47 and one (16.7%) had a source that was not clearly described.143 Two systems (33.3%) aided health care providers with tasks for diagnosis,34,47 two (33.3%) for chronic disease management,47,143 one (16.7%) for laboratory test ordering,154 one (16.7%) for initiating discussions with patients,143 and two (33.3%) for additional clinical tasks.36,37,47 All 6 of the systems (100%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter.34,36,37,39,47,143,154 Three of the interventions (50%) did not have a response requirement,34,47,154 and in three studies (50%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.36,37,39,143 In three studies (50%), the recommendations were integrated within a CPOE or EHR system;39,47,154 one (16.7%) was delivered online,143 one (16.7%) via a standalone system,36,37 and one (16.7%) via fax or computer printout.34 The recommendations were automatically delivered to the health care provider in four studies (66.7%),34,39,47,154 and in two studies (33.3%) the mode was not clearly described.36,37,143 Four studies (66.7%) received a “Good” quality score,34,36,37,47,154 one (16.7%) had a “Fair” score, 143 and one (16.7%) received a “Poor” score.39

None of the 10 key papers reported data describing the impact of CDSSs on patient satisfaction. Of the studies that reported patient satisfaction data, one reported that patients who were treated by intervention providers who utilized a discharge planning application had a higher perception of discharge preparedness and satisfaction with medication information.36,37 Holbrook et al. (2009)143 found that 75.9 percent of patients who received care from intervention providers who accessed a Web-based diabetes tracker to aid in therapeutic planning were more satisfied with the quality of their diabetes care. Kline et al. (2009)34 reported that more intervention patients who were treated by intervention providers who received a printout of pretest probability of acute coronary syndrome were satisfied with the explanation of the medical problem than those patients in the control group. Feldstein et al. (2006)154 observed that patients who received a new study drug, which subsequently required baseline laboratory testing found electronic recommendations presented to the physician during the patient visit, automated voice messages to the patient, and a call from a pharmacy team member all to be acceptable. Apkon et al. (2005)47 reported that intervention patients who used problem-knowledge couplers to report their chief complaint and guide provider decisionmaking were less satisfied with the overall visit; however, intervention patients were more satisfied with their interaction with the provider than those in the control group. An additional study by Tierney et al. (2005)39 assessed patient satisfaction with the physician’s communication abilities and pharmacy and found there was no effect on patient satisfaction between those who were treated by intervention providers who received guideline-based recommendations for the management of asthma and COPD and those patients who were treated by control providers.

From the research included in this section, we concluded that there is limited evidence that clinician use of CDSSs had a positive effect on patient satisfaction.34,36,37,143,154 This observation showing intervention patients were more satisfied than those in the control group is based on studies that included evaluation periods of at least 2 years36,37,143 and were published in 2009.34,36,37,143 Notably, two studies did not find that provider use of CDSSs increased satisfaction with the care received or overall visit.39,47

Impact on Economic Outcomes

Cost. We identified 22 of the 148 eligible studies (14.9%) that specifically examined the impact of CDSSs/KMSs on cost. These studies are summarized in Table I-12 of Appendix I.

Of these 23 studies, 14 (63.6%) were conducted in the U.S.,26,27,29,30,39,40,47,48,108,110,148,151,173175 6 (27.3%) in Europe,56,57,86,103,123,129,130,163 1 (4.5%) in multiple countries,23 and 1 (4.5%) did not report a location.176 Eleven (50%) of the studies were implemented in an academic setting,23,29,30,39,40,48,108,148,151,173,175 10 (45.5%) in a community setting,26,27,47,56,57,86,103,110,129,130,163,174,176 and 1 (4.5%) did not report a setting.123 Five studies (22.7%) evaluated the systems in the inpatient environment23,29,30,48,148 and 17 (77.3%) in the ambulatory environment.26,27,39,40,47,56,57,86,103,108,110,123,129,130,151,163,173176 Duration of the evaluation period across the studies ranged from 25 days 176 to 2.5 years.110 Seventeen interventions (77.3%) were implemented using a system developed within the specific health care organization,23,26,27,29,39,40,48,86,103,108,110,123,148,151,173176 and 5 (22.7%) were implemented using a commercially available system.30,47,56,57,129,130,163 Five systems (22.7%) aided health care providers with tasks for diagnosis,23,47,123,148,175 five (22.7%) for pharmacotherapy,23,29,30,56,57,110 nine (40.9%) for chronic disease management,26,27,39,40,47,86,108,110,129,130,163 six (27.3%) for laboratory test ordering,29,48,123,151,175,176 two (9.1%) for initiating discussions with patients,103,174 and seven (31.8%) for additional clinical tasks.47,56,57,103,123,148,173,174 Twenty of the systems (90.9%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,23,29,30,39,40,47,48,56,57,86,103,108,110,123,129,130,148,151,163,173,175,176 and 2 (9.1%) delivered recommendations outside of the health care provider–patient encounter.26,27,174 Five of the interventions (22.7%) required a mandatory response,110,129,130,148,151,175 two (9.1%) required the health care provider to justify the reason for not complying with the recommendation,48,86 five (22.7%) did not have a response requirement,23,47,56,57,173,176 three (13.6%) required a noncommittal acknowledgement,29,40,108 and in seven studies (31.8%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.26,27,30,39,103,123,163,174 In 11 studies (50%), the recommendations were integrated within a CPOE or EHR system;29,30,39,47,48,56,57,108,148,176 3 (13.6%) were delivered via fax or computer printout,26,27,173,174 1 (4.5%) was integrated within a CPOE or EHR and via delivered via fax or computer printout, 40 4 (18.2%) via a standalone system,23,86,123,129,130 and 3 (13.6%) had other formats.40,110,163 The recommendations were automatically delivered to the health care provider in 17 studies (77.3%);23,26,27,29,30,39,40,47,48,56,57,86,108,110,151,173176 In four studies (18.2%), the health care provider had to initiate an action to receive the recommendation,103,123,129,130,148 and one (4.5%) study did not have a mode clearly reported.163 Ten studies (45.5%) received a “Good” quality score,23,26,27,29,30,40,47,108,110,129,130,176 7 (31.8%) had a “Fair” score,48,56,57,86,148,151,174,175 and 5 (22.7%) received a “Poor” score.39,103,123,163,173

Two high-quality, recently published papers26,27,129,130 in which the CDSS interventions were thoroughly described were examined in detail to guide observations about this group of studies. Cleveringa et al. (2008)129,130 evaluated a standalone system that focused on decreasing cardiovascular risk in 3391 patients with type 2 diabetes over 12 months by including an algorithm based on the Dutch type 2 diabetes diagnostic and treatment guidelines. They found that use of the CDSS to provide patient-specific treatment recommendations reduced cardiovascular risk, but it was more costly as patients in the intervention group incurred higher total costs than those in the control group (€1,415, P = NS; ~ $1,967). Khan et al. (2010)26,27 assessed guideline-based diabetes recommendations to improve appropriate testing and they reported a significant reduction in hospitalization expenses for all subjects in the intervention group ($3,113.19 versus $3,480.14, P = 0.02) and the following intervention subgroups: seniors (age 65 years and older) ($3,699.26 versus $4,264.36, P = 0.004) and men ($3,098.26 versus $3,712.22, P = 0.03). A significant reduction in emergency department expenses was also found for all subjects in the intervention group ($414.30 versus $301.51, P < 0.0001) and for the following subgroups: seniors ($270.45 versus $443.27, P < 0.0001); men ($299.18 versus $410.91; P < 0.0001); and women ($307.80 versus $417.45, P < 0.009).

However, though there was an enormous variability in the studies reporting cost data, other studies found a cost savings between $6,000 (through recommendations for the appropriate use of abdominal radiograph orders) and $84,194 (through reminders about the appropriate use of antimicrobials). Of those reporting costs savings, Cobos et al. (2005)86 reported a significant cost savings by reducing the number of lipid-lowering drug prescriptions during the 1-year evaluation period between 20.8 and 24.9% from a CDSS that provided hypercholesterolemia treatment and followup visit recommendations.86 A second study published in 2008110 reported that a telemedicine intervention for the medication management of cardiovascular risk found that the intervention resulted in cost savings for outpatient costs (−$288) (95% CI −$25 to −$550) and total costs (−$2,311) (95% CI −$266 to −$4667).

From the research included in this section, we concluded that although one key paper found that the intervention increased costs, there is modest evidence from 13 studies (59.1%), including a second key paper conducted in the academic and community inpatient and ambulatory settings, that locally and commercially developed CDSSs demonstrated a trend toward lower treatment costs, total costs, and greater cost savings than the control groups and other non-CDSS intervention groups (e.g., patient education intervention, pharmacist intervention). These interventions were integrated in a CPOE or EHR system and nonintegrated (paper-based, online system, or standalone system); delivered recommendations automatically (system-initiated) and required user action to receive the recommendation (user-initiated); and provided recommendations synchronously at the point of care and asynchronously outside the point of care. 23,26,27,29,30,40,48,56,57,86,110,148,151,174,175 However, the majority of these studies evaluated locally developed CDSSs integrated in a CPOE or EHR system that automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care and did not require a mandatory response in the community ambulatory settings. This observation showing a trend toward lower costs and greater cost savings is supported by evidence from five studies that included evaluation periods longer than 1 year40,56,57,86,110,174 and nine studies with more than 2000 patients.23,26,27,29,30,48,56,57,86,151,175 Notably, all except two studies were published prior to 2008.26,27,110

Cost-effectiveness. We identified 6 of the 148 eligible studies (4.1%) that specifically examined the cost-effectiveness of CDSSs/KMSs or the impact of CDSSs/KMSs on the cost-effectiveness of care. These studies are summarized in Table I-13 of Appendix I.

Of these six studies, two (33.3%) were conducted in in Europe56,57,129,130 and four (66.7%) in Canada.62,63,95,144 Four of the studies (66.7%) were implemented in an academic setting62,63,95,144 and two (33.3%) in a community setting.56,57,129,130 All six studies (100%) evaluated the systems in the ambulatory environment.56,57,62,63,95,129,130,144 Duration of the evaluation period across the studies ranged from 10 weeks 63 to 15 months.62 Three interventions (50%) were implemented using a system developed within the specific health care organization,62,95,144 two (33.3%) were implemented using a commercially available system,56,57,129,130 and one (16.7%) did not clearly describe a source.63 One system (16.7%) aided health care providers with tasks for diagnosis,62 one (16.7%) for pharmacotherapy,56,57 one (16.7%) for chronic disease management,129,130 and four (66.7%) for additional clinical tasks.56,57,63,95,144 All six of the systems (100%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter.56,57,62,63,95,129,130,144 One of the interventions (16.7%) required a mandatory response,129,130 four (66.7%) did not have a response requirement,56,57,62,95,144 and in one study (16.7%) it was unclear to the abstractor if such requirement was present.63 In one (16.7 %) study, the recommendations were integrated within a CPOE or EHR system;56,57 four (66.7%) were delivered via fax or computer printout62,63,95,144 and one (16.7%) via a standalone system.129,130 The recommendations were automatically delivered to the health care provider in five (83.3%) studies,56,57,62,63,95,144 and the health care provider had to initiate an action to receive the recommendation in one study(16.7%).129,130 One (16.7%) study received a “Good” quality score,129,130 and five (83.3%) had a “Fair” score.56,57,62,63,95,144

One high-quality, recently published paper129,130 was examined in detail to guide observations about this group of studies. As described in the previous section, Cleveringa et al. (2008)129,130 evaluated a standalone system that provided clinicians with treatment recommendations for decreasing cardiovascular risk factors for type 2 diabetic patients and related to the resulting benefits cost-effectiveness. They found that the intervention group incurred higher total costs (€1,415; ~$1,967) and exceeded the study’s established willingness to pay quality-adjusted life year threshold of €20,000 (~$27,808). The remaining studies found that the intervention group tended to be more cost-effective in providing recommended preventive care, screenings, and treatment than usual care or other interventions (e.g., patient letters, telephone reminders). Rosser et al. (1992)144 assessed the cost-effectiveness of three interventions for improving provider compliance with reminders for tetanus vaccination. The effectiveness of each intervention was assessed based on provider time, time to prepare and deliver recommendations, and supply costs of mailing patient reminder letters. Among the three groups, they found that the cost per additional vaccination was $0.43 or $0.22 depending on the salary level for the physician reminders; $5.43 or $4.43 depending on the nurse salary level for the telephone reminders; and $6.05 for the patient letter reminders. McDowell et al. (1989)62 evaluated the cost-effectiveness of three interventions for improving blood pressure screening and assessed the effectiveness based on staff and material costs of delivering the recommendations. They reported that the cost per blood pressure reading was $1.70 or $1.33 depending on the salary level for physician reminders; $31.27 or $22.47 depending on the nurse salary level for telephone reminders; and $14.37 for the patient letter reminders. Fretheim et al. (2006)56,57 evaluated the cost-effectiveness of prescribing recommendations for antihypertensive and cholesterol-lowering drugs and estimated that the cost of using the CDSS was $183 per additional patient being started on a thiazide.

From the research included in this section, we concluded that there is conflicting evidence from the ambulatory setting regarding the cost-effectiveness of CDSSs that provided recommendations to providers synchronously at the point of care. This observation showing the interventions were more cost-effective for performing recommended process measures than the control groups is supported by studies with evaluation periods of at least 1 year and studies evaluated with more than 2000 patients.56,57,62,144 However, three studies reported that the intervention was not cost-effective.63,95,129,130 Notably, none of those studies that found favorable evidence on the cost-effectiveness of CDSSs were published after 2008; the most recent study was published in 2006,56,57 one in 1992,144 and the other in 1989.62

Impact on Use and Implementation Outcomes

Health care provider acceptance. We identified 24 of the 148 eligible studies (16.2%) that specifically examined the impact of health care provider acceptance of CDSSs/KMSs. These studies are summarized in Table I-14 of Appendix I.

Of these 24 studies, 17 (70.8%) were conducted in the U.S., 22,25,45,60,70,79,98,119,124,140,148,158,173,174,177179 5 (20.8%) in Europe,86,120122,164,170 and 2 (8.3%) in Canada.73,127 Ten of the studies (41.7%) were implemented in an academic setting,22,25,60,124,140,148,158,173,177,178 5 (20.8%) in a community setting,86,120122,170,174 3 (12.5%) in both academic and community settings,98,119,164 2 (8.3%) in a VA setting,70,79 1 (4.2%) in both academic and VA settings,45 and 3 (12.5%) for which the setting was not clearly described.73,127,179 Three studies (12.5%) evaluated the systems in the inpatient environment,98,148,158 19 (79.2%) in the ambulatory environment,22,45,60,70,73,79,86,119122,124,127,140,164,170,173,174,178,179 1 (4.2%) in a long-term care facility,177 and 1 (4.2%) in the emergency department.25 Duration of the evaluation period across the studies ranged from 1 month170 to 2.5 years.25 Twenty-one interventions (87.5%) were implemented using a system developed within the specific health care organization,22,25,45,60,70,73,79,86,98,120122,124,127,148,158,164,170,173,174,177,179 2 (8.3%) were implemented using a commercially available system,119,178 and 1 study (4.2%) did not clearly describe a source.140 Three systems (12.5%) aided health care providers with tasks for diagnosis,98,148,178 9 (37.5%) for pharmacotherapy,22,25,73,79,119,124,127,170,177 5 (20.8%) for chronic disease management,22,86,120122,164 4 (16.7%) for laboratory test ordering,22,60,70,140 1 (4.2%) for initiating discussions with patients,174 and 12 (50%) for additional clinical tasks.22,25,45,60,98,140,148,158,164,173,174,179 Twenty-one of the systems (87.5%) delivered recommendations in real time to enable decisionmaking during the health care provider–patient encounter,22,25,45,60,70,73,79,86,98,119122,124,127,140,148,158,170,173,177,179 2 (8.3%) delivered recommendations outside of the health care provider–patient encounter,174,178 and 1(4.2%) in both real time and outside of the health care provider–patient encounter.164 Four of the interventions (16.7%) required a mandatory response,25,119,148,178 7 (29.2%) required the health care provider to justify the reason for not complying with the recommendation,70,79,86,127,140,158,164 3 (12.5%) did not have a response requirement,124,173,177 1 (4.2%) required a noncommittal acknowledgement,22 1 (4.2%) required both a mandatory response and a reason for not complying,60 and in 8 studies (33.3%), it was assumed that there was no user response requirement or it was unclear to the abstractor if such requirement was present.22,45,73,98,120122,170,174,179 In 11 studies (45.8%), the recommendations were integrated within a CPOE or EHR system;25,70,73,119122,124,127,148,158,177 6 (25%) were delivered via fax or computer printout,22,60,79,140,173,174 3 (12.5%) via a standalone system,86,170,179 and 4 (16.7%) had other formats or combinations of formats.45,98,164,178 The recommendations were automatically delivered to the health care provider in 18 studies (75%),22,25,45,60,70,73,79,86,119,127,140,158,170,173,174,177179 and the health care provider had to initiate an action to receive the recommendation in 6 studies (25%).98,120122,124,148,164 Nine studies (37.5%) received a “Good” quality score,22,25,70,73,79,119,124,158,164 11 (45.8%) had a “Fair” score,60,86,98,120122,127,140,148,170,174,177 and 4 (16.7%) received a “Poor” score.45,173,178,179

Three high-quality, recently published papers25,70,119 in which the CDSS interventions were thoroughly described were examined in detail to guide observations about this group of studies. Fortuna et al. (2009)119 evaluated prescribing alerts for heavily marketed hypnotic medications on health care provider acceptance. They found that only 23% of providers felt that recommendations that included alternative treatment suggestions and information on prescribing, patient education materials, and copayment for heavily marketed medications changed their prescribing decisions. Regarding health care provider acceptance, the Sundaram et al. (2009)70 study found that providers were more likely to adhere to reminders to test for HIV rather than reminders to perform HIV risk assessment (11% versus 5%, P < 0.01). The reasons for not following recommendations due to lack of time or disagreement with the recommendation in general or for a specific patient visit decreased from the pre-intervention to post-intervention survey although more clinicians reported an increase in the recommendation not being received concurrently with the patient visit during the post-intervention survey. Terrell et al. (2009)25 assessed prescribing alerts that targeted potentially inappropriately prescribed medications for elderly patients in the emergency department and reported that providers accepted only 43% of the recommendations, which included recommendations for alternative treatment.

From the research included in this section, we concluded that evidence suggests that high levels of acceptance (at a rate greater than 75%) of recommendations from CDSSs are the exception98,170,178 rather than the rule. We recognize, however, that many of the successful CDSS studies did not assess user acceptance but still showed that systems were effective, implying that they were accepted and used. In the 19 studies (79.2%) that reported provider acceptance rates, 9 studies from the academic and community ambulatory settings found rates of acceptance between 50 and 75 percent of locally developed CDSSs that automatically delivered system-initiated (push) recommendations to providers synchronously at the point of care.22,86,120122,158,170,173,174,178 This observation showing provider acceptance of CDSSs greater than 50 percent is supported by evidence from six studies that included an evaluation period longer than 1 year 22,86,120122,174,178 and five studies that were evaluated with more than 2000 patients.22,86,98,120122 Further, only two of those studies demonstrating provider acceptance greater than 50 percent were evaluated with more than 100 providers— McDonald et al.22 included 130 providers (115 residents, 11 faculty member physicians, 4 nurses), and Rothschild et al.158 included 453 junior house staff (first-, second- and third-year residents). Notably, Dykes 2010 et al.98 was the only study that demonstrated provider acceptance of the recommended action greater than 50% that was published after 2008. While representing only a limited subset of studies, in these studies there was no significant effect of a mandatory clinician response on provider acceptance.86,178 Five studies captured some of the reasons clinicians did not accept the recommendations, citing disagreement with the recommended action for that specific visit, 45,70 clinical judgment based on the patient’s medical history,79,127 lack of facilities to fulfill lifestyle and relaxation recommendations,164 lack of time,70,127 incorrect drug or disease information,127 and not clinically important.127

Health care provider satisfaction. We identified 19 of the 148 eligible studies (12.8%) that specifically examined health care provider satisfaction with CDSSs/KMSs. These studies are summarized in Table I-15 of Appendix I.

Of these 19 studies, 12 (63.2%) were conducted in the U.S., 3638,47,68,70,81,110,117,119,124,126,173 5 (26.3%) in Europe,32,90,91,103,118,167,168 1 (5.3%) in multiple countries,171 and 1 (5.3%) with a location not reported.112 Four of the studies (21.1%) were implemented in an academic setting,36,37,90,91,124,173 eight (42.1%) in a community setting,32,47,68,81,103,110,117,118 four (21.1%) in both academic and community settings,112,119,126,167,168 two (10.5%) in a VA setting,38,70 and one (5.3%) for which the setting was not reported.171 One study (5.3%) evaluated the systems in the inpatient environment,36,37 14 (73.7%) in the ambulatory environment,32,47,68,70,81,103,110,117119,124,126,167,168,173 2 (10.5%) in both inpatient and ambulatory environments,38,112 1 (5.3%) in the emergency department,90,91 and 1 (5.3%) for which the environment was not reported.171 Duration of the evaluation period across the studies ranged from 3 months171 to 4.5 years.38 Twelve interventions (63.2%) were implemented using a system developed within the specific health care organization,32,3638,70,81,103,110,117,124,126,167,168,173 5 (26.3%) were implemented using a commercially available system,47,68,118,119,171 and 2 studies (10.5%) with a source that was not clearly described.90,91,112 Three systems (15.8%) aided health care providers with tasks for diagnosis,47,81,90,91 7 (36.8%) for pharmacotherapy,38,110,112,119,124,126,167,168 4 (21.1%) for chronic disease management,32,47,