NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Shojania KG, Ranji SR, Shaw LK, et al. Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies (Vol. 2: Diabetes Care). Rockville (MD): Agency for Healthcare Research and Quality (US); 2004 Sep. (Technical Reviews, No. 9.2.)

Cover of Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies (Vol. 2: Diabetes Care)

Closing the Quality Gap: A Critical Analysis of Quality Improvement Strategies (Vol. 2: Diabetes Care).

Show details

3Results

Search Yield and Results of Article Review Process

Figure 1 depicts the article search and review process, with the results at each step. The MEDLINE® search using PubMed® yielded a total of 3,601 citations. Searching the EPOC database produced an additional 104 articles deemed relevant for full abstraction, of which 12 articles104115 reporting 16 trials, met full inclusion criteria. (Conversely, the MEDLINE® search identified 10 articles82, 87, 89, 116122 evaluating 11 trials, which were not indexed in the EPOC database.) The manual search yielded an additional 77 articles, though only two of these23, 24 met the inclusion criteria for this review.††

Figure 1. Search Strategy and article review process.

Figure

Figure 1. Search Strategy and article review process. Figure 1 Legend EPOC = Cochrane Effective Practice and Organization of Care database, described above, contains the results of extensive electronic searches of multiple (more...)

A total of 529 articles merited full-text review. Of these, 139 were deemed “not a quality improvement or not an evaluation” and were excluded. This relatively large number of articles outside the scope of the review reflects the fact that 97 of the citations contained no abstract, and so they could not be screened out at Stages 1 or 2. Other reasons for exclusion after full text review included: excluded topic (22 articles); study design failed to meet the criteria for RCTs, quasi-RCTs, CBAs, or ITS (176 articles); no eligible outcomes (42 articles); duplicate or overlapping articles (14 articles); publication prior to 1980 (three articles); and other reasons (six articles), including one abstract not yet published or available as a manuscript from the authors.123 One published article could not be obtained.124, 125 (Figure 1 identifies all six of the articles excluded as “other,” and Appendix G lists all citations excluded after the full text review, along with reasons for the exclusions.)

As shown in Figure 1, a total of 126 articles merited full abstraction. The intervention in 68 of these articles consisted solely of patient education or promotion of self-management. These articles are listed in Appendix D and their results will likely be analyzed in another volume of this series that focuses on patient education and self-management. Those articles involving patient education in combination with other interventions were included in the present review. The study sample for the review consisted of 58 articles, reporting a total of 66 comparisons (Table 1).

Features of the Included Studies

Demographics. Table 1 displays the included 66 trials, along with descriptions of the trials with regard to setting, design, and QI strategies. (Appendix E presents structured summaries of the results for each of these studies.) Roughly half (29 articles reporting 34 comparisons) were published in the 1990s, 88, 96, 97, 106, 108110, 115, 116, 120, 122, 126144 while 24 articles (reporting 26 comparisons) were published in 2000 or later.82, 87, 89, 98, 104, 107, 112114, 117119, 121, 145155 Only five articles (reporting six comparisons) were published in the 1980s.105, 111, 156158 Roughly half of the comparisons (36 studies; 55% overall) were conducted in the United States. Twelve studies 82, 87, 104, 126, 129, 130, 134, 140, 142144, 154 (involving 13 comparisons) selected patients with poor glycemic control, poor adherence to medications or clinic attendance, the presence of specific comorbid conditions (e.g., hypertension, hyperlipidemia, coronary artery disease, obesity, tobacco use), or advanced illness (e.g., specific and previously documented diabetic complications such as nephropathy, neuropathy, or retinopathy).

Many studies omitted key data elements. For instance, among the 33 studies reporting mean reductions in HbA1c, 10 articles (reporting 12 comparisons) did not provide standard deviations or standard errors of the mean,88, 89, 112, 121, 137, 141, 143, 152, 153, 157 and four studies provided no baseline HbA1c values for either study group88, 107, 120, 156 Similarly, among the 26 studies reporting at least one measure of provider adherence, nine articles (reporting 10 comparisons) included no baseline values for any of the adherence outcomes.110, 111, 119, 120, 122, 127, 130, 154, 155

Methodologic features. Forty seven (71%) of the included trials had a randomized design,82, 8789, 97, 98, 104, 106, 107, 110, 112, 114117, 121, 126133, 135, 136, 139, 140, 143150, 152157 while one used a quasi-randomized design,96 and another 18 were controlled before-after studies.105, 108, 109, 111, 113, 118120, 122, 134, 137, 138, 141, 142, 151, 158 The investigators identified numerous uncontrolled before-after studies, but none of these met the adopted EPOC criteria for inclusion as a time series, which mandates data from at least three time points in the pre- and post-intervention periods.159 Thirty-five of the included trials involved clustering (i.e., unit of analysis differed from unit of allocation‡‡).8789, 96, 105, 107110, 113115, 119, 121, 122, 129, 132, 137, 138, 141, 145, 147149, 151153, 155, 157, 158 The number of clustered units per trial ranged from a low of one clinic, team, or firm per study group96, 109, 114, 119, 137, 153 to a high of 247 (with clinicians as the unit of allocation).149 Only three clustered trials reported ICC values.8789 One of these studies87 reported ICC values for each of the seven adherence outcomes identified in the study. These values ranged from 0.02 to 0.33, and the researchers used the median (0.18) to calculate the effective sample size for the adherence outcomes in this study. The additional two studies reported ICC values for measures of disease control. In one case,88 the ICC = 0.045 and 0.047. The other89 used a value of 0.07 in performing the power calculation, though the rationale for choosing this number was unclear. Since the remaining studies did not report ICC values, the investigators assigned values based on data from the literature.67

Reported outcomes. The included studies reported a wide range of specific outcomes, with 51 studies reporting at least one measure of disease control and 26 reporting at least one measure of provider adherence. Only nine studies meeting the inclusion criteria (which required studies to report at least one measure of disease control or provider adherence) reported patient adherence outcomes. Thirty eight studies reported changes in serum HbA1c in the format of mean and standard deviation for each study group, while 13 studies reported other measures related to changes in serum HbA1c (e.g., the percentage of patients with serum HbA1c falling within a certain range). Twenty-one studies reported an outcome involving blood pressure control, although only 15 reported the mean systolic or diastolic blood pressure, and just eight reported sufficient information to permit quantitative analysis (i.e., a standard deviation in at least one of the measurement periods for each study group and mean values from both periods).

Because of the variety of adherence outcomes, the investigators focused the analysis on a summary measure of provider adherence. As described in the Methods section, they calculated for each study the net change attributable to the intervention for all adherence outcomes reported and then used the outcome with the median effect as the data contributed by that study for this summary adherence measure. (Although nine studies included outcomes related to patient adherence, they exhibited little overlap in terms of the type of adherence outcome reported or the QI strategies employed. Consequently, patient adherence outcomes were not included in the present analysis.)

Provider adherence outcomes were reported in 26 studies, however only 17 had sufficient data to permit quantitative analysis. Adherence outcomes related to appropriate monitoring of serum HbA1c in 19 comparisons, management of hypertension or coronary artery disease in 14 studies, and to monitoring of laboratory values other than serum HbA1c or glucose in 20 comparisons (e.g., serum lipids, urine microalbumin). Thirteen studies reported adherence outcomes related to screening for diabetic complications of the foot, and 14 studies measured provider adherence in connection with screening for ophthalmologic complications.

Among the disease control outcomes, the analysis focused on changes in serum HbA1c (results for studies reporting usable data on changes in blood pressure are shown in Appendix E, Table E2). All outcomes were standardized so that a positive number for any change (or for any coefficient in the regression analysis) corresponds to an improvement, while a negative value reflects an undesirable change. As described previously, all outcomes involving changes in mean serum HbA1c refer to the net reduction in serum HbA1c (i.e., the intervention was associated with a positive reduction in HbA1c, as desired). Adherence outcomes also were standardized so that adherence was measured in terms of the desired process in such a way that positive changes always reflect an improvement in care.

Types and numbers of quality improvement strategies. As shown in Table 2a, the most common type of QI intervention fell into the broad category of organizational change (40 comparisons including 24 RCTs), followed by patient education (28 comparisons, 23 RCTs), and provider education (24 comparisons including 16 RCTs). Apart from studies161164 of incentives directed at patients for the purpose of reinforcing patient education or self-care, only one study131 evaluated financial incentives. (These other studies161164 will be included in the forthcoming volume of QI strategies focused on patient education and self-management.) The investigators reported results for the specific QI strategies, including analyses collapsing some of the categories with clear overlap (e.g., patient education, promotion of self-management, and patient reminders).

Table 2a. Number and design of included studies for each quality improvement strategy.

Table 2a

Number and design of included studies for each quality improvement strategy.

Table 2b depicts the number of different QI strategies examined per study. Fourteen studies evaluated single-component interventions. Fifty-two trials involved interventions employing more than one of the nine QI strategies in the taxonomy, with five being the maximum number of strategies involved in any single intervention.117, 133, 148, 150, 157

Table 2b. Number of quality improvement strategies per study intervention.

Table 2b

Number of quality improvement strategies per study intervention.

The median number of QI strategies per intervention was two. Though not a QI strategy per se, the researchers also abstracted information on the role played by clinical information systems in the trials.

Analysis by Outcome Measures

Trial design was a significant negative predictor of effect size for both outcomes (i.e., trials with a randomized design reported smaller improvements in glycemic control, and for provider adherence). For studies reporting impacts on glycemic control, sample size also exhibited a significant inverse correlation with the magnitude of effect (i.e., the larger studies showed smaller reductions). This inverse correlation persisted among the randomized trials and suggested an independent effect of sample size, rather than confounding due to a tendency of larger trials to have a randomized design. Consequently, the tables showing the associations between QI strategies and targeted outcomes are stratified generally by sample size for glycemic control, and by study design for both outcomes.

Publication bias, to be discussed below, provides the most likely explanation for the striking patterns shown in the tables stratifying reported results by sample size and by trial design. Thus, in addition to illustrating the magnitude of the relationships between effect sizes and study features, the tables also summarize the likely effect sizes of each QI type based on the subset of studies with the least apparent bias.

Effect of QI Strategies on Glycemic Control

Table 3a shows the median reductions in serum HbA1c achieved by interventions employing different QI strategies. Among the 38 comparisons with sufficient data regarding changes in mean serum HbA1c in the study and control groups, the median effect on serum HbA1c was an absolute reduction of 0.48% (IQ range: 0.20%, 1.38%). For specific QI strategies, trials that included provider education (alone or in combination with other QI strategies) had the highest median effect, with a median absolute decrease in serum HbA1c equal to 1.10% (IQ range: 0.56%, 1.50%). Trials that included promotion of self-management showed the lowest median reduction in serum HbA1c (0.40%; IQ range: 0.20%, 0.60%).

Table 3a. Association between type of quality improvement strategy and glycemic control stratified by study sample size*.

Table 3a

Association between type of quality improvement strategy and glycemic control stratified by study sample size*.

Stratifying the results by study sample size shows that the larger studies reported generally smaller effects. For instance, the 10 trials in the lowest quartile of sample size reported a median reduction in serum HbA1c of 1.35% (IQ range: 0.81%, 1.73%), while the 10 trials in the highest quartile reported a median reduction of only 0.10% (IQ range: 0.10%, 0.33%). Similarly, the 19 trials falling in the lower two quartiles of sample size reported a median reduction in serum HbA1c of 1.30% (IQ range: 0.41%, 1.49%), while the 19 trials in the upper two quartiles reported a median reduction of only 0.21% (IQ range: 0.10%, 0.55%). As shown in Table 3a, this pattern was consistent for all but one of the QI strategies. (Provider reminders represent the one exception, as reported reductions in serum HbA1c were approximately equivalent for studies in the upper and lower quartiles of sample size.)

Table 7a confirms the inverse relationship suggested by visual inspection as statistically significant. Specifically, the Spearman rank correlation coefficient for the relationship between sample size and improvement in serum HbA1c was -0.46 (95% CI: -0.09, -0.72; p=0.02). This relationship does not appear to reflect confounding based on trial design, as the same correlation existed among randomized trials (Spearman correlation coefficient = -0.48; 95% CI: -0.03,-0.77; p=0.04).

Table 7a. Regression results for impact of general study features on glycemic control and provider adherence.

Table 7a

Regression results for impact of general study features on glycemic control and provider adherence.

The relationship between sample size and reported effects strongly suggests publication bias, such that smaller studies reporting negative or less impressive results are not as likely to be published. For this reason, the authors have presented the results for impacts on glycemic control stratified by quartiles of sample size wherever possible. (The issue of publication bias is explored further in the Discussion section.)

Effect of QI Strategies on Provider Adherence

Table 3b exhibits the median improvements in the summary measure of provider adherence achieved by interventions employing the various QI strategies. Among the 17 trials with sufficient data regarding changes in provider adherence, the achieved median effect was an attributable 4.9% increase in adherence (IQ range: 3.8%, 15.0%). Self-management appears to have the largest median effect in Table 3b, but this result reflects a single study152 in which the intervention also included components of patient education and organizational change. (The issue of other interventions as a potential confounding presence is addressed in the analysis of specific QI types.)

Table 3b. Association between improvements in provider adherence* and type of quality improvement strategy stratified by study sample size†.

Table 3b

Association between improvements in provider adherence* and type of quality improvement strategy stratified by study sample size.

Apart from self-management, the largest median effects on provider adherence were associated with provider education and audit and feedback. Among the 11 trials with some component of provider education, either alone or in combination with other QI strategies, the median increase in adherence was 5.6% (IQ range: 4.15%, 17.2%). Nine trials utilizing audit and feedback also achieved a 5.6% median increase in adherence (IQ range: 3.4%, 16.4%).

Visual inspection of Table 3b suggests no striking variation in the effect sizes across quartiles of sample size, and Table 7a confirms the correlation to be smaller than for studies reporting changes in serum HbA1c and lacking in statistical significance (Spearman correlation coefficient = -0.22; 95% CI: 0.29, -0.63; p=0.4). However, trial design was a highly significant predictor in the regression analysis (p=0.0008; Table 7a). Based on the mean post-intervention differences between the study and control groups, the parameter estimate of -0.30 for the impact of trial design implies that randomization generally decreased the improvement in provider adherence associated with an intervention by 15.3% (see text accompanying Table 7a).

Table 4a bears out this relationship by showing the median effects associated with randomized and non-randomized trials, both overall and for each QI strategy. For those studies detailing impacts on glycemic control, RCTs reported a median reduction in serum HbA1c of 0.39% compared with 1.40% for non-randomized trials, and this difference was statistically significant (p=0.008 for Mann-Whitney test; Table 7c). This difference diminished only slightly with the restriction of the analysis to trials in the upper two quartiles of sample size (p=0.03; data not shown). Table 4b bears out the persistence of a relationship between sample size and magnitude of reductions in serum HbA1c, even among randomized trials.

Table 7c. Significance tests (Mann-Whitney) for median effects associated with selected methodologic features and QI strategies.

Table 7c

Significance tests (Mann-Whitney) for median effects associated with selected methodologic features and QI strategies.

Table 4b. Impacts on glycemic control and provider adherence stratified by trial design and sample size*.

Table 4b

Impacts on glycemic control and provider adherence stratified by trial design and sample size*.

Effect of the Number of QI Strategies per Intervention

Trials utilizing combinations of QI strategies were more likely to exert a positive effect. The six trials involving single-faceted interventions had no overall effect on glycemic control (Table 5a). The median reduction in serum HbA1c reported by these trials was 0.00 (IQ range: -0.08, 0.16) (i.e., some studies reported increases in serum HbA1c). By contrast, the 32 trials involving interventions with at least two strategies reported a median absolute reduction in serum HbA1c of 0.60% (IQ range: 0.30%, 1.40%). The Mann-Whitney test comparing these median effects was statistically significant at p=0.01 (Table 7c). The difference diminished only slightly when the analysis was restricted to trials in the upper two quartiles of sample size (p=0.03; data not shown).

Table 5a. Association between improvement in glycemic control and number of quality improvement strategies stratified by study sample size*.

Table 5a

Association between improvement in glycemic control and number of quality improvement strategies stratified by study sample size*.

The relationship between number of QI strategies and magnitude of effect was slightly weaker for provider adherence (Table 5b). The 14 trials involving interventions with at least two QI strategies were associated with a median increase in provider adherence of 5.3% (IQ range: 4.5%,16.1%) compared with the three single-faceted trials, which reported a median increase in adherence of 3.0% (IQ range: 2.0%,3.5%). The Mann-Whitney test for this comparison suggests rejection of the null hypothesis (p=0.04; Table 7c), that these two medians are the same. This p-value would not withstand correction for multiple comparisons, but the beneficial impact of multifaceted interventions was one of our a priori hypotheses (as described in the Methods section).

Table 5b. Association between improvement in provider adherence and number of quality improvement strategies stratified by study sample size*.

Table 5b

Association between improvement in provider adherence and number of quality improvement strategies stratified by study sample size*.

For trials reporting impacts on glycemic control, the apparent superiority of interventions with more than one QI strategy persisted when the analysis was restricted to randomized trials (Table 6). The 23 randomized comparisons of interventions involving at least two distinct QI strategies reported a median reduction in serum HbA1c of 0.41% (IQ range: 0.25%, 0.94%) attributable to the intervention, as opposed to 0.00% (IQ range: 0.10%, 0.00%) for the five randomized comparisons involving single-faceted interventions. Even with the relatively small number of studies involved, these medians are unlikely to be drawn from the same population (p=0.008 for Mann-Whitney test). Among trials in the upper two quartiles of sample size, interventions with at least two strategies retained their association with greater median reduction in serum HbA1c, though the difference between the medians was less striking (p=0.01; Table 7c). The greater impact of interventions with at least two QI types persisted even when the analysis was restricted to interventions involving organizational change, which would generally be regarded as more complex and more intense than other QI types. In other words, even in studies of single-faceted interventions employing a form of organizational change as the sole QI strategy, the addition of at least one more strategy was found to increase the overall effect (0.71% vs. 0.05% median reduction in serum HbA1c; p=0.002).

Table 6. Associations between number of quality improvement strategies and improvements in glycemic control and provider adherence stratified by trial design.

Table 6

Associations between number of quality improvement strategies and improvements in glycemic control and provider adherence stratified by trial design.

The investigators further explored the relationship between number of QI strategies and magnitude of effects using an alternate classification scheme in which important subtypes of provider education and organizational change were treated as their own category. Specifically, the broad category of provider education was replaced by three categories—workshops or meetings,55 distribution of educational materials,165 and educational outreach56—and organizational change was replaced by four strategies—disease or case management,17 changes to team structure or personnel, modification of medical records systems, and “other organizational change.” This alternate classification scheme resembles that used in the Cochrane review of QI interventions for diabetes care,16 and is more consistent with other reviews focusing on these specific strategies.17, 55, 56, 165

Under this alternate classification of the QI strategies, five studies still were catagorized as single-faceted,104, 112, 116, 121, 127 but the median number of strategies increased from two to three and the maximum number of strategies increased from five to six.153, 157 Appendix H (Tables H4ac) presents the same relationships discussed above and shown in Tables 5a, 5b, and 6, but using the results of this alternate classification in which major substrategies are promoted to their own category of QI strategy (Tables H4a, H4b, and H4c). The relationship between increased numbers of QI strategies and magnitude of effects appears somewhat stronger for studies reporting impacts on glycemic control. Though the analysis is not shown, the Mann-Whitney test of the difference in median effects between single and multifaceted interventions using the alternate classification scheme had greater significance (p=0.005).

Analysis by Type of QI Strategy

Visual inspection of Tables 3a, 3b, and 4a suggests no striking differences among the various QI strategies. However, studies that included provider education or audit and feedback, alone or in combination with other strategies, were among those associated with the largest effects on both outcomes. The effects also exhibited less erosion with stratification by sample size or by trial design.

Tables 8a through 11b show the median effects on glycemic control and provider adherence for specific QI strategies. These tables attempt to address several particular limitations inherent to this analysis:

1.

Several of the nine major QI strategies in our taxonomy may include substrategies that are sufficiently distinct to warrant their own category. To address this heterogeneity within some of the QI strategies, the tables of specific QI strategies compare specific substrategies to the overall set of QI strategies—not just other strategies within the same category. For example, “disease management” is compared with all other interventions, and not just interventions designated as “organizational change.” Similarly, educational meetings are compared with all other strategies, not just those designated as having some component of provider education. (Provider education and organizational change are analyzed in this manner in Tables 8a, 8b, and 11a, 11b, respectively.)

2.

Several of the categories currently defined as separate QI strategies may overlap with other QI strategies such that they might reasonably have been designated a substrategy within those categories (e.g., patient education might subsume promotion of self-management or even patient reminders; provider reminders might subsume facilitated relay of clinical data to providers). Tables 9a and 9b address this issue for patient education by presenting median effects for various ways of collapsing patient education, self-management, and patient reminders. Tables 10a and 10b present similar analyses for provider reminders and the facilitated relay of clinical data.

3.

The apparent benefit of any particular strategy is confounded by the presence of other strategies in the same intervention. Comparing interventions with a particular strategy to those interventions with no such component provides some estimate of the attributable effect of a given strategy (e.g., the median effect of all studies with provider education versus the median effect of all interventions with no component of provider education). Nevertheless, no definitive statements can be made about the effects of individual QI strategies because most studies used more than one strategy. The researchers also performed linear regression as a means of assessing the relative benefits of a particular QI strategy.

Table 8a. Association between improvements in glycemic control and specific substrategies of provider education stratified by study sample size*.

Table 8a

Association between improvements in glycemic control and specific substrategies of provider education stratified by study sample size*.

Table 11b. Association between improvements in glycemic control and specific substrategies of organizational change stratified by study design.

Table 11b

Association between improvements in glycemic control and specific substrategies of organizational change stratified by study design.

Table 8b. Association between improvements in glycemic control and specific substrategies of provider education stratified by study design.

Table 8b

Association between improvements in glycemic control and specific substrategies of provider education stratified by study design.

Table 11a. Association between improvements in glycemic control and specific substrategies of organizational change stratified by study sample size*.

Table 11a

Association between improvements in glycemic control and specific substrategies of organizational change stratified by study sample size*.

Table 9a. Association between improvements in glycemic control and specific substrategies of patient education stratified by sample size*.

Table 9a

Association between improvements in glycemic control and specific substrategies of patient education stratified by sample size*.

Table 9b. Association between improvements in glycemic control and specific substrategies of patient education stratified by study design.

Table 9b

Association between improvements in glycemic control and specific substrategies of patient education stratified by study design.

Table 10a. Association between improvements in glycemic control and specific substrategies of provider reminder stratified by sample size*.

Table 10a

Association between improvements in glycemic control and specific substrategies of provider reminder stratified by sample size*.

Table 10b. Association between improvements in glycemic control and specific substrategies of provider reminder stratified by trial design.

Table 10b

Association between improvements in glycemic control and specific substrategies of provider reminder stratified by trial design.

Provider Education

Interventions with some component of provider education, alone or in combination with other QI strategies, produced significantly larger improvements in glycemic control. Such interventions had a median absolute reduction in serum HbA1c of 1.10% (IQ range: 0.56%,1.50%), compared with 0.40% (IQ range: 0.10%, 1.08%) for interventions with no provider education component. The Mann-Whitney test suggests that these two medians are unlikely to be equivalent (p=0.02; Table 7c). But since the impact of provider education does not have a specific relation to any of the a priori hypotheses, this p-value would have to be adjusted for multiple hypothesis testing, in which case it would lose its significance (p= 0.2–0.3 if the number of comparisons were taken to be 10–15).

Visual inspection of Table 8a gives the impression that the greater effects associated with interventions having some component of provider education, compared with those with no such component, persisted among larger studies (1.50% median reduction in serum HbA1c vs. 0.20%). This comparison loses its significance, however, even without adjusting for multiple hypothesis testing (p=0.30 for Mann-Whitney test). Restriction of the analysis to RCTs also results in loss of significance (p=0.06 before adjustment for multiple comparisons), despite the persistent appearance of a larger effect in Table 8b.

Interventions involving provider education also reported greater improvements in provider adherence than did interventions without any educational component for providers. But the relative difference was less striking than for glycemic control, and it diminished when the analysis was restricted to RCTs (Table 8b). Interestingly, provider education also was the only QI strategy to have even borderline significance in the regression analysis (Table 7b), with a coefficient of 0.25 (95% CI: 0.00, 0.51%; p = 0.05). At the same time, a specific benefit for provider education did not relate to any of the a priori hypotheses, so this p-value requires adjustment for multiple comparisons and would remove the appearance of a significant result.

Table 7b. Regression results for impacts of quality improvement strategies by strategy type and by the number of strategies per intervention*.

Table 7b

Regression results for impacts of quality improvement strategies by strategy type and by the number of strategies per intervention*.

As acknowledged in the preceding section, the designation of “provider education” as a single category—including components as diverse as workshops and conferences, educational outreach, and distribution of printed materials—is somewhat arbitrary. These components were compared to the overall set of QI strategies, and not just to other strategies with some element of provider education, as if educational meetings and dissemination of educational materials were regarded as their own categories. Using this more general approach to the comparisons (Tables 8a and 8b), interventions with educational meetings or workshops appeared more effective than interventions without them. This also is true of interventions involving the distribution of educational materials, compared with those lacking the distributed materials component (p=0.03 and p=0.06, respectively). Only one study of educational outreach reported effect on glycemic control in a format compatible with this analysis, preventing meaningful comparisons with interventions lacking educational outreach.

Putting aside the issue of multiple hypothesis testing, the small numbers of studies make confounding by the presence of other interventions a significant possibility. The one study of educational materials, for example, that fell in the upper two quartiles of sample size involved a fairly intensive case management intervention.109 There is no way to assess the impact attributable to the component involving educational materials.

Confounding by the presence of other interventions is still quite probable, even with larger numbers of comparisons. Across all sample sizes, for instance, eight comparisons89, 96, 109, 115, 153, 157 evaluated interventions involving educational materials distributed to providers. These comparisons reported a median reduction in serum HbA1c of 0.91% (IQ range: 0.52%,1.48%) compared with 0.40% (IQ range: 0.1%, 1.25%) for the 30 interventions involving no distribution of educational materials. The eight comparisons with interventions including educational materials involved problem-based learning in one trial judged to involve no other QI strategies,96 a multifaceted intervention including components of audit and feedback and disease/case management in addition to provider education,153 and patient reminders and disease or case management in addition to provider education.109 Other comparisons included elements of patient education, patient reminders, and provider reminders in addition to provider education,157 a Web-based decision support tool,89 and an intensive, multifaceted intervention including benchmarking, computerized decision support, and frequent interaction with participating patients and providers.135 Thus, even among these eight interventions involving the distribution of educational materials and also reporting an effect on glycemic control, the apparent benefit of educational material actually might reflect the benefits of the other intervention components involved.

Patient Education, Promotion of Self-management, and Patient Reminders

Tables 9a and 9b present the median effects on glycemic control and provider adherence for patient education, promotion of self-management, and patient reminders, alone or in combination with other QI strategies. The 18 trials involving patient education achieved a median reduction in HbA1c of 0.70% (IQ range: 0.34%, 1.45%), compared with the median reduction of 0.39% (IQ range: 0.10%, 0.81%) seen in the 20 studies with no patient education component (Mann-Whitney p=0.08). The regression analysis detected no significant effect for patient education, in terms of glycemic control or the summary measure of provider adherence.

The investigators were unable to detect any important effects in the analysis for self-management or patient reminders. Collapsing the patient education, patient reminders and promotion of self-management strategies into a single, broad category added nine more studies but left the median effect relatively unchanged at 0.8% (IQ range: 0.33%, 1.44%). Moreover, self-management and patient reminders—separately or as a collapsed category—produced roughly the same median effects as interventions without any of these QI strategy components.

Provider Reminders and Facilitated Relay of Clinical Data

Among comparisons of all sizes, neither provider reminders nor facilitated relay of clinical data to providers achieved results substantially different from all other QI strategies, or all comparisons without any component of either of these strategies (Table 10a). Among larger studies, both strategies achieved marginally increased reductions in serum HbA1c compared to interventions without these strategies, but this benefit disappeared for RCTs (Table 10b). Neither strategy produced any apparent benefit for provider adherence beyond what was achieved by all other strategies, and by all interventions without components of either of these two strategies.

Audit and Feedback

The five trials utilizing audit and feedback137, 145, 153, 157 reported a median reduction in HbA1c of 0.71% (IQ range: 0.41%, 1.40%), compared with 0.47% (IQ range: 0.20%, 1.30%) for trials absent audit and feedback—though these medians are unlikely be different (p=0.5 for Mann-Whitney test). The improvement in provider adherence seen with audit and feedback (median improvement of 5.6% [IQ range: 3.4%, 16.4%]) also was superior to that achieved by interventions without any audit and feedback component (median improvement of 4.5% [IQ range: 4.0%, 5.1%]), though this difference also was non-significant (p=0.4).

The results for audit and feedback also may illustrate another form of publication bias related to quality of reporting, rather than sample size or trial design. Of the five comparisons (reported in four publications137, 145, 153, 157) evaluating the impact on glycemic control of an intervention involving audit and feedback, four comparisons137, 153, 157 reported no standard deviations for any of the reported serum HbA1c group means. (Consequently, the regression coefficient shown in Table 7b reflects the comparison of the post-intervention effect size for a single study of audit and feedback145 with the 26 other comparisons involving no component of audit and feedback). The single comparison that provided sufficient data to warrant inclusion in the regression analysis reported a net reduction in serum HbA1c attributable to the intervention of only 0.10% (from this single study), which appears significantly lower than the median reduction of 0.47% (95% CI: 0.24%, 0.99%) associated with all studies lacking any component of audit and feedback.

The researchers were unable to adequately capture any objective measure of “intensity” for audit and feedback (or any QI strategy), and therefore did not adjust for any such measure in the analysis. Consequently, it is possible that the single trial of audit and feedback included in the regression analysis was a particularly low intensity form of this general strategy and/or the 26 interventions with no audit and feedback component involved some high intensity versions of QI strategies other than feedback. The former possibility appears unlikely, as the aforementioned comparison of audit and feedback145 also involved a computerized decision support system used to guide physicians in matters of diagnostics, history recording, the physical exam, additional tests, and treatment, as well as providing recommendations for key management decisions. Nevertheless, any inferences regarding the relative benefit (or lack thereof) of audit and feedback, compared with the merits of other QI strategies, would be highly speculative given only one (or even five137, 145, 153, 157) trials as a basis for comparison.

Organizational Change

Organizational changes were present in 27 of the 38 comparisons reporting changes in mean serum HbA1c, but the investigators were able to calculate a value for the summary measure of provider adherence in only six of the 17 trials (Tables 11a and 11b). While organizational change as a broad category had little apparent impact on glycemic control, it was the disease or case management and changes to the existing medical record system (e.g., implementation of a specialized diabetes patient registry, or a more general electronic medical record) strategies that achieved median reductions in serum HbA1c notably greater than the interventions absent these strategies. This appeared to be the case across the entire sample of studies and in the subset of interventions that included some component of organizational change. For changes to the medical record (e.g., implementation of a clinical information system), the five studies classified as implementing this type of organizational change reported a median reduction in serum HbA1c of 1.40% (IQ range: 1.40%, 1.90%), compared with a median reduction of 0.40% (IQ range: 0.10%, 0.80%) for the 33 trials with no such component. This comparison was judged significant after the Mann-Whitney test was applied to the two medians (p=0.007; Table 7c), however the five trials involving this type of organizational change all appeared in the lower two quartiles of sample size (Table 11a).

The comparison of the median effects associated with interventions involving disease management and those interventions without any such component appeared to have a significant impact on the median reduction in serum HbA1c (p=0.009; Table 7c). But this appearance of statistical significance would not endure an adjustment for multiple hypothesis testing (given the 10–15 basic comparisons made). It should be noted that a trend towards a significant difference persisted among non-randomized trials in the upper two quartiles of sample size (p=0.003), although less so among randomized trials (p=0.06 without adjusting for multiple comparisons).

Adding disease management to an intervention was associated with less substantial incremental improvement in provider adherence (Table 11b). Because of the inverse associations with trial design and study period (discussed below), the researchers repeated the regression analysis of the impact of disease management with inclusion of trial design and study year as predictors. This analysis, however, left the parameter estimate and associated confidence interval relatively unchanged (data not shown). That disease management had little impact on provider adherence is perhaps to be expected, given the focus on structured followup and patient management, rather than aspects of provider behavior.

Only one trial examined changes to the medical record with provider adherence, and it reported less improvement than did studies without this intervention component. Changes to team personnel or structure produced unimpressive effects on glycemic control. Only two studies employed this intervention component and also reported provider adherence. The single RCT did achieve a larger improvement in generalized provider adherence than the 13 RCTs without changes to team structure or personnel, while the non-randomized trial did not (Table 11b).

Additional Analyses-Clinical Information Systems

Clinical information system is a broad term encompassing systems performing a wide variety of functions. A general feature that serves to distinguish clinical information systems from administrative information systems is that the former require data entry or data retrieval by clinicians at the point of care.166 The researchers identified interventions using a clinical information system for any of the following purposes: trial participant identification or enrollment, provider reminder delivery, clinical decision support, provider-to-provider communications enhancements, or clinical performance auditing.

As shown in Tables 12a and 12b, 20 trials (30%) involved some role for a clinical information system, though this role was limited to identifying or enrolling eligible participants in 6 of the trials.121, 129, 133, 143, 152, 155 Interventions that used a clinical information system in at least one such capacity achieved greater reductions in glycemic control than did interventions in which this component played no role (Table 12a). The difference, however, was not statistically significant (p=0.10 for Mann-Whitney test; Table 7c), even without adjustment for multiple comparisons. Moreover, even the appearance of a benefit for interventions with some role for a clinical information system diminished substantially in larger studies and those with a randomized design (Tables 12a and 12b).

Table 12a. Association between improvements in glycemic control and various roles for clinical information systems stratified by quartiles of sample size*.

Table 12a

Association between improvements in glycemic control and various roles for clinical information systems stratified by quartiles of sample size*.

Table 12b. Association between improvements in provider adherence and glycemic control for various roles for clinical information systems stratified by trial design.

Table 12b

Association between improvements in provider adherence and glycemic control for various roles for clinical information systems stratified by trial design.

The roles listed above clearly have the potential to differ widely in their effect (e.g., provision of decision support, versus mere identification of eligible participants in the intervention). At the same time, focusing on the specific roles for this intervention suggested no apparent benefit for decision support, auditing clinical performance, or any of the other roles examined (Tables 12a and 12b).

Effects of Study Setting and Methodologic Features

Study Setting

Table 7a shows relationships between key study outcomes and country (e.g., U.S. versus non-U.S.), study period (i.e., the midpoint of the observation period for the trial), and patient selection. Patient selection refers to any explicit efforts to enrich the study population for more complex patients, defined in terms of comorbid conditions, presence of diabetic complications, problems with treatment adherence, or poor access to care (e.g., uninsured patients).

The only statistically significant finding among these relationships is a negative correlation between study period and provider adherence, meaning that more recent trials had a tendency toward smaller improvements in adherence (p= 0.004). The relatively low p-value would retain conventional statistical significance, even with correction for as many as 15 comparisons. More important, this finding relates to one of the investigators' a priori hypotheses. As stated in Methods section (Page 24), one reason for entertaining this hypothesis was that baseline adherence might have improved in response to past QI efforts, making further improvement more difficult. The mean baseline adherence in control and intervention groups across all studies was 51 ±18%. Although not depicted in Table 7a, baseline adherence did exhibit a substantial positive correlation with study period, with a Spearman rank correlation of 0.6 (p=0.006).

Methodologic Features

The most striking finding in Table 7a is the highly significant negative correlation between use of a randomized design and the generalized measure of provider adherence, with randomization reducing the effect size by roughly 30% (p=0.0007). Again, this p value would retain its statistical significance with adjustment for multiple comparisons. Moreover, this was another of the researchers' a priori hypotheses. As there were only three non-randomized trials reporting impacts on provider adherence,105, 108, 113 this statistical significance reflects a striking difference in effect size. As shown in Table 6, these three non-randomized trials reported a median improvement in provider adherence of 18% (IQ range: 17.2%, 21.0%), compared with a median of 4.5% (IQ range: 3.5%, 5.4%) for the 14 randomized comparisons.

A substantial negative correlation also existed between sample size and effect size (i.e., larger studies tended to show smaller effects), further confirming the trend toward larger effects for smaller studies seen in Tables 312. Also, as shown in Table 7a, the rank correlation coefficients for this relationship were -0.46 (p=0.02) and -0.22 (p=0.4) for glycemic control and provider adherence, respectively. While of smaller magnitude, the correlation for provider adherence still is noteworthy. (The statistically non-significant result likely reflects the small number of studies reporting this outcome.) Also, though not shown in Table 7a, the inverse correlation between sample size and effect size changed very little among RCTs alone. Only six of the RCTs had a cluster design, so the p-value lost statistical significance, but the magnitude of the correlation increased overall (Spearman rank correlation coefficient = -0.54; p=0.3).

The relatively low yield of the manual search likely reflects the existing contribution of hand searching to the EPOC registry. Also, a substantial number of articles identified by the manual search involved patient education or self-management only. While they did not meet inclusion criteria for the present review, they will likely meet criteria for inclusion in a forthcoming volume of this Series focusing on patient education and self-management.

Cluster trials allocate participants at one level (e.g. providers randomized to intervention or control group), but collect and analyze data at the level of individual patients or clinical encounters. Analyzing such studies at the patient level produces so-called unit of analysis errors 160 unless investigators adjust for correlation within each cluster. As described in Methods and illustrated in Appendix F, we calculated an “effective sample size” to adjust for clustering effects whenever the unit of analysis and unit of study group allocation were not the same.

Footnotes

††

The relatively low yield of the manual search likely reflects the existing contribution of hand searching to the EPOC registry. Also, a substantial number of articles identified by the manual search involved patient education or self-management only. While they did not meet inclusion criteria for the present review, they will likely meet criteria for inclusion in a forthcoming volume of this Series focusing on patient education and self-management.

‡‡

Cluster trials allocate participants at one level (e.g. providers randomized to intervention or control group), but collect and analyze data at the level of individual patients or clinical encounters. Analyzing such studies at the patient level produces so-called unit of analysis errors 160 unless investigators adjust for correlation within each cluster. As described in Methods and illustrated in Appendix F, we calculated an “effective sample size” to adjust for clustering effects whenever the unit of analysis and unit of study group allocation were not the same.

PubReader format: click here to try

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (5.4M)

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...