Key Findings and Strength of Evidence

The key findings of this review, based on 128 unique RCTs and observational studies, as well as five reviews, most of which were fair-quality, are summarized in Tables 20, 21, and 22 below. The factors used to determine the overall strength of evidence grades are summarized in Appendix J. Changes in overall prescribing were reported in all studies, while attempts to measure changes in appropriate or inappropriate prescribing were reported in nine studies (7%), and antibiotic resistance was reported in one study. In addition to the sparseness of reporting on the outcome of appropriate prescribing, the few studies that attempted to assess appropriate prescribing had important limitations in outcome definition and ascertainment methods and lack of consistency in methods across studies. Reporting on actual use of antibiotics by the patient was also rare; only studies of delayed prescribing report patient self-report of filling the prescription, with use assumed.

Table 20

Interventions with evidence for improving or reducing antibiotic prescribing in acute RTI and of not causing adverse consequences.

Table 21

Interventions with evidence for improving or reducing antibiotic prescribing in acute RTI and no or insufficient evidence on adverse consequences.

Table 22

Interventions with evidence of improved prescribing for acute RTI but mixed evidence of adverse consequences.

Based on the direction and strength of evidence for benefits (prescribing and/or resistance) and adverse consequences (e.g. reconsultations), interventions are grouped into four categories as described in the Methods section above: (1) Interventions with evidence of improved or reduced prescribing of antibiotics and evidence of not increasing adverse consequences; (2) Interventions with evidence of improved prescribing of antibiotics and no, insufficient, or mixed evidence on adverse consequences; (3) Interventions with evidence of no effect on prescribing of antibiotics; (4) Interventions with evidence of a negative effect on prescribing of antibiotics. For all outcomes, although we sought to determine whether strategies differed based on various patient, clinical, and contextual factors, this was not possible due to the potential confounding influences of a wide variety of other factors. No intervention had high-strength evidence. Given the large number of interventions to consider, those with insufficient evidence are not discussed here.

Differences in Outcomes According to Potential Moderators of Effect

Methods for Assessing Appropriate Prescribing

The methods for assessing appropriate prescribing fell into three categories: (1) ICD-9 codes or diagnostic category, (2) adherence to a specific guideline's recommendations for antibiotic use, and (3) duration of symptoms for pharyngitis or sinusitis. Although we sought to assess whether the definition of appropriateness affects the apparent effectiveness of interventions, this was not possible due to the potential confounding influences of a wide variety of other factors.

Intended Target of Intervention

The intended target of the interventions varied in the education interventions, where the reductions in prescribing were greater when the target was the patient or parent, and somewhat less when the target was the clinician or combined groups. However, direct comparisons were not available and the ranges in rates of reduction overlapped across the groups such that a clear pattern could not be established. It was clear that combining patient and clinician education did not result in clearly greater reductions. Clinical outcomes, including patient or parent satisfaction were not significantly affected. With interventions aimed at improving communication, only clinician-targeted interventions were found to have beneficial effects, although the patient-targeted evidence was very limited. Other interventions were either aimed only at clinicians (e.g., point-of-care tests), or always included both clinicians and patients (e.g., delayed prescribing).

Specific Acute Respiratory Tract Infections

The results for studies that either enrolled patients with specific acute RTIs, or reported results stratified by type of RTI, are presented in Table 23, below. Interventions with mixed results by RTI type were patient education (with evidence of effectiveness for pharyngitis but not for acute otitis media), clinician education (with evidence of effectiveness in acute otitis media and pharyngitis but not sinusitis), combined patient and clinician education (with evidence of effectiveness in bronchitis but mixed evidence for pharyngitis and sinusitis), and the addition of clinician communication training to guideline education (which was found effective for sinusitis but not for bronchitis). Three interventions were found to have a significant effect in improving antibiotic prescribing across three RTI types; electronic decision support and two multifaceted interventions. Both involved clinician and patient education, but one added CRP testing and the other added academic detailing and practice profiling. We had no evidence on the effect of other patient characteristics on any outcome (i.e., signs and symptoms [nature and duration], previous medical history [e.g., frailty, comorbidity], prior RTIs, and prior use of antibiotics, age, ethnicity, socioeconomic status, and educational level attained).

Table 23

Effectiveness of interventions in improving antibiotic prescribing by respiratory tract infection type.

Seasonal Influences

Most of the studies were timed for the season with highest prevalence of disease, mainly winter months, and no clear pattern could be discerned in the results based on this factor. Local tailoring was typically done for educational interventions (e.g., using ethnically sensitive materials). Comparisons of no tailoring versus tailoring or between degrees or methods of tailoring were not possible due to the wide variation in the combinations of specific intervention details, population, and outcome measurement across studies.

Baseline Prescribing Rates

A key background factor may be baseline prescribing rates, which varied extremely widely across studies (from a low of <10% to greater than 90%) and situations where the background prescribing was declining during the study period. While this is likely true, the poor reporting of this information severely limits the ability to analyze the potential impacts. Other background contextual factors (i.e., known patterns of disease activity [e.g., an influenza epidemic, a pertussis outbreak], or system-level characteristics) were not studied explicitly and were reported inadequately to allow analysis.

We did not find evidence on other factors as potential effect modifiers (i.e., clinician characteristics such as specialty, number of years in practice, type of clinic organization, geographic region, and population served or diagnostic method or definition used, the clinician's perception of the patient's illness severity, or the clinician's diagnostic certainty).

Findings in Relationship to What Is Already Known

There are a number of existing systematic reviews and guidelines that have contributed to our understanding of what works for targeted populations, interventions, or diseases. The reviews are generally more narrowly focused on specific types of interventions, but broadly they have concluded that multifaceted educational interventions, clinician education, delayed prescribing, CRP, and procalcitonin may be effective in certain settings.^{13-16,19,38-42} Our conclusions overlap with these findings, but are not identical in that our results add evidence on more point-of-care tests and electronic decision support, as well as concluding that clinician education alone does not currently show net benefit. Reasons for these differences include the addition of a large volume of newer evidence, the use of a formal system to grade the strength of the evidence and to identify interventions with benefit and that do not cause increased adverse consequences, and the scope of interventions considered (e.g., point-of-care tests). However, a very recent systematic review of outpatient antimicrobial stewardship programs that had a broader scope than this review (including cost outcomes, antibiotic selection outcomes, and a broader range of diagnoses) had similar findings for key interventions: education, delayed prescribing, communication training, electronic decision support, and point-of-care testing.⁴³ They also found that evidence on prescribing rates was mixed with audit and feedback methods.

Specific interventions that have been recommended by professional organizations and societies include delayed prescribing for children with nonsevere symptoms and persistent sinusitis (American Academy of Pediatrics), patient and family education for uncomplicated acute bronchitis (Michigan Quality Improvement Consortium [MQIC] and the American College of Chest Physicians), and rapid strep testing for pharyngitis (MQIC and the Infectious Disease Society of America). Our findings expand on the evidence used to create these recommendations.

Applicability

As planned in our protocol for this review, we focused our reporting on applicability of the body of evidence on the subgroups specified in Key Questions 1 through 4 (subquestions a through e) within the elements of the PICOTS framework.

Implications for Clinical and Policy Decisionmaking

In an effort to appropriately reduce prescribing of antibiotics for acute RTIs, clinicians and policymakers need to make choices among the relevant interventions based on the best evidence, taking into account the characteristics of the setting in which the intervention is to be applied. Although the ultimate goal is reduction in antibiotic resistance, while not adversely affecting clinical outcomes, antibiotic resistance was understudied. Although the most logical intermediate outcome would be changes in appropriate antibiotic use, it, too, was understudied. Therefore it was necessary to consider the most widely studied, but proxy, outcome of overall prescribing to evaluate effectiveness. However, the reliability and validity of overall prescribing as a proxy for appropriate prescribing may vary because the ratio of inappropriate to appropriate prescribing can range so widely based on patient, provider and setting factors and the meaningfulness of the reductions is unclear due to a general lack of established minimally important difference parameters. Based primarily on overall prescribing, the best evidence to date supports the use of four interventions from different categories outlined in this report (education, electronic decision support, and procalcitonin). However, these interventions have varying resource use in both implementation and maintenance, and evidence on sustainability is not available. Even without considering these issues, the difficulty is that the evidence is inadequate to guide selection of the best intervention for a given setting or patient population. Among the interventions with the best evidence, however, there are some elements that could be considered in making decisions about implementation. With combined patient and clinician education programs, patient education can be simple, for example, waiting room posters featuring a letter from a local clinician. Clinician education programs should be locally tailored and the balance of program intensity and clinician participation needs to be taken into consideration. Electronic decision support systems have been shown to improve prescribing for bronchitis and acute otitis media and may be easily implementable in electronic medical record systems. The required resources to initiate a program and for clinicians to use such systems has not been studied, but ease of use (i.e. pop-up systems that do not require clinicians to seek out the information) may be key to ensuring adequate levels of use to result in benefit. For procalcitonin, while there is agreement across algorithms in terms of thresholds for antibiotic prescribing, they were developed for use in adults and use in children led to increased antibiotic use. Additional work is need to evaluate the tradeoffs in resource use required, the impact of the test turn-around time, specific populations where it is best used, and its sustainability as an intervention.

Although it is likely that combinations of interventions will result in improved results and possibly greater sustainability, the benefits and impact on resource utilization are largely unclear.

Limitations of the Review Process

Potential limitations in our process include the exclusion of non-English language publications. To explore the impact of this limitation we reviewed the English-language abstracts of studies with full text published in other languages for apparent eligibility. We identified 24 potentially relevant non-English language studies with English abstracts, of which only one was an RCT (of CRP testing) whose findings as reported in the abstract did not differ from the included studies. The remainders were mostly observational studies whose design and eligibility would require review of the full text, but none were evaluating interventions that we did not have evidence about from English-language publications. Therefore we do not believe that exclusion of non-English language studies has significantly affected the conclusions of this review. Please refer to Appendix C for citations of non-English language studies with English abstracts that were excluded from this review.

Another potential limitation involves our literature search strategies. We conducted extensive literature searches with carefully constructed electronic database strategies that underwent peer review and multiple iterations (Appendix A). However, we found that this topic area is difficult to search for as there are no standard search terms that cover the interventions and outcomes of interest. Thus, it is possible we were unable to identify all potentially relevant studies. In our early discussions with our TEP, we established 1990 as the earliest year that studies would be relevant, but agreed upon using good-quality systematic reviews to identify studies published between 1990 and 2000 for efficiency. It is possible that in using this method we may have missed some older studies, if those reviews had not identified them. To overcome this possible limitation, we utilized our TEP members to assist in identifying missing studies by sharing our included study list with them early on, contacted manufacturers of point-of-care tests, and searched reference lists and bibliographies of included studies. Each of these methods was successful in identifying additional citations for consideration. We also note that there was limited ability to assess potential publication and reporting bias, due to the few opportunities to pool studies and the lack of availability of study protocols.

The final limitation to note is the exclusion of observational studies that did not either control for potential confounding, or were simple before-after studies without a time-series design. We established this criterion to focus our efforts on better evidence (i.e., evidence with lower risk of bias). However, in doing so, it is possible that an important study was missed. In an attempt to overcome this limitation, we allowed any form of controlling for confounding, including simple stratification of results by potential confounders.

Gaps in the Evidence Base

Several gaps and serious limitations of the evidence base limited our ability to reach strong conclusions with regard to several aspects of this review. We used the framework proposed by Robinson et al¹⁹³ to outline these limitations; classifying identified gaps as insufficient or imprecise information, biased information, inconsistency or unknown consistency, and not providing the right information. The gaps are also organized according to the PICOTS framework, by which issues relating to limitations of applicability are identified as well. These are summarized in Table 25 below. Issues pertaining to the overall body of evidence for this report include study design and conduct, the specific details of interventions, choice of comparators and more. In our sample, only 39 percent of the RCTs were cluster randomized. Since many of the interventions were applied at the level of the clinician or even the clinic, allocating the intervention at the patient level was not ideal because of the risk for contamination of samples (i.e., patients and clinicians may talk to each other about the intervention and influence outcomes). Thus, the most appropriate design for most trials of these interventions is a cluster-randomized trial. While we cannot know the direction of the bias introduced by potential contamination, it is likely that lack of clustering could reduce the observed impact of an intervention or differences between interventions.

Table 25

Evidence gaps for interventions to improve use of antibiotics in acute respiratory tract infections.

In our review, we identified some indications for which specific interventions were beneficial or not beneficial for specific outcomes. The limitation here is that not all studies reported the outcomes stratified by such population groups, and some reported groupings of population groups that may have incorporated multiple specific indications, for example studies that stratified results by LRTI versus upper RTI. Some interventions had evidence only in one age group, for example, studies CRP testing were only available in adults.

We were limited in our ability to combine studies and to draw strong conclusions in part due to the variation in the specific details of interventions within a single category. For example, while we found multiple studies of enhancing clinician communication skills, the methods used varied enough that combining these studies led to significant statistical heterogeneity that was not resolved with subgrouping or sensitivity analyses. Other examples are in the group of studies on clinic-based methods to educate patients or parents. These interventions varied widely, with each study representing a “one-off” intervention (e.g., videos featuring local pediatricians, videos with animation, posters of “commitment letters”). While these may be viewed as being locally tailored, they varied enough that we could not combine them, and collectively they do not provide a cohesive picture of the benefits of educating patients using a core set of principles. Unfortunately, the variation in both categories and specific details of interventions used in multifaceted intervention studies seriously prevented drawing meaningful conclusions from an area of research that is likely to hold the key to identification of the most effective intervention.

Similarly, we found that the comparisons made by studies to date are too varied to be as useful as they could be in drawing meaningful conclusions. For example, delayed prescribing as an intervention was compared with always providing a prescription in some studies and with not providing a prescription in other studies. These comparisons are less generalizable to other study designs where the comparison is to usual care or to a competing intervention. In addition, the majority of studies do make comparisons to a usual care group, with fewer studies evaluating comparisons of competing interventions.

The specific outcomes reported and how they were measured also varied and created difficulties in combining similar studies and drawing strong conclusions. A simple example is the outcomes in Key Question 1 regarding the comparative effectiveness of interventions to improve appropriate antibiotic use in acute RTIs. The biggest gap in evidence is consistent reporting resistance to antibiotics and improvement in appropriate prescribing, the two most relevant outcomes for this topic. The few studies that did report appropriate prescribing had important limitations in outcome definition and ascertainment methods and lack of consistency in methods across studies. The methods fall into three categories: ICD-9 codes or diagnostic category, adherence to a specific guideline's recommendations for antibiotic use, and duration of symptoms for pharyngitis or sinusitis. None of the studies provided detailed information on how the information was obtained or assessed. Dependence on ICD-9 codes alone is a limited approach in that patient-level characteristics that may indicate the need for antibiotic therapy are not assessed. Use of a guideline to determine appropriateness of prescribing is also limited in that the determination of whether a decision adhered to the guideline or not is subjective and requires both access to adequate patient-level data and clinical knowledge. While the duration of symptoms beyond a suggested cutoff may be an indicator for when antibiotics are needed, this information alone is inadequate to make a precise determination.

Related to either overall or appropriate prescribing outcomes, there is a gap in consistently defined goals for the necessary change or difference in prescribing that will result in meaningful benefits, such as reductions in antibiotic resistance in intervention communities. While most of the RCTs did conduct power calculations to determine adequate sample sizes, the delta used to determine these ranged widely, with no reasoning given for the selection of the difference. For example, it is not clear that a difference in antibiotic prescribing of 15 percent is enough to make differences in key outcomes such as resistance, patient outcomes, satisfaction, and resource use. Without such information, it is difficult to evaluate the magnitude of difference seen in studies even when statistically significant. Similar concerns can be raised about other outcomes. For example, symptom improvement was often measured using mean change, without any parameters for judging the importance of the change/difference (e.g., differences in change of temperature of less than one degree Fahrenheit).

The potential for increased risk of hospitalization within 1 month of the index visit found with CRP testing, communication training and the combination is concerning and deserves further scrutiny (Table 24). The evidence of potential increased risk comes largely from three trials; a single, large (N=4,264), fair-quality factorial design trial of CRP testing, communication training or the combination conducted in clinics, a smaller (N=431) study with similar design, and a small study of CRP testing only, conducted in EDs (N=139). The larger multifactorial study presented only an analysis considering CRP test use with or without communication training compared with usual care or communication training alone. After adjusting for potential confounders, this study found a non-statistically significant increased risk with use of CRP testing (22 versus 8 events). An analysis of only CRP use versus only usual care was not done. The small study of only CRP testing found a similar non-significant increased risk; however, in five other studies there were no hospitalizations in either group.

Table 24

Risk of hospitalization at 1-month post index visit.

Based on events reported in the larger study, communication training also resulted in a non-statistically significant increase in risk, and the combination of the two interventions resulted in a statistically significant increased risk, although the estimates we provide are unadjusted. For the combination of CRP testing and communication training, reported in two similar multifactorial trials, we found a statistically significant increased risk, although this pooled estimate is unadjusted for potential confounders.

The reasons for a potential increased risk are unclear, since the studies were not designed to examine this outcome in depth. Since the absolute numbers of events was low, the estimates are likely to be unstable and could change with additional data.

For other Key Questions, the real gap is in outcome reporting in general. Few studies reported on clinical consequences of reduced prescribing, and those that did were inconsistent in definitions and methods. Studies of the rapid strep test are a good example. While it is a guideline-recommended test intended to inform prescribing decisions for moderate to severe pharyngitis, no study measured outcomes other than prescribing. No study of any intervention explicitly attempted to measure resource use. Given the clear differences in the potential for differential cost (both monetary and intangible costs) this is a major gap in understanding which intervention or combination of interventions is best in which situation. In studies that measured secondary outcomes (i.e., adverse consequences), adequate statistical power to identify statistically significant differences was uncommon. Where differences were found, it is not clear that the differences were important (clinically, economically, or from the patient's perspective).

We were limited in drawing conclusions about how the effects of the strategies may differ in specific subgroups based on previous medical history (e.g., frailty, comorbidity), prior use of antibiotics, ethnicity, socioeconomic status, clinician characteristics, and other subgroups because studies rarely conducted subgroup analyses on these factors and these factors were not commonly reported, limiting our ability to look across studies. Due to potentially confounding influences of a wide variety of sources of variability, it is difficult to establish a relationship between any one subgroup characteristic and outcome.

With regard to settings, there is a potentially major issue with attempting to use study results from studies in settings outside the United States. There may be cultural differences that result in wide variation in baseline prescribing, the application and uptake of specific interventions, and system-level differences that make this evidence nongeneralizable to the US setting. Given that 55 percent of included studies were conducted outside the United States, this is potentially a serious limitation.

We note that we identified several good-quality systematic reviews that were related to our report topic, but we were only able to use them to crosscheck lists of included studies for two main reasons. The gaps in knowledge left by these reviews, for our purposes, were related to mainly to scope, although some were not used due to the time since literature searching was completed. For the most part the reviews included either broader populations (a wider range of diagnoses) or narrower interventions (focusing on only one intervention, or one intervention type).

Future Research Needs

Based on the gaps and weaknesses identified through the systematic review of the literature, the following areas present an opportunity for new research to support healthcare decisions (Table 26). Although potentially difficult and time and resource-intensive, future studies of interventions to improve antibiotic prescribing in acute RTIs should add great value to our understanding of how to best address this important public health issue by having the following methodological features:

Table 26

Future research recommendations based on evidence gaps.

A recent report on ways to improve research evaluating antimicrobial stewardship programs echoes our findings above.¹⁹⁵ The authors stress that studies should move beyond measuring primarily economic outcomes and include key clinical outcomes such as resistance, incidence of adverse clinical consequences of antibiotic use, e.g. clostridium difficile in high-risk populations, should consider the difficulty in measuring outcomes that can obfuscate the findings, to consider the interventions as multifactorial, complex interventions, and the need to study the interventions in multiple settings.