Summary

10.1. Introduction: the characteristics of audit and feedback

Audit and feedback is a strategy that is widely used in European countries in various ways. The spectrum ranges from mandatory schemes run by government bodies to voluntary, smaller-scale initiatives led by professionals. Some audit and feedback initiatives aim to improve accountability (for example, towards the public, the payer, etc.), while others aim at continuous quality improvement and professional development. In some countries audit and feedback strategies are described as clinical audits, underlining their focus on clinical practice (in contrast to, for example, financial audits). All audit and feedback initiatives have in common the intention to encourage professionals to change their clinical practice when needed by showing them how they perform compared to descriptive or normative benchmarks or targets.

An audit is a review of professional performance based on explicit criteria or standards, preferably developed on the basis of evidence-based clinical guidelines or pathways (see Chapters 9 and 12). Performance information is subsequently fed back to professionals, showing how they perform in relation to their peers, standards or targets. In addition, there should be a formal process to identify possible actions in order to change current practice and to improve performance.

Audit and feedback can be used for any area of healthcare, i.e. preventive, acute, chronic and palliative care. Audits can attempt to assess individual health professionals’ performance or that of teams, departments, hospitals or regions. The audit may focus on various indicators of quality measured in terms of structures, processes or outcomes of care (see also Chapter 3). Audits could also focus on any of the three domains of healthcare quality (effectiveness, safety, patient-centredness), as well as on many other aspects of performance, such as timeliness, efficiency and equity. However, in practice most audits focus on processes of care and/or patient outcomes that are strongly correlated with processes of care, and the focus of most initiatives has been on effectiveness and patient safety.

Audits can be based on routinely available information, such as administrative databases, electronic patient records or medical registries, or they may be based on purposefully collected data from medical records or direct observations. Audit and feedback initiatives can be internal (conducted by individual or local groups of practitioners for their own practice), or external (conducted by professional bodies, research groups or government structures). In either case the indicators measured can be determined by outside sources (i.e. top-down) or by the recipients of the feedback (i.e. bottom-up) or by a combination of both (Foy et al., 2002). Ultimately, the approach taken depends on the underlying purpose (for example, accountability versus quality improvement or knowledge translation/implementation of guidelines).

Feedback can be delivered in different ways, which can be categorized in terms of the source (for example, administrators, senior colleagues or peers), the recipients (for example, individuals or groups), formats (for example, verbal or written), frequency (for example, monthly or yearly), and content (for example, including level of aggregation of data, type of comparison, clarity of message and action plan). The feedback of performance information can be performed in ways that involve varying amounts of interaction or engagement with recipients, depending on the level of interest and availability of resources.

Audits can be a prerequisite for accreditation and certification (see Chapter 8), and feedback can be linked to economic incentives (see Chapter 14). Furthermore, performance data can be made publically available to contribute to informed patient choice (see Chapter 13). However, in most cases feedback is confidential rather than public. In contrast to other quality improvement strategies, such as accreditation, certification and supervision (see Chapter 8), which are focused on healthcare organizations or institutions, audit and feedback is most often focused on improving performance of health professionals.

10.2. Why should audit and feedback contribute to healthcare quality?

Health professionals are usually assumed to be highly motivated individuals dedicated to providing high-quality care. However, it is well documented that many patients do not receive recommended care and that there are great variations in medical practice (Ash et al., 2006; Wennberg, 2014), which cannot be explained by illness, patient preferences or medical science. Part of the explanation for this phenomenon is likely that professionals have a limited ability to accurately assess their own performance (Davis et al., 2006). Therefore, information about how they perform compared to descriptive or normative standards can be an important motivator for change amongst health professionals (Godin et al., 2008).

Like many other quality improvement strategies, audit and feedback has been conceptualized as a cyclical process that involves five steps (see Fig. 10.1): (1) preparing for audit; (2) selecting criteria; (3) measuring performance; (4) making improvements; and (5) sustaining improvements (Benjamin, 2008). Roughly the first half of the cycle is concerned with auditing of professional performance, while the second half of the cycle starts with feeding this information back to professionals. However, audit and feedback will result in quality improvements only if the feedback leads to changes that improve clinical practice.

Fig. 10.1

The audit and feedback cycle. Source: based on Benjamin, 2008, with modifications

Whether or not practice changes take place depends on various factors, which have been explored qualitatively in numerous studies (see Brown et al., 2016). Many theories exist to explain how audit and feedback may lead to changes in professional practice. Some theories focus on change within the professionals, others on change within the social setting or within the organizational and economic context (Grol et al., 2007). According to these theories, audit and feedback may change the awareness of the recipients and their beliefs about current practice, which will subsequently result in changes of clinical practice. In addition, audit and feedback may change perceived social norms and direct attention to a specific set of tasks or subgoals (Ivers et al., 2012).

The extent to which audit and feedback successfully accomplishes this desired reaction depends upon the features of the intervention itself, the targeted behaviour change, and how these interact with features of the recipient and their environment (see below). Well designed feedback considers all these factors and seeks to minimize emotional responses of defensiveness while shifting the recipient’s attention towards the specific, achievable tasks needed to achieve best possible patient outcomes (Payne & Hysong, 2016).

10.3. What is being done in Europe?

The UK and the Netherlands are the countries in Europe that have the longest history of audit and feedback. In both countries audit and feedback initiatives developed on a voluntary basis in the 1970s and 1980s. Later, from 1991, the UK was the first country that required hospital doctors to participate in audit. Within a few years other health professionals were required to join multiprofessional clinical audits. In Germany and France audit and feedback initiatives emerged mostly in the 1990s. Table 10.1 provides an overview about some prominent audit and feedback programmes in Europe.

Table 10.1

A selection of some audit and feedback programmes in Europe.

In the UK various actors are active in the field of audit and feedback. The National Clinical Audit Programme is run by the Healthcare Quality Improvement Partnership (HQIP). National audits are performed for about 30 clinical conditions, including acute (for example, emergency laparotomy) and chronic conditions (for example, diabetes). These audits focus mostly on specialist inpatient and outpatient service providers, who are assessed with regard to all three dimensions of quality: effectiveness, patient safety and patient experience. Audits rely on various data sources, and assess performance in relation to numerous indicators of structures, processes and outcomes. Benchmark reports are provided to local trusts and annual reports are published for each of the clinical conditions. In the area of primary care the most important national audit programme is the Quality and Outcomes Framework (QOF). However, the main purpose of QOF is to distribute financial incentives (representing around 15% of GP income), and indicators were developed externally. GPs are also required to undertake audit and feedback as part of their revalidation scheme, which was launched in 2012. Furthermore, medical students are taught audit, and there is some teaching for GP trainees. Finally, there is a National Quality Improvement and Clinical Audit Network (NQICAN), which brings together 15 regional clinical audit/effectiveness networks from across England. NQICAN supports staff working in quality improvement and clinical audit in different health and social care organizations, providing practical guidance and support.

In the Netherlands audit and feedback activities historically started in primary care and were initiated by GPs. More recently, audit and feedback has expanded also to secondary inpatient and outpatient care and is more embedded in broader quality assurance initiatives. A Dutch Institute for Clinical Audit (DICA) was set up in 2009 and medical specialist societies use DICA to measure quality and communicate about it. DICA runs registers for cancer patients (colorectal, breast, upper gastrointestinal and lung), collects patient-reported outcome measures, and provides feedback reports to professionals. Almost all hospitals have established quality improvement strategies based on feedback reports from DICA, which also allow them to measure improvements over time. In addition, a comprehensive clinical and organizational audit is part of primary care practice accreditation. Furthermore, almost all GPs are part of one of 600 educational pharmacotherapy groups existing in the country, each consisting of GPs and pharmacists. These groups use audits of prescribing data as a starting point for discussions.

In Germany audit and feedback efforts also exist at several levels of the healthcare system. The most important audit and feedback initiative is the mandatory external quality assurance programme introduced for all hospitals in 2001. It is the responsibility of the Federal Joint Committee, which includes representatives of providers (for example, hospitals) and sickness funds. By 2014 the programme covered 30 specific areas of inpatient care (for example, cholecystectomy, or community-acquired pneumonia), which were assessed on the basis of more than 400 process and outcome indicators, including also patient-safety indicators (AQUA, 2014). Providers have to comply with specific quality documentation requirements in order to provide data for the audits. Collected data are analysed and forwarded to professional expert sections who may initiate a peer review process if the data suggest potential quality problems. Public disclosure of data was introduced in 2007. Smaller programmes cover amongst other things disease management programmes (DMPs) and ambulatory dialysis. In addition, professional associations may have their own audit systems, for example, for reproductive medicine, producing annual reports and providing feedback to providers.

In Italy the Emilia-Romagna region requires GPs to join a Primary Care Team. GPs are mandated to collaborate and share information and to engage in improving the quality of healthcare services provided to patients. Primary Care Teams receive quality reports featuring structure, process and outcome indicators computed on the basis of data from the regional healthcare administrative database, an anonymous comprehensive and longitudinal database linkable at the patient and provider level. The GPs in each team are asked to identify at least one critical area of the report and initiate quality improvement activities in their practice accordingly. The reports are not meant to be “punitive”; rather, the reports are intended to promote teamwork and coordination, and encourage clinical discussion. GPs seem to have a positive view of the reports (Maio et al., 2012; Donatini et al., 2012).

In Finland audit and feedback is used mostly in health centres. One fifth of all health centres participate in yearly quality measurements, based on two-week samples of treatment of patients, organized by Conmedic, a primary care quality consortium. Quality measurement always includes indicators for diabetes and cardiovascular care but also several other areas of care, which may vary from year to year based on decisions of health centres. Measured care areas have included fracture prevention, smoking cessation, interventions for risky alcohol consumption, dementia and self-care. The purpose of the audit and feedback is to inform local quality improvement activities. In addition, all intensive care units collect information on all patients, and the information is reported back to the professionals. Both audit and feedback systems started in 1994. The audit and feedback is voluntary, driven by health professionals. Audit data are fed back at group level. Another interesting initiative in Finland is the evidence-based decision support system (EBMeDS) developed by Duodecim, the Finnish Medical Society. EBMeDS is linked to many patient record systems and provides direct feedback and decision support to practitioners.

In Ireland a National Office of Clinical Audit (NOCA) was established in 2012. Its objective is to maintain clinical audit programmes at national level. They offer different audit programmes (major trauma, national intensive care unit, national orthopaedic register, hip fracture and hospital mortality) and publish national reports on some audit areas. National clinical audits are ongoing reviews of clinical practice that use structural, process and outcome measures to find room for improvement. NOCA emphasizes the importance of action based on audit output and supports hospitals in learning from their audit cycles. The comprehensiveness of data has improved over the years; for example, the most recent report on hip fractures contains data from all 16 eligible hospitals.

At the European level, guidelines on clinical audit for medical radiological practices, including diagnostic radiology, nuclear medicine and radiotherapy, were published by the EU Commission in 2010. These provide recommendations on how to approach clinical audit in radiological practice and suggest the inclusion of structure, process and outcome indicators for comprehensive audits. However, it remains unclear how far these guidelines have been implemented at national level.

To our knowledge, no systematic research has been conducted to assess or compare the use of audit and feedback across European healthcare systems. However, the informal overview provided in this section illustrates the large variation not only in terms of what is audited, but also how the feedback is delivered and ownership of the programmes.

10.4. The effectiveness and cost-effectiveness of audit and feedback

A systematic review from Cochrane on the effects of audit and feedback was first published in 2000 and has since been updated twice (2006 and 2012). Table 10.2 summarizes characteristics of 140 studies included in the 2012 update of the review (Ivers et al., 2012). Almost half of all studies were conducted in the USA. In most studies audit and feedback was combined with other quality improvement strategies such as clinician education, educational outreach (also called academic detailing) or reminder systems, and the targeted professionals were most often physicians.

Table 10.2

Characteristics of 140 audit and feedback intervention trials included in Ivers et al., 2012.

Audited information included mostly process indicators; it was mostly focused on aggregate patient data (for example, proportions of patients not receiving guideline consistent care), and on individual providers instead of groups of providers. Feedback was usually provided in writing, and in almost half of the studies it was provided only once. In more than half of the studies feedback was provided to individuals and it mostly showed comparisons with the performance of peers. In response to the feedback, professionals were required to either increase (41%) or decrease (21%) their behaviour, but they usually did not receive detailed instructions about how to change their behaviour.

Table 10.3 provides an overview of the main results of the meta-analyses performed as part of the 2012 Cochrane review of audit and feedback trials. The largest number of studies reported results comparing the compliance of professionals with desired practice using dichotomous outcomes (for example, the proportion of professionals compliant with guidelines). These studies found a small to moderate effect of audit and feedback. The median increase of compliance with desired practice was 4.3% (interquartile range (IQR) 0.5% to 16%). Although the median effect may be perceived as relatively small, a quarter of the studies included in the primary analysis showed larger than 16% absolute improvement in health professionals’ compliance with desired behaviour.

Table 10.3

Main results of audit and feedback studies included in Ivers et al., 2012.

Relatively few studies reported effects of audit and feedback on patient outcomes, including dichotomous outcomes (for example, smoking status) or continuous outcomes (for example, blood pressure). Studies reporting dichotomous outcomes found a minimal discernible effect, while studies reporting continuous outcomes found a comparatively large positive outcome (17%). In summary, the review confirmed the conclusions of earlier reviews that audit and feedback can be a useful and effective intervention for improving professional practice and potentially patient outcomes.

The large variation in reported results, with a quarter of studies reporting relatively large effects (i.e. absolute improvements in desired practice >16%), suggests that audit and feedback, when optimally designed, delivered and implemented, can play an important role in improving professional practice. However, it also implies that poorly designed audit and feedback schemes will have a minimal or no effect. This underlines the need to focus attention on the design and implementation of audit and feedback schemes.

A meta-regression included in the Cochrane review showed that baseline performance, characteristics of the feedback and the type of change in practice required by the intervention can explain part of the variation in effect size (see Table 10.3). For example, when feedback is presented both verbally and in written format, the median effect is 8% higher than when feedback is presented only verbally. Similar differences in effect sizes exist if the feedback is delivered by a supervisor or senior colleague compared to the investigators, when the frequency is increased from once only to weekly and when the feedback contains both an explicit, measurable target and a specific action plan. However, all the findings of the meta-regression should be taken as tentative, as they are based on indirect analyses and ecological bias.

Not surprisingly, the meta-regression also found that the effect of audit and feedback is larger among health professionals with low baseline performance. In addition, it seems that feedback is more effective for less complex changes in professional behaviour (such as prescriptions) than for more complex ones (such as the overall management of patients with chronic disease), although it is plausible that feedback may be useful if it targets very specific behaviour changes related to chronic disease management.

Furthermore, the meta-regression showed that sources of feedback associated with the lowest effect size are “professionals’ standards review organization” and “representative of the employer or purchaser”. This is an important finding in line with previous qualitative work, which suggested that feedback with a punitive tone is less effective than constructive feedback (Hysong, Best & Pugh, 2006). Also, Feedback Intervention Theory (Kluger & DeNisi, 1996) suggests that feedback directing attention towards acceptable and familiar tasks (as opposed to feedback that generates emotional responses or causes deep self-reflection) is more likely to lead to improvements.

Finally, Table 10.3 presents separately results from studies where audit and feedback was carried out alone and results for interventions where audit and feedback was combined with other interventions. Although combined interventions appeared to have a larger median effect size than studies where audit and feedback was implemented alone, the difference was not statistically significant. These findings are consistent with other reviews (O’Brien et al., 2007; Forsetlund et al., 2009; Squires et al., 2014), which found that there is no compelling evidence that multifaceted interventions are more effective than single-component ones. Therefore, it remains unclear whether it is worth the additional resources and costs to add other interventions to audit and feedback.

The cost-effectiveness of audit and feedback in comparison with usual care has not been evaluated in any review to date. In general, cost-effectiveness analyses are rare in the quality improvement literature (Irwin, Stokes & Marshall, 2015). However, it is clear that the costs of setting up an audit and feedback programme will vary depending on how the intervention is designed and delivered. Local conditions, such as the availability of reliable routinely collected data, have an important impact on the costs of an intervention. If accurate data are readily available, audit and feedback may prove to be cost-effective, even when the effect size is small.

Only very few reviews investigating the effectiveness of audit and feedback compared with other quality improvement strategies are available. The Cochrane review included 20 direct comparisons between audit and feedback and other interventions but it remained unclear whether audit and feedback works better than reminders, educational outreach, opinion leaders, other educational activities or patient-mediated interventions. One review compared the influence of 11 different quality improvement strategies, including audit and feedback, on outcomes of diabetes care (Tricco et al., 2012). Findings consistently indicated across different outcome measures (HbA1c, LDL levels, systolic and diastolic blood pressure) that complex interventions, such as team changes, case management and promotion of self-management, are more effective than audit and feedback in improving outcomes. However, cost-effectiveness was not considered in this review. The greater effectiveness of complex, system-level interventions compared to audit and feedback suggests that audit and feedback does not work well if the desired patient-level outcomes are not exclusively under the control of the provider receiving the feedback.

In summary, substantial evidence shows that audit and feedback improves care across a variety of clinical settings and conditions; further trials comparing audit and feedback with no intervention are not needed. However, given that the effect size differs widely across different studies, it is important to focus future research on understanding how audit and feedback systems can be designed and implemented to maximize the desired effect.

10.5. How can audit and feedback be implemented? What are the organizational and institutional requirements?

Different recommendations exist to provide guidance for the design of best practice audit and feedback schemes (Copeland, 2005; Ivers et al., 2014a; Brehaut et al., 2016; McNamara et al., 2016). Copeland (2005) is a practical handbook for clinical audit published by NHS England. Ivers (2014a, 2014b) made recommendations based on findings from the Cochrane review and the collective experience of a wide range of experts working in audit and feedback who gathered at a meeting in 2012. Brehaut et al. (2016) summarize recommendations that build upon findings from Ivers (2014a, 2014b) and add evidence from an additional series of interviews with experts from a range of disciplines. Finally, McNamara (2016) is a report prepared for the Agency for Healthcare Research and Quality in the United States that summarizes all the above, and incorporates real-world experience of those who have implemented audit and feedback strategies. Table 10.4 summarizes the main recommendations of the four sources, although the evidence supporting these statements is sometimes relatively weak.

Table 10.4

Tentative “best practices” when designing and implementing audit and feedback.

The first step of an audit and feedback process is to identify the problem and the local resources to solve it in order to define the focus of the intervention. The topic should be a priority for the organization and the patients it serves – and be perceived as a priority by the recipients of the feedback – and typically involves high-volume, high-risk and/or high-cost issues where there is known variation in performance. In addition, the audit should focus on care areas where there is clear evidence about what care is effective and appropriate, and for whom, implying that audits should focus on clinical practices for which strong recommendations according to the GRADE approach (Grading of Recommendations Assessment, Development and Evaluation) exist (Guyatt et al., 2008).

Concerning the audit component, it is important that the audited data are perceived to be valid and that the indicators assess structure, processes and/or outcomes that the recipients of feedback would have expected and/or intended to achieve and that they would feel capable of improving within the measurement interval. If goal-commitment and/or self-efficacy to achieve high performance in the indicator are not present, co-interventions may be needed for the feedback to achieve its desired results (Locke & Latham, 2002). It has been suggested that the key source of information for audits should be the medical record and routinely collected data from electronic systems (Akl et al., 2007). However, medical records are not always available or suitable for extracting the data needed, and it is necessary to pay attention to the reliability and validity of the data as well as to the appropriateness of the sample. In particular, the validity of records can vary depending on the type of information being extracted (Peabody et al., 2004), especially in outpatient settings. In some cases clinical vignettes or case reports have been shown to be a more valid source of information about practice behaviours than records (Peabody et al., 2004; Stange et al., 1998). In other cases, the use of patient-reported experience or outcome measures might be a promising approach, so long as the measures are validated and perceived as actionable (Boyce, Browne & Greenhalgh, 2014).

Concerning the feedback component, feedback is likely to be more effective when it is presented both verbally and in writing than when using only one modality and when the source (i.e., the person delivering the feedback) is a respected colleague rather than unknown investigators or employers of purchasers of care. Source credibility matters a great deal (Ferguson, Wakeling & Bowie, 2014).

Audit and feedback schemes should always include clear targets and an action plan specifying the steps necessary to achieve the targets (Gardner et al., 2010). Ideal targets are commonly considered to be specific, measurable, achievable, relevant and time-bound (Doran, 1981). In addition, feedback should include a comparison with achievable but challenging benchmarks (for example, comparing performance to the top 10% of peers) (Kiefe et al., 2001).

Furthermore, audit and feedback requires a supportive organizational context. This includes commitment to a constructive (i.e. non-punitive) approach to continuous quality improvement and to iterative cycles of measurement at regular, predictable intervals (Hysong, Best & Pugh, 2006). In addition, many mediating structural factors may impact on care and on the likelihood of clinical audit to improve care, such as staffing levels, staffing morale, availability of facilities and levels of knowledge. Finally, the recipients may require skills and/or resources to properly analyse and interpret the audited data and they need to have the capacity to act upon it. This is especially true if the feedback does not provide patient-level information with clear suggestions for clinical action (meaning resources may be needed to conduct further analyses) or if the feedback highlights indicators that require organizational changes to address (such that change-management resources may be needed).

It is rarely possible to design each component of an audit and feedback scheme in an optimal way. Therefore, it is useful to perceive the individual components outlined in Table 10.4 as “levers” to be manipulated when working within setting-specific constraints. For example, if circumstances dictate that the delivery of feedback cannot be repeated in a reasonable timeframe, extra attention should be paid to other aspects of the intervention, such as the source of the feedback. In addition, co-interventions, tailored to overcome identified barriers and boost facilitators, may help if feedback alone seems unlikely to activate the desired response (Baker et al., 2010).

10.6. Conclusions for policy-makers

Audit and feedback is a quality strategy that is widely used in European countries in various ways. The various programmes presented in Table 10.1 may provide inspiration for policy-makers aiming to introduce similar programmes in their countries. The available evidence suggests that audit and feedback can contribute to improving quality measured in terms of processes (for example, adherence to guidelines) or outcomes (for example, reduction in blood pressure) (see Table 10.3). Recently, a number of large-scale initiatives using audit and feedback have shown success with a focus on safety in the prescription of medicines (Guthrie et al., 2016; Dreischulte et al., 2016).

Several aspects have to be taken into account when implementing audit and feedback (see Table 10.4). Feedback is more effective when baseline performance is low since the room for improvement of practice is greater and because the mechanism of action requires a noteworthy discrepancy between desired and expected performance. The effect of feedback is greater when the source of feedback is a respected colleague, when it is provided regularly both verbally and in written reports, and when it includes both measurable targets and an action plan for changing practice (Ivers et al., 2012, 2014b).

There is inconclusive evidence about the effectiveness of audit and feedback compared with other quality improvement strategies, such as reminders, educational outreach, opinion leaders, other educational activities or patient-mediated interventions. In addition, it remains somewhat unclear whether audit and feedback is more effective when combined with other interventions, and whether the cost of these additional interventions can be justified. In general, cost-effectiveness of audit and feedback in comparison with other strategies remains largely unexplored.

Ultimately, most decisions about audit and feedback must largely be guided by local circumstances, barriers and facilitators, and pragmatic considerations. Organizational support, including time and resources for professionals as well as provision of data, is crucial. When audit and feedback is utilized, careful attention to the way it is designed and delivered may increase its effectiveness.

In summary, it would be fair to say that, in comparison to most other quality improvement or implementation strategies, a strong answer does exist to the question of “Should audit and feedback be implemented to improve processes of care?” In most circumstances the correct answer is Yes! Small to moderate absolute improvements in desired practice are achievable, depending on the measures in the audit and the design of the feedback. However, a strong answer to the question “How could policy-makers best implement audit and feedback and how should it be combined with other interventions?” cannot be given in light of the available evidence. Most likely, the correct answer is: It depends!

This chapter offers a series of tentative recommendations and best practices based on the current evidence base and relevant theory. To guide policy-makers, a shift is needed in the implementation research towards a comparative effectiveness paradigm, prioritizing studies that assess not whether audit and feedback works, but how best to conduct feedback and how best to combine it with other interventions (Ivers et al., 2014b). Whenever policy-makers are planning to implement audit and feedback initiatives, they could partner with researchers to prospectively test different approaches and iteratively improve the impact of their programmes while contributing in important ways to the implementation science literature (Ivers & Grimshaw, 2016).

References

• Akl EA, et al. NorthStar, a support tool for the design and evaluation of quality improvement interventions in healthcare. Implementation Science. 2007;2:19. [PMC free article: PMC1913538] [PubMed: 17594495]
• AQUA. Qualitätsreport 2014. Göttingen: Institut für Qualitätsförderung und Forschung im Gesundheitswesen (AQUA); 2014.
• Ash S, et al. Who is at greatest risk for receiving poor-quality health care. New England Journal of Medicine. 2006;354(24):2617–19. [PubMed: 16775246]
• Baker R, et al. Tailored interventions to overcome identified barriers to change: effect on professional practice and health care outcomes. Cochrane Database of Systematic Reviews. 2010;(3):CD005470. [PMC free article: PMC4164371] [PubMed: 20238340]
• Benjamin A. Audit: how to do it in practice. BMJ. 2008;336(7655):1241–5. [PMC free article: PMC2405828] [PubMed: 18511799]
• Brehaut J, et al. Practice Feedback Interventions: 15 Suggestions for Optimizing Effectiveness. Annals of Internal Medicine. 2016;164(6):435–41. [PubMed: 26903136]
• Boyce MB, Browne JP, Greenhalgh J. The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research. BMJ Quality and Safety. 2014;23(6):508–18. [PubMed: 24505110]
• Brown B, et al. A meta-synthesis of findings from qualitative studies to audit and feedback interventions. UK: National Institute of Health Research; 2016.
• Colquhoun H, et al. Reporting and design elements of audit and feedback interventions: a secondary review. BMJ Quality and Safety. 2017;(1):54–60. [PubMed: 26811541]
• Copeland G. A Practical Handbook for Clinical Audit. London: Clinical Governance Support Team, Department of Health Publications; 2005.
• Davis D, et al. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. Journal o f the American Medical Association. 2006;296(9):1094–102. [PubMed: 16954489]
• Donatini A, et al. Physician Profiling in Primary Care in Emilia-Romagna Region, Italy: a Tool for Quality Improvement. Population Health Matters (formerly Health Policy Newsletter. 2012;25(1):10.
• Doran G. There’s a S.M.A.R.T. way to write management’s goals and objectives. AMA Management Review. 1981;70:35–6.
• Dreischulte T, et al. Safer Prescribing – A Trial of Education, Informatics, and Financial Incentives. New England Journal of Medicine. 2016;374:1053–64. [PubMed: 26981935]
• Ferguson J, Wakeling J, Bowie P. Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review. BMC Medical Education. 2014;14:76. [PMC free article: PMC4011765] [PubMed: 24725268]
• Forsetlund L, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews. 2009;(2):CD003030. [PMC free article: PMC7138253] [PubMed: 19370580]
• Foy R, et al. Attributes of clinical recommendations that influence change in practice following audit and feedback. Journal of Clinical Epidemiology. 2002;55(7):717–22. [PubMed: 12160920]
• Gardner B, et al. Using theory to synthesise evidence from behaviour change interventions: the example of audit and feedback. Social Science and Medicine. 2010;70(10):1618–25. [PubMed: 20207464]
• Godin G, et al. Healthcare professionals’ intentions and behaviours: a systematic review of studies based on social cognitive theories. Implementation Science. 2008;3:36. [PMC free article: PMC2507717] [PubMed: 18631386]
• Grol R, et al. Planning and studying improvement in patient care: the use of theoretical perspectives. Millbank Quarterly. 2007;85(1):93–138. [PMC free article: PMC2690312] [PubMed: 17319808]
• Guthrie B, et al. Data feedback and behavioural change intervention to improve primary care prescribing safety (EFIPPS): multicentre, three arm, cluster randomised controlled trial. BMJ (Clinical Research Edition). 2016;354:i4079. [PMC free article: PMC4990081] [PubMed: 27540041]
• Guyatt G, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ (Clinical Research Edition). 2008;336(7650):924–6. [PMC free article: PMC2335261] [PubMed: 18436948]
• Hysong S, Best R, Pugh J. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implementation Science. 2006;1:9. [PMC free article: PMC1479835] [PubMed: 16722539]
• Irwin R, Stokes T, Marshall T. Practice-level quality improvement interventions in primary care: a review of systematic reviews. Primary Health Care Resea rch and Development. 2015;16(6):556–77. [PubMed: 26004929]
• Ivers N, Grimshaw J. Reducing research waste with implementation laboratories. Lancet. 2016;388(10044):547–8. [PubMed: 27511773]
• Ivers N, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews. 2012;(6):CD000259. [PubMed: 22696318]
• Ivers N, et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. Journal of General Internal Medicine. 2014a;29(11):1534–41. [PMC free article: PMC4238192] [PubMed: 24965281]
• Ivers N, et al. No more “business as usual” with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implementation Science. 2014b;9:14. [PMC free article: PMC3896824] [PubMed: 24438584]
• Kiefe C, et al. Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. Journal of the American Medical Association. 2001;285(22):2871–9. [PubMed: 11401608]
• Kluger A, DeNisi A. The Effects of Feedback Interventions on Performance: a Historical Review, a Meta-Analysis, and a Preliminary Feedback Intervention Theory. Psychological Bulletin. 1996;119(2):254–84.
• Locke E, Latham G. Building a Practically Useful Theory of Goal Setting and Task Motivation; a 35-Year Odyssey. American Psychologist. 2002;(9):705–17. [PubMed: 12237980]
• McNamara, et al. Confidential physician feedback reports: designing for optimal impact on performance. Rockville, Maryland: AHRQ; 2016.
• Maio V, et al. Physician Profiling in Primary Care in Emilia-Romagna Region, Italy: A Tool for Quality Improvement. Populati on Health Matters (formerly Health Policy Newsletter. 2012;25(1) Article 10.
• O’Brien M, et al. Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database of Systematic Reviews. 2007;(4):CD000409. [PMC free article: PMC7032679] [PubMed: 17943742]
• Payne V, Hysong S. Model depicting aspects of audit and feedback that impact physicians’ acceptance of clinical performance feedback. BMC Health Services Research. 2016;16:260. [PMC free article: PMC4944319] [PubMed: 27412170]
• Peabody J, et al. Measuring the quality of physician practice by using clinical vignettes: a prospective validation study. Annals of Internal Medicine. 2004;141(10):771–80. [PubMed: 15545677]
• Squires J, et al. Are multifaceted interventions more effective than single-component interventions in changing health-care professionals’ behaviours? An overview of systematic reviews. Implementation Science. 2014;(9):152. [PMC free article: PMC4194373] [PubMed: 25287951]
• Stange K, et al. How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patient visits. Medical Care. 1998;36(6):851–67. [PubMed: 9630127]
• Tricco A, et al. Effectiveness of quality improvement strategies on the management of diabetes: a systematic reviews and meta-analysis. Lancet. 2012;379(9833):2252–61. [PubMed: 22683130]
• Wennberg J. Forty years of unwanted variation – and still counting. Health Policy. 2014;114(1):1–2. [PubMed: 24373550]