Chapter  149:  Effectiveness of Continuing Medical Education

Key Question 1: Is there evidence that particular methods of delivering CME are more effective in: a) imparting knowledge to physicians, b) changing physician attitudes, c) acquiring skills, d) changing physician practice behavior, or e) changing clinical practice outcomes?

Key Question 2: Do changes in knowledge, attitudes, skills, practice behavior, or clinical practice outcomes produced by CME persist over time (greater than or equal to 30 days)?

Key Question 3: What is the evidence from systematic reviews about the effectiveness of simulation methods in medical education outside of CME?

• Eight reviews evaluated the role of simulation in skill acquisition, while two reviews evaluated the role of simulators in knowledge acquisition. Simulation methods used in these reviews included computer simulations (e.g., learning cardiovascular physiology), virtual reality (e.g., learning laparoscopic procedures), standardized patients (e.g., learning effective communication), and manikins (e.g., learning physical diagnosis).

• Overall, the direction of evidence pointed to the effectiveness of simulation training, especially in psychomotor skills (i.e., procedures or physical examination techniques) and communication skills.

• The strength of the evidence was considered low, due to the small number of appropriate studies, the scarcity of quantitative data, and a number of study limitations.

Key Question 4: Which characteristics of the audience by themselves or in combination with other characteristics influence the effectiveness of certain educational techniques?

• Thirteen studies examined the influence of audience characteristics on the educational intervention. These included such characteristics as age, gender, practice setting, and years in practice, among others.

• The small and heterogeneous studies available did not allow us to reach definitive conclusions regarding the influence of audience characteristics on the effectiveness of CME.

Key Question 5: Which external factors by themselves or in combination with other factors reinforce the effects of CME in changing behavior?

• Five studies examined the influence of external factors on the educational intervention. The small and heterogeneous studies available did not allow us to reach definitive conclusions regarding the influence of external factors on the effectiveness of CME.

Key Question 6: What is the reported validity and reliability of the methods that have been used for measuring the effects of CME in terms of: a) imparting knowledge, b) changing attitudes, c) acquiring skills, d) changing practice behavior, or e) changing clinical practice outcomes?

• Forty-five of 136 articles (33 percent) reported the validity and/or reliability within the study population of at least one evaluation method for assessing the effectiveness of CME, for a total of 61 methods. Validity refers to the degree to which a method truly measures what it is intended to measure. Reliability refers to the consistency or reproducibility of measurements.

• Of the 61 evaluation methods with validity or reliability reported, 29 evaluation methods were drawn from previous studies, and 24 were created for the current studies. For eight methods, the source was unclear. Authors frequently did not report reliability testing within the new study population for methods found to be reliable in other populations.

• Of 61 evaluation methods with validity or reliability reported, 16 (26 percent) included descriptions of validity alone, 28 (46 percent) included descriptions of reliability alone, and nine (15 percent) had descriptions of both validity and reliability. For six methods (10 percent), the methods were described as valid and/or reliable, but the specific type of validity or reliability was not reported.

• The most common type of outcome evaluated by valid and/or reliable evaluation methods involved practice behaviors, for 31 out of 61 methods (51 percent). Knowledge or cognitive skills were evaluated by 15 methods (24 percent). Attitudes were evaluated by seven methods (12 percent). Skills were evaluated by 11 methods (18 percent). Clinical outcomes (with or without practice behaviors) were evaluated by 8 methods (13 percent).

• Among these 61 methods, content validity (i.e., the degree to which an instrument accurately represents the skill or characteristic it was designed to measure) was the most commonly reported type of validity (27 percent).

• Among these 61 methods, inter-rater reliability (28 percent) and internal consistency (23 percent) were the most common types of reliability reported. Inter-rater reliability is the degree to which measurements are the same when obtained by different persons. Internal consistency is a measure of how well items reflecting the same construct yield similar results.

Conclusions

Overall, despite the generally low quality of the evidence, most of the studies reviewed suggested that CME is effective, at least to some degree, in not only achieving, but also in maintaining the objectives studied. Despite the wide variety of CME techniques, media, and exposures used, and the heterogeneity of the studies, we found common themes among studies which applied across objectives. For example, when assessing the effectiveness of CME across domains, print media seemed to be less effective than live media, and multimedia activities generally seemed more effective than single media. In addition, interactive techniques seem to be more effective than non-interactive ones, and multiple exposures to the CME activity seem to be more effective than a single exposure. Thus, the evidence supports consideration of these attributes of effective educational interventions when designing a CME course.

We evaluated the effect of simulation methods in medical education by conducting a review of systematic reviews. Although we found that simulation training generally was effective, especially in the dissemination of psychomotor skills (e.g., procedures or physical examination techniques), studies which examined simulation did not review outcomes along the entire continuum of domains (i.e., knowledge through clinical outcomes), and were heterogeneous enough that few other conclusions could be drawn.

We also studied whether certain internal (audience) and external characteristics or factors may affect the effectiveness of CME. We found that the small and heterogeneous studies available did not allow us to reach definitive conclusions regarding the influence of audience characteristics or external factors on the effectiveness of CME.

Limitations

This review has several important limitations. The heterogeneous nature of the studies precludes a quantitative summary of the effectiveness of CME. The educational interventions studied targeted different types of audiences, using multiple types of objectives across diverse content areas. This makes it difficult to generalize results from one field of CME to another.

Furthermore, we cannot draw firm conclusions about the effectiveness of CME because of the generally low quality of study designs, the variable quality of reporting in studies, and the lack of valid and reliable CME evaluation tools. Although we used a comprehensive search strategy, we cannot rule out some degree of publication bias. The review does point out a lack of standardization of approaches to CME research in general, including the lack of standardization for definitions of controls. The CME literature in general lacks standardization of terminology related to media type, educational techniques, and exposure volume, which makes it difficult to determine the impact of these factors on the effectiveness of CME.

Future Research Implications

• Educators should develop strategies for identifying and prioritizing the gaps in our knowledge about CME that should be the focus of further research.

• More randomized controlled studies of CME should be performed with clear definition of intervention and control groups and measurement of effectiveness at multiple points post-intervention. Such studies should focus on high priority areas given the resource limitations that educators face in conducting research on CME.

• To advance such research on CME, leaders in medical education could develop a national agenda on what is needed most to improve the effectiveness of CME.

• In developing a national agenda for research on CME, educational leaders should establish a clear definition of what constitutes CME. For example, does quality improvement or practice improvement alone constitute CME?

• Future research on CME should include development of more standardized approaches to describe CME interventions, media, techniques, and exposure volumes.

• Further studies should examine emerging methods of CME such as Internet-based CME that could be available to clinicians at the point of care.

• Future research on CME should be based on a sound conceptual model of what influences the effectiveness of CME.

Data Abstracted to Assess the Effectiveness of CME (Key Questions 1 and 2)

Table 2

Definitions of techniques/educational methods

Table 2

Definitions of techniques/educational methods

Technique/Educational method	Definition
Academic detailing	Detailing provided by an institution or hospital
Audience response systems	Addresses knowledge objectives. Used in combination with live lectures or discussion groups, these are computerized feedback tools that allow the teacher/instructor to pose a question to a large group and receive immediate feedback from each learner which is collated and presented on a screen. Instructor may choose to alter content based on audience response
Case-based learning	Addresses higher order knowledge and skill objectives. Actual or authored clinical cases are created to highlight learning objectives; clinical material is presented and followed with questions usually determined by the instructor
Clinical experiences	Addresses skill, knowledge and attitudinal objectives. Generally refers to a preceptorship or observership with an expert, as in attending a specialty clinic or operating room
Demonstration	Addresses skill and or knowledge (knows how) objectives; can be presented live, or with video or audio. Teacher determines amount and pace of content
Discussion group	Addresses knowledge, especially application or higher order knowledge, or affective objectives; usually requires preparation with readings, or another experience, such as viewing a videotape, or a role play. Can be facilitated by instructor, but group often determines content
Feedback	The provision of information about an individual's performance to learners
Lecture	Presentation of knowledge content; live, video, audio or slide presentation available online. Teacher/instructor determines amount and pace of content
Mentor/Preceptor	Addresses higher order cognitive, skill and affective objectives. Learner is paired with a mentor who may observe, review documentation of performance, advise, coach, and facilitate learning
Point of care	Addresses knowledge and higher order cognitive objectives (decision-making). Information which is provided at the time of clinical need, integrated into chart or electronic medical record
Problem-based learning or team-based learning	Addresses higher order knowledge objectives, metacognition and some skill (group work) objectives. A clinical scenario is presented to a team, who identify the learning objectives, assign information-seeking tasks, and return to share information and answer questions about the case. Can be facilitated or non-facilitated
Programmed learning	Addresses knowledge objectives. Content is delivered in sequential steps, which are tested with the learner, before moving to the next, usually more complicated step. Pace is determined by the learner, but objectives are set by the program (teacher). Can be delivered in text or online
Readings	Presentation of knowledge content or background for attitudinal objectives. Requires learner to complete; can be done at learner's pace. Teacher/instructor directed or self-directed (e.g., journals, newsletters, searching online)
Role play	Addresses skill, knowledge and affective objectives. Learners assume role of patients and/ or clinicians in practicing focused encounters around training problems, usually when standardized patients are unavailable. Encounter may be recorded and reviewed or followed with a discussion group. Rarely used as sole method of education
Simulation (other than standardized patient or role-play)	Addresses knowledge and skill objectives; ability to simulate potentially addresses higher order integrative objectives, such as responding to an emerging clinical situation, understanding the unfolding of a protein structure, working in teams. Technology can be used for simulation training of procedures, as in endoscopy virtual reality trainers or anesthesia simulators. Includes also models, such as joint injection and suture. Requires active participation of learner; can use multiple learners in some scenarios
Standardized patient	Addresses skill and some knowledge and affective objectives. Usually used for communication skills training and assessment, the standardized patient or simulated patient is trained in a specific patient scenario and presentation of a clinical problem. Encounter may be audio or videotaped and timed. Review offers opportunity for reflection and “replay” of the scenario
Writing/Authoring	Addresses knowledge and affective objectives. Can include authoring test items and participation in test development. Journaling is used frequently for affective objectives, and may be followed with discussion groups or review with a mentor

Additionally, each reviewer independently completed a study quality form and a reporting of adult learning principles form. The study quality form was based on the Jadad criteria⁸ to assess randomization, blinding, and withdrawals, and included additional questions regarding power calculations. On the reporting of adult learning principles form, reviewers assessed how well the article reported the extent to which the CME activity incorporated adult learning principles, such as enabling learners to be active contributors to their learning, relating the curriculum to learners' current experiences, and tailoring the curriculum to learners' past experiences (Romsai Boonyasai, personal communication). The questions on the reporting of adult learning principles form was derived from a review of adult education.⁹

Data Abstracted from Systematic Reviews on the Effectiveness of Simulation in Medical Education (Key Question 3)

Table 3

Definitions of simulation types

Table 3

Definitions of simulation types

Simulation type	Definition
Full simulation	Whole room or whole patient simulations
Partial task simulation	The use of products to learn or practice a specific skill, such as intubation heads, central venous line chests, intraosseous line legs, or umbilical artery cannulation trainers
Computer simulation	The use of computer programs that allow the student to practice decision making skills, specific knowledge sets such as Advanced Cardiac Life Support (ACLS) trainers and trauma management trainers
Virtual reality	The use of advanced computerized technology to allow students to learn or practice how to perform cardiac catheterizations, colonoscopies, bronchoscopies, ureteroscopies, laparoscopic surgery, hysteroscopy, arthroscopy, ocular surgery, intravenous line placement, etc.
Standardized patient	The use of individuals trained to play the roles of patients, family members, or others to allow students to practice physical exam skills, history taking skills, communication skills, etc.
Role play	Participants play roles of patients, family members, or others to allow practice of communication skills, etc.

Each reviewer also independently completed a study quality form. The study quality form was based off of the QUORUM statement¹⁰ and assessed the reporting of the study question, search methods, inclusion/exclusion criteria, analysis, quality assessment, and conclusions. Additional questions regarding assessment of publication bias were included.

Data Abstracted to Assess the Influence of Audience Characteristics and/or External Factors on the Effectiveness of CME (Key Questions 4 and 5)

For studies addressing the influence of audience characteristics and/or external factors (Key Questions 4 and 5), an audience characteristics/external factors form (see Appendix E, Data Abstraction Review Forms) was completed. Data abstracted to this form included the audience characteristic or external factor that was being analyzed, whether a primary goal of the study was to assess the effects of this audience characteristic or external factor, any covariates used in the analysis, and a qualitative summary of the results. Additionally, reviewers abstracted data regarding general study characteristics, CME activity characteristics, outcomes, study quality, and the reporting of adult learning principles.

Data Abstracted to Assess the Validity and/or Reliability of Tools (Key Question 6)

Data regarding validity and reliability of methods used to assess the effectiveness of CME (Key Question 6) were abstracted to a validity/reliability of tools form (see Appendix E, Data Abstraction Review Forms). Articles need not have used the specific terms “validity” or “reliability” to be included in Key Question 6. If authors did not label the specific type of validity or reliability reported, we classified the type based on the definitions from Reed et al.¹¹ Articles that used a previously validated/reliable method were included if the authors described the method as valid/reliable or described a process or statistic used for psychometric testing. Reviewers also abstracted data regarding general study characteristics, CME activity characteristics, outcomes, study quality, and the reporting of adult learning principles.

Knowledge Outcomes

A total of 22 studies addressing 23 objectives demonstrated improvements in knowledge.¹⁹³³–⁵³ This represents 79% of the 28 studies with an adequate control group. Of these, 6 studies³³–³⁸ addressing 6 objectives did not clearly report the duration of evaluation, 1 study³⁹ addressing 1 objective demonstrated short-term improvement in knowledge, and 15 studies addressing 16 objectives demonstrated long-term improvements in knowledge. The studies were too heterogeneous too identify any global similarities, but any specific information related to the type of media, educational technique or exposure volume will be addressed in those subsequent sections.

Four studies addressing four objectives failed to show improvements in knowledge.³²⁵⁴–⁵⁶ No study demonstrated a regression in knowledge. Of these four studies, one study³² did not clearly report on the duration of evaluation while the three remaining studies evaluated long-term knowledge changes. No study was identified that evaluated short-term knowledge and failed to show a change in knowledge. Of the three studies that considered long-term outcomes, the study by Elliott et al demonstrated trends toward improved knowledge but these slight improvements lacked statistical significance.⁵⁵ In the remaining two studies, one failed to show improvements in knowledge regarding bioterrorist attacks through the voluntary participation in a web-based educational program and the other failed to show an improvement in knowledge regarding blood pressure control after a mailed CME program.

Two studies addressing two objectives demonstrated mixed results.⁵⁷⁶² One study did not clearly report on the duration of evaluation and one reported on long-term effects. The study that did not clearly report duration evaluated the impact of an educational intervention aimed at medical care evaluation committees and demonstrated a statistical improvement in only one of the three committees.⁵⁷ The study by Chodosh et al considered an evaluation nine months after the intervention. In this study, more intervention group physicians answered knowledge-related questions on capacity determination for patients with possible dementia, but found no differences between intervention group and control group on questions regarding dementia evaluation, patient safety, or depression treatment.

In summary, 78% of the studies with an adequate control group demonstrated that CME activities were effective at improving knowledge with the majority (68%) of these studies demonstrating long-term improvements in knowledge.

Short-term and long-term effects of CME media methods on knowledge. Studies were classified by objectives met and evaluation duration as described above and by media utilized for the educational intervention. The media classifications included multi-media, single media and single versus multi-media.

As stated in the above section, 22 studies addressing 23 objectives demonstrated improvements in knowledge. Seven studies addressing eight objectives evaluated a single media intervention.^{3940434548505356} Three of the studies utilized the internet and the remaining four utilized live media. The three studies that utilized the internet demonstrated improved short-term knowledge in one study³⁹ and long-term knowledge in two studies.⁴³⁵⁰ All four of the studies that utilized live media demonstrated long-term improvements in knowledge. Twelve studies addressing 12 objectives utilized multimedia interventions to demonstrate knowledge benefits.¹⁹³³–^{37414246495152} Five studies did not clearly report on the duration and seven studies demonstrated long-term improvements in knowledge. Three studies compared a single media-based intervention to a multimedia intervention.³⁸⁴⁴⁴⁷ All three studies included at least one concurrent control group that only utilized print material and in all three cases multi-media outperformed the print-based single media intervention group.

The four studies that did not demonstrate improvements in knowledge utilized single media (print) in one study⁵⁶ that evaluated long-term knowledge, multimedia (live, audio and print) in one study³² in which evaluation duration was not clearly defined, multimedia (live and print) in one study⁵⁵ that considered long-term knowledge changes, and one study⁵⁴ that compared single to multi-media for long-term benefits. In this study the comparison was between a live intervention and a live internet with non-real time reading material.

Of the two studies that demonstrated mixed results, one study⁵⁷ utilized a single media intervention (live) and the other study⁶² utilized multimedia intervention (live internet with not real time print).

When grouped solely by media classification, 9 studies addressing 10 objectives used a single media with 7 demonstrating benefits, 15 studies addressing 15 objectives considered multi-media interventions with 12 demonstrating benefits, and 4 studies addressing 4 objectives compared single to multi-media with 3 of these studies demonstrating that multi-media interventions had greater benefit.

Given the heterogeneity of the studies the only recognized trends were that multimedia seems better than a single media intervention and that print interventions are either not beneficial or very weak in their ability to lead to improved knowledge.

Short-term and long-term effects of CME educational techniques on knowledge.Studies were classified by objectives met and evaluation duration as described above and by technique utilized for the educational intervention. The technique classifications included multiple techniques, single technique and single versus multiple techniques. Two of the studies that have not been previously included because of a lack of a control group can be utilized in this section because the concurrent comparison group did indeed utilize a different technique permitting some head to head comparisons.²⁵⁶⁶ This means that for this section there are 30 studies addressing 31 objectives.

Only one study that met its objectives utilized a single technique, academic detailing, and this study demonstrated improvements in long-term knowledge.¹⁹ Eighteen studies addressing 18 objectives utilized multiple techniques to improve knowledge.³³–³⁷³⁹⁴¹–⁴⁶⁴⁸–⁵³ Five studies³³–³⁷ that demonstrated improved knowledge did not clearly report the duration, 1 study³⁹ demonstrated short-term knowledge improvement and 12 studies demonstrated long-term knowledge improvements. The majority of the studies that demonstrated improvement but did not clearly report the duration included case-based learning as a technique in combination with techniques ranging from discussion groups to independent reading. Of the 12 studies that demonstrated long-term improvements in knowledge the majority integrated multiple techniques and most commonly combined case-based learning with discussion groups with independent reading, several utilized standard lectures with readings, and some technique combinations ranged from lecture with a standardized patient to lecture with team-based training. Three studies addressing four objectives compared a single technique versus multiple techniques demonstrated improvements in knowledge.³⁸⁴⁰⁴⁷ One study did not clearly report duration and two studies addressing three objectives demonstrated long-term knowledge improvements. The study that did not clearly report duration demonstrated a greater benefit to the combination of case-based learning with readings when compared to readings alone.³⁸ One study that addressed two objectives compared problem-based learning to the combination of lecture and discussion groups and demonstrated that that problem-based learning group was more effective regarding knowledge of diagnosis and management of headache. The other study compared discussion groups with readings to readings alone and the combination was more effective at increasing knowledge regarding compliance adherence.

Four studies of four objectives did not demonstrate improved knowledge.³²⁵⁴–⁵⁶ One study that utilized multiple techniques that included lecture, discussion groups, role playing and feedback failed to demonstrate improved knowledge and did not clearly report the duration.³² Two studies failed to demonstrate improvements in knowledge despite utilizing multiple techniques.⁵⁵⁵⁶ One combined lecture with case-based learning with discussion groups and readings and failed to show improvements in pain knowledge, while the other combined readings with chart cue materials and failed to show improvements in knowledge of hypertension. One study compared a single technique, lecture to a combination of techniques that included lecture with case-based learning and readings and failed to show an improvement in knowledge of bioterrorism.

The two studies that were added to this section deserve a greater explanation. Greenberg et al compared lecture to case-based learning and the case-based learners demonstrated improvements in post-test questions regarding common pediatric problems as compared to 29% improvement in the lecture group.²⁵ Unfortunately, these improvements seen in the case-based group were not significant at six and nine months after the intervention. Heale et al compared three groups, one receiving lectures, one case-based learning with discussion groups, and the final group received problem-based learning.⁶⁶ This study also failed to demonstrate benefits of any technique either in short or long-term knowledge. Thus these two studies, when added to the four above studies that did not demonstrate improvements means a total of six studies did not show improved knowledge.

One of the two studies that demonstrated mixed results did not clearly report duration while the other considered long-term knowledge improvements.⁵⁷⁶² The study⁵⁷ that did not report duration compared a combination of case-based learning with discussion groups to control while the other compared lecture with discussion group to control.

When grouped solely by technique classification then two studies addressing two objectives utilized a single technique with one study showing improvements through the use of academic detailing and one showing no knowledge improvement through the use of readings alone. Twenty-three studies addressing 23 objectives utilized multiple techniques with 18 demonstrating benefits, 3 no benefits, and 2 mixed results. Five studies addressing six objectives compared single versus multiple techniques. Two studies addressing two objectives demonstrated benefits of multiple techniques as compared to single, one study addressing two objectives demonstrated benefits to single technique(problem-based learning) compared to multiple and two studies addressing two objectives did not demonstrate any benefit of single as compared to multiple techniques.

The outcomes from this section are also heterogeneous but it does appear that multiple techniques that most commonly include case-based learning seem to be more likely to be associated with improvements in knowledge.

Short-term and long-term effects of the amount of exposure on knowledge. Studies were classified by objectives met and evaluation duration as described above and by exposure volume for the educational intervention. The exposure volume classifications included multiple exposures, single exposure and single versus multiple exposures. The two studies added in the previous section do not apply to this section given the lack of control groups. In addition, one additional study³⁸ did not adequately describe the exposure and thus is excluded from analysis in this section leaving 27 studies addressing 28 objectives.

There were 21 studies addressing 22 objectives which demonstrated an improvement in knowledge.¹⁹³³–³⁷³⁹–⁵³ Five studies addressing five objectives evaluated a single exposure volume with one study³⁶ not clearly reporting the duration and four studies^44454853 demonstrating long-term improvements. Three of these considered knowledge at 6 months after intervention and one at 24 months after intervention. Twelve studies addressing 12 objectives utilized multiple exposure volumes. Three studies did not clearly report duration,³³³⁵³⁷ one study demonstrated short-term knowledge gains,³⁹ and the final eight studies all demonstrated long-term knowledge gains.^1941434649–⁵² The shortest time interval to evaluation in these long-term beneficial studies was 3 months and the longest was 15 months. Four studies addressing five objectives compared a single exposure to multiple exposures and demonstrated knowledge gains. One study that did not clearly report duration demonstrated that multiple exposures were better than a single exposure at improving knowledge of office-based dermatologic procedures.³⁴ The other three studies addressing four objectives all demonstrated that multiple exposures were better than a single exposure at improving knowledge.⁴⁰⁴²⁴⁷

Four studies addressing four objectives did not demonstrate knowledge improvements. No study was identified that evaluated a single exposure only. Three studies evaluated multiple exposures with one³² not clearly reporting the duration and the other two⁵⁵⁵⁶ failing to demonstrate long-term knowledge improvements. One study that compared single versus multiple exposures did not show any difference from baseline knowledge at one and six months after intervention in either group.⁵⁴

No studies were identified that utilized a single exposure only or compared a single versus multiple that demonstrated mixed results. Two studies demonstrated mixed results and both utilized multiple exposures.⁵⁷⁶² One study⁵⁷ did not clearly report on duration and the other study demonstrated mixed results at the nine month evaluation.

When grouped solely by exposure volume then all five studies that evaluated a single exposure demonstrated improved knowledge. Twelve of the 17 studies that utilized multiple exposures demonstrated knowledge improvements with an additional 2 demonstrating mixed results. The majority (67%) of these were able to demonstrate long-term knowledge benefits. Of the five studies that compared a single exposure to multiple exposures, four (80%) demonstrated a greater benefit to multiple exposures as compared to a single exposure.

In summary, despite the heterogeneity of these studies it appears that despite the fact that all five studies that utilized a single exposure demonstrated benefit the head-to-head comparison studies imply that when possible multiple exposures produces better knowledge gains.

Summary of the effects of CME on knowledge. The heterogeneity of the studies precludes firm conclusions, but the trends demonstrated that CME is effective at producing both short-term and long-term knowledge gains and that when possible, multimedia, multiple techniques, and multiple exposures should be used.

Attitude Outcomes

A total of 22 studies addressing 26 objectives demonstrated improvements in attitude.¹³³⁴–^{363940424750525368}–⁷⁸ This represents 71% of the 31 studies with an adequate control group. Of these, six studies addressing seven objectives did not clearly report the duration of evaluation,¹³³⁴–³⁶⁶⁸⁶⁹ one study addressing two objectives demonstrated short-term improvement in attitudes,³⁹ and 15 studies addressing 17 objectives demonstrated long-term improvements in attitude.^{40424750525370}–⁷⁸ The studies were too heterogeneous too identify any global similarities, but any specific information related to the type of media, educational technique or exposure volume will be addressed in those subsequent sections.

Four studies addressing four objectives failed to show improvements in attitude.^37525562 No study demonstrated a regression in attitude. Of these four studies, one study³⁷ did not clearly report on the duration of evaluation while the three remaining studies evaluated long-term attitudinal changes. No study was identified that evaluated short-term attitudes and failed to show a change in attitudes. Of the three studies that considered long-term outcomes, the study by Elliott et al demonstrated trends toward improved attitudes but these slight improvements lacked statistical significance.⁵⁵ In the remaining two studies, one failed to show improvements in attitudes regarding providers' perceptions of quality of care nine months after intervention.⁶² The other failed to show an improvement in attitude regarding efficacy of cholesterol lowering practices.⁵²

Two studies addressing two objectives demonstrated mixed results.⁶²⁸¹ Both studies reported on long-term effects. The study by Chodosh et al considered an evaluation nine months after the intervention. In this study, more intervention group physicians endorsed the statement, “Older patients with dementia are difficult to manage in primary care,” but no other differences in attitudes regarding dementia were identified.⁶² The study by Norris et al utilized an evaluation six months after intervention and noted that intervention group providers had significant improvements in three of ten tested attitudes, including they felt by self report that they were more likely to counsel patients regarding physical activity than control providers.⁸¹

In summary, 85 percent of the studies with an adequate control group demonstrated that CME activities were effective at improving attitudes with the majority (68%) of these studies demonstrating long-term improvements in attitudes.

Short-term and long-term effects of CME media methods on physician attitudes.Studies were classified by objectives met and evaluation duration as described above and by media utilized for the educational intervention. The media classifications included multi-media, single media and single versus multi-media.

As stated in the above section, 22 studies addressing 26 objectives demonstrated improvements in attitudes. Seven studies addressing eight objectives evaluated a single media intervention.^{13394050537172} Two of the studies³⁹⁵⁰ utilized the internet, one¹³ utilized a computer-based program and the remaining four utilized print media.^40537172 The two studies addressing three objectives that utilized the internet demonstrated improved short-term attitudes in one study and long-term attitudes in the other. All four of the studies that utilized print media demonstrated long-term improvements in attitudes. The one study that utilized computer-based education for its intervention did not clearly report the duration. Twelve studies addressing 15 objectives utilized multi-media interventions to demonstrate attitudinal benefits.³⁴–^{36425268697375}–⁷⁸ Five studies addressing six objectives did not clearly report on the duration and seven studies addressing nine objectives demonstrated long-term improvements in attitudes. Three studies compared a single media-based intervention to a multi-media intervention.⁴⁷⁷⁰⁷⁴ All three studies included at least one concurrent control group that only utilized print material and in all three cases multi-media outperformed the print-based single media intervention group. All three studies showed long-term improvements in attitudes.

The four studies that did not demonstrate improvements in attitude all utilized multimedia.^37525562 One study combined live with print and did not clearly report on duration.³⁷ The other 3 studies considered long-term attitudinal changes at 9 months in one study and 15 months in the other 2 studies. The study that considered outcomes at nine months utilized a combination of live internet with print material while the other two studies combined live with print in one and live with video and print in the other.

Both of the two studies that demonstrated mixed results utilized multimedia approaches.⁶²⁷⁹ One study utilized live internet with print material and the other study utilized live in conjunction with print and a followup phone call.

When grouped solely by media classification, then 7 studies addressing 8 objectives utilized a single media with all 7 demonstrating attitudinal benefits, 18 studies addressing 26 objectives considered multimedia interventions with 12 demonstrating benefits, and 3 studies addressing 3 objectives compared single to multi-media with all 3 of these studies demonstrating that multimedia interventions had greater benefit.

Given the heterogeneity of the studies the only recognized trends were that multimedia appears better than a single media intervention and that print interventions are either not beneficial or very weak in their ability to lead to improve attitudes.

Short-term and long-term effects of CME educational techniques on physician attitudes. Studies were classified by objectives met and evaluation duration as described above and by technique utilized for the educational intervention. The technique classifications included multiple techniques, single technique and single versus multiple techniques. One study that had not been previously included because of a lack of a control group can be utilized in this section because the concurrent comparison group did indeed utilize a different technique permitting some head to head comparisons.⁶⁶ This means that for this section there are 23 studies addressing 27 objectives.

Only one study that met its objectives utilized a single technique, reading, and this study did not clearly report duration.¹³ Seventeen studies addressing 21 objectives utilized multiple techniques to improve attitudes. Five studies³⁴–³⁶⁶⁸⁶⁹ addressing 6 objectives that demonstrated improved attitudes did not clearly report the duration, 1 study³⁹ addressing 2 objectives demonstrated short-term attitudinal improvement and 11 studies^4250525371–⁷³⁷⁵–⁷⁸ addressing 13 objectives demonstrated long-term attitudinal improvements.

The majority of the studies that demonstrated improvement but did not clearly report the duration included case-based learning as a technique in combination with techniques ranging from discussion groups to independent reading. Of the 11 studies that demonstrated long-term improvements in attitude the majority integrated multiple techniques and most commonly combined case-based learning with discussion groups with independent reading, several utilized standard lectures with readings, and some technique combinations ranged from lecture with a standardized patient to lecture with team-based training. Four studies addressing four objectives compared a single technique versus multiple techniques demonstrated improvements in attitudes.^40477074 All four studies demonstrated long-term knowledge improvements. Three of these four studies utilized readings as the sole technique and one utilized a discussion group. All four studies demonstrated that multiple techniques were better than single techniques.

Four studies of four objectives did not demonstrate improved attitudes.^37525562 One study that utilized multiple techniques that included live and print failed to demonstrate improved attitudes and did not clearly report the duration.³⁷ Two studies failed to demonstrate improvements in attitudes despite utilizing multiple techniques.⁵²⁶² One combined lecture with discussion group and failed to show improvements in provider perceptions about quality of care for patients with dementia, while the other combined case-based learning with discussion with readings, with standardized patients and failed to show improvements in attitudes regarding cholesterol lowering practices. Only one study compared a single technique versus multiple and failed to show an improvement in provider attitudes regarding pain.⁵⁵

The one study that was added to this section deserves a greater explanation. Heale et al⁶⁶ compared three groups, one receiving lectures, one case-based learning with discussion groups, and the final group received problem-based learning. This study demonstrated improvements in attitudes that were greatest in the problem-based learning participants.

Both studies that demonstrated mixed results assessed long-term attitudinal change⁶²⁸¹ and both utilized multiple techniques. These included lecture with a point-of-care opinion leader in one study and lecture with discussion groups in the other.

When grouped solely by technique classification then one study addressing one objective utilized a single technique and it demonstrated attitudinal improvement. Twenty-two studies addressing 26 objectives utilized multiple techniques with 17 demonstrating benefits, 3 no benefits, and 2 mixed results. Five studies addressing five objectives compared single versus multiple techniques. Four studies demonstrated greater attitudinal change with the utilization of multiple techniques as compared to a single technique and one study showed no improvement.

The outcomes from this section are also heterogeneous but it does appear that multiple techniques that most commonly include case-based learning seem to be more likely to be associated with improvements in attitudes.

Short-term and long-term effects of the amount of exposure on physician attitudes.Studies were classified by objectives met and evaluation duration as described above and by exposure volume for the educational intervention. The exposure volume classifications included multiple exposures, single exposure and single versus multiple exposures. The one study added in the previous section does not apply to this section given the lack of control group. In addition, one additional study⁵⁸ did not adequately describe the exposure and thus is excluded from analysis in this section leaving 24 studies addressing 25 objectives.

There were 22 studies addressing 25 objectives which demonstrated an improvement in attitude.¹³³⁴–^{363940424750525368}–⁷⁸ Seven studies addressing eight objectives evaluated a single exposure volume with two studies³⁶⁶⁸ not clearly reporting the duration and five studies^5370717477 addressing six objectives demonstrating long-term improvements. Eleven studies addressing 12 objectives utilized multiple exposure volumes. Three studies¹³³⁵⁶⁹ did not clearly report duration, one study³⁹ demonstrated short-term knowledge gains, and the final seven studies^{50527273757678} addressing eight objectives all demonstrated long-term knowledge gains. Four studies addressing five objectives compared a single exposure to multiple exposures and demonstrated attitudinal improvements.^34404247 One study³⁴ addressing two objectives did not clearly report duration, while the remaining three studies all demonstrated long-term improvements in attitudes.

Four studies addressing four objectives did not demonstrate attitude improvements. No study was identified that evaluated a single exposure only or that performed a comparison between single and multiple exposures. Three studies evaluated multiple exposures with one³⁷ not clearly reporting the duration and the other two⁵⁵⁶² failing to demonstrate long-term knowledge improvements.

No studies were identified that utilized a single exposure only or compared a single versus multiple that demonstrated mixed results. Two studies demonstrated mixed results and both utilized multiple exposures.⁶²⁸¹ Both studies assessed long-term attitudes.

When grouped solely by exposure volume then all seven studies that evaluated a single exposure demonstrated improved attitudes. Eleven of the 17 studies that utilized multiple exposures demonstrated attitudinal improvements with an additional two demonstrating mixed results. The majority (64%) of these were able to demonstrate long-term attitudinal benefits. Of the four studies that compared a single exposure to multiple exposures all demonstrated a greater benefit to multiple exposures as compared to a single exposure.

In summary, despite the heterogeneity of these studies it appears that there is a trend toward multiple exposures being of greater benefit for attitudinal change then a single exposure, although it must be pointed out that all seven studies that evaluated a single exposure indeed demonstrated improvements in attitudes.

Summary of the effects of CME on physician attitudes.The heterogeneity of the studies precludes firm conclusions, but the trends demonstrated that CME is effective at producing both short-term and long-term attitudinal gains and that when possible, use of multimedia, multiple techniques, and multiple exposures should produce better attitudinal outcomes.

Skills Outcomes

Of those 15 studies, 10 had skill outcomes that met the objectives of the study.^{32333640497284}–⁸⁷ One study had three skill outcomes that all met the objectives,⁸⁴ and another study had two reported skill outcomes that met the objectives.³³ The remaining studies each had one skill outcome. Two studies had skill outcomes that did not achieve the study objectives,¹³⁷⁷ and one study had mixed results that did not clearly meet the study objectives.⁸⁸ Two studies compared different methods and techniques of CME without a separate control group that did not receive CME;⁶⁰⁸⁰ therefore, the overall effectiveness of CME cannot be discerned from these studies. Given that 10 out of 13 studies that included control groups and reported on skills (13 out of 16 outcomes) met the objectives and given the varied nature of the studies, the literature does indicate that CME is effective in this area, particularly at developing cognitive skills. Little can be said about the effectiveness of CME for psychomotor skills given the paucity of data in this area.

Short-term effects of CME media methods on skills.The media methods of CME that were included in the studies that met the study objectives regarding skills included seven that used live media,^{32333640497284} four that included print materials,^32334985 two that included video methods,³²³⁶ two that included audio methods,³²⁴⁹ two that used the Internet (not real time),⁸⁶⁸⁷ and one that used computers (off-line).³³ However, in several cases the same media methods were used in all experimental arms and therefore no conclusions can be drawn from the study outcomes applying to specific CME methods. This situation applied to live methods in one study,⁴⁰ audio methods in one study,⁴⁹ and print materials in two studies.⁴⁹⁸⁵ In addition, one study was not clear about which groups received particular methods including live, print, and computer (off-line) media.³³ The studies that did not clearly meet skills objectives included live,^{77 88} video,^{77 88} and computer (off-line) methods.¹³ Two studies directly compared different methods of CME but did not include a control group without CME.⁶⁰⁸⁰ One of these studies showed no difference between print, computer (off-line), and live methods.⁸⁰ However, another study showed that live methods were superior to video and print combined.⁶⁰ Given this result and the dominance of live methods among the studies that met their skills objectives, the data suggests that live methods have the greatest impact on the effectiveness of CME regarding skills outcomes. Given the paucity of data and the varied results, little can be said about the relative effectiveness of other CME media methods on affecting skills.

Long-term effects of CME media methods on skills.The six studies that addressed long-term skill outcomes beyond 30 days and met their skills objectives used a variety of media methods including four using live methods,^40497291 two using print materials,⁴⁹⁸⁵ one using audio methods,⁴⁹ and one using the Internet (not real time).⁸⁶ However, the one study that used audio methods had audio in all experimental arms,⁴⁹ the two that used print media did so in all groups,⁴⁹⁸⁵ and one of the four that used live methods did so in all groups.⁴⁰ Therefore, the experimental effect on long-term outcomes in studies that met their skills objectives was seen only for three studies using live media,⁴⁹⁷²⁸⁴ and one study that used the Internet (not real time).⁸⁶ The one study that did not meet its objectives regarding long-term skill outcomes used live and video as the methods.⁷⁷ The studies that directly compared different methods of CME did not address long-term skill outcomes.^{60 80} Based on the limited data, it is difficult to draw conclusions on particular media methods of CME that have a greater or lesser impact on long-term skill outcomes.

Short-term effects of CME educational techniques on skills.Varied educational techniques were used in the studies that met their objectives regarding short-term skill outcomes. These included lectures in six studies,^323640497284 discussion groups in five studies,^3249728485 readings in five studies,^3233498586 case-based learning in four studies,^36498687 feedback in three studies,³²⁴⁹⁸⁶ role play in three studies,³²⁷²⁸⁴ and clinical experiences in two studies.³²⁷² Listserv,⁸⁵ programmed learning,⁸⁶ problem-based learning,⁴⁰ audio-taped encounters,⁷² and standardized patients,⁸⁴ were seen in one study each in the studies that met their skills objectives. Of note, readings were used as a technique in both experimental and control groups in one study.⁸⁵ In the studies where the skills objectives were not clearly met by the outcomes, the following techniques were used: demonstration and lecture in two studies,⁷⁷⁸⁸ and readings,¹³ discussion groups,⁸⁸ feedback,⁸⁸ programmed learning,⁸⁸ and role play⁸⁸ in one study each. Two studies that did not include a control group compared the techniques of readings versus readings with demonstration versus mentor/preceptor with simulation.⁸⁰ No difference was seen with these different techniques. In another study without a control group, the techniques of demonstration and simulation were compared with demonstration and simulation plus feedback.⁶⁰ The group that included feedback was significantly better when skills acquisition was assessed. Finally, two of the studies that met their objectives regarding skills and included control groups did a comparison of individual techniques of delivering CME. One study compared problem-based learning with lecture, and problem-based learning was significantly better.⁴⁰ Another study compared discussion groups and readings to discussion groups and readings with feedback.⁸⁵ In this study, feedback had no additional effect. Given the limited number of studies, the wide variety of techniques described, and the conflicting results, it is difficult to draw conclusions about the educational techniques that have the greatest and least effect on skills.

Long-term effects of CME educational techniques on skills.The educational techniques used in the studies that met the study objectives regarding long-term skill outcomes included discussion groups in four studies,^49728485 lectures in four studies,^40497284 readings in three studies,⁴⁹⁸⁵⁸⁶ role play in two studies,⁷²⁸⁴ case-based learning in two studies,⁴⁹⁸⁶ and feedback in two studies.⁴⁹⁸⁶ The techniques of clinical experiences,⁷² listserv,⁸⁵ programmed learning,⁸⁶ problem-based learning,⁴⁰ audio-taped encounters,⁷² and standardized patients,⁸⁴ were seen in one study each that met skills retention objectives. The one study that did not meet its objectives regarding long-term skill outcomes used the techniques of demonstration, lecture, and simulation.⁷⁷ The two studies that compared different techniques in CME without a control group did not address long-term skill outcomes.⁶⁰⁸⁰ However, in two studies that did have a control group, one showed that problem based learning is superior to lectures in long-term skill outcomes,⁴⁰ but the other showed no advantage of feedback over demonstration and readings alone.⁸⁵ Given the limited number of studies and the varied techniques, it is difficult to draw conclusions about the educational techniques that that have a greater or lesser effect on long-term skill outcomes.

Short-term effects of the amount of exposure on skills.The majority of the studies that met their skills objectives had multiple exposures to the CME activity. Seven of these studies used multiple exposures,^3240497284–⁸⁶ while one used a single exposure.³⁶ It was unclear how many exposures there were in two of the studies whose skills outcomes met the study objectives.³³⁸⁷ The studies that did not meet the skills objectives included two that used a single exposure,⁷⁷⁸⁸ and one that used multiple exposures.¹³ These results suggest that multiple exposures to CME for skills objectives is superior to a single exposure.

Long-term effects of the amount of exposure on skills.All six of the studies that addressed long-term skill outcomes and met the study objectives used multiple exposures to CME.^40497284–⁸⁶ The one study that did not meet its study objectives regarding long-term skill outcomes used a single exposure.⁷⁷ These results support multiple exposures as having a greater impact on long-term skill outcomes.

Practice Behavior Outcomes

Fifty studies with evaluation duration greater than 30 days met 58 objectives, suggesting long-term retention of CME effectiveness.¹⁹⁴⁴–^{49525362707275}–^7881828295–¹²⁵ Among these, evaluation duration ranged from 6 months or less after the educational intervention (17 studies) to 1 year or greater (30 studies).

A wide mix of objectives was studied. For example, 12 objectives were related to medication prescribing, eight were related to screening standards, 14 were related to physician counseling behaviors (mostly smoking cessation, but also dietary counseling, sexual practices counseling, etc.), 11 were related to guideline adherence, and the remainder were related to physician behaviors pertaining to other topics.

Twenty nine total studies, reporting on 38 objectives, did not meet objectives. Of these, twenty four studies reporting on 33 objectives were evaluated at greater than 30 days.^{141643567073749699101119122126}–¹³⁷ Two studies with evaluation duration of 30 days or less did not meet objectives.¹⁸¹³⁸ Three studies did not report evaluation duration and did not meet objectives.^1357139

Nine studies, evaluating 9 objectives, showed mixed results in terms of their objectives being met.^{2042717388137140}–¹⁴² One study was unclear as to whether it met objectives.¹⁰¹ Fourteen studies, evaluating 17 objectives lacked a control group and therefore did not allow us to assess effectiveness.²³–²⁵⁵⁹⁶⁴–^6783143–¹⁴⁷

Overall, CME interventions were effective in the short- and long-term achievement of practice behavior objectives.

Short-term and long-term effects of CME media methods on practice behavior. The different types of media evaluated included single-media live presentations (20 studies), single-media print materials (9 studies), internet (1 study), other single media (2 studies), multimedia (57 studies), and also single versus multimedia comparisons (15 studies).

Of 20 studies using single live media, 10 studies met 11 objectives, three studies did not meet objectives, three studies showed mixed results and four studies did not have a control group. Of these 20 studies, nine studies with evaluation duration of greater than 30 days met 10 objectives,^{4548537282102107121125} suggesting that the use of single live media had a favorable long-term effect on practice behavior objectives. One study had an evaluation duration of 30 days or less and met objectives using single live media.⁹⁴ Three studies using single live media reported evaluation duration greater than 30 days but showed mixed results.^2071140 Four studies lacked a control group, and no meaningful conclusions could be drawn with regard to the comparative effectiveness of single live media.^24256466 Three studies did not meet practice behavior objectives using single live media. Of these, one did not report evaluation duration,⁵⁷ one reported evaluation duration of 30 days or less,¹³⁸ and one reported evaluation duration greater than 30 days.¹³³

Out of nine studies with ten objectives that examined the impact of single print media, only one met objectives,¹⁵ but it did not report evaluation duration. One study did not meet objectives using single print media and did not report evaluation duration.¹³⁹ One study with evaluation duration of 30 days or less did not meet objectives using single print media.¹⁸ Four studies (five objectives) with evaluation duration greater than 30 days did not meet objectives using single print media.^1456130134 Two studies using single print media reported evaluation duration greater than 30 days, but lacked a control group and no meaningful conclusions could be drawn with regard to the comparative effectiveness of single print media.²³¹⁴³ The evidence suggests that single print media is not effective in the short- or long-term achievement of practice behavior objectives.

The only study using single internet media reported an evaluation duration greater than 30 days and it was unclear whether it met objectives.⁴³ One study did not meet objectives using other single media and did not report evaluation duration.¹³ One study using other single media did not report evaluation duration and lacked a control group.¹⁴⁷ One study with evaluation duration greater than 30 days did not report the media used and lacked a control group.¹⁴⁵

Out of 57 studies (78 total objectives) using multimedia, 40 studies met 47 objectives, 14 studies did not meet 19 objectives, 4 studies showed mixed results, and 4 studies lacked a control group. Of the 40 studies which met objectives, 31 studies with 37 objectives were evaluated at greater than 30 days suggesting that multimedia-based CME has a favorable long-term effect on practice behaviors.^194649526275–^{7881959899101103104106108}–^110112113116–¹²⁰¹²²–¹²⁴¹⁴⁸ Nine studies met 10 objectives using multimedia but did not report evaluation duration.^{323536686979879293} One study using multimedia did not report evaluation duration; it also lacked a control group.¹⁴⁶ Four studies using multimedia reported an evaluation duration greater than 30 days, and showed mixed results.^427388142 Three studies (six objectives) with evaluation duration greater than 30 days lacked a control group and precluded meaningful conclusions.^6583144 Fourteen studies with 19 objectives with evaluation duration greater than 30 days did not meet objectives using multimedia.^{167399101119122126}–^{129131132135136} The evidence suggests that multimedia may have a positive short- and long-term effect on practice behavior objectives.

Out of 15 studies comparing single media and multimedia, 10 studies met 11 objectives and all were evaluated at an interval of greater than 30 days after the educational intervention,^{4447709697100105111114115} suggesting that both single media and multimedia have a positive short and long-term effect on practice behavior objectives, and that multimedia have an advantageous effect. One study comparing single media and multimedia did not report evaluation duration and lacked a control group.⁵⁹ Two studies comparing single media and multimedia reported evaluation duration greater than 30 days and showed mixed results.¹³⁷¹⁴¹ One study reported evaluation duration greater than 30 days, but lacked a control group.⁶⁷ Four studies with evaluation duration greater than 30 days did not meet seven objectives.^707496137

One study did not report the type of media used, reported evaluation duration greater than 30 days, lacked a control group, and was inconclusive about meeting objectives.¹⁴⁵

The evidence suggests that both single media and multimedia may have positive short- and long-term effects on practice behavior, with use of multimedia being advantageous.

Short-term and long-term effects of CME educational techniques on practice behavior.A total of 11 studies reporting on 12 objectives evaluated the impact of a single technique on practice behavior objectives. One study met objectives using a single technique but did not report evaluation duration.¹⁵ One study with evaluation duration of 30 days or less met objectives using a single technique.⁹⁴ One study with evaluation duration greater than 30 days met objectives using a single technique.¹⁹ One study did not report evaluation duration and did not meet objectives using a single technique.¹³ One study with evaluation duration of 30 days or less did not meet objectives using a single technique.¹⁸ Two studies with evaluation duration of greater than 30 days did not meet objectives using a single technique.¹⁴¹⁶ One study was judged inconclusive because of mixed results.²⁰ Three studies using a single technique reported evaluation duration of greater than 30 days, but lacked a control group.²³–²⁵ Two of these three studies did not reach statistical significance. This suggests that a using a single technique may not have a short- or long-term positive effect on practice behavior objectives.

A total of 76 studies with 98 objectives evaluated the short- and long-term impact of multiple techniques on practice behavior objectives. Eight studies met 11 objectives using multiple techniques but did not report evaluation duration^{3235366869799293} and thus did not allow us to distinguish between short- and long-term effects of CME. Thirty-nine studies with evaluation duration greater than 30 days met 45 objectives using multiple techniques.⁴⁴–^{4648495253627276}–^{78818282959899101}–¹¹³¹¹⁶–¹²⁰¹²²–¹²⁴

Two studies did not report evaluation duration and did not meet objectives using multiple techniques.⁵⁷¹³⁹ One study with evaluation duration of 30 days or less did not meet objectives using multiple techniques.¹³⁸ Sixteen studies with evaluation duration of greater than 30 days reporting on 22 objectives did not meet objectives using multiple techniques.^{435673101119122126}–¹²⁹¹³¹–¹³⁶

One study using multiple techniques did not report evaluation duration and lacked a control group.¹⁴⁶ One study using multiple techniques did not report evaluation duration and showed mixed results.¹⁴⁹ Seven studies reporting on 10 objectives using multiple techniques reported an evaluation duration of greater than 30 days but lacked a control group.^64656783143–¹⁴⁵ Six studies showed mixed results.^{42717388140142} One of the six showed a positive effect which was lost 6 months after the intervention.⁷³

The evidence suggests that the use of multiple techniques in CME may have an overall positive short- and/or long-term effect on practice behavior objectives.

A total of 18 studies reporting on 24 objectives compared the use of single and multiple educational techniques in CME.^{4759667074879697100114115121125130137141147148} Ten studies with evaluation duration greater than 30 days met 12 objectives comparing single and multiple techniques,^{47709697100114115121125148} and indicated that multiple techniques may have an advantageous short- and long-term effect on practice behavior objectives. Two studies using single versus multiple techniques did not report evaluation duration and neither had a control group, precluding further meaningful conclusions.⁵⁹¹⁴⁷ Two studies using single versus multiple techniques reported an evaluation duration of greater than 30 days and showed mixed results,¹³⁷¹⁴¹ One study with evaluation duration greater than 30 days lacked a control group. Five studies with evaluation duration of greater than 30 days reporting on eight objectives did not meet objectives and thus were unable to identify a difference when using single versus multiple techniques.^707496130137

Short-term and long-term effects of the amount of exposure on practice behavior.A total of 37 studies reporting on 41 objectives evaluated the impact of single exposure to the CME activity. Two studies met objectives using single exposure, but did not report evaluation duration.³⁶⁶⁸ One study with evaluation duration of 30 days or less met objectives using a single exposure to the CME activity.⁹⁴ Sixteen studies with evaluation duration greater than 30 days met 18 objectives using single exposure to the CME activity.^{4445485370778298100102}–^{104106107111114} This suggests that single CME exposure may have a positive short- and long-term effect on practice behavior objectives. Two studies with evaluation duration of 30 days or less did not meet objectives using single exposure to the CME activity.¹⁸¹³⁸ Only six studies with evaluation duration greater than 30 days did not meet objectives using single exposure to the CME activity.^{14167074128129} Five studies with evaluation duration greater than 30 days lacked a control group,²⁴²⁵⁶⁴–⁶⁶ although one of the five did not meet objectives for either intervention group.²⁴ Six studies with evaluation duration greater than 30 days showed mixed results.^207188140–¹⁴²

A total of 55 studies (72 objectives) evaluated the impact of multiple exposures to the CME activity. Five studies met objectives using multiple exposures to the CME activity, but did not report evaluation duration.^1532356993 Thirty studies with evaluation duration greater than 30 days met 36 objectives using multiple exposures to the CME activity.^{19464952627275767881959799101105108}–^110112113115–^121123125148 One study using multiple exposures to the CME activity did not report evaluation duration and lacked a control group.¹⁴⁷ Five studies of evaluation duration greater than 30 days lacked a control group²⁴²⁵⁶⁴–⁶⁶ whereas two studies showed mixed results.⁷³¹³⁷ No meaningful conclusions could be drawn from these studies. Two studies did not meet objectives using multiple exposures to the CME activity, and did not report evaluation duration.¹³⁵⁷ Sixteen studies with evaluation duration greater than 30 days did not meet 21 objectives using multiple exposures to the CME activity.^{43567399101119126127130132}–¹³⁷¹⁵⁰ Overall, we conclude that multiple exposure volume to the CME activity may have positive short- and long-term effects on practice behaviors.

A total of eight studies reporting on 17 objectives did a head-to-head comparison between single and multiple exposures.^{424759798396122144} Whereas four of these studies indicated that multiple exposures may be better than single exposure, six showed mixed or negative results, thus not allowing us to draw any strong conclusions. One study met objectives using single versus multiple exposures to the CME activity, but did not report evaluation duration.⁷⁹ Three studies with evaluation duration greater than 30 days met objectives using single versus multiple exposures to the CME activity.^4796122

One study using single versus multiple exposures to CME activity did not report evaluation duration and had no control group.⁵⁹ Two studies using single versus multiple exposures to CME activity reported evaluation duration greater than 30 days but did not have a control group.⁸³¹⁴⁴ No meaningful conclusions could be drawn from these studies. One study using single versus multiple exposures to CME activity reported evaluation duration greater than 30 days and showed mixed results.⁴²

Two studies with evaluation duration greater than 30 days did not meet objectives using single versus multiple exposures to the CME activity.⁹⁶¹²²

One study did not meet objectives and did not report exposure to CME activity or evaluation duration.¹³⁹

Two studies met objectives using other exposures to the CME activity, but did not report evaluation duration.⁸⁷⁹² One study with evaluation duration greater than 30 days met objectives using other exposures to the CME activity.¹²⁴ One study using other exposure to CME activity did not report evaluation duration and lacked a control group.¹⁴⁶

Clinical Outcomes

Long-term effects of CME on clinical outcomes.Thirty-three studies, reporting on 42 clinical outcomes, measured the long-term effect of CME, i.e., more than 30 days following the educational intervention.^{22434755566572}–^{75777881849599109111}–^{115117120131132135137152}–¹⁵⁶ Fourteen of these studies were successful in achieving the desired effect of the CME intervention on clinical outcomes.^{22434774788495109111115152}–¹⁵⁵ One study showed mixed results, impacting frequency of office visits but not emergency room visits or hospitalizations.¹²⁰ In the remaining 23 studies, either no effect of CME was observed, or the effect was uncertain due to ambiguous results or problems in study design. Of the 14 studies that did show a long-term effect, six reported on direct measures of health status of the target patient population. These outcomes were arthritis pain and disability,⁹⁵ depression,²²¹⁵² general health and function,¹⁰⁹ emotional distress,⁸⁴ and lost work due to back pain.¹⁵³ Eight studies reported on health-related behaviors or attitudes. These outcomes were: percent of patients taking medication,¹⁵² patient adherence with antibiotics,⁴⁷ patient satisfaction with care,⁷⁴ frequency of physician visits,¹¹¹ hospitalizations,⁷⁸ hospital length of stay,¹⁵⁵ and smoking cessation rates.¹¹⁵¹⁵⁴ One study reported a mixed outcome, “quality of practice,” which combined direct measures of patient health status, such as whether the blood pressure was below 130/80 mm Hg, with behavioral or physician-related outcomes, such as whether the physician had recorded a family history of diabetes.⁴³

Short-term effects of CME media methods on clinical outcomes. Only one study was available to assess the relative effectiveness of different types of media (live, print, internet, or multiple) on short-term clinical outcomes (less than 30 days). In this study, a print intervention improved adherence with beta-blocker use.¹⁸ Five studies did not report on duration of clinical outcome.^{32 33 146 151 157} Thus, no conclusions could be drawn about the differential effectiveness of CME media in the short term.

Long-term effects of CME media methods on clinical outcomes. Of the studies that had information about the effectiveness of different single media forms of CME on long-term clinical outcomes, five used a live CME intervention;^{7284152154155} four of these five achieved the stated goal of the study. Two studies used print media and neither achieved its objective.⁵⁶¹⁵⁷ Another used Internet-based CME and did achieve its objective.⁴³ Most of the studies, however, used multiple CME media. Twenty-two studies used multiple media CME in comparison to a control.^{5565737375757778819599109112113117120120131131132135156} In four of these, the study achieved its stated aim. Seven studies compared multiple media CME to single media CME.^{4774111114115137153} Six of these achieved the stated aim; each found multiple media CME to be more effective than single media in improving clinical outcomes.

Short-term effects of CME educational techniques on clinical outcomes. A total of 15 different educational techniques were identified in the studies that reported clinical outcomes: readings, conference calls, academic detailing, discussion groups, lectures, point of care CME, feedback, physician visits, case-based learning, role-play, standardized patients, demonstrations, clinical experiences, simulation, and problem-based learning. Only one CME technique was evaluated singly in comparison to control for short-term clinical outcomes: provision of educational readings was associated with increased use of beta-blockers.¹⁸

Long-term effects of CME educational techniques on clinical outcomes. Only one study evaluated the effect of a single CME technique in comparison to control for long-term clinical outcomes. In this study, the use of conference calls was associated with improvements in depression.²² With only one such study, no conclusions could be drawn regarding the comparative effectiveness of single CME techniques on long-term clinical outcomes. Most of the studies evaluated used multiple CME techniques. Thirty-eight studies reported on the use of multiple simultaneous CME techniques in comparison to control. Twelve of these reported that the desired clinical outcome of the CME intervention was achieved.^{434774788495109111115152153155} Two studies yielded mixed results, i.e., some of the outcomes showed improvement while others did not.¹²⁰¹⁵¹ No individual CME techniques were common to the studies that did or did not achieve their stated objective; thus, one cannot draw any conclusions regarding the differential effectiveness of specific educational techniques. Five studies compared single to multiple CME interventions.^{4774115137152} In three of the five studies, the use of multiple simultaneous CME techniques was superior to the use of a single CME technique (readings).

Short-term effects of the amount of exposure on clinical outcomes. Only one study assessed the short-term effect of CME on clinical outcomes (less than 30 days). In this study, a one-time print intervention improved adherence with beta-blocker use.¹⁸ Five studies did not report on duration of clinical outcome,^{3233146151157} and all other studies reported long-term outcomes. Thus, no conclusions could be drawn about the differential effectiveness of amount of CME exposure (one-time vs. multiple exposures) on short term outcomes.

Long-term effects of the amount of exposure on clinical outcomes. Seven studies evaluated the long-term effect of a single CME exposure on clinical outcomes.^{657477111114152154} Four of these studies reported that the CME objective had been met.^74111152154 In one study, the objective was not met.¹¹⁴ Of the remaining two studies, one lacked a well-defined control group⁶⁵ and one yielded unclear results.⁷⁷ Most studies employed multiple CME exposures. In 24 studies, the multiple CME exposures were compared to a control with no CME.^{22435556727375757881849599109112113115117120131132135137156} In seven of the studies, the objectives were met.^{2243788495109115} In 16 studies, the objectives were not met or it was unclear if they were met.^{5556727375758199112113117131132135137156} One study produced mixed results, as described above.¹²⁰ In one study, a single printed CME intervention and a combination of printed material plus tutorial were compared with control.⁴⁷ While the CME intervention was deemed successful in changing clinical outcomes, the use of multiple CME exposures (tutorial plus reading) outperformed a single CME exposure (reading alone) in only 2 of 5 outcomes studied. In summary, both one-time and multiple exposure CME interventions have produced changes in clinical outcomes in about half of the studies, but it is unclear whether multiple exposure CME produces better results than one-time CME.

Characteristics of the Systematic Reviews

Types of simulation studied. A wide variety of simulation-based methods were identified in these reviews. Two reviews²⁶³⁰ included studies that had evaluated virtual reality, two reviews³¹¹⁵⁹ included studies with full simulation, three reviews^28158160 included studies with standardized patients or role play, five reviews²⁹–^31158159 included studies with partial task simulation, and six reviews²⁸–^31158159 included studies with computer simulation. All reviews had studies that compared simulation-based training with another type of simulation-based training, other educational intervention, standard training, no education, or no training.

Study populations and study designs. Three reviews^26158160 restricted inclusion criteria to studies that had enrolled only medical students or physicians-in-training. Other reviews also included studies that enrolled, in addition to medical students and physicians-in-training, fully-trained physicians, nurses, allied health professionals, and non-medical personnel. One review³¹ included only randomized controlled trials, while other reviews included non-randomized controlled trials and prospective trials in addition to randomized controlled trials.

Effectiveness of Simulation in Teaching Procedural Skills

Table 5

Quantity and direction of evidence for effectiveness (more...)

Table 5

Quantity and direction of evidence for effectiveness of simulation

Learning objectives	Number of reviews which addressed learning objective	Number of studies which addressed effectiveness of simulation as an educational method	Direction of evidence
Psychomotor Skills	62629–31158159	63	Favors simulation
Communication Skills	2158160	14	Favors simulation
Cognitive Skills	228159	37	Mixed results

Three reviews evaluated the effectiveness of virtual reality in teaching surgical skills. In virtual reality, the surgical field is represented in three dimensions which may help in learning more accurate surgical planning and procedures. One meta-analysis²⁶ found that training in a virtual reality environment significantly decreases the total amount of time required for task completion. There was also a trend toward a decreased error rate which did not reach a statistically significant level. Another systematic review²⁹ found four studies that evaluated the role of virtual reality in training surgical techniques. Two studies in this review found improvement in surgical skills after training with a virtual reality simulator while two other studies found no significant improvement after training with virtual reality simulators. A third systematic review³¹ found that trainees trained on computer simulation perform better than those who received no training, however, studies found an inconsistent benefit of computer simulation if simulation-trained students were compared with those who received standard training. This systematic review found only one study in which computer simulation was found to be superior to a physical training model. On the other hand, this review found that physical or model simulation may be superior to no training and standard training as instructions from mentors or manuals.

Video simulation was studied in one review.³¹ Video simulation was not superior to standard training or no training, and there was insufficient evidence to support the superiority of computer simulation to video simulation.

One review³⁰ evaluated the effectiveness of simulators for training in gastrointestinal endoscopy and concluded that flexible sigmoidoscopy can be applied for clinical training of residents and fellows for better patient comfort only. However, this review did not find enough evidence to support the use of simulators for clinical training in gastrointestinal endoscopy to improve clinical outcomes.

Another review¹⁵⁹ found one study with a cross-over design that reported a better post-test score (on a 22-object written test) of the anesthesia residents who were trained on a simulator as compared to those who were not trained.

Effectiveness of Simulation in Teaching Physical Examination

Two systematic reviews evaluated the effectiveness of simulation in teaching physical examination. One review¹⁵⁸ found that use of standardized patients to teach breast examination to medical students was associated with better performance in a clinical skills examination. This review further found that use of standardized patients or breast examination models was associated with improved ability of the students to detect breast lumps. The second review¹⁵⁹ found that use of a patient simulator was associated with improved practical skills as measured on a post-test examination.

Effectiveness of Simulation in Teaching Communication Skills

Two reviews evaluated the effectiveness of simulation in teaching communication skills. One review¹⁵⁸ found that when students were taught communication skills by patients with cancer, including training for giving bad news, students were more likely to respond empathetically to patients and better able to communicate bad news than students who were not taught communication skills by patients with cancer. This review further found that role-playing may be important in teaching oncology-specific communication skills as well as in communicating bad news. The second review¹⁶⁰ found that use of standardized patients and role-play was effective in teaching medical students smoking cessation counseling skills. This review further found that the use of standardized patients in teaching tobacco cessation skills to medical students was associated with increased confidence of students in their smoking cessation counseling skills.

Effectiveness of Simulation in Knowledge Acquisition

We found two reviews²⁸¹⁵⁹ that addressed the effectiveness of simulation in knowledge acquisition. Hmelo²⁸ found that computer assisted models are effective in teaching pathophysiologic principles to medical students. The pooled effect size, which measures the combined magnitude of the effect of intervention across 33 studies, was 0.63 for use of computer assisted learning (in favor of simulation). One study included in Ravert¹⁵⁹ looked at computer-based trauma simulation to teach trauma management; individual-study effect sizes ranged from -0.04 to 0.35 (did not favor simulation or were neutral). This review¹⁵⁹ did not report on the pooled effect size.

Features of High-Fidelity Simulators for Effective Learning

One review²⁷ systematically evaluated the features of high-fidelity simulators essential for effective learning. This review found that the following features of a high-fidelity simulator were important for effective learning: should provide feedback during learning experience, allow repetitive practice, can integrate in overall curriculum, has increasing levels of difficulty, is adaptable to multiple learning strategies, can allow clinical variation in a simulated environment, provides controlled environment to make and detect mistakes without consequences, provides individualized and standardized learning, defines outcomes clearly, and must have face validity-realism of simulator.

Summary

Overall the direction of evidence points to the effectiveness of simulation training, especially in psychomotor skills (e.g., procedures or physical examination techniques) and communication skills, but the strength of the evidence was considered low, due to the small number of appropriate studies, the scarcity of quantitative data, and other limitations. Several factors may be responsible for the inadequate quality of evidence in support of this method. In our view the most important factor is the lack of widely-accepted and standardized methods to quantify the competency in procedural or communication skills. In addition, the high cost of simulation methods and difficulty in introducing clinical realism in a simulated environment are other factors that may be responsible for inadequate quality of evidence in this field.

Audience Characteristics

Six studies examined the effect of years in practice on the educational intervention.^{3353798092140} Beaulieu, et al. suggested that physicians with less than 11 years of experience ordered fewer unnecessary screening tests than those with more experience, however, these results were not stratified by the educational technique actually employed by individual physicians.¹⁴⁰ Two other studies suggested that physicians with greater experience who underwent educational interventions had improvements in attitudes³³ and self-reported practice behavior.⁵³ However, none of the studies examined revealed a relationship between years in practice and acquiring knowledge, acquiring skills, or changing practice outcomes. Similarly, age had no influence on the outcomes of educational interventions in the 6 studies that reported on the effects of age.^{397980101105161}

Of the five studies^39537980161 that analyzed the effect of gender, only one showed a significant association. Leopold, et al. showed that women improved more than men in confidence with an objective performance of knee joint injections after an educational intervention consisting of printed material, hands-on instruction, or video instruction.⁸⁰ Only one paper described the influence of race on the effectiveness of the educational intervention.⁷⁹ Grady, et al. suggested that non-whites who underwent a presentation on mammography screening followed by cue enhancement (i.e., chart stickers and clinic posters) improved their screening rates more than whites.⁷⁹ This study also suggested that the intervention had a greater effect on solo practitioners compared to those in other practice settings. Three studies that examined the influence of board certification on educational outcomes primarily focused on internists and family practitioners and failed to show an association between certification and the desired outcome.^5379105

Conclusions and Limitations. We cannot reach definitive conclusions regarding the influence of audience characteristics on the effectiveness of specific educational techniques due to the heterogeneity of the educational interventions and characteristics examined. Furthermore, there are very limited data regarding any specific characteristic and the overall quality of the existing data on these questions are suboptimal.

External Factors

The literature is limited about the role of external factors, by themselves or in combination with other factors, in reinforcing the effects of CME on changing behavior. Very few studies explicitly stated that such factors were examined independently or collected data regarding these factors. Only one study rigorously examined external factors as a primary outcome.⁷⁹ Grady et al. studied whether token monetary rewards in addition to an educational intervention and chart cues increased the rate of mammography referral. While chart cues were effective in increasing mammogram referral and completion rates, the addition of a token monetary incentive provided no added benefit.

The offering of CME credit would not intuitively appear to be an external motivating factor for behavior change. Yet, two studies specifically examined the role of offering CME credit for this purpose, so it was included in our analysis. Both of these studies looked at results in association with earning CME credit for the educational activity.¹⁰⁶¹⁵⁶ Chassin and colleagues examined in a subgroup analysis the effect of offering CME credit for attendance at educational programs designed to decrease inappropriate x-ray pelvimetry rates in 64 study hospitals.¹⁶² They found no significant difference among intervention participants with respect to the offering of CME credit; both groups had a comparable decrease in pelvimetry use. Messina et al. found a potential association of offering CME credit for a physician educational program with an increase in the use of screening mammography in women who had never undergone mammography, but it did not reach statistical significance. The trend did not hold true for previous mammogram users.¹⁵⁶ These findings may be because CME credit may be an inducement to attend a CME activity but may not be sufficient to engage the participant in active learning.

Some CME courses utilize the signature of a “commitment-to-change” statement as an external motivating factor to improve clinical outcomes. Two studies examined the effectiveness of such a practice.¹⁶¹¹⁶³ Mazmanian and colleagues randomized 110 physicians to signature versus non-signature groups. While they found that those expressing an intent to change were more likely to change practice behavior (as documented by self-report on a follow-up survey), the act of signing such a commitment-to-change statement had no effect.¹⁶¹ In a much smaller study of 16 physicians attending a geriatrics course, Pereles et al. found that the physicians who were asked to make written commitments for practice changes (n=7) made more changes than controls at both one and three months followup. The results are of unclear statistical significance given the small study numbers.¹⁶³

Conclusions and Limitations. There are several barriers to collecting data on external factors. First, it is methodologically difficult to offer incentives (such as CME credit or financial reward) in a controlled fashion. Second, most evaluation of external factors is based on self-report. Finally, small study sizes often preclude a valid analysis of external factors in subgroup analyses due to lack of adequate power. Consequently, it is difficult to draw conclusions regarding the effectiveness of external factors in enhancing CME effects on behavior.

Background

Valid and reliable evaluation tools are necessary to demonstrate the effectiveness of CME interventions. The validity of the evaluation method is “the degree to which the method truly measures what it is intended to measure.”¹¹ A valid evaluation method accurately measures achievement of the stated objective of the educational intervention, whether it involves knowledge, attitudes, skills, practice behaviors, or clinical outcomes. The reliability of the evaluation method is “the consistency or reproducibility of measurements.”¹¹ A reliable evaluation method allows educators to have confidence in their assessments of learning across multiple measurements. As one measurement expert emphasizes, “small amounts of unreliability may cause misclassification errors and large score differences on retesting.”¹⁶⁴

An evaluation method may be statistically reliable without being valid for the objective intended by the investigators. However, a method cannot be valid without being reasonably reliable. Thus, Downing argues that “reliability is a necessary but not sufficient condition for validity, and reliability is a major source of validity for all assessments.”¹⁶⁴

An educational study may employ a newly created evaluation method or one that previously has been shown to be valid and/or reliable in another study population. The creation of a new evaluation method consumes time and resources for pilot testing, cognitive testing, and psychometric analyses to determine the validity and reliable of the new method. However, a previously used method may not be a valid measure for a new educational intervention if it does not map appropriately to the stated objective. Also, the reliability of a method changes as it is applied in different populations and ideally should be re-measured each time.

Results

We found reports of the validity and/or reliability of at least one evaluation method in 46 out of 136 total articles (33.8 percent). Among these 46 articles, 11 reported on the validity and/or reliability of more than one method: eight studies described two methods;^{424348526072137144} two studies described three methods;⁵⁵⁸⁴ and one study described four methods.³² Thus, 61 evaluation methods were accompanied by validity or reliability data. For the results below, percentages are based on the total number of methods - rather than articles - since some articles reported on multiple methods.

Among these 61 evaluation methods, 30 (49.2 percent) were drawn from previous studies and 28 (45.9 percent) were created for the current studies. For 3 methods (4.9 percent), it was not clearly reported whether the method was newly created or previously used. For 22 of the 30 previously used methods, the authors reported that reliability had been assessed: 13 within the current study population, 8 within previous study populations, and 1 within current and previous study populations. However, only 14 methods were presented with specific statistical data to support this reliability. For 12 of the 30 newly created methods, the authors reported that pilot and/or cognitive testing was performed.

Knowledge or cognitive skills were evaluated by 15 methods (24.5 percent).

Attitudes were evaluated by seven methods (11.5 percent). Two methods focused exclusively on attitudes, while five methods evaluated a combination of attitudes and knowledge / cognitive skills.

Skills (communication, psychomotor, or procedural) were evaluated by 11 methods (18.0 percent). One method evaluated physical exam skills in an educational setting. A combination of skills (communication, psychomotor, or procedural) and practice behaviors were measured by 10 methods, using standardized patients to visit physicians at their practice setting or analyzing interactions with real patients.

Practice behaviors (without clinical outcomes) were evaluated by 20 methods (32.8 percent). Seven methods used self-report by physicians of their practice behaviors. Three methods used patients' report of their physicians' behaviors in their medical care. Ten used chart review of medical records and/or claims data.

Clinical outcomes (with or without practice behaviors) were evaluated by 8 methods (13.1 percent). Two studies used chart review. One used reports by patients or families of their attitudes. One study used patient satisfaction. Two studied patient reports of their own behavior — including medication adherence and participation in preventive screening — as the outcome. One study used patients' reports of preventative services provided by their physicians. Three used measures of the patient's health.

The following articles provide notable examples for reporting of the validity or reliability of educational outcome measures:

• Knowledge or cognitive skills: Fordis et al⁴² describes the development of a knowledge test for cholesterol management, including: content validation by experts; description of the test and response options; piloting and item number reduction; and high internal consistency reliability (Cronbach alpha = 0.79).

• Attitudes: Mann et al⁵² describes the development of a knowledge and attitudes test about cholesterol management, including: content validation by experts; need for >90 percent agreement on question inclusion and consistency; description of the test and response options; pilot testing with internal consistency testing in the pilot population (KR20 = 0.60); and test-retest reliability analysis using control group test scores.

• Skills (communication, psychomotor, or procedural): Roter et al⁸⁴ describes a coding system to rate physicians' proficiency in managing standardized patients' emotional distress, including: blinding of coders; dichotomous coding system; and internal consistency reliability (Cronbach alpha = 0.20–0.62 for shorter scales, 0.76–0.81 for longer scales, and 0.62 for overall score).

• Practice behaviors: Sibley et al¹³⁶ describes chart abstraction for quality of care, including: content validity of pre-determined criteria by experienced clinicians; rating system; training of blinded nurse-abstractors; and high interrater and intrarater reliability (kappa >0.8).

• Clinical outcomes: Roter et al⁸⁴ describes use of the General Health Questionnaire-28 to detect emotional distress in patients, including: citation of original source for this previously validated questionnaire; repeated reliability testing with the study population, yielding high internal consistency (Cronbach alpha = 0.90–0.92).

Table 7

Number of methods evaluated for each type of validity (more...)

Table 7

Number of methods evaluated for each type of validity or reliability, organized by outcome type

Type of Validity	Definition*	# of Methods	Knowledge or cognitive skills†	Attitudes†	Skills (communication or psychomotor)†	Practice Behaviors†	Clinical Outcomes†
Content	Degree to which an instrument accurately represents the skill or characteristic it is designed to measure, based on people's experience and available knowledge	16	15213236384042464851–545785142	351–53	0	253136	0
Concurrent criterion	Degree to which an instrument produces the same results as another accepted or proven instrument that measures the same variable	8	1142	182	125	616257282137144	247137
Predictive criterion	Degree to which a measure accurately predicts expected outcomes	1	0	0	0	178	0
Construct	Degree to which an instrument measures the theoretical construct it intends to measure	5	3364385	0	0	253104	0
Type of Reliability	Definition	# of Methods	Knowledge or cognitive skills†	Attitudes†	Skills (communication or psychomotor)†	Practice Behaviors†	Clinical Outcomes†
Internal consistency	How well items reflecting the same construct yield similar results	19	12364042454849525355636485	8454952535562–64	23284	6324549537284	355‡84
Inter-rater	Degree to which measurements are the same when obtained by different persons	16	185	0	632608488138140	133242768488108118132134136138140144	2132137
Intra-rater	Degree to which measurements are the same when repeated by the same person	2	0	0	0	248136	0
Equivalence	Degree to which alternate forms of the same measurement instrument produce the same results	4	3173654	0	0	153	0
Test-retest	Degree to which the same test produces the same results when repeated under the same conditions	5	352‡53	25253	0	32253134	0
Validity: not specifically reported		4	0	172	3324372	33272121	181
Reliability: not specifically reported		5	160	0	132	232121	181

*

Definitions were obtained from Reed, et al.¹¹

†

Methods referenced may target more than one type of outcome and are listed under each applicable column.

‡

Two assessment methods within the same article.

Validity was reported for 31 of 61 evaluation methods (50.8 percent). Content validity was reported for 16 methods. The specific “experts” who reviewed the assessment were reported for 11 of these 15 methods. Concurrent criterion validity was reported for 8 methods. Predictive criterion validity was reported for only 1 method, which involved a comparison between physicians' reports of asthma management behaviors and patients' reports of physician behaviors. Construct validity was reported for 5 methods, usually through “known-group validity” (establishing construct validity by demonstrating better scores among those with higher levels of training or clinical experience). High statistical validity was only demonstrated for two methods.⁷²¹³⁷ Five methods were described as valid without specific details. Thus, the vast majority of CME studies offered no or limited psychometric data for the validity of their evaluation methods.

Reliability was reported for 43 of 61 evaluation methods (70.5 percent). Internal consistency reliability was reported for 19 methods, including 13 learner instruments, 1 observer instrument for audio-taped interactions, 1 standardized patient instrument, and 3 clinical patient instruments. Inter-rater reliability was reported for 16 methods, including 9 medical data abstractions and 6 skills assessments. Intra-rater reliability was assessed for 2 medical data abstraction studies. Equivalence reliability was reported for 4 methods, and test-retest reliability for 5 methods. Four methods were described as reliable without specific details. When reported, statistical tests yielded primarily modest evidence of reliability based on Cronbach-alpha, Kappa, or correlation statistics.

Conclusions

• Forty-six of 136 articles (34 percent) reported the validity and/or reliability of at least one evaluation method for assessing the effectiveness of CME.

• Thirty methods were drawn from previous studies, and 28 were created for the current studies. For 3 methods, the source was unclear. Authors did not commonly report reliability testing within the new study population for methods found to be reliable in other populations.

• The most common type of outcome evaluated by valid and/or reliable evaluation methods involved practice behaviors, for 20 out of 61 methods (34 percent).

• Of 61 evaluation methods with validity or reliability reported, 16 (26 percent) included descriptions of validity alone, 29 (48 percent) included descriptions of reliability alone, and ten (16 percent) had descriptions of both validity and reliability. For six methods (10 percent), the methods were described as valid and/or reliable, but the specific type of validity or reliability was not reported.

• Among these 61 methods, content validity was the most commonly reported type of validity (26 percent).

• Among these 61 methods, internal consistency (31 percent) and inter-rater (28 percent) were the most common types of reliability reported.

Although many studies of the effectiveness of CME have considered the validity or reliability of their evaluation methods, relatively few studies have used methods that have strong evidence of both construct and criterion validity. In addition, relatively few studies have used evaluation methods that have strong evidence of each of the specific types of reliability (internal consistency, inter-rater, intra-rater, equivalence, and test-retest). We therefore conclude that the overall strength of evidence on the effectiveness of CME is limited by weaknesses in the evaluation methods that have been used. To strengthen the evidence base on the effectiveness of CME, it will be necessary to commit additional resources to the development of valid and reliable evaluation methods. This may be quite challenging because of the limited resources that generally are available to clinician-educators. Where appropriate, educators may save time and resources by using previously validated and reliable methods, but they must demonstrate the validity of these methods for their specific educational outcomes and the reliability of these methods for their particular study populations.