A survey study of evidence-based medicine training in US and Canadian medical schools

Purpose: The authors conducted a survey examining (1) the current state of evidence-based medicine (EBM) curricula in US and Canadian medical schools and corresponding learning objectives, (2) medical educators’ and librarians’ participation in EBM training, and (3) barriers to EBM training. Methods: A survey instrument with thirty-four closed and open-ended questions was sent to curricular deans at US and Canadian medical schools. The survey sought information on enrollment and class size; EBM learning objectives, curricular activities, and assessment approaches by year of training; EBM faculty; EBM tools; barriers to implementing EBM curricula and possible ways to overcome them; and innovative approaches to EBM education. Both qualitative and quantitative methods were used for data analysis. Measurable learning objectives were categorized using Bloom’s taxonomy. Results: One hundred fifteen medical schools (77.2%) responded. Over half (53%) of the 900 reported learning objectives were measurable. Knowledge application was the predominant category from Bloom’s categories. Most schools integrated EBM into other curricular activities; activities and formal assessment decreased significantly with advanced training. EBM faculty consisted primarily of clinicians, followed by basic scientists and librarians. Various EBM tools were used, with PubMed and the Cochrane database most frequently cited. Lack of time in curricula was rated the most significant barrier. National agreement on required EBM competencies was an extremely helpful factor. Few schools shared innovative approaches. Conclusions: Schools need help in overcoming barriers related to EBM curriculum development, implementation, and assessment. Implications: Findings can provide a starting point for discussion to develop a standardized competency framework.


INTRODUCTION
Evidence-based medicine (EBM) has been defined as ''the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients [which involves] integrating individual clinical expertise with the best available external clinical evidence from systematic research'' [1].It affects both patient outcomes and trainees' practice-based learning and improvement [2,3].Its importance is reflected in an interdisciplinary panel convened by the Institute of Medicine (IOM) that recommended all health care trainees and professionals practice EBM [4].
Although US and Canadian medical school accreditation standards include the acquisition and practice of EBM skills [5], research-based literature on undergraduate medical education training in EBM is sparse.A 2002 study of EBM training in internal medicine clerkships found that 38.5% of the 109 responding US medical schools had a formal EBM curriculum during the third year and/or fourth year [2].EBM curricular materials and evaluation tools varied among these schools.That study also noted lack of time for student's EBM training in the school curriculum and inadequately trained faculty among the study respondents.The authors concluded that EBM had been integrated into the formal curriculum in relatively few clerkships.
A review by Maggio et al. of 2006 to 2011 publications characterizing worldwide EBM educational initiatives with medical students also suggested that educational setting, learner level, instructors in general, skills covered, and teaching methods varied greatly across educational interventions [6].Seven of the twenty articles identified by the authors were from the United States; and three articles focused on preclinical settings, another three on clinical settings, and one on both preclinical and clinical settings.In 2006-2007, Meats et al. surveyed UK medical schools to determine the extent to which EBM training played a role in the overall curriculum.Considerable variation was found in curricular methods and content among the 63% of responding schools.The survey findings also showed few opportunities for students to practice or assess their EBM skills, pressing curriculum time constraints, few trained A supplemental appendix is available with the online version of this journal.
faculty, and inadequate teaching materials.The authors proposed that medical schools integrate EBM training into clinical rotations, train faculty, and develop high-quality teaching resources.They also stressed the need to establish a national medical school EBM curriculum [7].
In A draft of the survey was shared with medical education researchers from the MESRE Section of the AAMC GEA for their feedback and then pilot-tested with medical educators before implementation.
The survey link was mailed in April 2011 to those curricular deans included in a list provided by the AAMC GEA, requesting recipients to assist with gathering baseline data to examine the current status of EBM training in US and Canadian medical schools.In total, 136 US and 13 Canadian medical school curricular deans were invited to complete the survey or, if appropriate, forward it to a designee familiar with the school's EBM curriculum.The survey was administered via SurveyMonkeyH and hosted by Tufts University School of Medicine.Responses were collected anonymously via SurveyMonkey, and biweekly follow-up messages were sent to encourage survey completion and a robust response rate.Data collection was closed 6 weeks after the survey link was initially mailed.

Analysis
Percentages and mean values were determined from the multiple-choice question responses.An inductive qualitative approach was employed to analyze the overall open-ended questions via open coding and identification of emergent themes [12].Open-ended comments to learning objective questions were analyzed only if they included measurable objectives, as described by the educational literature [13] (e.g., A survey study of evidence-based medicine training ''understand how to…'' or simply listing a content area or activity such as ''critical appraisal'' or ''EBM practice in the clinical setting'' were considered unmeasurable).The measurable learning objectives included in the analysis described specific and observable EBM knowledge and skills that students were expected to attain (e.g., ''Define evidence-based medicine (EBM) in one's own words,'' ''Analyze population health data using appropriate measures'').Each of the authors individually coded the measurable learning objectives, by year of training, matching each to one of Bloom's six cognitive domain categories in order to understand levels of thought processes associated with the learning objectives most commonly set in EBM programs [14].Authors together reviewed any coding variations to reach consensus.
Seventy percent of respondents were from public institutions, 29% from private institutions, and 1% from a federal institution.This distribution roughly matches the national breakdown of public/private/ federal medical schools, except that the number of private medical schools in the national breakdown is slightly higher [15].Almost two-thirds (62%) of these institutions had school enrollments between 401 and 800 students; 45% had a class size between 121 and 200 students; and 37% had 40-120 students.
Almost all (95%) respondents agreed with Sackett et al.'s definition [1] of EBM.This same percentage indicated that EBM was part of their medical schools' formal curricula.

Learning objectives
The 115 respondents reported a total of 900 EBM learning objectives covering the 4-year medical school curriculum.Of those, 482 (54%, as reported by 58 respondents) were measurable and thus coded.Figure 1 displays the distribution of Bloom's 6 cognitive domain categories [14] associated with the measurable learning objectives by year of medical school training.Interestingly, the number of measurable learning objectives decreases with year of training, with 233 for first-year and 22 for fourth-year students.Table 1 provides illustrative examples of the measurable learning objectives reported by respondents for each of the 6 cognitive domain categories.Table 2 reflects the most frequent curriculum content included in the measurable learning objectives, by year of training.

Curricular activities
Figure 2 reflects the frequency with which formal EBM training is taught in various educational settings by training year.Not surprisingly, lectures, small group, and hands-on settings were most frequent for first-and second-year students; the clinical setting was most common for third-year students; and ''none'' was most common for fourth-year students.Less frequently, the ''other'' category included the following: & first year: online exercises, computer-based selfstudy, student projects, courses in critical appraisal

Assessment approaches
Figure 3 displays how the responding schools formally assessed the extent to which the learning objectives were met across the four years of medical school.Written exams and global written faculty observation-that is, recorded general observations rather than recorded observation based on checklists or rubrics-were the approaches most frequently cited.Again, not surprisingly, the former were more frequently used to assess first-and second-year students, while the latter was most often employed for third-and fourth-year students.Although OSCEs and verbal exams were used far less frequently overall, they were used more often for third-year students than for the remaining three years.
The ''other'' assessment category cited included the following, by trainee year: & first-year students: small group and/or team-based learning activities & second-year students: individual student assignments & third-year students: student presentations at grand rounds, computerized assessment, patient case presentations, and journal club activities & fourth-year students: student presentations Faculty Figure 4 summarizes the faculty reported to be teaching EBM in each training year.Faculty categories included basic scientists, clinicians, librarians, allied health staff (e.g., nurses, pharmacists, and physical therapists), and ''other'' (e.g., statisticians and faculty with specific titles connected to EBM educational activities).Clinicians, by far, were the most frequent EBM faculty for all four training years.

Barriers to implementing an EBM curriculum
Over half of the respondents (59%) indicated encountering at least 1 barrier listed on the survey when implementing their schools' EBM curricula.Almost half (45%) rated ''lack of time in curriculum'' as a significant barrier; 41% rated ''students' perceived importance of this training versus traditional basic science or clinical courses/program'' and ''difficulty integrating EBM with clinical care'' as somewhat significant.Twenty-eight percent rated ''lack of EBM knowledgeable faculty'' as a significant barrier, and 27% rated ''lack of faculty interest'' as somewhat significant.One-fifth (20%) of respondents listed ''other'' barriers to implementing EBM curricula at their schools.Over half of these were related to faculty expectations and preparation for teaching EBM (e.g., faculty role modeling or inconsistent faculty expectations).Other barriers rated as at least somewhat significant included finding appropriate-level textbooks, finding agreement on the definition of EBM and the need for a formal EBM curriculum, having minimal contact time with students, and ''wrestling with the software.''Resource access and knowledgeable library staff were not perceived to be barriers to implementing the school's EBM curriculum, garnering a rating of ''not significant'' by 71% (''poor access to EBM-related e-resources'') and 72% (''lack of knowledgeable library staff in the EBM process'').Likewise, ''lack of evidence of value of using EBM'' was not perceived to be a barrier (54%).''National agreement on required EBM competencies'' was rated most frequently as extremely helpful in overcoming barriers (41%).One-third also rated ''revision of local curriculum to include time allocation for EBM'' as extremely helpful.Fifty-four percent rated ''evidence of improved patient outcomes from use of EBM'' as either helpful or extremely helpful.Forty percent rated ''easy faculty access to enhancing faculty knowledge of EBM'' as helpful in overcoming barriers.In addition, 32% and 27% rated ''easy librarians' access to enhancing librarians knowledge of EBM'' and ''closer working relationship with librarians'' as somewhat helpful, respectively.Of the 11 respondents who freely commented on ''other'' factors that might help their schools overcome these barriers, 45% mentioned the need for faculty development on EBM skills.Other single suggestions included the need for a mandate from the administration, creation of accountable care organizations and new reimbursement models for clinicians, and integration of EBM training vertically into the curriculum.

Innovative approaches to teaching EBM
Only ten respondents shared innovative approaches that their schools were implementing in teaching EBM.Five indicated that EBM was fully integrated into each clerkship rotation and linked to daily specific patient-care tasks.Three described faculty development initiatives with residents to expand the number of available faculty and thus improve student exposure to EBM practice on the wards.One reported that senior students prepared EBM teaching exercises that were used by junior students, and another mentioned implementing an EBM OSCE station at the end of medical school.

DISCUSSION
Respondents' familiarity with their schools' curricula suggested that the targeted audience had been reached.In addition, almost all respondents indicated  A survey study of evidence-based medicine training that their schools had a formal EBM curriculum and endorsed the EBM definition embraced in this survey, a definition that complied with accreditation standards, as well as a universal understanding of what EBM entails across training programs.
Findings indicated that schools perceived a lack of time in the curriculum as a significant barrier to implementing their EBM curricula, as has been noted in previous studies on undergraduate medical education EBM curricula [2,7].For example, Aiyer et al. found a ''lack of [EBM] curriculum'' in internal medicine clerkships in 34% of 109 US responding schools [2].In addition, findings also indicated that having some national agreement on required EBM competencies would help medical schools immensely with implementation in the curriculum.They might benefit from a common developmental EBM competency framework that would further inform and guide their EBM curricular efforts.Meats et al., who conducted a similar study in the United Kingdom, also suggested such a national framework for EBM education [7].Findings from this survey study, specifically the outcome data on EBM learning objectives reported in Table 1, could provide a starting point for a national discussion of required EBM developmental competencies in US and Canadian medical education organizations and accreditation bodies.Other barriers mentioned by respondents, such as preparing faculty to teach EBM and finding appropriate-level textbooks, could be addressed by exposing faculty to existing EBM curricular resources published in medical education online repositories, such as MedEdPortal [16,17].
Among those measurable learning objectives that survey respondents reported, knowledge application was the predominant cognitive category across the four years, consistent with the definition of EBM and its use to improve patient-care practice.The content associated with those objectives focused on comprehension of core EBM concepts in the first two years; in the third year, application of critical appraisal, database searches, and library resources to patient care, and synthesis (formulation) of answerable clinical questions; and in the fourth year, application of an EBM approach to patient care, alluding to management, informed decision making, and synthesis of patient-care plans.This content distribution across the four years already illustrated a developmental progression from pre-clerkship to clerkship education; however, due to a disproportionate emphasis on respondents' application of EBM concepts, bolstering other cognitive domains might be necessary.For example, preclinical students need the opportunity to define (knowledge domain) and describe (comprehension domain) EBM concepts before being asked to apply that knowledge.In the later years, it would be reasonable to increase the analysis, synthesis, and evaluation of EBM concepts.However, given that half of the educational objectives reported were not measurable, faculty development opportunities appear warranted for most EBM faculty to ensure creation of sound learning objectives.
Findings also suggested that formal EBM curricular activities were predominately integrated into other curricular activities, allowing students to acquire EBM knowledge and skills in a broader training and practice context.Recent AAMC data on the percentages of schools teaching EBM as an independentversus an integrated-course also showed that most schools incorporated EBM training into other curricular activities [17].Still, the question that developers of both preclinical and clinical curricula face is whether all concepts should be taught before they are applied.
The findings also indicated that EBM curricular activities more commonly occurred in the first and second years of medical school, rather than during the clinical years.Activities dispersed within lectures and/or didactic sessions during years 3 and 4 decreased by 54%, interspersed activities in small group sessions decreased by 40%, and hands-on classroom or clinical-skills center practice decreased by almost 30%.These findings suggested that the fourth-year curriculum was generally not as standardized as that of the first three years of medical school.The decrease in EBM curricular activities in the clinical years might also be explained by a lack of time, particularly at clinical settings, which was reported in previous studies [2] and perceived as a significant barrier in this survey study.The challenging task of standardizing and monitoring clinical curricula at different clinical teaching sites might also explain this decrease in EBM formal curriculum activities at some schools where students' clinical rotations take place at different sites.The emphasis of EBM training in the preclinical years without reinforcement of this training in the clinical years could put at risk students' transfer of EBM knowledge and skills to actual clinical practice.
Similarly, the study implied that formal assessment approaches tend to fade as training advances through the clinical years: most schools acknowledged not assessing their students' EBM skills in their end-ofmedical school OSCE.Given that written exams and global observational assessment were the assessment approaches more frequently cited by respondents, OSCEs and/or simulation exercises appear to be needed to accurately assess knowledge application.Expanding assessment methods across the curriculum would also benefit students and faculty in monitoring the vertical and horizontal integration of an EBM curriculum.
As other scholars previously noted [6], findings also revealed that clinicians were most often identified as EBM instructors across the four years at the surveyed schools, with basic scientists and librarians also fairly frequently providing instruction, particularly in the first two years of medical school.Increasing the amount of interprofessional education in EBM might relieve clinicians from shouldering the bulk of the EBM curriculum.Perhaps teaming basic scientists, librarians, allied health staff, and other professionals (such as statisticians) with clinicians across the four years would emphasize for students the reality that EBM is an interprofessional activity that can benefit Blanco et al.
from the participation of a wide range of experts (not just clinicians).In fact, this multidisciplinary approach might increase the sustainability of the EBM curriculum not only across, but beyond the medical school curriculum.As Maggio et al. suggest [6], integrating interprofessional education, which is consistent with the current model of health care delivery, into EBM curricula is warranted.
Survey results also indicate the use of a wide variety of EBM tools, with PubMed Clinical Queries and the Cochrane database most frequently cited.Given the number of clinical tools synthesizing and appraising evidence that have been introduced since the early stages of the EBM movement, the extent of this list is not surprising.It is also not surprising that the most robustly used tool (PubMed Clinical Queries) is freely available and offered by numerous health sciences libraries via easily accessible tutorials.This survey did not include PubMed as an option because it was not viewed primarily as an EBM tool.Some respondents might have selected PubMed Clinical Queries as a subset of PubMed, if in fact they as users did essentially view PubMed as an EBM tool.Therefore, this finding should be interpreted with caution.
Of the few schools commenting on innovative approaches to teaching EBM, full integration of EBM into each clerkship rotation via specific patient-care tasks or into an end-of-medical school OSCE was highlighted, as well as having trainees (e.g., residents or senior medical students) train junior medical students.These reported innovations address some of the barriers that the survey revealed, such as the lack of faculty development and the lack of a variety of assessment approaches.More innovative approaches to teaching EBM should also be fostered, as Maggio et al. also suggested [6].

Limitations
An examination of and/or comparisons among the four-year curricula of the individual schools could not be conducted because no identifiers were collected in order to preserve data de-identification.In addition, although invited deans were requested to forward the survey to a designee familiar with the school's EBM curriculum and respondents indicated being familiar with their school curricula, the extent to which those who responded were familiar with the details of the EBM curricula in all four years is unknown.Survey fatigue could also explain the finding that EBM training appeared to fade in the third and fourth years, and could explain the few innovations that were reported, because these questions were posed toward the end of an admittedly long survey.However, the lack of EBM curricula in clinical years and the need for integrating medical education trends into EBM curricular initiatives were also reported in previous studies.Nonetheless, the authors believe overall that the findings provide a credible snapshot of a sample of US and Canadian medical schools' EBM curricular practices and challenges.

CONCLUSIONS
Medical educators, in collaboration with librarians, need to examine how schools might overcome barriers in developing, implementing, and assessing an EBM curriculum.Furthermore, clinicians might partner with librarians and other health professionals to standardize a definition of and training in EBM.Senior academic leaders should introduce clear, quantifiable instructional time for EBM within and across curricula.Finally, national professional groups-such as the AAMC-GEA, the Society of Directors in Medical Education Research (SDRME), and AAHSL-might offer grant opportunities to promote inter-institutional collaborations in EBM education and increase rigorous program evaluation approaches to EBM learning outcomes.Findings of this survey study could also serve as a starting point for formulating a common developmental EBM competency framework.
& Rick B. Forsman, MLS, MPA, FMLA, founder of the AAHSL-MESRE Task Force, provided administrative support during the study design and implementation.He also participated in the data collection and preliminary analysis.
Maggio et al. called for authors to provide more detailed descriptions of their interventions and employ more rigorous research methods.They also recommended that educators consider trends in medical education, such as interprofessional education and online training, when designing EBM curricular interventions.

Figure 1
Figure 1Evidence-based medicine (EBM) knowledge-based measurable learning objectives associated with the educational activities of first through fourth years of responding medical schools

Figure 2
Figure 2 Educational activities providing formal EBM training across the four years

Figure 3
Figure 3 EBM formal assessment approaches across the four years of medical school

4
Faculty teaching EBM by year of medical school training * Resp.5respondents from different medical schools.

&
Jacqueline D. Doyle, MS, AHIP, FMLA, Arizona Health Science Library, was a member of the original AAHSL-MESRE Task Force and participated in the study design and data collection.& Louise S. Miller, MA, CAE, executive director, AAHSL, provided administrative support during the data analysis and manuscript writing.& Nicole Love, BA, curriculum manager, Office of Educational Affairs, Tufts University School of Medicine, provided administrative support during the study design and executed and monitored the data collection process.& Ann Maderer, EdD (candidate), curriculum and administrative director, Office of Educational Affairs, Tufts University School of Medicine, contributed to the graphic data representation design and manuscript reviews.& Members of the Steering Committee of the MESRE Section of the AAMC GEA reviewed and provided feedback on the survey instrument and the preliminary full report of findings.& Patricia S. O'Sullivan, EdD, director, Research and Development in Medical Education, School of Medicine, University of California-San Francisco, reviewed and provided feedback on the preliminary full report of the findings.& Bridget C. O'Brien, PhD, assistant professor, Department of Medicine and Office of Research and Development in Medical Education, University of California-San Francisco, and director of evaluation, SFVA Center of Excellence for Primary Care, re-A survey study of evidence-based medicine training viewed and provided feedback on the draft manuscript.

Table 1
Illustrative examples of measurable learning objectives Define evidence-based medicine (EBM) in one's own words.N Define criteria for inferring causality from statistical associations including the Surgeon General and Hill Criteria.N List the 4 core prerequisites for an effective screening test.Comprehension N Describe the steps in the EBM process.N Identify the 5 common types of clinical questions: diagnosis, etiology/harm, prevention, prognosis, and therapy.N Explain the difference between statistical significance and clinical significance, type I and type II error.Application N Locate a Medical Subject Headings (MeSH) term in the context of broader or narrower MeSH terms.N Utilize the 7 steps in the modified systemic approach to answering drug information questions.N Interpret research findings reported in scholarly journals.Analysis N Differentiate between disease and patient-oriented evidence.N Analyze population health data using appropriate measures.N Discriminate among types of information sources in terms of their currency, format (for example, a review vs. an original article), authority, relevance, and availability.Synthesis N Ask an answerable question, specific to the patient, intervention, comparison, and outcome (PICO).N Craft foreground questions for both diagnosis and treatment using the PICO format.N Construct well-defined clinical questions from case scenarios and patients, designed to improve general knowledge about a topic, to help make decisions regarding the assessment of risk and the use of diagnostic tests.Evaluation N Appraise the evidence for validity and relevance to the PICO.N Evaluate a website according to the major criteria established by the ''Guidelines for Medical and Health Information Sites on the Internet'' published in JAMA, March 22/29, 2000 (pp.1600-6).N Critically appraise retrieved evidence (Is this study well done?What are the results?)

Table 2
Most frequent curriculum content included in measurable learning objectives, by year of medical school training * Content described in a given year was repeated in the years that followed.A survey study of evidence-based medicine training tronic Library, STAT!Ref, USML Easy, Clinical Practical Guidelines [sic], Clinicaltrials.gov,Bandolier, PubMed, and Vividesk were reported in the ''Other'' category.