|
|
BMJ. 2004 October 30; 329(7473): 1017. doi: 10.1136/bmj.329.7473.1017. | PMCID: PMC524555 |
What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review Arri Coomarasamy, specialist registrar in obstetrics and gynaecology1 and Khalid S Khan, consultant obstetrician and gynaecologist1 1 Education Resource Centre, Birmingham Women's Hospital, Birmingham, B15 2TG Objective To evaluate the effects of standalone versus clinically integrated teaching in evidence based medicine on various outcomes in postgraduates. Design Systematic review of randomised and non-randomised controlled trials and before and after comparison studies. Data sources Medline, Embase, ERIC, Cochrane Library, DARE, HTA database, Best Evidence, BEME, and SCI. Study selection 23 studies: four randomised trials, seven non-randomised controlled studies, and 12 before and after comparison studies. 18 studies (including two randomised trials) evaluated a standalone teaching method, and five studies (including two randomised trials) evaluated a clinically integrated teaching method. Main outcome measures Knowledge, critical appraisal skills, attitudes, and behaviour. Results Standalone teaching improved knowledge but not skills, attitudes, or behaviour. Clinically integrated teaching improved knowledge, skills, attitudes, and behaviour. Conclusion Teaching of evidence based medicine should be moved from classrooms to clinical practice to achieve improvements in substantial outcomes. The knowledge and skills needed for critical appraisal of literature and practice of evidence based medicine (EBM) are often taught through standalone courses and workshops in classrooms away from clinical practice. An early (and now out of date) review showed that in these educational interventions, gains in knowledge were poorer among postgraduates than undergraduates. 1 Without reinforcement in subsequent practice, even the modest knowledge gains from such courses are likely to deteriorate over time. Postgraduate and continuing education received in this way is unlikely to lead to any meaningful changes in clinical care. In theory, teaching and learning that is integrated into routine practice should bring greater benefits. We examined the effects of postgraduate teaching in EBM and explored the effect of the teaching methods (whether standalone or integrated into clinical practice) on various outcomes. We searched Medline, Embase, ERIC, Cochrane controlled trials register (CCTR), Cochrane database of systematic reviews (CDSR), database of abstracts of reviews of effects (DARE), Health Technology Assessment database (HTA), Best Evidence, Best Evidence Medical Education (BEME), and Science Citation Index (SCI) using the following search terms and their word variants: “evidence”, “critical”, “appraisal” or “journal club” combined with “AND” to “teach$”, “learn$”, “instruct$”, or “education”. We also searched reference lists of known systematic reviews. 1-4 The final electronic search was conducted in April 2004. We included studies that evaluated the effects of postgraduate EBM or critical appraisal teaching compared with a control group or baseline before teaching, using a measure of participants' learning achievements or patients' health gains as outcomes. Learning achievement was assessed separately for knowledge, critical appraisal skills, attitudes, and behaviour. Knowledge relates to issues such as remembering materials as well as grasping the meaning—for example, defining and understanding the meaning of number needed to treat (NNT). If this knowledge can then be applied accurately to given problems this will be regarded as a gain in critical appraisal skills—for example, the ability to generate a number needed to treat when baseline risks and odds ratios are provided. Spontaneously acknowledging a need for the use of a certain piece of knowledge or skill in practice will be regarded as a change in attitude—for example, recognising without prompting the need for different NNTs for different clinical scenarios and intending to calculate the respective NNTs for different levels of risk. Finally, a change in behaviour occurs when one seeks the necessary information and applies the knowledge and skills to solve the issue in practice—for example, searching the literature, finding relevant baseline risks and odds ratios, and calculating necessary NNTs to guide clinical practice. We excluded studies on teaching of EBM in undergraduate education. We graded the quality of the evidence in these articles as either level 1 (randomised controlled trials) or level 2 (non-randomised studies that either had a comparison with a control group or a before and after comparison without a control group). We could not use meta-analysis because of the obvious heterogeneity in features, quality, and assessment tools in individual studies. We weighted our conclusions by quality of methods. The literature search identified 42 potentially useful citations. We examined the full manuscripts of all of these citations and identified 23 articles (including one yet to be published) relevant for inclusion in our review (tables 1 and 2). 5-26 Of the 19 articles that we rejected, 15 examined populations unsuitable for our review (for example, undergraduates or non-medical staff), two examined an unsuitable intervention (for example, the effect of dissemination of EBM guidelines rather than teaching of EBM), and two were reviews of primary studies already included in our review. Of the 23 included studies, four were randomised (level 1) trials (R Taylor, personal communication), 5,22,23 and 19 were non-randomised studies (level 2), comprising seven non-randomised controlled studies and 12 before and after comparison studies. Teaching methods included workshops, seminars, and journal clubs alone or in various combinations (tables 1 and 2). Eighteen studies (R Taylor, personal communication), 5-21 including two randomised trials (R Taylor, personal communication) 5 (level 1), evaluated a standalone teaching method ( table 1), while five studies, 22-26 including two randomised trials, 22,23 evaluated an integrated teaching method ( table 2). Integrated teaching focused on training in EBM components (such as question formulation, literature searching, and critical appraisal) in real time clinical ward rounds or basing the EBM teaching sessions on encounters with patients on the wards and in clinics. The outcomes reported were knowledge, skills, attitude, and behaviour. None of the studies assessed patients' health. | Table 1Primary studies of standalone teaching of critical appraisal skills and EBM in postgraduate trainees |
| Table 2Primary studies of clinical practice based (integrated) EBM teaching among postgraduate trainees |
Does knowledge improve? Of the 23 studies, 17 assessed knowledge (). The weight of evidence, including the evidence from the three randomised trials (R Taylor, personal communication) 5 that reported on this outcome, indicated an improvement in knowledge from both teaching methods. Do critical appraisal skills improve? Nine of the studies assessed critical appraisal skills (). The only randomised trial that reported this outcome in the standalone group did not find an improvement (R Taylor, personal communication). Of the six non-randomised studies that reported this outcome in the standalone group, three found an improvement. On the other hand, both the studies, including a randomised trial, 22 which reported skills as an outcome in the integrated teaching group found an improvement. Therefore, on balance, there is weak evidence that standalone courses improve appraisal skills and good evidence, including evidence from a randomised trial, 22 that the integrated approach leads to gains in appraisal skills. Do attitudes change? Six studies assessed change in attitudes, three each in both teaching groups (). In the standalone teaching group the three studies, including a randomised trial (R Taylor, personal communication), did not find a change in attitudes. In the integrated teaching group, however, all studies, including one randomised trial, 22 found an improvement in attitudes. Therefore there is compelling evidence that teaching integrated into clinical practice changes attitudes about the role of EBM or critical literature appraisal in medicine, while a standalone approach does not. Does behaviour change? Fourteen studies assessed the outcome of behavioural change after EBM or critical appraisal teaching, including four randomised trials, two in each teaching group (). The two randomised trials (R Taylor, personal communication) 5 in the standalone group found no change in behaviour, and both randomised trials in the integrated teaching group observed a change in behaviour. 22,23 These findings from the randomised evidence were found to be consistent with the findings of the non-randomised studies, with four of seven studies in the standalone group not showing a change in behaviour and all three non-randomised studies in the integrated teaching group showing benefit. The improvements noted in behaviour included changes in reading habits 24 and choice of information resources, 25 as well as substantial outcomes such as changes in management of patients 23,26 and guidelines. 26Do patients' health outcomes improve? None of the studies evaluated health outcomes. As the integrated teaching approach showed that it was possible to change behaviour, however, this holds the potential for improving health outcomes. The translation of changes in behaviour into complex outcomes such as better care of patients may not be a linear one, as improving care is likely to be affected by many factors, only one of which may be the practice of EBM. Moreover, such improvements in patients' outcomes are likely to occur over a long period of time and among many other changes, making them difficult to identify in studies of evaluation of teaching or practice of EBM and critical appraisal skills. To our knowledge, a comparison of the effects of standalone versus integrated teaching in critical appraisal skills and EBM has not been done before. In addition to not making the distinction between standalone and integrated courses, 2-4 several existing reviews have generally considered undergraduates and postgraduates together. There is empirical evidence, however, that the outcomes of teaching EBM markedly differ between undergraduates and postgraduates, with smaller gains in knowledge among the postgraduates. 1 Moreover, adult learning theory suggests that the determinants of learning in the two groups are different, with postgraduate learning tending to be driven by self motivation and relevance to clinical practice, whereas undergraduate learning is generally driven by external factors such as curriculum and examinations. 27 This suggests that effectiveness of educational interventions in teaching critical appraisal skills and EBM should be evaluated separately for postgraduate and continuing education, which we have done. Studies examining the effectiveness of educational interventions may suffer from various weaknesses. Even a randomised controlled study, which is generally regarded as the optimum method for settling questions of effectiveness, is not immune to many of these weaknesses. These weaknesses include difficulty with standardising the educational intervention(s), contamination between the two arms of a study, inability to blind the study participants and the teachers from the educational intervention(s) leading to selective cointervention, and finally difficulty with measuring outcomes due to the lack of valid and reliable assessment tools. Some of these factors make randomised trials unfeasible in educational settings, thus necessitating other designs such as non-randomised controlled and before and after studies. We included all three designs in our review. We have shown that while standalone teaching and integrated teaching are both effective in improving the knowledge base, it is clinically integrated teaching of EBM that is likely to bring about changes in skills, attitudes, and behaviour (). Changes in attitudes are likely to be important in bringing about sustained changes in behaviour, which may ultimately benefit care of patients. It is therefore important that teachers of critical literature appraisal and EBM consciously find ways of integrating and incorporating teaching of critical appraisal into routine clinical practice. Where resources and facilities are available, such teaching can form part of a “real time” ward round with the dual purposes of teaching EBM skills and attempting to improve care with best available evidence. 28,29 If the provisions for real time teaching are not available, then even traditional teaching settings, such as a journal club, 30 can be adapted to be based on real and current clinical problems, thus illustrating that the process is not merely an academic exercise but that it informs care. The purpose of EBM is to integrate best research evidence with clinical skills and patients' values and preferences. 31 Teaching EBM should not only equip practitioners with knowledge and skills but also foster their attitudes and encourage the practice of EBM. This is because the ultimate aim of improving care could not be achieved with changes in knowledge and skills alone—it would also require changes in attitudes and behaviour. Critical appraisal and EBM teaching that is integrated into clinical practice seems more effective in improving such substantial outcomes including behavioural changes. Teachers of critical appraisal and EBM should aim to bring teaching out of classrooms into the clinic, but this will require a greater effort. Future studies should focus not only on substantial outcomes such as behaviour and health outcomes but also on longer term outcomes as there is the potential for decay of learning outcomes over time. Summary points Critical literature appraisal and evidence based medicine (EBM) can be taught through standalone courses or through instructional methods that incorporate teaching into routine clinical care Several randomised and non-randomised studies have evaluated the effects of teaching EBM to postgraduates Both standalone courses and integrated teaching improve knowledge Improvements in skills, attitudes, and behaviour, however, come about when teaching is integrated into clinical practice; standalone courses bring about no change It is important to incorporate EBM teaching into clinical practice, but this would require a sustained effort well beyond standalone instruction |
1. Norman GR, Shannon SI. Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: a critical appraisal. CMAJ
1998;158: 177-81. [PubMed] 2. Parkes J, Hyde C, Deeks J, Milne R. Teaching critical appraisal skills in health care settings. Cochrane Database Syst Rev
2001;(3): CD001270. [PubMed] 3. Taylor R, Reeves B, Ewings P, Binns S, Keast J, Mears R. A systematic review of the effectiveness of critical appraisal skills training for clinicians. Med Educ
2000;34: 120-5. [PubMed] 4. Coomarasamy A, Taylor R, Khan KS. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Med Teach
2003;25: 77-81. [PubMed] 5. Linzer M, Brown JT, Frazier LM, DeLong ER, Siegel WC. Impact of a medical journal club on house-staff reading habits, knowledge, and critical appraisal skills. A randomized control trial. JAMA
1988;260: 2537-41. [PubMed] 6. Gehlbach SH, Bobula JA, Dickinson JC. Teaching residents to read the medical literature. J Med Educ
1980;55: 362-5. [PubMed] 7. Kitchens JM, Pfeifer MP. Teaching residents to read the medical literature: a controlled trial of a curriculum in critical appraisal/clinical epidemiology. J Gen Intern Med
1989;4: 384-7. [PubMed] 8. Green ML, Ellis PJ. Impact of an evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med
1997;12: 742-50. [PubMed] 9. Fu CHY, Hodges B, Regehr G, Goldboom DS, Garfinkel P. Is a journal club effective for teaching critical appraisal skills? A controlled trial with residents in psychiatry. Acad Psychiatry
1999;23: 205-9. 10. Bazarian JJ, Davis CO, Spillane LL, Blumstein H, Schneider SM. Teaching emergency medicine residents evidence-based critical appraisal skills: a controlled trial. Ann Emerg Med
1999;34: 148-54. [PubMed] 11. Smith CA, Ganschow PS, Reilly BM, Evans AT, McNutt RA, Osei A, et al. Teaching residents evidence-based medicine skills: a controlled trial of effectiveness and assessment of durability. J Gen Intern Med
2000;15: 710-5. [PubMed] 12. Ross R, Verdieck A. Introducing an evidence-based medicine curriculum into a family practice residency—is it effective? Acad Med
2003;78: 412-7. [PubMed] 13. Mulvihill MN. Faculty development and resident training in epidemiology and biostatistics. Mt Sinai J Med
1981;48: 350-2. [PubMed] 14. Seelig CB. Affecting residents' literature reading attitudes, behaviors, and knowledge through a journal club intervention. J Gen Intern Med
1991;6: 330-4. [PubMed] 15. Langkamp DL, Pascoe JM, Nelson DB. The effect of a medical journal club on residents' knowledge of clinical epidemiology and biostatistics. Fam Med
1992;24: 528-30. [PubMed] 16. Seelig CB. Changes over time in the knowledge acquisition practices of internists. South Med J
1993;86: 780-3. [PubMed] 17. Hillson SD, Schlossberg LA. Effects of journal club on critical review. Clin Res
1993;41: 559A. 18. Caudill TS, Johnson MMS, Rich EC. The effect of journal club on medicine resident interaction with original research literature. Clin Res
1993;41: 816A. 19. Ibbotson T, Grimshaw J, Grant A. Evaluation of a programme of workshops for promoting the teaching of critical appraisal skills. Med Educ
1998;32: 486-91. [PubMed] 20. Kellum JA, Rieker JP, Power M, Powner DJ. Teaching critical appraisal during critical care fellowship training: a foundation for evidence-based critical care medicine. Crit Care Med
2000;28: 3067-70. [PubMed] 21. Schoenfeld P, Cruess D, Peterson W. Effect of an evidence-based medicine seminar on participants' interpretations of clinical trials: a pilot study. Acad Med
2000;75: 1212-4. [PubMed] 22. Bradley DR, Rana GK, Martin PW, Schumacher RE. Real-time, evidence-based medicine instruction: a randomized controlled trial in a neonatal intensive care unit. J Med Libr Assoc
2002;90: 194-201. [PubMed] 23. McGinn T, Seltz M, Korenstein D. A method for real-time, evidence-based general medical attending rounds. Acad Med
2002;77: 1150-2. [PubMed] 24. Khan KS, Pakkal M, Brace V, Dwarakanath LS, Awonuga A. Postgraduate journal club as a means of promoting evidence-based obstetrics and gynaecology. J Obstet Gynaecol
1999;19: 231-4. [PubMed] 25. Grad R, Macaulay AC, Warner M. Teaching evidence-based medical care: description and evaluation. Fam Med
2001;33: 602-6. [PubMed] 26. Haines SJ, Nicholas JS. Teaching evidence-based medicine to surgical subspecialty residents. J Am Coll Surg
2003;197: 285-9. [PubMed] 27. Knowles MS, Downie CM, Basford P. Teaching and assessing in clinical practice. London: University of Greenwich, 1998: 23-38. 28. Sackett DL, Straus SE. Finding and applying evidence during clinical rounds: the “evidence cart.” JAMA
1998;280: 1336-8. [PubMed] 29. Deshpande N, Publicover M, Gee H, Khan KS. Incorporating the views of obstetric clinicians in implementing evidence-supported labour and delivery suite ward rounds: a case study. Health Info Libr J
2003;20: 86-94. [PubMed] 30. Dwarakanath LS, Khan KS. Modernizing the journal club. Hosp Med
2000;61: 425-7. [PubMed] 31. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: how to practice and teach EBM. Edinburgh: Churchill Livingstone, 2000.
|
This article has been 
