OBJECTIVE: To develop and validate an instrument for measuring knowledge and skills in evidence based medicine and to investigate whether short courses in evidence based medicine lead to a meaningful increase in knowledge and skills. DESIGN: Development and validation of an assessment instrument and before and after study. SETTING: Various postgraduate short courses in evidence based medicine in Germany. PARTICIPANTS: The instrument was validated with experts in evidence based medicine, postgraduate doctors, and medical students. The effect of courses was assessed by postgraduate doctors from medical and surgical backgrounds. INTERVENTION: Intensive 3 day courses in evidence based medicine delivered through tutor facilitated small groups. MAIN OUTCOME MEASURE: Increase in knowledge and skills. RESULTS: The questionnaire distinguished reliably between groups with different expertise in evidence based medicine. Experts attained a threefold higher average score than students. Postgraduates who had not attended a course performed better than students but significantly worse than experts. Knowledge and skills in evidence based medicine increased after the course by 57% (mean score before course 6.3 (SD 2.9) v 9.9 (SD 2.8), P<0.001). No difference was found among experts or students in absence of an intervention. CONCLUSIONS: The instrument reliably assessed knowledge and skills in evidence based medicine. An intensive 3 day course in evidence based medicine led to a significant increase in knowledge and skills.