Background: Evidence-based practice (EBP) is now a component of most medical curricula. Summative assessment instruments are often of debatable quality, do not cover the full spectrum of EBP or lack authenticity.
Aim: To develop and evaluate the quality of an authentic assessment instrument for use in summative assessment of general practice trainees.
Methods: An assignment was designed based on the ask, acquire, appraise and apply steps of EBP. Content validity was evaluated by external EBP experts. Concurrent validity was tested with the Fresno test. Inter-rater agreement and internal consistency were measured. Acceptability and feasibility were also assessed.
Results: EBP experts agreed that the instrument had good content validity. Concurrent validity was good (disattenuated intraclass correlation coefficient 0.75). Inter-rater agreement varied from 0.70 to 0.83. Internal consistency was high (Cronbach's alpha 0.70-0.86). The procedure was feasible but only moderately acceptable to students.
Conclusion: Our authentic assignment provided a valid, reliable and feasible procedure to assess our students. Acceptability was moderate, probably due to teething problems in instructions given and unfamiliarity with the format. Consequential validity data are lacking and would be of value. Our instrument could be an interesting alternative to other validated tests that may be less authentic.