Purpose: Course evaluations by students are a standard tool that U.S. universities use to monitor the quality of their product. Here, the authors examine an alternative method of monitoring instructional quality that differs from traditional approaches in that it does not rely on students' ratings. The authors sought to glean relevant diagnostic information about course effectiveness from in-class exams used to assess students' learning progress (i.e., cognitively diagnostic assessments that explicitly target instructional content).
Method: The authors used data from an end-of-course, cumulative exam given in 2015 and in 2016 to 200 first-year medical students. They mapped the exam questions to 4 attributes and analyzed the students' overall mastery of the content tested and the percentage of students mastering each attribute.
Results: Analyses of the cognitively diagnostic assessment data revealed the percentage of the cohort who achieved/failed to achieve mastery of each of the attributes, discreet mastery profiles that distinguish among students with similar scores, and the percentage of the cohort within each of the 16 attribute mastery profiles. Analysis allowed the authors to evaluate how well the course content was delivered.
Conclusions: Cognitively diagnostic assessments enable in-class tests to appraise which skills specified in the curriculum have/have not been mastered by the students and how many students have mastered/failed to master which particular skills. Hence, if the learning goals have been well defined at the beginning of a course, then cognitively diagnostic assessments can show to what degree the instructional objectives have actually been accomplished.