Purpose: Schools undergoing curricular reform are reconsidering the optimal timing of Step 1. This study provides a psychometric investigation of the impact on United States Medical Licensing Examination Step 1 scores of changing the timing of Step 1 from after completion of the basic science curricula to after core clerkships.
Method: Data from four schools that recently moved the examination were analyzed in a pre-post format using examinee scores from three years before and after the change. The sample included scores from 2008 through 2016. Several confounders were addressed, including rising national scores and potential differences in cohort abilities using deviation scores and analysis of covariance (ANCOVA) controlling for Medical College Admission Test (MCAT) scores. A resampling procedure compared study schools' score changes versus similar schools' in the same time period.
Results: The ANCOVA indicated postchange Step 1 scores were higher compared with prechange (adjusted difference = 2.67; 95% confidence interval: 1.50-3.83, P < .001; effect size = 0.14) after adjusting for MCAT scores and rising national averages. The average score increase in study schools was larger than changes seen in similar schools. Failure rates also decreased from 2.87% (n = 48) pre change to 0.39% (n = 6) post change (P < .001).
Conclusions: Results suggest moving Step 1 after core clerkships yielded a small increase in scores and a reduction in failure rates. Although these small increases are unlikely to represent meaningful knowledge gains, this demonstration of "noninferiority" may allow schools to implement significant curricular reforms.