Investigating the Impact of Preparation Strategies on USMLE Step 1 Performance

Abstract

Introduction

The United States Medical Licensing Exam (USMLE) Step 1 is the first of a series of three board exams. It serves both as a licensing exam as well as a screening tool for residency directors. Scores have continued to rise on the USMLE Step 1 over the past decade (Sutton et al., 2014). The number of medical students applying to competitive residency programs is also on the rise (Rinard & Mahabir, 2010). Residency directors at competitive programs use the USMLE Step 1 score in conjunction with class ranking to screen potential applicants (Lotfipour et al., 2008). Those students meeting certain academic and professional criteria are offered an interview. USMLE Step 1 scores correlate with how well a student will do on a residency board exam (Kay, Jackson, & Frank, 2015). A primary goal of medical educators is to promote success for students and to help them fulfill their career ambitions. The key to this process is providing an excellent medical curriculum and offering informative guidance for USMLE Step 1 preparation (Lieberman et al., 2010). This study examined the extent to which test preparation courses help improve scores on the USMLE Step 1 as well as when to take the exam following the second year of medical school. Common courses and preparatory programs including Doctors in Training, First Aid and Board Review Series books, and Kaplan and USMLE World Question banks were evaluated in correlation to score differences on the USMLE Step 1. Testing time interval following second year medical school was broken into quintiles. Students preparing for this high-stakes examination used various strategies for preparation and outcomes varied accordingly for the different preparation strategies employed. Not surprisingly, students in the first quartile of the medical school class performed better on USMLE Step 1 than students in the fourth quartile. The objective of this study is to investigate which preparation strategies correlate with high scores on the USMLE Step 1 in order to improve academic guidance regarding the best methods to prepare for the exam.

Methods

The West Virginia University (WVU) Institutional Review Board committee approved this study. Students from the WVU School of Medicine graduating classes of 2013-2014 who chose to participate were given a cover letter that instructed him or her to the purpose of the research and to his or her role as a participant. A survey (Appendix 1) in the form of a handout was designed for current WVU medical students who completed the second year curriculum and passed the USMLE Step 1 exam. The survey included the date of the exam, date of expected graduation, and resources used for exam (First Aid, Board Review Series, USMLE World Question Bank, Kaplan Question Bank, and Doctors in Training). A numerical analysis template was created for each student with data from the survey. The number of students who used each resource was tabulated and the time period for taking the exam was broken up into quintiles. The data set was de-identified for personal information and subsequently paired with USMLE Step 1 exam scores and class quartile rank based on a numerical match system. The mean USMLE Step 1 score was calculated for students who used each reference material and was reported with standard error. A student's t-test was used to compare between groups. The effect size was calculated to determine practical significance. A Pearson's coefficient was determined for quartile ranking and USMLE Step 1 scores. The coefficient of determination was likewise calculated. P<0.05 was considered statistically significant for all values.

Results

From the graduating classes of 2013 and 2014, there were a total of 165 students. Of those, 164 students completed the survey (99.39%). The majority of students used multiple resources to prepare for the exam. The resources can be broadly grouped into books, training programs, and question banks. Of those students who completed the survey, 97.56% used the First Aid book and 42.07% used the Board Review Series books. 51.83% used the Doctors in Training program. 88.41% used the USMLE Question bank and 15.85% used the Kaplan Question Bank. Those students who used the Doctors in Training program inherently used the First Aid book because it was included as the course reference (Figure 1).

Open in a separate window

Figure 1

Frequency Table

Students who used USMLE World Question bank had higher USMLE Step 1 scores (M=229, SE=1.4) than students who did not use this resource (M=215, SE=4.2). The difference was statistically significant t(162) = 3.3, p<0.05. The calculated effect size (d=.77) suggests that the difference in mean scores was also practically significant. No other statistically significant differences were found between groups (Figure 2).

Open in a separate window

Figure 2

Study Tool Means and Standard Deviations

The time since finishing second year medical school to taking the exam was broken into quintiles. The time interval until taking the exam did not have an effect on USMLE Step 1 scores, unlike for other medical exams (Malangoni et al., 2012). Those students who took the test closest to the end of second year medical school or closest to the beginning of third year medical school had slightly lower scores but these differences were not statistically significant (Figure 3).

Open in a separate window

Figure 3

Timing of Exam

Not surprisingly, a statistically significant difference in USMLE Step 1 scores was seen between class ranking quartiles. Those students who did well on the medical school curriculum performed well on USMLE Step 1. A strong negative correlation was seen for USMLE Step 1 scores and class ranking quartiles (r = −0.689). The 4th quartile students had the lowest USMLE Step 1 scores. The coefficient of determination is 0.47. Approximately 47% of the variance in USMLE Step 1 scores can be explained by knowing the class ranking quartile for the student. Our data showed no significant difference in USMLE Step 1 scores between students in the classes of 2013 and 2014 (2013: mean 226.4, N= 90 and 2014: mean 227.4, N=75) (Figure 4).

Open in a separate window

Figure 4

Step 1 Scores and Quartiles

Discussion

Commercial test preparation courses, books, and question banks have become increasing popular due to the high-stake nature of the USMLE Step 1. Students often ask medical educators about the best methods to prepare for the exam. In response to these questions, some schools have even started a seminar series to discuss process-oriented exam preparation (Strowd & Lambros, 2010). The best advice to give students is mixed. Over the past several years, studies have reported conflicting results. The National Board of Medical Examiners Comprehensive Basic Science Self-Assessment, which most students take at the end of second year medical school, only accounts for 67% of the variance on USMLE Step 1 scores (Morrison et al., 2010). Therefore, test preparation does play some role in exam performance. Zhang and colleagues however showed that exam performance was closely related to class rank and was independent of preparation technique (Zhang, Rauchwarger, Toth, & O'Connell, 2004). Furthermore, scores on the Medical College Admissions Test (MCAT) correlate closely with performance on USMLE Step 1(Julian, 2005). Additionally, in a large group of students commercial test preparation courses provided no improvement on USMLE Step 1 scores (Werner & Bull, 2003). A novel approach, peer-designed review courses, on the other hand have been tied to better performance on the USMLE Step 1 (Alcamo, Davids, Way, Lynn, & Vandre, 2010). Therefore, traditional preparation techniques may need to be re-evaluated in favor of the more team based and problem solving peer review courses.

In this paper, we showed a unique series of results. Similar to other groups we found that commercial test courses had no statistical significance on exam score (Thadani, Swanson, & Galbraith, 2000). Not surprisingly, we also found that students in the top quartiles performed better on USMLE Step 1. These results were similar to what Zhang and colleagues found at University of Miami School of Medicine (UMSM). With the use of ANCOVA and the second year GPA as a covariate, Zhang's group found that participation in preparation courses had no significant effect on student USMLE Step 1 performance when second-year GPA was controlled (Zhang et al., 2004). The novel finding we did report is that the use of the USMLE World question bank provided a statistically significant improvement on USMLE Step 1 scores. The question bank is interactive and requires students to engage with the resource. Similar to peer-review courses it may be an improved alternative to traditional examination preparation.

Group study is a common method utilized by medical students in preparation for exams and recent evidence has supported its effectiveness. Medical education curriculums have started including problem based learning courses. The benefit of problem-based learning is the application of knowledge through discussion and team engagement.

Problem based learning has been shown to increase retention of material and enhance performance on exams (Imanieh, Dehghani, Sobhani, & Haghighat, 2014). A recent study employed the same techniques for USMLE Step 1 preparation. The Ohio State University College of Medicine (OSUCOM) conducted a study comparing the Step 1 scores of students who participated in a weekly peer designed and peer-led review course during the second year of medical school to other students at OSUCOM who did not participate in the preparation course. When the Step 1 scores were examined using ANCOVA, the participants of the peer-designed and peer-led review course scored significantly higher than the nonparticipants (Alcamo et al., 2010).

Another topic that students frequently consider is when to take the exam. Previously, when the USMLE Step 1 was administered as a paper-and-pencil test, it was administered only twice annually. The examination is now computerized and students have the option to take the USMLE Step 1 at anytime. Students often have anxiety about when to set their test date and frequently ask advice from medical educators. Students at WVU School of Medicine have approximately 7 weeks between the end of the second year curriculum and the start of third year clinical work. The school does not currently offer a sponsored review course, so most students either choose to study on their own for several weeks with several study resources or pay for a month long electronic study course such as Doctors in Training. The time until taking the exam was broken into quintiles. We reported no statistically significant difference in scores based on when the exam was taken. Pohl and colleagues also analyzed the relationship between students Step 1 scores and the amount of time that had passed since the completion of their second year medical course work. The study concluded that as long as the exam was scheduled with 2 months of finishing second year course work there was no statistically significant difference in scores based on when the exam was taken (Pohl, Robeson, Hojat, & Veloski, 2002).

Conclusion

We conducted this study in order to provide tangible evidence that medical educators may use in advising students on USMLE Step 1 preparation. Similar to other groups we have shown that commercial based preparation courses provided no statistically significant benefit on USMLE Step 1 scores. We also showed that the timing for when the exam is taken does not correlate with better or worse scores. What we did find is that study resources, which require student engagement and interaction, correlate with better Step 1 scores. Future work will examine the benefit of peer-review courses and increased problem based learning strategies. Problem based learning may be a way in which students can improve scores on this high-stakes exam. As the average score on USMLE Step 1 continues to increase, medical educators are charged with providing insightful advice to students on how best to succeed on this exam as well as successfully match into residency programs. Novel preparation approaches that engage students and help them participate in the learning process are sure to help students succeed.

Acknowledgements

Glossary Terms

Appendix 1

Open in a separate window

Footnotes

References