Send to

Choose Destination
See comment in PubMed Commons below
Acad Emerg Med. 2017 Aug 20. doi: 10.1111/acem.13296. [Epub ahead of print]

Simulation for Assessment of Milestones in Emergency Medicine Residents.

Author information

Emergency Medicine, Hennepin County Medical Center, University of Minnesota Medical School.
Emergency Medicine.
University of Illinois College of Medicine at Peoria.
Lehigh Valley Health Network.
Emergency Medicine, Emory University.
Emergency Medicine, University of Iowa.
Emergency Medicine, Hofstra University North Shore Long Island Jewish SOM, Northwell Health Center.
Emergency Medicine, University of Illinois College of Medicine at Peoria.
Department of Anesthesiology, Vanderbilt University.
Emergency Medicine, University of Chicago.
Division of Education, Lehigh Valley Health Network.
Emergency Medicine, Mount Sinai Hospital, New York.
Emergency Medicine, Kaiser Permanente, Los Angeles Medical Center.



All residency programs in the United States are required to report their residents' progress on the Milestones to the Accreditation Council for Graduate Medical Education (ACGME) biannually. Since the development and institution of this competency-based assessment framework, residency programs have been attempting to ascertain the best ways to assess resident performance on these metrics. Simulation was recommended by the ACGME as one method of assessment for many of the milestone subcompetencies. We developed 3 simulation scenarios with scenario specific Milestone based assessment tools. We aimed to gather validity evidence for this tool.


We conducted a prospective observational study to investigate the validity evidence for 3 mannequin-based simulation scenarios for assessing individual residents on Emergency Medicine (EM) milestones. The subcompetencies (i.e. PC1, PC2, PC3) included were identified via a modified Delphi technique using a group of experienced EM simulationists. The scenario specific checklist items were designed based on the individual milestone items within each EM subcompetency chosen for assessment and reviewed by experienced EM simulationists. Two independent live raters who were EM faculty at the respective study sites scored each scenario following brief rater training. The interrater reliability (IRR) of the assessment tool was determined by measuring intraclass correlation coefficient (ICC) for the sum of the checklist (CL) items as well as the global rating scales (GRS) for each scenario. Comparing GRS and CL scores between various PGY levels was performed with ANOVA.


Eight subcompetencies were chosen to assess with 3 simulation cases, using 118 subjects. Evidence of test content, internal structure, response process and relations with other variables were found. The ICCs for the sum of the CL items and the GRSs were > 0.8 for all cases, with one exception (clinical management GRS 0.74 in sepsis case). The Sum of Checklist items and GRSs (p<0.05) discriminated between post graduate (PGY) levels on all cases. However, when the specific CL items were mapped back to milestones in various proficiency levels, the milestones in the higher proficiency levels (level 3 & 4) did not often discriminate between various PGY levels. Level 3 milestone items discriminated between PGY levels on 5 out of 12 occasions they were assessed, and Level 4 items discriminated only 2 out of 12 times they were assessed.


Three simulation cases with scenario specific assessment tools allowed evaluation of EM residents on proficiency levels 1-4 within 8 of the EM Milestone subcompetencies. Evidence of test content, internal structure, response process and relations with other variables were found. Good to excellent IRR and the ability to discriminate between various PGY levels was found for both the sum of CL items and GRSs. However, there was a lack of a positive relationship between advancing PGY level and the completion of higher level milestone items (levels 3 and 4). This article is protected by copyright. All rights reserved.

PubMed Commons home

PubMed Commons

How to join PubMed Commons

    Supplemental Content

    Full text links

    Icon for Wiley
    Loading ...
    Support Center