Format

Send to

Choose Destination
J Surg Res. 2015 Jun 15;196(2):302-6. doi: 10.1016/j.jss.2015.03.018. Epub 2015 Mar 18.

Crowd-sourced assessment of surgical skills in cricothyrotomy procedure.

Author information

1
Department of Electrical Engineering, University of Washington, Seattle, Washington. Electronic address: navaa@u.washington.edu.
2
Department of Otolaryngology - Head and Neck Surgery, University of Washington, Seattle, Washington.
3
M.D.C. School of Medicine, Stanford University, Stanford, California.
4
Department of Electrical Engineering, University of Washington, Seattle, Washington.
5
Department of Urology, Seattle Children's Hospital Seattle, Washington.

Abstract

BACKGROUND:

Objective assessment of surgical skills is resource intensive and requires valuable time of expert surgeons. The goal of this study was to assess the ability of a large group of laypersons using a crowd-sourcing tool to grade a surgical procedure (cricothyrotomy) performed on a simulator. The grading included an assessment of the entire procedure by completing an objective assessment of technical skills survey.

MATERIALS AND METHODS:

Two groups of graders were recruited as follows: (1) Amazon Mechanical Turk users and (2) three expert surgeons from University of Washington Department of Otolaryngology. Graders were presented with a video of participants performing the procedure on the simulator and were asked to grade the video using the objective assessment of technical skills questions. Mechanical Turk users were paid $0.50 for each completed survey. It took 10 h to obtain all responses from 30 Mechanical Turk users for 26 training participants (26 videos/tasks), whereas it took 60 d for three expert surgeons to complete the same 26 tasks.

RESULTS:

The assessment of surgical performance by a group (n = 30) of laypersons matched the assessment by a group (n = 3) of expert surgeons with a good level of agreement determined by Cronbach alpha coefficient = 0.83.

CONCLUSIONS:

We found crowd sourcing was an efficient, accurate, and inexpensive method for skills assessment with a good level of agreement to experts' grading.

KEYWORDS:

Crowd sourcing; Emergency cricothyrotomy procedure; Surgical skill assessment

PMID:
25888499
PMCID:
PMC5945282
DOI:
10.1016/j.jss.2015.03.018
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Elsevier Science Icon for PubMed Central
Loading ...
Support Center