Format

Send to

Choose Destination
J Pediatr Gastroenterol Nutr. 2018 Dec;67(6):e111-e116. doi: 10.1097/MPG.0000000000002089.

Validity Evidence for Direct Observation of Procedural Skills in Paediatric Gastroscopy.

Author information

1
Department of Gastroenterology, Dudley Group Hospitals NHS Foundation Trust, Dudley.
2
Joint Advisory Group, Royal College of Physicians.
3
Department of Paediatric Gastroenterology, Royal Free Hospital, London.
4
Department of Paediatric Gastroenterology, John Radcliffe Hospital, Oxford.
5
Department of Gastroenterology, Torbay and South Devon NHS Foundation Trust, Torquay.
6
Department of Gastroenterology, Norfolk and Norwich University Hospital, Norwich.
7
Department of Paediatric Gastroenterology, Alder Hey Children's Hospital, Liverpool.
8
Department of Paediatric Gastroenterology, Sheffield Children's NHS Foundation Trust, Western Bank, Sheffield, UK.

Abstract

OBJECTIVES:

Direct observation of procedural skills (DOPS) are competence-assessment tools in endoscopy. Formative paediatric gastroscopy DOPS were implemented into the UK curriculum in 2016 but lack validity evidence; we aimed to assess validity evidence using a recognised contemporary validity framework.

METHODS:

We performed a prospective UK-wide analysis of formative paediatric gastroscopy DOPS submitted to the e-Portfolio over 1 year. Internal structure validity was assessed using interitem correlations between DOPS items, average domain, and skillset scores and with the overall competency rating. Overall competence scores and mean DOPS scores were compared by trainee seniority and procedure count (discriminative validity). Receiver operating characteristic curve analysis was performed to explore if DOPS scores could be used to delineate procedural competency (consequential validity).

RESULTS:

A total of 157 DOPS assessments were completed by 20 trainers for 17 trainees. Strengths of correlations varied between DOPS components, with overall competency correlating most with technical-predominant items, domains and skillsets. Both the overall assessor's rating and mean DOPS scores increased with trainee seniority (P < 0.001) and lifetime procedure count (P < 0.001). Overall competency could be delineated using mean DOPS scores (area under receiver operating characteristic curve 0.95, P < 0.001), with a threshold of 3.9 providing optimal sensitivity (94.4%) and specificity (89.7%).

CONCLUSIONS:

Competencies in paediatric gastroscopy, as assessed using DOPS, vary in their correlation with overall competence and increase with trainee experience. Formative DOPS thresholds could be used to indicate readiness for summative assessment. Our study therefore provides evidence of internal structure, discriminative, and consequential validity in support of formative paediatric gastroscopy DOPS.

Supplemental Content

Full text links

Icon for Wolters Kluwer
Loading ...
Support Center