Format

Send to

Choose Destination
Med Educ. 2015 Oct;49(10):1004-15. doi: 10.1111/medu.12809.

Use of an error-focused checklist to identify incompetence in lumbar puncture performances.

Author information

1
Department of Medicine, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
2
W21C, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
3
Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada.
4
Department of Critical Care Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
5
Department of Surgery, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
6
Department of Clinical Neurosciences, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.

Abstract

CONTEXT:

Checklists are commonly used in the assessment of procedural competence. However, on most checklists, high scores are often unable to rule out incompetence as the commission of a few serious procedural errors typically results in only a minimal reduction in performance score. We hypothesised that checklists constructed based on procedural errors may be better at identifying incompetence.

OBJECTIVES:

This study sought to compare the efficacy of an error-focused checklist and a conventionally constructed checklist in identifying procedural incompetence.

METHODS:

We constructed a 15-item error-focused checklist for lumbar puncture (LP) based on input from 13 experts in four Canadian academic centres, using a modified Delphi approach, over three rounds of survey. Ratings of 18 video-recorded performances of LP on simulators using the error-focused tool were compared with ratings obtained using a published conventional 21-item checklist. Competence/incompetence decisions were based on global assessment. Diagnostic accuracy was estimated using the area under the curve (AUC) in receiver operating characteristic analyses.

RESULTS:

The accuracy of the conventional checklist in identifying incompetence was low (AUC 0.11, 95% confidence interval [CI] 0.00-0.28) in comparison with that of the error-focused checklist (AUC 0.85, 95% CI 0.67-1.00). The internal consistency of the error-focused checklist was lower than that of the conventional checklist (α = 0.35 and α = 0.79, respectively). The inter-rater reliability of both tools was high (conventional checklist: intraclass correlation coefficient [ICC] 0.99, 95% CI 0.98-1.00; error-focused checklist: ICC 0.92, 95% CI 0.68-0.98).

CONCLUSIONS:

Despite higher internal consistency and inter-rater reliability, the conventional checklist was less accurate at identifying procedural incompetence. For assessments in which it is important to identify procedural incompetence, we recommend the use of an error-focused checklist.

PMID:
26383072
PMCID:
PMC4584502
DOI:
10.1111/medu.12809
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Wiley Icon for PubMed Central
Loading ...
Support Center