Format

Send to

Choose Destination
Int J Emerg Med. 2014 Sep 26;7:34. doi: 10.1186/s12245-014-0034-3. eCollection 2014.

Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?

Author information

1
Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran ; Program in Public Health, Department of Population Health and Disease Prevention, University of California Irvine, 653 E. Peltason Dr., Irvine 92697, CA, USA.
2
Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran ; Klinik f├╝r Innere Medizin III, Kardiologie, Angiologie und Internistische Intensivmedizin, Universit├Ątsklinikum des Saarlandes, Homburg/Saar 66421, Germany.
3
Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran.
4
Department of Emergency Medicine, Iran University of Medical Sciences, Tehran 14496, Iran.
5
Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran ; Department of Neurology, Saarland University Medical Center, Homburg/Saar 66421, Germany.
6
Kamyar Clinic, Tehran 51406, Iran.
7
Department of Emergency Medicine, School of Medicine, University of California Irvine, Irvine 92697, CA, USA.

Abstract

BACKGROUND:

We aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).

FINDINGS:

A SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient. In 22 out of 30 (73.3%) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30%), underestimation (30%), and judging the diagnostic value in an opposite direction (13.3%). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs. The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).

CONCLUSIONS:

Although SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.

KEYWORDS:

Clinical judgment; Decision-making; Diagnosis; Evidence-based medicine; Likelihood ratio; Script concordance test; Visual analog scales

Supplemental Content

Full text links

Icon for BioMed Central Icon for PubMed Central
Loading ...
Support Center