Send to

Choose Destination
See comment in PubMed Commons below
J Clin Epidemiol. 2010 Nov;63(11):1256-63. doi: 10.1016/j.jclinepi.2010.01.012. Epub 2010 Apr 28.

Calculating kappas from adjusted data improved the comparability of the reliability of triage systems: a comparative study.

Author information

  • 1Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands.



It is difficult to compare the reliability of triage systems with the kappa statistic. In this article, a method for comparing triage systems was developed and applied to previously conducted triage reliability studies.


From simulations with theoretical distributions, the minimum, normal, and maximum weighted kappa for 3- to 5-level triage systems were computed. To compare the reliability of triage systems in previously conducted triage reliability studies, the normal kappa was calculated. Furthermore, the reported quadratically weighted kappas were compared with the minimum, normal, and maximum weighted kappa to characterize the degree and direction of skewness of the data.


The normal kappa was higher in 3-level triage systems (median: κ=0.84) compared with 4-level (median: κ=0.37) and 5-level (median: κ=0.57) systems. In 3-level triage systems, the percentages observed agreement were unequally distributed, which resulted in small quadratically weighted kappas. In 4- and 5-level systems, the percentages observed agreement were more equally distributed compared with 3-level systems, which resulted in higher quadratically weighted kappa values.


When comparing triage systems with different numbers of categories, one should report both the normal and quadratically weighted kappa. Calculating normal kappas from previously conducted triage reliability studies revealed substantial theoretical differences in interrater reliability of triage systems than previously reported.

Copyright © 2010 Elsevier Inc. All rights reserved.

[PubMed - indexed for MEDLINE]
PubMed Commons home

PubMed Commons

How to join PubMed Commons

    Supplemental Content

    Full text links

    Icon for Elsevier Science
    Loading ...
    Write to the Help Desk