Send to

Choose Destination
Emerg Med J. 2019 Nov 25. pii: emermed-2018-208268. doi: 10.1136/emermed-2018-208268. [Epub ahead of print]

Inter-rater reliability in a bespoke scoring tool: the Paediatric Observation Priority Score.

Author information

College of Medicine Biological Sciences and Psychology, University of Leicester, Leicester, UK
SAPPHIRE Group, Health Sciences, University of Leicester, Leicester, UK.
Paediatric Emergency Medicine Leicester Academic (PEMLA) Group, Leicester Hospitals, Leicester, UK.



Inter-rater reliability (IRR) is rarely determined for scoring systems used to recognise deterioration in children. Thus, the primary objective of this study was to determine the IRR of the Paediatric Observation Priority Score (POPS), a bespoke paediatric scoring system for ED use. The IRR of both the overall POPS and its individual parameters are to be investigated.


This is an experimental, single-centre study based in the Leicester children's ED, England. A purposive sample of nursing, healthcare assistants and doctors were recruited from October 2017 to December 2017. Eleven prerecorded video assessments were shown to 45 participants asked to generate a POPS for each child. The participants were blinded to each other's scores and the triage POPS. IRR among the participants POPS was measured using the intraclass correlation coefficient (ICC). Fleiss kappa was used to determine the IRR of individual parameters.


Overall, the ICC across all assessments by staff was 0.84 (95% CI 0.71 to 0.95). Fleiss kappa for parameters ranged from 0.87 to 1 for oxygen saturation, 0.48 to 0.91 for work of breathing, 0.55 to 1 for response, 0.46 to 0.87 for gut feeling, 0.53 to 1 for medical history, 0.76 to 1 for heart rate, 0.44 to 0.96 for respiratory rate and 0.51 to 1 for temperature.


This study suggests there is statistically moderate to excellent IRR of the POPS when assessing a variety of clinical presentations between different healthcare professionals with a range of experience.


clinical assessment, effectiveness; paediatrics; performance improvement; risk management; triage

Conflict of interest statement

Competing interests: AB is a National Institute for Health Research Academic Clinical Fellow.

Supplemental Content

Full text links

Icon for HighWire
Loading ...
Support Center