Send to

Choose Destination
J Pharm Policy Pract. 2018 May 3;11:11. doi: 10.1186/s40545-018-0137-y. eCollection 2018.

Evaluating inter-rater reliability of indicators to assess performance of medicines management in health facilities in Uganda.

Author information

Management Sciences for Health, Plot 15, Princess Anne Drive, Bugolobi, P.O. Box 71419, Kampala, Uganda.
2Harvard Pilgrim Health Care Institute, 401 Park Drive Suite 401,, Boston, MA 02215 USA.
3Ministry of Health, Pharmacy, Division, Lourdel Road, Wandegeya, Kampala Uganda.



To build capacity in medicines management, the Uganda Ministry of Health introduced a nationwide supervision, performance assessment and recognition strategy (SPARS) in 2012. Medicines management supervisors (MMS) assess performance using 25 indicators to identify problems, focus supervision, and monitor improvement in medicines stock and storage management, ordering and reporting, and prescribing and dispensing. Although the indicators are well-recognized and used internationally, little was known about the reliability of these indicators. An initial assessment of inter-rater reliability (IRR), which measures agreement among raters (i.e., MMS), showed poor IRR; subsequently, we implemented efforts to improve IRR. The aim of this study was to assess IRR for SPARS indicators at two subsequent time points to determine whether IRR increased following efforts to improve reproducibility.


IRR was assessed in 2011 and again after efforts to improve IRR in 2012 and 2013. Efforts included targeted training, providing detailed guidelines and job aids, and refining indicator definitions and response categories. In the assessments, teams of three MMS measured 24 SPARS indicators in 26 facilities. We calculated IRR as a team agreement score (i.e., percent of the MMS teams in which all three MMS had the same score). Two sample tests for proportions were used to compare IRR scores for each indicator, domain, and overall for the initial assessment and the following two assessments. We also compared the IRR scores for indicators classified as simple (binary) versus complex (multi-component). Logistic regression was used to identify supervisor group characteristics associated with domain-specific and overall IRR scores.


Initially only five (21%) indicators had acceptable reproducibility, defined as an IRR score ≥ 75%. At the initial assessment, prescribing quality indicators had the lowest and stock management indicators had the highest IRR. By the third IRR assessment, 12 (50%) indicators had acceptable reproducibility, and the overall IRR score improved from 57% to 72%. The IRR of simple indicators was consistently higher than that of complex indicators in the three assessment periods. We found no correlation between IRR scores and MMS experience or professional background.


Assessments of indicator reproducibility are needed to improve IRR. Using simple indicators is recommended.


Complex indicators; Data quality audit; Data reproducibility; IRR; Inter-rater reliability; Medicines management indicators; Performance assessment quality; Simple indicators

Conflict of interest statement

This study evaluated IRR of medicines management data as part of the national capacity-building strategy SPARS carried out by MMS under the Ministry of Health, Uganda. The study did not involve patients, human or personal health data, human tissue, or animals. Therefore, the study did not require ethical approval or a waiver. All observations and data collection were conducted with the permission of Ministry of Health, the District Health Officers, the facility in-charges, and the MMS. The study constituted a Ministry of Health initiated data quality evaluation and is approved by the Ministry of Health. All authors of the manuscript have read and agreed to its content and are accountable for all aspects of the accuracy and integrity of the manuscript in accordance with ICMJE criteria.That the article is original, has not already been published in a journal, and is not currently under consideration by another journal. Consent for publication from the Ministry of Health has been obtained.The authors declare that they have no competing interests.Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplemental Content

Full text links

Icon for BioMed Central Icon for PubMed Central
Loading ...
Support Center