An AUC-like index for agreement assessment

The commonly used statistical measures for assessing agreement of readings generated by multiple observers or raters, such as intraclass correlation coefficient (ICC) or concordance correlation coefficient (CCC), have well-known dependency on the data's normality assumption and, thereby, are heavily influenced by data outliers. Here, we propose a novel agreement measure (rank-based agreement index, rAI) by estimating agreement from the data's overall ranks. Such a nonparametric approach provides a global measure of agreement, regardless of the data's exact distributional form. We have shown rAI as a function of the overall ranks of each subject's extreme values. Furthermore, we propose an agreement curve, a graphic tool that aids visualizing the extent of the agreement, which strongly resembles the receiver operating characteristic (ROC) curve. We further show that rAI is a function of the area under the agreement curve. Consequently, rAI shares some important features with the area under the ROC curve (AUC). An extensive simulation study is included. We illustrate our method with two cancer imaging study data sets.