Format

Send to

Choose Destination
Acad Radiol. 2005 Oct;12(10):1293-300.

Incorporating utility-weights when comparing two diagnostic systems: a preliminary assessment.

Author information

1
Department of Biostatistics, Graduate School of Public Health, Department of Radiology, Imaging, Suite 4200, Magee-Women's Hospital, University of Pittsburgh, Pittsburgh, PA 15216, USA.

Abstract

RATIONALE AND OBJECTIVES:

We sought to develop a new index that incorporates utility-weights when assessing the overall performance of a diagnostic system and to provide a statistical test for comparing two indices in a paired study design.

MATERIALS AND METHODS:

The area under the receiver operating characteristic (ROC) curve (AUC) was used as the basis for constructing a new index. The index we propose represents a weighted average of class-specific AUCs each of which relates to a class of pairs of actually negative (normal) and actually positive (abnormal) cases with a specific predetermined utility (or clinical importance). For each pair of normal-abnormal cases, the utility is defined a priori and based on external (covariate) information. In the proposed approach utility-weights represent the relative importance (utility) of discriminating between different types of normal and abnormal cases (pairs of the same type are combined in the classes termed utility-classes). We also describe a simple nonparametric procedure for comparing the proposed indices as computed from paired data. Computer simulations were conducted to evaluate the behavior of the type I error of the proposed test in the simple albeit important instance of two utility-classes.

RESULTS:

The new index provides an extension of the commonly used area under the ROC curve. It allows for incorporation of utility-weights into the analysis and reduces to the conventional AUC index when all assigned utility-weights are equal to unity. Computer simulations indicate that in the considered scenario of two utility-classes, the type I error of the proposed test is comparable to that of the conventional nonparametric test for equality of AUC indices.

CONCLUSIONS:

The proposed index and the statistical test provide a practical approach of incorporating utilities when comparing diagnostic systems.

PMID:
16179206
DOI:
10.1016/j.acra.2005.05.028
[Indexed for MEDLINE]

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center