Stat Med. 2008 Jan 30;27(2):157-72; discussion 207-12.

Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.

Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS.

Source

Department of Mathematics and Statistics, Framingham Heart Study, Boston University, Boston, MA 02215, USA. mpencina@bu.edu

Abstract

Identification of key factors associated with the risk of developing cardiovascular disease and quantification of this risk using multivariable prediction algorithms are among the major advances made in preventive cardiology and cardiovascular epidemiology in the 20th century. The ongoing discovery of new risk markers by scientists presents opportunities and challenges for statisticians and clinicians to evaluate these biomarkers and to develop new risk formulations that incorporate them. One of the key questions is how best to assess and quantify the improvement in risk prediction offered by these new models. Demonstration of a statistically significant association of a new biomarker with cardiovascular risk is not enough. Some researchers have advanced that the improvement in the area under the receiver-operating-characteristic curve (AUC) should be the main criterion, whereas others argue that better measures of performance of prediction models are needed. In this paper, we address this question by introducing two new measures, one based on integrated sensitivity and specificity and the other on reclassification tables. These new measures offer incremental information over the AUC. We discuss the properties of these new measures and contrast them with the AUC. We also develop simple asymptotic tests of significance. We illustrate the use of these measures with an example from the Framingham Heart Study. We propose that scientists consider these types of measures in addition to the AUC when assessing the performance of newer biomarkers.

Comment in

Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). [Stat Med. 2008]
Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929).Ware JH, Cai T. Stat Med. 2008 Jan 30; 27(2):185-7.
Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina, R. B. D'Agostino Sr, R. B. D'Agostino Jr, R. S. Vasan, Statistics in Medicine (DOI: 10.1002/sim.2929). [Stat Med. 2008]
Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina, R. B. D'Agostino Sr, R. B. D'Agostino Jr, R. S. Vasan, Statistics in Medicine (DOI: 10.1002/sim.2929).Greenland P. Stat Med. 2008 Jan 30; 27(2):188-90.
Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). [Stat Med. 2008]
Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929).Cook NR. Stat Med. 2008 Jan 30; 27(2):191-5.
Comments on 'Evaluating the added predictive ability of a new marker' by M. Pencina, R. D'Agostino, R. D'Agostino Jr, R. Vasan, Statistics in Medicine (DOI: 10.1002/sim.2929). [Stat Med. 2008]
Comments on 'Evaluating the added predictive ability of a new marker' by M. Pencina, R. D'Agostino, R. D'Agostino Jr, R. Vasan, Statistics in Medicine (DOI: 10.1002/sim.2929).Kraemer HC. Stat Med. 2008 Jan 30; 27(2):196-8.
The need for reorientation toward cost-effective prediction: comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). [Stat Med. 2008]
The need for reorientation toward cost-effective prediction: comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929).Chi YY, Zhou XH. Stat Med. 2008 Jan 30; 27(2):182-4.
Integrated discrimination improvement and probability-sensitive AUC variants. [Stat Med. 2010]
Integrated discrimination improvement and probability-sensitive AUC variants.Van Calster B, Van Huffel S. Stat Med. 2010 Jan 30; 29(2):318-9.
Net reclassification improvement and decision theory. [Stat Med. 2009]
Net reclassification improvement and decision theory.Vickers AJ, Elkin EB, Steyerberg E. Stat Med. 2009 Feb 1; 28(3):525-6; author reply 526-8.
Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). [Stat Med. 2008]
Comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929).Pepe MS, Feng Z, Gu JW. Stat Med. 2008 Jan 30; 27(2):173-81.
The need for reorientation toward cost-effective prediction: comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). [Stat Med. 2008]
The need for reorientation toward cost-effective prediction: comments on 'Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond' by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929).Greenland S. Stat Med. 2008 Jan 30; 27(2):199-206.

PubMed

Result Filters

Display Settings:

Send to:

Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.

Source

Abstract

Comment in

Publication Types, MeSH Terms, Grant Support

Publication Types

MeSH Terms

Grant Support

LinkOut - more resources

Full Text Sources

Other Literature Sources

Supplemental Content

Save items

Related citations in PubMed

Cited by over 100 PubMed Central articles

Related information

Recent activity

Simple NCBI Directory

Getting Started

Resources

Popular

Featured

NCBI Information