Format

Send to

Choose Destination
J Clin Epidemiol. 2015 Mar;68(3):279-89. doi: 10.1016/j.jclinepi.2014.06.018. Epub 2014 Aug 30.

A new framework to enhance the interpretation of external validation studies of clinical prediction models.

Author information

1
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508GA Utrecht, The Netherlands. Electronic address: T.Debray@umcutrecht.nl.
2
Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.
3
Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508GA Utrecht, The Netherlands.

Abstract

OBJECTIVES:

It is widely acknowledged that the performance of diagnostic and prognostic prediction models should be assessed in external validation studies with independent data from "different but related" samples as compared with that of the development sample. We developed a framework of methodological steps and statistical methods for analyzing and enhancing the interpretation of results from external validation studies of prediction models.

STUDY DESIGN AND SETTING:

We propose to quantify the degree of relatedness between development and validation samples on a scale ranging from reproducibility to transportability by evaluating their corresponding case-mix differences. We subsequently assess the models' performance in the validation sample and interpret the performance in view of the case-mix differences. Finally, we may adjust the model to the validation setting.

RESULTS:

We illustrate this three-step framework with a prediction model for diagnosing deep venous thrombosis using three validation samples with varying case mix. While one external validation sample merely assessed the model's reproducibility, two other samples rather assessed model transportability. The performance in all validation samples was adequate, and the model did not require extensive updating to correct for miscalibration or poor fit to the validation settings.

CONCLUSION:

The proposed framework enhances the interpretation of findings at external validation of prediction models.

KEYWORDS:

Case mix; Generalizability; Prediction model; Reproducibility; Transportability; Validation

PMID:
25179855
DOI:
10.1016/j.jclinepi.2014.06.018
[Indexed for MEDLINE]
Free full text

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center