Send to

Choose Destination
Diagn Progn Res. 2017;1:12. doi: 10.1186/s41512-017-0012-3. Epub 2017 Apr 13.

Validation of prediction models: examining temporal and geographic stability of baseline risk and estimated covariate effects.

Author information

Institute for Clinical Evaluative Sciences, G106, 2075 Bayview Avenue, Toronto, Ontario M4N 3M5, Canada.
Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada.
Schulich Heart Research Program, Sunnybrook Research Institute, Toronto, Canada.
Department of Public Health, Erasmus MC-University Medical Center Rotterdam, Rotterdam, The Netherlands.
Predictive Analytics and Comparative Effectiveness Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA, USA.
Department of Medical Statistics and Bioinformatics, Leiden University Medical Center, Leiden, The Netherlands.
Peter Munk Cardiac Centre and Joint Department of Medical Imaging, Division of Cardiology, Department of Medicine, University of Toronto, Toronto, Canada.



Stability in baseline risk and estimated predictor effects both geographically and temporally is a desirable property of clinical prediction models. However, this issue has received little attention in the methodological literature. Our objective was to examine methods for assessing temporal and geographic heterogeneity in baseline risk and predictor effects in prediction models.


We studied 14,857 patients hospitalized with heart failure at 90 hospitals in Ontario, Canada, in two time periods. We focussed on geographic and temporal variation in baseline risk (intercept) and predictor effects (regression coefficients) of the EFFECT-HF mortality model for predicting 1-year mortality in patients hospitalized for heart failure. We used random effects logistic regression models for the 14,857 patients.


The baseline risk of mortality displayed moderate geographic variation, with the hospital-specific probability of 1-year mortality for a reference patient lying between 0.168 and 0.290 for 95% of hospitals. Furthermore, the odds of death were 11% lower in the second period than in the first period. However, we found minimal geographic or temporal variation in predictor effects. Among 11 tests of differences in time for predictor variables, only one had a modestly significant P value (0.03).


This study illustrates how temporal and geographic heterogeneity of prediction models can be assessed in settings with a large sample of patients from a large number of centers at different time periods.


Clinical prediction model; Geographic variation; Hierarchical regression model; Risk prediction; Temporal variation; Validation

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center