• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information

Studies of external validation: Framingham-Anderson48;49

Validation study IDValidating population, location, date, description Follow upPopulation characteristics
Inclusion
Exclusion
Co-morbidity
Outcome Changes to methods.
Methods used for validation
Results
Attempt at recalibration
Authors conclusions: applicability in studied population. Reviewer notes
De Henauw et al.236Charleroi and Ghent, Belgium

1985–1992

MONICA project centres: population based survey

Comparison of predicted rates with separate incidence data.

No prospective study
Age 25–64
N=3x2000 in 3 different time periods
Approximately 50% men
CHDCompared predicted risk with differences in CHD incidence in two regions, and changes in incidence in two regions over time.Authors note low participation rates in surveys
Predicted riskCharleroi>Ghent by 5–21%
CHD incidenceCharleroi>Ghent by 50%
Predicted change in CHD incidenceCharleroi- no significant trend
Ghent- no significant trend
Actual change in CHD incidenceCharleroi increase
Ghent decrease
Kornitzer etal. 200067Belgian Physical Fitness Study

Three communes randomly chosen from all 42 Belgian districts. Participants in an RCT

1980–1984

10 year follow up
Age 40–55
N=2186
Men

Free of clinical CHD
CHD mortality

Calibration: graphical representation of CHD mortality in quartiles of risk
Calibration
A steady increase in mortality from CHD across quartiles of risk. Relative risk of coronary death, highest versus lowest quartile, 14.2
Kornitzer et al. 200067Belgian Interuniversity Research on Nutrition and Health

Three communes randomly chosen from all 42 Belgian districts. Participants in an RCT

1980–1984

10 year follow up
Age 50–74
N=1758
Women

Free of clinical CHD
CHD mortality

Calibration: graphical representation of CHD mortality in quartiles of risk
Calibration
A steady increase in mortality from CHD across quartiles of risk. Relative risk of coronary death, highest versus lowest quartile, 3.5
Brindle et al.244Caerphilly and Speedwell Studies (two populations from Caerphilly, Wales and Speedwell, Bristol)
Phase 1
Baseline: 1979–83
10 years follow- up.
Phase 2
Baseline: 1982–88
10 years follow- up.
Phase 1:
Age 45–63
N=3213
100% men

Individuals with any previous cardiovascular disease.

Phase 2:
Age 47–67
N=2467
100% men
CVD mortality
CHD mortality
Myocardial infarction

Calibration: predicted and observed

Discrimination: Area under ROC

Sensitivity, specificity, positive and negative likelihood ratios were calculated at different CHD risk thresholds
Phase 1Phase 2Abstract only. Additional data obtained from first author.

Concluded that because the predicted risk increases relatively more than the observed risk from Phase 1 to Phase 2, the predicted versus observed ratio moves to over- prediction (ie greater than 1) in Phase 2.

The likelihood ratio of a negative test improves ((ie is lower) in Phase 2, meaning that a patient who is predicted to be low risk according to a threshold can be better reassured than in Phase 1
Calibration
P:OP:O
CVD mortality:0.80 (0.70 to 0.93)1.16 (1.00 to 1.36)
CHD mortality:0.97 (0.83 to 1.15)1.39 (1.18 to 1.71)
MI:1.24 (1.11 to 1.42)1.37 (1.22 to 1.56)
Discrimination
AUROCAUROC
CVD mortality:0.70 (0.67 to 0.74)0.71 (0.67 to 0.76)
CHD mortality:0.69 (0.65 to 0.74)0.70 (0.65 to 0.75)
MI:0.66 (0.62 to 0.69)0.68 (0.64 to 0.71)
Simon et al. 1997254Renault Volvo Coeur Study

France and Sweden

1993

Random selection of car-plant workers in France and Sweden
Age 45–50
N=approximately 1800 (90% of 2000 with data)
100% men
France 52% blue collar
Sweden 39% blue collar

No history of CHD
CHD

Comparison of Framingham risk in similar populations in 2 countries with different CHD rates
5 year Framingham risk of CHD was similar in French (3.6% SD 2.72) and Swedish (3.88% SD 2.84) car-plant workers. However CHD mortality rate is 50% higher in Sweden than France.Despite the expected higher rate of CHD death in Sweden compared with France, the Framingham risk scores did not differ between cohorts.

Expected numbers of MI cases in the age group studied was low.
Hense et al. 2003237Augsburg Germany

1984/85
1989/90

WHO MONICA cross-sectional survey

Follow up 10.5 years
Age 35–64
N=5786
49% men

Free of IHD on ECG and angina pectoris

Complete risk factor and follow up data
MI and CHD equations combined

Silent and unrecognised MI not evaluated. Sudden death inclusion different.
Non-fasting blood
Self-reported diabetes
Smokers: at least 1 cigarette/day

Calibration:
predicted: observed (P:O)

Discrimination: area under ROC

Miller 1991 approach
CalibrationPopulation had middle to low coronary event incidence compared with other MONICA cohorts (low compared with other German cohorts)

Predicted risk was about twice the absolute risk actually observed and a marked increment in risk difference with rising age.
Non fatal MI plus fatal CHD
P:O35–44 years45–54 years55–64 years
Men2.22.11.9
Women5.72.82.4
Graphical representation of observed and predicted risk revealed a consistent and marked excess of predicted risk with rising observed risk

Discrimination
Area under ROC
Men0.78 (95%CI 0.73–0.84) –146 events
Women0.88 (95%CI 0.80–0.96) –32 events
Miller approach: negative intercept different from 0 (p<0.001). Beta coefficient not significantly different from 1.
Hense et al. 2003237PROCAM

Munster, Germany

1979–1985

Workers from 52 companies and local government authorities

Follow up 11.6 years for men and 11.1 years for women
Age 35–64
N=8682
64% male

No previous MI or stroke

Complete risk factor and follow up data
MI and CHD combined

Silent and unrecognised MI not evaluated. Sudden death inclusion different
Fasting blood
Single BP measure
Diabetes questionnaire
Smokers: at least 1 cigarette/day

Calibration: predicted: observed (P:O)

Discrimination: area under ROC

Miller 1991 approach
CalibrationPopulation had middle to low coronary event incidence compared with other MONICA cohorts.

Predicted risk was about twice the absolute risk actually observed and a marked increment in risk difference with rising age.
Non fatal MI plus fatal CHD
P:O35–44 years45–54 years55–64 years
Men2.51.71.6
Women2.83.32.8
Discrimination
Area under ROC
Men0.73 (95%CI 0.70–0.75) –307 events
Women0.77 (95%CI 0.69–0.85) –31 events
Miller approach: negative intercept different from 0 (p<0.001). Beta coefficient not significantly different from 1.
Ferrario et al. 200537
Menotti et al. 2000238Joint European ESC, EAS, ESH coronary risk chart 1998 Based on Framingham Anderson 1991

Seven Countries Study of Cardiovascular Diseases - Italian Rural Areas Study

Crevalcore, Northern Italy. Montegiorgio, Central Italy. Prospective population study in 2 rural communities

1960

25 year follow up
Age 40–59
N=1656
100% male

Free of coronary heart disease. Complete data.
CHD

Casual blood samples.
Cholesterol measured by Anderson and Keys method.

Compared prediction with a function derived from the cohort.
Framingham risk function based coronary risk chart overestimates absolute coronary risk in countries characterised by a lower incidence of coronary events.There is little information on external validation.
Furuya et al. 2001263Kanagawa prefecture, Japan

People attending health check up in area with higher IHD mortality rate than whole of Japan.

1998

2 year follow up
N=1652
31% 62468}male
CHD
Compared predicted risk with reports from Japanese cohort studies
Predicted probabilities are higher than values reported from Japanese cohort studies.Framingham overestimates in Japanese population.
Jee et al. 2003253Korean Medical Insurance Corporation (KMIC) Study

Republic of Korea

1990–1992

Insurance company medical examination

8 years follow up
Age 35–59
N=183,675
63% male

Attended 2 health examinations
CHD (Fatal and non-fatal CHD)

Comparison of P:O
Framingham predicted risk underestimated observed risk in men aged <45 and overestimated at age >50. In women, the Framingham equation underestimated risk in all age groups
P:O
Results are presented per 100 person-years
MenWomen
35–390.70.4
40–440.90.5
45–491.00.7
50–541.20.6
55–591.40.7
Grover et al. 2000265Canadian
Working Group on Hypercholesterolaemia and other Dislipidaemias 1998/2000. Includes Framingham Anderson risk calculation

North America

10 clinics participating in Lipid Research Clinics prevalence and follow up studies

1972–1976

Average 12.2 year follow up
N=2218
Age 40–70 (men)
Age 50–70 (women)

Free of CVD. Not taking digitalis or anti-arrhythmic medication. Not pregnant.
Complete data
Death due to coronary artery disease (definite or suspected)

Discrimination: area under ROC
Discrimination1998 Canadian lipid screening guidelines discriminated better in the LRC cohort than NCEP ATP, NCEP ATP2 and previous Canadian CCC guidelines. This may be attributed to incorporation of HDL cholesterol into model
AUROC0.83 (SD 0.02)
0.83 men
0.82 women
Kjeldsen et al. 2000266Losartan
Intervention for Endpoint reduction in hypertension study (LIFE)

Scandinavia, UK and USA

1995–1997

RCT

Follow up when 479 events reported
Age 55–80
N=9,194
46% men in study overall

Essential hypertension and ECG-LVH

No cardiac conditions or non-cardiac conditions that may limit survival
MI, stroke or CV deathFramingham risk score strongly stratified patients into risk categoriesThe Framingham risk score was found to be useful for predicting cardiovascular complications in hypertensive patients with ECG-LVH recruited to an RCT.

No other information on calibration and discrimination provided.
Tertiles of Framingham riskObserved event rate
Low2.8%
Middle4.7%
Highest8.1%
Nijhuis et al. 2002239Rotterdam Study

Rotterdam, NL Representative population sample.

1990–1993

7 year follow up
Age 55–69
Age 69–80
N=5431
Approximately 39% men
CHD

Discrimination: ROC analyses
Discrimination
Age 55–69 Area under ROC curve 0.708
Age 69–80 Area under ROC curve 0.663
Inclusion of family history of MI and ankle brachial pressure improved prediction.
Van der Meer et al 2003197
Van der Meer et al. 2002240
Rotterdam Study

Rotterdam, Netherlands

Nested case-control study from population based cohort study

1990–1993

4–8 years
Age 55+ (mean 70)
N=657
61% men

No history of MI
Myocardial infarction

Age >75 included

Diabetes: medication or non-fasting glucose 200+mg/dl

Discrimination: area under ROC; sensitivity and specificity
DiscriminationInclusion of CRP in Framingham risk function did not improve discrimination
AUROC (SE)0.746 (0.021)
AUROC (SE) in age <75 years0.752 (0.025)
Upper 20% of risk distribution
Specificity86.8
Sensitivity38.9
Upper 10% of risk distribution
Specificity94.4%
Sensitivity20.4%
Milne et al. 2003248
Milneet al 2003249
Fletcher-Challenge – University of Auckland Heart and Health Study

New Zealand.

Worksite (72%) and electoral rolls (28%)

1992–1993

5 year follow up
Age 35–74
N=6354
73% men

No self-reported previous cardiovascular events
Cardiovascular death or hospitalisation 5 year probability
HDL not measured in worksite group (72%) Values used for men 1.1 women 1.4
Non fasting
Diabetes: doctor diagnosis or treatment
Smoking: daily or quit <1 year

Calibration: predicted: observed

Discrimination: 2x2 contingency tables for true and false positives and negatives. Area under ROC. Sensitivity and specificity.
Calibration
Graphical representation shows reasonable agreement between observed and predicted CVD events in all but older women (70–74 years)

Discrimination
8.1% of men and 5.4% of women predicted to be at high risk of CVD (>15% in 5 years) but this accounted for c25% of events. Low risk (<5% in 5 years) also accounted for c25%.
59% of cohort were men aged under 55

5-year cardiovascular mortality rate was low (4.7/1000)

Only modest ability to predict which individuals will have an event
AUROC(95% CI)
Men0.74(0.73,0.75)
Women0.77(0.74,0.80)
At 15% 5 year riskSpecificity 90% (men ) and 90% (women)
Sensitivity 20% (men) and 27% (women)
At <10% 5 year risk specificity becomes poor
At >15% 5 year risk sensitivity is very poor
Brindle et al.242Renfrew Paisley Study

Scotland

1972–1976

General urban population

10 year follow up
Age 45–64
N=12304
46% men (overall in study)

Complete risk factor information

Non-manual occupational coding
Manual occupational coding
  1. CVD death
  2. CHD death
HDL not measured in the study
CalibrationUnder-estimation of risk was worse in people from manual social classes and in people from areas of high deprivation. Authors note that future assessment of risk should take into account socio-economic position.
P:O (95% CI)
CVD deathCHD death
Manual0.52 (0.48,0.56)0.56 (0.51,0.63)
Non-manual0.69 (0.60,0.81)0.78 (0.66,0.92)
Discrimination
AUROC for CVD outcome only
Manual0.720
Non-manual0.744
WOSCOPS 1998267WOSCOPS
West Scotland

1989–1995

RCT of pravastatin lipid lowering medication

4.4 year follow up
Age 45–64
N=1251 placebo
N=1803
pravastatin 100% men

Moderately hypercholesterolaemic.

No history of vascular disease, MI or coronary event, angiography within 6 months of randomisation into trial, cancer. LDL >6.0 also excluded.
CVD (fatal MI, other cardiovascular death, definite or suspected non-fatal MI, CABG or PTCA)

Serum cholesterol was the mean of two measurements
Calibration“Remarkable agreement between observed CHD event rate and predicted in placebo group”

Graphical representation of observed risk at levels of predicted risk suggest predicted risk is much higher than observed risk at lower levels of risk in both pravastatin and placebo group.
PlaceboP:O 1.09p=0.58 (z-score test)
PravastatinP:O 1.27p=0.026 (z-score test)
Graphical representation of observed risk at levels of predicted risk

Framingham method predicted relative risk reduction of 24% associated with pravastatin. Observed relative risk reduction was 35%
Brindle et al. 2003243British Regional Heart Study

24 representative samples from UK towns.
Selected to represent the range of CVD mortality in UK. Random selection from 1 general practice in each town

1978–1980

10 year follow up
Age 40–59
N=6643
Men

Free of CVD with complete risk factor information
CHD death and fatal/non fatal CHD equations
Changes to methods: Diabetes-recall of doctor diagnosis
LVH-Minnesota coded
Cholesterol-Lieberman-Buchard method
HDL cholesterol-mg/phosphotungstate precipitation
No ECG
No unrecognised MI group

Calibration: predicted: observed (P:O)

Discrimination: sensitivity and specificity at 2 levels of risk
CalibrationReasonably consistent at ranking individual according to relative risk

Framingham methods over-estimate coronary risk in representative British male population. This could be reduced by simple adjustment

Use of 30% CHD 10 year risk threshold fails to identify most men who go on to have a CHD event
P:OCHD deathfatal/non fatal CHD
All areas1.471.57
S.England1.651.71
Midlands & Wales2.011.71
N.England1.271.54
Scotland1.501.28
47% over-prediction of CHD mortality risk (p<0.0001) was similar at all risk levels. The over-prediction of absolute risk was greatest for people at highest risk

Discrimination
30% fatal/non-fatal CHD 10 year risk threshold:
sensitivity 16%; specificity 94%
15% fatal/non-fatal CHD 10 year risk threshold:
sensitivity 75%; specificity 55%
Area under ROC curve 0.73

Recalibration: calculated score for each individual divided by the amount of over prediction. Predicted risk became close to observed risk at all levels of risk. For CHD death and CHD events χ2 for goodness of fit decreased from 30.2 to 3.4 and 155.3 to 24.6 respectively
30% fatal/non-fatal CHD 10 year risk threshold:
sensitivity 1.8%; specificity 99.6%
15 fatal/non-fatal CHD 10 year risk threshold:
sensitivity 37%; specificity 85%
Chambers et al. 2001245West London, UK

Cross-sectional
GP lists

Not prospective

1995–1998
Age 35–60
N=1025
100% men
49% white European
51% Indian Asian
CHDCompared estimates of risk with reported CHD mortality rates. Framingham risk score underestimated CHD mortality by approximately 50% in Indian Asians. Reasonable estimate of CHD mortality rate in European whites.
Guzder et al. 2002255Poole Diabetes Centre

Poole, UK

Community based cohort of newly diagnosed diabetics

1996–1998

Median follow up 4.2 years (SD 0.62)
Mean age 59 (SD 0.5)
N=428
57% men

Newly diagnosed diabetics
CVD

Calibration
P:O
Calibration

P:O=0.67
McEwan et al. 2004256Cardiff Diabetes Database

Cardiff, UK.
Diabetes register.
Population study demographic characteristics are believed to reflect those of UK diabetics.

1996

4 year follow up
Age 30–74
N=938
58% men

Type 1 or 2 diabetic patients with full data-set
CHD

Calibration: mean 10-year predicted probability.

Discrimination: graphical representation of proportion of cases developing/ not developing at different levels of risk. Specificity and sensitivity. AUROC

Recalibration: quasi-Newton optimisation algorithm
Calibration
There was considerable overlap in the predicted probabilities between those who subsequently did and did not develop CHD

Discrimination
Discriminatory power poor when applied to this population
Mean 10 year predicted probability of CHD (SD)
Discriminatory power of Framingham risk equation poor in this patient population

Use of imputed HDL cholesterol proved unreliable.
CHDNo CHD
Men0.27 (0.10)0.23 (0.10)
Women0.23 (0.09)0.19 (0.09)
AUROC
Men0.64
Women0.66
Recalibration
Only minor improvements using coefficients derived from Cardiff diabetes database (AUROC men 0.65, women 0.68)
Ramachandran et al. 2000247Whickham NE England, UK

Prospective heart disease study

1972–1974

20 year follow up
Age 30–75
N=1700
44% male
100% white

Free from heart disease. Ex-smokers excluded
CHD

20 year follow up

HDL not measured. Values of 1.15mmol/l used fro men, 1.4mmol/l for women

Calibration: graphical representation of predicted and observed risk by risk below or above 1.5%/ year
Mean 20 year risk in people developing heart disease with those not.

Discrimination: consideration of confidence intervals
Calibration
P:O good agreement at a predicted event rate >1.5%/ year – no significant difference between observed and expected CHD event rates (p=0.85). At lower event rates the predictive model significantly underestimates the number of observed events (p<0.01)
Framingham model reliably predicts the absolute risk of heart disease in white men and women in the UK when the annual risk is above 1.5%. The Framingham model underestimates the risk when the absolute risk is lower.
Heart diseaseNo heart disease
Mean 20 year risk (CI)30.5% (29.2,31.8)20.5% (19.7,21.4)
Discrimination
Wide confidence intervals in predicted: observed graph indicate that there is significant overlap between risk scores in those participants who developed heart disease and those who did not
Orford et al. 2002262Normative Aging Study

Boston, USA.
Community
screening-volunteers

1961

10 year follow up
Age 30–74
N=1393
Men
c 98% white


Normo-tensive with no known chronic medical conditions. No diabetes
CHD

Calibration: predicted and observed

Discrimination: Area under ROC
Calibration
European model overestimated incidence of total CHD in highrisk group (expected 20–40%, observed 20%)

Discrimination
AUROC c-statistic 0.58 (5 risk categories)
Accuracy of absolute risk prediction, particularly at the extremes of risk, is imperfect.

In an analysis of the same cohort but with 2280 men published in abstract form (Orford et al. 2000261) the authors note that the ESC model overestimated the incidence of CHD in the high risk group
Bastuji-Garin et al. 2002142
Bastuji Garin et al. 2001268
Western Europe and Israel

INSIGHT RCT participants randomised to nifedipine or coamiloride

1994–1996

Follow up at least
3 years
Age 55–74
N=4147
c45% male

Hypertensive men and women without previous cardiovascular events. Also 1 or more risk factor: smoking, hypercholesterolaemia, diabetes mellitus, LVH, LV strain on ECG, family history of MI
CVD
CHD
Stroke

3.7 years prediction
LVH set to absent

Calibration:
predicted: observed

Discrimination: area under ROC
Calibration
P:O in 6 countries/ regions (range)
Framingham CHD Anderson 1991 equation over-estimated CVD and CHD risk but not stroke risk. Large differences between countries, greater over-estimation in countries with low incidence. Precludes use of a single European threshold to define patients at high risk. However “models may be used within each country, provided that cut-off points defining high risk patients have been determined within each country
CVD2.6 (2.0–4.0)
CHD2.3 (2.0–7.0)
Stroke1.0 (0.3–2.0)
Discrimination
CHD AUROC 0.66 (0.2)

From: Appendix 6, Studies of external validation of risk-scoring methods (part complete)

Cover of A Systematic Review of Risk Scoring Methods and Clinical Decision Aids Used in the Primary Prevention of Coronary Heart Disease (Supplement)
A Systematic Review of Risk Scoring Methods and Clinical Decision Aids Used in the Primary Prevention of Coronary Heart Disease (Supplement) [Internet].
NICE Clinical Guidelines, No. 67S.
Beswick AD, Brindle P, Fahey T, et al.
Copyright © 2008, Royal College of General Practitioners.

PubMed Health. A service of the National Library of Medicine, National Institutes of Health.