• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of bmjBMJ helping doctors make better decisionsSearch bmj.comLatest content
BMJ. Feb 23, 2002; 324(7335): 459–464.
PMCID: PMC65668

Comparison of estimates and calculations of risk of coronary heart disease by doctors and nurses using different calculation tools in general practice: cross sectional study

R J McManus, clinical research fellow,a J Mant, senior lecturer,a C F M Meulendijks, research student,b R A Salter, H M Pattison, senior lecturer,a A K Roalfe, medical statistician,a and F D R Hobbs, professor and head of departmenta on behalf of the Midlands Research Practice Consortium

Abstract

Objective

To assess the effect of using different risk calculation tools on how general practitioners and practice nurses evaluate the risk of coronary heart disease with clinical data routinely available in patients' records.

Design

Subjective estimates of the risk of coronary heart disease and results of four different methods of calculation of risk were compared with each other and a reference standard that had been calculated with the Framingham equation; calculations were based on a sample of patients' records, randomly selected from groups at risk of coronary heart disease.

Setting

General practices in central England.

Participants

18 general practitioners and 18 practice nurses.

Main outcome measures

Agreement of results of risk estimation and risk calculation with reference calculation; agreement of general practitioners with practice nurses; sensitivity and specificity of the different methods of risk calculation to detect patients at high or low risk of coronary heart disease.

Results

Only a minority of patients' records contained all of the risk factors required for the formal calculation of the risk of coronary heart disease (concentrations of high density lipoprotein (HDL) cholesterol were present in only 21%). Agreement of risk calculations with the reference standard was moderate (κ=0.33-0.65 for practice nurses and 0.33 to 0.65 for general practitioners, depending on calculation tool), showing a trend for underestimation of risk. Moderate agreement was seen between the risks calculated by general practitioners and practice nurses for the same patients (κ=0.47 to 0.58). The British charts gave the most sensitive results for risk of coronary heart disease (practice nurses 79%, general practitioners 80%), and it also gave the most specific results for practice nurses (100%), whereas the Sheffield table was the most specific method for general practitioners (89%).

Conclusions

Routine calculation of the risk of coronary heart disease in primary care is hampered by poor availability of data on risk factors. General practitioners and practice nurses are able to evaluate the risk of coronary heart disease with only moderate accuracy. Data about risk factors need to be collected systematically, to allow the use of the most appropriate calculation tools.

What is already known on this topic

Recent guidelines have recommended determining the risk of coronary heart disease for targeting patients at high risk for primary prevention

Estimates of risk have been shown to be inaccurate

General practitioners and practice nurses can use risk calculation tools accurately when given patient data in the form of scenarios

What this study adds

Many patients do not have adequate information in their records to allow the risk of coronary heart disease to be calculated

When data about risk factors were available, risk calculations made by general practitioners and practice nurses were moderately accurate compared to a reference calculation

When adequate information about risk factors is not available, subjective estimates are a reasonable alternative to calculating risk

Introduction

Cardiovascular disease is the most important cause of death in the world today,1 with a lifetime risk at age 40 of 1 in 2 for men and 1 in 3 for women in developed countries.2 In the United Kingdom, 66 000 people aged under 75 die from cardiovascular disease each year.3 The British government has set a target of a reduction in cardiovascular disease in this age group of at least two fifths by 2010. Prevention on this scale requires major action to reduce the adverse consequences of risk factors for cardiovascular disease.4,5 Specific interventions will need to be used for hypercholesterolaemia,6,7 hypertension,8,9 smoking,10 and diabetes11,12 (primary prevention strategies), and people already suffering from cardiovascular disease will need to be treated (secondary prevention).1315

People with the highest risk of coronary heart disease stand to gain the most, in absolute terms, from interventions, and effective methods to identify such people are required. To date, most emphasis has been placed on secondary prevention; those most likely to benefit from secondary prevention are identified relatively easily on the basis of a history of cardiovascular disease. Identification of patients suitable for primary prevention is less straightforward, and recent guidelines have recommended the use of formal risk scoring with an algorithm based on the Framingham equation to identify high risk patients.1619 The recent national service framework for coronary heart disease in the United Kingdom will eventually require primary prevention of coronary heart disease on the basis of risk calculations for coronary heart disease.20 Although the Framingham equation has been shown to predict events when it was applied retrospectively to placebo populations recruited to statin trials,21 evidence for the use of risk calculations in primary care is conflicting.

General practitioners and practice nurses are able to use risk tables accurately when given patient data in the form of scenarios.22 However, accuracy was only moderate in a study that compared risks calculated in daily practice by general practitioners and practice nurses using the New Zealand risk tables with a reference calculation of risk.23

An alternative to calculation is for clinicians to use their knowledge of the patient to estimate the risk subjectively. This method is quick and does not need a full set of risk factors to provide an answer—but studies that evaluated this approach showed disappointing results. A Canadian study found that doctors accurately estimated the relative risk for hypothetical patients, but that they systematically overestimated the absolute risk of developing coronary heart disease.24 Another study, in the United Kingdom, compared subjective estimates of risk with calculations made using the Framingham equation and found that general practitioners consistently underestimated the risk.25

We are not aware of any study that directly compared subjective estimates of risk with formal calculations on the basis of data available routinely in patients' records. This study aimed to assess the accuracy with which general practitioners and practice nurses apportion the risk of coronary heart disease, in a quasi-routine clinical setting, by estimating and calculating, with four different calculation tools, the risk of coronary heart disease, by using a randomly selected sample of patient records, quota sampled for varied risk factors for cardiovascular disease.

Methods

We recruited 18 practices from the Midlands Research Practices Consortium—a research network that comprises more than 70 general practices scattered across the West Midlands. The network's overall practice populations are generalisable, in terms of socioeconomic gradient and type of location, to England and Wales.

Each practice performed computerised searches to identify patients in five groups at risk of coronary heart disease: those with hypertension, type 1 or 2 diabetes or hypercholesterolaemia; those who were current smokers; and those with current coronary heart disease (defined as history of angina, previous myocardial infarction, or previous coronary artery bypass or coronary angioplasty). We asked practices to use their own disease registers in the searches, because this represents the way that groups at risk of coronary heart disease would be identified by risk scoring in routine practice. Once the five groups had been identified, we randomly selected 10 sets of notes by choosing two patients from each group. Quota sampling was used to ensure that the sample contained no fewer than three women and no fewer than three patients over 65 years of age.

Four different risk calculation tools, representing the most widely used tools and a spectrum of different formats (risk chart or electronic calculator), were selected for comparison. The tools were the Sheffield table (a paper chart),26 the New Zealand hypertension guideline risk table (a paper chart),19 the joint European societies' recommendations on prevention of coronary heart disease (a wall chart),27 and the joint British societies' recommendations on the prevention of coronary heart disease in clinical practice (a computer program, which is also available as a paper chart).28 Each tool uses slightly different data to calculate risk (see table table11).

Table 1
Number of records for 110 patients without a history of cardiovascular disease in which data on risk factors needed were available*

In each practice, a general practitioner and a practice nurse estimated and calculated risks separately and blinded to each other in a dedicated non-clinical session (to approximate to an audit session in the practice). The practitioners were asked first to subjectively estimate the coronary heart disease risk for the 10 chosen patients, using only information in the patients' notes and computer records. They were then asked to calculate the risk, using the four different calculation tools, from the information available in the patient's records. To prevent an ordering effect, each practice used the calculation tools in a different predetermined sequence. Instructions about how to use the calculation tools were as given in the first published descriptions of the tools.19,2628 For each patient, the general practitioner or practice nurse recorded the risk calculation (or explained why they were not able to calculate it).

Analysis

The risks obtained with each of the tools were divided into high or low risk. The recommendations in the national service framework for coronary heart disease define high risk as a >30% risk of coronary heart disease at 10 years, and they say that interventions, such as statins, should be used in primary prevention for patients with risks higher than this. The nearest equivalent to this level provided by each table was chosen as the cut-off value: >3% annual risk of coronary heart disease for the Sheffield table, >20% risk of cardiovascular disease at five years for the New Zealand table (note the apparently higher risk band used because cardiovascular risk rather than coronary heart disease risk was calculated), >20% risk of coronary heart disease at 10 years for the European chart, and >30% risk of coronary heart disease at 10 years for the British program.

All estimates and calculations were compared with one prospectively derived “reference standard” score of risk, which was calculated for each patient by using a computerised version of the original Framingham equation16 and data independently collected from the patients' records by researchers. Where data—for example, concentrations of total cholesterol or high density lipoprotein (HDL) cholesterol—were missing, age and sex adjusted national averages were used in the reference calculation.29 In line with current British recommendations, the reference standards were classified as high risk if the risk of coronary heart disease at 10 years was greater than 30% or if the patient had pre-existing cardiovascular disease.20

We performed analyses with SPSS for Windows (version 10). We compared estimates and calculations of risk with reference calculations by using the K statistic to test agreement above chance,30 and estimates made by general practitioners and practice nurses for the same patient were also compared. We separately examined risks calculated for patients with pre-existing cardiovascular disease to find out in how many scores for the risk of coronary heart disease had been calculated inappropriately. We determined the sensitivity and specificity of each method compared with the reference calculation.

Results

Availability of relevant data in routine patient records

The records of 70 patients contained evidence of cardiovascular disease; these patients could be classified as being at high risk of coronary heart disease without risk estimation or calculation. Risk calculation was appropriate for the remaining 110 patients—sex and age were documented in all cases. Blood pressure and smoking status were recorded for most patients, although cholesterol concentrations, particularly of HDL cholesterol, were present in fewer records (table (table1).1). The number of patients who could be classified from these data with each calculation tool varied from 92 (51%) for the British program and New Zealand table to 161 (89%) for the Sheffield table (table (table1).1).

Subjective risk estimation and formal risk scoring

Subjective estimates by practice nurses agreed with the reference calculation in 125 (69%, κ=0.38) cases and with general practitioner estimates in 131 (71%, κ=0.45) cases (table (table2).2). The number of formal (objective) risk calculations attempted and the proportion correct varied (table (table2).2). Nurses using the British program achieved the closest agreement with the reference calculation (κ=0.65), but attempted the fewest calculations with this tool (96, 53%). General practitioners achieved the closest agreement with the reference calculation when using the British program (κ=0.55), the New Zealand table (κ=0.53), and the Sheffield table (κ=0.51), but they attempted fewer calculations with the former (99 v 134 v 131). The greatest number of calculations was attempted with the European table (148, 82% by practice nurses and 149, 83% by general practitioners), but little agreement with the reference calculation was seen (κ=0.38 for practice nurses, 0.33 for general practitioners).

Table 2
Calculations of estimates of risk of coronary heart disease by practice nurses and general practitioners compared with reference standard.* Values are numbers (percentages) unless otherwise specified

Some risk calculations were completed for people for whom risk scoring was inappropriate—for example, patients with a history of coronary heart disease, who automatically are eligible for secondary prevention (table (table3).3). Partly because of this systematic error in scoring, the overall trend using objective risk scoring was towards an underestimation of risk (table (table2).2).

Table 3
Inappropriate classification of 70 patients with cardiovascular disease as at low risk for coronary heart disease

Repeatability of subjective estimates and formal risk calculations—comparing results from general practitioners and practice nurses for the same patient—was moderate (between κ=0.47 for estimates and κ=0.58 for scoring risk with the British program) (table (table4).4).

Table 4
Repeatability of calculations of risk of coronary heart disease by general practitioners and practice nurses. Values are numbers (percentages) unless otherwise specified

Sensitivity was lowest for practice nurses using the Sheffield table to calculate risk (58%) and highest for general practitioners using the British program (80%) (table (table5).5). Specificity was lowest for practice nurses using the New Zealand table (63%) and highest for practice nurses using the British program (100%).

Table 5
Sensitivity and specificity of results when healthcare professionals determined whether patients are at high risk of coronary heart disease.* Values are numbers (percentage, 95% confidence interval)

Discussion

In this study, general practitioners and practice nurses were asked to assess whether people were at high or low risk of coronary heart disease, consistent with the recommendations of the national service framework for coronary heart disease. The greatest overall accuracy was achieved when primary care clinicians used the British program to calculate coronary heart disease risk.

Limitations of formally calculating risk

An important limitation of using formal calculations to assess the risk of coronary heart disease in primary care is that information necessary for determining risk factors may not be available. This is a particular problem for the New Zealand table and British program, which use concentrations of HDL cholesterol (the notes of only a fifth of patients (23/110) contained this information when patients with existing coronary heart disease were excluded). In contrast, it was possible to calculate risk for most patients with the Sheffield and European tools.

A second problem is that the calculations may be used inappropriately—risk scores were calculated in up to 40% of patients with coronary heart disease, even though such patients can be classified as high risk because of their disease history alone. This tended to be less of a problem with the British program, perhaps because the instructions about not applying the program to such patients were clearer.

Accuracy of tools

When enough data are available, general practitioners and practice nurses can achieve reasonable accuracy with the different calculation tools. The British program was the tool that was used most reliably—it gave the best results in terms of sensitivity for general practitioners and practice nurses and specificity for practice nurses. With the possible exception of the British program, risk calculators were no more accurate than subjective assessment for determining the risk of coronary heart disease. This is likely to be because the calculators were used in inappropriate patients and because of the poor availability of data needed for calculations.

For the New Zealand table and the British program, risk was calculated for several patients in whom data about concentrations of HDL cholesterol were not available. Although it should only have been possible to use the table to calculate risk in 22 out of 110 patients without known coronary heart disease, nurses and general practitioners gave results for 78 patients and 64 patients, respectively. In most of these cases, the clinician seems to have assumed that the concentration of HDL cholesterol was 1 mmol/l.

Accuracy of tools in identifying patients at high or low risk

Assessing whether people are at high or low risk of coronary heart disease is the most clinically relevant way of using the risk calculation tools, because a dichotomous decision has to be made as to whether or not to treat each patient. Cut-off values closest to 30% risk at 10 years were chosen for each calculation tool; this will have resulted in some variation in the results from the different tools and the reference standard since, with the exception of the British program, the cut-off values were slightly different from this level. As expected, a higher threshold (such as the 3% per year risk used in the Sheffield table) gave a less sensitive but more specific calculation, but a lower threshold (such as the 20% 10 year risk used in the European chart) did not give the expected more sensitive and less specific assessments. It is not clear why; perhaps the choice of >20% 10 year risk as the cut-off value for the European chart (necessary because of the categories used in that chart) may have led to its relatively poor performance.

Accuracy of estimates and calculations

Past studies have not compared the results of subjective estimates with calculations in the same patients. Grover et al asked 253 general practitioners attending an educational meeting to determine risk for four theoretical patients.24 The general practitioners systematically overestimated the absolute baseline risk of developing coronary disease and the risk reductions associated with specific interventions. Montgomery et al compared absolute risk estimations for 397 patients with hypertension with risk scores calculated with the Framingham equation.25 Estimates were correct in 21% of patients; 63% of risks were underestimated and 16% overestimated. Neither study analysed the results in terms of the choices between high and low risk and whether to initiate formal preventive treatment or not, which is the most important decision for primary care.

Accurate use of tools

Few studies have looked at how accurately professionals use tables to calculate risk. In a Scottish study that used case histories to test how accurately general practitioners and practice nurses used the Sheffield and New Zealand tables and the British program, high and low risk thresholds were used, and enough data were provided to make all calculations possible; the table and program performed best in this study, producing calculations that were correct in most cases.22 Peters et al found only moderate agreement between the reference standard and risk scores calculated by 34 general practitioners and five practice nurses who used the New Zealand table for 228 hypertensive patients (weighted κ=0.56).23

Limitations of our study

The main limitations of our study are that the general practitioners and practice nurses performed the estimates and calculations in protected time outside of consultations, and that they were all drawn from practices with an interest in research. Results achieved within the time pressures of a consultation would be expected to be worse than those in our study.

The lack of information (especially about concentrations of HDL cholesterol) in the patient records meant that we had to use estimates to calculate the reference standard, as has been the case in past studies.23 Although such estimates may introduce some error, the reference standard will still represent the best evaluation of risk that is possible, given what is known about the patient.

Implications for clinical practice

The present study did not directly measure the accuracy with which general practitioners and practice nurses use a particular table. However, the results suggest that problems with extracting data from records and applying the tools only to appropriate patients may be at least as important as simple errors in calculations.

Adequate training is required to ensure that the risk calculation tools are used only for appropriate patients. The availability of data about risk factors needed for these tools, especially concentrations of total cholesterol and HDL cholesterol, needs to be improved—a point emphasised in the national service framework for coronary heart disease. In the absence of complete data, subjective estimates of risk seem to be a reasonable alternative.

Supplementary Material

[extra: More information on Consortium members]

Footnotes

Funding: This study was funded by the Midlands research practice consortium, which is supported by Budget 1 NHS research and development competitive funding.

Competing interests: None declared.

Further information on members of the Midlands Research Practice Consortium is available on bmj.com

References

1. Murray CJ, Lopez AD. Mortality by cause for eight regions of the world: global burden of disease study. Lancet. 1997;350:141–142. [PubMed]
2. Lloyd-Jones DM, Larson MG, Beiser A, Levy D. Lifetime risk of developing coronary heart disease. Lancet. 1999;353:89–92. [PubMed]
3. Department of Health. Our healthier nation. London: Stationery Office; 1998.
4. Kannel WB, Neaton JD, Wentworth D, Thomas HE, Stamler J, Hulley SB, et al. Overall and coronary heart disease mortality rates in relation to major risk factors in 325,348 men screened for the MRFIT. Am Heart J. 1986;112:825–836. [PubMed]
5. West of Scotland Coronary Prevention Study. Identification of high-risk groups and comparison with other cardiovascular intervention trials. Lancet. 1996;348:1339–1342. [PubMed]
6. Downs JR, Clearfield M, Weis S, Whitney E, Shapiro DR, Beere PA, et al. for the AFCAPS/TexCAPS Research Group. Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS JAMA 1998. 2791615–1622.1622 [PubMed]
7. Shepherd J, Cobbe SM, Ford I, Isles CG, Lorimer AR, MacFarlane PW, et al. Prevention of coronary heart disease with pravastatin in men with hypercholesterolemia. N Engl J Med. 1995;333:1301–1307. [PubMed]
8. Hansson L, Zanchetti A, Carruthers SG, Dahlof B, Elmfeldt D, Julius S, et al. Effects of intensive blood-pressure lowering and low-dose aspirin in patients with hypertension: principal results of the hypertension optimal treatment (HOT) randomised trial. Lancet. 1998;351:1755–1762. [PubMed]
9. MacMahon SW, Cutler JA, Neaton JD, Furberg CD, Cohen JD, Kuller LH, et al. Relationship of blood pressure to coronary and stroke morbidity and mortality in clinical trials and epidemiological studies. J Hypertens. 1986;4(suppl):S14–S17. [PubMed]
10. Prescott E, Hippe M, Schnohr P, Hein HO, Vestbo J. Smoking and the risk of myocardial infarction in women and men: longitudinal population study. BMJ. 1998;316:1043–1047. [PMC free article] [PubMed]
11. UK Prospective Diabetes Study Group. Tight blood pressure control and risk of macrovascular and microvascular complications in type 2 diabetes (UKPDS 38) BMJ. 1998;317:703–713. [PMC free article] [PubMed]
12. UK Prospective Diabetes Study (UKPDS) Group. Intensive blood-glucose control with sulphonylureas or insulin compared with conventional treatment and risk of complications in patients with type 2 diabetes. Lancet. 1998;352:837–885. [PubMed]
13. Scandinavian Simvastatin Survival Study Group. Randomised trial of cholesterol lowering in 4444 patients with coronary heart disease: the Scandinavian Simvastatin Survival Study (4S) Lancet. 1994;344:1383–1389. [PubMed]
14. Medical Research Council general practice research framework. Thrombosis prevention trial: randomised trial of low intensity oral anti-coagulation with warfarin and low–dose aspirin in the primary prevention of ischaemic heart disease in men at increased risk. Lancet. 1998;351:233–241. [PubMed]
15. Mehta RH, Eagle KA. Secondary prevention in acute myocardial infarction. BMJ. 1998;316:838–842. [PMC free article] [PubMed]
16. Anderson KM, Odell PM, Wilson PWF, Kannel WE. Cardiovascular disease risk profiles. Am Heart J. 1991;121:293–298. [PubMed]
17. Second Joint Task Force of European and Other Societies on Coronary Prevention. Prevention of coronary heart disease in clinical practice. Eur Heart J. 1998;19:1434–1503. [PubMed]
18. National Cholesterol Education Program. Second report of the expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (adult treatment panel II) Circulation. 1994;89:1329–1445.
19. National Health Committee. Guidelines for the management of mildly raised blood pressure in New Zealand. Wellington: Ministry of Health; 1998. www.nzgg.org.nz/library/gl_complete/bloodpressure/index.cfm#contents (accessed 5 Sept 2001).
20. Department of Health. National service framework for coronary heart disease. London: HMSO; 2000.
21. Wallis EJ, Ramsay LE, Ul Haq I, Ghahramani P, Jackson PR, Rowland-Yeo K, et al. Coronary and cardiovascular risk estimation for primary prevention: population validation of a new Sheffield table. BMJ. 2000;320:671–676. [PMC free article] [PubMed]
22. Isles CG, Ritchie LD, Murchie P, Norrie J. Risk assessment in primary prevention of coronary heart disease: randomised comparison of three scoring methods. BMJ. 2000;320:690–691. [PMC free article] [PubMed]
23. Peters TJ, Montgomery AA, Fahey T. How accurately do primary health care professionals use cardiovascular risk tables in the management of hypertension? Br J Gen Pract. 1999;49:987–988. [PMC free article] [PubMed]
24. Grover SA, Lowensteyn I, Esrey KL, Steinert Y, Joseph L, Abrahamowicz M. Do doctors accurately assess coronary risk in their patients? Preliminary results of the coronary health assessment study. BMJ. 1995;310:975–978. [PMC free article] [PubMed]
25. Montgomery AA, Fahey T, Mackintosh C, Sharp DJ, Peters TJ. Estimation of cardiovascular risk in hypertensive patients in primary care. Br J Gen Pract. 2000;50:127–128. [PMC free article] [PubMed]
26. Ramsay LE, Haq IU, Jackson PR, Yeo WW, Pickin DM, Payne JN. Targeting lipid-lowering drug therapy for primary prevention of coronary disease: an updated Sheffield table. Lancet. 1996;348:387–388. [PubMed]
27. Wood D. European and American recommendations for coronary heart disease prevention. Eur Heart J. 1998;19 (Suppl A):A12–A19. [PubMed]
28. British Cardiac Society; British Hyperlipidaemia Association; British Hypertension Society; endorsed by the British Diabetic Association. Joint British recommendations on prevention of coronary heart disease in clinical practice. Heart. 1998;80(suppl 2):1–29. http://heart.bmjjournals.com/cgi/content/full/80/suppl_2/S1 (accessed 21 Dec 2001). [PMC free article] [PubMed]
29. Erens R, Primatesta P. Health survey for England 1998. London: Stationery Office; 1999. pp. 1–368.
30. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed]

Articles from BMJ : British Medical Journal are provided here courtesy of BMJ Group
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • MedGen
    MedGen
    Related information in MedGen
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...