Logo of brjgenpracRCGP homepageJ R Coll Gen Pract at PubMed CentralBJGP at RCGPBJGP at RCGP
Br J Gen Pract. Jul 1, 2005; 55(516): 529–533.
PMCID: PMC1472770

Case-mix and variation in specialist referrals in general practice

Caoimhe O Sullivan, MA, Medical Statistician
R&D Directorate, University College London Hospital NHS Trust
Rumana Z Omar, PhD, Senior Lecturer and Gareth Ambler, PhD, Lecturer
Department of Statistical Science, University College London
Azeem Majeed, MD, FRCGP, FFPH, Professor of Primary Care



The potential of a comprehensive measure of patient morbidity to explain variation in referrals to secondary care has not previously been examined in the UK.


To examine the relative role of age, sex and morbidity as defined by the Johns Hopkins ACG Case-Mix System in explaining variations in specialist referrals in general practice.

Design of study

Retrospective study of a cohort of patients followed for 1 year.


Two hundred and two general practices, with a total list size of 1 161 892, contributing data to the General Practice Research Database.


Each patient was assigned an ACG and morbidity group, based on their diagnoses, age and sex. The variability in referrals explained by these factors was examined using multilevel logistic regression models by splitting it into variation between practices and variation between patients within practices.


The annual median (range) percentage of patients referred was 14.8% (range = 2.4–24.4%). The percentage of patients referred increased with age and morbidity. Morbidity explained 30.4% of the total variation in referrals (composed of variability between and within practices). Age and sex only explained 5.3% of the total variation. The variation attributable to practices was approximately 5%, thus most of the variation occurred within practices. Morbidity was also identified as a better predictor of referral compared to age and sex.


Morbidity explains almost six times more of the variation in general practice referrals than age and sex, although about two-thirds of the variation remains unexplained. Most of the unexplained variation is due to differences within rather than between practices. The amount of variability in referrals between practices may be less than implied by previous studies based on aggregate information. The implications are that any investigation of specialist referrals from general practice should be interpreted cautiously, even after adjustment for age, sex and morbidity.

Keywords: case-mix, morbidity, referral and consultation


One of the key roles of GPs in the UK's NHS is to act as gatekeepers, with responsibility for referring patients to specialist services. In an ideal situation, GPs would refer those patients who would benefit most from specialist care, while retaining the management of other patients within primary care. The reality is that many studies have found large variations in general practice referral rates.1 These are often poorly explained by factors such as practice demography or the socioeconomic characteristics of the areas where practices are located.1 Variations in referrals have important implications for NHS spending and patients' access to specialist health services. Because the government is striving for significant improvements in waiting times to see specialists, GPs are finding their referral practices coming under increased monitoring.2

Monitoring practices' referral rates, or other aspects of practice activity, runs the risk that practices that look after less-well populations may be unfairly scrutinised or penalised for having ‘excessively’ high rates. Hence, understanding the relationship between patients' clinical characteristics and referrals is important. Despite this, few studies in the UK have examined this association.1,3,4 In contrast, studies in the US and Canada have used case-mix systems to adjust for differences in patients' clinical characteristics when investigating primary care referrals.5,6

How this fits in

Referrals from general practice vary widely and are coming under increasing scrutiny, both in the UK and elsewhere. Previous studies have shown that age and sex explain relatively little variation in referrals. The potential of diagnostic-based morbidity measures to explain variation in general practice referrals has not been explored in the UK. This paper demonstrates that diagnostic-based morbidity measures explain more of the variation in referrals than age and sex alone. Therefore, diagnostic-based case-mix measurement may have a role in producing adjusted measures of practice performance, resource utilisation, and practice mortality. Referrals that have not been adjusted for case-mix should be interpreted cautiously. Even after adjustment for age, sex and morbidity, the bulk of the variation in referrals remains unexplained.

The aim of this study was to use the Johns Hopkins Adjusted Clinical Group (ACG) Case-Mix System (http://www.acg.jhsph.edu) to investigate how well patient-level measures of case-mix explain the variability in referrals and predict referrals by general practices in the UK. We chose to use the ACG system because it is the only case-mix system specifically designed for use in primary care, and has been widely developed and validated.7,8 It was developed at Johns Hopkins University, Baltimore, on the basis of a body of research that found that clustering of morbidity is a better predictor of resource use than individual diseases.7,8 The ACG system assigns all international classification of diseases, ninth revision, clinical modification (ICD-9-CM) codes to one of 32 diagnosis groups, known as Aggregated Diagnosis Groups (ADGs). Each diagnosis can be placed in an ADG based on five clinical criteria: duration; severity; diagnostic certainty; aetiology; and specialty care.9-11 Each patient can be assigned to one of about 100 mutually exclusive ACGs depending on their combination of diagnoses, age and sex, with the result that individuals within a given ACG will have experienced a similar pattern of morbidity over a given year (http://www.acg.jhsph.edu).9


We used data from the General Practice Research Database (GPRD) on 1 323 611 patients registered in 211 general practices across England and Wales in 1997.12 The database records include demographic information, disease incidence and medical, surgical and psychiatric referrals for each patient. Several studies have assessed the validity of these data.13,14

General practices contributing data to the GPRD follow guidelines for the recording of administrative, diagnostic, and referral data. Diagnoses were recorded using OXMIS or Read codes, and these were later converted to ICD-9-CM codes. Each patient was assigned to an ACG group, based on their combination of diagnoses, age, and sex.9 The ACGs were ranked according to their morbidity burden in the US and then grouped into six mutually exclusive categories, each containing roughly the same number of ACGs.4 We refer to these categories of ACGs as morbidity groups, ranging from group 1 (healthiest) to group 6 (least healthy). Age was divided into four groups representing children, young adults, older adults and the elderly (0–15, 16–34, 35–64 and ≥65 years respectively).

Patients were excluded from analysis if they were registered with a practice for less than 180 days. Practices were excluded if less than 2% of their registered patients were referred or if their deprivation code was missing. Since only a small percentage of patients were referred more than once, we combined patients with one or more referrals in a single category and treated referral as a binary outcome.

Statistical methods

Multilevel logistic regression models with random intercepts were used, and these take into account the clustered nature of the data (patients within practices).15 Four models with different sets of predictors were considered:

  • Model 1: no predictors;
  • Model 2: age and sex;
  • Model 3: morbidity;
  • Model 4: age, sex and morbidity.

These models allow us to split the variation in referrals into two of its components: variation between practices and variation between patients within practices. The percentage of total variation in referrals explained by each of the models was estimated using an R2 measure developed for multilevel models by Snijders and Bosker.16 We made graphical comparisons of the agreement between the observed referrals and those predicted by the models for each practice. Receiver Operating Characteristic (ROC) areas were calculated to assess how well the models were able to discriminate between those patients who were referred and those who were not.17 A value of 1 for the ROC area indicates that the associated model discriminates perfectly between patients with high and low referrals, while an area of 0.5 indicates that the model discriminates no better than chance. MlwiN v1.10 software was used for multilevel modelling;18 Stata 8.0 was used for all other analyses.


After exclusions, the data consisted of 1 161 892 registered patients from 202 practices. The median practice size was 5055 (range = 1364–14587) and the median percentage of patients referred by practice was 14.8% (range = 2.4–24.4%). Overall, 14.7% of patients were referred to a specialist at least once during the study period. Twelve per cent of patients had only one referral, while 2% had two referrals and only 0.4% had three or more referrals. Table 1 summarises the distribution of patients and referrals across the 202 practices for each age, sex and morbidity group. The overall percentage of referrals increases steadily by age group, with 7.5% of patients aged 0–15 years referred compared to 21.1% of patients in the ≥65 age group, and was higher for females (17.1% against 12.2%). There is an even stronger association between referrals and morbidity group. Table 2 shows the odds ratios for referral for Models 2 to 4. The odds of referral increase significantly with increasing age (Model 2), and they increase more strongly with increasing morbidity (Model 3). The odds of referral for the least healthy patients relative to the healthiest patients is 36.2 (95% CI = 35.3 to 37.1). The results for morbidity in Model 4 were similar to those for Model 3, but age was no longer statistically significant. We therefore focused on comparisons between Models 2 and 3.

Table 1
Patient characteristics.a
Table 2
Model odds ratios and percentage of variation explained.a

Patient morbidity explained considerably more of the total variability in referrals than patients' age and sex alone (30.4% against 5.3%, Table 2). The variation at the practice level was estimated to be approximately 5% of the total variation in referrals (Table 2, Model 1). Of the total variation in referrals, 3.6% remained unexplained at the practice level and 66.1% remained unexplained at the patient level after adjusting for morbidity. The corresponding figures are 4.5% and 90.2% when adjusting for age and sex.

Figures 1 and and22 compare the predicted and observed percentages referred by practice. If the model predictions are accurate, the points will lie close to the line of equality. The range of predicted practice referrals is narrow for Model 2 (11.6–15.2%) (Figure 1). Model 3 (Figure 2) is a marked improvement on Model 2, with predicted referrals ranging from 6.2–18.8%, closer to those of the observed range. However, even after adjusting for morbidity, there remained substantial departures from the line of equality. The area under the ROC curve was 0.616 (95% CI = 0.614 to 0.617) for Model 2 and 0.768 (95% CI = 0.766 to 0.769) for Model 3. This suggests that the morbidity model discriminates significantly better between patients who were referred and those who were not than the model with age and sex alone.

Figure 1
Observed versus predicted referrals by practice for Model 2.
Figure 2
Observed versus predicted referrals by practice for Model 3.


Summary of main findings

Even after excluding practices with very low referrals (less than 2% of patients referred annually), there was still a wide range in patients referred by practice (2.4–24.4%). Patient morbidity explains considerably more of the total variability in referrals than patient age and sex. We found that about 5% of the total variation in referrals was occurring at the practice level, and this supports research suggesting that the amount of variability between practices may be less than implied by previous studies based on aggregate information.19 However, it may not be straightforward to understand the significance of the size of variability with respect to decisions on policy-related issues.19 Furthermore, the findings suggest that there is considerable variation in referrals between patients within practices; and that morbidity explains substantially more of this (around 24% more) than age and sex. There are often many factors other than the diagnostic label that influence whether the patient is referred or not, and this could explain why a large proportion of the variability remains unexplained.1 We also found that morbidity was a better predictor of practice referrals than age and sex. Thus, morbidity holds promise as a targeting tool to predict which patient groups are most likely to require referrals.

Strengths and limitations of this study

This was a large primary care-based study that used data from practices contributing to the GPRD, which has been extensively validated and shown to be of high quality.12,13 The GPRD practices are reasonably representative of the age–sex profile of the UK population, and their referral rates are similar to national averages.19 However, there is some under-representation of inner-city practices and the average size of the practices is greater than the national average.19 The database provided us with individual level data, thus avoiding the ‘ecological fallacy’ (that is, associations at the practice level may not hold at the individual level), and we incorporated the clustering of patients in practices in our statistical models. This was the first large-scale study in the UK to control for diagnostic-based morbidity groupings specifically designed for use in primary care when examining variation in specialist referrals in general practice. However, the ACG system was derived for use in the US, and so may need some further adaptation to maximise its utility in the UK.11 Finally, the system depends on diagnostic codes recorded by GPs during consultations and hence, differences in the way that GPs record similar conditions on their practice computers could introduce bias.

Comparison with existing literature

Many previous studies have reported wide variations in general practice referral rates. Studies that have adjusted for age and sex have typically found that the observed variation decreased by less than 10%, in line with our own findings.19 This could be improved upon by adjusting for other factors, such as socioeconomic deprivation or practice characteristics, but most of the variation in referrals still remains unaccounted for.1 The potential of a morbidity measure based on each patient's medical history to explain general practice referrals has not previously been investigated in the UK. Studies from the US suggest that it is better than age and sex in explaining variation in referrals, again in line with our own findings.6

Implications for general practice

Variation in general practice referrals is an important issue for clinicians, managers, patients and politicians. Age and sex seem to explain little of this. The Johns Hopkins ACG Case-Mix System explains substantially more variation, although most remains unexplained. More research is needed to assess the cost-effectiveness of implementing and administrating such a system.

The unexplained variation seems to occur largely within practices, so that the GP decision of whether to refer a patient varies, even for patients with a similar age, sex and morbidity group. This may be partly due to patient factors that cannot be easily measured, such as the perceived patient pressure or their level of confidence in their GP.21,22 The GP's perception of medical need, experience in treating the condition for which the patient was referred and the local provision of specialist services may also be important.21,22 Understanding other reasons for variation in referrals could help to ensure more efficient use of NHS resources.

The findings suggest that any investigation of how practices are using specialist services should be interpreted cautiously even after adjustment for age and sex. In contrast, measures such as diagnostic-based morbidity are likely to give more useful information, but caution should be exercised even after adjusting for these, as much of the variability in use of services is likely to remain unexplained. Similar caveats are likely to apply to other areas of primary care, such as prescribing costs, mortality rates and the quality measures in the new GP contract. The association of these measures with case-mix should be explored to ensure that practices that seem to have a high intensity of resource use, that have high death rates, or that do not achieve quality targets are not inappropriately scrutinised or penalised for this.


We thank Kevin Carroll for his help in extracting the data and assigning the ACGs for this study.


Funding body

Caoimhe O Sullivan is funded by a Department of Health Researcher Development Award (RDA01/019). Azeem Majeed is funded by a Department of Health Career Scientist Award (CSA/2000/007)

Ethics committee

Approval from the GPRD Scientific and Ethical Committee (304)

Competing interests



1. O'Donnell CA. Variation in GP referral rates: what can we learn from the literature? Fam Pract. 2000;17:462–471. [PubMed]
2. Rowlands G, Graham L, Keene D, et al. PCT/primary care referrals: current and future trends. BJHCM. 2004;10(4):110–115.
3. Hippisley-Cox J, Hardy C, Pringle M, et al. The effect of deprivation on variations in general practitioners' referral rates: a cross sectional study of computerised data on new medical and surgical outpatient referrals in Nottinghamshire. BMJ. 1997;314:1458–1461. [PMC free article] [PubMed]
4. Forrest C, Majeed A, Weiner JP, et al. Primary care physicians' specialty referral rates in the United Kingdom versus the United States. BMJ. 2002;325:370–371. [PMC free article] [PubMed]
5. Franks P, Zwanziger J, Mooney C, Sorbero M. Variations in primary care physician referral rates. Health Serv Res. 1999;34:323–329. [PMC free article] [PubMed]
6. Salem-Schatz S, Moore G, Rucker M, Pearson SD. The case for case-mix adjustment in practice profiling. When good apples look bad. JAMA. 1994;272(11):871–874. [PubMed]
7. Starfield B, Weiner J, Mumford L, Steinwachs D. Ambulatory care groups: a categorization of diagnoses for research and management. Health Serv Res. 1991;26(1):53–74. [PMC free article] [PubMed]
8. Weiner JP, Starfield BH, Steinwachs DM, Mumford LM. Development and application of a population-oriented measure of ambulatory care case-mix. Med Care. 1991;29:452–472. [PubMed]
9. Weiner JP, Abrams C. ACG virtual library: version 4.5 (February 2000). ACG software documentation/users manual. Baltimore: Health Services Research & Development Centre, The Johns Hopkins University; 1991–2001.
10. Majeed A, Bindman AB, Weiner JP. Use of risk adjustment in setting budgets and measuring performance in primary care I: how it works. BMJ. 2001;323:604–607. [PMC free article] [PubMed]
11. Majeed A, Bindman AB, Weiner JP. Use of risk adjustment in setting budgets and measuring performance in primary care II: advantages, disadvantages, and practicalities. BMJ. 2001;323:607–610. [PMC free article] [PubMed]
12. Lawson DH, Sherman V, Hollowell J. The General Practice Research Database. Scientific and Ethical Advisory Group. QJM. 1998;91:445–452. [PubMed]
13. Hollowell J. The General Practice Research Database: quality of morbidity data. Popul Trends. 1997;87:36–40. [PubMed]
14. Jick H, Jick SS, Derby LE. Validation of information recorded on general practitioner based computerised data resource in the United Kingdom. BMJ. 1991;302:766–768. [PMC free article] [PubMed]
15. Goldstein H. Multilevel statistical models. London: Edward Arnold; 1995.
16. Snijders TAB, Bosker RJ. Multilevel analysis: an introduction to basic and advanced multilevel modeling. Sage Publishers; 1999.
17. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29–36. [PubMed]
18. Rasbash J, Browne W, Goldstein H, et al. A user's guide to MlwiN version 2.1. London: Multilevel Models Project, Institute of Education, University of London; 2000.
19. Davis P, Gribben B, Lay-Yee R, Scott A. How much variation in clinical activity is there between general practitioners? A multi-level analysis of decision-making in primary care. Health Serv Policy. 2002;7(4):202–208. [PubMed]
20. Office for National Statistics. Key health statistics from general practice. Series MB6 No. 2. London: The Stationery Office; 1998.
21. Armstrong D, Fry J, Armstrong P. Doctors' perceptions of pressure from patients for referral. BMJ. 1991;302:1186–1188. [PMC free article] [PubMed]
22. Little P, Dorward M, Warner G, et al. Importance of patient pressure and perceived pressure and perceived medical need for investigations, referral, and prescribing in primary care: nested observational study. BMJ. 2004;328:444–448. [PMC free article] [PubMed]

Articles from The British Journal of General Practice are provided here courtesy of Royal College of General Practitioners
PubReader format: click here to try


Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


  • PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...