Logo of brjgenpracRCGP homepageJ R Coll Gen Pract at PubMed CentralBJGP at RCGPBJGP at RCGP
Br J Gen Pract. Jan 1, 2009; 59(558): e16–e24.
PMCID: PMC2605545

Risk taking in general practice: GP out-of-hours referrals to hospital

Jenny C Ingram, BSc, PhD, Research Fellow
RDS-SW(Bristol), Education Centre, University Hospitals Bristol NHS Foundation Trust, Bristol
Michael W Calnan, MSc, PhD, Professor
School of Social Policy, Sociology and Social Research, University of Kent, Canterbury
Rosemary J Greenwood, MSc, Medical Statistician
RDS-SW(Bristol), Education Centre, University Hospitals Bristol NHS Foundation Trust, Bristol
Terry Kemple, FRCP, FRCGP, DCH, GP
Horfield Health Centre, Bristol
Sarah Payne, BSc, PhD, Reader
School for Policy Studies, University of Bristol, Bristol
Michael Rossdale, MA, DCH, MRCGP, GP



Emergency admissions to hospital at night and weekends are distressing for patients and disruptive for hospitals. Many of these admissions result from referrals from GP out-of-hours (OOH) providers.


To compare rates of referral to hospital for doctors working OOH before and after the new general medical services contract was introduced in Bristol in 2005; to explore the attitudes of GPs to referral to hospital OOH; and to develop an understanding of the factors that influence GPs when they refer patients to hospital.

Design of study

Cross-sectional comparison of admission rates; postal survey.


Three OOH providers in south-west England.


Referral rates were compared for 234 GPs working OOH, and questionnaires explored their attitudes to risk.


There was no change in referral rates after the change in contract or in the greater than fourfold variation between those with the lowest and highest referral rates found previously. Female GPs made fewer home visits and had a higher referral rate for patients seen at home. One-hundred and fifty GPs responded to the survey. Logistic regression of three combined survey risk items, sex, and place of visit showed that GPs with low ‘tolerance of risk’ scores were more likely to be high referrers to hospital (P<0.001).


GPs' threshold of risk is important for explaining variations in referral to hospital.

Keywords: emergency service, hospital; out-of-hours medical care; risk


Admissions to hospital are an increasing source of pressure on the NHS,1 and unplanned admissions represented 37% of hospital admissions in the UK in 2005–2006.2 Emergency admissions at night and weekends are distressing for patients and disruptive for hospitals, and many of these admissions result from referrals from GP out-of-hours (OOH) providers. Research has shown that there is a greater than fourfold variation between the top and bottom quartiles in OOH admission rates between GPs working for the same OOH service and caring for the same patient population,3 suggesting that doctors' decision making varies and plays an important part in determining admission rates. Qualitative research with the same group of OOH GPs suggests this may be due to lack of confidence, aversion to risk, feelings of isolation, and poor knowledge of alternatives to admission. The results suggest that interventions designed to address these problems may have a considerable impact on emergency admission rates.4

In 2005, the new GP contract allowed GPs to opt out of OOH care in the evenings, nights, and at weekends, and also made local primary care trusts (rather than GPs) responsible for the provision of OOH services. GPs are also able to choose their shifts and, consequently, where they work (primary care centre or making home visits).

Following the introduction of the new general medical services (GMS) contract in Bristol in January 2005, the aims of the current study were to compare the referral rates before and after the contract change; to investigate the attitudes of GPs to referral to hospital OOH; and to develop an understanding of the factors that influence GPs when they refer patients to hospital. It also aimed to develop an attitude-assessment tool to identify doctors with low and high referral rates to hospital which could be used in GP training sessions.


Using statements derived from major themes from the qualitative study,4 a questionnaire was designed, piloted with 10 GPs in a neighbouring primary care trust, and modified based on their results. The final questionnaire comprised 41 Likert statements, 36 of which covered seven attitude dimensions of confidence and experience, tolerance of risk and uncertainty, complaints by patients or relatives, attitudes to hospital admission, patient-related factors, motivation, and organisational factors; an eighth dimension on job satisfaction had five additional questions based on previous research (Appendix 1).5 Data were also collected about age, years since qualification, type and size of practice, and what type of referrer GPs thought they were. Open-ended questions asked for GPs' opinions about alternatives to hospital which they felt might change their referral rates to hospital OOH and during their daytime work (Appendix 2).

A list of GP names was obtained from the OOH database (Adastra Software Ltd) for the three OOH providers around Bristol (one of which was the basis of the audit in 2004). Referral rates to hospital (OOH) were available for all GPs; however, only those who saw 20 or more face-to-face consultations in a 20-month time frame (1 January 2005 to 31 August 2006) were included in the survey. Questionnaires were sent to all GPs who had a current address on the OOH services databases. Reminder letters and questionnaires were sent after 3 weeks to the non-responders, and a personal telephone call made 2 weeks later by one of the research GPs for the study. Those who returned the questionnaire were paid £20.

Questionnaire data were entered onto an Excel® spreadsheet and combined with the OOH referral rates. Referral rates were expressed as a face-to-face rate and a total rate. The total rate included those contacts made by phone where they received ‘doctor advice’, whereas the face-to-face rate only included home visits and patient attendance at the primary care centre.

How this fits in

Emergency admissions to hospital are an increasing source of pressure on the NHS and many result from referrals from GP out-of-hours (OOH) providers. There is a greater than fourfold variation in OOH admission rates between GPs working for the same OOH services; research suggests that decision making varies and plays an important part in determining admission rates. GPs with a lower tolerance of risk are more likely to be high referrers to hospital. Providing a GP with special interest in primary care OOH service who could access alternatives to admission might enable GPs to alter their referral practices.

Statistical methods

The analysis was carried out in two parts. First, an analysis of all the GPs on the database modelled whether the associations found were similar to those seen before the contract changed.3 For this the GPs were banded into quartiles, based on their admission rates, and logistic regression was used to determine the odds of being admitted to hospital if seen by a GP in the top quartile, compared with that of being seen by a GP in the bottom quartile. Logistic regression was based on patient contacts, and STATA (version 9) accounted for multiple contacts per GP using clustering by GP. Linear regression explored any associations found, including place of visit and sex differences, with their referral rates to hospital. Comparison of referral rates for individual GPs with data available before and after the contract changed was also made using Spearman's rank correlations and Wilcoxon signed rank tests.

The second analysis involved only the GPs who had returned questionnaires. Comparisons for those who did and did not return questionnaires were carried out using χ2 test for sex and Mann–Whitney U tests for the number of contacts and referral rates. The correlation between the type of referrer that the GP perceived themselves to be, and their actual referral pattern was examined using Kendall's tau-b.

Factor analysis was used to investigate the 41 attitude statements, to determine whether there were any distinct subscales similar to those used in the design of the questionnaire on attitudes to referral to hospital from OOH providers. Different rotations were considered, including varimax and direct oblimin, to explore the clinical relevance of the subscales. Cronbach's alpha was used to identify potentially redundant items. The statements were also investigated individually to see if any of them were independently associated with referral rates.

Attitude scores were calculated for the eight questionnaire attitude dimensions, by summation of the items from each dimension. Items that were negatively correlated with the others were reversed and Spearman's rank correlation coefficients calculated to examine the association between referral rates and the eight dimensions. Similar correlations were calculated for individual items and referral rates using a square root transformation of referral rates to produce a normal distribution. Linear regression modelling was then carried out using the statistically significant items combined as a risk score with sex and place of visit.

Where the individual items were used, a statistical significance level of 0.01 was used (rather than 0.05) to reduce the chance of a type I error. A formal Bonferroni correction was not used as this was too stringent given the hypothesis-generating nature of the study. For regression modelling, P<0.05 significance was retained.


Data available from the Adastra OOH database for three OOH providers covering the 20-month period since January 2005 included 77 748 patient contacts with 496 OOH workers. After excluding those with no face-to-face contacts recorded (n = 37), those who had fewer than 20 face-to-face contacts (n = 212), those who were not GPs (n = 6), and a further seven duplicate names, 234 names remained. These GPs had 73 453 total patient contacts (n = 51 688 face-to-face), with 6353 admissions to hospital (n = 4913 from face-to-face contacts).

Admission rates

Admission rates were available for 234 GPs who had from 20 to 2580 face-to-face patient consultations. Face-to-face admission rates varied from 0% for a GP who had 26 face-to-face consultations, to 38.5% for a GP who had 91 face-to-face consultations. Referral rates were compared between the highest and lowest quartiles both before and after the change in contract. Table 1 shows that since the change in contract the odds ratio for being admitted to hospital if seen by a GP in the highest quartile was 4.49 (95% confidence interval [CI] = 3.98 to 5.06) compared with being seen by a GP in the lowest quartile, which was similar to odds ratios for the 3 years prior to the change in contract. The referral rates were also very similar, with the overall rates being 9.54% before the change in contract and 9.50% after the contract changed.

Table 1
Comparison of being admitted to hospital if seen by a GP in each of the rates of admission quartiles from face-to-face contacts before and after introduction of the new general medical services contract.a

Referral rates were also compared for 80 individual GPs with data available for both periods of time. There was a significant correlation in their referral rates (Spearman's r = 0.57, P<0.001) before and after the contract changed. Median referral rates for this group of GPs were not significantly different (Wilcoxon P = 0.17) before (9.2%) and after (9.8%) the contact changed. There were no significant differences in the referral rates of those who continued with OOH work and those who did not (Mann–Whitney P = 0.887). Although fewer female GPs continued with the new contract (44% compared to 51% before), the proportion of males to females was not significantly different.

There was a difference in the number of patients seen at home or the primary care centre for male and female doctors, in that females made fewer home visits (16% of contacts) compared with males (21%) and seemed to choose more shifts based at the primary care centre. The distribution of referral rates for male and female doctors by the site of the consultation (primary care centre or home) was also different, as shown in Table 2, being highest for females when visiting patients at home. Linear regression modelling of referral rates controlling for place of visit showed that these sex differences were statistically significant (P<0.001), indicating that female GPs referred more patients to hospital than male GPs.

Table 2
Referral rate (%) to hospital out of hours by GP sex and place of consultation.

Postal survey

Of the 234 names, 17 were excluded as there was no current address, and another 23 were only registered with one OOH provider that did not distribute the questionnaires, leaving an eligible total of 194 GPs. Six of these returned their questionnaires indicating that they did not wish to take part, and 38 did not return a questionnaire, resulting in 150 completed questionnaires (77.3% return rate). There were no differences in sex or referral rates between those who did and did not return their questionnaires.

There was a significant correlation (P = 0.002, Kendall's tau-b = 0.21) between what type of referrer (high, medium, low) GPs thought they were and their referral rates, but interestingly only 5% (n = 7) of them felt that they were high referrers. The correlation was low, indicating that there was a general lack of awareness about their own referral rate compared to those of others.

Factor analysis did not produce any clinically relevant subscales even after trying different rotations. The confirmatory factor analysis of the eight attitude dimensions suggested that there might be further subscales present within some of the dimensions, but due to the low number of items within each dimension it was not possible to identify stable subscales within them. Cronbach's alpha did not indicate that the omission of any items would lead to a substantial improvement.

Spearman's rank correlation showed no statistically significant associations between any of the eight dimensions of the attitude survey and referral patterns, as shown in Table 3. A sample of 150 participants has 80% power to detect correlations of 0.23 or greater as statistically significant at the 0.05 level, and none of these correlations was greater than 0.13.

Table 3
Correlations between questionnaire attitude dimensions and referral rates (n = 150).

Analysis of the 41 individual items produced three items from the tolerance of risk and uncertainty dimension with statistically significant correlations with referral rates as displayed in Table 4. These indicated that GPs who believe that they are cautious have higher referral rates; those who believe they are carrying out risk assessment and those who believe they are good at living with uncertainty and risk both have lower referral rates.

Table 4
Item-by-item correlation coefficients and P-valuesa for the 41 attitude items with referral rates (both total admitted rates and rates for those admitted following face-to-face consultations).

A risk score was computed by summing these responses (items 4, 7, and 8) related to tolerance of risk and uncertainty. Linear regression modelling transformed the face-to-face referral rate with sex, controlling for place of visit, showed that female doctors were significantly more likely to refer patients to hospital than male doctors (P = 0.039). However, when the risk score was included in the model, the sex difference was no longer significant and the risk score had a stronger association with referral rate (P<0.001), indicating that it is GPs with a lower tolerance of risk that are more likely to admit patients to hospital OOH.

Free-text comments on the questionnaires suggested that GPs might change their referral behaviour if there were better access to intermediate care, more information about patients, and some follow-up or feedback about patients.


Summary of main findings

This study has confirmed the authors' previous finding3 with the current group of 234 GPs working for three OOH organisations, that there remains a greater than fourfold variation in referral rates between the highest and lowest quartiles of GPs since January 2005. Referral rates have also not changed since the new GMS contract started. This is remarkable, since the study was comparing three post-contract OOH organisations with one pre-contract one and a different group of GPs, who had all opted to do OOH work. Previous findings from the authors' qualitative study,4 which suggested that GP perception of risk had an influence on decisions to refer, were also confirmed. The survey evidence showed that those who are comfortable living with risk are likely to be low referrers to hospital, and this was also evident in the qualitative study where high-referring GPs were typically cautious, believed it was better to refer if in doubt, and expressed anxiety about the consequences of a decision not to admit, for both the patient and themselves.4 The authors had believed that the changes in the new contract would attract different types of GPs to OOH for different reasons and motives, and that those who were not comfortable with it would opt out. However, this was not reflected by GPs' referral decisions.

Strengths and limitations of the study

The strength of the study is the high response rate in the survey from those who received it. Thus the results can be considered as representative of the views of GPs working in OOH services. Limitations include the use of only three GP OOH services and the relatively large numbers of items that were used to try to assess whether attitudes were associated with referral behaviour. While a large majority of these attitudinal items were derived directly from the qualitative study, their validity as a measure had not been evaluated in previous research. Also, the small number of items that were statistically significant did not enable the development of a comprehensive assessment training tool as initially intended. However, the analysis did identify three risk items that might be used as an ‘attitude to risk’ tool to predict referral rates.

Comparison with existing literature

This study showed that female GPs refer more patients to hospital OOH, and they seemed to choose shifts based in the primary care centre rather than doing home visits. When this was allowed for in the analysis, GPs with low scores for tolerance of risk were more likely to be high referrers to hospital. This supports research evidence from daytime referral practices,6 which suggests GPs' threshold of risk is important for explaining variations in referral. However, males and females work in different ways, with females being more patient centred,7 and perhaps more risk averse. On the other hand, male doctors have a higher output than females, but this is offset by males experiencing more litigation and disciplinary action.8

Most GPs seemed to know what type of referrer they were, but a few felt that they were lower referrers than was actually seen in their referral rates. They indicated, as was evident in the previous study,4 that they might alter their referral practices if reliable and accessible alternatives to hospital were readily available to keep patients at home, while additional training, feedback, and opportunities for a second opinion without referring might also affect some GPs' referral decisions.

Understanding how different doctors make decisions,9 and the difficulties they have estimating and managing risks,10 is important in planning changes in their performance. Different doctors have different ways of managing uncertainty and may be unaware of how they compare with others; poor performers often have misplaced confidence in their performance.11 Giving doctors educational feedback on their performance compared to the optimal can change the clinical performance of some doctors,11 and may have an impact on referral rates OOH.

Implications for future research or clinical practice

Some GPs have suggested that a separate defined OOH specialty with recognised qualifications and bespoke or mandatory training in primary care OOH services might provide a better service.12 An ‘attitude to risk’ assessment tool could be used in such training.

Further research could explore in greater depth why female GPs may be more risk averse. Another important area to investigate would be how to facilitate access to intermediate care or other alternatives to emergency admission, and feedback to GPs about patient outcomes, to provide a greater insight into what OOH practitioners would value and whether patients' quality of life could be improved.


We thank all the GPs who took part, Rob Simpson for managing the Adastra data, and Hannah Spiers for questionnaire data management.

Appendix 1. Attitudes to referral to hospital from out-of-hours questionnaire.

The following statements, some of which have been made by GPs, represent a range of views about the influences on decisions to refer to hospital or not from out-of-hours (OOH). Please indicate how far you agree with these statements by selecting the appropriate level of agreement.

Confidence and experienceAgree stronglyAgree a littleNeither agree nor disagreeDisagree a littleDisagree strongly

  1. I have a continual feeling that I haven't quite done enough.
  2. I feel I don't know whether I'm doing the right thing or not.
  3. Experience has taught me that being cautious sometimes isn't necessarily a bad thing.

Tolerance of risk and uncertaintyAgree stronglyAgree a littleNeither agree nor disagreeDisagree a littleDisagree strongly

  • 4.
    When it comes to medicine I'm quite cautious.
  • 5.
    As a GP you think that you can deal with most things most of the time.
  • 6.
    I think my risk assessment is reasonably good, and I'm reasonably safe.
  • 7.
    All GPs take risks; it's risk assessment in general practice all the time.
  • 8.
    GPs are good at living with uncertainty and risk.
  • 9.
    I don't worry about my decisions after I've made them.
  • 10.
    I sometimes go back and check on the patient's outcome after a shift has finished.

Complaints by patients or relativesAgree stronglyAgree a littleNeither agree nor disagreeDisagree a littleDisagree strongly

  • 11.
    I don't worry about a complaint being made about me.
  • 12.
    I have had complaints made against me.
  • 13.
    Fear of litigation influences my practice.

Attitudes to hospital admissionAgree stronglyAgree a littleNeither agree nor disagreeDisagree a littleDisagree strongly

  • 14.
    A benefit of admission is that it reduces the risk of being sued.
  • 15.
    It's easier to send somebody in and get the hospital to check them out than to leave them at home.
  • 16.
    Admitting someone to hospital enables me to get a second opinion.
  • 17.
    Admitting someone to hospital means I can speed up urgent tests.
  • 18.
    Admitting someone to hospital puts them in danger of being ‘over-tested’.

Patient-related factorsAgree stronglyAgree a littleNeither agree nor disagreeDisagree a littleDisagree strongly

  • 19.
    The most important factor in my decision to admit is the patient's clinical status.
  • 20.
    I am more likely to admit a person if they want to be admitted.
  • 21.
    If a family wants me to admit their relative I would be more inclined to admit them.
  • 22.
    I see myself as a negotiator, ‘selling’ my decision (whatever that is) to family and carers.
  • 23.
    If members of the family say there's nobody to look after someone, I see that as a problem for the family rather than the doctor.
  • 24.
    I am more likely to admit someone if they live alone.
  • 25.
    I am more likely to admit someone if they are poorly housed.

MotivationAgree stronglyAgree a littleNeither agree nor disagreeDisagree a littleDisagree strongly

  • 26.
    I like the nature of the clinical work in OOH care.
  • 27.
    I work in OOH because I like the financial rewards.
  • 28.
    I enjoy working with my OOH organisation.
  • 29.
    I dislike the work in OOH because it is different from routine GP work.
  • 30.
    I do not enjoy working with the larger network of GPs, nurses, and other staff OOH.
  • 31.
    I work in OOH because I feel responsible for the quality of care that our patients receive OOH.

Organisational factorsAgree stronglyAgree a littleNeither agree nor disagreeDisagree a littleDisagree strongly

  • 32.
    I think intermediate care is probably much better for the patient if they can have it.
  • 33.
    I have experienced difficulty arranging intermediate care for someone.
  • 34.
    Hospital nurses are less likely than hospital doctors to confront you about your decision to admit.
  • 35.
    I feel guilty when I admit a patient.
  • 36.
    We are increasingly under pressure not to admit people.

Job satisfaction (in your work as a doctor)Agree stronglyAgree a littleNeither agree nor disagreeDisagree a littleDisagree strongly

  • 37.
    I feel that some parts of my work do not make sense.
  • 38.
    My work still interests me as much as it ever did.
  • 39.
    Assuming that pay and conditions were similar, I would just as soon do non-medical work.
  • 40.
    I find enjoyment in my work.
  • 41.
    My work involves a great deal of wasted effort on my part.

Appendix 2. Background factors questionnaire.

Please complete the following or circle the appropriate alternative.

An external file that holds a picture, illustration, etc.
Object name is bjgp59-e16S1.jpg

aAbbreviations ‘Brisdoc’, ‘Frendoc’, and ‘Nordoc’ are used by three local out-of-hours services covering three areas of the city of Bristol.


Funding body

Avon Primary Care Research Collaborative small grant scheme (05-IPE4)

Ethical approval

Southmead Research Ethics Committee (ref 05/Q2002/62) August 2005

Competing interests

The authors have stated that there are none

Discuss this article

Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss


1. Roland M, Dusheiko M, Gravelle H, Parker S. Follow up of people aged 65 and over with a history of emergency admissions: analysis of routine admission data. BMJ. 2005;330(7486):289–292. [PMC free article] [PubMed]
2. Hospital Episode Statistics. 2005/6. http://www.hesonline.org.uk (accessed 11 Jul 2008)
3. Rossdale M, Kemple T, Payne S, et al. An observational study of variation in GPs' out-of-hours emergency referrals. Br J Gen Pract. 2007;57(535):152–154. [PMC free article] [PubMed]
4. Calnan M, Payne S, Kemple T, et al. A qualitative study exploring variations in general practitioners' out-of-hours referrals to hospital. Br J Gen Pract. 2007;57(542):706–713. [PMC free article] [PubMed]
5. Calnan M, Corney R. Job satisfaction in general practice: a longitudinal study. Int J Health Sci. 1993;5:51–58.
6. Wilkin D. Patterns of referral: explaining variations. In: Roland MO, Coulter A, editors. Hospital referrals. Oxford: Oxford University Press; 1992. pp. 76–91.
7. Roter DL, Hall JA, Aoki Y. Physician gender effects in medical communication: a meta-analytic review. JAMA. 2002;288(6):756–764. [PubMed]
8. Firth-Cozens J. Effects of gender on performance in medicine [editorial] BMJ. 2008;336(7647):731–732. [PMC free article] [PubMed]
9. Groopman J. How doctors think. Boston: Houghton Mifflin; 2007.
10. Gigerenzer G. Reckoning with risk. London: Allen Lane; 2002.
11. Potchen E. Measuring observer performance in chest radiology: some experiences. J Am Coll Radiol. 2006;3(6):423–432. [PubMed]
12. Twisselmann B. GPs' 24 hour responsibility: summary of responses. BMJ. 2007;335(7624):787–788. [PMC free article] [PubMed]

Articles from The British Journal of General Practice are provided here courtesy of Royal College of General Practitioners
PubReader format: click here to try


Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


  • PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...