• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of qualsafetyQuality and Safety in Health CareCurrent TOCInstructions for authors
Qual Saf Health Care. Apr 2003; 12(2): 122–128.
PMCID: PMC1743685

Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls


Measuring the quality of health care has become a major concern for funders and providers of health services in recent decades. One of the ways in which quality of care is currently assessed is by taking routinely collected data and analysing them quantitatively. The use of routine data has many advantages but there are also some important pitfalls. Collating numerical data in this way means that comparisons can be made—whether over time, with benchmarks, or with other healthcare providers (at individual or institutional levels of aggregation). Inevitably, such comparisons reveal variations. The natural inclination is then to assume that such variations imply rankings: that the measures reflect quality and that variations in the measures reflect variations in quality. This paper identifies reasons why these assumptions need to be applied with care, and illustrates the pitfalls with examples from recent empirical work. It is intended to guide not only those who wish to interpret comparative quality data, but also those who wish to develop systems for such analyses themselves.

Full Text

The Full Text of this article is available as a PDF (110K).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.
  • Davies HT, Marshall MN. Public disclosure of performance data: does the public get what the public wants? Lancet. 1999 May 15;353(9165):1639–1640. [PubMed]
  • Nutley S, Smith PC. League tables for performance improvement in health care. J Health Serv Res Policy. 1998 Jan;3(1):50–57. [PubMed]
  • Rainwater JA, Romano PS, Antonius DM. The California Hospital Outcomes Project: how useful is California's report card for quality improvement? Jt Comm J Qual Improv. 1998 Jan;24(1):31–39. [PubMed]
  • Romano PS, Rainwater JA, Antonius D. Grading the graders: how hospitals in California and New York perceive and interpret their report cards. Med Care. 1999 Mar;37(3):295–305. [PubMed]
  • Davies HT. Public release of performance data and quality improvement: internal responses to external data by US health care providers. Qual Health Care. 2001 Jun;10(2):104–110. [PMC free article] [PubMed]
  • Davies HT, Crombie IK. Assessing the quality of care. BMJ. 1995 Sep 23;311(7008):766–766. [PMC free article] [PubMed]
  • Davies HT, Crombie IK. Interpreting health outcomes. J Eval Clin Pract. 1997 Aug;3(3):187–199. [PubMed]
  • Crombie IK, Davies HT. Beyond health outcomes: the advantages of measuring process. J Eval Clin Pract. 1998 Feb;4(1):31–38. [PubMed]
  • Campbell SM, Braspenning J, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002 Dec;11(4):358–364. [PMC free article] [PubMed]
  • Noseworthy JH, Ebers GC, Vandervoort MK, Farquhar RE, Yetisir E, Roberts R. The impact of blinding on the results of a randomized, placebo-controlled multiple sclerosis clinical trial. Neurology. 1994 Jan;44(1):16–20. [PubMed]
  • Leyland AH, Boddy FA. League tables and acute myocardial infarction. Lancet. 1998 Feb 21;351(9102):555–558. [PubMed]
  • Jenkins KJ, Newburger JW, Lock JE, Davis RB, Coffman GA, Iezzoni LI. In-hospital mortality for surgical repair of congenital heart defects: preliminary observations of variation by hospital caseload. Pediatrics. 1995 Mar;95(3):323–330. [PubMed]
  • Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB, Mukamal K, Phillips RS, Banks NJ, Davis DT., Jr Does the Complications Screening Program flag cases with process of care problems? Using explicit criteria to judge processes. Int J Qual Health Care. 1999 Apr;11(2):107–118. [PubMed]
  • Stange KC, Zyzanski SJ, Smith TF, Kelly R, Langa DM, Flocke SA, Jaén CR. How valid are medical records and patient questionnaires for physician profiling and health services research? A comparison with direct observation of patients visits. Med Care. 1998 Jun;36(6):851–867. [PubMed]
  • Goldacre Michael J, Griffith Myfanwy, Gill Leicester, Mackintosh Anne. In-hospital deaths as fraction of all deaths within 30 days of hospital admission for surgery: analysis of routine statistics. BMJ. 2002 May 4;324(7345):1069–1070. [PMC free article] [PubMed]
  • Druss B, Rosenheck R. Evaluation of the HEDIS measure of behavioral health care quality. Health Plan Employer Data and Information Set. Psychiatr Serv. 1997 Jan;48(1):71–75. [PubMed]
  • Morgan K, Prothero D, Frankel S. The rise in emergency admissions--crisis or artefact? Temporal analysis of health services data. BMJ. 1999 Jul 17;319(7203):158–159. [PMC free article] [PubMed]
  • Wakefield DS, Hendryx MS, Uden-Holman T, Couch R, Helms CM. Comparing providers' performance: problems in making the "report card" analogy fit. J Healthc Qual. 1996 Nov-Dec;18(6):4–10. [PubMed]
  • Hartz AJ, Kuhn EM. Comparing hospitals that perform coronary artery bypass surgery: the effect of outcome measures and data sources. Am J Public Health. 1994 Oct;84(10):1609–1614. [PMC free article] [PubMed]
  • Iezzoni LI. Using risk-adjusted outcomes to assess clinical practice: an overview of issues pertaining to risk adjustment. Ann Thorac Surg. 1994 Dec;58(6):1822–1826. [PubMed]
  • Iezzoni LI. The risks of risk adjustment. JAMA. 1997 Nov 19;278(19):1600–1607. [PubMed]
  • Shwartz M, Iezzoni LI, Ash AS, Mackiernan YD. Do severity measures explain differences in length of hospital stay? The case of hip fracture. Health Serv Res. 1996 Oct;31(4):365–385. [PMC free article] [PubMed]
  • Iezzoni LI, Shwartz M, Ash AS, Mackiernan YD. Does severity explain differences in hospital length of stay for pneumonia patients? J Health Serv Res Policy. 1996 Apr;1(2):65–76. [PubMed]
  • Green J, Wintfeld N. Report cards on cardiac surgeons. Assessing New York State's approach. N Engl J Med. 1995 May 4;332(18):1229–1232. [PubMed]
  • Sowden AJ, Deeks JJ, Sheldon TA. Volume and outcome in coronary artery bypass graft surgery: true association or artefact? BMJ. 1995 Jul 15;311(6998):151–155. [PMC free article] [PubMed]
  • Posnett J. Is bigger better? Concentration in the provision of secondary care. BMJ. 1999 Oct 16;319(7216):1063–1065. [PMC free article] [PubMed]
  • Spiegelhalter David J. Mortality and volume of cases in paediatric cardiac surgery: retrospective study based on routinely collected data. BMJ. 2002 Feb 2;324(7332):261–263. [PMC free article] [PubMed]
  • Komaromy M, Lurie N, Osmond D, Vranizan K, Keane D, Bindman AB. Physician practice style and rates of hospitalization for chronic medical conditions. Med Care. 1996 Jun;34(6):594–609. [PubMed]
  • Reid FD, Cook DG, Majeed A. Explaining variation in hospital admission rates between general practices: cross sectional study. BMJ. 1999 Jul 10;319(7202):98–103. [PMC free article] [PubMed]
  • Majeed A, Bardsley M, Morgan D, O'Sullivan C, Bindman AB. Cross sectional study of primary care groups in London: association of measures of socioeconomic and health status with hospital admission rates. BMJ. 2000 Oct 28;321(7268):1057–1060. [PMC free article] [PubMed]
  • Giuffrida A, Gravelle H, Roland M. Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes. BMJ. 1999 Jul 10;319(7202):94–98. [PMC free article] [PubMed]
  • Majeed FA, Cook DG, Anderson HR, Hilton S, Bunn S, Stones C. Using patient and general practice characteristics to explain variations in cervical smear uptake rates. BMJ. 1994 May 14;308(6939):1272–1276. [PMC free article] [PubMed]
  • Hofer TP, Hayward RA. Identifying poor-quality hospitals. Can hospital mortality rates detect quality problems for medical diagnoses? Med Care. 1996 Aug;34(8):737–753. [PubMed]
  • Hofer TP, Hayward RA. Can early re-admission rates accurately detect poor-quality hospitals? Med Care. 1995 Mar;33(3):234–245. [PubMed]
  • Park RE, Brook RH, Kosecoff J, Keesey J, Rubenstein L, Keeler E, Kahn KL, Rogers WH, Chassin MR. Explaining variations in hospital death rates. Randomness, severity of illness, quality of care. JAMA. 1990 Jul 25;264(4):484–490. [PubMed]
  • Thomas JW, Hofer TP. Accuracy of risk-adjusted mortality rate as a measure of hospital quality of care. Med Care. 1999 Jan;37(1):83–92. [PubMed]
  • Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG. The unreliability of individual physician "report cards" for assessing the costs and quality of care of a chronic disease. JAMA. 1999 Jun 9;281(22):2098–2105. [PubMed]
  • Florey CD. Sample size for beginners. BMJ. 1993 May 1;306(6886):1181–1184. [PMC free article] [PubMed]
  • Mant J, Hicks N. Detecting differences in quality of care: the sensitivity of measures of process and outcome in treating acute myocardial infarction. BMJ. 1995 Sep 23;311(7008):793–796. [PMC free article] [PubMed]
  • Marshall EC, Spiegelhalter DJ. Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates. BMJ. 1998 Jun 6;316(7146):1701–1705. [PMC free article] [PubMed]
  • Bland JM, Altman DG. Some examples of regression towards the mean. BMJ. 1994 Sep 24;309(6957):780–780. [PMC free article] [PubMed]
  • Hayes RJ. Methods for assessing whether change depends on initial value. Stat Med. 1988 Sep;7(9):915–927. [PubMed]
  • Black N. High-quality clinical databases: breaking down barriers. Lancet. 1999 Apr 10;353(9160):1205–1206. [PubMed]
  • McCarthy EP, Iezzoni LI, Davis RB, Palmer RH, Cahalane M, Hamel MB, Mukamal K, Phillips RS, Davies DT., Jr Does clinical evidence support ICD-9-CM diagnosis coding of complications? Med Care. 2000 Aug;38(8):868–876. [PubMed]
  • Bickell NA, Chassin MR. Determining the quality of breast cancer care: do tumor registries measure up? Ann Intern Med. 2000 May 2;132(9):705–710. [PubMed]
  • Fielding JE, Sutherland CE, Halfon N. Community health report cards. Results of a national survey. Am J Prev Med. 1999 Jul;17(1):79–86. [PubMed]
  • Mohammed MA, Cheng KK, Rouse A, Marshall T. Bristol, Shipman, and clinical governance: Shewhart's forgotten lessons. Lancet. 2001 Feb 10;357(9254):463–467. [PubMed]
  • Adab Peymané, Rouse Andrew M, Mohammed Mohammed A, Marshall Tom. Performance league tables: the NHS deserves better. BMJ. 2002 Jan 12;324(7329):95–98. [PMC free article] [PubMed]
  • Lawrance RA, Dorsch MF, Sapsford RJ, Mackintosh AF, Greenwood DC, Jackson BM, Morrell C, Robinson MB, Hall AS. Use of cumulative mortality data in patients with acute myocardial infarction for early detection of variation in clinical practice: observational study. BMJ. 2001 Aug 11;323(7308):324–327. [PMC free article] [PubMed]
  • Mannion Russell, Davies Huw T O. Reporting health care performance: learning from the past, prospects for the future. J Eval Clin Pract. 2002 May;8(2):215–228. [PubMed]
  • Davies Huw T O, Washington A Eugene, Bindman Andrew B. Health care report cards: implications for vulnerable patient groups and the organizations providing them care. J Health Polit Policy Law. 2002 Jun;27(3):379–399. [PubMed]
  • Marshall M, Davies H. Public release of information on quality of care: how are health services and the public expected to respond? J Health Serv Res Policy. 2001 Jul;6(3):158–162. [PubMed]

Articles from Quality & Safety in Health Care are provided here courtesy of BMJ Group


Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


  • PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...