• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of qualsafetyQuality and Safety in Health CareCurrent TOCInstructions for authors
Qual Saf Health Care. Dec 2002; 11(4): 358–364.
PMCID: PMC1758017

Research methods used in developing and applying quality indicators in primary care


Quality indicators have been developed throughout Europe primarily for use in hospitals, but also increasingly for primary care. Both development and application are important but there has been less research on the application of indicators. Three issues are important when developing or applying indicators: (1) which stakeholder perspective(s) are the indicators intended to reflect; (2) what aspects of health care are being measured; and (3) what evidence is available? The information required to develop quality indicators can be derived using systematic or non-systematic methods. Non-systematic methods such as case studies play an important role but they do not tap in to available evidence. Systematic methods can be based directly on scientific evidence by combining available evidence with expert opinion, or they can be based on clinical guidelines. While it may never be possible to produce an error free measure of quality, measures should adhere, as far as possible, to some fundamental a priori characteristics (acceptability, feasibility, reliability, sensitivity to change, and validity). Adherence to these characteristics will help maximise the effectiveness of quality indicators in quality improvement strategies. It is also necessary to consider what the results of applying indicators tell us about quality of care.

Full Text

The Full Text of this article is available as a PDF (120K).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.
  • Forrest D, Hoskins A, Hussey R. Clinical guidelines and their implementation. Postgrad Med J. 1996 Jan;72(843):19–22. [PMC free article] [PubMed]
  • Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993 Nov 27;342(8883):1317–1322. [PubMed]
  • McGlynn EA, Asch SM. Developing a clinical performance measure. Am J Prev Med. 1998 Apr;14(3 Suppl):14–21. [PubMed]
  • Seddon ME, Marshall MN, Campbell SM, Roland MO. Systematic review of studies of quality of clinical care in general practice in the UK, Australia and New Zealand. Qual Health Care. 2001 Sep;10(3):152–158. [PMC free article] [PubMed]
  • McGlynn EA. Six challenges in measuring the quality of health care. Health Aff (Millwood) 1997 May-Jun;16(3):7–21. [PubMed]
  • Davies HT, Crombie IK. Assessing the quality of care. BMJ. 1995 Sep 23;311(7008):766–766. [PMC free article] [PubMed]
  • Eddy DM. Performance measurement: problems and solutions. Health Aff (Millwood) 1998 Jul-Aug;17(4):7–25. [PubMed]
  • Mant J, Hicks N. Detecting differences in quality of care: the sensitivity of measures of process and outcome in treating acute myocardial infarction. BMJ. 1995 Sep 23;311(7008):793–796. [PMC free article] [PubMed]
  • Palmer RH. Process-based measures of quality: the need for detailed clinical data in large health care databases. Ann Intern Med. 1997 Oct 15;127(8 Pt 2):733–738. [PubMed]
  • Tarlov AR, Ware JE, Jr, Greenfield S, Nelson EC, Perrin E, Zubkoff M. The Medical Outcomes Study. An application of methods for monitoring the results of medical care. JAMA. 1989 Aug 18;262(7):925–930. [PubMed]
  • McGlynn EA. The outcomes utility index: will outcomes data tell us what we want to know? Int J Qual Health Care. 1998 Dec;10(6):485–490. [PubMed]
  • Wensing M, Jung HP, Mainz J, Olesen F, Grol R. A systematic review of the literature on patient priorities for general practice care. Part 1: Description of the research domain. Soc Sci Med. 1998 Nov;47(10):1573–1588. [PubMed]
  • Campbell SM, Hann M, Hacker J, Burns C, Oliver D, Thapar A, Mead N, Safran DG, Roland MO. Identifying predictors of high quality care in English general practice: observational study. BMJ. 2001 Oct 6;323(7316):784–787. [PMC free article] [PubMed]
  • Ramsay J, Campbell JL, Schroter S, Green J, Roland M. The General Practice Assessment Survey (GPAS): tests of data quality and measurement properties. Fam Pract. 2000 Oct;17(5):372–379. [PubMed]
  • Wensing Michel, Vedsted Peter, Kersnik Janko, Peersman Wim, Klingenberg Anja, Hearnshaw Hilary, Hjortdahl Per, Paulus Dominique, Künzi Beat, Mendive Juan, et al. Patient satisfaction with availability of general practice: an international comparison. Int J Qual Health Care. 2002 Apr;14(2):111–118. [PubMed]
  • van den Hombergh P, Grol R, van den Hoogen HJ, van den Bosch WJ. Practice visits as a tool in quality improvement: mutual visits and feedback by peers compared with visits and feedback by non-physician observers. Qual Health Care. 1999 Sep;8(3):161–166. [PMC free article] [PubMed]
  • Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med. 1996 Sep 26;335(13):966–970. [PubMed]
  • Howie JG, Heaney DJ, Maxwell M. Measuring quality in general practice. Pilot study of a needs, process and outcome measure. Occas Pap R Coll Gen Pract. 1997 Feb;(75):i–32. [PMC free article] [PubMed]
  • Campbell SM, Roland MO, Shekelle PG, Cantrill JA, Buetow SA, Cragg DK. Development of review criteria for assessing the quality of management of stable angina, adult asthma, and non-insulin dependent diabetes mellitus in general practice. Qual Health Care. 1999 Mar;8(1):6–15. [PMC free article] [PubMed]
  • Brook RH, McGlynn EA, Shekelle PG. Defining and measuring quality of care: a perspective from US researchers. Int J Qual Health Care. 2000 Aug;12(4):281–295. [PubMed]
  • Pringle M. Preventing ischaemic heart disease in one general practice: from one patient, through clinical audit, needs assessment, and commissioning into quality improvement. BMJ. 1998 Oct 24;317(7166):1120–1124. [PMC free article] [PubMed]
  • Pringle M. Clinical governance in primary care: participating in clinical governance. BMJ. 2000 Sep 23;321(7263):737–740. [PMC free article] [PubMed]
  • Hearnshaw HM, Harker RM, Cheater FM, Baker RH, Grimshaw GM. Expert consensus on the desirable characteristics of review criteria for improvement of health care quality. Qual Health Care. 2001 Sep;10(3):173–178. [PMC free article] [PubMed]
  • Campbell SM, Cantrill JA. Consensus methods in prescribing research. J Clin Pharm Ther. 2001 Feb;26(1):5–14. [PubMed]
  • Baker R, Fraser RC. Is ownership more important than the scientific credibility of audit protocols? A survey of medical audit advisory groups. Fam Pract. 1997 Apr;14(2):107–111. [PubMed]
  • McColl A, Roderick P, Gabbay J, Smith H, Moore M. Performance indicators for primary care groups: an evidence based approach. BMJ. 1998 Nov 14;317(7169):1354–1360. [PMC free article] [PubMed]
  • Naylor CD. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet. 1995 Apr 1;345(8953):840–842. [PubMed]
  • Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, Marteau T. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999 Oct;4(4):236–248. [PubMed]
  • Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995 Aug 5;311(7001):376–380. [PMC free article] [PubMed]
  • Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The reproducibility of a method to identify the overuse and underuse of medical procedures. N Engl J Med. 1998 Jun 25;338(26):1888–1895. [PubMed]
  • Campbell SM, Roland MO, Buetow SA. Defining quality of care. Soc Sci Med. 2000 Dec;51(11):1611–1625. [PubMed]
  • Ibrahim JE. Performance indicators from all perspectives. Int J Qual Health Care. 2001 Dec;13(6):431–432. [PubMed]
  • Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000 Oct;32(4):1008–1015. [PubMed]
  • Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984 Sep;74(9):979–983. [PMC free article] [PubMed]
  • Brook RH, Chassin MR, Fink A, Solomon DH, Kosecoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Technol Assess Health Care. 1986;2(1):53–63. [PubMed]
  • Cantrill JA, Sibbald B, Buetow S. Indicators of the appropriateness of long-term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility, and reliability. Qual Health Care. 1998 Sep;7(3):130–135. [PMC free article] [PubMed]
  • Rockwell MA. The Delphi procedure: knowledge from goat viscera? N Engl J Med. 1973 Jun 14;288(24):1298–1299. [PubMed]
  • Marsh L, Dawson TM. Treatment of early Parkinson's disease. BMJ. 2000 Jul 1;321(7252):1–2. [PMC free article] [PubMed]
  • Campbell SM, Roland MO, Quayle JA, Buetow SA, Shekelle PG. Quality indicators for general practice: which ones can general practitioners and health authority managers agree are important and how useful are they? J Public Health Med. 1998 Dec;20(4):414–421. [PubMed]
  • Normand SL, McNeil BJ, Peterson LE, Palmer RH. Eliciting expert opinion using the Delphi technique: identifying performance indicators for cardiovascular disease. Int J Qual Health Care. 1998 Jun;10(3):247–260. [PubMed]
  • Naylor CD. What is appropriate care? N Engl J Med. 1998 Jun 25;338(26):1918–1920. [PubMed]
  • Hicks NR. Some observations on attempts to measure appropriateness of care. BMJ. 1994 Sep 17;309(6956):730–733. [PMC free article] [PubMed]
  • Ayanian JZ, Landrum MB, Normand SL, Guadagnoli E, McNeil BJ. Rating the appropriateness of coronary angiography--do practicing physicians agree with an expert panel and with each other? N Engl J Med. 1998 Jun 25;338(26):1896–1904. [PubMed]
  • Leape LL, Hilborne LH, Schwartz JS, Bates DW, Rubin HR, Slavin P, Park RE, Witter DM, Jr, Panzer RJ, Brook RH. The appropriateness of coronary artery bypass graft surgery in academic medical centers. Working Group of the Appropriateness Project of the Academic Medical Center Consortium. Ann Intern Med. 1996 Jul 1;125(1):8–18. [PubMed]
  • Kahn KL, Rogers WH, Rubenstein LV, Sherwood MJ, Reinisch EJ, Keeler EB, Draper D, Kosecoff J, Brook RH. Measuring quality of care with explicit process criteria before and after implementation of the DRG-based prospective payment system. JAMA. 1990 Oct 17;264(15):1969–1973. [PubMed]
  • Shekelle PG, MacLean CH, Morton SC, Wenger NS. Assessing care of vulnerable elders: methods for developing quality indicators. Ann Intern Med. 2001 Oct 16;135(8 Pt 2):647–652. [PubMed]
  • Gray D, Hampton JR, Bernstein SJ, Kosecoff J, Brook RH. Audit of coronary angiography and bypass surgery. Lancet. 1990 Jun 2;335(8701):1317–1320. [PubMed]
  • Scott EA, Black N. Appropriateness of cholecystectomy in the United Kingdom--a consensus panel approach. Gut. 1991 Sep;32(9):1066–1070. [PMC free article] [PubMed]
  • Hemingway H, Crook AM, Feder G, Banerjee S, Dawson JR, Magee P, Philpott S, Sanders J, Wood A, Timmis AD. Underuse of coronary revascularization procedures in patients considered appropriate candidates for revascularization. N Engl J Med. 2001 Mar 1;344(9):645–654. [PubMed]
  • Campbell SM, Hann M, Hacker J, Durie A, Thapar A, Roland MO. Quality assessment for three common conditions in primary care: validity and reliability of review criteria developed by expert panels for angina, asthma and type 2 diabetes. Qual Saf Health Care. 2002 Jun;11(2):125–130. [PMC free article] [PubMed]
  • Eccles M, Clapp Z, Grimshaw J, Adams PC, Higgins B, Purves I, Russell I. North of England evidence based guidelines development project: methods of guideline development. BMJ. 1996 Mar 23;312(7033):760–762. [PMC free article] [PubMed]
  • Smeele IJ, Grol RP, van Schayck CP, van den Bosch WJ, van den Hoogen HJ, Muris JW. Can small group education and peer review improve care for patients with asthma/chronic obstructive pulmonary disease? Qual Health Care. 1999 Jun;8(2):92–98. [PMC free article] [PubMed]
  • Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998 Sep 26;317(7162):858–861. [PMC free article] [PubMed]
  • Schers H, Braspenning J, Drijver R, Wensing M, Grol R. Low back pain in general practice: reported management and reasons for not adhering to the guidelines in The Netherlands. Br J Gen Pract. 2000 Aug;50(457):640–644. [PMC free article] [PubMed]
  • Frijling BD, Spies TH, Lobo CM, Hulscher ME, van Drenth BB, Braspenning JC, Prins A, van der Wouden JC, Grol RP. Blood pressure control in treated hypertensive patients: clinical performance of general practitioners. Br J Gen Pract. 2001 Jan;51(462):9–14. [PMC free article] [PubMed]
  • Campbell SM, Hann M, Roland MO, Quayle JA, Shekelle PG. The effect of panel membership and feedback on ratings in a two-round Delphi survey: results of a randomized controlled trial. Med Care. 1999 Sep;37(9):964–968. [PubMed]
  • Coulter I, Adams A, Shekelle P. Impact of varying panel membership on ratings of appropriateness in consensus panels: a comparison of a multi- and single disciplinary panel. Health Serv Res. 1995 Oct;30(4):577–591. [PMC free article] [PubMed]
  • Fraser GM, Pilpel D, Kosecoff J, Brook RH. Effect of panel composition on appropriateness ratings. Int J Qual Health Care. 1994 Sep;6(3):251–255. [PubMed]
  • Huff ED. Comprehensive reliability assessment and comparison of quality indicators and their components. J Clin Epidemiol. 1997 Dec;50(12):1395–1404. [PubMed]
  • Siu AL, McGlynn EA, Morgenstern H, Beers MH, Carlisle DM, Keeler EB, Beloff J, Curtin K, Leaning J, Perry BC, et al. Choosing quality of care measures based on the expected impact of improved care on health. Health Serv Res. 1992 Dec;27(5):619–650. [PMC free article] [PubMed]
  • Thomson R, Lally J. Clinical indicators: do we know what we're doing? Qual Health Care. 1998 Sep;7(3):122–122. [PMC free article] [PubMed]
  • Enthoven AC. A promising start, but fundamental reform is needed. BMJ. 2000 May 13;320(7245):1329–1331. [PMC free article] [PubMed]
  • Wilkinson EK, McColl A, Exworthy M, Roderick P, Smith H, Moore M, Gabbay J. Reactions to the use of evidence-based performance indicators in primary care: a qualitative study. Qual Health Care. 2000 Sep;9(3):166–174. [PMC free article] [PubMed]
  • Baker R. Managing quality in primary health care: the need for valid information about performance. Qual Health Care. 2000 Jun;9(2):83–83. [PMC free article] [PubMed]
  • Cradock J, Young AS, Sullivan G. The accuracy of medical record documentation in schizophrenia. J Behav Health Serv Res. 2001 Nov;28(4):456–465. [PubMed]
  • Wyatt JC, Wright P. Design should help use of patients' data. Lancet. 1998 Oct 24;352(9137):1375–1378. [PubMed]
  • Wu L, Ashton CM. Chart review. A need for reappraisal. Eval Health Prof. 1997 Jun;20(2):146–163. [PubMed]
  • Luck J, Peabody JW, Dresselhaus TR, Lee M, Glassman P. How well does chart abstraction measure quality? A prospective comparison of standardized patients with the medical record. Am J Med. 2000 Jun 1;108(8):642–649. [PubMed]
  • Salzer MS, Nixon CT, Schut LJ, Karver MS, Bickman L. Validating quality indicators. Quality as relationship between structure, process, and outcome. Eval Rev. 1997 Jun;21(3):292–309. [PubMed]
  • Kahn KL, Park RE, Vennes J, Brook RH. Assigning appropriateness ratings for diagnostic upper gastrointestinal endoscopy using two different approaches. Med Care. 1992 Nov;30(11):1016–1028. [PubMed]
  • Phelps CE. The methodologic foundations of studies of the appropriateness of medical care. N Engl J Med. 1993 Oct 21;329(17):1241–1245. [PubMed]
  • Merrick NJ, Fink A, Park RE, Brook RH, Kosecoff J, Chassin MR, Solomon DH. Derivation of clinical indications for carotid endarterectomy by an expert panel. Am J Public Health. 1987 Feb;77(2):187–190. [PMC free article] [PubMed]
  • Kravitz RL, Laouri M, Kahan JP, Guzy P, Sherman T, Hilborne L, Brook RH. Validity of criteria used for detecting underuse of coronary revascularization. JAMA. 1995 Aug 23;274(8):632–638. [PubMed]
  • Selby JV, Fireman BH, Lundstrom RJ, Swain BE, Truman AF, Wong CC, Froelicher ES, Barron HV, Hlatky MA. Variation among hospitals in coronary-angiography practices and outcomes after myocardial infarction in a large health maintenance organization. N Engl J Med. 1996 Dec 19;335(25):1888–1896. [PubMed]
  • Ferlie EB, Shortell SM. Improving the quality of health care in the United Kingdom and the United States: a framework for change. Milbank Q. 2001;79(2):281–315. [PMC free article] [PubMed]
  • Donabedian A. The quality of medical care. Science. 1978 May 26;200(4344):856–864. [PubMed]
  • Lomas J, Anderson G, Enkin M, Vayda E, Roberts R, MacKinnon B. The role of evidence in the consensus process. Results from a Canadian consensus exercise. JAMA. 1988 May 27;259(20):3001–3005. [PubMed]

Articles from Quality & Safety in Health Care are provided here courtesy of BMJ Group


Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


  • PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...