NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Institute of Medicine (US) Roundtable on Evidence-Based Medicine; Olsen LA, Aisner D, McGinnis JM, editors. The Learning Healthcare System: Workshop Summary. Washington (DC): National Academies Press (US); 2007.

Cover of The Learning Healthcare System

The Learning Healthcare System: Workshop Summary.

Show details

8Structuring the Incentives for Change


A fundamental reality in the prospects for a learning healthcare system lies in the nature of the incentives for inducing the necessary changes. Echoed throughout this report are calls for incentives that are structured and coordinated to drive the system and culture changes, as well as to establish the collaborations and technological developments necessary to build learning into every healthcare encounter. Public and private insurers, standards organizations such as National Committee for Quality Assurance (NCQA) and the Joint Commission (formerly JCAHO), and manufacturers have the opportunity to shape policy and practice incentives to accelerate needed changes. In this chapter, representatives from private and public payer organizations, a manufacturer, and a standards organization give their perspectives on how the field might support specific and broad policy changes that provide incentives for systemwide progress. These perspectives engage only a sampling of the sorts of incentives to development of a learning healthcare system, but they represent important focal points for stakeholder alignment.

Alan Rosenberg of WellPoint offers a perspective from the health insurers industry on opportunities for encouraging both evidence development and application. For payers, incentives for change are structured around their three major functions: reimbursement, benefit plan design, and medical policy formation. Rosenberg gives several examples of past and current efforts by insurance providers to encourage evidence generation and application. Steven Phurrough discusses the Centers for Medicare and Medicaid Services’ (CMS’s) opportunities for influencing change through regulatory and payment processes but also notes that its leadership in implementing policies that assist in developing evidence might encourage similar efforts throughout the healthcare system. A very powerful approach taken by CMS in this respect is Coverage with Evidence Development (CED), in which reimbursement is conditional on the creation or submission of additional data. Phurrough describes recent revisions of CED that define two CED components: Coverage with Appropriateness Determination and Coverage with Study Participation, as well as the range of gaps in the evidence base that such policies may help to fill.

Wayne A. Rosenkrans and colleagues discuss the importance of providing incentives for appropriate evidence creation and appropriate use of evidence. Establishing standards of evidence and processes for reimbursement decisions would help manufacturers design better studies and work to provide the evidence needed. In addition, flexibility to account for individual variation needs to be built into the system. Evidence generated must be put to good use through thoughtful structuring of guidelines, decision support, and outreach to providers and patients. New models for product development such as evidence-based drug development will accelerate the creation of a learning healthcare system by making clear breaks with outdated approaches. For example, embedding the creation of effectiveness data into the process of drug development or developing a continuous process of evidence creation in partnership with the regulators and payers might provide sufficient financial incentives to drive change.

Margaret E. O’Kane discusses the important role that standards organizations have played in improving the quality of health care by measuring performance and identifies the significant barriers to extending this across the healthcare system: the absence of evidence from basic science, to complex comorbidities, subpopulations, and comparative effectiveness; the lack of usable guidelines; difficulty in obtaining data; and lack of accountability throughout the system. All are significant impediments to improving quality and efficiency of care. O’Kane calls for payment reform, regulatory reform, and liability reform as the key instruments to confront these challenges. System-wide change requires new ways of thinking about gathering, managing, and deploying knowledge and an evolution in our approaches.


Alan Rosenberg, M.D.

WellPoint Inc.

Private insurers have many opportunities to structure the incentives for change both in evidence development and in evidence-based application. This paper addresses both the private insurance industry as a whole and gives specific examples from my own organization, WellPoint Inc., which provides health benefits through its affiliated companies for 34 million Americans. The area to be covered is very broad, but this overview provides a sense of the opportunities that the private insurers have for collaboration with others in health care to build a learning health-care system for evidence-based medicine.

Three major functions of private insurers provide such opportunities: reimbursement and financing methodologies, including reimbursement to physicians, other practitioners, and healthcare institutions, can be and are currently used to structure incentives; claims data through their analysis can be and are used to support this environment; benefit plan design (BPD) and, within BPD, medical policy formation also can be and are used.

This paper also discusses two initiatives—data integration for electronic health records (EHRs) and consumer-focused initiatives—along with the methods used, the importance of collaboration, and some of the hurdles from the private insurer perspective in achieving these goals.

Private insurers have and will continue to fund significant components of evidence development and evidence-based applications (Blumenthal 2006). Certain of these are still evolving but worthy of special note. Several health plans now provide benefit coverage for individuals in National Institutes of Health (NIH) sponsored trials. The Centers for Medicare and Medicaid Services has recently embarked on a model of coverage based on participation in registries such as the implantable cardioverter defibrillator registry. Several insurers, including WellPoint, helped provide funding for the development of this registry. WellPoint also recently announced its collaboration, including economic support, with the Society of Thoracic Surgeons (STS), for its cardiac surgery registry. Use of the data from these registries to support evidence development and evidence-based care remains a priority. WellPoint uses such data, including data from STS and Joint Commission on Accreditation of Healthcare Organizations (JCAHO) registries to determine variable reimbursement for facilities in a variety of its pay for performance (P4P) programs. Payers recognize that evidence for and analysis of the effectiveness of P4P programs are still under evaluation (Rosenthal et al. 2005). At the same time, payers continue to fund this important and significant modification in traditional insurance-based reimbursement methodology. Today, P4P programs include enhanced reimbursement for physicians who perform better on a wide array of evidence-based measures including mammography rates, immunization rates, diabetic retinal eye exam rates, and rates of measurement of hemoglobin A1c levels. WellPoint has already paid to physicians and medical facilities more than 100 million dollars through these programs.

The second area for review is how we can collaborate with others to advance evidence development through claim data analysis. WellPoint databases have claims history for 34 million Americans and integrated pharmacy claims data on 17 million of these. Through analysis of this dataset, and by other insurers as well, post-market surveillance can be undertaken. WellPoint, with its affiliate HealthCore, did submit information to the Food and Drug Administration (FDA) on cyclooxygenase-2 (COX-2) inhibitor complications based on analysis of this dataset. Health-Core is currently working with a variety of academic research centers and specialty societies on projects using this dataset to evaluate comparative effectiveness including projects on cardiac care, controller effectiveness in asthma, and an analysis of blood growth factors. Some of the individuals and organizations contributing to this workshop summary have worked for years and developed sophisticated models of data analysis. Ongoing efforts and collaborations will continue to tap these valuable resources for evidence development. In addition to WellPoint’s initiatives, the Blue Cross Blue Shield Association (BCBSA) announced its plan to bring together an array of claims datasets that will eventually include claims data on more than 70 millions Americans. Unlike currently available CMS datasets, but similar to WellPoint’s, these data will include integrated pharmacy and medical claims data.

Claims data are also an important component in evaluating evidence application. While having many limitations, efficiency is a real advantage of claims-based quality measures. The National Committee for Quality Assurance has catalyzed and significantly supported this effort by the development of standards for quality and preventive health initiatives through its Health Plan Employer Data and Information Set (HEDIS) measurement set. This is an example of a long-term collaboration involving practitioners, insurers, and employers that has significantly moved this field forward. Insurers continue to use this and other nationally developed measurement sets to provide feedback to the practitioner community participating in networks. However the defined datasets remain limited, and ongoing development of broader, nationally recognized, consistently defined, and efficient-to-gather standardized measurement sets needs to continue. In addition to NCQA’s ongoing work, the majority of large insurers continue to actively engage in and support recent efforts by the AQA (previously known as the Ambulatory care Quality Alliance). Benefit plan design and, specifically, medical policy formation support evidence-based application. A key component of most health insurance benefit plans is that covered services are those that are medically necessary and services that are investigational are not covered. The benefit plans are legal contracts, and definitions of “medical necessity” and “investigational” vary, but they are routinely incorporated into these legal contracts. Therefore these definitions are at the core of insurers’ benefit determinations. Based on the benefit plan language, insurers can and do develop evidence-based models for those things that may be considered medically necessary and those that may be considered investigational or not medically necessary. Insurers use inputs from a variety of evidentiary sources, including literature-based review, governmental review bodies, Cochrane reviews, and other technology assessment entities. (See publicly posted coverage or medical policies on numerous BCBSA-affiliated entities and Aetna web sites.) In addition, WellPoint uses specialty society and academic medical center input regarding medical policy and medical necessity determinations, working actively with more than 30 specialty societies. This process is an active effort at developing evidence-based evaluation that includes input from the practice community on when the evidence, in conjunction with national practice patterns, has developed to support benefit coverage. Collaboration in this regard is essential. However, litigation continues to be a significant mitigation in the insurers’ ability to fully apply evidence-based medical necessity benefit determinations.

One significant initiative at WellPoint to better support the diffusion of claims information into practice is data integration for electronic health records. This is an effort to use our claim and pharmacy information datasets to support clinical practices. Through this initiative, WellPoint is making available claims-based data regarding diagnoses, procedures, and pharmaceutical information to emergency rooms on a real-time basis. This has already been undertaken in Missouri and is under way in California.

Insurers are actively involved in efforts to reengage their members through consumer-directed health plans. Initial evaluations regarding the effect of these programs have been published (Newhouse 2004). Some of the consumer-directed opportunities that these programs offer include disease management programs; web-based information, including films on conditions or procedures; mail-based information that is procedure and disease specific; web-based, paper, and personalized mailings with health information based on combinations of claim and pharmacy information; and safety information on drug recalls and new drug alerts.

Health insurers have excellent opportunities to support evidence development and evidence-based application. Private insurers also understand that we are only one part of the mosaic and that better value will be created through collaboration. This collaboration needs to occur across the industry and across the healthcare sector with manufacturers, with government, with academic and medical communities, and with our members to move healthcare delivery to a more evidence-based approach. Examples of collective opportunities include the following:

  • With government: develop national registries, support and conduct research, educate, develop national standards;
  • With the academic medical community: provide evidence-based care, conduct research, establish evidence-based guidelines, educate practitioners and patients;
  • With members and patients: strengthen demand for evidence-based along with preference-based care;
  • With manufacturers: conduct research, support full publication of data, innovate and bring to market health-enhancing products; and
  • With employers and insurers: enhance individual and physician decision support, integrate data sources, educate, develop performance-based reimbursement plans, and provide evidence-based coverage determinations.

There are also significant hurdles faced by insurers. One of the most significant is the litigation environment and the large dollar court settlements that do not align with evidence-based practices. For a bit of context, AHRQ’s budget is approximately $330 million dollars a year, but insurers have had and continue to have $50 million to $100 million dollar settlement costs for failing to provide benefit coverage for an intervention that has inadequate evidentiary support. The most broadly described has been autologous bone marrow transplantation for breast cancer, but new examples occur each and every year. Insurers recognize that we are not viewed favorably in terms of court cases and have very significant risk—this continues to make it difficult for us to maintain strong evidence-based decision making. The opportunity here is to develop some safe harbors built around a national evidence-based process similar to Britain’s National Institute for Clinical Excellence (NICE) (Rawlins 2004). Second, insurers recognize that we are not the healthcare sector that most people come to as a trusted partner for getting health information. These ongoing trust issues with the consumer and medical community make communications by us less effective. We are working hard to transform this perception through continued collaborations, but we recognize we have a way to go.

The last significant hurdle that must be addressed is cost. One important example of this is the discussion of conversion from the Ninth Revision of the International Classification of Diseases (ICD-9) to the Tenth Revision (ICD-10). Healthcare researchers generally agree that this will increase the availability of useful clinical information in claim databases. However the cost to systems for conversions is enormous. The ICD-9 to ICD-10 conversion cost to the health insurance industry alone (not including cost to physician practices or hospitals) is estimated to be between $432 million and $913 million, according to a Robert E. Nolan company report and subsequently by independent IBM reports. Therefore careful evaluation of the true value of these changes versus the total cost remains an ongoing imperative—each dollar spent on this is a dollar that cannot be spent on one of the other initiatives outlined.


Steve Phurrough, M.D., M.P.A.

Centers for Medicare and Medicaid Services

What opportunities does CMS have to develop policies that can provide incentives for the developments necessary to build learning—evidence development and application—into every healthcare encounter? Issues around trial design, rapid technology advancement, and healthcare culture are crucial and of concern to CMS, and have been discussed in earlier chapters. This discussion focuses on options that CMS has to influence these changes.

CMS influences changes either through its regulatory and payment processes or through its leadership role in the healthcare system. Over the last few years, CMS has attempted to use the latter more broadly, with its emphasis on public health and not just paying claims. This role is very important in itself but with better prospects when accompanied with payment or regulatory changes.


There are certain prerequisites to these policy changes. For all health-care experiences to become learning experiences will require some technologies not currently in place. This global population-level learning requires information technology (IT) changes that allow simpler, faster, and more complete data collection and exchanges. Telephonic and paper surveys, multiple data entries, chart reviews for data, and extensive data validations are all impediments to this learning experience. Solutions are complex, but the technology exists. A national standard for the operational architecture for electronic health records is an essential first step. Many facilities, clinics, and offices are hesitant to enter into EHRs, fearful that standards may change and expensive systems become obsolete. While CMS does not have responsibility for this, it is interested in being part of that conversation.

Standard definitions for medical terminology are also necessary. Clinical trials, registries, FDA post-market approval studies, and payers do not have consistent definitions of basic medical terms necessary to simplify data collection. An industry-sponsored registry for a particular device may pay for data on a particular patient that also has a requirement to have the same data submitted to a payer. The definitions of the data elements may differ, and hospitals spend significant time sorting out the differences. However, although the time issue is significant, comparing results from the two data collections may not be useful because of this.

The Department of Health and Human Services (HHS) has contracted with the American National Standards Institute (ANSI) in cooperation with the Healthcare Information and Management Systems Society, the Advanced Technology Institute, and Booz Allen Hamilton to develop a widely accepted and useful set of standards specifically to enable and support widespread interoperability among healthcare software applications. ANSI created a Healthcare IT Standards Panel (American National Standards Institute 2006) to assist in the development of the U.S. Nationwide Health Information Network (NHIN) by addressing issues such as privacy and security within a shared healthcare information system. This panel recently presented its first set of 30 harmonized standards to the American Health Information Community in October 2006. A number of officials from both the public and the private sectors voiced concerns about the standards. This initial effort demonstrates the difficulty of this task.

Much of the learning discussed in this Institute of Medicine (IOM) workshop requires collecting data in ways that may not have the same scientific base as the typical clinical trial methodological sciences. Health-care research has a fairly well thought out and well designed research methodology, but the research methodology for collecting, analyzing, and utilizing data in other manners—data from claims databases, registries, emerging death data—is much less clear. Increasing the opportunities to collect more information without a good scientific basis for how it will be used—and without broad acceptance in the healthcare community that it is valid evidence—is a fruitless endeavor. Good science behind the use of these types of data is needed.


Moving from prerequisites to policy options, CMS has several opportunities that could increase learning. Improving the information included with claims data could rapidly advance knowledge, particularly around safety and adverse events. Moving from ICD-9 to ICD-10 adds a greater level of detail for diagnoses and procedures than we currently have. The success of this option will require providers and coders to accurately select the more definitive diagnoses found in ICD-10.

Significant concerns have been raised about the ability of the healthcare industry to track adverse events from medical devices and to notify patients when those events occur or when a recall is initiated. If a particular implantable device is recalled, providers and hospitals have tremendous difficulties in identifying patients who have those devices. Payers, such as Medicare, have significant interest in ensuring that their beneficiaries are notified of the recall. That information is not available to payers. The FDA has begun discussions on an initiative to add unique product identifiers to medical devices, identifiable through bar coding, radio frequency, or other automated techniques. While this would allow easier recording of this information in hospital records, it still does not simplify notification of patients. Having data fields in the claims system that could be populated through the systems required by the FDA would allow payers to notify their beneficiaries when recalls or adverse events occur. In addition, device specific information on claims forms has the potential, when matched with subsequent claims data from patients’ interactions with the healthcare system, to identify other events not recognizable through the current reporting systems.

Using claims data is not without problems. Claims forms are payment instruments and are completed with the goal of maximizing payments. In some instances, this may result in a lack of clarity as to the major diagnoses or procedures involved in the care of the patient. Using claims data also makes risk stratification difficult and results in outcomes that do not reflect true practices.

Another opportunity that CMS has already employed is to create requirements that condition payment on the submission of additional data. The CMS quality programs are representative of this. Hospitals currently participate in several quality initiatives that were required as a condition of the annual inpatient rate increase. In other instances, CMS has provided additional payment for information—the oncology demonstration is an example of that. CMS is currently working with the public to develop appropriate quality measures for ambulatory services as well. While the collection of this type of information has become de rigueur for CMS and most health plans, it is still challenging to present this information in manner that is beneficial to the public. In early 2005, CMS released a draft guidance document on Coverage with Evidence Development that outlined a policy on requiring additional evidence as a condition of payment. Several National Coverage Determinations (NCDs) over the previous two years had used this option, and the 2005 guidance document outlined how this process would be used in the future. Following extensive public comments and public forums, CMS issued a revised document in the summer of 2006. This revised document answered many of the concerns addressed by the public. In essence, the guidance document outlined two CED components: Coverage with Appropriateness Determination (CAD) and Coverage with Study Participation (CSP).

CAD requires additional data collection to ensure that the item or service is being provided in a manner consistent with the guidelines outlined in the CMS reasonable and necessary determination. CAD will be required when CMS is concerned that the data collected on a claims form are insufficient to determine that the item or service was appropriately provided as outlined in the NCD. The following are some concerns that may lead to a coverage decision that requires CAD as a condition of coverage:

  • If the newly covered item or service should be restricted to patients with specific conditions and criteria;
  • If the newly covered item or service should be restricted for use by providers with specific training or credentials;
  • If there is concern among clinical thought leaders that there are substantial opportunities for misuse of the item or service; and
  • If the coverage determination significantly changes how providers manage patients utilizing this newly covered item or service.

CSP may be an option in a national coverage determination when CMS does not find sufficient evidence to determine that the item or service is reasonable and necessary under section 1862(a)(1)(A). This option would condition payment on participation of the beneficiary in a clinical research study outlined in the NCD. The following list includes some of the evidentiary findings that might result in CSP:

  • Available evidence may be a product of otherwise methodologically rigorous evaluations but may not have evaluated outcomes that are relevant to Medicare beneficiaries.
  • The available clinical research may have failed to address adequately the risks and benefits to Medicare beneficiaries of off-label or other unanticipated uses of a drug, biologic, service, or device.
  • Available clinical research studies may not have included specific patient subgroups or patients with disease characteristics that are highly prevalent in the Medicare population.
  • New applications may exist for diagnostic services and devices that are already on the market, but there is little or no published research that supports a determination of reasonable and necessary for Medicare coverage at the time of the request for an NCD.
  • Sufficient evidence about the health benefits of a given item or service to support a reasonable and necessary determination is available only for a subgroup of Medicare patients with specific clinical criteria and/or for providers with certain experience or other qualifications. Other patient subgroups or providers require additional evidence to determine if the item or service is reasonable and necessary.
  • CSP research conducted may include a broader range of studies than randomized clinical trials to include observational research. However, all studies must conform to the standards that will be developed by the Clinical Research Policy.

In rare instances, for some items or services, CMS may determine that the evidence is very preliminary and not reasonable and necessary for Medicare coverage, but, if the following criteria are met, CSP might be appropriate:

  • The evidence includes assurance of basic safety;
  • The item or service has a high potential to provide significant benefit to Medicare beneficiaries; and
  • There are significant barriers to conducting clinical trials.

The implantable cardioverter defibrillator (ICD) registry is one instance where payment for some beneficiaries was conditional on data submission to a registry. Hospitals implanting ICDs for primary prevention are required to submit several data elements to a national registry. This registry was initially managed by one of CMS’ Quality Improvement Organizations. However, this has now been transferred to the National Cardiovascular Data Registry managed by the American College of Cardiology and the Heart Rhythm Society. The first year of the registry resulted in data on almost 45,000 ICD implantations with a compliance rate greater than 95 percent. The initial review of these data produced results not seen in the large trials for these devices. In those trials, no deaths were recorded during the initial hospitalization for the implantation. In the registry, a number of deaths were recorded. This provides a cautionary warning to both patients and providers that ICD implantation is not without risk.

The current CED concept limits its use to those few instances in which we believe that the evidence does not support coverage but is suggestive of a benefit and some additional data collection supported by CMS might provide that additional evidence. While yet to be clearly delineated, there is the potential for this concept to be broadened to the extent that new technologies receive CMS coverage at an earlier time in return for continuous data collection over the life cycle of the technology and a more streamlined process for technologies to enter and leave the CMS coverage toolbox. This concept, somewhat similar to ongoing discussions at FDA, will require significant public discussion and potential legislative action.

CMS does have limits in its abilities to influence learning behavior. Its policy options are essentially payment options. We do think there is a leadership role for government; as we implement on a small scale a particular policy that we think will assist in developing more evidence, we think that particular leadership role will trickle down into other settings. In our current healthcare system, physician learning typically comes from training and experience. Patient learning comes from physicians or the Internet—a fairly inaccurate database. Movement toward a system in which both providers and patients are basing decision about their health care on good, reliable information is a challenge. It is a challenge that CMS and HHS are engaged in and we look forward to engaging the rest of the community in making this work.


Wayne A. Rosenkrans, Jr., Ph.D., Catherine Bonuccelli, M.D., and Nancy Featherstone, M.B.A.

AstraZeneca Pharmaceuticals

From the point of view of a pharmaceutical firm, the opportunities and challenges in moving toward a learning healthcare system fall into three categories: appropriate evidence creation, appropriate use of evidence, and evidence-based drug development.

Appropriate Evidence Creation

In evidence creation, policies should be pursued that encourage asking the right questions or, rather, avoiding asking the wrong questions. Specifically, for a question worth asking, we need to ensure that the methodology for answering that question is matched to the need and that we are applying the right level or standard of evidence to the question. The right question will depend on the severity of the health condition and the nature of the intervention. For serious life-threatening health conditions, there is a greater willingness to accept less than perfect information because critical decisions must be made by doctors and patients in a short time. In asymptomatic populations at risk for developing an illness, there is a greater desire for certainty before widespread use. Transparency in the question generation process between all stakeholders in health care is essential to the process.

Thus there is a dichotomy in the questions being asked and in the evidentiary standards used to answer each. Who defines what that standard is? Randomized trials frequently fail to find differences when there indeed are some. On the other hand, decision-modeling approaches frequently identify differences where there are none. How should this conundrum be engaged—what is the tie-breaker? When assessing the comparative effectiveness of therapeutic options, all of the evidence needed to fully assess the options is rarely available to decision makers; in fact it may never be available. At some level a decision must be made despite the lack of evidence. From the perspective of pharmaceutical companies, to generate the desired evidence we need to know what standards are going to be required. We have a long-standing set of standards with the FDA for demonstrating safety and efficacy, but lack standards of evidence for demonstrating effectiveness.

Clarification of the standards to be used to evaluate evidence is an area in need of development. For example, the overall size and scope of some Health Technology Assessment (HTA) guidelines reach so far beyond that of the Academy of Managed Care Pharmacy (AMCP) dossier requirements as to be virtually impossible to meet as an evidentiary standard, even if we were to spend substantial amounts of time and money. Not only that, but there is little consistency between the various HTA requirements. We also know that most health plans don’t conduct a thorough review of the AMCP dossier, the only standard we have today. What is the expectation here? We need a set of standards defining which evidence will be used for what purposes, transparency in the process of how evidence is used to make those decisions, and how that evidence translates into treatment guidelines and reimbursement policy.

Also, evidence by its nature is population based, but each individual patient responds differently to a given therapy. As we begin to recognize clinical differences in response to therapy, we need to work to identify the needs of patient subpopulations and individuals and tailor the delivery of care appropriately. There must be a process for exceptions or appeals of coverage rules that are established on population-based evidence, and this appeals process must be efficient and easy to navigate. We need to grapple with the challenge of tailoring population-based evidence to individualization of therapy. Physicians need to estimate the individual patient’s response when initiating therapy and to make adjustments based on actual patient response. Evolving knowledge around the molecular nature of disease leading to personalized, or stratified, medicine may provide an answer here. Utilizing sophisticated diagnostic technology to identify therapeutic responders offers the promise of bridging the evidence gap from populations to the individual.

Appropriate Use of Evidence

Most of the current dialogue is around creating new evidence, rather than how we use the evidence we already have or how we embed it into practice. However, if we did nothing more than consistently use the evidence that already exists, we could significantly improve patient care. Appropriate use of evidence can be addressed in two areas of focus: the individuals involved, namely the patient and the provider, and the tools available, namely decision support tools and reimbursement incentives. There are clear benefits to educating patients with evidence-based health communications. Educated patients are believed to be more likely to strive toward effectively managing their own health—if they are provided meaningful and relevant information. Several obstacles hinder both the effective communication of good evidence to patients and the patients’ subsequent action to embed that evidence into creating better health. The sheer volume and sometimes questionable quality of the information can work against us. In translating complex scientific material into usable content for patients, the patient’s own medical literacy affects how he or she receives and responds to information, and failure to take cultural or ethnic differences into account may also interfere with effective communication. Finally, people being people, there are behavioral issues related to different beliefs, attitudes, and social milieus. We need to better understand how patient behavior impacts response to evidence-based therapy recommendations in order to create more effective communication paradigms.

As the primary treatment decision makers, physicians are in the best position to embed evidence into clinical practice. The physician-patient dyad is the most important component of the healthcare continuum. While formal physician training provides doctors with a baseline education in effectively treating patients, the evidence base is constantly evolving, and “best practices” can change at an unbelievable pace. As with patients, the sheer volume of information is working against us, as are cultural and behavioral issues. We need to understand how to support continuous, efficient physician learning and how physicians respond to a variety of media. New strategies for how to communicate evidence to the practicing physician are required.

Health information technology (HIT) can enable healthcare professionals to access real-time patient records and compare them to thousands of other patients with similar profiles to determine treatment options with the greatest probability of producing the desired results. Adoption of computerized decision-support systems paired with electronic health records has been shown to improve patient compliance with clinical guidelines. Personal health records may provide a vehicle for patients to own—and understand—their own health status and data. However we need to be cognizant of several issues here as well. The use of HIT to the degree of sophistication at Cleveland Clinic, Intermountain Health, and the Veterans Administration (VA) is not yet widespread. The cost hurdles to embedding the kind of sophistication seen at the major medical centers in small-practice offices and hospitals are significant, and there continue to be issues with standards and vocabularies for information exchange. We also need to be careful about what is embedded in these decision support systems. We need to avoid embedding financially based decision criteria that are counterproductive to good care. The emphasis needs to be on improving the quality of care by gaining intelligence from data gathered from those for whom services and treatments have been delivered.

Probably the most effective way of driving evidence into the system today is to follow the money. Payers have as much if not more power to embed evidence than the physician does. Reimbursement decisions by payers can effectively embed evidence-based practice patterns into clinical practice through decisions to cover or not to cover a product, device, procedure, or service. A payer is often seen as a trusted authority on evidence assessment and thus able to influence physicians’ beliefs about best clinical practices. Payers have more resources than individuals and physicians to analyze large volumes of complex and sometimes conflicting information. Yet are the right questions being asked? Are the questions designed to assess treatment impact on health outcome or are they geared toward cost savings only? Were the correct methodologies used? There are both challenges and drawbacks to embedding evidence through reimbursement including provider incentives based on process efficiency rather than outcome of therapy.

Evidence-Based Drug Development

Pharmaceutical manufacturers spend most of their resources developing drugs to help people—an expensive undertaking. One estimate (DiMasi et al. 2003) found that it took an average of $802 million to develop a drug—with another $95 million in post-launch costs. Surveying a group of leaders in research and development (R&D) recently suggested that this number may have increased to $1.4 billion. Adding the costs of doing effectiveness research on top of doing research on efficacy and safety (estimated from internal costs for outcomes and epidemiological trials) may lead to a total between $2 billion and $2.3 billion to develop a new drug. The additive costs of comparative outcomes research extend beyond economics. The larger patient populations required to adequately power these studies significantly increase the resources needed, including logistical and manufacturing issues. The sterile environment of the traditional clinical trial is needed to reduce bias and other confounding elements, but the traditional clinical trial doesn’t reflect real-world clinical use. These costs are unsustainable and we are coming to a tipping point rapidly in being able to afford innovation.

One of the things that could change is the way that we develop new medicines: we need new models, and one possibility is a concept of evidence-based drug development—embedding the creation of effectiveness data in addition to efficacy and safety data into the process of drug development rather than as an afterthought. We do drug development episodically today—Phase I, II, III, regulatory approval, and launch. Generating evidence of effectiveness largely occurs in Phase IV if at all. Could we think about changing this episodic process to something that looks very different, a more continuous process of evidence creation generating an ever-increasing databank of safety, efficacy, and effectiveness in partnership with not only the regulators but the payers as well? FDA’s critical path, the NIH Roadmap, and CMS’s Quality Roadmap are all beginning to align along this need for change, but much remains to be done in the regulatory and reimbursement arenas to generate sufficient financial incentive and drive the change.

Technologically, the development process should leverage information from qualified biomarkers, surrogate markers, and development simulations, in concert with regulatory authorities, prior to first patient exposure. Once available to some patients, use of the evolving EHR systems at major payer and academic medical centers can begin a process of continuous data capture and simulation confirmation in broadening patient numbers. At a point agreed between the innovator, regulator, and possibly the payer there is an initial release where the innovator can begin to recoup investment cost in a proscribed patient base while additional data accumulate until there is a full release of the new therapy. This is a very different way of doing drug development, a concept that drug development never stops, or maybe the learning development system.

It is not possible for any one stakeholder to solve these issues, and if we are going to make evidence-based medicine work for our healthcare system we all need to work together. This includes the pharmaceutical and biotech industry, provider organizations, government agencies, the diagnostic and device companies, payers, and technology providers. As we work through this new world for health care we need to keep the patient in the middle of our dialogue, which must be about putting patient health first.


Margaret E. O’Kane

National Committee for Quality Assurance

Foundational to any exploration of evidence development in health care is a restatement of the central mission of health care: to improve health. In many ways we have moved too far away from this mission and pursued a series of tangential goals that, while important, have led us to lose sight of where we are going. If we are to achieve the mission, healthcare regulators, accreditors, purchasers, consumers, and providers must build their efforts around this larger goal.

Today, the healthcare quality movement is being held hostage to ever-rising costs and the failure to manage evidence effectively. Our current payment system is irrational, wasteful, and inequitable by any measure; as a result, healthcare costs are being driven to the boundaries of society’s ability and willingness to pay. Over the last six years, the average cost of health insurance for a family of four Americans has jumped 87 percent, a larger increase than any other sector of our economy (Kaiser Family Foundation 2006). Healthcare costs have eaten into corporate profits and family incomes. Even in 2005, when insurance inflation dropped to 7.7 percent, that increase and the accompanying shift of costs to employees ate up much of the average 3.8 percent rise in Americans’ wages (Kaiser Family Foundation 2006). Until we deal with costs, our efforts to improve quality will always take a back seat. Yet we know that these two issues are intricately entwined. Poor-quality care can cost more than high-quality care. If a patient gets the right care at the right time for the conditions that ail her, there will often be less need for more expensive, high-cost treatment of complications in the future. High-quality care also leads to greater productivity on the job and healthier and happier families at home.

Addressing the Evidence Gaps

We need to think in terms of populations. If we really think clearly about how health care affects populations, we will all be better off. In their impressive book Epidemic of Care, George Halvorson and George Isham presented us with a visual representation of population-based health that offers a way of thinking about the health of the American public. It reflects the continuum of health of the population of the United States, of a state, of a city, of a community, or even of a health plan. As time goes by, however, some of those people move into “at-risk” or “high- risk” or “early-symptom” categories. This representation has been adapted in Figure 8-1 to illustrate how we need to keep healthy people healthy, and to identify those who are at risk and move them back to the healthy stage through treatment and behavioral modification. As a population moves further toward high risk and active illness, we must give people the appropriate care and support to keep them as healthy as possible. Finally, when people are living with active disease or are at the end of life, we need to help them manage their illness, coordinate their complex care, alleviate their pain, and assist them in dying with dignity. We also know that 80 percent of healthcare dollars are spent on the 20 percent of the population who are in this last stage of health. Sadly, much of that care is of poor quality or unnecessary and neither prolongs nor improves life. If we can reduce the amount of money we spend there—through improved quality in earlier stages and a more rational use of care in the latter stages—not only could we help address our overall cost problems but we could redirect some of that money toward the earlier stages of the continuum that provide a better payback in health for the healthcare dollar.

FIGURE 8-1. A value-based healthcare system moves people to the left—and keeps them there.


A value-based healthcare system moves people to the left—and keeps them there. SOURCE: Essential Guide to Health Care Quality, NCQA 2007.

NCQA has adopted this framework to guide our efforts to accredit health plans and measure the performance of healthcare providers. We have developed a paradigm that holds the plan accountable for the quality of care and service provided to its members. The plans, in turn, often hold providers accountable for delivering care that is evidence-based and of high quality. In the health plan accreditation process, NCQA assesses the performance of health plans in three ways. First, we establish standards for structural and procedural activities meant to ensure patients’ rights, appropriate utilization management, quality improvement, customer service, and access to care. Second, we require plans to measure and report on their performance through HEDIS, a set of standardized, evidence-based measures. This unique approach allows us to evaluate how often patients receive the right care at the right time. Finally, we ask patients about their experiences with their health plans and providers. The Consumer Assessment of Health Plans and Providers (CAHPS) examines both a plan’s direct administrative performance and members’ experience with the delivery system. Together, these three parts combine to form an accreditation program that has become a model for public and private oversight of health care. It is the only accreditation program in health care where performance measurement drives a substantial proportion of the scoring.

Clearly, when one considers the list of HEDIS measures (Table 8-1), there are many important aspects of care that are being measured. It is important to note that for what we measure, there have been notable gains for patients, purchasers, and the health plans that report. NCQA has documented enormous improvements on the issues that it does measure. For example, the use of beta-blockers has increased substantially over the last decade. Health plans are now reporting 95 percent of patients being discharged from the hospital after a heart attack or major cardiac procedure on a beta-blocker, up from 69 percent in 1996 when NCQA began measuring this (National Committee for Quality Assurance 2006). Children are nearly three times as likely to have had all their recommended immunizations as they were in 1997. Diabetics are twice as likely to have their cholesterol controlled (< 130 mg/dL) as in 1998. These are just a few examples. We are extremely proud of the ground that we broke here because we know that there are people alive and healthier today because of this improvement (Figure 8-2). This transparency and accountability for quality is something really worth promoting—because we know it works. However, there are also many aspects of care that are not being measured, and this is where we need to focus as we move forward.

TABLE 8-1. HEDIS Effectiveness of Care Measures.


HEDIS Effectiveness of Care Measures.

FIGURE 8-2. Selected HEDIS improvements: 1999–2004.


Selected HEDIS improvements: 1999–2004.

Despite our progress on accountability, significant barriers remain to robust measurement of performance across the healthcare system. These include the following:

  • The absence of evidence. Despite tremendous growth in funding of biomedical research in the United States and abroad, there are significant gaps in our evidence base. The process of managing the evidence base toward a return on investment in health can only be described as haphazard. Often research is conducted in areas that are already well understood, while areas that may be clinically important remain uncharted. As a result, in many areas, we simply do not know what constitutes effective care. There are many examples, such as the basic science behind effective treatment for esophageal cancer and the appropriate care for patients with multiple conditions. We lack clinical trial data for the elderly. We do not know about the comparative effectiveness of many treatments so that we cannot evaluate how treatment “A” compares to treatments “B” or “C” in terms of risks, outcomes, and costs. David Eddy has mapped the areas of what we know in diabetes and heart care, and he uses the metaphor “islands of knowledge.” We have some islands of knowledge, supported by repetitive and redundant trials, while there remains a vast sea of ignorance that is unexplored. There is a tremendous need to proactively manage the evidence base. The funders of research must make sure that we are being strategic on where we need to develop more evidence. So it isn’t just turning research into practice, it is figuring out where there are basic research gaps. The best way to identify priorities would be to look at issues that affect large numbers of patients with a high cost of disease burden and expense and where practice is not solidly evidence based. Issues could be prioritized using quality-adjusted life years per dollar spent. Finally, evidence-based guidelines for patients with a combination of conditions are needed, but randomized clinical trials rarely address more complex patients.
  • The absence of care guidelines. Even where the evidence exists, there is often a failure to convert it into usable guidelines because we have a significant failure to develop consensus about the evidence. Medical specialty societies develop their own guidelines, often producing conflicting guidelines for the same patients, and this creates a lot of confusion. The model of specialty societies developing their own guidelines without some kind of overall organizing structure needs rethinking. For example, orthopedists, neurosurgeons, and internists disagree on when to do surgery for back pain. As a result, there is no cohesion in approaches or agreement on what is appropriate. We also have some guidelines that do not go far enough. For example, the guidelines for screening for depression or cholesterol levels in the general population do not specify the appropriate intervals. Here again, however, the evidence base that could determine the appropriate guideline has not been adequately marshaled to do so.
  • The lack of or difficulty in obtaining data. There are also important barriers to getting the data to assess performance. NCQA had to abandon an otitis media measure because health plans were unable to find the data in patients’ medical charts. This is true in many other areas.
  • The lack of accountability. Appearances to the contrary, only a small part of the current healthcare system is being held accountable through quality measurement and reporting. Among health plans, only health maintenance organizations (HMOs) and point of service (POS) plans are called on to report HEDIS. Yet those plans account for only a third of all Americans enrolled in private health plans. Only a handful of preferred provider organizations (PPOs), which cover 64 percent of privately insured Americans, collect and report HEDIS (National Committee for Quality Assurance 2006).

Another significant barrier is the relative absence of performance measurement at the provider level. Some progress has been made in recent years in developing and implementing hospital focused measures. The Joint Commission on Accreditation of Healthcare Organizations, the Leapfrog Group, and the Hospital Quality Alliance have each developed measures that are now in broad use. There has been less progress on the physician side of the ledger. NCQA, working with the American Medical Association and its Physician Consortium for Performance Improvement, has developed physician-level HEDIS measures for certain conditions and has been working in partnership with CMS) to develop additional measures at the specialty level. Much more work needs to be done before we are even close to a system of physician-level accountability.

NCQA has developed a set of programs designed to recognize high-performing physicians in diabetes and heart or stroke care, as well as practices that utilize clinical information systems to improve care across the board. These recognition programs also form the basis for several pay for performance initiatives operating around the United States, including the Bridges to Excellence program sponsored by large employers and the Integrated Healthcare Association program in California that works with large medical groups. NCQA also has been working with a variety of partners to develop a new CAHPS survey for clinician groups.

Measuring at the physician level presents other challenges as well. When it comes to individual doctors, we have a limited, insufficient sample size to assess performance. Yet more than half of American doctors practice in one- or two-member practices. Except for some proceduralists, individual physicians simply do not treat enough patients with a particular condition to draw valid conclusions. We need to recognize these limitations and factor them into our planning and implementation in order to safeguard the integrity of the measurement process.

The advent of electronic health records will help decrease the burden and expense of data collection—but only if we get the right kind of EHRs. The current generation of EHRs leaves much to be desired in terms of their usefulness to improve quality. Partly, this is because processes in health care are often unique to the practice microenvironment and, to some extent, standardized software requires standardized process. Perhaps the biggest potential for EHRs is their theoretical ability to provide customized reminders and warnings for each patient based on specific medical history, diagnoses, and situations. Comprehensive EHRs can also support appropriate decision making, but the current generation of products generally does not have these capabilities. In fact, many current EHRs don’t even have the capability of producing performance data after the fact.

How Can We Advance?

If we are to achieve a system that is truly accountable for quality, we must overcome the politics, the warring measures, resource use-cost measures, and political opposition that stand in the way of comprehensive quality assessment. We also need to address issues of appropriateness. Another problem that must be addressed is unclear accountabilities. There needs to be some entity or organization larger than the individual provider that is ultimately responsible for coordinating care across boundaries and practice settings and taking responsibility for at-risk patients after discharge. Once again, the sickest 20 percent of patients account for 80 percent of medical costs and there are important opportunities to improve quality and reduce costs with this group.

As we attempt to move practice toward a stronger evidence base, it is crucial that we chart an approach to comprehensive evidence management in order to exercise responsible stewardship for scarce healthcare resources. This will require cooperation across stakeholder groups and specialty and disease organizations, as well as substantial public funding. We need a comprehensive function of evidence management (Box 8-1). This enterprise would identify the gaps, encourage priority research areas, and promote paradigms of continuous learning and evidence development.

Box Icon

BOX 8-1

Comprehensive evidence management will do the following. Identify the gaps Encourage priority research areas

In order to have the greatest impact, consider the establishment of an evidence stewardship board to do three things: (1) evaluate the current medical evidence and publish guidelines; (2) identify priority areas for new evidence development; and (3) consider ways to move beyond clinical trials in an era when much more real-time information can be gleaned from practice and used to study what works. Such a board would be analogous to the U.S. Preventive Services Task Force, but covering the broad scope of medical practice.

There are four basic policy enablers and each must play a role to drive systematic improvements in health care (Box 8-2).

Box Icon

BOX 8-2

Policy Enablers Needed. Payment reform Regulatory reform

First, we must change the way that we pay for health care in this country. Payment reforms such as pay for performance, and other current models, are a good beginning, but they are generally based on fee-for-service payment. With new technology we all know that the old models of care, even some team-based care, are hopelessly inefficient. We must stop providing financial rewards for what we know is inefficient and ineffective medicine. The move to Diagnosis-Related Groups (DRGs) in the early1980s and the Resource-Based Relative Value Scale (RBRVS) in the early 1990s were significant advances, moving us away from true fee-for-service toward a system that is based more on the average cost of performing a group of services. It is time for a new generation of payment reform that utilizes bundled payments with strong incentives for efficient delivery of high-quality care. This kind of payment demands true quality transparency at the same time.

Second, we need regulatory reform. The role of policy makers is to encourage promising new models of care; figure out which ones are taking us toward higher-quality, affordable health care; and then push relentlessly to make the whole delivery system move in that direction. Old and outmoded regulatory requirements need to be dropped as new and more powerful techniques are adopted.

Third, liability reform is also part of the change management process. The current litigious environment is an impediment to good medicine, which requires a special partnership between patients and their healthcare providers. We need to make a new deal with society. We need to modify our practices going forward, and in return, we ask that we have some protection as we seek to improve the delivery system.

Finally, we need real patient activation. It is critical that we recognize that patient-centered health care requires that we do more than make data available or even provide it to patients and expect them to do the right thing. A very important area that needs to be more fully explored is determining the best ways to motivate patients. We need to take a comprehensive look at how both providers and the health system as a whole can help people make healthier choices, without creating dependency or increasing costs. We need to address issues such as health literacy and the need for culturally and linguistically appropriate services in order to communicate with patients in a manner that they understand and that motivates them to make lifestyle changes to improve their health and play an active role in managing their conditions.

At the end of the day, our purpose in quality measurement is not to derive knowledge in order to deliver it back into the same process. We will only maximize the benefits of performance measurement by addressing the structural elements that must be in place for systemic improvement. We need to find different ways to gather information in the first place; then transformation becomes not only desirable but almost unavoidable. As we explore new models for knowledge development, there are important considerations for how to effectively use them. Also very importantly, regulators, accreditors, and certifiers must work hard to evolve their approaches to accountability to support and encourage and not stifle knowledge development. Old paradigms based on assumed relationships between structure and outcomes need to be either justified or replaced with more evidence-based constructs if regulators, accreditors, and certifiers are to be drivers and enablers of progress rather than impediments to it.


  1. American National Standards Institute. Healthcare IT Standards Panel. 2006. [accessed April 4, 2007]. Available from http://www​​_activities​/standards_boards_panels/hisb/hitsp​.aspx?menuid=3.
  2. Blumenthal D. Employer-sponsored insurance—riding the health care tiger. New England Journal of Medicine. 2006;355(2):195–202. [PubMed: 16837686]
  3. DiMasi J, Hansen R, Grabowski H. The price of innovation: new estimates of drug development costs. Journal of Health Economics. 2003;22(2):151–185. [PubMed: 12606142]
  4. Kaiser Family Foundation. Employer Health Benefits Annual Survey. 2006.
  5. NCQA (National Committee for Quality Assurance). State of Health Care Quality Report. 2006.
  6. NCQA (National Committee for Quality Assurance). Essential Guide to Health Care Quality. 2007.
  7. Newhouse J. Consumer-directed health plans and the RAND Health Insurance Experiment. Health Affairs. 2004;23(6):107–113. [PubMed: 15584103]
  8. Rawlins M. NICE work—providing guidance to the British National Health Service. New England Journal of Medicine. 2004;351(14):1383–1385. [PubMed: 15459297]
  9. Rosenthal M, Frank R, Li Z, Epstein A. Early experience with pay-for-performance: from concept to practice. Journal of the American Medical Association. 2005;294(14):1788–1793. [PubMed: 16219882]
Copyright © 2007, National Academy of Sciences.
Bookshelf ID: NBK53501


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (2.0M)

Related information

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...