• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Institute of Medicine (US) Roundtable on Value & Science-Driven Health Care. Learning What Works: Infrastructure Required for Comparative Effectiveness Research: Workshop Summary. Washington (DC): National Academies Press (US); 2011.

Cover of Learning What Works

Learning What Works: Infrastructure Required for Comparative Effectiveness Research: Workshop Summary.

Show details

1The Need and Potential Returns for Comparative Effectiveness Research


Understanding the comparative effectiveness of candidate interventions or similar conditions is essential for improving the development and delivery of effective health care. Many publications have highlighted the shortfalls of health care delivered in the United States, including the frequency of medical errors (IOM, 2000); wide variation in practice patterns, driven more by services available than medical needs (Fisher and Wennberg, 2003); the slow translation of research discoveries into medical practice (Balas and Boren, 2000; Woolf, 2008); the limited quality of the evidence developed to guide healthcare decision makers (Atkins et al., 2004; Califf, 2004; IOM, 2008a; Tunis and Pearson, 2006); and the adverse consequences of care administered with adequate evidence (IOM, 2001). While each highlights a different problem or concern with the current healthcare system, collectively these findings reveal systemic inadequacies in current approaches to developing evidence to help guide the health decisions of policy makers, physicians, and patients.

Underscoring the pressing need for better insights into the relative effectiveness of therapeutics and treatments are the rising and unsustainable costs of health care and the relatively low returns for those high-cost investments. In 2009, spending on health care totaled $2.5 trillion, or over 17 percent of the nation's gross domestic product. Healthcare costs are becoming increasingly burdensome, with annual out-of-pocket costs to consumers steadily increasing (KFF, 2009). Some experts suggest that medical costs due to illness and injury contribute to a significant proportion, perhaps half, of bankruptcies filed by American families (Himmelstein et al., 2005). The Congressional Budget Office estimates that if left unchecked, health expenditures will rise to 25 percent of the gross national product by 2025 (CBO, 2007).

Developing and using information on which treatments work best for whom is imperative to achieving better value from national healthcare expenditures. Of the more than $2.5 trillion spent in 2009 on health in the United States, available estimates indicate that less than one-tenth of 1 percent has been devoted to such research (AcademyHealth, 2005; Moses et al., 2005). Recently, policy makers have demonstrated substantial interest in comparative effectiveness research (CER) (Jacobson, 2007), with attention and discussion focused on increased funding and on the structure, placement, and governance of an entity or agency charged with developing CER information (Kupersmith et al., 2005; Wilensky, 2006). With the passage of the American Recovery and Reinvestment Act of 2009, $1.1 billion were made available to the National Institutes of Health (NIH), the Agency for Healthcare Research and Quality (AHRQ), and the Secretary of Health and Human Services for the conduct of CER and to encourage data resource development and use for such analyses.1 These funds provided an important down payment on efforts to move to a system focused on delivering high-value care and driven by the best evidence, and formal recommendations have been made by the Institute of Medicine (IOM) (2009) and the Federal Coordinating Council for CER (FCC, 2009). With the 2010 passage of the ACA, and establishment of the Patient-Centered Outcomes Research Institute (PCORI), the capacity for sustained investment has developed. Appendices C, D, and E offer additional background.

The infrastructure needed to expand capacity for CER extends beyond developing data resources (e.g., registries, databases, data networks). Innovative research strategies are needed to improve the efficiency and relevance of clinical research as well as to ensure the appropriate translation and use of CER information by decision makers. Consideration is also needed of how best to align the substantial promise offered by health information technologies—to gather and disseminate needed data and information—with the needs of CER. These technologies offer opportunities to reduce costs and improve the quality of health care (e.g., e-prescribing, remote monitoring, public health records, electronic health records [EHRs]) and will increase access to new types of data and modes of communication (Litan, 2008). Adopting such innovations requires infrastructure development. Careful investments in the requisite workforce, systems, and technologies can also enhance the nation's capacity to learn from health care delivered.

Consideration of such long-term strategies as well as the identification of areas where appropriate investment and coordination will enable immediate progress were the focus of the July 30–31, 2008, workshop, Learning What Works: Infrastructure Required for Comparative Effectiveness Research. The meeting's discussions were motivated by many of the issues discussed above and the resulting need to explore key elements and opportunities for infrastructure development (see Box 1-1 on p. 64).

Box Icon

BOX 1-1

Issues Motivating the Discussion. Substantial demand for greater insights into the comparative clinical effectiveness of clinical interventions and care processes to improve the effectiveness and value of health care. Expanded interest and activity in (more...)

Expanded capacity for comparative effectiveness research can provide information and insights helpful to important care decisions of patients, providers, and policy makers, but progress will require informed and careful investment in key infrastructure elements. Summarized below are possible implications of CER for healthcare stakeholders, an assessment of activities under way, and options to enhance national CER currently under consideration. The workshop's two keynote presentations offered additional perspectives on infrastructure needs by describing a long-term vision and potential returns for a healthcare system informed by CER.

Mark B. McClellan reflects on the core elements of a robust and sustainable capacity for CER and how these immediate needs might also fit into a long-term strategy to support the functions of a learning health system. Noting that form should follow function, he outlines four key evidence gaps that should inform infrastructure development: (1) baselines for evaluations, such as disease models and natural histories; (2) safety; (3) comparative effectiveness of interventions; and (4) comparative effectiveness of treatment strategies and practice patterns. Efforts should focus on all of these areas that fall short in order to develop a healthcare delivery system that provides better outcomes for each kind of patient at much lower cost. Gail R. Wilensky notes that the potential returns from increased investment in CER are enormous, and she offers some suggestions on the elements required for progress, including establishing a center charged with creating better information. It will also be important to develop and use the approaches, data resources, and analyses most useful to producing the information needed and to recognize that all stakeholders need to be a part of the decision-making process.

Background material for the workshop was assembled and prepared by staff of the IOM's Roundtable on Value & Science-Driven Health Care, founded in 2006 to provide a trusted venue for major healthcare stakeholders to consider and advance their mutual interests in the enhanced development and use of evidence in health care. The Roundtable has defined science-driven health care broadly to mean “to the greatest extent possible, the decisions that shape the health and health care of Americans—by patients, providers, payers, and policy makers alike—will be grounded on a reliable evidence base, will account appropriately for individual variation in patient needs, and will support the generation of new insights on clinical effectiveness” (IOM Roundtable on Value & Science-Driven Health Care, 2009). An expanded capacity to develop evidence on the comparative benefits and risks of healthcare treatments and strategies is an essential step toward a learning health system, and it has been the principal focus of the Roundtable's working group on sustainable capacity. At the working group's request, in 2007 IOM staff authored an issue overview white paper, Learning What Works: The Nation's Need for Evidence on Comparative Effectiveness in Health Care (IOM, 2007). This background brief also provided context for the July 30–31 workshop, Learning What Works: Infrastructure Required for Comparative Effectiveness Research, and was included in the meeting's briefing materials. It is summarized below and included in full in Appendix A.


, , , , , and .

IOM Roundtable on Value & Science-Driven Health Care staff

A core objective for the nation is achieving the best health outcome for every patient. This objective cannot be accomplished until better evidence is available upon which to base healthcare decisions and until existing knowledge is applied more effectively. Each need is vitally important. It is known, for example, that failure to deliver proven interventions is a substantial challenge to the quality of health care for Americans, and it is a key concern of the IOM Roundtable on Value & Science-Driven Health Care (IOM, 2007). Yet, with the current pace of change, the most rapidly growing problem is the healthcare system's inability to produce the needed evidence in a timely fashion. Medical-care decision making is now strained, at both the level of the individual patient and the level of the population as a whole, by the growing number of diagnostic and therapeutic options for which evidence is insufficient to make a clear choice. The consequences can be seen in the broad geographic variation in the intensity of services delivered for the same outcome, in the occurrence of medical errors, in patient and provider confusion about which interventions deliver the most value, and in the costs of care.

A testament to innovation is the fact that new pharmaceuticals, medical devices, biologics, and procedures are introduced constantly, and the pace is quickening. From 1991 to 2003 the number of medical device patents per year doubled, and biotechnology patents tripled. Between 1993 and 2004 there was an 80 percent increase in the number of prescriptions received by Americans. A recent review suggests that half or more of the growth in medical spending in recent years is attributable to changes in technology.

In addition to the growth in application of drugs, devices, biologics, and procedures, the world of health care is about to experience dramatic new insights into the genetic variation in individual responses to different diagnostic and treatment interventions (AdvaMed, 2004; Biotechnology Industry Organization, 2006; Foster et al., 2002; Gelijins and Rosenberg, 1994). The age of personalized medicine will soon be a reality, if the capacity can be developed to contend with these insights. Today the average clinical encounter already requires a health provider to manage more variables than would be considered reasonable given what is known about the capabilities of the human mind. Over the next decade, that same encounter will require contending with perhaps an order of magnitude more complex (IOM, 2007).

These developments hold fundamental implications for health prospects, and, to capture and use them effectively and efficiently, a proportionate commitment is required to understand their advantages and appropriate applications. It is both a capacity investment and a resource allocation problem. Of the nation's more than $2.5 trillion in 2009 health expenditures, only a tiny fraction was devoted to CER. If only 1 percent of the nation's healthcare spending were devoted to understanding the effectiveness of the care purchased, the total for effectiveness research would come to approximately $20 billion annually—about 10 times the amount in 2009. In contrast, even accounting for the support from all private and public sources, the aggregate national commitment to assessing the effectiveness of clinical interventions is far below the standard that any company would expect to invest in work to evaluate and improve its products.

Regardless of individual perspectives on reform of the many challenging issues in health policy today, there is little question about the critical need for patients and providers to have better information with which to make their decisions about the comparative advantages of healthcare options. What follows is a summary of the issues and options and is intended to inform discussions of how to proceed on this matter of central importance to health and health care. It does not provide recommendations.

Implications for Stakeholders

For patients, the stakes are very clear. Every patient should be able to feel confident that there is solid evidence that the care received is the care most appropriate to the circumstances. Yet, increasingly this is not the case. In a 2005 survey, 60 percent of Americans said they didn't believe that the United States had the best healthcare system in the world, 41 percent said they knew of a time when they or a family member had received the wrong care, and 56 percent said there should be more investment in clinical and health services research. Health providers feel similar tensions. No health professional should be put in the position of uncertainty about the evidence in support of the care provided at his/her behest. Yet, with the pace of advances in medical procedures, pharmaceuticals, devices, and biotechnology, a sometimes confusing array of choices is presented for patients, their healthcare providers, and the healthcare organizations in which care is delivered. The integrity and reputation of healthcare delivery organizations is dependent on their ability to ensure the quality and appropriateness of the care delivered within their walls. Any decision support system is only as good as the information built into the model and should include the comparative advantages or disadvantages of different diagnostic and therapeutic options.

Healthcare manufacturers, focused as they are on returns on investment, inherently understand the importance of improving the value proposition in patient care. But their stakes go deeper. Manufacturers directly bear the economic burden of delays and inefficiencies when information is not available about the advantage of their products, not to mention the challenges of public and shareholder backlash when problems are identified too late. Without a sizable improvement in our evaluation capacity, the slower pace of understanding how and when interventions work best will retard the application of innovative treatments.

From a purchasing perspective, the need for better information is of central importance to those who pay for health care: patients, employers, insurers, and the government. Over half of the nation's health expenditures are borne by the private sector, including a sizable share by employers. For the fourth consecutive year, chief executive officers of U.S. companies have cited healthcare costs as their number one economic concern. Employers now pay 78 percent more for health care than 5 years ago, and it has been suggested by some that this increased financial burden makes it more difficult for American companies and workers to compete in the global marketplace. Often acting on behalf of employers, insurers represent the front line of the economic choices that have to be made about payment for healthcare services. This means drawing conclusions about the comparative advantages or disadvantages of proposed diagnostic or treatment interventions in the face of a paucity of such information, especially information applicable to real-world circumstances. As a payer, the government accounts for about 45 percent of health expenditures in the United States, including care that it delivers directly in its own facilities. Whether as a payer or a provider, the government has a central interest in ensuring that its clients receive the care that is most appropriate and of the greatest value.

Current Activities in Clinical Effectiveness Research

Currently, activities to assess the effectiveness of healthcare interventions are broad but underresourced and fall far short of the need (IOM, 2007). CER can be described as either primary or secondary. Primary refers to the direct generation of evidence through the use of a specific experimental methodology. Secondary refers to the synthesis of evidence from multiple primary studies in order to draw conclusions for practice. Within the overall umbrella of CER, the most practical need is for studies of comparative effectiveness, the comparison of one diagnostic or treatment option to one or more others (Wilensky, 2006).

The largest investment in CER has been made by industry, with industry-sponsored clinical trials representing a significant portion of health manufacturer investments in research and development (R&D). For example, about 40 percent of pharmaceutical R&D investments goes to the phase 3 and phase 4 trials, which have particular relevance to clinical effectiveness (PhRMA, 2006). Many of these studies are conducted with academic investigators, and others are managed by contract research organizations. Relatively few of the studies are comparative, or head-to-head, studies.

Outside of industry, several government agencies support CER, including AHRQ, which has a specific mandate and a small appropriation for CER. In 2005, the total appropriations to all federal agencies—the NIH, the Veterans Health Administration, the Department of Defense, the Centers for Medicare & Medicaid Services (CMS), the Food and Drug Administration (FDA), AHRQ, and the Centers for Disease Control and Prevention—for all health services research amounted to about $1.5 billion, and only a modest portion of this was devoted to clinical effectiveness research, far below the industry level. Additional work, also modest, is undertaken by certain of the larger healthcare delivery organizations. Evidence synthesis activity is supported by the insurance industry, professional societies, healthcare organizations, and government. AHRQ has established a network of 13 AHRQ-sponsored evidence-based practice centers that review literature and produce evidence reports, including comparative effectiveness reviews. Organizations interested in evidence reviews will often draw upon syntheses performed by several well-established technology assessment entities (IOM, 2008).

Activities and Needs Related to Comparative Effectiveness Research

Although there is a great deal of interest and activity surrounding the topic of clinical effectiveness, the aggregate research capacity is very thin, and the products fall substantially short of the need. Because of the scant resources available for the support of primary CER—head-to-head studies—much of the work is, of necessity, secondary evidence synthesis. Yet the most pressing needs that clinicians and their patients have are for reliable studies upon which to base their decisions. The elements of the needs have been characterized in various ways, and can be grouped into the key areas indicated in Box 1-1. The key challenges that must be faced in each of these areas are summarized in Table 1-1 (Buto and Juhn, 2006; Clancy, 2006; Health Industry Forum, 2006; Hopayian, 2001; Kupersmith et al., 2005; Rowe et al., 2006).

TABLE 1-1. Prominent Comparative Effectiveness Research Activities and Needs—Key Challenges.


Prominent Comparative Effectiveness Research Activities and Needs—Key Challenges.

Models for a Stronger Approach to Comparative Effectiveness Research

To narrow the rapidly growing gap between the available evidence on clinical effectiveness and the evidence necessary for sound clinical decision making, various organizations and recent public articles have called for the creation of a new entity and a quantum increase in spending—several billion dollars—on CER. The various approaches to building the required capacity can be grouped into four categories according to the funding patterns for their support (Box 1-2). Each of the approaches is based on an existing or recent model. Although presented as discrete models for discussion purposes, they are not mutually exclusive.

Box Icon

BOX 1-2

Models for Enhancing Capacity. Incremental model Executive branch agency model

The most straightforward public-funded approach is an expanded and appropriated mandate to an existing or newly created federal agency, and the agency whose mandate most closely parallels these priorities is AHRQ. Through its Effective Health Care program, AHRQ has an existing framework into which many elements of the identified needs can easily fit. Other executive branch models include locating the primary capacity in the NIH, putting it elsewhere in the Department of Health and Human Services, or creating it as a free-standing operational federal agency.

Other possibilities include approaches that are privately funded, although this raises issues of independence and objectivity, as well as approaches with a blend of public and private funding, which could have various governing and execution structures. In the latter category are those approaches based on the quasi-governmental federally funded research and development centers (FFRDCs), which are funded primarily by the federal government but which are allowed to have up to 30 percent of their funding from private sources. The FFRDCs are private entities managed by nongovernmental organizations and are based on the examples of free-standing independent quasi-governmental entities such as the Federal Reserve Board, which serves as the nation's central banking system, and the IOM and the Transportation Research Board (TRB) at the National Academies. TRB, from its National Academies locus, houses publicly and privately funded work in transportation that is conceptually similar in structure to what is envisioned for CER (IOM, 2008a; Kupersmith et al., 2005; Wilensky, 2006).

Decision and Implementation Considerations

Weighing the relative strengths and weaknesses of the various models can begin with certain touchstone principles that have been suggested to help guide their consideration. These include the characteristics of the approaches with respect to the following:

  • Scientific credibility: ability to gain the trust and confidence of the public, the scientific community, and the other stakeholders involved.
  • Political independence: well-insulated from the political processes that interests from all perspectives will seek to leverage.
  • Stakeholder neutrality: ability to engage with all stakeholders—patients, providers, employers, manufacturers, and insurers—in an independent, even-handed fashion.
  • Participatory governance: affording the opportunity for relevant stakeholders to engage as appropriate in setting priorities and agendas, while safeguarding the scientific integrity.
  • Investigator integrity: management and conduct of the research processes, and the determination and validation of research results completely insulated from outside influence.
  • Agenda flexibility: organizational decision making, resource allocation, and program conduct with the flexibility to respond quickly to emerging issues and changing circumstances.
  • Infrastructure efficiency: use where possible existing capacity for the establishment of scientific standards and for the management and conduct of studies.
  • Transparency of processes and results: specification and availability of the data on which determinations are based, and clarity as to the processes and tools used in their evaluation.

Other implementation considerations include those related to funding and program management. As noted earlier, funding estimates are in the range of several billion dollars. This is a sizable amount, although it is not particularly large in the context of the total U.S. health expenditures or in the context of the efficiencies that could be gained. Suggestions for funding mechanisms range from direct annual federal appropriation or a small set-aside from the Medicare Trust Fund to the structuring of proportionately matching contributions, including set-asides from Medicare fund expenditures, from private health insurance premiums, or from manufacturers' R&D expenditures (Health Industry Forum, 2006; Hopayian, 2001; Kupersmith et al., 2005; Wilensky, 2006). There can be many variations on these themes, but, ultimately, the source of the funds invested is not so important as the value of the return for the outcomes and efficiency of the nation's health care.

Independent Approaches Most Commonly Discussed

Because of the challenges to increasing CER primarily through a simple appropriation to an existing agency—such as the difficulty of marshaling an appropriation at a sufficient level, a lack of political independence, a limited ability to draw on other agencies—much of the recent discussion has focused on three of the independent models, often with blended public and private funding. Table 1-2 presents these as the federal agency, independent board, and hybrid models.

TABLE 1-2. Comparative Effectiveness Research Enterprise Models.


Comparative Effectiveness Research Enterprise Models.

As independent entities, each of these approaches assumes the establishment of a governing board composed of stakeholders and charged with priority setting, broad budget allocation, and fiduciary responsibility for the program of activities. The approaches differ in their degree of insulation between the stakeholder priority setting and the conduct of the scientific studies, as well as in the ways those studies would be managed, the involvement of existing agencies, and the reporting of results (Buto and Juhn, 2006; Kupersmith et al., 2005; Wilenksy, 2006).

PCORI, established under the ACA (2010) as an independent non-profit organization to assist in informing the health decisions of “patients, clinicians, purchasers, [and] policy-makers,” fits this model. The ACA appropriated to the PCORI Trust Fund $10 million, $50 million, and $150 million for fiscal year 2010-2012. Additionally, $150 million plus $1 per Medicare part A and B enrollee has been appropriated for 2013 and $150 million plus $2 for each A/B enrollee, each year from 2014-2019. As outlined in the Act, PCORI will set a national agenda for research priorities, fund entities that conduct priority research, improve clinical effectiveness research methods, and ensure transparency and broad dissemination of its findings. It will be overseen by a Governing Board, comprised of 19 members appointed by head of the Government Accountability Office, as well as 2 ex officio representatives from the Agency for Healthcare Research and Quality and the National Institutes of Health. For more information on PCORI, see Appendix E.

Concluding Observations

As ever-increasing options evolve in health care, current gaps in knowledge and practice about which care works best will persist or worsen without the appropriate information on which to base healthcare decisions. The rate with which new interventions are introduced into the medical marketplace is currently outpacing the rate at which information is generated about their effectiveness and the circumstances of their best use. If trends continue, the ability to deliver appropriate care will be strained and may be overwhelmed. A substantially increased capacity to conduct and evaluate research on the clinical effectiveness of interventions brings many potential opportunities for improvement across a wide spectrum of healthcare needs. In time, the enhanced capacity to identify and apply the most appropriate care will improve health and also support innovation, by identifying the areas where it is needed most. The options reviewed above offer a sense of the possibilities and opportunities, but the need for better information is pressing.


, M.D., Ph.D., Director.

Engelberg Center for Health Care Reform, Brookings Institution; IOM Roundtable on Value & Science-Driven Health Care

The IOM Roundtable on Value & Science-Driven Health Care has characterized the key elements of the learning health system needed to achieve the goal that by 2020, 90 percent of clinical decisions will be supported by accurate, timely, and up-to-date clinical information (IOM, 2007). Central to this vision is a sharper focus on research that compares the effectiveness of clinical and health policy interventions in real-world populations. The growing support for CER represents an important first step toward the Roundtable's long-term goal. As infrastructure needs for expanding the nation's capacity for CER are identified and prioritized, it is essential to consider how these immediate needs might support the development of a healthcare system that learns—a system that promotes innovation and improves care by efficiently identifying and disseminating knowledge about practices that lead to better outcomes and higher-value health care.

Key infrastructure for a learning health system will encompass three core elements: data networks, methods, and workforce. Outlined below are the important advances needed in each of these areas and how it might be possible to learn from existing capacity to create an effective, efficient CER infrastructure. It is crucial that this infrastructure be thoughtfully developed, as it will be central to determining the impact of an expanded CER effort on cost and quality.

The chapters that follow provide additional insights into the “how” of CER infrastructure by identifying opportunities to develop the core elements of a robust, sustainable capacity for CER; how these elements interact and reinforce one another; and how to build upon, link, and improve existing public and private system elements.

Data and Information Networks

Distributed data networks for securely and efficiently sharing relevant clinical and claims data are necessary for moving to a system that uses information captured at the point of care to influence practice patterns. These networks will be crucial to the CER infrastructure because they allow data owners—commercial health plans, the federal government, and others—to share only summary data in response to specific queries; individual-level data remains protected within the data owners' systems. This “virtual” approach to linking databases can result in statistically sound study results while addressing concerns about patient privacy. The functioning of these networks—and the ability to incorporate findings into EHRs—depends on further development of standards. Incentives will likely need to be put in place in order to encourage the creation of linkages and the adoption of standards.

Finally, work is needed to identify and develop successful demonstration projects and pilot models—AHRQ and the FDA, for instance, are already supporting research in this area—that can be built upon and quickly brought to scale.

Research Methods

Capitalizing on the opportunities presented by emerging clinical data and information networks requires innovative approaches to clinical trials in order to allow them to be conducted under conditions of actual practice, enabling estimates of real-world effectiveness. It will also be necessary to have improved statistical and epidemiologic methods to address the limitations of nonrandomized studies employing heterogeneous but much richer and larger-scale data sources. Similarly, methods will be needed to predict patient-level responses to interventions from population-level data. The past several decades have witnessed dramatic methodological advancement in other fields, such as financial services, Web and Internet search technologies, aerospace, and flight dynamics, but such methods have not yet been fully applied in health care. As these new data resources and methodologies become more widely available, and as the challenges of finding better ways to use them are addressed, the nation is on the verge of a tremendous opportunity to improve health care.


These critical advances must be driven by a larger and well-trained workforce, prepared to conduct studies in what might be thought of as an expanded or new field of “treatment evaluation in healthcare delivery,” as both AcademyHealth and the IOM have proposed. Consideration must be given to how the many existing training and educational programs and approaches can be built upon to develop a broad, cross-disciplinary workforce with advanced capabilities in biostatistics, epidemiology, decision analysis, health economics, health services research, and program evaluation.

Learning from Existing Approaches

The pluralistic nature of the U.S. healthcare system means that careful planning and coordination is required to successfully implement new approaches to collecting and using data, designing and conducting clinical trials, and recruiting and training a workforce for CER. Many view this heterogeneity as a barrier to the successful creation of a CER infrastructure. However, there is an upside to pluralism, as variation can be helpful in improving practice by affording opportunities to learn more quickly which interventions and policies work and which do not. Thus, a CER infrastructure should promote sharing and learning from the diverse experiences of all its stakeholders. Public–private partnerships can serve as an ideal vehicle for ensuring that all of the stakeholders within the healthcare system are represented within the CER enterprise and have the ability to coordinate with one another. Various public and private organizations would have an incentive to participate in such an effort because all would benefit from the more rapid development of evidence on the effectiveness of practices and treatment strategies.

Evidence Gaps That Inform Infrastructure Needs

In addition to identifying the critical components of a CER infrastructure, dialogue is needed on how the infrastructure can best support the learning health system. The concept of “form follows function” suggests that the development of this infrastructure should be guided by the kinds of research questions that need to be answered. The remainder of this paper will be devoted to examining four key gaps in evidence that could be closed by a learning health system and that should inform CER infrastructure development.

Establishing Baselines for Evaluations—Disease Models and Natural Histories

To move beyond evaluating the average impact of a treatment in a population and toward targeted, personalized medicine, researchers need to understand how particular types of patients are being treated. There has already been some initial progress in this area, as researchers have begun to create and use such tools as natural histories and disease models to clarify how different patients experience disease states and how they respond to different kinds of treatments and, potentially, to new additions to treatments. These models can be developed with large epidemiologic data sets—with the recognition that the data describe actual medical practices and the experiences of patients who are treated differently. Establishing such descriptive baselines is necessary for moving into more direct evaluations of the impacts of interventions on particular subpopulations of patients.

This approach has a clear resonance with the public and policy makers. For example, the public has increasingly taken advantage of the healthcare resources on the Internet, particularly Web sites and services such as PatientsLikeMe, which allow users to share experiences and gain insights on their own conditions based on the experiences of others. In a similar vein, the FDA and other scientific research groups are working to improve their understanding of exactly how different kinds of subpopulations of patients experience their conditions.

Current progress on understanding how subgroups of patients are treated can be bolstered and accelerated by appropriate infrastructure development, specifically the creation and implementation of complete standards for data collection, clinical trials, and EHRs. The potential is clear. At a recent meeting of the American Health Information Community, a number of research groups discussed how a better understanding of particular disease models with particular kinds of patients is beginning to emerge. This initial success in bridging a particular type of gap in evidence can be accelerated with appropriate infrastructure development. Expanding and improving data on treatment patterns in subgroups will also help move the biomedical research enterprise toward developing personally relevant target information that can be used to improve care for particular types of patients. Distributed data networks, as described above, are ideal for this type of work, especially since large amounts of data may be necessary in order to produce statistically significant results in studies of subpopulations.


Outstanding questions related to patient safety represent a second gap in evidence. For several reasons, some safety issues may be relatively straightforward to address as more extensive infrastructure for evidence generation is developed. First, patient safety has received broad public attention and has been backed by strong bipartisan legislation in the form of the FDA Amendments Act of 2007 (FDAAA), which endorsed the creation of a national, virtual infrastructure for quickly learning about the association between medical product use and adverse reactions and, potentially, about products' benefits in subgroups. Tracking adverse events does not have be statistically challenging, particularly in instances where safety issues should occur only very rarely, if at all. If an adverse event occurs much more frequently in a particular population of patients, it is possible to reach a responsible conclusion about excess risk without conducting a randomized controlled study. Other safety issues may be more difficult to resolve; for instance, increased adverse-event rates may be difficult to discern when medical products are used over long periods of time and when the adverse events are common healthcare problems, like heart attacks. In these situations, the difference in rates between a group treated one way and a group treated another way may be modest. Another challenge requiring further investigation is the presence of selection bias—patients who take certain drugs are likely to have different, unobserved characteristics than those with a different medication regimen—which makes it more difficult to establish a true, causal effect of a treatment. Sometimes, follow-up studies may be necessary if observational methods provide conflicting or otherwise unsatisfactory results.

The FDA, through its Sentinel Initiative, is already well under way in its efforts to develop the needed public–private infrastructure to support post-market safety monitoring of medical products. The Reagan-Udall Foundation, a nonprofit foundation created by the FDAAA to advance regulatory and product-development science at the FDA—and to involve private sector support as well—is also working to promote the establishment of this kind of network. Initially, this network will only be used for safety monitoring. However, as it evolves, it may be able to accommodate studies that compare the safety and effectiveness of treatments in different subgroups, for example. Thus, the Sentinel Network is likely to be an important part of an infrastructure for evidence generation.

Comparative Effectiveness of Interventions

A third gap concerns comparative effectiveness; at this time, the nation does not have a sufficiently evidence-based system for deciding among treatment options. A recent study concluded that more that 40 percent of the American College of Cardiology/American Heart Association (ACC/AHA) clinical practice guidelines are based on low-quality evidence (Tricoci et al., 2009), according to the evidence-grading system of the ACC and AHA.

These shortcomings in the evidence base create certain challenges that can be addressed with infrastructure development. A primary challenge is the relatively small effect of alternative treatments on patient outcomes observed in comparative effectiveness studies. A difference of only a few percentage points in outcomes—while perhaps clinically important to some patients—might be difficult to detect except in a very large study. These differences would also perhaps be more likely to be subject to confounding if the treatments being compared are not fully randomized, as in the traditional clinical trial. Conducting carefully randomized studies, in real-world situations, of these kinds of practical treatment questions can be difficult as well as costly and time consuming. In fact, by the time a large randomized trial is completed, the information may be outdated. If the information of value to patients and providers is the impact of the treatment in particular kinds of patients, a key challenge is how to move beyond approaches that generate evidence about an overall average effect—in one population versus another—to the efficient development of information relevant to particular types of patients.

The Clinical Antipsychotic Trials in Intervention Effectiveness (CATIE), which led to significant insights about the use of alternative antipsychotic medications, suggest how future CER could focus on improving the evidence of treatment effectiveness on patient subgroups. It is important to emphasize that CATIE revealed not which treatment is better on average, but rather it advanced understanding of which kinds of patients might be treated best with one approach or another at a particular stage in their treatment. Despite the importance of the CATIE study, the evidence on comparative effectiveness that it generated has not yet resulted in an FDA labeling change. As presence on a medical product label is viewed as the gold standard for determining whether evidence is of the highest quality available, the absence of particular evidence from a label can make it difficult for such evidence to be widely accepted by physicians.

A considerable amount of work is currently under way to conduct comparative effectiveness trials and to use data that has been collected for other purposes to develop comparative effectiveness information. There are also ongoing studies to determine the limitations, the methodological challenges, and the improvement in data-collection methods that are necessary to increase the value of these data. This work to improve CER methods has the potential to help develop a truly effective learning health system, but the process will hold many challenges—foremost among them is the absence of agreement on the amount and type of evidence needed for decision-making purposes.

Practice Patterns and Treatment Strategies

The fourth type of evidence gap relates to the need to understand effective treatment strategies and policies. The lack of evidence on how to best deliver care is distinct from the need to understand the differential impacts of specific treatments in different populations, yet both are critical. Treatment strategies and policies alike must be compared in their ability to produce the best outcomes, at the lowest cost, for particular populations. The complexity of medicine is increasing exponentially in terms of the array of treatments available and how they can be used in combination for particular patients. Some estimate that there are hundreds of comparative effectiveness studies under way. Even if that number were doubled or increased tenfold, the current capacity cannot contend with the impending exponential growth in complexity of medical decision making. This complexity stems from the increasing array of medical technologies and combinations of treatments, especially for the growing number of patients with chronic diseases. When physicians have little information about how these technologies and strategies affect outcomes in their patients—particularly those with complex problems—they may provide care that is only marginally beneficial or even harmful.

The geographic variations in the kinds of care delivered to similar Medicare patient populations are at least partly the result of the lack of evidence or consensus on treatment strategies. Experts have suggested that it should be possible to reduce costs in Medicare by 20 percent or more without consequences for patient outcomes—if these variations could be addressed. These variations in costs from area to area are the result of the set of sometimes subtle differences in practice patterns, especially for chronic-disease management. Among a population of patients, for example, the rate at which they are seen for follow-up varies significantly. Thus, relevant effectiveness questions include: What proportion of patients make it to more frequent follow-up? How often are they referred to specialists, and to which type of specialists? Which diagnostic tests are done and when? What minor procedures are performed on these patients and when? In this context, the evidence generated from head-to-head comparisons of treatments in experimental settings is unlikely to address the root causes of this variation.

Resolving questions related to differences in practice will likely require other methods besides randomized clinical trials (RCTs). To compare such practices and determine which relevant policies factor into variation, research needs to account for changes in delivery systems, changes in benefit designs, and changes in payments to providers that could influence how certain practices might lead to better outcomes at a lower cost. Such assessments should be part of the science of healthcare delivery, and knowledge gained through such studies could influence practice.

These questions can be studied in real-world medical practice, where similar patient populations are exposed to different health policies and therefore may face different treatment options and strategies. These kinds of studies could help close the gap between what is known to work and what is actually delivered in medical practice. Such studies, for example, would help answer questions regarding the lack of long-term adherence to certain medications among the chronically ill. More broadly, comparisons of treatment strategies could enhance our understanding of the underlying issues related to the coordination and integration of care—the lack of which constitutes a major problem in our healthcare system today.

The infrastructure needed to address these challenges should involve broad collaboration among many stakeholders, including AHRQ, other researchers, health plans, employers, consumers and patients, and provider groups. Consensus is needed to identify the best methodological approaches for developing the kind of evidence that can show which reforms in payments, benefits, and support systems for healthcare professionals and for consumers can lead to the best results. In this case, methods development should focus on improving observational methods since strategies and policies can only be studied in real-world practice and are not amenable to the idealized academic clinical trial setting. Such studies, however, could be very useful in uncovering key opportunities for improving outcomes while lowering costs.

Closing Observations

This paper provided examples of what an infrastructure can do to advance knowledge about which care is best as well as some insights about the different elements of infrastructure that can support a learning health system. Unless new infrastructure is informed by current gaps in evidence—and the ultimate goal of delivering care that produces better outcomes for different patient types at much lower costs—it will not be possible to close all the gaps that exist today. Fortunately, there is a tremendous opportunity to meaningfully expand the evidence base, as a result of the advances and collaborations discussed in the chapters that follow. The efforts of all stakeholders are necessary to transform health care into a system that learns much more effectively from actual practice.


, Ph.D., Senior Fellow.

Project HOPE

Interest in the potential of comparative clinical effectiveness information as a strategy to help Americans learn to “spend smarter” has been growing among those at both ends of the political spectrum, and it can best be understood as part of the concern about healthcare quality and value, and the drive toward the increased use of evidenced-based medicine. Other countries have focused on the use of comparative effectiveness information primarily as a strategy for new drug approval in their national health systems. The potential economic gains are even greater for medical procedures where even less comparative effectiveness information has traditionally been available, since procedures account for much more of the healthcare dollar. Substantial attention has been given to the important decisions that need to be made about the appropriate structure, placement, financing, and function of an agency devoted to comparative effectiveness. It will be equally important to focus on how best to align financial incentives to encourage the use of better information in clinical decision making. The potential for better information to improve health outcomes and help moderate spending increases is enormous, and an understanding of this information is dependent upon how to capture some of the potential savings that CER could bring.

As a prelude to this discussion, I want to note that statistically significant information will not necessarily have clinical or policy importance, in terms of guiding clinical decision making. Early in my career as a health economist, I codirected the National Medical Care Expenditure Survey (NMCES) at what was then called the National Center for Health Services Research. One of the lessons that I learned is that when very large samples are being assessed—such as 40,000 individuals, the sample size for the NMCES—almost any difference is statistically significant, but many of those differences were not relevant in terms of driving any conceivable policy decision. In the case of very small samples, on the other hand, what appear to be large differences may, in fact, not be statistically significant, which means that they should be used only with great caution in making policy decisions.

The problems that can arise from information bias are a major reason it may be important to consider new data collection, including the possibility of new prospective trials and other costly data-collection strategies, even when it looks like observed differences are substantial. This need stems from the possibility of self-selection or biased selection being introduced in analyses of observed data. An obvious example concerns the presumed advantages of hormone-replacement therapy (HRT). Prior to the Women's Health Initiative studies, relationships had been observed between the use of HRT and a variety of positive outcomes, such as improved cardiovascular health or lower rates of dementia. Unfortunately, data from the Women's Health Initiative showed these supposed advantages to be a function of selection bias related to the characteristics of the women who were using HRT. Therefore, the sizeable differences that had been observed were, in fact, not meaningful in terms of causal interpretation.

Such cases serve as reminders that it is important in designing a study not only to lay out the hypotheses and the data that will serve to support or not support the hypotheses but also to take great care in searching for correlations among the independent variables and other potential drivers of statistical bias in the data to be sampled. In addition, the differences that are likely to exist between various groups will help to determine the necessary sample size needed for the study. Finally, some determination will be needed as to whether the likely differences are ones that would be relevant at a clinical or policy level.

With that introduction, let me turn now to the kinds of data that will be relevant for comparative effectiveness analyses, remembering that the focus for these analyses is generally a medical condition and the various alternative strategies that can be used for treating that medical condition. It will be important to ensure that data are collected for various subgroups in the population that may be differentially affected by a particular medical condition. At the moment, these distinctions may be defined in terms of age and sex or other demographic characteristics, but ultimately it may be possible to differentiate probable outcomes based on an individual's genotype, phenotype, or metabolic type.

While there has been debate about the data that are most appropriate to use for comparative effectiveness, it can be argued that data need to be collected from as many sources as is possible. This includes not only the so-called gold standard of double-blinded RCTs, but the use of real-world prospective trials that Sean R. Tunis and others have been developing allow for inclusion of individuals with comorbidities, epidemiological studies, medical record analyses, registry data, administrative data, and so forth. There have been occasions where researchers have spoken as though only data reflecting the results of double-blinded RCTs should be regarded as appropriate for decision making, but comparative effectiveness analyses need to include data from many sources, although it will be important to make clear the robustness of the data collection strategies and methodologies used in the analyses. Presumably the conclusion made from the data will reflect the robustness of the data and the statistical analyses used in the assessment.

All data have limitations and are subject to error, including the results from RCTs. Specifying these limitations and biases and correcting for them wherever possible is appropriate and should be made available as part of the data release. It will also be important to find ways to reduce the costs and time required for the collection of new prospective data, given the amount of new data collection that is likely to be needed. Efforts by Bryan Luce in developing his Pragmatic Approaches to Comparative Effectiveness initiative, along with the work of Don Berry that makes use of Bayesian statistical approaches to establish shorter end points in certain types of clinical trials, represent other important efforts in this vein.

Even with these strategies to reduce the costs of new prospective trials, it is the anticipated need for a substantial amount of new data that makes the cost involved with comparative effectiveness significant. My guess is that, when fully operational, such efforts could cost several billions of dollars a year—perhaps in the neighborhood of $4 billion to $6 billion a year—although an investment of several hundred million dollars would probably be enough to make a serious start.

The first step in considering an analysis of the comparative effectiveness of various treatments for a particular medical condition is to assess the data that already exist and the analyses that have already been done. It now appears that this step may also require significant investments in time and effort. The IOM report Knowing What Works in Health Care: A Roadmap for the Nation served as a wake-up call that making better and more effective use of the information that exists is harder and more challenging than many of us had previously thought (IOM, 2008). Obtaining systematic reviews of existing data will also be more controversial than had previously been recognized.

Setting priorities for comparative effectiveness analyses should be informed by a two-step process: first, focus on those medical conditions—Medicare diagnosis-related groups might be a useful proxy—that are high-cost/high-volume areas in health care; and second, focus on those medical conditions that are subject to substantial variation in terms of how they are treated. As an economist who is looking at comparative effectiveness as a way to learn how to “spend smarter,” I suggest that the best place to focus early efforts would be those conditions on which a lot of money is spent and for which there is a great deal of geographic variation, since that suggests that differences in opinion exist about how best to treat the condition or, in any case, that differences exist in how the condition is actually treated. It would also be appropriate to look at issues of clinical relevance, disease burden, and the various subgroups that are particularly affected. Such considerations would certainly help determine, at a policy level, the relative importance of given interventions.

An important early step for more effective CER will be the creation of either a new center or a series of centers—my preference would be for a single center—that is part of the government or is a public–private enterprise and that is responsible for funding comparative clinical effectiveness studies. Unlike some of my colleagues, I believe it would be unwise—both at a technical level and, even more importantly, at a political level—to include cost-effectiveness analysis as part of the activities of a center for comparative clinical effectiveness. Cost-effectiveness information should be a component in reimbursement decisions made by payers and even in clinical decision making by clinicians and individuals, but these analyses should be kept separate and carried out in separate places.

One reason is that the amount of effort required to increase knowledge about comparative clinical effectiveness is of a much different magnitude in terms of the kinds of studies, the cost of the studies, and the length of time these studies will require. Also, at a technical level, some of the issues involved in cost effectiveness, particularly when it comes to such issues as discount rates, get into areas for which there are no definitive answers. Moreover, the number that one chooses to use has a significant effect on the outcome calculated. Other kinds of technical challenges that arise in cost-effectiveness analysis involve which cost to use and whose perspective to use: Should it be society's perspective? Medicare's? The employer's? Other issues are when in the lifetime of a technology the cost is measured and whose costs are being considered.

As important as the technical reasons are for keeping comparative clinical effectiveness analyses separate from comparative cost-effectiveness analyses, the political reasons are even more important. Cost-effectiveness analyses have long been held in suspicion by industry and many patient advocacy groups as a strategy to prevent them from providing or receiving the latest innovations and technologies in medical care. While this is an issue that ultimately will have to be dealt with, without better information about the likely effects of different medical interventions in treating various medical conditions, particularly high-cost conditions, it will not be possible to effectively make use of the information on cost effectiveness. It is therefore urgent that the nation make the investment in comparative clinical effectiveness information and keep it as protected as possible, for all users—clinicians, patients, and payers, both public and private—to have available. It will be crucial to have a common data source that we can turn to that captures what is known about the likely clinical outcomes of various kinds of treatments for various subgroups of the population before we get into the next round of much more difficult decision making about how we make use of that information. Having said that, we need to have better information about cost effectiveness as well. Fortunately, that can be funded more quickly and at substantially less cost. A portion of the funding stream that is used to fund studies in comparative clinical effectiveness can be provided to CMS to fund studies in the cost effectiveness of alternative medical treatments.

In determining the best way to fund CER, it is important to take into account the issue of the preferred versus the practical. The preferred strategy would be to use a direct appropriation, as is the case with the NIH, since the information generated is clearly a public good as the economist uses the term. Unfortunately, the practical reality is that relying on a direct appropriation is likely to produce an unreliable funding stream. An alternative would be to use a combination of direct appropriations with fees, which would resemble an all-payer system, for people who are covered by private plans as well as a contribution from the public players such as Medicare. If it appeared that a direct appropriation to CMS for cost-effectiveness analysis was unreliable, a small portion of the funding stream could be diverted to CMS in order to fund the cost-effectiveness studies that are important for Medicare. It will also be important to ensure that the information on cost effectiveness that is generated is valid in terms of objectivity and credibility—just as information needs to be for comparative clinical effectiveness—or it will not be trusted. However, unlike the comparative clinical effectiveness information, payers could generate better information about cost effectiveness as long as they do so in a way that keeps the generation of the information transparent. Alternatively, the cost-effectiveness analysis could done by AHRQ or other entities in the federal government, and it could continue to be done in the private sector by private-sector payers as well.

The question then becomes how do you begin to use this information? Several important principles apply. First, the concept of expecting and allowing for different players to use this information differently is very important. If there is a sense that a single entity can and is making decisions about how information on comparative clinical effectiveness and cost effectiveness will be used, there will be a great deal of resistance by patient advocates as well as by industry. That does not mean that such information should not be part of a realignment of financial incentives to reward clinicians and institutions in terms of how they practice and to encourage positive patient behaviors, but rather that it ought not be relegated to a single unitary decision-making entity. Such a monopoly would be inappropriate in a country as large and diverse as ours, and it would also be a political nightmare for politicians.

As part of the need to realign financial incentives so that physicians and other clinicians, as well as institutions, are rewarded for producing good clinical outcomes, a first step could be to have information available from comparative clinical effectiveness as part of a change in reimbursement policies. This would be consistent with the development of discussions on value-based insurance, where the amount of the copayment varies with the likelihood of a good clinical outcome for a particular intervention. The objective would be that procedures likely to produce good clinical outcomes for patients in particular categories or subcategories of the population would have low copayments or no copayments, while those medical procedures unlikely to have good clinical outcome would be made more expensive, although not disallowed. In this sense, comparative effectiveness information would be considered not so much as a coverage issue as a reimbursement issue.

Changes in the statutes governing Medicare would be required before we could begin to think about copayments on a variable basis, but that approach is better than some alternatives that have been proposed. Currently there is no statutory authority, when it comes to either coverage or reimbursement, to allow the agency to introduce concepts of cost or cost effectiveness. The ideas outlined here provide a way to introduce these concepts into the reimbursement process and to do so in a way that allows different private payers to use the information differently. It is one of the many changes that will need to occur.

In addition to major investments in comparative clinical effectiveness, it will be important to use a variety of strategies to improve evidence development. One of the ways this could occur would be by tying the local coverage decision making that now goes on under Medicare with evidence development. There has been a great deal of discussion over the last couple of decades about whether it is appropriate for carriers, the local payers, to have their own coverage authority, at least on an interim basis, before a national coverage decision is made. Before I had been sworn in as administrator of the Health Care Financing Administration, now CMS, I had thought that one of my goals would be to remove local coverage decision making on the grounds that this is a national program and that the benefits ought to be the same everywhere. In ensuing discussions, however, it became apparent that to do so would introduce a very conservative bias to the coverage of new innovations and technologies in Medicare. If local carriers are going to continue making interim coverage decisions, these decisions should only occur with evidence development. Such a change would need to be statutorily driven rather than have the agency attempt to do this administratively, but it seems to be the kind of change that could be introduced to advance evidence development in as many ways as is possible. This would make it possible to harness the diversity that exists in the U.S. healthcare delivery system in order to improve the knowledge base. The concept proposed here is just one example of how to continue having the local coverage variation that already exists, but to do so in ways that still contribute to knowledge and, therefore, to improved decision making in the future.

In conclusion, the following steps need to occur. First, a center or entity should be established that is charged with creating better information on comparative clinical effectiveness. Initially, such a center could be started with an investment of perhaps a few hundred million dollars. In the long term, such a center would require funding on the order of several billion dollars to sustain its effectiveness. Second, priority setting ought to be based on both cost and geographic variation, at least as general guidance, but with allowances made to include the economic and clinical burdens of disease in making decisions. Third, it should be recognized that all stakeholders need to be a part of the decision-making process. Better to have them on the inside participating in the decision making about what is analyzed and how to treat various types of information than to have them attacking the process from the outside.

As noted earlier, it is important to generate information on cost, but the estimates should be done separately from the comparative clinical effectiveness analysis, both in the public and the private sector. It is important also to have credibility, objectivity, and transparency associated as much with the cost analysis as with the generation of comparative clinical effectiveness information. In addition, we need to recognize that as important as it is to have information on clinical or cost effectiveness out there, the necessary gains are not likely to be made unless the reimbursement system is changed to make use of the information through value-based insurance, through changing how we reimburse clinicians and institutions, and through rewarding the kind of behavior that needs to be encouraged rather than just paying more for doing more.

It will be particularly important to begin to give CMS the legislative authority to introduce what is known about clinical and cost effectiveness into its reimbursement decisions. As indicated, this would be preferable to having comparative clinical effectiveness become part of the coverage decision, which is too heavy a burden to use going forward.

And finally, it is important to understand that if the growth in medical spending is to be slowed from its current rate of 2.5 percent faster than the economy to something more tolerable, it must be recognized that such a slowing will mean less increased cashflow over time than industry, clinicians, institutions, and patients have been used to seeing come into the system. None of them are likely to appreciate the consequences, and there will likely be charges that clinicians are being prevented from providing the best care possible to their patients. Not only may patient advocates and clinicians feel they are being denied “the best care that is out there,” but industry may also feel that it is being prevented from having the opportunity to sell what could be the lifesaving or quality-improving strategy that people want. Having credible information to indicate the contrary will be a critical first step—although only a first step—if the United States is ever to learn how to treat patients better and spend smarter.


  1. ACA (Patient Protection and Affordable Care Act). Public Law 111-148. 2010.
  2. AcademyHealth. Placement, coordination, and funding of health services research within the federal government, AcademyHealth Report. Sep, 2005. [September 3, 2010]. 2005. http://www​.academyhealth​.org/files/publications​/placementreport.pdf.
  3. AdvaMed (Advanced Medical Technology Association). The medical technology industry at a glance. 2004. [September 3, 2010]. p. 16. http://www​.omnex.com​/training/iso13485/articles​/ADVAMED-Medical-Device-Industry-Facts.pdf. Chart 3.6.
  4. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schunemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW Jr, Zaza S. Grading quality of evidence and strength of recommendations. British Medical Journal. 2004;328(7454):1490. [PMC free article: PMC428525] [PubMed: 15205295]
  5. Balas EA, Boren SA. Managing clinical knowledge for healthcare improvements, Yearbook of medical informatics. Schattauer V, editor. Stuttgart, Germany: Schattauer Publishing; 2000. pp. 65–70.
  6. Biotechnology Industry Organization. Chart: Total biotechnology patents granted per year. 2006. [July 15, 2010]. http://www​.bio.org/speeches​/pubs/er/statistics.asp.
  7. Buto K, Juhn P. Can a center for comparative effectiveness information succeed? Perspectives from a health care company. Health Affairs. 2006;25:w586–w588. [PubMed: 17090556]
  8. Califf RM. Defining the balance of risk and benefit in the era of genomics and proteomics. Health Affairs. 2004;23(1):77–87. [PubMed: 15002630]
  9. CBO (Congressional Budget Office). Research on the comparative effectiveness of medical treatments. Dec, 2007. [July 15, 2010]. http://www​.cbo.gov/ftpdocs​/88xx/doc8891/12-18-ComparativeEffectiveness.pdf.
  10. Clancy CM. Getting to “smart” health care. Health Affairs. 2006;25(6):w589–w592. [PubMed: 17090557]
  11. FCC (Federal Coordinating Council for Comparative Effectiveness Research). Federal Coordinating Council Report to the President and to Congress on comparative effectiveness research. 2009. [September 3, 2010]. http://www​.hhs.gov/recovery​/programs/cer/cerannualrpt.pdf.
  12. Fisher ES, Wennberg JE. Health care quality, geographic variations, and the challenge of supply-sensitive care. Perspectives in Biology and Medicine. 2003;46(1):69–79. [PubMed: 12582271]
  13. Foster L, Nanu D, Alcock S. Patenting in the biopharmaceutical industry—comparing the U.S. with Europe. Drug Plus International. 2002;1(1)
  14. Gelijins AC, Rosenberg N. The dynamics of technological change in medicine. Health Affairs. 1994;13(3):28–46. [PubMed: 7927160]
  15. Health Industry Forum. Comparative Effectiveness Forum: Executive summary. 2006. [July 20, 2010]. http://healthforum​.brandeis​.edu/meetings​/materials/2006-30-Nov./ExecBrief.pdf.
  16. Himmelstein D, Warren E, Thorne D, Woolhander S. Illness and injury as contributors to bankruptcy, Health Affairs. 2005. pp. W5–W63. Web Exclusive. http://content​.healthaffairs​.org/cgi/content/full/hlthaff​.w5.63/DC1.
  17. Hopayian K. The need for caution in interpreting high quality systematic reviews. British Medical Journal. 2001;323(7314):681–684. [PMC free article: PMC1121240] [PubMed: 11566835]
  18. IOM (Institute of Medicine). To err is human: Building a safer health system. Washington, DC: National Academy Press; 2000.
  19. IOM (Institute of Medicine). Crossing the quality chasm. Washington, DC: National Academy Press; 2001.
  20. IOM (Institute of Medicine). Learning what works best: The nations need for evidence on comparative effectiveness in health care. 2007. [August 5, 2010]. http://www​.iom.edu/ebm-effectiveness.
  21. IOM (Institute of Medicine). Knowing what works in health care: A roadmap for the nation. Washington, DC: The National Academies Press; 2008.
  22. IOM (Institute of Medicine). Initial national priorities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009.
  23. IOM Roundtable on Value & Science-Driven Health Care. Charter and vision statement: Roundtable on value & science-driven health care. Washington, DC: 2009.
  24. Jacobson G. United States Congressional Research Service. Comparative clinical effectiveness and cost effectiveness research: Background, history and overview. 2007. [July 15, 2010]. http://www​.cbo.gov/ftpdocs​/88xx/doc8891/12-18-ComparativeEffectiveness.pdf.
  25. KFF (Kaiser Family Foundation). Trends in health care costs and spending. 2009. [January 19, 2011]. http://www​.kff.org/insurance​/upload/7692_02.pdf.
  26. Kupersmith J, Sung N, Genel M, Slavkin H, Califf R, Bonow R, Sherwood L, Reame N, Catanese V, Baase C, Feussner J, Dobs A, Tilson H, Reece EA. Creating a new structure for research on health care effectiveness. Journal of Investigative Medicine. 2005;53(2):67–72. [PubMed: 15810492]
  27. Litan RE. Vital signs via broadband: Remote health monitoring transmits savings, enhances lives. 2008. [August 5, 2010]. http://www​.betterhealthcaretogether​.org/www/docs/broadband​.pdf.
  28. Moses H 3rd, Dorsey ER, Matheson DH, Thier SO. Financial anatomy of biomedical research. Journal of the American Medical Association. 2005;294(11):1333–1342. [PubMed: 16174691]
  29. PhRMA (Pharmaceutical Research and Manufacturers of America). Pharmaceutical industry profile. 2006. [August 5, 2010]. p. 49. Table 4. http://www​.phrma.org​/files/attachments/%2006​%20Industry%20Profile.pdf.
  30. Rowe JW, Cortese DA, McGinnis JM. The emerging context for advances in comparative effectiveness assessment. Health Affairs. 2006;25(6):w593–w595. [PubMed: 17090558]
  31. Tricoci P, Allen JM, Califf RM, Smith SC Jr. Scientific evidence underlying the ACC/AHA clinical practice guidelines. Journal of the American Medical Association. 2009;301(8):831–841. [PubMed: 19244190]
  32. Tunis SR, Pearson SD. Coverage options for promising technologies: Medicare's “coverage with evidence development” Health Affairs. 2006;25(5):1218–1230. [PubMed: 16966717]
  33. Wilensky GR. Developing a center for comparative effectiveness information. Health Affairs. 2006;25(6):w572–w585. [PubMed: 17090555]
  34. Woolf SH. The power of prevention and what it requires. Journal of the American Medical Association. 2008;299(20):2437–2439. [PubMed: 18505953]

American Recovery and Reinvestment Act. 2009. HR1, 111th Cong, 1st Sess.



American Recovery and Reinvestment Act. 2009. HR1, 111th Cong, 1st Sess.

Copyright © 2011, National Academy of Sciences.
Bookshelf ID: NBK64791
PubReader format: click here to try


Related information

  • PMC
    PubMed Central citations
  • PubMed
    Links to pubmed

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...