NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Institute of Medicine (US) Roundtable on Translating Genomic-Based Research for Health. Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary. Washington (DC): National Academies Press (US); 2009.

Cover of Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health

Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary.

Show details

2Generating Evidence for Decision Making


Steven Teutsch, M.D., M.P.H.

County of Los Angeles Department of Public Health

Decisions affecting health care must be acceptable and legitimate to the people they will affect, Teutsch began. The legitimization of health policy decisions requires prospective agreement about the evidentiary standards that will be used. This is a deliberative and inclusive process to develop an understanding of the different types of decisions to be made, and the nature and importance of the evidence that is appropriate for each. There is no simple formula or prescription for decision making. Each decision is based not only on the evidence, but also the context in which each decision is being made. Transparency of the process is also important, so that it is clear what information was used in making the decision.

Evidentiary Threshold

The translational process can be viewed as moving from gene discovery to application in a health context, to health practice, and finally to understanding the health impact (Figure 2-1). The critical step in translation is the development of an evidence-based guideline that allows the technology to move from research into clinical or public health practice. A key question in developing guidelines, Teutsch said, is how high the evidence bar should be. By employing a lower threshold, technologies can move more rapidly from research into practice. The consequences are that less information is available on the clinical validity of the technology, and almost no information is available about the clinical use. This lack of information can lead to negative insurance coverage decisions. There is the potential for increased harms because less is known about the technology, but also the potential for increased benefits by providing the technology sooner to those who may need it. Requiring a lower evidentiary bar means a greater dependence on models and expert opinion. Because technologies can enter practice more easily, a lower bar might stimulate innovation, thereby making more technologies available.

FIGURE 2-1. The translational process.


The translational process. SOURCE: Teutsch, 2009.

If the evidentiary bar is high, more will be known about the validity and utility of the technology, and payers can make better decisions about reimbursement. On the other hand, a higher threshold for evidence makes moving technologies into practice more difficult, which can potentially lower the incentive for innovation. More is known about the technology, resulting in a diminished potential for harms, but it will take a longer time to bring the product to those who can benefit from it.

When making an evidence-based decision, several questions must be answered:

  • What decision must be made?
  • How does the nature of that decision affect the evidentiary standards that should be applied?
  • What are the relevant contextual issues?
  • How will information (both scientific and contextual) be integrated and applied?
  • What processes are needed to legitimize the decision process?

There is a dynamic relationship between evidence-based decision making and evidence review and synthesis (Figure 2-2). Decisions may pertain to regulation, coverage, guidelines, quality improvement metrics (e.g., pay-for-performance), or individual care decisions made by a clinician and/or patient. The decision maker should first frame the key questions to be answered and determine the level of rigor required. Then evidence reviewers should synthesize data from studies as well as desired economic information. With quantitative scientific evidence in hand, the decision makers should also consider budget constraints, values and preferences, equity issues, acceptability, and other contextual issues before making a decision.

FIGURE 2-2. Dynamic relationship between evidence review and synthesis and evidence-based decision making.


Dynamic relationship between evidence review and synthesis and evidence-based decision making. SOURCE: Teutsch and Berger, 2005.

Quantitative Information for Decision Making

Quantitative information needed for decision making includes data on effectiveness, such as the level of certainty there will be an impact, and the magnitude of the effect, or net benefit. Cost and cost-effectiveness data are also important, as are any data regarding how the new technology compares to existing alternatives. Clinical effectiveness and cost effectiveness are usually assessed in relationship to therapeutic or diagnostic alternatives.

A matrix, such as the one under development by America’s Health Insurance Plans, can be useful to help payers compare two technologies with regard to net benefit and certainty (Figure 2-3). Technologies that have large net benefit and high certainty would be good candidates for coverage. On the other hand, products with limited or low certainty and equal net benefit are not ready for broad use. Some will have incremental benefits, but high certainty, and others will have new technology that is unproven, but has potential. Different insurance groups are likely to make different coverage decisions. Payers should be able to articulate what their criteria are, or how high the evidentiary bar is going to be, so a technology developer can decide whether to invest in developing the technology.

FIGURE 2-3. Comparative clinical effectiveness matrix.


Comparative clinical effectiveness matrix. SOURCE: Developed by the America’s Health Insurance Plans (AHIP) Evidence Based Medicine Roadmap Group, Personal communication, S. Pearson, Institute for Clinical and Economic Review (ICER), July 9, 2009. (more...)

The key effectiveness questions relate to the following:

  • Efficacy: Can the technology work in controlled conditions?
  • Harms: What are the possible harms?
  • Effectiveness: Does it work in practice?
  • Trade-offs: What is the balance of harms and benefits?
  • Comparative effectiveness: Does it work better than alternatives currently in use?
  • Subpopulations: Are there specific groups for whom it is likely to be a technology of choice?

As one example of a framework to determine how high the evidentiary bar should be for clinical management decisions, Teutsch cited the work of Djulbegovic and colleagues (2005) on cancer. The framework lays out proposed evidentiary standards for clinical applications as a function of treatment goals and acceptable regret. Considering the various goals of treatment—including cancer prevention in healthy individuals, palliative therapies, procedures that offer incremental improvement in terms of survival, or curative measures—how much certainty is needed before a technology should be used? How much regret will there be if the technology used is ineffective or even harmful?

In the prevention arena, Teutsch said, the evidentiary bar is very high because the interventions are being delivered to people who are otherwise healthy. The Evaluation of Genomic Applications in Practice and Prevention (EGAPP) working group, established by the Centers for Disease Control and Prevention, recently published its methods for evidence-based evaluation of genetic tests (Teutsch, 2009). Genome-based products first were categorized by application: diagnostic, screening, risk assessment and susceptibility, prognostic, or predicting therapeutic response. EGAPP then established the criteria that would be used when assessing clinical validity and utility issues (Table 2-1).

TABLE 2-1. Categories of Genetic Test Applications and Some Characteristics of How Clinical Validity and Utility Are Assessed.


Categories of Genetic Test Applications and Some Characteristics of How Clinical Validity and Utility Are Assessed.

One approach to answering the quantitative questions is the ACCE model for evaluating data on emerging genetic tests. The model breaks down the information needed into four main areas (from which the name is derived): Analytic validity, Clinical validity, Clinical utility, and Ethical, legal, and social implications (Haddow and Palomaki, 2004). At the center of the circle in Figure 2-4 is the disorder to which the genetic test will be applied, and the setting in which the testing will be done. From there, an analytic framework is constructed by answering more than 40 targeted questions in each of the 4 areas.

FIGURE 2-4. The ACCE method for multidisciplinary evaluation of genetic tests.


The ACCE method for multidisciplinary evaluation of genetic tests. SOURCE: CDC, 2007.

EGAPP has been working within the ACCE framework to articulate the evidentiary standards that could or should be applied to evaluation of genetic tests. Table 2-2 presents a hierarchy of data sources and study designs for the analytic validity, clinical validity, and clinical utility components of evaluation. Looking at clinical utility, for example, meta-analysis of randomized controlled trials (RCTs) would be the strongest form of evidence. A good single RCT may be adequate, but less strong. The list then covers other study designs that are progressively less desirable, such as controlled trials that are not randomized, or cohort studies, with case series or expert opinion being the least desirable form of evidence.

TABLE 2-2. Hierarchies of Data Sources and Study Designs for the Components of Evaluation.


Hierarchies of Data Sources and Study Designs for the Components of Evaluation.

Contextual Information for Decision Making

Numerous contextual issues can inform the decision to introduce a test into practice. Clinical applications differ widely, and it is important to consider the severity of the condition, subgroup differences, the availability of alternatives, the severity and frequency of harms, and the risk of overuse or inappropriate use of the test. Economics is also considered from a contextual perspective. Many decision makers are interested not only in cost-effectiveness, but also budget impact, budget constraints, and value. Legal and ethical considerations include federal and state regulatory constraints, as well as issues of precedent, and regret as a result of introducing or not introducing a test. Feasibility of the test in question refers to the current level of use, the infrastructure required to use the test properly, and the acceptability of the test to all partners and stakeholders, particularly patients. Decisions should be made in the context of the preferences and values of those who are going to be affected by the decision. Finally, there are administrative issues, such as options for targeting or limiting the use of the test to patients who would benefit most, and how to consider possible further evidence.

Decision-Factor Matrix

In the end, Teutsch said, a systematic process is needed to ensure fairness and reasonableness in decision making. This process includes: clear “rules of the road” for the technology developers, patient advocacy groups, and others; a deliberative process incorporating both quantitative and qualitative or contextual information; transparency; and an appeals processes so that when other issues arise, they can be addressed, and the decision changed where appropriate.

Teutsch presented a draft of a decision matrix, plotting different decisions that are likely to be made for any test or technology against a set of quantitative and qualitative information that might need to be generated. His example (Figure 2-5) suggests that a regulator may be primarily interested in efficacy, safety, and the legal and ethical constraints. These aspects, however, would be less likely to impact individual decisions. Rather, effectiveness, as well as cost, may be of great interest in practice. Each type of user will have important criteria, some secondary considerations, and other information that may not be directly relevant. The important point, Teutsch said, is that different decision makers require different kinds of information, and it is important to be able to generate that information for them.

FIGURE 2-5. Example of a hypothetical decision-factor matrix.


Example of a hypothetical decision-factor matrix. * Administrative feasibility of management, e.g., limiting coverage to people who meet specific criteria. Legend:

In refining the approach to standards of evidence, Teutsch said in conclusion, it will be important to rethink the hierarchy of evidence in terms of the many different applications and new types of evidence. When is it appropriate to use predictive modeling, for example? Another critical issue is how research efforts are aligned with application needs. The evolving role of observational data must be accommodated, and appropriate methods must be used to make better decisions when the evidence is insufficient.


Wylie Burke, M.D., Ph.D.


A question was asked as to whether the appeals process mentioned by Teutsch would address passive challenges, such as a need for change identified as a result of horizon scanning, as well as active challenges. Teutsch responded that there may be information that was not taken into consideration in the original decision, and the appeals processes can help address that issue. But in general, one should be proactive about the information generation process. In trial design, for example, it is important to ensure representation from the appropriate groups, and that may require participation of the affected groups in the development of the study.

A participant noted that the methodology outlined focuses on the test or the technology itself, and asked if the questions would change when the focus was on whether or not to screen for a condition. Teutsch responded that one needs to have a specific clinical scenario in mind, and that assessments should not be done in the abstract.

Another participant expressed concern about the decision matrixes considering low efficacy and harm as if they were similar in impact, and suggested that a distinction be made. Teutsch said the vocabulary varies, but in his perspective, efficacy refers to benefits, and effectiveness refers to the balance of the benefits and potential harms. On some occasions, risk of substantial harm may be acceptable because of the potential for substantial benefits, while at other times the equation will be different. He agreed there is a need to be clear about whether one is talking about benefits or harms, and to whom they accrue.

Copyright © 2009, National Academy of Sciences.
Bookshelf ID: NBK32309


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (882K)

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...