NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Institute of Medicine (US) Roundtable on Translating Genomic-Based Research for Health. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington (DC): National Academies Press (US); 2011.

Cover of Generating Evidence for Genomic Diagnostic Test Development

Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary.

Show details

5Considerations Moving Forward

Key Points Raised by Speakers

  • Developing new models for coverage of tests post-FDA clearance could aid the development of clinical utility evidence.
  • Patient management and outcomes should be the focus of intervention development.
  • There needs to be greater accounting of and transparency in experimentation for biomarkers.
  • The public needs to be educated on the need for evidence of utility for new interventions.

In light of the barriers and complexities associated with the various models of evidence generation discussed and of the need for more timely evidence gathering in order to better meet the needs of patients and providers, the workshop participants considered what the next steps should be in generating new evidence for the development of genomic diagnostic tests.


Roger Klein, a member of EGAPP, asked why medical practice often diverges from the recommendations issued by evidence-based review groups. For example, there are a number of tests for thrombophilia that have been cleared by the FDA. These are very commonly ordered assays, yet there is limited evidence that the results of these tests offer any clinically useful information for most people.

Dan Hayes responded that tests are often ordered because they are available and easy to order and there is little cost to the doctor for using them inappropriately. Physicians order marker tests as part of general data gathering about the patient. “A bad marker is as bad as a bad drug,” he said, adding that providers are making critical treatment decisions based on information that may be wrong. Practitioners could do much better in not overtreating patients who will not benefit and in identifying patients who will benefit. This is where marker development is as important as pharmaceutical development, Hayes said. Becker added that the idea of being able to manage clinical effect (e.g., the hoped-for effect against a tumor or the range of adverse events) is something that lies in the realm of clinical practice. It is a matter of judgment and experience. The question in the mind of a clinician deciding whether to adopt a new test is, What does it add to what is already done in day-to-day practice? Hayes added that clinicians are sometimes of the opinion that practice guidelines are made without considering individual patients and the nuances that come with each. Such contextual issues are difficult to build into the guidelines.

One approach that could affect practice would be for third-party payers who agreed to cover a genomic diagnostic test to then deny coverage of a treatment if the provider treated the patient differently than what the test result had directed (but only if the recommendations from the results are clear cut). Blue Cross and Blue Shield plans, for example, do not pay for trastuzumab treatment for a HER2-negative patient because such a test result indicates it is not a medically appropriate treatment. However, some participants said, this approach does raise concerns about patient autonomy and about patients' rights to change their minds on treatment course in certain circumstances.


Moderator Sharon Terry of Genetic Alliance pointed out the variation in evidence-based decisions. For example, FDA recently cleared the breast cancer recurrence test, MammaPrint, for more broad use in all age groups, while the Blue Cross and Blue Shield Association TEC determined that Oncotype DX met its criteria but MammaPrint did not, and EGAPP has concluded there is insufficient evidence to recommend for or against either test.

Piper responded that the Oncotype DX test was reviewed by TEC several times, and the final vote to cover the test was very close. Similarly, the final EGAPP recommendation was not arrived at easily. (As discussed by Calonge in Chapter 3, the insufficient evidence conclusion for Oncotype DX was classified as “encouraging indirect evidence.”) These nuances do not come across because only the final decision or recommendation is released. TEC assessments summarize the available evidence, Piper noted, but are not a practice guideline, while EGAPP is able to incorporate some of these contextual issues. This difference, she said, may add to some of the disparities between recommendations.

Hayes said that the ASCO committee was widely criticized and accused of being United States–centric for having recommended the 21-gene test, Oncotype DX, which was developed by a company in the United States, and not recommending the 70-gene test, MammaPrint, from an Amsterdam-based company. Hayes explained that ASCO's Oncotype DX decision was based on the review of studies that asked clinically relevant questions regarding node-negative, estrogen receptor–positive women and tamoxifen that were applicable to practice. Becker said that one of the most important differences between the tests was the fact that Oncotype DX claimed to be able to therapeutically manage patients, while MammaPrint could not specifically make that claim. It came down to clinical utility versus clinical validity.

A test that is cleared or approved by FDA may not necessarily be reimbursed by payers, Becker noted. Clinical validation ties the test to the disease and diagnosis, which may lead to a prognosis, but the payer is focused on the utility. This system, where regulatory approval is disassociated from reimbursement approval, has been in place for a long time in the United States, and, Becker said, “It is not a system which was tailored for the rapidly evolving circumstances that we see now.” A participant warned against linking FDA clearance and reimbursement too closely, as this may lead to other problems. “Either the bar at the FDA would be too high for the commerce aspect or too low for what a rational system should be spending its resources on.”

Currently, a device developer might allot three to five years in its expected development timeline to navigate the FDA clearance process, a participant said. If the target is changed to clinical utility, that could easily push this phase of development out to 7 to 10 years. If payers will reimburse for devices after they are cleared or approved by the FDA, it gives the manufacturer an opportunity both to sell the product and to prepare and conduct clinical utility trials. Developing biospecimen banks will help close the gap between what FDA requires and what third-party payers require. Coverage with evidence development is a valuable tool as well, as it allows for provisional coverage while studies are being conducted.


Piper reiterated that research is too often driven by the desire to secure grants, publications, FDA approval, and reimbursement, when the real focus should be whether the intervention will make a difference in managing the patient. This should be the starting point in choosing tests to evaluate.

It was suggested by several participants that the Institute of Medicine roundtable could help define the top 10 genomics clinical questions that need to be answered. Such a list would not have to be the definitive list, but it could provide well-justified examples and help flesh out other questions that would also be appropriate. It will also be important to compare these questions with the priorities of the medical system at large, one participant commented. The point was made again that the success of Oncotype DX was due to the fact that the clinical question came first, then the developer figured out what kind of dataset was needed to answer the question and found the most appropriate previously conducted clinical trial. As another participant noted, there needs to be a focus on addressing the salient clinical questions.

With regard to building the genomics evidence infrastructure, one participant suggested starting with such a list of ten different, clinically important questions and assembling groups to decide how best to answer those questions and to determine the infrastructure that would need to be developed to facilitate that approach. Some of the answers will come from analysis of biobanked samples, but there may also be ways to answer questions outside of clinical trials. It would be very informative to solve a set of specific, important problems over the next five to ten years and see what evidentiary approaches emerge.

A participant pointed out that research has changed over time as NIH has required grantees to include minorities, children, and women in studies and has required that data-sharing plans be devised. What if NIH required grantees to demonstrate the potential for translation into practice?

Another participant observed that device manufacturers tend to design trials that place the product into a lower category for FDA clearance so that the product can be brought to market more quickly. While it may speed up approval, this approach does not deliver information about the true value of the test for patients and clinicians.

It was noted that one of the themes that has emerged in every roundtable workshop is the idea of having a framework that allows for access to high-quality biological samples connected to clinical data and which facilitates discovery and validation of genetic and genomic biomarkers. “We have to build the sandbox that everybody will play in,” said a participant. It was suggested that GAPPNet (a stakeholders' group formed by CDC) could offer a forum for taking discussions and ideas from the roundtable workshops and fleshing out how to translate them into practice by pulling together public and private partners to conduct pilot projects to test the ideas in different contexts.


One participant offered several observations relevant to moving forward with an evidence generation model for genomic tests:

  • Individual learning curves vary considerably, and learning is an ongoing process that extends beyond medical/health professional school.
  • Part of the process is learning to be comfortable with being wrong.
  • Variability among both consumers and providers of health care results in different messages being given at different times and in different situations.
  • Familiarity with new tests is variable (e.g., knowing what actions to take in the face of strong test results).
  • There is a general lack of comfort with and understanding of change.
  • There is a lack of comfort with ambiguity.
  • There is no perfect evidence and no absolute truth. We need to set a standard of excellence that providers can be comfortable with.


A participant said that the evidence base is damaged by selective publication of clinical trials. Although manufacturers are required to register all clinical trial protocols publicly in, the participant said she believed that not all trial results are being reported.

Hayes agreed that has been a great step forward. The Consolidated Standards of Reporting Trials have helped to ensure transparency in clinical trial reporting, as have the requirements by many journals that authors provide details about basic research studies, such as the specific reagents used. But for some reason, the translational area in between basic and clinical research has been left behind. Hayes suggested that many journal editors, who are generally very careful about not accepting clinical trials that are not registered in or that do not provide essential experimental details, have abrogated their responsibility in terms of biomarkers, because markers are not valued in the way that clinical trials are. Hayes mentioned a new effort to initiate a registry for tumor marker studies. This registry would help ensure that studies are being conducted with prospectively written protocols and that negative results come to light.

It was further suggested that the roundtable engage major journal editors and urge them to pay more attention to tumor marker studies with regard to design, quality, and transparency.


Participants discussed the need for a public education effort regarding genomic interventions. Terry said that many people believe the FDA to be primarily interested in the public's health and not in commerce and assume that agency clearance implies that the product is beneficial and should be adopted. On the other hand, a participant noted, the public often perceives FDA as being too slow or hindering access to products (such as happened with products targeting the human immunodeficiency virus). Another participant added that the current environment is one of “immediate-communications” and that health literacy and public understanding about evidence are highly variable. While eager for new products to be available on the market, the public in general does not demand evidence to support claims about new interventions. As a participant said, “If we could simply educate the public to ask [for evidence] every time they hear a claim, I think it would help us.” It was also noted that whole-genome analysis will enable providers to give patients their entire genome, but it will not be of much practical use unless the genome can be related to various kinds of clinical evidence.

Copyright © 2011, National Academy of Sciences.
Bookshelf ID: NBK62402


Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...