5Panel: Where Are the Gaps?

Publication Details

Expanding on the previous panel presentations, three experts were asked to provide perspective on gaps in the systems for evaluation of genome-based technologies and health care.


Foley Hoag, LLP

Watson and Crick’s 1952 article asserting that DNA was a double helix presented four or five different tracks of evidence, Quinn said. Isolated, any one of those tracks of evidence was insufficient to conclude that DNA was a double helix, but together, the combined evidence provided a clear case for their proposed structure. When talking about evidence-based medicine, there is always the discussion that more data are needed. The critical step, however, occurs in the human brain, which absorbs the data, makes judgments about it, and integrates it with other known information.

Most would agree, Quinn said, that the data have established that monoclonal antibodies which bind epidermal growth factor receptor (EGFR) will not inhibit tumor growth in patients whose tumors have a mutated KRAS gene. If the tumor does not have a KRAS mutation, then EGFR monoclonal therapy has a chance of being effective. One piece of information that is not always discussed, however, is that EGFR monoclonal antibodies do not work very well overall, with 5, 10, maybe 15 percent of patients who have tumors with wild-type KRAS showing a response. Taking all the observations into account, it is generally agreed that a randomized controlled trial (RCT) to test the clinical utility of a KRAS gene test would not be appropriate.

There are also good basic science data on genetic variation in the response to tamoxifen or warfarin, but there is much more confusion about whether the genetics are clinically relevant. In warfarin management, for example, there are many additional considerations, such as height, weight, concomitant medications, diet, and compliance. In this case, RCTs would seem to be very important.

The way an Institutional Review Board (IRB) views a trial impacts whether an RCT is conducted. A study of the published literature found that RCTs that have gone through IRB review have a 40 to 60 percent success rate for the hypothesis being tested in the trial (Djulbegovic and Bercu, 2002). If there was a 10 percent chance of success, no one would fund the trial and an IRB would not approve it. Similarly, if there was 90 percent chance of success, the same thing would occur. A trial needs to fall somewhere in this 40–50–60 percent range to garner IRB approval.

Something that happens fairly often with diagnostic tests is that the 40 to 60 percent success range is already exceeded based on known information. For a genetic test, retrospective data may suggest an 80 or 90 percent likelihood of a particular result. Already, the predicted success rate of the hypothesis is outside of that 40–60 percent range where, based on documentation, an IRB will tend to approve it. An insurer, or another decision maker, may want to see data from an RCT, but the type of trials they are asking for would not likely be approved by an IRB.

It is important for companies bringing a diagnostic test into the marketplace, or facing insurer decisions, to remember that a product faces very different value propositions across its life cycle, Quinn said. In the early investment phase, intellectual property, barriers to competition, and development risk are important considerations. When seeking regulatory approval, meeting Food and Drug Administration (FDA) standards for safety and efficacy are paramount. After approval, the focus is on demonstration of clinical utility and comparative effectiveness. Quinn recalled the studies described by Ginsburg (see Chapter 4) where patients whose gene expression profile predicted low risk were in a control (observation) group; however, if the gene profile predicted high risk, there was a control group as well as a therapy group. One problem with this approach, Quinn said, is that the genetic test is not being compared to something else. True, the results of the genetic test can help assign therapy, he said, but if the gene panel costs $400, and a $50 antibody to do the same thing exists but was not part of the trial, that is a comparative question that an insurer would ask.

Once a test has been approved for insurer coverage, adoption in the marketplace becomes the focus. Based on his experience as an insurer medical director, Quinn said adoption is also driven by economics, whether there is a profit margin for the physician or the institution. If a test does not bring in a profit, it is unlikely to be adopted. If there is a $30 profit margin, it will probably be adopted fairly quickly. Adoption also depends on the patient perception of the benefit. If the patient sees no benefit, then compliance or acceptance will be poor.

With regard to medical education, the basic biology of genetics is not that complicated, Quinn said. But to teach about all of the actual genes involved would be a mountain of information. For example, six genes are involved in tamoxifen dynamics, and five alleles for each of the six genes. Quinn said the educational challenge is to impart the right concepts to the physicians at the right time, rather than strictly educating about basic concepts or the details of thousands of genes.

In reviewing the way technology advances outside of medicine, Quinn noted that advances in the technology itself is only one part of the adoption and benefit of the technology. Often there is a long rollout period to understand different uses of the technology. Global Positioning System (GPS) devices, for example, are far better now than they were 4 or 5 years ago, but not because anything about GPS technology has changed. The satellites have been in place for 10 or 20 years, but the way the interfaces work and how people use them has changed. A panel conducted in 1980 or 1985 would probably have concluded that personal computers were not useful or cost effective, and that people should not have personal computers. But technology stumbles forward, there is investment over time, and people learn how to use the technology better. There is a risk of killing programs prematurely, Quinn said, just as there is the risk of asking for things that are actually impossible.

There have been several generations of radiation therapies for prostate cancer over the last couple of decades, Quinn said. First was standard radiation, approximating fields where the prostate would be, then conformal radiation, which was somewhat more accurate, then a technology called Intensity Modulated Radiation Therapy (IMRT), which is still more accurate. Now there is proton beam radiotherapy, which some centers have aggressively put into practice for prostate cancer treatment, and still another technology called stereotactic radio beam therapy. None of those have come out with a lot of data. IMRT was adopted fairly quickly once the Current Procedural Terminology (CPT) code was assigned. But if implementation had been slowed down, it could have created a difficult situation—when a company spends $200 million to develop an IMRT machine, it cannot then lay off the employees, mothball its equipment, and wait 10 years for outcomes data, Quinn said. How the process should work is not exactly clear, he said, but if RCTs were the standard required for technology advance, radiation therapy would still be given by laying a towel on the patient and placing a lump of radium on it. There would not be the somewhat messy, somewhat garbled lurching forward, which is the way technologies advance, whether they are automobiles, television sets, or medical technologies.


University of Washington

Evaluation of Genomic Applications in Practice and Prevention (EGAPP) is an initiative from the Centers for Disease Control and Prevention tasked with establishing an evidence-based process for assessing genetic tests. Berg highlighted recommendations that EGAPP has released to date regarding the use of four specific genetic tests (EGAPP, 2009).

The first review focused on CYP450 genetic testing related to the use of selective serotonin reuptake inhibitors for the treatment of depression. The EGAPP working group found insufficient evidence to support a recommendation and discouraged the routine use of CYP450 testing until further clinical trials are completed.

The next review involved UGT1A1 genotyping in patients with meta-static colorectal cancer and the use of irinotecan. Again, the working group found the evidence insufficient to recommend for or against genotyping as a way to predict adverse drug effects.

In its review of Lynch syndrome, EGAPP found sufficient evidence to recommend offering genetic testing for Lynch syndrome, but decided that the evidence was insufficient to recommend a specific genetic testing strategy. Several approaches could be taken, Berg said, and the evidence was not sufficient to favor one over another.

Finally, EGAPP reviewed breast cancer gene expression profiling and found insufficient evidence to make a recommendation for or against its use because it was not possible to assess the balance of benefits and harms of the proposed use of the test. EGAPP encouraged further development and evaluation of the technologies.

Berg offered this brief EGAPP activity summary not only to provide the Roundtable with the current working group findings, but also to point out a recurring theme—the conclusion that there is often “insufficient” evidence to make recommendations. Most people now agree that more evidence on new technologies is needed.

To fill in the gaps in the systems for generating evidence, Berg highlighted three issues from the day’s discussions, all related to funding. The first issue relates to funding and infrastructure. Several speakers discussed the importance of funding infrastructure for data collection, registries, sample banking, or other activities. The second issue is funding for the research itself. The right balance of public–private partnerships and public support needs to be implemented. Finally, it is important to continue to fund “thinking.” One of the activities the Roundtable does well is to thoughtfully consider a variety of topics. Participants leave Roundtable workshops with a renewed sense of priorities.

Thinking of outcomes beyond morbidity and mortality is important. EGAPP is preparing a manuscript discussing various outcomes, including some not mentioned at the workshop, such as the value of information to family members. Issues of economic analysis and modeling need attention. A vast number of polymorphisms seem to be related with fairly modest relative risks to cardiovascular disease, diabetes, and other disorders. There are not likely to be RCTs to study these, but modeling could provide useful information.

Berg said that as a family physician, and not an expert in genetics, each patient in his office is basically a clinical trial with a single participant. He collates all of the information he can extract from the patient history and from biological samples, and works with the patient to develop a strategy that will have the largest predicted balance of benefits compared to harms. But unintended and unexpected consequences always arise.

The main question is not whether more comprehensive genetic information is going to advance medicine in 5 or 10 years, or even 50 or 100 years, but whether it ever can. The problem is that genetic knowledge increases the number of factors that must be brought to bear on each clinical encounter with the patient. The more factors there are, the less likely a physician will be able to find good comparators for the patient sitting in front of him or her and be able to make predictions about care. Genetic variation is infinite. Multiply that by the number of environmental factors, and the answer is that the number of useful comparators will always be near zero. Every patient is unique genetically and in his or her environmental experience. If individual whole-genome profiles become common, a real concern will be providing information that will help inform, not hinder, clinical practice. Significant harm could result from looking for correlations, and trying to characterize patients in ways that actually complicate their care. Berg urged continued funding for dialogue such as the Roundtable, to help set goals for where genetic testing should be 10, 15, 20 years into the future, adequately addressing what is possible versus what would be useful.


University of California–San Francisco, School of Pharmacy

The Center for Translational and Policy Research on Personalized Medicine (TRANSPERS) at the University of California–San Francisco (UCSF) was launched in late 2008, and is focused on evidence-based assessment of personalized medicine and health outcomes. Phillips highlighted four steps needed to close the evidence gaps for personalized medicine and genomic technologies (Phillips, 2008).

The first step is to document the gaps in knowledge about actual clinical practices. This is accomplished by data analysis and through forums that bring together different perspectives. UCSF has developed approaches to analyzing medical records and claims data to better understand what is happening in the real world. Economic analyses are also important. Second, documentation, procedures, and interpretation of genetic tests should be standardized. This would help improve communication between laboratories and clinicians. A third area, which Phillips noted was not covered during the workshop, is providing incentives to close gaps. Policies can be developed that encourage generation of the type of data needed. One example is a policy implemented by UnitedHealthcare that requires clinicians to submit documentation of a positive HER2 test with the first trastuzumab claim. Lastly, creative approaches need to be developed to build the evidence base. Phillips noted that a variety of creative approaches were discussed during the workshop, including better coding and public–private partnerships.

Private Payers

One area that was not addressed in any depth during the workshop is the role of private payers. The seven largest health plans in the United States represent 100 million patients. Reimbursement issues are a significant barrier to moving the field of genome-based medicine forward. TRANSPERS held roundtable discussions and board meetings with health plans to understand what kind of evidence they are seeking, how they make decisions, how they interpret the evidence, and what incentives could drive collection of the right kind of evidence so that payers can make appropriate decisions. Payers agreed that is important to address the evidence gaps and the generation of evidence. Many factors lead to gaps, and a variety of solutions can help to close those gaps. Some solutions that appear obvious may be infeasible; for example, there are some barriers to using genetic modifiers for CPT codes.

Phillips was surprised by how different the payers are, and how different their product coverage decisions can be. They all agree, however, that the biggest challenge is the lack of clinical outcomes data. To help address this, TRANSPERS is working on methods to link claims data and patient charts so that use and outcomes can be tracked.

Payers are interested in developing evidence frameworks, but one finding that has become clear, Phillips said, is that one framework cannot be applied across the board to all payers, or relative to all topics. In addition, clinical utility is not the only endpoint. Contextual factors must be considered, such as FDA approval, political pressure, or physician demand, and TRANSPERS is developing a taxonomy of evidence gaps.

Private payers bring an important perspective to the evidence debate, but better mechanisms are needed to facilitate their involvement. TRANSPERS offers one mechanism for payer input, but more are needed. In closing, Phillips said it is useful to consider how various stakeholders view evidence gaps, and it is important to work with the stakeholders on using their data.


Wylie Burke, M.D., Ph.D.


Knowledge and Contemplation

A participant commented on how rapidly patients are admitted and discharged, noting that most of that limited time is consumed by doing something to patients to justify the admission and discharge. This leaves residents and medical students with no time to think because contemplation is not reimbursed. Medical schools require more and more subjects in less and less time, all of which are evaluated by written examinations. There is more focus on passing the standard exams, and less interest in thinking. Physicians and scientists need time to think about what they are doing. Otherwise, the participant said, medicine will be ruled by algorithms and practice, which is not good for science or for human care.

Berg concurred with the comment about medical education, but questioned whether medical knowledge is really increasing geometrically. Some of the EGAPP reports start with hundreds of articles to review and boil down to four or five of substance. The number of times that research is truly changing practice is very small. Many articles add important information to a larger body of evidence on a topic, but Berg argued that evidence that changes practice is not actually increasing that fast.


A participant said that “coverage with evidence development” is essentially asking payers to begin funding research as part of their mission. Some research is funded through standard payment mechanisms. For example, children’s oncology treatment is often done in the context of collecting data. But to move toward coverage with an evidence development model, the payer needs incentives to be involved, because the margin in most cases is fairly thin. The participant supported the TRANSPERS effort to develop a taxonomy of gaps, and agreed there is great variety in the methodologies that payers use for technology assessment. A menu of frameworks for how payers might approach this would be very helpful, as well as a repository of evidence cataloging the gaps.

Phillips responded that she sees industry as paying for the evidence development, rather than payers. In other words, payers would not cover a technology unless the evidence was produced.

Quinn noted that there could be occasions where the new therapy was the same price as an existing therapy, making it cost neutral to cover, and entering information into a disease registry could be a requirement of coverage.

A participant asked about the role of the Centers for Medicare & Medicaid Services (CMS) in the generation of evidence. Does the fact that most Medicare coverage is for those aged 65 years and over play a role in CMS’s knowledge or understanding of genetic issues?

Quinn responded that the Medicare system basically has no policies regarding genetic testing. Medicare is considering whether or not to cover genetic testing related to warfarin. Many genetic health issues would be uncommon in Medicare’s general population, but 10 percent of Medicare patients, or 4 million people, are under 65.

Phillips added that the popular thought is that private payers tend to follow Medicare decisions. But even Medicare is not a monolithic plan, and local and national coverage decisions can differ. Companies will sometimes seek a local decision that then preempts a national decision. Payers are all considering the same factors, but how they interpret those factors is where the variation occurs.

Technology Development

A participant said market behavior is one difference between health care technologies and technologies such as the personal computer. The people who recognize and reap the benefits of personal computers, and the people who pay for them, are the same. Therefore, if the technology is good, it develops into a market. This is not the case in health care. Also, in the case of personal computers or similar technologies, the benefits are realized almost immediately. In the case of chronic diseases, the benefits may be realized many years later, or they may be imperceptible. For example, if the technology reduces the risk of developing a cancer from 10 to 5 percent, 90 percent of people would not normally benefit, and there is no way to know who the 5 percent who would benefit are. The beneficiaries are not paying for the technology. It is a different kind of market structure.

An audience member stressed the importance of the patient perspective in development and translation. For example, he said, a challenge to conducting RCTs comparing surgery versus radiation therapy was that patients were not willing to be randomized to a treatment group. They had clear preferences for either surgery or radiation therapy. Another example he offered was that it was patients who saw the positive impact of mammogram screening for breast cancer who lobbied Congress for prostate-specific antigen screening for prostate cancer.

A participant commented that companies should be thinking about prospective evaluations of medications during the development process, so that treatments come out with prospectively evaluated companion diagnostics.

Another participant said that technology should not be released without at least some system in place to study the outcomes. A system of prioritizing is needed to determine what needs to be studied by RCT and what can be studied by observational data. Medicine has a long history of new technologies, such as computerized tomography (CT) scans or mammography, coming out with compelling, intuitive information, which then makes it almost impossible to study via a classic RCT. These are the situations where it is absolutely necessary to study the impact of this technology on human health. Berg agreed, and said that as genetics research identifies more variables in the characteristics of individuals, it exponentially increases the size of randomized trials necessary to look for differences in outcome.

Are Genetic Tests unique?

Panelists were asked whether the same standards that apply to all other aspects of medicine should apply to genetic tests, or if there is something special or different about genetic tests in terms of the need for evidence, the associated politics, or the public demand.

Quinn responded that in general, there is nothing special about genetic tests. Germ-line testing is perhaps a little different because it is conducted once in a lifetime. Berg agreed, with the exception of the impact genetic testing can have on family members of the patient. He cited the EGAPP analysis of Lynch syndrome, where genetic testing was recommended because of the potential benefit to family members. The tests that one does in medical care otherwise generally do not have implications for family members. Burke said that from the clinical genetics perspective, the tests that tend to fall out as different are the ones with high predictive value, which are generally the ones for single-gene diseases.

International Collaboration

A participant stated that in the basic research arena, there has been significant international collaboration through the Genomewide Association Studies, which leveraged the different strengths of different nations. He asked the panel to comment on the extent to which international collaboration might be useful for translational research, noting that many other nations have complementary or more extensive registries, better electronic medical records, and single-payer systems.

Berg agreed and said colleagues in other countries are more likely to be able to deliver on promises of research because they have health care systems that make coverage decisions differently than the United States, and they may not implement practices until they have a certain level of evidence. Where the United States has struggled to conduct RCTs on prostate cancer, trials in Europe are proceeding well because they have enough patients who have not been screened to be able to conduct the studies. The health technology assessment programs in the United Kingdom and Australia are quite sophisticated, and Berg said there is much to be gained from international cooperation on issues of evidence, not only on straightforward issues such as screening, but also on some of the treatment questions that have been discussed.

Prostate Cancer Radiotherapy as a Translation Case Study

A participant from Blue Cross Blue Shield expanded on Quinn’s example of prostate cancer radiotherapy. Escalation in cost with relatively limited evidence about the value proposition is of concern, she said. The move from conformal radiation to IMRT increased costs per case from about $10,000 to $40,000. The known advantage of IMRT is a slight decrease in incidence of proctitis. That is a huge cost impact for treatment of a common disease. Cost estimates for proton beam radiotherapy will be considerably higher. It is important to understand whether these procedures add value to the health care system, or create distortions, especially when 45 million people are uninsured. She added that there are also distortions in the delivery of care. When Massachusetts adopted universal coverage, they found they lacked primary care physicians. The participant said there is an overemphasis on intensity of care and specialties, and an erosion of the core of the health care system, the primary care physician.

Quinn agreed that there is not a great deal of evidence for the escalation in cost of radiotherapy, and noted that the radiation oncology association has not commented about the increased costs. The participant added that there has been no willingness on the part of the professional group to promote comparative trials among radiotherapies or in comparison to other therapy. There is almost a lock on the development of information, but the costs keep growing.

Berg commented that prostate cancer is a very instructive example. The U.S. Preventive Services Task Force continues to be skeptical about screening for prostate cancer, and actually advises against it at age 75. Nonetheless, Medicare implemented screening when former Senator Robert Dole developed prostate cancer, and members of Congress approved Medicare coverage of the screening test. There still is no RCT showing that radiation treatment at any stage of the disease provides any benefit. Only one RCT shows that radical prostatectomy increases survival. Yet despite the insufficient evidence base, screening continues on a large scale. The problems facing genetic testing are similar to problems elsewhere in the system. Policy decisions and coverage decisions are made not only in the absence of the evidence, but sometimes in defiance of the evidence. Prostate cancer is an interesting case study where some things have worked well, while others continue to be baffling.

A participant said that IMRT is the classic physician conundrum. Radiation therapists try to deliver a dose to the tumor, which can be done with a few crude beams. IMRT was really a physics algorithm that allowed outlining of exactly where the tumor was, facilitating delivery of a radiation dose to the tumor and delivering less to the normal tissues. It became hard for doctors to not use IMRT, and to keep irradiating normal structures at high doses when unnecessary. Genetics is different in many respects, but the field is evolving in a way that clinicians have to decide how to best treat their patient given a great deal of uncertainty. The equalizer is going to be computer technology, and the ability of the physician to obtain data, proper analyses, and proper consensus when they need them.

Another participant said more large-scale randomized trials of IMRT or proton beam therapy are not what is needed now. What is needed is a better basic genetic understanding of prostate cancer so that men could be stratified by risk, and determinations could be made about which patients do not need treatment.

An audience member added that when considering prostate cancer treatment, those framing the question failed to ask what would be the incremental benefit over doing nothing. They considered one therapy versus another. There were no differences among the therapies in terms of benefit. The main difference was in terms of harms, and they were relatively modest.

Priorities for the Next 5 years

A challenge for the Roundtable, a member said, is that genome-based health is such a diffuse and enormous field with broad stakeholder representation. He asked the panel what they thought should be priorities for translating genomics into health care for the next 5 years. What is the best investment of intellectual and tangible resources?

Berg said family physicians in primary care take all comers, and never know what will be behind the door. Having tests or innovations that can be applied in that setting is no different for a genetic test than for anything else. Many factors compete for the attention of clinicians right now. Although most would agree that it would be great to find opportunities to use genetic tests, so far in his practice, Berg did not know of anyone using a genetic test for a common clinical condition. To be a viable business model, the tests need to be applicable to common clinical scenarios. Like any other innovation, it would need to be fast enough and cheap enough, and have demonstrated improvement in clinical outcomes in order to be adopted into practice.

Phillips said the Roundtable has a unique opportunity to bring the many different perspectives together, and to look across issues in an objective way. More generally, personalizing medicine will continue to be of interest, whether through genetics, family history, or various other means. There is a push toward comparative effectiveness and toward maintaining quality while reducing cost. The Roundtable’s priorities should be developed within that context of where the health care system is going overall.