Importance of a Well-Designed Development Plan

As described in Chapter 2, an important aspect of test development is a well-designed development plan. Components of a test development plan include starting with a clinically meaningful question, developing a candidate test on a training set of specimens, locking down the candidate test, and employing rigorous test validation procedures. The availability of appropriate archival tissue for clinical/biological validation can greatly facilitate rigorous development procedures. If appropriate archival samples are not available for assessment of clinical utility, then the development plan should include a prospective clinical trial design.

Tests developed in universities (or where the developmental process is started in universities) are likely to grow out of basic research on the biological meaning of the omics elements that comprise the test and gradually evolve into more formal test development. This is illustrated by the studies of HER2 and MammaPrint, although the development of MammaPrint crossed into a more formalized approach after Agendia was formed. In contrast, companies are likely to start with a focus on a specific test that will eventually have commercial value and thus generally have a clear development plan early on, as seems to be the case with Genomic Health. According to a presentation to the IOM committee by Steven Shak, chief medical officer of Genomic Health, Oncotype DX had a clearly articulated development plan from the start. Investigators defined the purpose of the test and subsequently created and implemented a multistep, multistudy approach to develop their omics-based test “to provide the evidence regarding analytic performance, clinical[/ biological] validity, and clinical utility to meet the needs of patients, physicians, payors, and regulators” (Shak, 2011).

Data and Code Availability

As described in Chapter 2, the committee recommends that data and metadata used for identification of an omics-based test should be made available in an independently managed database in standard format, and that code and fully specified computational procedures for omics-based tests should be made sustainably available. Chapters 2 and 5 discuss the importance of sharing data within the scientific community, especially in the setting of omics research, where data and analyses are particularly complex. Sharing data can enable external verification of the results and allow other investigators to generate additional insights from the data. In Chapter 5, the committee recommends that journals and funders require the public availability of data, metadata, prespecified analysis plans, code, and fully specified computational models.

The importance of data availability was highlighted in the OvaCheck case study, in which publicly available datasets enabled external researchers to uncover serious problems in experimental design (Baggerly et al., 2004) that ultimately demonstrated that the published computational model (Petricoin et al., 2002) was based on artifacts and not on relevant biological signals.

In the Duke case, external investigators did not have full access to the data and code, and this limited the ability to independently evaluate the tests to determine whether they were valid (Baggerly and Coombes, 2009; Baron et al., 2010). Conflicting and unclear information in the papers and cited references regarding the data and statistical methods contributed to the inability of colleagues in the scientific community to understand and replicate the generation of the computational models (Baggerly, 2011; McShane, 2010; Review of Genomic Predictors for Clinical Trials from Nevins, Potti, and Barry, 2009). When Baggerly and Coombes attempted to assess the validity of the tests, they determined that there was insufficient information to reproduce the published results using the available data and the methods published in the Nature Medicine paper (Baggerly, 2011; Potti et al., 2006).

A review of the six commercially available tests (Table 6-2) illustrates that public availability of all omics-based test data and code has not been the standard of practice. The field of omics is early in its development and the standards for data sharing have been unclear and only slowly evolving toward more transparency. Commercial interests and protection of proprietary information may also limit the public availability of some data and information.

TABLE 6-2

Data Availability.

NOTE: FDA = Food and Drug Ad ministration, GEO = Gene Expression Omnibus, RT-PCR = rcvcrsc-transcriptasc polymerase chain reaction.

^aPersonal communication, Laura van ‘t Veer, Agcndia, November 1, 2011.

^bPersonal communication, Steve Rosenberg, CardioDX, December 12, 2011.

The cases highlight several examples in which test developers explicitly note the availability of data. For example, Paik et al. (2004), Deng et al. (2006), and Rosenberg et al. (2010) report the computational model for Oncotype DX, AlloMap, and Corus CAD, respectively. Both tests developed as laboratory-developed tests (LDTs) had published computational models (Oncotype DX and Corus CAD); only one FDA-cleared test has a published computational model (AlloMap). Discovery microarray data are available for MammaPrint, AlloMap, and Corus CAD (Deng et al., 2006; van ‘t Veer et al., 2002).¹ Buyse et al. (2006) report that raw microarray data and clinical data for the MammaPrint clinical validation study were deposited with the European Bioinformatics Institute ArrayExpress database. Although there are examples of developers reporting the availability of a test’s computational model or data used in discovery or validation, as Table 6-2 shows, there often is not enough information publicly available for external investigators to fully reproduce a test.

The IOM committee recognized that it might not always be possible to make this information publicly available due to the protection of intellectual property. For publicly funded research, the committee recommends that code and fully specified computational procedures should be made available at the time of publication or at the end of funding. For commercially developed tests, code and fully specified computational procedures would be submitted for FDA review if seeking approval or clearance, or would be described in a publication in the case of an LDT (see Chapter 2). Companies that seek FDA clearance or approval for their tests would have had to submit data to FDA as part of the 510(k) clearance processes or premarket approval (PMA) processes, respectively, but only the information reported in the FDA decision summary is made publicly available.

Avoidance of Overlap of Discovery and Validation Specimens

One of the challenges of omics research is the lack of available specimens on which to conduct correlative science and exploration of candidate omics-based tests (reviewed in IOM, 2010). Partly due to this challenge, a number of tests have been developed with overlapping training and validation datasets. In the cases that the committee reviewed, two commercial omics-based tests used overlapping training and development datasets at some point in their development processes: MammaPrint and AlloMap (Table 6-3). Of the samples used in the development of the MammaPrint computational model, 78 percent were reused in the van de Vivjer et al. (2002) study, leading to criticism about the overlap between training and validation datasets (Kim and Paik, 2010; Ransohoff, 2003, 2004). A subsequent validation study (Buyse et al., 2006) was performed, and suggested that overfitting due to the use of training samples was likely a problem in the 2002 study. In the AlloMap case, the first validation study used 63 specimens that had not been used in previous development of the test. This second validation included the 63 primary validation samples plus samples from patients who had contributed samples to the gene discovery and/or diagnostic development phases of the study. Overlap of training and validation datasets can lead to a number of problems in test discovery and development, including overstatement of the accuracy of an omics-based test and incorrect error estimation (Leek et al., 2010). The OvaCheck case study illustrates the importance of independent validation datasets (Diamandis, 2004). If test samples are obtained independently, preferably drawn from another institution at a different point in time, and prepared and analyzed separately, then the risk of batch effects is greatly reduced. As described in Chapter 2, test developers should clearly separate training and validation datasets in order to avert these problems, which are among the most common causes of a test ultimately failing clinical validation.

TABLE 6-3

Statistical and Bioinformatics Validation Considerations.

Locking Down All Aspects of the Test Prior to Evaluation for Clinical Use

In Chapter 4, the committee recommends that a validated omics-based test should not be changed during a clinical trial. The omics-based test should be locked down prior to this stage of development (more information on the importance on locking down an omics-based test can be found in Chapters 2, 3, and 4). As noted by Baggerly and colleagues and by McShane, the computational models developed by a Duke University lab were not locked down prior to use in the clinical trials (Baggerly, 2011; McShane, 2010). This was a serious shortcoming in the development of the Duke omics-based tests. For three other cases, the developers explicitly stated in the study publication that their test was locked down before clinical validation: Oncotype DX (Paik et al., 2004), Tissue of Origin (Monzon et al., 2009; Pillai et al., 2011), and Corus CAD (Rosenberg et al., 2010). Communication with test developers provided additional confirmation that AlloMap, OVA1, and MammaPrint were locked down (Table 6-3).

Interaction with FDA

In Chapter 4, the committee recommends that FDA clarify the regulation of omics-based tests by developing and finalizing a guidance or regulation defining which omics-based tests require FDA review, the type of review required, and when this review should occur. A similar guidance should be developed and finalized for oversight of LDTs that are currently not reviewed by FDA.

Review of the committee’s case studies demonstrates that companies have pursued both LDT and FDA pathways for developing an omics-based test. The use of multiple pathways indicates a lack of clarity and consistency on the regulatory requirements for omics-based tests. Five of the commercially available tests that the committee examined are performed exclusively by each company’s proprietary Clinical Laboratory Improvement Amendments of 1988 (CLIA)-certified laboratory.² Two companies did not seek FDA clearance and market their tests as LDTs: Genomic Health (Oncotype DX) and CardioDx (Corus CAD). Four companies received FDA 510(k) clearance of their tests: Agendia (MammaPrint), Pathwork Diagnostics (Tissue of Origin), Vermillion (OVA1), and XDx (AlloMap). The publicly available 510(k) clearance decision documents summarize the analytical and clinical validation results that the company/sponsor submitted to FDA. However, FDA clearance does not mean that a test has clinical utility, and lack of FDA clearance does not mean that a test does not have clinical utility. This is an important distinction between the FDA approval process for drugs versus devices (which includes omics-based tests).

In Chapter 5, the committee also recommends that FDA communicate the investigational device exemption (IDE) requirements for omics-based tests used in clinical trials. Commercial developers, such as those examined in the case studies, may be more familiar with IDE requirements than academic institutions. In several of the case studies, companies and FDA held a pre-IDE meeting to determine whether an IDE would be required for the company’s test development process. FDA determined that an IDE was not needed for both the AlloMap and Tissue of Origin tests because the test was not directing patient therapy in the studies proposed to assess the test.³ Physicians can now use the test for that purpose, however. Agendia reported that it received an IDE for MammaPrint that helped clarify the process and requirements for the de novo 510(k),⁴ and Vermillion reported that it received an IDE for OVA1.⁵ Two ongoing prospective studies (the TAI-LORx and RxPONDER trials) direct patient management on the basis of the Oncotype DX Recurrence Score. For both trials, information required for approval of investigational use of Oncotype DX in the trial was submitted as part of an investigational new drug application to FDA.⁶ In the Duke case study, the investigators did consult FDA regarding the need for an IDE (FDA, 2011a). In 2009, FDA sent a letter to Duke stating that the omics-based tests being studied in the three clinical trials named in the IOM statement of task needed to go through the IDE process (Chan, 2009). In response, the investigators made some changes to the protocol of the studies, and Duke contacted FDA for further clarification about whether an IDE was still required (FDA, 2011a; Potti, 2009). The Duke Institutional Review Board (IRB) determined that an IDE was not needed when it did not receive a reply from FDA⁷ (FDA, 2011a). However, in retrospect, the Duke IRB recognized that an IDE should have been obtained for the omics-based tests because the tests were used to direct patient management in the clinical trials (FDA, 2011a).

Regardless of which pathway is taken to market, consultation with FDA can be beneficial and is recommended. For example, the developers of the OVA1 test sought FDA input, and this early dialogue with FDA prompted Vermillion to include two different cut-off values for the test, depending on a patient’s menopausal status (Fung, 2010). Although Genomic Health did not meet with FDA, the company indicated that it benefited from past experience working with FDA and from the extensive background material FDA provides on its website about assay validation.⁸

Clinical/Biological Validation Characteristics

The choice of study design for the clinical/biological validation studies of the tests varied widely among the case studies (Table 6-4). Designs included retrospective studies, a prospective–retrospective study (using archived specimens from previously conducted, formal clinical trials that evaluated treatment options that might be affected by the test’s use), and prospective trials in which the test was not directing therapy. In the Duke case, the investigators attempted to validate some of the tests using cell lines, or in one case, clinical samples from patients with breast cancer (Bonnefoi et al., 2007). However, prospective clinical trials in which the omics-based tests were used to direct patient therapy were initiated prematurely, before clinical/ biological validity had been established.

TABLE 6-4

Choice of Trial Designs for Clinical/Biological Validation.

As described in Chapter 3, the committee recommended that the identity of the specimens used for clinical/biological validation of the omics-based test should be blinded to the individuals performing and interpreting the test results during clinical/biological validation of the test when feasible. In some instances, this may entail working with another independent group to conduct validation studies. In the case of Oncotype DX, Genomic Health completed reverse-transcriptase polymerase chain reaction (RT-PCR) analysis of the specimens used for clinical validation and supplied the results to the National Surgical Adjuvant Breast and Bowel Project, who conducted the analyses evaluating the association between recurrence score and clinical outcome. In the clinical validation for MammaPrint by Buyse et al. (2006), the data were housed at TRANSBIG,⁹ and the statistical analyses were conducted by the International Drug Development Institute. For Corus CAD, the publication describing the clinical/biological validation noted that an investigator at the Scripps Translational Science Institute was responsible for the data analyses, while the laboratory work was performed at CardioDX.

The development and validation pathways for Oncotype DX and MammaPrint offer some interesting comparisons. Kim and Paik (2010) note that both MammaPrint and Oncotype DX went through their respective development steps either “by design or by demand from the community.” Investigators largely chose different strategies for test discovery and development, including the type of tissue used in discovery, the type of risk score (dichotomous variable versus low-, intermediate-, and high-risk categories); different regulatory approaches; and a different intended use population. Oncotype DX was designed for patients with estrogen-receptor-positive, lymph-node-negative, early-stage breast cancer. MammaPrint has a potentially larger indication, including patients with estrogen-receptor-negative tumors. Despite these differences, MammaPrint and Oncotype DX have shown around 80 percent agreement in outcome classification (Fan et al., 2006), although they only have one gene in common.

Assessment of Clinical Utility

Clinical utility is defined as “evidence of improved measurable clinical outcomes, and [a test’s] usefulness and added value to patient management decision-making compared with current management without [the] test” (Teutsch et al., 2009, p. 11). Assessment of clinical utility is not part of the FDA’s evaluation of a test, and generally, clinical utility is determined after a test or device is on the market, sometimes decades later. Clinical utility is different from the intrinsic attributes of a test’s performance characteristics, such as sensitivity and specificity. Clinical utility is not concerned with how a test performs, but rather how its use influences health outcomes. As described in Chapter 4, the ideal way to assess clinical utility is through prospective randomized controlled trials addressing, for example, whether the use of the new test results in an increase in the length or quality of life, a significant increase in progression-free survival, or avoidance of unnecessary treatment (especially toxic treatment) by patients who are unlikely to benefit. Clinical trials assessing clinical utility are important because they can inform how a test is used in practice.

As described in Chapter 4, prospective trial designs in which biomarkers direct patient management (as in Figures 4-4 and 4-5) are useful approaches to assess the clinical utility of the biomarker. However, they are also the study designs that pose the highest risk to trial participants, because treatment decisions are determined by the test result. The justification for using these designs depends substantially on the amount of information known about the test. For example there were no reported validation attempts using clinical tumor samples from patients with lung cancer for the cisplatin test developed by the Duke University lab, even though the first trial in which the cisplatin test was used to guide therapy was the NCT00509366 trial for advanced lung cancer. This type of trial design may have been premature, given the lack of tumor-specific validation of the cisplatin test.

Evidence on clinical utility evolves as new information about a test emerges, and the absence of data on clinical utility should not be interpreted as a lack of utility. A defining feature of both the MammaPrint and Oncotype DX development stories is the need for a large, prospective trial to provide more information on each test’s utility in clinical practice. In both cases, the prospective trial was not initiated until after the test was on the market and widely available for clinical use as a prognostic factor. The results of the prospective studies TAILORx and MINDACT will provide prospective information on the clinical utility of Oncotype DX and MammaPrint, respectively, for the first time.

The Role of Investigators and Institutions in Scientific Oversight

Chapter 5 comprehensively discusses the roles of responsible parties in the conduct of omics research and omics-based test development. Here, the roles of the investigators and institutions are highlighted for their central role in ensuring rigorous omics research and test development. As illustrated in the Duke case study (Appendix B), transparency and open communication are integral to the conduct of science, whether it be reporting of data and code, disclosure of conflicts of interest, or reporting of potential breaches in scientific procedures.

Investigators are responsible for the accuracy of their data, the fairness of their conclusions, and responding appropriately to criticism. Investigators ensure that clinical research is conducted with the engagement of appropriate scientific expertise, including the involvement of individuals with proper biostatistics and bioinformatics expertise, and that the research has the approval of relevant review bodies. It also is important for all members of a research team to understand the aims and intricacies of collaborative studies and for coauthors of a publication to keep each other informed about constructive criticism of the work and ways to improve ongoing research.

Institutions play an important role in establishing a culture of scientific integrity and transparency, including setting expectations of behavior, achievement, and integrity, and providing safe environments for reporting irregularities to prevent lapses in scientific integrity. Institutions are directly charged with being the “oversight” bodies when specific scientific questions or challenges arise, including investigating questions of misconduct, or simply in investigating “soundness of science.” Oversight processes that will maintain integrity even in the presence of institutional conflicts of interest, both financial and non-financial (such as factors that impact an institution’s reputation) may be especially important in addressing this charge. Closer attention to such conflicts may have been helpful in avoiding the events that occurred in the Duke case (see Appendix B).