In this report, we addressed four key questions targeting the four objectives of the assignment. For Key Question 1, we addressed whether it is feasible to clarify a comprehensive framework or a limited set of frameworks for evaluating genetic tests by modifying existing frameworks. This key question encompasses all test evaluation areas, including analytic validity, clinical validity, clinical utility, and societal impact. In the report, we define evaluation frameworks as conceptual approaches to the evaluation of a health care technology and to organizing the relevant evidence. Evaluation frameworks are tools for clarifying the scope of the questions to be addressed in the review and the nature of evidence necessary for answering the questions.

Overall, the Workgroup and the ECRI Evidence-based Practice Center (EPC) agreed that, for different stakeholders (e.g., patients, providers, payers, regulators, and test developers), the priority in the issues that need to be addressed in evaluation of genetic tests could be different. These different stakeholders may need somewhat different frameworks for their evaluation tasks. However, the perspectives of patients matter the most and should serve as the ultimate guide to the evaluation efforts by all other stakeholders. It would be a valuable effort to clarify a set of frameworks for evaluating genetic tests from patients' perspectives.

Based on the findings of the targeted review and the input from the Workgroup, the ECRI EPC presented a set of frameworks for some common testing scenarios, including diagnosis in symptomatic patients, screening in asymptomatic patients, risk/susceptibility assessment, treatment monitoring, prognostic assessment, and pharmacogenetic evaluation. These frameworks are primarily based on the frameworks—draft and published—developed by the Evaluation of Genomic Applications in Practice and Prevention initiative (EGAPP) Working Group for evaluating genetic tests.^{5,8,40,41,52,58} Each framework presented in this report includes a diagram that visually presents the relationships among the population, testing, subsequent interventions, and outcomes. Under each framework, analytic validity, clinical validity, clinical utility of the test, as well as potential harms associated with the testing and subsequent interventions, are evaluated. Under the frameworks, both intermediate and health outcomes are also examined for evaluating the clinical validity of the test. The key research questions generated under these frameworks particularly emphasize the importance of comparing the effectiveness of the test with that of current standard-of-care testing strategies.

In this report, we also presented frameworks for evaluating germline-mutation testing, which is far more complicated to evaluate than somatic mutation testing. We further provided examples demonstrating how to customize the frameworks presented from patients' perspectives to meet the needs of other stakeholders for evaluation of genetic tests. It is our hope that the proposed frameworks will help to decrease the inconsistency among the evaluations performed by different assessors. However, we acknowledge that these frameworks might not address all needs that an assessor may have when evaluating certain tests. There is always a possibility that some customization of the frameworks might be needed before they can be appropriately applied to the specific evaluation scenario.

The other three key questions addressed in this report all focused on analytic validity. For Key Question 2, we were asked to examine different approaches to literature searching to assess evidence on variability in genetic testing and, if possible, to recommend an optimal approach to literature searching. To address this key question, we performed a targeted review of the search strategies used in completed evidence reports and sought input from the Workgroup experts. Our targeted review found that a lack of published data remained a major challenge to evaluating analytic validity of genetic tests.^4-8 Searches for unpublished data are often needed in order to address analytic validity issues in evaluation of genetic tests.

In this report, we recommend a comprehensive search strategy for identifying data—published or unpublished—for evaluation of analytic validity. Particularly, we summarized common sources of unpublished data with guidance from the Workgroup. Potential strengths and limitations of these gray literature sources were also compared. We believe that a systematic search of published and gray literature sources would help to identify the data that are required for the evaluation of analytic validity. However, whether the data identified meet the minimum quality standards would need to be critically evaluated by the technology assessors.

For Key Question 3, we addressed whether it is feasible to apply existing quality rating criteria to analytic validity studies on genetic tests and evaluated whether there is an optimal set of quality rating criteria for those studies. For this report, we only focused on quality of individual studies, including such concepts as systematic bias and reporting adequacy, and validity of statistical analysis. In addition, external validity (or generalizability) should be examined from the perspective of the intended audience. Rating the overall strength of evidence base is beyond the scope of the report.

Our targeted review found that there was no consensus among the authors of these evidence reports on what criteria should be used for judging the quality of analytic validity studies. Our review identified one instrument that was designed specifically for assessing the quality of analytic validity studies.³ However, this instrument is difficult to use without major revisions. Having identified a need to construct a comprehensive, but easy-to-use, quality instrument for assessing the quality of analytic validity studies, we proposed a set of criteria to be utilized for this purpose.

The list was proposed by synthesizing the quality criteria from an instrument proposed by the EGAPP group³ and other sources (e.g., the criteria used in completed evidence reports and the guidance documents used by regulators for reviewing test packets). The list consists of 17 items that cover various quality aspects including internal validity, reporting quality, and other factors potentially causing biases (e.g., funding sources). Given the time frame for this report, it was not feasible for us to empirically validate the list. However, we believe this list provides a practical tool for choosing important criteria for identifying major quality flaws in analytic validity studies.

For Key Question 4 of the report, we evaluated existing gaps in evidence on sources and contributors to variability in performance of genetic tests. To address the question, we used three case studies of tests to demonstrate the issues test evaluators may experience when attempting to evaluate the analytic validity of tests. We chose three tests that represent the primary categories of testing: molecular tests, cytogenetic tests, and biochemical tests. We searched for literature and information on analytic validity to investigate the gaps in evidence sources.

Upon review of these case studies, we found that there are several major challenges systematic reviewers must overcome when conducting systematic reviews of the analytic validity of the tests. One challenge is lack of consistent evidence on analytic validity that is generated using scientifically sound methods. This occurs due to various reasons such as lack of appropriately validated samples for test validation and lack of “gold standard” reference methods techniques.^1,2,9 To fill the gaps, a higher level of collaboration among the research community, professional societies, and test developers would be needed in efforts such as increasing the availability of appropriately validated samples that can be used for test validation, developing effective reference methods, and building sample-splitting or sample-sharing programs.

Another challenge is how to obtain information that already exists about the analytic validity of tests. With few exceptions, searches of the published medical literature often yielded limited to no information on analytic validity of the tests being evaluated. What we found in these case studies echoes what other authors have found in their work.^4-8 According to the Workgroup, the data that are useful for evaluation of analytic validity may be scattered in different places, such as clinical laboratories and proficiency testing programs. An extensive search of gray literature may provide some data; but in the long run, stakeholders in the field of genetic testing should work together to establish more effective mechanisms to make the data accessible to all parties who might need them.

The experts who participated in our Workgroup suggested some solutions to fill this gap, including regulatory mandates for release of data or creation of a database of deidentified validation data (i.e., the laboratory from which the data came would not be identified) by an independent accrediting organization such as the College of American Pathologists or by a professional society such as The Association for Molecular Pathology. However, until and unless there are policies that provide incentives for laboratories to release what is generally considered proprietary analytic validity data, those doing systematic reviews of genetic tests, particularly laboratory-developed tests, will be left using the types of search strategies for published and unpublished information discussed in this report.