NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Trikalinos TA, Dahabreh IJ, Wong J, et al. Future Research Needs for the Comparison of Percutaneous Coronary Interventions with Bypass Graft Surgery in Nonacute Coronary Artery Disease: Identification of Future Research Needs from Comparative Effectiveness Review No. 9 [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2010 Sep. (Future Research Needs Papers, No. 1.)

Cover of Future Research Needs for the Comparison of Percutaneous Coronary Interventions with Bypass Graft Surgery in Nonacute Coronary Artery Disease

Future Research Needs for the Comparison of Percutaneous Coronary Interventions with Bypass Graft Surgery in Nonacute Coronary Artery Disease: Identification of Future Research Needs from Comparative Effectiveness Review No. 9 [Internet].

Show details


This Future Research Needs document is intended to inform and support researchers and those who fund research to ultimately enhance the body of comparative effectiveness evidence so that it is useful for decisionmakers. The current document is a pilot project undertaken by the Tufts EPC that emulates the envisioned process for developing Future Research Needs documents: after the completion of a CER, EPCs would prioritize the identified evidence gaps into research needs, and suggest potential research designs. The premise is that not all evidence gaps are of equal importance, and because resources are finite, future research should address the most important among the answerable evidence gaps using the most efficient research methods.

By their very nature Future Research Needs documents refer to the same clinical context as the CER they are based on. This means that they pertain to similar patient populations, interventions and comparators. For the current pilot we refer to patients with nonacute CAD in whom revascularization is warranted; the intervention is PCI variants and the comparator is CABG variants.

Management of coronary artery disease without revascularization (e.g., only with optimal medical therapy) is a valid clinical option in patients with stable CAD, but it was not included in the Stanford CER and is not considered in this document. We comment on the implications of this decision in the Discussion Section.

Overview of the Stepwise Approach

We sought a practical approach to identifying evidence gaps and prioritizing them into research needs. We used a continuum of qualitative and quantitative methodologies, as outlined below. In the following paragraphs we provide details on our methods, and our rationale.

  1. Initial list of evidence gaps (List 1). We generated an initial list of evidence gaps based on the Stanford CER and initial feedback from a group of key informants.
  2. Interim expanded list of evidence gaps (List 2). We searched the literature to identify trials that have been published after the Stanford CER. We also searched to identify ongoing studies in the field. We then proceeded to further discuss the initial list of evidence gaps (List 1) in one-to-one interviews with key informants taking into account information from the literature searches. This resulted in an interim expanded list of evidence gaps, which included more potential target topics for future research compared to the initial list.
  3. Pruned (final) list of important evidence gaps (List 3). Here we decided to prune the length of the interim list of evidence gaps (List 2) to contain its scope. This step was not anticipated in the beginning of the pilot project but it was considered necessary in view of the expansion of the evidence gaps list at the prior steps. We pruned the list of the previous step internally and according to post hoc criteria, but without direct and explicit feedback from the key informants. We took into consideration data from the literature and the inputs of key informants in the previous steps.
  4. Recommendations for future research. We recommended specific research designs for future research. To this end we used ad hoc criteria to assess the importance of each evidence gap in List 3 to the US setting. Further we estimated the feasibility of different research designs, asked for additional key informant feedback, and performed focused modeling analyses to pinpoint for which parameters it is most important to obtain additional information through future research.

Step 1. Generating the Initial List of Evidence Gaps

We formed a group of key informants to help us better understand the evidence gaps identified in the Stanford CER (Table 1) and how they relate to clinical practice and the current trends in the field.

Identification of Key Informants and First Round of Key Informant Feedback

We identified six key informants, namely a medical officer of a funding agency for cardiovascular research, a medical director in a major payor, an interventional cardiologist, a CABG surgeon, a general cardiologist, and a clinical researcher and methodologist with contributions to the topic who was also an author of the Stanford CER. We did not use a particular method to identify key informants, such as random sampling from a large pool of candidates. A patient representative was not included because we deemed that the patient perspective is peripheral to this exercise. We elaborate on our decision not to include a patient representative in the Discussion Section.

We did not use a formal process such as a Delphi process to elicit and compile input from key informants. Instead, key informants provided initial feedback via a teleconference. In the teleconference we started discussing which subpopulations, interventions, comparators and outcomes represent evidence gaps based on the Stanford CER. The teleconference did not exhaust the discussion. Based on the feedback from the teleconference we generated the initial list of evidence gaps (List 1, described in the Results Section and in Appendix B).

Step 2. Generating the Expanded List of Evidence Gaps

We then proceeded to further refine the list of evidence gaps. To inform additional discussions with the key informants, we searched for relevant, recently published or ongoing studies that were not included in the Stanford CER.

Trials Published After the Stanford CER and Additional Ongoing Studies

The Stanford CER was published in 2007, and reviewed evidence through 2006. To assess its currency and whether any of the identified evidence gaps have been addressed in the meanwhile, we searched PubMed for randomized controlled trial reports between 2006 and 2010 (last search August 4, 2010). We also searched to identify registered ongoing interventional (randomized or nonrandomized) or observational studies (e.g., prospective cohorts) that included patients with nonacute CAD and could inform the comparison of PCI with CABG. Our search strategies are listed in Appendix A.

Second Round of Key Informant Feedback

We then invited key informants to participate in one-to-one teleconferences to continue the discussion of the evidence gaps also considering recent evidence and ongoing studies. Four of the six key informants participated in one-to-one interviews. The interviews aimed to identify the key informants’ perception of the major questions in the field, and if applicable, their rationale on why specific questions may be more important than others. Key informants suggested additional areas for future research that were related to the comparison of PCI vs. CABG, but were not in the scope of the Stanford CER. This resulted in an interim expanded list of evidence gaps (List 2 in Appendix B), which included additional potential target topics for future research. Finally, key informants provided feedback on criteria to prioritize specific research designs (to be used in Step 4).

Step 3. Generating the Final List of Important Evidence Gaps

The expanded list of evidence gaps was too long to effectively develop research designs for, so we eliminated less important gaps based on data from the literature and inputs from the key informants provided during the previous steps.

First, we classified the gaps in the interim expanded list into four thematic areas of future research. The four thematic areas are not overlapping, could be pursued “independently,” and are amenable to different types of study designs (and therefore do not all require extensive resources to pursue). The four thematic areas were:

  1. Comparative effectiveness and safety of PCI vs. CABG
  2. The role of testing to inform choice of revascularization procedure
  3. Enhancing patient participation
  4. Assessing performance

Only the first thematic area was directly related to the scope of the Stanford CER. The remaining were added based on key informant input in the previous steps.

We developed the final list of important evidence gaps (List 3 in Appendix B) by pruning the interim expanded list (List 2) within each of the four thematic areas using post hoc criteria, as described below.

Most Important Evidence Gaps in Comparative Treatment Effectiveness and Safety (First Thematic Area)

We considered as more pressing research gaps that pertained to subpopulations representing a high “disease burden” among patients with nonacute CAD in the US. We estimated the frequency of subpopulations representing evidence gaps in 4 analyses of large clinical registries1 (Appendix Table B3)8–11 as a crude proxy of the corresponding “disease burden.” We selected the 6 subpopulations with the largest percentage in the publication by Hannan et al.,8 which is based on two comprehensive registries from the state of New York, and had the largest sample size. We favored interventions that are in routine use in current clinical practice, and that are likely to be routinely used in the midterm future (i.e., for the following 5 years). We deemed that any future research should collect as much information on outcomes as practically feasible.

Evidence Gaps in the Role of Testing To Inform Choice of Revascularization Procedure (Second Thematic Area)

Based on key informant input from previous steps we distinguished evidence gaps related to the ability of testing to predict patient response to treatment with PCI or CABG. Testing could therefore have an effect on patient outcomes, in that it would affect treatment decisions, which in turn affect patient outcomes.2 The Stanford CER did not review testing to predict revascularization outcomes, and therefore we did not have a systematic overview the state of the evidence available. Instead of explicitly selecting individual tests or combinations of tests for further study, we considered an index list of 4 types of pretreatment testing (invasive and noninvasive) in the following step.3

Evidence Gaps in Enhancing Patient Participation (Third Thematic Area)

The third thematic area of evidence gaps pertained to understanding patient preferences and facilitating shared decisionmaking between patients and their physicians. This area of research was not in the scope of the Stanford CER. We selected two particular gaps that were amenable to observational or experimental study, were indicated as important by the key informants, and covered the range of key informant comments on enhancing patients’ voice.

Evidence Gaps in Assessing Performance (Fourth Thematic Area)

Finally, based on prior key informant input, there are no validated process-based performance measures that could quantify optimal care. The potential payoff of validated performance measures at the level of a health system is substantial. Therefore, we considered this as an important evidence gap to be addressed through future research.

Step 4. Making Recommendations for Future Research

We proposed specific research designs to address the evidence gaps in the pruned (final) list of the previous step. We considered the four thematic areas of future research separately; in principle, the evidence gaps in the four thematic areas are nonoverlapping and can be pursued independently.

We prioritized research needs based on the evidence gaps in the pruned final list (List 3) and considering predefined prioritization criteria. We solicited explicit input from key informants on the relative importance of the evidence gaps in the final list, and obtained insights from modeling analyses. We considered the following ad hoc criteria to prioritize research designs: feasibility in terms of research costs, feasibility in terms of projected study duration, likelihood that the study will have nontrivial findings,4 likelihood that the study will provide unbiased results (to inform clinical practice), and likelihood that ongoing research will address the evidence gap.

Third Round of Key Informant Input

All key informants were asked to rank the evidence gaps in the pruned list and provide any additional feedback via an e-mailed questionnaire. The key informants were asked to consider the qualitative criteria of Appendix Table B4 in their ranking. We did not automatically accept the suggestions of the key informants, but considered it in our prioritization of research designs together with the aforementioned criteria.

Focused Modeling To Identify Important Parameters To Address in Future Research

One would perform further research in topics where decisions have to be made, but there is substantial remaining uncertainty on parameters that can affect these decisions. For example, consider the choice between PCI and CABG among elderly patients.5 For which parameters is it most important to obtain additional information through future research? Assuming a reasonably simple specification of this decisional problem, such parameters may be the prevalence of procedural deaths or procedural strokes; the relative effects for procedural deaths or strokes across the compared interventions; the frequency of long term deaths, myocardial infarctions, strokes, or repeat revascularizations; the corresponding relative effects across revascularization options; patient preferences (utilities); and immediate and downstream health care costs. This has practical implications, because different parameters are naturally amenable to different research designs.6

To gain insights on important parameters we developed simple mathematical models to analyze the choice between PCI and CABG. Details on our quantitative approaches (modeling strategy, assumptions, data sources, statistical analyses) are presented in the Methods Section of Appendix D. Briefly, we followed an operational process to develop simple decision models that compare different treatment options (PCI with BMS, PCI with DES, CABG). We used Markov models with the following health states: (1) asymptomatic (no prior stroke), (2) recent myocardial infarction (no prior stroke), (3) asymptomatic (post stroke), (4) recent myocardial infarction (post stroke), (5) repeat stroke in patients who had had a prior stroke; (6) dead (Appendix Figure D1). Estimates for model parameters were derived from published sources, including the Stanford CER,12 a subsequent meta-analysis of individual patient data from RCTs;17 a network meta-analysis of DES and BMS,14 three large recently published randomized trials (CARDia,16 COURAGE,13 and SYNTAX15) and three cost-effectiveness analyses.18–20 The previous studies were identified from key informant input or from focused literature searches. We used network meta-analysis to obtain consistent estimates for all treatment effects.21 We analyzed the models using a time horizon of 10 years, as this is our time horizon for making recommendations for future research.

Specification of Three Index Models

It was not feasible to perform quantitative analyses for all identified subpopulations in the pruned final list of research gaps, because we would have to develop separate models for each one.7 Instead, we decided to develop models for three index subpopulations and extrapolate any insights to the rest. We chose the index subpopulations based on ease of modeling and availability of good data to parameterize the decision models:

  • RCT-type participants:” a reference scenario simulating a cohort of 65 year old patients with nonacute CAD and no major comorbidities. The decision is between revascularization with DES, BMS or CABG. Most data to parameterize the models were obtained from RCTs. This model is based on more robust data than the other two models (Appendix D).
  • “Elderly participants (older than 75 years):” a cohort of elderly (75 year old) CAD patients with nonacute CAD. The decision is between revascularization with DES, BMS and CABG. Because there is not a lot of information on elderly patients specifically, this model makes extrapolations based on specific assumptions (Appendix D).
  • “Diabetics:” a cohort of 65-year old diabetic patients. The decision is between revascularization with DES and CABG. As was the case with the model on the elderly, we make several assumptions to parameterize this model (Appendix D).

Modeling Analyses

We performed three types of quantitative analyses:

  1. We analyzed our deterministic decision models using quality-adjusted life years (QALY) as the decision relevant quantity. Our aim was to identify “influential” parameters, i.e., parameters that exert the maximum influence on the decision relevant quantity in one-way sensitivity analyses.8 Theoretically, those parameters should be considered as research priorities, since reducing the uncertainty around them would have the biggest effect on the decision uncertainty.9
  2. We then included costs and performed cost-effectiveness analyses. The decision relevant quantities were incremental cost-effectiveness ratios between treatment pairs. Again we identified influential parameters using one-way sensitivity analyses. As in the decision analysis approach, influential parameters are more likely to represent priority research needs compared to less influential ones.
  3. Finally, we recast the models as probabilistic models and performed value of information analyses. First we calculated the expected value of perfect information (EVPI). EVPI is expressed in monetary units and represents the opportunity cost incurred by having to make decisions based on imperfect information. EVPI represents an upper bound to the expected returns of future research. EVPI can be considered as the value of reducing uncertainty for the overall decision tree, i.e. it can be considered as indicative of the value of future research on a broad field, but it cannot prioritize specific research topics or guide study designs. Then we calculated the expected value of perfect information for groups of parameters (EVPPI), which places an upper bound to the value of research on specific (groups of) parameters. EVPPI estimates for the specific (groups of) parameters can be used to select the specific topics that future research should address. In EVPPI analyses we organized groups of parameters of interest so that each group could be addressed by a single future study (Appendix D).

Interpretation of Modeling Results

We perform modeling analyses to only to identify “influential parameters” and to rank them according to their relative “influence.” This is an atypical use of modeling, in that we stop short of providing insights on treatment choices for different assumptions and circumstances. However this is a conscious choice:10 Ranking of influential parameters is likely to remain stable even if more elaborate models are used, or if better data are used to populate parameters for which we made simplifying assumptions. In contrast, exact values may change substantially.

Candidate Study Designs

Candidate study designs will differ across types of research needs. Effectiveness or efficacy of treatments can be most definitively addressed in RCTs, and secondarily in well conducted nonrandomized comparative observational studies. In contrast, eliciting patient preferences can be meaningfully performed with nonexperimental designs (e.g., in a survey). We list the candidate study designs for different types of questions of future research (Table 2).

Table 2. Candidate study designs for addressing different types of research needs.

Table 2

Candidate study designs for addressing different types of research needs.

Broadly speaking and without considering other factors such as feasibility, an RCT is the most suitable study design to obtain unbiased estimates of effectiveness or efficacy of specific interventions in specific populations (other options are listed in Table 2). While one could suggest using RCTs to compare various test-and-treat strategies for patient management, this is often not possible.22 The first stage should be to evaluate the performance of tests in predicting differential response to the treatments of interest, and decide whether an RCT is necessary in a second step (see Results Section).22 This can be achieved by modeling the treatment-by-test-results interaction in patients who received the treatments of interest and have been followed up for sufficient time to observe their response. An attractive design is to “nest” the study of predictive performance of testing in an RCT that compares the interventions of interest, by applying the test at baseline. An alternative is to reanalyze data of existing RCTs, provided that test results are available for all participants at baseline (or are missing at random in a minority of RCT participants). Other options are listed in the Table. Finally, RCTs are not appropriate for eliciting patient preferences, or for developing decision support tools or performance measures. Surveys of patients or qualitative research studies are possible study designs for these latter cases, as discussed in the Results Section.

Feasibility of Study Designs

Studies that do not require new data collection are in principle feasible, provided that access to existing data can be agreed upon or has already been granted. An analysis of an existing registry can be completed within a year. A meta-analysis of individual patient data can be conducted in a time horizon of two years.13 The feasibility of such studies, generally, does not depend on the desired sample size.

We considered that a study of primary data collection would be infeasible if it were too expensive or complex to conduct; if it required too long a followup, say beyond 5–7 years; or if it relied on information or data that is not yet available. We acknowledge that deliberations on feasibility are inherently subjective.

Generally, RCTs are among the most expensive research designs. Recently completed or ongoing large efficacy RCTs may be examples of “expensive” research. Some of the largest recent efficacy RCTs in nonacute CAD have sample sizes in the neighborhood of 2500 patients.14 Using this as a reference we commented on the feasibility of other research designs based on the following assumptions:

  • Prospective nonrandomized comparative trials are less expensive to conduct than efficacy RCTs of similar size.
  • Cohorts or case control studies of the predictive performance of tests for response to treatments are substantially less expensive to perform than efficacy RCTs of similar size.
  • Surveys of, e.g., patients to elicit preferences should be generally economically feasible, as they are expected to be much cheaper than a large RCT of a few thousand people.

Sample Size Calculations for RCTs

We performed sample size calculations using standard formulae for a two-sided chi-squared test at the 0.05 level of significance. We assumed a true relative effect of 0.80 favoring the intervention arm, an allocation ratio of 1:1, no loss to followup, no crossover between treatments, and no sequential monitoring. We made power calculations for 3 and 5 years of followup assuming a range of constant annual event rates in the comparator intervention corresponding to 5-year cumulative proportion of primary events at 5, 10, 15, 20, 30 or 40 percent. To estimate the duration of a trial so that the mean followup is 5 years we assumed a minimum follow up of 2.5 years, an accrual period of 5 years and a constant accrual rate throughout the accrual period. Because of our simplifying assumptions, we probably overestimate the power attained at various total sample sizes.

Handling Conflicts of Interest

In order to minimize conflicts of interest we suggested proposed research designs internally, using predetermined criteria and incorporating insights from modeling analyses. Key informants, all of whom were screened for potential conflicts of interest were consulted to ensure that important evidence gaps were considered and to identify criteria for prioritization, but they were not directly involved in the final prioritization.



These were selected among those identified in the Stanford CER using the following criteria: The analysis included patients revascularized in the 1990’s in North America or Europe, described at least 1000 patients treated with PCI and 1000 treated with CABG, reported patient characteristics and performed multivariate statistical analyses.


Here we are interested in testing to guide treatment choice (via predicting differential response to treatment). We do not refer to other settings such as screening of asymptomatic individuals, diagnosis, or treatment monitoring. In particular screening of asymptomatic individuals was identified as a very interesting research area, but it was deemed to be outside the scope of our exercise.


Invasive testing with coronary arteriography, non-invasive computerized tomography angiography (CTA) or magnetic resonance angiography (MRA), resting or exercise single photon emission computerized tomography (SPECT), and noninvasive exercise treadmill testing with or without echocardiography.


If there is prior information from large scale analyses of randomized data suggesting, e.g. a treatment effect modification.


Actually, this is one of the important evidence gaps in the first thematic area (comparative effectiveness of PCI vs. CABG).


To better define prevalence of procedural or long term events one may opt to analyze registries or perform observational studies. To get more information on treatment effect modification one would have to reanalyze existing comparative data or perform new RCTs. Other research designs would be more suitable to other parameters (see subsequent paragraph on “Candidate research designs”).


Initially we envisioned that in our discussions with the Key Informants we would identify only one or two important subpopulations in which we would perform modeling analyses. However, this was not the case.


In one-way sensitivity analyses we change the value of each parameter over a prespecified uncertainty range (its corresponding 95% confidence interval) while keeping all other parameters at their baseline values, and record the effect on the decision relevant quantity.


This is a simplification. One-way sensitivity analyses underestimate the uncertainty inherent in the model and cannot handle correlated parameters.


This is admittedly a defensive stance. It is very influenced by the fact that our models are not developed with the same rigor as some of the well-known and elaborate models that are used to analyze clinical decisions; they have not been calibrated using external data; and their predictions have not been validated in external data. They are operational models that are constructed to make use of the summary information obtained from an evidence report, that can still offer broad insights.


A variety of research designs may be pertinent, ranging from qualitative research in focus groups, to randomized trials of using versus not using the decision support aid. We do not expand here, but discuss specifics in the Results and Discussion sections.


A variety of research designs may be pertinent to developing performance measures. We do not expand here, but discuss specifics in the Results and Discussion sections.


A meta-analysis of individual patient data can take longer to complete than an analysis of an existing and available database. There are logistical complications including but not limited to identification of data sources, convincing investigators to participate, standardizing definitions of interventions and outcomes, complying with HIPAA, and harmonizing datasets.


For example COURAGE13 (NCT00007657) compared optimal medical therapy and PCI in 2,287 patients, FREEDOM22 (NCT00086540) compares PCI and CABG in approximately 2,400 patients, and STICH23,24 (NCT00023595) compared CABG and medical treatment in 2,136 patients.

Image appendixes.app4f1


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (1.0M)
  • Disable Glossary Links

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...