NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Hoomans T, Seidenfeld J, Basu A, et al. Systematizing the Use of Value of Information Analysis in Prioritizing Systematic Reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012 Aug.

Cover of Systematizing the Use of Value of Information Analysis in Prioritizing Systematic Reviews

Systematizing the Use of Value of Information Analysis in Prioritizing Systematic Reviews [Internet].

Show details

Theory

Introduction

In this section, we describe an algorithm intended to guide the process of identifying the effective and efficient use of VOI in prioritizing systematic reviews. To develop this algorithm, we first identify the conditions under which VOI is likely to be valuable as an overall strategy to prioritize systematic reviews. We then seek to outline a logical sequence of stages in which full modeling, minimal modeling, maximal modeling, and conceptual VOI are considered in order to minimize the costs and burden of analyzing the value of systematic reviews. Following this, we detail a multistage algorithm for deciding about when to invest in VOI and what specific approach to VOI to use in different contexts. Finally, we discuss how prioritization decisions can be made on the basis of estimates of the value of information from performing systematic reviews. The algorithm we develop is intended to systematize the use of VOI analysis in prioritizing systematic reviews.

Conditions of Value of VOI Analysis in Systematic Reviews

Following the general principles of VOI (Part 1, Principles, above), we can identify several conditions under which it may be worthwhile to invest in performing a VOI analysis to inform priorities for systematic reviews. A first condition is that existing research indicates that none of the conceptual VOI elements for a particular topic have a zero or very low value, as described under Conceptual VOI (in Part 2) of this report. This implies that some difference is anticipated in the (net) benefits of the treatment or health interventions being compared, and that the synthesis of evidence from multiple studies might change the degree of uncertainty about the benefits of those options compared with relying on a single research study. In addition, the findings from a systematic review must be considered likely to be durable for at least some period of time, the information obtained must be considered likely to be implemented into practice, and the population of affected patients must not be very small. As a second condition, prioritization exercises with VOI are only valuable when the costs of VOI are less than the value expected from performing a systematic review net of its costs. An implicit assumption is that either the resources for systematic reviews are limited relative to the set of reviews that could be performed or that the resources for systematic reviews have alternative uses so that some prioritization among systematic review topics is necessary. A third condition that is necessary for investment in VOI to be worthwhile is that the research for which a decision to proceed might be affected by the systematic review is expected to be costly; if studying the question was very inexpensive, one would just do the study without either VOI or systematic review. More commonly, if a clinical study to address a particular question is expected to be very costly, then costly full-modeling or maximal modeling VOI studies may be rational. These VOI applications would not only inform whether an additional primary research study should be performed but also to identify an efficient study design.

These conditions make it possible to define a practical and efficient process for the use of VOI to inform priorities for systematic reviews by outlining a logical sequence of stages to select the VOI approach that will minimize cost and burden of analyzing VOI for systematic reviews. This process begins by applying the lowest cost VOI method and proceeds to higher cost methods for a given clinical question. Thus, the sequence we propose is: (1) conceptual VOI; (2) consideration of maximal modeling VOI in which costs are covered among multiple topics for systematic review simultaneously; (3) consideration of a minimal modeling approach to VOI in individual review topics; and (4) consideration of full modeling VOI. When minimal modeling or conceptual VOI is selected, there may be a tradeoff between reducing the complexity and cost of modeling compared with the full characterization of the uncertainty in the benefits of particular health interventions and more accurate estimates of the value of information from applying full modeling or maximal modeling approaches to VOI. How this tradeoff is best navigated is likely to depend on the conditions, including timing and budget restrictions, under which decisions about research spending are to be made, and the extent to which it is important to address additional questions such as the value of information on specific parameters that may affect specific clinical or health policy decisions.

Before describing our approach to VOI of systematic reviews in more detail, we should note that we do not devote much attention to the possibility that a VOI analysis has already been developed for a specific decision problem. This is mainly because there are still relatively few problems for which VOI analysis has been performed and those for which it has been performed would need to be adapted to the specific question for which a systematic review is being considered. Nevertheless, as a first step, it would surely make sense to review the literature in a topic area to assess whether a VOI has been performed and consider adapting that VOI if less costly VOI approaches do not adequately address the VOI of the potential systematic review being considered.

Algorithm for Selecting the VOI Approach To Prioritize Systematic Reviews

As shown in Figure 3, we propose an algorithm that describes a multistage process for identifying the effective and efficient approach to VOI in prioritizing the performance of a systematic review to address a particular clinical question or research topic. This algorithm starts with the use of a conceptual approach to VOI, in which the expected change in uncertainty of the benefits of the treatment or interventions under consideration, the implementation and durability of review findings, and the size of the affected patient population are considered to assess whether there is likely to be any meaningful value of synthesizing existing evidence on individual topics nominated for systematic review (i.e., none of the elements of conceptual VOI approach zero). If this approach does not suggest a low conceptual VOI for a particular topic, the algorithm suggests clustering topics being considered for review for the potential use of “maximal” models that can simultaneously assess the potential value of multiple systematic reviews within or across clinical domains. When topics cannot be clustered or when a maximal modeling approach to VOI is deemed impractical or undesirable, the next step is to consider minimal modeling VOI. A minimal modeling approach to VOI is only possible when some data are readily available on comprehensive outcome measures for the clinical or policy decision that is to be informed. Finally, if none of the other VOI methods are feasible, the algorithm proposes considering the use of full modeling VOI. Since a full modeling approach to VOI would likely be more costly than a systematic review, this approach will be worthwhile only when a potentially valuable but costly clinical trial or other (observational) study for collecting additional primary data on a particular topic is so likely to be planned that performing a VOI analysis for research design and sample size calculations is likely anyway. As we discuss further under Other Issues, below, in that case, it may be useful to simply perform a full modeling VOI study immediately.

Figure 3 is a flowchart describing the algorithm for identifying the effective and efficient approach to calculating VOI in informing priorities for systematic reviews. The flow chart begins with the consideration of any potential clinical question or reseach topic for systematic review, and is a guide to identifying the most appropriate approach to VOI for a given topic based on a series of susequent questions. This figure is described further in the section entitled “Theory, Introduction, Algorithm for Selecting the VOI Approach to Prioritize Systematic Reviews” as follows: “As shown in Figure 3, we propose an algorithm that describes a multi-stage process for identifying the effective and efficient approach to VOI in prioritizing the performance of a systematic review to address a particular clinical question or research topic. This algorithm starts with the use of a conceptual approach to VOI, in which the expected change in uncertainty of the benefits of the treatment or interventions under consideration, the implementation and durability of review findings, and the size of the affected patient population are considered to assess whether there is likely to be any meaningful value of synthesizing existing evidence on individual topics nominated for systematic review (i.e., none of the elements of conceptual VOI approach zero). If this approach does not suggest a low conceptual VOI for a particular topic, the algorithm suggests clustering topics being considered for review for the potential use of ‘maximal’ models that can simultaneously assess the potential value of multiple systematic reviews within or across clinical domains. When topics cannot be clustered or when a maximal modeling approach to VOI is deemed impractical or undesirable, the next step is to consider minimal modeling VOI. A minimal modeling approach to VOI is only possible when some data are readily available on comprehensive outcome measures for the clinical or policy decision that is to be informed. Finally, if none of the other VOI methods are feasible, the algorithm proposes considering the use of full modeling VOI. Since a full modeling approach to VOI would likely be more costly than a systematic review, this approach will be worthwhile only when a potentially valuable but costly clinical trial or other (observational) study for collecting additional primary data on a particular topic is so likely to be planned that performing a VOI analysis for research design and sample size calculations is likely anyway. As we discuss further under Other Issues, below, in that case, it may be useful to simply perform a full modeling VOI study immediately.”

Figure 3

Algorithm for identifying the effective and efficient approach to calculating VOI in informing priorities for systematic reviews. VOI: value of information

Stage 1. Use of Conceptual VOI to Bound Value in Systematic Reviews and Formal Quantification of VOI

The first stage of the algorithm we propose is to assess the conceptual VOI of performing a systematic review in an individual review topic. With this approach, information on each of the conceptual elements (e.g., the expected change in uncertainty about treatment benefits from evidence synthesis, and the durability of such review findings) is used to determine the population-level VOI from the review of evidence from existing research studies in order to provide informative bounds on the value of systematic reviews in individual topics without formally quantifying such VOI estimates through more complex modeling exercises. When information is available that suggests that any of these elements approximates zero, the product of these terms (and hence the VOI) will almost always be zero unless some other element is exceptionally large. For topics in which the values for the conceptual VOI are low, it is not likely that prioritizing and reviewing evidence in a systematic review would be an effective means of research spending.

To assess whether the conceptual VOI is likely to be low, research on VOI elements at this stage of the algorithm is meant to quickly identify any conceptual elements that have values that approximate zero. A pragmatic method for this would be to assess the values for each of the elements of VOI through a quick scan, followed by a more comprehensive search for data on values in the elements initially identified as likely to be low in the quick scan. Such information may be available from the literature identified through MEDLINE®/PubMed, national statistics bureaus, the National Health Institutes, registries and post marketing surveillance studies, or the use of expert opinion (see also Table 2). The order in which the value of each of the conceptual elements of VOI is assessed could be determined by the ease with which information on these elements can be collected. For example, population size is relatively easy to determine, while information on implementation or durability of review findings may be more difficult to find. Given that VOI will be low if any of these elements approach zero, it may be most efficient to use judgment about whether any of these elements are likely close to zero and then focus initially on those conceptual elements of VOI until any one of them is found to approach zero.

Stage 2. Consideration of Maximal Modeling VOI When Topics Cluster Within or Across Clinical Domains

For systematic review topics in which modeling cannot be excluded because of low conceptual VOI, the second stage of the algorithm considers whether a maximal modeling approach offers a possibility to simultaneously analyze the value of performing systematic reviews on separate but related topics clustering in a particular clinical domain. The clustering of topics may be based on the specific relationship of particular diseases and their treatments, perhaps along the lines of pathophysiology or clinical pathways, for which one often relies on the opinions from experts. For example, the screening, diagnosis and treatment of patients with prostate cancer can be perceived as topics clustering in the domain of prostate cancer. The single comprehensive models used for maximal modeling VOI are often organized around disease and treatment processes or health care programs.

By simultaneously calculating the value of multiple systematic reviews, the costs of performing VOI are minimized across the individual review topics. While maximal models may have to be newly constructed, it may be more efficient to use existing models for this purpose. Examples of such models would include the Coronary Heart Disease Policy Model or perhaps one of the decision-analytic models like CORE Diabetes Model or CDC-RTI Cost-Effectiveness model he already available in diabetes care.15,112114 In identifying models that are already available in the clinical domain of interest, and that are applicable for maximal modeling approaches to VOI, expert opinion, environmental scans, or literature searches may be useful. Clearly, an existing model may need to be adapted before applying it to perform VOI in a specific decision context, for example, by adjusting or updating data input. To evaluate maximal models, simulation or bootstrapping can be performed using software like MS Excel, Stata or WinBUGS. A maximal modeling approach to VOI may be particularly desirable when multiple potential uses of the model could be envisioned. In effect, this may be especially relevant in the context of prioritizing systematic reviews because such approaches to evidence synthesis are often set up to identify research gaps and direct the planning of new research.

Even when individual topics for systematic reviews can be clustered, maximal modeling approaches to VOI to prioritizing systematic reviews and other research studies may still be considered impractical or undesirable. This may be because of the perceived burden to constructing new ‘maximal’ models or adapting existing ones, or because of the limited appreciation of establishing a more sustainable infrastructure for future VOI analyses. In those situations where a choice is made to not use maximal modeling VOI, the algorithm suggests to assess the value in systematic review topics individually rather than simultaneously, starting with the consideration of applying minimal modeling VOI to identify research priorities. For example, for a clinical question that could be clustered with other topics to apply a maximal modeling approach, but for which data to apply minimal modeling VOI might be readily available, a minimal modeling approach might be preferred on the basis of speed with which it could be applied if a decision about starting a trial was being actively considered.

Stage 3. Use of Minimal Modeling VOI When Data on Comprehensive Outcomes Measures Are Available

The third stage of prioritizing systematic reviews among individual, potentially valuable review topics is to consider performing VOI calculations with only minimal modeling based on data on comprehensive outcomes measures (e.g., life expectancy, QALYs, and costs or net benefits) that can be used to readily address the clinical or health policy decision in question. The data needed for such a minimal modeling approach to VOI may often be available from existing clinical trials, observational studies, or meta-analyses. It may be thereby useful to break down the term minimal modeling into no modeling (i.e., when comprehensive outcomes are directly measured), and limited modeling (i.e., when some modeling is needed to calculate the comprehensive outcomes measure, for example by combining quality of life with life expectancy). VOI calculations based on minimal modeling can be done via bootstrapping/simulation using raw data or distributions for health outcomes on costs, and survival data, or even through equation-based computations. Minimal modeling VOI can thereby be implemented in software like R, Stata or WinBUGS, and templates for these types of analyses are readily available.7

As noted above, if a VOI study already exists in a topic area, measures for VOI can also be derived from these studies. This requires careful attention as to whether the evidence on specific diseases and/or treatments in these studies is readily applicable to the specific context in which a particular clinical question is to be addressed. If no such studies can be found and no comprehensive measures of relevant outcomes are readily available, the next step would be to consider the use of a full modeling approach to VOI to prioritize systematic reviews.

Stages 4 and 5. Use of Full Modeling VOI When Additional Collection of Primary Data Is Expected to Be Both Valuable and Costly

For systematic review topics in which the VOI algorithm suggests the potential for full modeling of a particular disease, its treatment and the different health states, it is important to consider whether further research is likely to be performed. The basic challenge in using full modeling VOI to prioritize a systematic review is that the construction of full models is too burdensome and a costly way to perform VOI to inform decisions about prioritizing low-cost studies such as systematic reviews. It would not make sense to perform a costly VOI study just to decide not to do a relative inexpensive systematic review since it makes more sense just to perform the systematic review.

Our algorithm suggests the use of a full modeling approach to VOI only when it seems so likely that further research is planned for collecting additional primary data, and this research is likely to be costly enough that it will make sense to perform a full modeling VOI study. VOI is then done both to prioritize the systematic review topic and help the efficient design of relevant studies, for example by suggesting appropriate sample size in trials, the most relevant outcomes to measure, or the appropriate length of followup of patients and patient cohorts. If that VOI analysis suggests that further research is not likely to reveal any evidence or is too costly, prioritization exercises for systematic reviews are not likely to be valuable. In practice, the work involved in performing the full modeling VOI may overlap so greatly with the work needed to perform the systematic review, that they will effectively both be completed. As such, it is hard to argue that the full modeling VOI is being used to prioritize the systematic review. Nevertheless, it would make sense to perform the full modeling VOI at this stage and since this may suggest that a systematic review could be of low value (perhaps compared with a review focusing on some part of the decision problem, or compared with studying the problem at all), it is still the case that the full modeling VOI might result in the decision not to complete a full systematic review.

Other Issues in the Use of VOI Estimates to Prioritize Systematic Reviews

In prioritizing topics nominated for systematic review, a differentiation can be made between (1) those topics for which estimates of the population-level VOI are provided through maximal, minimal or full modeling; (2) those topics for which conceptual VOI indicates that there is no or limited value in the review of evidence; and (3) those topics for which application of VOI does not appear practical so that approaches other than VOI need to be used to inform priorities for systematic reviews. Under the assumption that resources or budgets are fungible between research and other uses, a strict economic analysis might suggest performing all systematic reviews for which VOI is calculated to exceed the costs associated the review and synthesis of existing evidence. However, if the costs of systematic reviews vary and the resources or budgets for research are limited, priority has to be given to performing the systematic review or set of reviews that maximizes the returns (i.e., population-level VOI net of the expected costs of systematic reviews) of research spending.

In practice, with typical levels of resources available to perform systematic reviews, the costs of performing systematic reviews may be so small relative to their benefits, or the capacity to perform systematic reviews may be so limited, that their costs can usually be neglected in prioritizing reviews. Priority setting may then rely solely on the assessment of population-level VOI of systematic reviews. When comparing the value of systematic reviews across individual topics or clusters of topics, however, it is important that estimates for the population-level VOI reflect the size of the affected patient populations, the probability of implementation of specific treatments or interventions, and the durability of evidence that would come from the review. Standardization should also account for potential differences in the perspective and time horizon of analysis, the use of health outcomes, costs, utilities as well as threshold values for cost-effectiveness or willingness-to-pay for an effectiveness outcome. Although the construction of full models may perhaps provide most accurate indication of the value of research to address a particular clinical question, lower cost VOI methods (i.e., conceptual VOI, maximal modeling, and minimal modeling) probably more often have practical and valuable application in informing priorities in systematic reviews.

Notably, the initial stage in prioritizing systematic reviews, prior to any use of the algorithm, is to generate a list of nominated review topics. This may be done on an ad-hoc basis or by more systematic approaches like environmental scans, literature searches or Delphi techniques. Since failure to consider a sufficiently large set of potential review topics may result in assigning high priority to a topic that would have received lower priority had additional topics been considered, it is critical that the list of topics nominated for systematic review include as many potential topics at possible.

Conclusion

In minimizing the costs of VOI as part of an overall strategy to use VOI to inform priorities for systematic reviews, we propose an algorithm that describes a multistage process to identify the effective and efficient approaches to performing value of information analysis. This process begins with conceptual VOI to identify when VOI of a systematic review is likely to be very low, followed by the clustering of review topics and consideration of the use of maximal models, and then consideration of minimal modeling using comprehensive outcome measures of the benefit of the alternative treatments or health interventions under study if data permits. Although full models may aid in the planning and design of research studies, we find rather limited conditions for its use in prioritizing systematic reviews. The valuable application of a full modeling approach to VOI is limited primarily to instances where a such an approach appears likely to be applied in any case because it seems very likely that a review would result in suggesting that a costly trial will be needed for which a full modeling VOI would be a logical investment. The algorithm we propose attempts to provide a systematic strategy with which to consider the use of VOI to prioritizing systematic reviews.

Views

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...