Value of information analysis for health research seeks to estimate the expected value of research projects at the population level. To do so, it begins with a person-level estimate of the value of research to reduce uncertainty surrounding the net benefit of alternatives treatments or interventions under consideration. Specifically, let θ describe a parameter vector that determines the net benefit of treatment option j to be studied, which we denote as NB(θ, j). Net benefit is most commonly defined as net monetary benefit, which can be calculated by taking the benefit in monetary terms of some improvement in health and subtracting any costs. The monetary value of improvements is most commonly calculated by multiplying the gain in health by some measure of the monetary value per unit of health gained (e.g., QALYs gained multiplied by $ per QALY).^* Furthermore, let max _jE_θNB(θ, j) describe the expected value of the decision from among j interventions that maximizes expected net benefit given current information. The value of research is defined by identifying the information set {I} consisting of a set of outcomes and associated probabilities that could result from a particular research activity. Equation 1 describes the expected value of information ( EVI) from research on a per-person basis.

Several additional factors need to be considered to translate the value of research at the person level to a population statistic potentially relevant for informing policy. First, because research has value for populations of people and over time ( t), it is important to account for the incidence ( Incidence_t) of the relevant condition (i.e., annual rate of new cases per member of the population) and the size of the relevant at-risk population ( Population_t). Another factor that should be considered is the likelihood that relevant information may be imperfectly implemented, and thus produce value for only a fraction of the population in whom it could have been applied ( Im plementation_t). In addition, VOI may account for the possibility that that future cohorts would not benefit from the research because the value of the information is not durable over time because improved treatments are introduced, and/or new clinical evidence emerges that may greatly increase or decrease the expected clinical benefit of a treatment independent of the research study being considered ( Durability_t). Finally, benefits accruing to more distant future cohorts may be valued less than benefits for less distant cohorts, causing benefits to future cohorts to be discounted at a rate β _twhere β _t< 1. Thus, the population-level expected value of information ( pEVI) is:

Equation 2 lays out the basic framework for the VOI framework when it is fully applied. In practice, application of this framework is almost never complete. In some cases, it is because cutting-edge theoretical issues such as value of information were simply not considered by the individuals performing the VOI analysis. In other cases, a factor is considered, but without much rigorous analysis. For example, durability is sometimes modeled by considering benefits that only accrue over a time horizon of 5 to 10 years, generally with little or no justification or consideration of the fact that the results of research may take some time to be implemented, or that irreversible decisions made today (such as surgery) may have highly durable effects. In other cases, critical issues such as the size of the affected population are modeled without much thought for the practical effects of research across populations.

For example, VOI analyses performed by the National Institute for Health and Clinical Excellence (NICE) typically are based on the size of the United Kingdom population (~60 million), which is 20 percent of the size of the U.S. population (300 million), 12 percent of the size of the European Union, and less than 1 percent of the world population (7 billion). Since research done in one country is generally also of value outside that country, it is clear that estimates of the value of research that take a single-country perspective can severely underestimate the value of research on a global scale, and that comparisons of the value of research from different countries based on the size of the local population can be severely misleading as to the net value of research.⁴ Of course, since the cost of research is usually borne by one country, there is some justification for the traditional practice of focusing on just that country’s population if that country discounts benefits to other countries.

Most commonly, however, VOI calculations are not fully implemented because of the lack of critical pieces of information. For example, it may be very difficult to fully characterize the possible outcomes of a research study, and their likelihood. Similarly, it may be difficult to meaningfully characterize the uncertainty in the net benefit of the alternatives under consideration, particularly when there is little or no prior clinical data on relevant outcomes. In these circumstances, decision models are often used in these to obtain estimates of comprehensive measures of (net) benefit, based on data on aspects of the decision in question. Decision models have the advantage of permitting calculation of the expected value of partial perfect information ( EVPPI), which describes the value of information on specific parameters in a complex decision model, and identifies the most important parameters to target for study. The use of decision models to perform VOI also has drawbacks. One big drawback is the issue of transparency in modeling and in assumptions (a common challenge for decision analysis in general). Perhaps the most important drawback is that VOI studies based on decision analyses can be very time consuming and complex so that the approach is too burdensome for practical application in some circumstances.

The practical challenges in applying VOI have led to both led to both theoretical and practical efforts to simplify the application of value of information approaches. Some factors, such as implementation and durability, are simply ignored on a routine basis. This causes VOI calculations in general to overestimate the true benefits of research. When it is difficult to characterize the extent to which a particular research study is likely to reduce uncertainty, the expected value of perfect information ( EVPI) is often used to provide an upper bound on the value of research by calculating the expected value of research that would eliminate all uncertainty in the net benefit of treatment. This is possible because EVPI depends only on the distribution of the net benefit of the treatment options being considered (Equation 3).

When even this uncertainty cannot be fully characterized—for example, because little or nothing is known about the effectiveness of the treatments being considered—EVPI can also be bounded by measures of the total burden of disease that could potentially be eliminated.5 The advantages of these bounding approaches are that they are easier to apply so that they can be used to triage potential topics if an upper bound suggests that the potential value of research is not large. They can also be applied when a great deal of potentially relevant information on uncertainty or the potential information that could come from research is lacking. However, a major limitation is that it is not generally possible to know how close such upper bounds would be to more complete analyses. Thus upper bounds will again be informative mostly when they suggest that the potential value of research is not large.

When good data are not available to characterize the uncertainty associated with a treatment, decision models are often used to characterize uncertainty in net benefit. These models typically describe a series of health states, a mathematical model to describe transitions among health states (e.g., a decision tree to describe the likelihood of events within periods and a Markov model to describe transitions over time), and a set of payoffs (e.g., utility and costs) associated with each health state. Because the construction of these models can be very complex and time consuming, approaches that are easier to apply would be helpful. This is especially true when a decision tree is to be used for VOI analysis, since that also necessitates characterizing the uncertainty associated with each parameter. Cost-effectiveness analyses done alongside clinical trials⁶ in which comprehensive outcomes measures (at least in terms of direct measures of QALYs and costs) are collected directly, avoid the need for the construction of models and also allow uncertainty in these outcomes to be directly quantified.

Minimal Modeling Approaches

We define minimal modeling approaches to VOI as those that model VOI without constructing a decision model of the disease and treatment process to characterize the uncertainty in net benefit associated with an intervention.

As previously discussed, minimal modeling approaches to VOI are feasible in certain circumstances. One situation in which minimal modeling VOI is feasible is when a prior clinical trial provides data on a comprehensive measure of the net benefit of the interventions examined. In general, this would require a trial that directly measures all health benefits in QALYs and all costs. To be valid representations of the net benefit of the treatments that are compared while avoiding modeling, such trials would need to measure these comprehensive outcomes until the point that there are no differences between the treatments examined. Examples would include studies that followed all patients to death or that followed all patients until they recovered. These approaches require no modeling to calculate the individual level of value of information, and we, therefore, term them “no modeling” approaches. VOI calculations based on no modeling can be done mathematically or via bootstrapping/simulation. Bootstrapping/simulation, replicating or resampling decision values, can be done using raw patient-level data on relevant parameters (i.e., a nonparametric approach) or by making parametric distributional assumptions.

Another situation in which it may not be necessary to build a full decision model of the disease and treatment process is when the treatment does not affect survival but only quality of life and quality of life is directly measured by a clinical trial. In such cases, which we term “limited modeling,” it is necessary to build a survival model, but the model does not require developing a full model of health states that predicts survival (e.g., progression of cancer between stages, psychosis to completed suicide). The recent analysis of the value of research on atypical antipsychotics is an example of such a study.⁷

Table 1 briefly outlines and summarizes these three the modeling approaches to VOI calculations (full, limited, and no modeling), their potential scope applicability, and the advantages and disadvantages of these alternative approaches.

Table 1

Modeling approaches to VOI calculations.

Footnotes

*

Alternatively, net health benefit can be calculated by taking health gains in QALYs and subtracting the health that could be obtained by applying the costs of the intervention in a health intervention that was at the threshold for cost-effectiveness (e.g., $50,000 per QALY).³