• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of bmcmrmBioMed Centralsearchsubmit a manuscriptregisterthis articleBMC Medical Research Methodology
BMC Med Res Methodol. 2001; 1: 8.
Published online Sep 6, 2001. doi:  10.1186/1471-2288-1-8
PMCID: PMC58837

On the probability of cost-effectiveness using data from randomized clinical trials

Abstract

Background

Acceptability curves have been proposed for quantifying the probability that a treatment under investigation in a clinical trial is cost-effective. Various definitions and estimation methods have been proposed. Loosely speaking, all the definitions, Bayesian or otherwise, relate to the probability that the treatment under consideration is cost-effective as a function of the value placed on a unit of effectiveness. These definitions are, in fact, expressions of the certainty with which the current evidence would lead us to believe that the treatment under consideration is cost-effective, and are dependent on the amount of evidence (i.e. sample size).

Methods

An alternative for quantifying the probability that the treatment under consideration is cost-effective, which is independent of sample size, is proposed.

Results

Non-parametric methods are given for point and interval estimation. In addition, these methods provide a non-parametric estimator and confidence interval for the incremental cost-effectiveness ratio. An example is provided.

Conclusions

The proposed parameter for quantifying the probability that a new therapy is cost-effective is superior to the acceptability curve because it is not sample size dependent and because it can be interpreted as the proportion of patients who would benefit if given the new therapy. Non-parametric methods are used to estimate the parameter and its variance, providing the appropriate confidence intervals and test of hypothesis.

Introduction

In reporting cost-effectiveness analyses alongside clinical trials, authors [1-4] have used various definitions, estimation methods and interpretations for acceptability curves. Acceptability curves provide an excellent means of quantifying the stochastic uncertainty of the estimated incremental cost-effectiveness ratio (ICER) in relation to a particular value ascribed to a unit of effectiveness. It is the certainty, expressed as a probability, that the current evidence would lead us to believe that some new therapy is cost-effective, insofar as the ICER is less than the value ascribed to a unit of effectiveness. In addition, acceptability curves provide an estimator for the ICER and its confidence limits. However, acceptability curves are often interpreted and expressed as the probability that the new therapy is cost-effective. It is argued below that this is subject to misinterpretation, and an alternative definition for a parameter representing the probability that the new therapy is cost-effective is introduced. Data from a clinical trial can be used to make statistical inference about this parameter. Furthermore, the inference provides a non-parametric estimator for the ICER and its confidence interval.

In a two-arm randomized control trial let eji and cji be the respective measures of effectiveness and cost for patient i on therapy j, where j = T (treatment), S (standard); i = 1, 2, . . .nj ; and nj is the number of patients randomized to therapy j. Typically, eji is the patient's survival time (perhaps quality-adjusted) from randomization to death or to the end of the period of interest. Let An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i5.gif. Define An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifc similarly. Let E(An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gife) = Δe and E(An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifc) = Δc, where E is the expectation function. Thus, the incremental cost-effectiveness ratio (ICER) is Δce, and is estimated by An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifc / An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gife. In addition, the incremental net benefit [5-9] (INB) is Δeλ-Δc, and is estimated by An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifeλ - An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifc, where λ is the value given to a unit of effectiveness. Typically the INB is expressed as a function of λ, allowing readers to provide the value they consider most relevant.

van Hout et al. [1] define the acceptability curve as "the probability that the [ICER] is under a certain acceptable limit", say λ. The acceptability curve, then, is a function of λ. If one assumes that the authors are referring to the true ICER ratio, the definition is Bayesian. In the same paper they define the acceptability curve in algebraic terms as An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i6.gif, where f is the joint probability density function for the random vector (An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gife,An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifc)'. Here the definition is not Bayesian, since it is the probability that, in repeated sampling, the random variable An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifeλ - An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifc (i.e., the observed net benefit) is greater than 0. In an illustration, the authors substitute the sample estimates for the model parameters in f to yield an empirical density function, call it An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i2.gif, and refer to the acceptability curve as the probability that the ICER is acceptable. Briggs and Fenn [2] refer to the acceptability curve as "the probability an intervention is cost-effective in relation to different values of" λ. For estimation they propose using the integration of An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i2.gif, as above, or determining the proportion of bootstrap re-samples in which An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifeλ - An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifc is greater than 0.

Briggs[3] provides a purely Bayesian approach by defining the acceptability curve as the probability that Δeλ - Δc is greater than 0. In an illustration the author interprets the acceptability curve as "probability of cost-effective". Assuming f is the density function for a bivariate normal random vector, and using an uninformative prior, the acceptability curve is given by A(λ)=An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i7.gifg(x)dx, where g is the probability density function for a normal random variable with mean An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifeλ - An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i1.gifc and variance An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i8.gif, where An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i9.gif and An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i10.gif are estimates of the variance of eji and cji, respectively, and An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i3.gif is an estimate of the correlation between eji and cji . This is exactly the same curve as determined by the integration of An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i2.gif, given above, and, due to the symmetry, is equal to 1 minus the p-value of the test of the hypothesis Δeλ - Δc < 0. In reporting the results of a cost-effectiveness analysis, Raikou et al. [4] interpret the acceptability curve as the "probability that intervention is cost effective".

By rewriting Pr(Δeλ - Δc > 0) as Pr(Δce < λ), assuming Δe > 0, the acceptability curve can be interpreted as the posterior distribution for the ICER. Defining A(λγ) = γ, the estimate for the ICER and its (1-α/2)100% Baysian limits are given by λ0.5, λα/2 and λ1-α/2, respectively.

The interpretation that the acceptability curve is the probability that the intervention is cost-effective is not entirely accurate and could easily be misunderstood by policy makers. Consider the situation in which the observed INB for treatment is very small, but due to a very large sample size the acceptability curve at the value of λ of interest is 0.99. Attaching the label "the probability that the intervention is cost-effective" to this quantity could mislead policy makers into thinking that treatment is highly beneficial compared to standard. What, in fact, is high is our confidence that the INB, however small, is not zero. A more accurate interpretation of the acceptability curve is that it is a measure of the certainty with which the current evidence would lead us to believe that treatment is cost-effective, i.e., Δeλ - Δc > 0. For a Bayesian, this is Pr(Δeλ - Δc) > 0, and for a frequentist, it is, assuming symmetry, 1 minus the p-value for the test of the hypothesis Δeλ - Δc < 0. This is not just the traditional confusion between statistical and clinical significance. In significance testing as the sample size increases, the variance of the estimator decreases, but the magnitude of the parameter being estimated stays the same. However, as sample size increase the magnitude of the acceptability curve for a given λ increases.

In the next section we provide a more accurate definition for the probability that treatment is cost-effective. The definition contains no element of certainty, and is the probability of the "next" patient realizing a larger net benefit if he or she is given treatment rather than standard. Using data from a clinical trial, non-parametric methods can be used to estimate this probability, and uncertainty

Methods

The probability that treatment is cost-effective

Let bji(λ) = ejiλ - cji. The quantity bji(λ) is the net benefit, expressed in money, realized by patient i on therapy j. An alternative to the acceptability curve for quantifying the probability that treatment is cost-effective is defined as: θ(λ) [equivalent] the probability, for a given λ, that a patient will receive a larger net benefit with treatment rather than standard. Its definition is not Bayesian because it is a probability statement about random variables, not population parameters. Nonetheless, it relates directly to the notion of the probability of treatment being cost-effective. A policy maker can genuinely interpret θ(λ) as the probability of the "next" patient realizing a larger net benefit if he or she is given treatment rather than standard. Such a direct interpretation is not provided by the acceptability curve. The acceptability curve is the probability that the average net benefit of a group of patients of the same size as in the clinical trial will be greater is they receive treatment rather than standard.

To estimate θ(λ) we borrow methodology from receiver operating characteristic curves [10]. An estimate of θ(λ) is given by:

An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i11.gif

If there are no ties, An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ) is the proportion of all the possible comparisons of a patient on treatment with a patient on standard in which the former was observed to have a larger net benefit. The estimate of the variance of An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ), denoted by An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i12.gif, is given by:

An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i13.gif

The 100(1 - α)% confidence interval, defined as An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ)±sθ(λ)z1-α/2, can be used to express the uncertainty regarding An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(θ), where Z1-α/2 is the 100(1 - α/2)th percentile of the standard normal distribution. We have made the assumption that in large samples An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ) will be normally distributed.The value λ0.5, defined at θ (λ0.5) = 0.5, provides a non-parametric estimator of the ICER, since it is that value of λ for which treatment and standard are equally cost-effective in the sense that Pr [bTk0.5) > bSi0.5)] = 0.5. The quantities λL and λU, defined as An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gifL)+ sθL)Z1-α/2 = 0.5 and An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gifU)- sθU)Z1-α/2 = 0.5, respectively, are the corresponding lower and upper non-parametric confidence intervals for the ICER.

The hypothesis H0: θ (λ) = 0.5 versus H1: θ (λ) > 0.5 can be tested at the level α by rejecting H0 if An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i17.gif. Rejecting H0 leads to the conclusion that the data provide evidence that treatment is cost-effective.

Suppose there is an interest in comparing θ (λ) between patient subgroups, say between males and females. To achieve this, let An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gifM(λ) and An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gifF(λ) be the estimator of θ (λ) for males and females, respectively, with corresponding estimated variances, An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i18.gif and An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i19.gif. Then the hypothesis that θ (λ) is the same for males and females can be rejected, at the α-level, 2-sided, if

An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i20.gif

There is an important distinction to be made between A(λ) and An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ). As sample size increases, A(λ) approaches 1 if Δe λ - Δc > 0, reflecting the certainty that treatment is cost-effective. If Δeλ - Δc ≤ 0, A(λ) approaches 0, reflecting the certainty that treatment is not cost-effective. The quantity An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ), being independent of sample size, approaches θ(λ) regardless of Δeλ - Δc. The certainty with which θ(λ) is estimated is reflected in sθ(λ) which is a decreasing function of the sample size.

Example

In a trial of symptomatic hormone resistant prostate cancer [11,12], 161 patients were randomized between prednisone alone (S) and prednisone plus mitoxantrone (T). Although there was no statistically significant difference in survival, there was better palliation with T. Cost data, including hospital admissions, outpatient visits, investigations, therapies and palliative care, were collected retrospectively on the 114 patients from the three largest centres. Survival was quality-adjusted using the EORTC quality of life questionnaire QLQ-C30. All patients were followed until death. The sample means and the sample variance-covariance information can be found in Table Table1.1. Cost is given in Canadian dollars (CAD$) and effectiveness in quality-adjusted life-weeks (QALW).

Table 1
Sample sizes and parameter estimates for prostate example

A plot of An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ), complete with 90% confidence intervals can be found in Figure Figure1.1. In this example all three plots have positive slope over the range of λ shown, although this would not be true for all examples. For example, by definition θ (0) = Pr(cTk - cSj < 0) and θ (λ) approaches Pr(eTk - eSj > 0) as λ becomes arbitrarily large, for any k and j. Therefore, any example in which Pr(cTk - cSj< 0) > Pr(eTk - eSj > 0) will have negative slope for some interval of positive λ. In the prostate example, since An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ) crosses the 0.5 horizontal at λ = -50, for any value greater than -50, the estimate of An external file that holds a picture, illustration, etc.
Object name is 1471-2288-1-8-i4.gif(λ) is greater than 0.5. The value -50 is a non-parametric estimate of the ICER, since for that value of λ, treatment and standard are equally cost-effective, in that the probability that a patient on treatment has the same net benefit as a patient on standard is 0.5. This compares to the parametric estimate of -134.

Figure 1
Estimate and confidence limits for θ as a function of λ.

Since the lower bound crosses the 0.5 horizontal at 334, for any value greater than 334 the hypothesis θ(λ) < 0.5 can be rejected at the 5% level of significance. Thus, a non-parametric upper bound of the ICER is 334. This compares to 378 using Fieller's theorem [13,14]. For this example a health policy maker can interpret the results as follows: for any positive λ, the estimated probability that treatment is cost-effective is greater than 50%; and for any λ greater than 334 per QALW, the probability that treatment is cost-effective is statistically significantly greater than 50%.

Discussion

As an alternative to the acceptability curve, the quantity θ(λ) is proposed as a definition for the probability that treatment is cost-effective. One advantage is that it is not sample size dependent, i.e. it is a population parameter. Another is that it has an appropriate interpretation, namely, it is the proportion of patients that realize a larger net benefit if given treatment rather than standard. The acceptability curve does not provide this, although the language often used regarding it, implies that it does. The use of θ(λ) should be helpful to policy makers, since it does not confuse the magnitude of the benefit with the certainty of its estimate.

Analysis regarding the quantity θ(λ) is not proposed as an alternative to traditional cost-effectiveness analysis for allocating health care resources. When allocating a fixed amount of resources to one of two new treatments, the proportion of patients receiving an increase in net benefit would be maximized by choosing the treatment with the larger θ(λ) to Δc ratio. However, this would not maximize net benefit, since the ratio may be larger only because the between-patient variability in cost and effectiveness is smaller, resulting in a larger θ(λ).

Non-parametric methods can be used to estimate θ(λ) and its variance. This provides the appropriate confidence intervals and test of hypothesis. In addition, non-parametric estimates of the ICER and its confidence intervals can be determined. This is of particular importance in the presence of highly skewed cost data.

Competing interests

None declared.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1471-2288/1/8/prepub

Acknowledgements

The author wishes to acknowledge the reviewers whose comments led to a much improved manuscript and to thank Gary Foster for help with the figure.

References

  • van Hout BA, Al MJ, Gordon GS, Rutten EFH. Cost, effects and C/E-ratios alongside a clinical trial. Health Economics. 1994;3:309–319. [PubMed]
  • Brigg A, Fenn P. Confidence intervals or surfaces? Uncertainty on the cost-effectiveness plane. Health Economics. 1998;7:723–740. doi: 10.1002/(SICI)1099-1050(199812)7:8<723::AID-HEC392>3.0.CO;2-O. [PubMed] [Cross Ref]
  • Briggs AH. A Bayesian approach to stochastic cost-effectiveness analysis. Health Economics. 1999;8:257–261. doi: 10.1002/(SICI)1099-1050(199905)8:3<257::AID-HEC427>3.0.CO;2-E. [PubMed] [Cross Ref]
  • Raikou M, Gray , Briggs , Stevens A, Cull C, McGuire A, Fenn P, Stratton I, Holman R, Turner R. Cost effectiveness analysis of improved blood pressure control in hypertensive patients with type 2 diabetes: UKPDS 40. British Medical Journal. 1998;317:720–726. [PMC free article] [PubMed]
  • Phelps CE, Mushlin AI. On the (near) equivalence of cost-effectiveness and cost-benefit analysis. International Journal of Technology Assessment in Health Care. 1991;7:12–21. [PubMed]
  • Ament A, Baltussen R. The interpretation of results of economic evaluation: explicating the value of health. Health Economics. 1997;6:625–635. doi: 10.1002/(SICI)1099-1050(199711)6:6<625::AID-HEC309>3.0.CO;2-O. [PubMed] [Cross Ref]
  • Stinnett AA, Mallahy J. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analsyis. Medical Decision Making. 1998;18 Suppl.:S68–S80. [PubMed]
  • Tambour M, Zethraeus , Johannesson M. A note on confidence intervals in cost-effectiveness analysis. International Journal of Technology Assessment. 1998;14:467–471. [PubMed]
  • Willan AR, Lin DY. Incremental net benefit in randomized clinical trials. Statistics in Medicine. 2001;20:1563–1574. doi: 10.1002/sim.789. [PubMed] [Cross Ref]
  • DeLong ER, DeLong DM, Clake-Pearson DL. Comparing the area under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–845. [PubMed]
  • Tannock IF, Osoba D, Stockler MR, Ernst DS, Neville AJ, Moore MJ, Armitage GR, Wilson JJ, Venner PM, Coppin CM, Murphy KC. Chemotherapy with mitoxantrone plus prednisone or prednisone alone for symptomatic hormone-resistant prostate cancer: a Canadian trial with palliative endpoints. Journal of Clinical Oncology. 1996;14:1756–1764. [PubMed]
  • Bloomfield DJ, Krahn MD, Neogi T, Panzarella T, Warde P, Willan AR, Ernst S, Moore MJ, Neville A, Tannock IF. Economic evaluation of chemotherapy with mitoxantrone plus prednisone for symptomatic hormone-resistant prostate cancer: based on a Canadian trial with palliative endpoints. Journal of Clinical Oncology. 1998;16:2272–2279. [PubMed]
  • Willan AR, O'Brien BJ. Confidence intervals for cost-effectiveness ratios: An application of Fieller's theorem. Health Economics. 1996;5:297–305. doi: 10.1002/(SICI)1099-1050(199607)5:4<297::AID-HEC216>3.0.CO;2-T. [PubMed] [Cross Ref]
  • Chaudhary MA, Stearns SC. Confidence intervals for cost-effectiveness ratios: an example from a randomized trial. Statistics in Medicine. 1996;15:1447–1458. doi: 10.1002/(SICI)1097-0258(19960715)15:13<1447::AID-SIM267>3.0.CO;2-V. [PubMed] [Cross Ref]

Articles from BMC Medical Research Methodology are provided here courtesy of BioMed Central

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...