Send to

Choose Destination
Value Health. 2017 Feb;20(2):200-205. doi: 10.1016/j.jval.2016.12.011. Epub 2017 Feb 10.

Validity and Reliability of Value Assessment Frameworks for New Cancer Drugs.

Author information

Partnership for Health Analytic Research, LLC, Beverly Hills, CA, USA. Electronic address:
Tufts Medical Center, Boston, MA, USA.
Memorial Sloan Kettering Cancer Center, New York, NY, USA.
Hematology Oncology of San Fernando Valley, Encino, CA, USA.
Health Sciences Program, California State University, East Bay, Hayward, CA, USA.
Department of Medicine, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA.
City of Hope National Medical Center, Duarte, CA, USA.
Eisai Inc., Woodcliff Lake, NJ, USA.
Partnership for Health Analytic Research, LLC, Beverly Hills, CA, USA.



Several organizations have developed frameworks to systematically assess the value of new drugs. These organizations include the American Society of Clinical Oncology (ASCO), the European Society for Medical Oncology (ESMO), the Institute for Clinical and Economic Review (ICER), and the National Comprehensive Cancer Network (NCCN).


To understand the extent to which these four tools can facilitate value-based treatment decisions in oncology.


In this pilot study, eight panelists conducted value assessments of five advanced lung cancer drugs using the ASCO, ESMO, and ICER frameworks. The panelists received instructions and published clinical data required to complete the assessments. Published NCCN framework scores were abstracted. The Kendall's W coefficient was used to measure convergent validity among the four frameworks. Intraclass correlation coefficients were used to measure inter-rater reliability among the ASCO, ESMO, and ICER frameworks. Sensitivity analyses were conducted.


Drugs were ranked similarly by the four frameworks, with Kendall's W of 0.703 (P = 0.006) across all the four frameworks. Pairwise, Kendall's W was the highest for ESMO-ICER (W = 0.974; P = 0.007) and ASCO-NCCN (W = 0.944; P = 0.022) and the lowest for ICER-NCCN (W = 0.647; P = 0.315) and ESMO-NCCN (W = 0.611; P = 0.360). Intraclass correlation coefficients (confidence interval [CI]) for the ASCO, ESMO, and ICER frameworks were 0.786 (95% CI 0.517-0.970), 0.804 (95% CI 0.545-0.973), and 0.281 (95% CI 0.055-0.799), respectively. When scores were rescaled to 0 to 100, the ICER framework provided the narrowest band of scores.


The ASCO, ESMO, ICER, and NCCN frameworks demonstrated convergent validity, despite differences in conceptual approaches used. The ASCO inter-rater reliability was high, although potentially at the cost of user burden. The ICER inter-rater reliability was poor, possibly because of its failure to distinguish differential value among the sample of drugs tested. Refinements of all frameworks should continue on the basis of further testing and stakeholder feedback.


convergent validity; inter-rater reliability; oncology; value frameworks

[Indexed for MEDLINE]
Free full text

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center