NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Modeling and Simulation in the Context of Health Technology Assessment: Review of Existing Guidance, Future Research Needs, and Validity Assessment
Methods Research Reports
Investigators: Issa J Dahabreh, MD, MS, Jeffrey A Chan, BA, Amy Earley, BS, Denish Moorthy, MD, Esther E Avendano, BA, Thomas A Trikalinos, MD, Ethan M Balk, MD, MPH, and John B Wong, MD.
Author Information and AffiliationsStructured Abstract
Background:
Despite rigorous systematic reviews of efficacy and effectiveness of health care interventions, patients, providers and policymakers may remain in doubt about what they should do because of uncertainty, tradeoffs among benefits and harms, and conflicting preferences. Modeling and simulation studies in health care can supplement systematic reviews to increase the usefulness of the evidence summary. The aims of this report are four-fold: (1) to summarize evidence- and consensus-based guidance on the conduct and reporting of health care modeling and simulation; (2) to summarize guidance from Health Technology Assessment (HTA) groups for modeling; (3) to prioritize future research needs to improve models; and (4) to provide an overview of methods for model calibration and validation.
Methods:
With guidance from a Technical Expert Panel and a Clinical and Policy Advisory team with clinical, methodological research, and policymaking expertise, we completed the following projects: For Aim 1, we conducted a systematic review of articles that provided evidence- or consensus-based recommendations for the conduct and reporting of health care modeling and simulation studies. We classified recommendation statements into four domains: model structure, data, consistency, and communication of model results. To contextualize the findings of the systematic review, we organized a meeting with a group of 28 stakeholders, including modelers, users of models, and funders of research. For Aim 2, we searched the web sites of 126 international agencies and institutes conducting HTA for real-world practices regarding when to apply modeling and simulation methods, which we summarized. For Aim 3, from the systematic review and from the stakeholders in Aim 1 we identified and collected suggestions for future research needs. Stakeholders prioritized those needs based on importance, desirability of new research, feasibility, and potential impact. For Aim 4, we searched for articles that compared or applied alternative validation methods for modeling and simulation. We extracted and summarized descriptions and comparisons of any methods and reported results for face validity, internal validity, external validity, cross-model validation, and calibration.
Results:
The systematic review of modeling recommendations (Aim 1) found 71 eligible articles. 90 percent of articles (n=64) provided recommendations regarding model structure. Almost all articles (n=68, 96%) also provided recommendations regarding obtaining appropriate data to populate models. Stakeholders highlighted the importance of establishing guiding principles for “good practice” but discouraged the use of “cookbook” checklists. Of the 71 articles, 38 (54%) provided suggestions for future research; stakeholders provided 28 additional suggestions. We found 21 HTA organizations provided guidance (Aim 2) through their web sites regarding the application of modeling and simulation in the context of conducting a HTA. The HTA organizations varied widely in what areas of modeling they provided guidance for and what specific recommendations they made. In general, HTA organizations favored incorporating models into HTA, provided recommendations on how to model data and structure, and recommended inclusion of costs in cost-effectiveness models. Future research needs that were prioritized (Aim 3) included questions about model data, model structure, consistency, and reporting. Studies comparing validation methods (Aim 4) provided information on model validation (face validity and internal, external, and cross-model validation) and calibration (varying specifications of the calibration problem with the same and different algorithms and use of alternative algorithms for the same calibration problem).
Conclusion:
Our systematic review and stakeholder meeting summary provides a comprehensive compendium of guidance documents for modeling and simulation studies, annotated with information on the domains covered by each document, and the methods used to arrive at specific recommendations. We also summarized modeling recommendations for HTA organizations. These processes enabled us to prioritize future research needs to form an empirical basis for and to improve recommendations for modeling. Our overview of model validation and calibration provides insights into the relative value and efficiency of different methods.
Contents
- Preface
- Technical Expert Panel
- Peer Reviewers
- Acknowledgments
- Preamble
- Chapter 1. Systematic Review of Recommendations for the Conduct and Reporting of Health Care Modeling and Simulation Studies
- Chapter 2. Review of Guidance from Health Technology Assessment Organizations
- Chapter 3. Future Research Needs for Health Care Modeling and Simulation Studies
- Chapter 4. A Review of Validation and Calibration Methods for Health Care Modeling and Simulation
- References
- Appendix A. Acknowledgments
- Appendix B. Systematic Review Search Strategy
- Appendix C. Health Technology Assessment Organizations
- Appendix D. HTA Organization Data Extraction
Suggested citation:
Dahabreh IJ, Chan JA, Earley A, Moorthy D, Avendano EE, Trikalinos TA, Balk EM, Wong JB. Modeling and Simulation in the Context of Health Technology Assessment: Review of Existing Guidance, Future Research Needs, and Validity Assessment. Methods Research Report. (Prepared by the Tufts Evidence-based Practice Center under Contract No. 290-2007-10055-I.) AHRQ Publication No.16(17)-EHC020-EF. Rockville, MD: Agency for Healthcare Research and Quality. January 2017. www.effectivehealthcare.ahrq.gov/reports/final.cfm.
This report is based on research conducted by the Tufts Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10055-I). The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.
The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.
AHRQ or U.S. Department of Health and Human Services endorsement of any derivative products that may be developed from this report, such as clinical practice guidelines, other quality enhancement tools, or reimbursement or coverage policies may not be stated or implied.
- 1
5600 Fishers Lane, Rockville, MD 20857; www
.ahrq.gov
- NLM CatalogRelated NLM Catalog Entries
- Review Evaluating the Potential Use of Modeling and Value-of-Information Analysis for Future Research Prioritization Within the Evidence-Based Practice Center Program[ 2011]Review Evaluating the Potential Use of Modeling and Value-of-Information Analysis for Future Research Prioritization Within the Evidence-Based Practice Center ProgramMyers E, Sanders GD, Ravi D, Matchar D, Havrilesky L, Samsa G, Powers B, McBroom A, Musty M, Gray R. 2011 Jun
- A research roadmap for complementary and alternative medicine - what we need to know by 2020.[Forsch Komplementmed. 2014]A research roadmap for complementary and alternative medicine - what we need to know by 2020.Fischer F, Lewith G, Witt CM, Linde K, von Ammon K, Cardini F, Falkenberg T, Fønnebø V, Johannessen H, Reiter B, et al. Forsch Komplementmed. 2014; 21(2):e1-16. Epub 2014 Mar 24.
- Review Decision and Simulation Modeling in Systematic Reviews[ 2013]Review Decision and Simulation Modeling in Systematic ReviewsKuntz K, Sainfort F, Butler M, Taylor B, Kulasingam S, Gregory S, Mann E, Anderson JM, Kane RL. 2013 Feb
- Review Guidance for the Conduct and Reporting of Modeling and Simulation Studies in the Context of Health Technology Assessment.[Methods Guide for Effectivenes...]Review Guidance for the Conduct and Reporting of Modeling and Simulation Studies in the Context of Health Technology Assessment.Dahabreh IJ, Trikalinos TA, Balk EM, Wong JB. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. 2008
- Review Avoiding and identifying errors in health technology assessment models: qualitative study and methodological review.[Health Technol Assess. 2010]Review Avoiding and identifying errors in health technology assessment models: qualitative study and methodological review.Chilcott J, Tappenden P, Rawdin A, Johnson M, Kaltenthaler E, Paisley S, Papaioannou D, Shippam A. Health Technol Assess. 2010 May; 14(25):iii-iv, ix-xii, 1-107.
- Modeling and Simulation in the Context of Health Technology Assessment: Review o...Modeling and Simulation in the Context of Health Technology Assessment: Review of Existing Guidance, Future Research Needs, and Validity Assessment
- Methods Guide for Medical Test ReviewsMethods Guide for Medical Test Reviews
- Methods Guide for Effectiveness and Comparative Effectiveness ReviewsMethods Guide for Effectiveness and Comparative Effectiveness Reviews
- Proceedings of a Workshop - Advancing Maternal Health Equity and Reducing Matern...Proceedings of a Workshop - Advancing Maternal Health Equity and Reducing Maternal Morbidity and Mortality
- Pharmacotherapy for Alcohol DependencePharmacotherapy for Alcohol Dependence
Your browsing activity is empty.
Activity recording is turned off.
See more...