NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Machine Learning Methods in Systematic Reviews: Identifying Quality Improvement Intervention Evaluations
Research White Papers
Investigators: Susanne Hempel, PhD, Kanaka D Shetty, MD, Paul G Shekelle, MD, PhD, Lisa V Rubenstein, MD, Marjorie S Danz, MD, Breanne Johnsen, BA, and Siddhartha R Dalal, PhD.
Author Information and AffiliationsStructured Abstract
Background:
Electronic searches typically yield far more citations than are relevant, and reviewers spend a substantial amount of time screening titles and abstracts to identify potential studies eligible for inclusion in a review. This is of particular relevance in complex research fields such as quality improvement. We tested a semiautomated literature screening process applied to the title and abstract screening stage of systematic reviews. A machine learning approach may allow literature reviewers to screen only a fraction of a search output and to use a predictive model to learn and then emulate the reviewers’ decisions. Once learned, the model can apply the selection process to an essentially unlimited number of citations.
Method:
Two independent literature reviewers screened 1,591 quasi-randomly selected citations in a training dataset used to predict decisions on the remaining citations in a MEDLINE search output of 9,395 citations. We explored different prediction algorithms and tested results against reference samples screened by experts in quality improvement. Qualitative (relevance cutoff determined in ROC curve) and quantitative predictions (probability rank order of citations) were determined.
Results:
The agreement between independent literature reviewers ranged from κ= 0.55 to 0.57. Across two reference samples, the predictive performance of the machine learning approach demonstrated 90.1 percent sensitivity, 43.9 percent specificity, and 32.1 percent PPV. This translates to a reduction of 36.1 percent in citation screening if applied. The predictive performance was affected by reviewer disagreements: a subgroup analysis restricted to citations both reviewers agreed on showed a sensitivity of 98.8 percent (specificity 43.9 percent).
Conclusion:
Machine learning approaches may assist in the title and abstract inclusion screening process in systematic reviews of complex, steadily expanding research fields such as quality improvement. Increased reviewer agreement appeared to be associated with improved predictive performance.
Contents
- Preface
- Acknowledgements
- Background
- Methods
- Results
- Discussion
- Conclusion
- References
- Appendix A Analytic Framework Machine Learning in Systematic Reviews
- Appendix B Search Strategy
- Appendix C Inclusion Screening Form for Publications Obtained as Full Text
- Appendix D Machine Learning Confusion Matrix, Text Terms Distinguishing Relevant and Irrelevant Citations, and Reviewer Disagreements
Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No. 290-2007-10062-I, Prepared by: Southern California Evidence-based Practice Center, RAND Corporation, Santa Monica, CA
Suggested citation:
Hempel S, Shetty KD, Shekelle PG, Rubenstein LV, Danz MS, Johnsen B, Dalal SR. Machine Learning Methods in Systematic Reviews: Identifying Quality Improvement Intervention Evaluations. Research White Paper (Prepared by the Southern California Evidence-based Practice Center under Contract No. 290-2007-10062-I). AHRQ Publication No. 12-EHC125-EF. Rockville, MD: Agency for Healthcare Research and Quality. September 2012. www.effectivehealthcare.ahrq.gov/reports/final.cfm.
This report is based on research conducted by the Southern California Evidence-based Practice Center under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10062-I). The findings and conclusions in this document are those of the author(s), who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well-informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.
This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.
None of the investigators have any affiliation or financial involvement that conflicts with the material presented in this report.
- 1
540 Gaither Road, Rockville, MD 20850; www
.ahrq.gov
- NLM CatalogRelated NLM Catalog Entries
- Review Assessing the Accuracy of Machine-Assisted Abstract Screening With DistillerAI: A User Study[ 2019]Review Assessing the Accuracy of Machine-Assisted Abstract Screening With DistillerAI: A User StudyGartlehner G, Wagner G, Lux L, Affengruber L, Dobrescu A, Kaminski-Hartenthaler A, Viswanathan M. 2019 Nov
- Review Supplemental Screening for Breast Cancer in Women With Dense Breasts: A Systematic Review for the U.S. Preventive Service Task Force[ 2016]Review Supplemental Screening for Breast Cancer in Women With Dense Breasts: A Systematic Review for the U.S. Preventive Service Task ForceMelnikow J, Fenton JJ, Whitlock EP, Miglioretti DL, Weyrich MS, Thompson JH, Shah K. 2016 Jan
- Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers.[Syst Rev. 2015]Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers.Rathbone J, Hoffmann T, Glasziou P. Syst Rev. 2015 Jun 15; 4:80. Epub 2015 Jun 15.
- Review Screening for Skin Cancer in Adults: An Updated Systematic Evidence Review for the U.S. Preventive Services Task Force[ 2016]Review Screening for Skin Cancer in Adults: An Updated Systematic Evidence Review for the U.S. Preventive Services Task ForceWernli KJ, Henrikson NB, Morrison CC, Nguyen M, Pocobelli G, Whitlock EP. 2016 Jul
- Semi-automated screening of biomedical citations for systematic reviews.[BMC Bioinformatics. 2010]Semi-automated screening of biomedical citations for systematic reviews.Wallace BC, Trikalinos TA, Lau J, Brodley C, Schmid CH. BMC Bioinformatics. 2010 Jan 26; 11:55. Epub 2010 Jan 26.
- Machine Learning Methods in Systematic Reviews: Identifying Quality Improvement ...Machine Learning Methods in Systematic Reviews: Identifying Quality Improvement Intervention Evaluations
- Radiotherapy Treatments for Head and Neck Cancer UpdateRadiotherapy Treatments for Head and Neck Cancer Update
- Comparative Effectiveness and Safety of Radiotherapy Treatments for Head and Nec...Comparative Effectiveness and Safety of Radiotherapy Treatments for Head and Neck Cancer
- Results - Radiotherapy Treatments for Head and Neck Cancer UpdateResults - Radiotherapy Treatments for Head and Neck Cancer Update
- Discussion - Radiotherapy Treatments for Head and Neck Cancer UpdateDiscussion - Radiotherapy Treatments for Head and Neck Cancer Update
Your browsing activity is empty.
Activity recording is turned off.
See more...