Questions about the reproducibility of scientific research have been raised in numerous settings and have gained visibility through several high-profile journal and popular press articles. Quantitative issues contributing to reproducibility challenges have been considered (including improper data measurement and analysis, inadequate statistical expertise, and incomplete data, among others), but there is no clear consensus on how best to approach or to minimize these problems.

A lack of reproducibility of scientific results has created some distrust in scientific findings among the general public, scientists, funding agencies, and industries. While studies fail for a variety of reasons, many factors contribute to the lack of perfect reproducibility, including insufficient training in experimental design, misaligned incentives for publication and the implications for university tenure, intentional manipulation, poor data management and analysis, and inadequate instances of statistical inference.

The workshop summarized in this report was designed not to address the social and experimental challenges but instead to focus on the latter issues of improper data management and analysis, inadequate statistical expertise, incomplete data, and difficulties applying sound statistic inference to the available data. Many efforts have emerged over recent years to draw attention to and improve reproducibility of scientific work. This report uniquely focuses on the statistical perspective of three issues: the extent of reproducibility, the causes of reproducibility failures, and the potential remedies for these failures.

Contents

• The National Academies of SCIENCES • ENGINEERING • MEDICINE
• PLANNING COMMITTEE ON STATISTICAL CHALLENGES IN ASSESSING AND FOSTERING THE REPRODUCIBILITY OF SCIENTIFIC RESULTS
• COMMITTEE ON APPLIED AND THEORETICAL STATISTICS
• BOARD ON MATHEMATICAL SCIENCES AND THEIR APPLICATIONS
• Acknowledgment of Reviewers
• 1. Introduction
  • WORKSHOP OVERVIEW
  • WORKSHOP THEMES
  • ORGANIZATION OF THIS REPORT
• 2. Overview and Case Studies
  • OPENING REMARKS FROM THE WORKSHOP CO-CHAIRS
  • PERSPECTIVES FROM STAKEHOLDERS
  • OVERVIEW OF THE STATISTICAL CHALLENGES OF REPRODUCIBILITY
  • CASE STUDIES
• 3. Conceptualizing, Measuring, and Studying Reproducibility
  • DEFINITIONS AND MEASURES OF REPRODUCIBILITY
  • REPRODUCIBILITY AND STATISTICAL SIGNIFICANCE
  • ASSESSMENT OF FACTORS AFFECTING REPRODUCIBILITY
  • REPRODUCIBILITY FROM THE INFORMATICS PERSPECTIVE
• 4. The Way Forward: Using Statistics to Improve Reproducibility
  • OPEN PROBLEMS, NEEDS, AND OPPORTUNITIES FOR METHODOLOGIC RESEARCH
  • REPORTING SCIENTIFIC RESULTS AND SHARING SCIENTIFIC STUDY DATA
  • THE WAY FORWARD FROM THE DATA SCIENCES PERSPECTIVE: RESEARCH
• References
• APPENDIXES
  • A. Registered Workshop Participants
  • B. Workshop Agenda
  • C. Acronyms

Rapporteur: Michelle Schwalbe.

Suggested citation:

National Academies of Sciences, Engineering, and Medicine. 2016. Statistical Challenges in Assessing and Fostering the Reproducibility of Scientific Results: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/21915.