Display Settings:

Format

Send to:

Choose Destination
PLoS One. 2009;4(2):e4607. doi: 10.1371/journal.pone.0004607. Epub 2009 Feb 25.

Decision-making in research tasks with sequential testing.

Author information

  • 1Program for Evolutionary Dynamics, Harvard University, Cambridge, Massachusetts, United States of America. pfeiffer@fas.harvard.edu

Abstract

BACKGROUND:

In a recent controversial essay, published by JPA Ioannidis in PLoS Medicine, it has been argued that in some research fields, most of the published findings are false. Based on theoretical reasoning it can be shown that small effect sizes, error-prone tests, low priors of the tested hypotheses and biases in the evaluation and publication of research findings increase the fraction of false positives. These findings raise concerns about the reliability of research. However, they are based on a very simple scenario of scientific research, where single tests are used to evaluate independent hypotheses.

METHODOLOGY/PRINCIPAL FINDINGS:

In this study, we present computer simulations and experimental approaches for analyzing more realistic scenarios. In these scenarios, research tasks are solved sequentially, i.e. subsequent tests can be chosen depending on previous results. We investigate simple sequential testing and scenarios where only a selected subset of results can be published and used for future rounds of test choice. Results from computer simulations indicate that for the tasks analyzed in this study, the fraction of false among the positive findings declines over several rounds of testing if the most informative tests are performed. Our experiments show that human subjects frequently perform the most informative tests, leading to a decline of false positives as expected from the simulations.

CONCLUSIONS/SIGNIFICANCE:

For the research tasks studied here, findings tend to become more reliable over time. We also find that the performance in those experimental settings where not all performed tests could be published turned out to be surprisingly inefficient. Our results may help optimize existing procedures used in the practice of scientific research and provide guidance for the development of novel forms of scholarly communication.

PMID:
19240797
[PubMed - indexed for MEDLINE]
PMCID:
PMC2643008
Free PMC Article

Images from this publication.See all images (4)Free text

Figure 1
Figure 2
Figure 3
Figure 4
PubMed Commons home

PubMed Commons

0 comments
How to join PubMed Commons

    Supplemental Content

    Full text links

    Icon for Public Library of Science Icon for PubMed Central
    Loading ...
    Write to the Help Desk