Response to Comment on "Estimating the reproducibility of psychological science"

Christopher J Anderson; Štěpán Bahník; Michael Barnett-Cowan; Frank A Bosco; Jesse Chandler; Christopher R Chartier; Felix Cheung; Cody D Christopherson; Andreas Cordes; Edward J Cremata; Nicolas Della Penna; Vivien Estel; Anna Fedor; Stanka A Fitneva; Michael C Frank; James A Grange; Joshua K Hartshorne; Fred Hasselman; Felix Henninger; Marije van der Hulst; Kai J Jonas; Calvin K Lai; Carmel A Levitan; Jeremy K Miller; Katherine S Moore; Johannes M Meixner; Marcus R Munafò; Koen I Neijenhuijs; Gustav Nilsonne; Brian A Nosek; Franziska Plessow; Jason M Prenoveau; Ashley A Ricker; Kathleen Schmidt; Jeffrey R Spies; Stefan Stieger; Nina Strohminger; Gavin B Sullivan; Robbie C M van Aert; Marcel A L M van Assen; Wolf Vanpaemel; Michelangelo Vianello; Martin Voracek; Kellylynn Zuni

doi:10.1126/science.aad9163

Response to Comment on "Estimating the reproducibility of psychological science"

Science. 2016 Mar 4;351(6277):1037. doi: 10.1126/science.aad9163.

Authors

Christopher J Anderson¹, Štěpán Bahník², Michael Barnett-Cowan³, Frank A Bosco⁴, Jesse Chandler⁵, Christopher R Chartier⁶, Felix Cheung⁷, Cody D Christopherson⁸, Andreas Cordes⁹, Edward J Cremata¹⁰, Nicolas Della Penna¹¹, Vivien Estel¹², Anna Fedor¹³, Stanka A Fitneva¹⁴, Michael C Frank¹⁵, James A Grange¹⁶, Joshua K Hartshorne¹⁷, Fred Hasselman¹⁸, Felix Henninger¹⁹, Marije van der Hulst²⁰, Kai J Jonas²¹, Calvin K Lai²², Carmel A Levitan²³, Jeremy K Miller²⁴, Katherine S Moore²⁵, Johannes M Meixner²⁶, Marcus R Munafò²⁷, Koen I Neijenhuijs²⁸, Gustav Nilsonne²⁹, Brian A Nosek³⁰, Franziska Plessow³¹, Jason M Prenoveau³², Ashley A Ricker³³, Kathleen Schmidt³⁴, Jeffrey R Spies³⁵, Stefan Stieger³⁶, Nina Strohminger³⁷, Gavin B Sullivan³⁸, Robbie C M van Aert³⁹, Marcel A L M van Assen⁴⁰, Wolf Vanpaemel⁴¹, Michelangelo Vianello⁴², Martin Voracek⁴³, Kellylynn Zuni⁴⁴

Affiliations

¹ Russell Sage College, Troy, NY, USA. nosek@virginia.edu.
² University of Würzburg, Würzburg, Germany.
³ University of Waterloo, Waterloo, Ontario, Canada.
⁴ Virginia Commonwealth University, Richmond, VA, USA.
⁵ University of Michigan, Ann Arbor, MI 48104, USA. Mathematica Policy Research, Washington, DC, USA.
⁶ Ashland University, Ashland, OH, USA.
⁷ Michigan State University, East Lansing, MI, USA.
⁸ Southern Oregon University, Ashland, OR, USA.
⁹ University of Göttingen, Institute for Psychology, Göttingen, Germany.
¹⁰ University of Southern California, Los Angeles, CA, USA.
¹¹ Australian National University, Canberra, Australia.
¹² Technische Universität Braunschweig, Braunschweig, Germany.
¹³ Parmenides Stiftung, Munich, Germany.
¹⁴ Queen's University, Kingston, Ontario, Canada.
¹⁵ Stanford University, Stanford, CA, USA.
¹⁶ Keele University, Keele, Staffordshire, UK.
¹⁷ Boston College, Chestnut Hill, MA, USA.
¹⁸ Radboud University Nijmegen, Nijmegen, Netherlands.
¹⁹ University of Koblenz-Landau, Landau, Germany.
²⁰ Erasmus Medical Center, Rotterdam, Netherlands.
²¹ University of Amsterdam, Amsterdam, Netherlands.
²² Harvard University, Cambridge, MA, USA.
²³ Occidental College, Los Angeles, CA, USA.
²⁴ Willamette University, Salem, OR, USA.
²⁵ Arcadia University, Glenside, PA, USA.
²⁶ University of Potsdam, Potsdam, Germany.
²⁷ University of Bristol, Bristol, UK.
²⁸ Vrije Universiteit Amsterdam, Amsterdam, Netherlands.
²⁹ Karolinska Institutet, Stockholm University, Stockholm, Sweden.
³⁰ Center for Open Science, Charlottesville, VA, USA. University of Virginia, Charlottesville, VA, USA. nosek@virginia.edu.
³¹ Harvard Medical School, Boston, MA, USA.
³² Loyola University, Baltimore, MD, USA.
³³ University of California, Riverside, CA, USA.
³⁴ Wesleyan University, Middletown, CT, USA.
³⁵ Center for Open Science, Charlottesville, VA, USA. University of Virginia, Charlottesville, VA, USA.
³⁶ University of Konstanz, Konstanz, Germany.
³⁷ Yale University, New Haven, CT, USA.
³⁸ Coventry University, Coventry, UK.
³⁹ Tilburg University, Tilburg, Netherlands.
⁴⁰ Tilburg University, Tilburg, Netherlands. Utrecht University, Utrecht, Netherlands.
⁴¹ University of Leuven, Leuven, Belgium.
⁴² University of Padova, Padova, Italy.
⁴³ University of Vienna, Vienna, Austria.
⁴⁴ Adams State University, Alamosa, CO, USA.

PMID: 26941312
DOI: 10.1126/science.aad9163

Abstract

Gilbert et al. conclude that evidence from the Open Science Collaboration's Reproducibility Project: Psychology indicates high reproducibility, given the study methodology. Their very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data. Using the Reproducibility Project: Psychology data, both optimistic and pessimistic conclusions about reproducibility are possible, and neither are yet warranted.

Publication types

Research Support, Non-U.S. Gov't
Comment

MeSH terms

Behavioral Research*
Psychology*
Publishing*
Research*