Format

Send to

Choose Destination
Psychol Methods. 2012 Dec;17(4):551-66. doi: 10.1037/a0029487. Epub 2012 Aug 27.

The ironic effect of significant results on the credibility of multiple-study articles.

Author information

1
Department of Psychology, University of Toronto Mississauga, Ontario, Canada. uli.schimmack@utoronto.ca

Abstract

Cohen (1962) pointed out the importance of statistical power for psychology as a science, but statistical power of studies has not increased, while the number of studies in a single article has increased. It has been overlooked that multiple studies with modest power have a high probability of producing nonsignificant results because power decreases as a function of the number of statistical tests that are being conducted (Maxwell, 2004). The discrepancy between the expected number of significant results and the actual number of significant results in multiple-study articles undermines the credibility of the reported results, and it is likely that questionable research practices have contributed to the reporting of too many significant results (Sterling, 1959). The problem of low power in multiple-study articles is illustrated using Bem's (2011) article on extrasensory perception and Gailliot et al.'s (2007) article on glucose and self-regulation. I conclude with several recommendations that can increase the credibility of scientific evidence in psychological journals. One major recommendation is to pay more attention to the power of studies to produce positive results without the help of questionable research practices and to request that authors justify sample sizes with a priori predictions of effect sizes. It is also important to publish replication studies with nonsignificant results if these studies have high power to replicate a published finding.

PMID:
22924598
DOI:
10.1037/a0029487
[Indexed for MEDLINE]

Supplemental Content

Full text links

Icon for American Psychological Association
Loading ...
Support Center