Format

Send to

Choose Destination
Psychon Bull Rev. 2016 Apr;23(2):640-7. doi: 10.3758/s13423-015-0913-5.

Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies.

Author information

1
Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands. aoj.cramer@gmail.com.
2
Faculty of Science and Information Technology, School of Psychology, University of Newcastle, Callaghan, New South Wales, Australia.
3
Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
4
Data Analytics, Price Waterhouse Coopers, Amsterdam, The Netherlands.

Abstract

Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

KEYWORDS:

Benjamini–Hochberg procedure; Factorial ANOVA; False discovery rate; Familywise error rate; Multiple comparison problem; Multiway ANOVA; Preregistration; Sequential Bonferroni; Type I error

PMID:
26374437
PMCID:
PMC4828473
DOI:
10.3758/s13423-015-0913-5
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Springer Icon for PubMed Central
Loading ...
Support Center