Robin Hood: A cost-efficient two-stage approach to large-scale simultaneous inference with non-homogeneous sparse effects

A classical approach to experimental design in many scientific fields is to first gather all of the data and then analyze it in a single analysis. It has been recognized that in many areas such practice leaves substantial room for improvement in terms of the researcher's ability to identify relevant effects, in terms of cost efficiency, or both. Considerable attention has been paid in recent years to multi-stage designs, in which the user alternates between data collection and analysis and thereby sequentially reduces the size of the problem. However, the focus has generally been towards designs that require a hypothesis be tested in every single stage before it can be declared as rejected by the procedure. Such procedures are well-suited for homogeneous effects, i.e. effects of (almost) equal sizes, however, with effects of varying size a procedure that permits rejection at interim stages is much more suitable. Here we present precisely such multi-stage testing procedure called Robin Hood. We show that with heterogeneous effects our method substantially improves on the existing multi-stage procedures with an essentially zero efficiency trade-off in the homogeneous effect realm, which makes it especially useful in areas such as genetics, where heterogeneous effects are common. Our method improves on existing approaches in a number of ways including a novel way of performing two-sided testing in a multi-stage procedure with increased power for detecting small effects.