Format

Send to

Choose Destination
Stat Sci. 2017 Aug;32(3):385-404. doi: 10.1214/16-STS604.

Principles of Experimental Design for Big Data Analysis.

Author information

1
School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia, 4000.
2
Department of Statistics, University of Oxford, Oxford, UK, OX1 3TG.
3
MRC Biostatistics Unit, Cambridge Institute of Public Health, Cambridge, UK, CB2 0SR.
4
Biostatistics Department, King's College London, UK, SE5 8AF.

Abstract

Big Datasets are endemic, but are often notoriously difficult to analyse because of their size, heterogeneity and quality. The purpose of this paper is to open a discourse on the potential for modern decision theoretic optimal experimental design methods, which by their very nature have traditionally been applied prospectively, to improve the analysis of Big Data through retrospective designed sampling in order to answer particular questions of interest. By appealing to a range of examples, it is suggested that this perspective on Big Data modelling and analysis has the potential for wide generality and advantageous inferential and computational properties. We highlight current hurdles and open research questions surrounding efficient computational optimisation in using retrospective designs, and in part this paper is a call to the optimisation and experimental design communities to work together in the field of Big Data analysis.

KEYWORDS:

active learning; big data; dimension reduction; experimental design; sub-sampling

Supplemental Content

Full text links

Icon for PubMed Central
Loading ...
Support Center