Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research

PLoS Biol. 2021 May 19;19(5):e3001009. doi: 10.1371/journal.pbio.3001009. eCollection 2021 May.

Abstract

The replicability of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets replicability has also been recently challenged, with the observation that the reduction of variability within studies can lead to idiosyncratic, lab-specific results that cannot be replicated. An alternative approach is to, instead, deliberately introduce heterogeneity, known as "heterogenization" of experimental design. Here, we explore a novel perspective in the heterogenization program in a meta-analysis of variability in observed phenotypic outcomes in both control and experimental animal models of ischemic stroke. First, by quantifying interindividual variability across control groups, we illustrate that the amount of heterogeneity in disease state (infarct volume) differs according to methodological approach, for example, in disease induction methods and disease models. We argue that such methods may improve replicability by creating diverse and representative distribution of baseline disease state in the reference group, against which treatment efficacy is assessed. Second, we illustrate how meta-analysis can be used to simultaneously assess efficacy and stability (i.e., mean effect and among-individual variability). We identify treatments that have efficacy and are generalizable to the population level (i.e., low interindividual variability), as well as those where there is high interindividual variability in response; for these, latter treatments translation to a clinical setting may require nuance. We argue that by embracing rather than seeking to minimize variability in phenotypic outcomes, we can motivate the shift toward heterogenization and improve both the replicability and generalizability of preclinical research.

Publication types

  • Research Support, Non-U.S. Gov't

MeSH terms

  • Animal Experimentation / standards*
  • Animals
  • Behavior, Animal / physiology
  • Brain Ischemia / metabolism
  • Humans
  • Meta-Analysis as Topic
  • Models, Animal
  • Phenotype
  • Reference Standards
  • Reproducibility of Results
  • Research Design / standards*
  • Research Design / trends
  • Stroke / physiopathology

Associated data

  • figshare/10.6084/m9.figshare.14527317.v4

Grants and funding

AMS was supported by a Discovery Early Career Researcher Award from the Australian Research Council (ARC DECRA: DE180101520: https://www.arc.gov.au/grants/discovery-program/discovery-early-career-researcher-award-decra), and formally by a Coffey Fellowship from the University of Sydney. SN was supported by the Australian Research Council Discovery Grants (DP180100818 and DP200100367: https://www.arc.gov.au/grants/discovery-program/discovery-projects). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.