Format

Send to

Choose Destination
See comment in PubMed Commons below
Stat Med. 2014 Jan 30;33(2):209-18. doi: 10.1002/sim.5925. Epub 2013 Jul 30.

Interpreting observational studies: why empirical calibration is needed to correct p-values.

Author information

1
Department of Medical Informatics, Erasmus University Medical Center Rotterdam, Rotterdam, The Netherlands; Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, U.S.A.

Abstract

Often the literature makes assertions of medical product effects on the basis of ' p < 0.05'. The underlying premise is that at this threshold, there is only a 5% probability that the observed effect would be seen by chance when in reality there is no effect. In observational studies, much more than in randomized trials, bias and confounding may undermine this premise. To test this premise, we selected three exemplar drug safety studies from literature, representing a case-control, a cohort, and a self-controlled case series design. We attempted to replicate these studies as best we could for the drugs studied in the original articles. Next, we applied the same three designs to sets of negative controls: drugs that are not believed to cause the outcome of interest. We observed how often p < 0.05 when the null hypothesis is true, and we fitted distributions to the effect estimates. Using these distributions, we compute calibrated p-values that reflect the probability of observing the effect estimate under the null hypothesis, taking both random and systematic error into account. An automated analysis of scientific literature was performed to evaluate the potential impact of such a calibration. Our experiment provides evidence that the majority of observational studies would declare statistical significance when no effect is present. Empirical calibration was found to reduce spurious results to the desired 5% level. Applying these adjustments to literature suggests that at least 54% of findings with p < 0.05 are not actually statistically significant and should be reevaluated.

KEYWORDS:

calibration; hypothesis testing; negative controls; observational studies

PMID:
23900808
PMCID:
PMC4285234
DOI:
10.1002/sim.5925
[Indexed for MEDLINE]
Free PMC Article
PubMed Commons home

PubMed Commons

0 comments
How to join PubMed Commons

    Supplemental Content

    Full text links

    Icon for Wiley Icon for PubMed Central
    Loading ...
    Support Center