Most recommended medical interventions reach P < 0.005 for their primary outcomes in meta-analyses

Background: It has been proposed that the threshold of statistical significance should shift from P-value < 0.05 to P-value < 0.005, but there is concern that this move may dismiss effective, useful interventions. We aimed to assess how often medical interventions are recommended although their evidence in meta-analyses of randomized trials lies between P-value = 0.05 and P-value = 0.005.

Methods: We included Cochrane systematic reviews (SRs) published from 1 January 2013 to 30 June 2014 that had at least one meta-analysis with GRADE (Grading of Recommendations Assessment, Development and Evaluation) assessment and at least one primary outcome having favourable results for efficacy at P-value < 0.05. Only comparisons of randomized trials between active versus no treatment/placebo were included. We then assessed the respective UpToDate recommendations for clinical practice from 22 May 2018 to 5 October 2018 and recorded how many treatments were recommended and what were the P-values in their meta-analysis evidence. The primary analysis was based on the first-listed outcomes.

Results: Of 608 screened SRs with GRADE assessment, 113 SRs were eligible, including 143 comparisons of which 128 comparisons had first-listed primary outcomes with UpToDate coverage. Altogether, 60% (58/97) of interventions with P-values < 0.005 for their evidence were recommended versus 32% (10/31) of those with P-value 0.005-0.05. Therefore, most (58/68, 85.2%) of the recommended interventions had P-values < 0.005 for the first-listed primary outcome. Of the 10 exceptions, 4 had other primary outcomes with P-values < 0.005 and another 4 had additional extensive evidence for similar indications that would allow extrapolation for practice recommendations.

Conclusions: Few interventions are recommended without their evidence from meta-analyses of randomized trials reaching P-value < 0.005.