The current approach to assessing the quality of evidence obtained from clinical trials focuses on three dimensions: the quality of the design (with double-blinded randomised controlled trials representing the highest level of such design); the statistical power (beta) and the level of significance (alpha). While these aspects are important, we argue that other significant aspects of trial quality impinge upon the truthfulness of the findings: biological plausibility, reproducibility and generalisability. We present several recent studies in critical care medicine where the design, beta and alpha components of the study are seemingly satisfactory but where the aspects of biological plausibility, reproducibility and generalisability show serious limitations. Accordingly, we argue for more reflection, definition and consensus on these aspects of the evaluation of evidence.