NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Structured Abstract
Objectives:
To examine associations between a set of trial quality criteria and effect sizes and to explore factors influencing the detection of associations in meta-epidemiological datasets.
Data Sources:
The analyses are based on four meta-epidemiological datasets. These datasets consist of a number of meta-analyses; each contained between 100 and 216 controlled trials. These datasets have “known” qualities, as they were used in published research to investigate associations between quality and effect sizes. In addition, we created datasets using Monte Carlo simulation methods to examine their properties.
Review Methods:
We identified treatment effect meta-analyses and included trials and extracted treatment effects for four meta-epidemiological datasets. We assessed quality and risk of bias indicators with 11 Cochrane Back Review Group (CBRG) criteria. In addition, we applied the Jadad criteria, criteria proposed by Schulz (e.g., allocation concealment), and the Cochrane Risk of Bias tool. We investigated the effect of individual criteria and quantitative summary scores on the reported treatment effect sizes. We explored potential reasons for differences in associations across different meta-epidemiological datasets, clinical fields and individual meta-analyses. We investigated factors that influence the power to detect associations between quality and effect sizes in Monte Carlo simulations.
Results:
Associations between quality and effect sizes were small, e.g. the ratio of odds ratios (ROR) for unconcealed (vs. concealed) trials was 0.89 (95% CI: 0.73, 1.09, n.s.), but consistent across the CBRG criteria. Based on a quantitative summary score, a cut-off of six or more criteria met (out of 11) differentiated low- and high-quality trials best with lower quality trials reporting larger treatment effects (ROR 0.86, 95% CI: 0.70, 1.06, n.s.). Results for evidence of bias varied between datasets, clinical fields, and individual meta-analyses. The simulations showed that the power to detect quality effects is, to a large extent, determined by the degree of residual heterogeneity present in the dataset.
Conclusions:
Although trial quality may explain some amount of heterogeneity across trial results in meta-analyses, the amount of additional heterogeneity in effect sizes is a crucial factor in determining when associations between quality and effect sizes can be detected. Detecting quality moderator effects requires more statistically powerful analyses than are employed in most investigations.
Contents
Prepared for: Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services1, Contract No. HHSA 290-2007-10056-I. Prepared by: Southern California Evidence-based Practice Center2
Suggested citation:
Hempel S, Miles J, Suttorp M, Wang Z, Johnsen B, Morton S, Perry T, Valentine D, Shekelle P. Detection of Associations between Trial Quality and Effect Sizes. Methods Research Report. Prepared by the Southern California Evidence-based Practice Center under Contract No. 290-2007-10062-I. AHRQ Publication No. 12-EHC010-EF. Rockville, MD: Agency for Healthcare Research and Quality; January 2012.
This report is based on research conducted by the Southern California Evidence-based Practice Center (EPC) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. HHSA 290 2007 10056 I). The findings and conclusions in this document are those of the author(s), who are responsible for its content; and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.
The information in this report is intended to help clinicians, employers, policymakers, and others make informed decisions about the provision of health care services. This report is intended as a reference and not as a substitute for clinical judgment.
This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.
None of the investigators has any affiliations or financial involvement that conflicts with the material presented in this report.
- 1
540 Gaither Road, Rockville, MD 20850; www
.ahrq.gov - 2
RAND Corporation, 1776 Main Street, Santa Monica, CA 90407
- NLM CatalogRelated NLM Catalog Entries
- Review Empirical Evidence of Associations Between Trial Quality and Effect Size[ 2011]Review Empirical Evidence of Associations Between Trial Quality and Effect SizeHempel S, Suttorp MJ, Miles JNV, Wang Z, Maglione M, Morton S, Johnsen B, Valentine D, Shekelle PG. 2011 Jun
- Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis.[Syst Rev. 2013]Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis.Hempel S, Miles JN, Booth MJ, Wang Z, Morton SC, Shekelle PG. Syst Rev. 2013 Nov 28; 2:107. Epub 2013 Nov 28.
- Review The Empirical Evidence of Bias in Trials Measuring Treatment Differences[ 2014]Review The Empirical Evidence of Bias in Trials Measuring Treatment DifferencesBerkman ND, Santaguida PL, Viswanathan M, Morton SC. 2014 Sep
- Review Low-Dose Aspirin for the Prevention of Morbidity and Mortality From Preeclampsia: A Systematic Evidence Review for the U.S. Preventive Services Task Force[ 2014]Review Low-Dose Aspirin for the Prevention of Morbidity and Mortality From Preeclampsia: A Systematic Evidence Review for the U.S. Preventive Services Task ForceHenderson JT, Whitlock EP, O'Conner E, Senger CA, Thompson JH, Rowland MG. 2014 Apr
- Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study.[Ann Intern Med. 2011]Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study.Dechartres A, Boutron I, Trinquart L, Charles P, Ravaud P. Ann Intern Med. 2011 Jul 5; 155(1):39-51.
- Detection of Associations Between Trial Quality and Effect SizesDetection of Associations Between Trial Quality and Effect Sizes
Your browsing activity is empty.
Activity recording is turned off.
See more...