NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Chou R, Fu R, Carson S, et al. Empirical Evaluation of the Association Between Methodological Shortcomings and Estimates of Adverse Events. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006 Oct. (Technical Reviews, No. 13.)

Cover of Empirical Evaluation of the Association Between Methodological Shortcomings and Estimates of Adverse Events

Empirical Evaluation of the Association Between Methodological Shortcomings and Estimates of Adverse Events.

Show details


Every health care intervention is associated with a risk of harmful or adverse events that must be balanced against the potential favorable outcomes.1 Recent highly publicized examples of harms associated with medical interventions emphasize the importance of accurate measurement and reporting of harms.24 In studies that report harms, estimates of adverse events may vary because of differences in the underlying risk of the populations studied,5, 6 differences in the delivery of the intervention (such as the skill of the surgeon7 or dosing of medication), and random or systematic errors in adverse event assessment due to deficiencies in the design or execution of the study. Systematic reviews should therefore assess the generalizability of study findings to other clinical settings and populations, and evaluate for potential biases in assessment and reporting of harms.

Studies have consistently demonstrated that assessment and reporting of harms in clinical trials is suboptimal, with adverse events infrequently defined, unclear or inadequate methods for identifying adverse events, poor description of severity of harms, and little space devoted to adverse event reporting.812 Guidelines for improving harms reporting have been published to help address perceived shortcomings in measurement, analysis, and reporting of harms data.13 The Cochrane Collaboration has also proposed draft guidelines for evaluating adverse events in systematic reviews.14 However, little is known about the empirical associations between such perceived shortcomings and estimates of harms.15 Empiric evaluations have been published on the association between methodological shortcomings and estimates of efficacy1619 and diagnostic test characteristics.20 Similar information is necessary to help readers appropriately judge the validity of studies reporting harms data.

The purpose of our study was to empirically assess the association between perceived methodological shortcomings and estimates of serious complications from surgical and medical interventions. We compared estimates of harms from studies meeting eight pre-defined criteria designed to assess the quality of harms reporting to studies not meeting the criteria. We hypothesized that estimates of harms would be understated in studies with methodological shortcomings as measured by the quality criteria, even after controlling for other clinical and study design-related variables that could affect adverse event rates.

Our main analyses were performed on a large data set of studies of carotid endarterectomy (CEA) for symptomatic stenosis. Based on the analyses of this data set, we developed a generic summary quality-rating instrument for studies reporting harms. In order to assess the reproducibility of our results, we also analyzed a smaller data set of studies of the same intervention (CEA) in patients with asymptomatic stenosis. In addition, because the association between methodological shortcomings and estimates of harms may differ across interventions, we also evaluated a third data set consisting of studies of the drug rofecoxib, a cyclooxygenase-2 selector inhibitor recently withdrawn from the market because of concerns over increased cardiovascular risk.21


  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (381K)

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...