Send to

Choose Destination
CMAJ Open. 2016 Apr 6;4(2):E132-40. doi: 10.9778/cmajo.20150036. eCollection 2016 Apr-Jun.

Current use of routinely collected health data to complement randomized controlled trials: a meta-epidemiological survey.

Author information

Stanford Prevention Research Center, Department of Medicine (Hemkens, Ioannidis), Stanford University School of Medicine, Stanford, Calif.; Basel Institute for Clinical Epidemiology and Biostatistics (Hemkens), University Hospital Basel, Basel, Switzerland; Department of Pediatrics, Division of Infectious Diseases (Contopoulos-Ioannidis), Stanford University School of Medicine, Stanford, Calif.; Palo Alto Medical Foundation Research Institute (Contopoulos-Ioannidis), Palo Alto, Calif.; Department of Health Research and Policy (Ioannidis), Stanford University School of Medicine; Meta-Research Innovation Center at Stanford (METRICS) (Ioannidis), Stanford, Calif.



Studies that use routinely collected health data (RCD studies) are advocated to complement evidence from randomized controlled trials (RCTs) for comparative effectiveness research and to inform health care decisions when RCTs would be unfeasible. We aimed to evaluate the current use of routinely collected health data to complement RCT evidence.


We searched PubMed for RCD studies published to 2010 that evaluated the comparative effectiveness of medical treatments on mortality using propensity scores. We identified RCTs of the same treatment comparisons and evaluated how frequently the RCD studies analyzed treatments that had not been compared previously in randomized trials. When RCTs did exist, we noted the claimed motivations for each RCD study. We also analyzed the citation impact of the RCD studies.


Of 337 eligible RCD studies identified, 231 (68.5%) analyzed treatments that had already been compared in RCTs. The study investigators rarely claimed that it would be unethical (6/337) or difficult (18/337) to perform RCTs on the same question. Evidence from RCTs was mentioned or cited by authors of 213 RCD studies. The most common motivations for conducting the RCD studies were alleged limited generalizability of trial results to the "real world" (37.6%), evaluation of specific outcomes (31.9%) or specific populations (23.5%), and inconclusive or inconsistent evidence from randomized trials (25.8%). Studies evaluating "real world" effects had the lowest citation impact.


Most of the RCD studies we identified explored comparative treatment effects that had already been investigated in RCTs. The objective of such studies needs to shift more toward answering pivotal questions that are not supported by trial evidence or for which RCTs would be unfeasible.

Supplemental Content

Full text links

Icon for HighWire Icon for PubMed Central
Loading ...
Support Center