• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of plosmedPLoS MedicineSubmit to PLoSGet E-mail AlertsContact UsPublic Library of Science (PLoS)View this Article
PLoS Med. Nov 2008; 5(11): e230.
Published online Nov 25, 2008. doi:  10.1371/journal.pmed.0050230
PMCID: PMC2586359

Bias, Spin, and Misreporting: Time for Full Access to Trial Protocols and Results

Although randomized trials provide key guidance for how we practice medicine, trust in their published results has been eroded in recent years due to several high-profile cases of alleged data suppression, misrepresentation, and manipulation [1–5, 39]. While most publicized cases have involved pharmaceutical industry trials, accumulating empiric evidence has shown that selective reporting of results is a systemic problem afflicting all types of trials, including those with no commercial input [6]. These examples highlight the harmful potential impact of biased reporting on patient care, and the violation of ethical responsibilities of researchers and sponsors to disseminate results accurately and comprehensively.

Biased reporting arises when two main decisions are made based on the direction and statistical significance of the data—whether to publish the trial at all, and if so, which analyses and results to report in the publication. Strong evidence for the selective publication of positive trials has been available for decades [7,8]. More recent cohort studies have focused on the misreporting of trials within publications by comparing journal articles either with documents from regulatory agencies [9–12] or with trial protocols from research ethics committees [13–16], funding agencies [17], research groups [18,19], and journals [20]. These cohort studies identified major discrepancies—favorable results were often highlighted while unfavorable data were suppressed; definitions of primary outcomes were changed; and methods of statistical analysis were modified without explanation in the journal article.

Linked Research Article

This Perspective discusses the following new study published in PLoS Medicine:

Rising K, Bacchetti P, Bero L (2008) Reporting bias in drug trials submitted to the Food and Drug Administration: A review of publication and presentation. PLoS Med 5(11): e217. doi:10.1371/journal.pmed.0050217

Lisa Bero and colleagues review the publication status of all efficacy trials carried out in support of new drug approvals from 2001 and 2002, and find that a quarter of trials remain unpublished.

New Evidence

In a new study published in PLoS Medicine, Lisa Bero and colleagues make an important contribution to the growing body of evidence that the randomized trial literature is skewed towards reporting favorable results [9]. The researchers identified trials from 33 new drug applications (NDAs) for new molecular entities approved by the United States Food and Drug Administration (FDA) in 2001–2002, and compared information from FDA reviews with journal articles. By including all NDAs from a variety of specialty fields, their findings have broad generalizability to pharmaceutical trials.

Overall, a substantial amount of primary outcome data submitted to the FDA was found to be missing from the literature. One quarter of trials in their sample were unpublished—predominantly those with unfavorable results. Not only were data suppressed for the unpublished trials, but an additional quarter of primary outcomes were omitted from journal articles of published trials. These findings are consistent with two recent reviews of FDA documents and journal articles [10,21], one of which was published in PLoS Medicine in September 2008 [21].

Bero and colleagues also identified important discrepancies between the primary outcomes, statistical analyses, and conclusions presented in NDAs versus those reported in journal articles. The vast majority of discrepancies favored the sponsor's new drug, suggesting biased reporting. While it is possible that the FDA requested modifications to the sponsor's analyses, these amendments should be mentioned in the FDA's statistical review; should not involve altering primary outcomes without explanation in the publication; and would not be expected to favor the sponsor's drug as often as was found in this study.

Biased reporting of results from NDA trials is particularly concerning because these journal articles are the only peer-reviewed source of information on recently approved drugs for health care providers, who will have had limited clinical experience with these new treatments. There are also substantial cost implications if the efficacy is overestimated and the drugs overused, as new molecular entities are among the most expensive pharmaceuticals on the market [22].

The Need for Increased Transparency

Since the interests of patients are of utmost importance, it is difficult to justify why health care providers and policy makers should have access to only a biased subset of information that is substantially different from that which regulatory agencies have at their disposal. Bero and colleagues' study highlights the importance of public access to key documents that have traditionally been deemed confidential—regulatory agency submissions and trial protocols. Both types of documents have unique properties that complement each other.

Regulatory agency submissions represent the final description of how the trial was conducted and analyzed prior to journal publication. However, details from these submissions are not publicly available in most countries. Although summaries of FDA reviews are posted on the FDA Web site, their content and availability is variable, and sections are often redacted [9,21,23]. Furthermore, regulatory agency submissions are prepared by companies after data analysis and may themselves be subject to biased reporting. Finally, only devices, pharmaceuticals, and biological agents require regulatory approval in the United States and other countries, meaning that trials examining other types of interventions (e.g., surgery, education)—which constitute 20% of published randomized trials [24]—would be excluded from reviews of regulatory agency documents. Pharmaceutical trials conducted post-approval would also be missed.

On the other hand, protocols constitute the most comprehensive description of study design prior to trial inception. Their content therefore cannot be influenced by the study results. However, access to trial protocols is particularly difficult to obtain [25,26]. As with summaries of FDA reviews, their content is also highly variable and often lacks sufficient detail [13–18,20]. The SPIRIT initiative (Standard Protocol Items for Randomized Trials) aims to address these deficiencies by producing evidence-based recommendations for key information to include in a trial protocol [27].

Time for Action

It is clear that the trial literature is biased, facilitated in part by limited oversight and difficulty in accessing detailed trial documents. Ongoing progress in trial registration and results disclosure represents a key initial step towards ensuring public access to basic information on trial methods and results [28–33]. Several journals have also acted by publishing protocols and requiring their submission with manuscripts [34–36].

However, much remains to be done—not only to establish reliable, comprehensive registration and results disclosure processes worldwide, but also to start heeding the calls for increased access to full protocols and regulatory agency submissions [14,23,33,37,38]. As shown by recent examples and studies highlighted above, misreporting of trials can be difficult to detect without access to detailed documents beyond what is currently available on registries and results databases. Only with full transparency can the validity of a randomized trial be judged.

The time has come to tackle the challenge of making key trial documents public. It has taken decades for trial registration and results disclosure to be implemented; hopefully, for the sake of patients, public access to full protocols and regulatory agency submissions will come much sooner.

Acknowledgments

I am very grateful to Professor Doug Altman for his helpful comments on a previous version of this paper.

Glossary

Abbreviations

FDA
Food and Drug Administration
NDA
new drug application

Notes

Note Added in Proof

Reference 39 was added after this article was already in proof.

Footnotes

An-Wen Chan is at the Mayo Clinic, Rochester, Minnesota, United States of America. E-mail: ude.oyam@newna.nahc

Funding:The author received no specific funding for this article.

Competing Interests: A-WC worked previously as a scientist with the World Health Organization's International Clinical Trials Registry Platform and currently chairs the SPIRIT initiative referenced in this article.

Provenance: Commissioned; not externally peer reviewed

References

  • McHenry LB, Jureidini JN. Industry-sponsored ghostwriting in clinical trial reporting: A case study. Account Res. 2008;15:152–167. [PubMed]
  • Jureidini JN, McHenry LB, Mansfield PR. Clinical trials and drug promotion: Selective reporting of study 329. Int J Risk Safety Med. 2008;20:73–81.
  • Psaty BM, Kronmal RA. Reporting mortality findings in trials of rofecoxib for Alzheimer disease or cognitive impairment: A case study based on documents from rofecoxib litigation. JAMA. 2008;299:1813–1817. [PubMed]
  • Curfman GD, Morrissey S, Drazen JM. Expression of concern reaffirmed. N Engl J Med. 2006;354:1193. [PubMed]
  • Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, et al. Selective serotonin reuptake inhibitors in childhood depression: Systematic review of published versus unpublished data. Lancet. 2004;363:1341–1345. [PubMed]
  • Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan A-W, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE. 2008;3:e3081. doi: 10.1371/journal.pone.0003081. [PMC free article] [PubMed]
  • Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess. 2000;4:1–115. [PubMed]
  • Dickersin K. How important is publication bias? A synthesis of available data. AIDS Educ Prev. 1997;9:15–21. [PubMed]
  • Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: A review of publication and presentation. PLoS Med. 2008;5:e217. doi: 10.1371/journal.pmed.0050217. [PMC free article] [PubMed]
  • Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med. 2008;358:252–260. [PubMed]
  • Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine—Selective reporting from studies sponsored by pharmaceutical industry: Review of studies in new drug applications. BMJ. 2003;326:1171–1173. [PMC free article] [PubMed]
  • Hemminki E. Study of information submitted by drug companies to licensing authorities. BMJ. 1980;280:833–836. [PMC free article] [PubMed]
  • Chan A-W, Hróbjartsson A, Jørgensen KJ, Gøtzsche PC, Altman DG. Discrepancies in sample size calculations and data analyses reported in randomized trials: Comparison of publications with protocols. BMJ. 2008. In press. [PMC free article] [PubMed]
  • Chan A-W, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA. 2004;291:2457–2465. [PubMed]
  • Hahn S, Williamson PR, Hutton JL. Investigation of within-study selective reporting in clinical research: Follow-up of applications submitted to a local research ethics committee. J Eval Clin Pract. 2002;8:353–359. [PubMed]
  • Pildal J, Chan A-W, Hróbjartsson A, Forfang E, Altman DG, et al. Does unclear allocation concealment in trial publications reflect poor methods or poor reporting of adequate methods? Cohort study of trial protocols and corresponding published reports. BMJ. 2005;330:1049–1052. [PMC free article] [PubMed]
  • Chan A-W, Krleža-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171:735–740. [PMC free article] [PubMed]
  • Scharf O, Colevas AD. Adverse event reporting in publications compared with sponsor database for cancer clinical trials. J Clin Oncol. 2006;24:3933–3938. [PubMed]
  • Soares HP, Daniels S, Kumar A, Clarke M, Scott C, et al. Bad reporting does not mean bad methods for randomised trials: Observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004;328:22–24. [PMC free article] [PubMed]
  • Al-Marzouki S, Roberts I, Evans S, Marshall T. Selective reporting in clinical trials: Analysis of trial protocols accepted by The Lancet. Lancet. 2008;372:201. [PubMed]
  • Lee K, Bacchetti P, Sim I. Publication of clinical trials supporting successful new drug applications: A literature analysis. PLoS Med. 2008;5:e191. doi: 10.1371/journal.pmed.0050191. [PMC free article] [PubMed]
  • Morgan SG, Bassett KL, Wright JM, Evans RG, Barer ML, et al. “Breakthrough” drugs and growth in expenditure on prescription drugs in Canada. BMJ. 2005;331:815–816. [PMC free article] [PubMed]
  • Turner EH. A taxpayer-funded clinical trials registry and results database. PLoS Med. 2004;1:e60. doi: 10.1371/journal.pmed.0010060. [PMC free article] [PubMed]
  • Chan A-W, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365:1159–1162. [PubMed]
  • Chan A-W, Upshur R, Singh JA, Ghersi D, Chapuis F, et al. Research protocols: Waiving confidentiality for the greater good. BMJ. 2006;332:1086–1089. [PMC free article] [PubMed]
  • Lurie P, Zieve A. Sometimes the silence can be like the thunder: Access to pharmaceutical data at the FDA. Law Contemporary Problems. 2008;69:85–97.
  • Chan A-W, Tetzlaff J, Altman DG, Gøtzsche PC, Hróbjartsson A, et al. The SPIRIT initiative: Defining Standard Protocol Items for Randomized Trials [conference abstract] German J Evid Quality Health Care (suppl) 2008;102:S27.
  • World Health Organization. International Clinical Trials Registry Platform. 2008. Available: http://www.who.int/ictrp/en/. Accessed 20 October 2008.
  • The PLoS Medicine Editors. Next stop, don't block the doors: Opening up access to clinical trials results. PLoS Med. 2008;5:e160. doi: 10.1371/journal.pmed.0050160. [PMC free article] [PubMed]
  • United States Congress. Food and Drug Administration Amendments Act of 2007, Title VIII, Section 801. Expanded clinical trial registry data bank. 2007. Available: http://www.govtrack.us/congress/billtext.xpd?bill=h110-3580. Accessed 20 October 2008.
  • Sim I, Chan A-W, Gulmezoglu AM, Evans T, Pang T. Clinical trial registration: Transparency is the watchword. Lancet. 2006;367:1631–1633. [PubMed]
  • DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, et al. Is this clinical trial fully registered?: A statement from the International Committee of Medical Journal Editors. JAMA. 2005;293:2927–2929. [PubMed]
  • Krleža-Jeric K, Chan A-W, Dickersin K, Sim I, Grimshaw J, et al. Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1) BMJ. 2005;330:956–958. [PMC free article] [PubMed]
  • PLoS Medicine. Guidelines for authors. 2008. Available: http://journals.plos.org/plosmedicine/guidelines.php#supporting. Accessed 20 October 2008.
  • McNamee D, James A, Kleinert S. Protocol review at The Lancet. Lancet. 2008;372:189–190. [PubMed]
  • Jones G, Abbasi K. Trial protocols at the BMJ. BMJ. 2004;329:1360. [PMC free article] [PubMed]
  • Lassere M, Johnson K. The power of the protocol. Lancet. 2002;360:1620–1622. [PubMed]
  • Hawkey CJ. Journals should see original protocols for clinical trials. BMJ. 2001;323:1309. [PMC free article] [PubMed]
  • Mitka M. Controversies surround heart drug study: Questions about Vytorin and trial sponsors' conduct. JAMA. 2008;299:885–887. [PubMed]

Articles from PLoS Medicine are provided here courtesy of Public Library of Science
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...