Instruments assessing risk of bias of randomized trials frequently included items that are not addressing risk of bias issues

Ying Wang; Maryam Ghadimi; Qi Wang; Liangying Hou; Dena Zeraatkar; Atiya Iqbal; Cameron Ho; Liang Yao; Malini Hu; Zhikang Ye; Rachel Couban; Susan Armijo-Olivo; Dirk Bassler; Matthias Briel; Lise Lotte Gluud; Paul Glasziou; Rod Jackson; Sheri A Keitz; Luz M Letelier; Philippe Ravaud; Kenneth F Schulz; Reed A C Siemieniuk; Romina Brignardello-Petersen; Gordon H Guyatt

doi:10.1016/j.jclinepi.2022.10.018

Instruments assessing risk of bias of randomized trials frequently included items that are not addressing risk of bias issues

J Clin Epidemiol. 2022 Dec:152:218-225. doi: 10.1016/j.jclinepi.2022.10.018. Epub 2022 Oct 28.

Authors

Ying Wang¹, Maryam Ghadimi², Qi Wang³, Liangying Hou³, Dena Zeraatkar⁴, Atiya Iqbal², Cameron Ho⁵, Liang Yao², Malini Hu⁶, Zhikang Ye², Rachel Couban⁷, Susan Armijo-Olivo⁸, Dirk Bassler⁹, Matthias Briel¹⁰, Lise Lotte Gluud¹¹, Paul Glasziou¹², Rod Jackson¹³, Sheri A Keitz¹⁴, Luz M Letelier¹⁵, Philippe Ravaud¹⁶, Kenneth F Schulz¹⁷, Reed A C Siemieniuk², Romina Brignardello-Petersen², Gordon H Guyatt²

Affiliations

¹ Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada. Electronic address: yingwwy@163.com.
² Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada.
³ Department of Social Medicine and Health Management, School of Public Health, Lanzhou University, Lanzhou, China.
⁴ Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
⁵ Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, Ontario, Canada.
⁶ Michael G. DeGroote School of Medicine, McMaster University, Hamilton, Ontario, Canada.
⁷ Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada.
⁸ University of Applied Sciences, Faculty of Business and Social Sciences, Osnabrück, Germany; Faculty of Rehabilitation Medicine, Department of Physical Therapy, University of Alberta, Edmonton Canada.
⁹ Department of Neonatology, University Hospital Zurich, University of Zurich, Zurich, Switzerland.
¹⁰ Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada; Meta-Research Centre Basel, Department of Clinical Research, University Hospital Basel, Switzerland.
¹¹ Gastro Unit, Copenhagen University Hospital Hvidovre, Copenhagen, Denmark.
¹² Institute for Evidence-Based Healthcare, Bond University, Gold Coast, Queensland, Australia.
¹³ Section of Epidemiology & Biostatistics at the School of Population Health, Faculty of Medical and Health Sciences, University of Auckland, New Zealand.
¹⁴ Department of Medicine, Lahey Hospital & Medical Center, Burlington, MA, USA.
¹⁵ Department of Internal Medicine, Escuela de Medicina, Pontificia Universidad Catolica de Chile, Alameda 340, Santiago, Chile.
¹⁶ Epidemiology and Statistics Sorbonne Paris Cité Research Center (CRESS), INSERM, Université Paris Descartes, Paris, France.
¹⁷ School of Medicine, University of North Carolina at Chapel Hill, USA.

PMID: 36424692
DOI: 10.1016/j.jclinepi.2022.10.018

Abstract

Objectives: To establish whether items included in instruments published in the last decade assessing risk of bias of randomized controlled trials (RCTs) are indeed addressing risk of bias.

Study design and setting: We searched Medline, Embase, Web of Science, and Scopus from 2010 to October 2021 for instruments assessing risk of bias of RCTs. By extracting items and summarizing their essential content, we generated an item list. Items that two reviewers agreed clearly did not address risk of bias were excluded. We included the remaining items in a survey in which 13 experts judged the issue each item is addressing: risk of bias, applicability, random error, reporting quality, or none of the above.

Results: Seventeen eligible instruments included 127 unique items. After excluding 61 items deemed as clearly not addressing risk of bias, the item classification survey included 66 items, of which the majority of respondents deemed 20 items (30.3%) as addressing risk of bias; the majority deemed 11 (16.7%) as not addressing risk of bias; and there proved substantial disagreement for 35 (53.0%) items.

Conclusion: Existing risk of bias instruments frequently include items that do not address risk of bias. For many items, experts disagree on whether or not they are addressing risk of bias.

Keywords: Instrument; Methodological quality; Randomized controlled trials; Risk of bias; Systematic reviews; Systematic survey.

MeSH terms

Bias
Humans
Publications*
Randomized Controlled Trials as Topic