Send to

Choose Destination
Environ Int. 2019 Sep;130:104884. doi: 10.1016/j.envint.2019.05.078. Epub 2019 Jul 9.

Development of outcome-specific criteria for study evaluation in systematic reviews of epidemiology studies.

Author information

U.S. Environmental Protection Agency, National Center for Environmental Assessment, United States. Electronic address:
U.S. Environmental Protection Agency, National Center for Environmental Assessment, United States.
Department of Environmental Health Sciences, Columbia University Mailman School of Public Health, United States.
Integrated Laboratory Systems, United States.
Department of Epidemiology and Biostatistics, University of Maryland School of Public Health, United States.
Department of Epidemiology, Tulane University School of Public Health and Tropical Medicine, United States.
Department of Global and Community Health, College of Health and Human Services, George Mason University, United States.
Department of Environmental Health, Harvard T. H. Chan School of Public Health, United States.
Department of Obstetrics and Gynecology, The Ohio State University College of Medicine, United States.
Division of Epidemiology, University of California Berkeley, United States.
Department of Child and Adolescent Psychiatry, Hassenfeld Children's Hospital at NYU Langone, United States.
U.S. Environmental Protection Agency, National Center for Environmental Assessment, United States; The Innocence Project, United States.



Systematic review tools that provide guidance on evaluating epidemiology studies are receiving increasing attention and support because their application facilitates improved quality of the review, consistency across reviewers, and transparency for readers. The U.S. Environmental Protection Agency's Integrated Risk Information System (IRIS) Program has developed an approach for systematic review of evidence of health effects from chemical exposures that includes structured approaches for literature search and screening, study evaluation, data extraction, and evidence synthesis and integration. This approach recognizes the need for developing outcome-specific criteria for study evaluation. Because studies are assessed at the outcome level, a study could be considered high quality for one investigated outcome, and low quality for another, due to differences in the outcome measures, analytic strategies, how relevant a certain bias is to the outcome, and how the exposure measure relates to the outcome. The objective of this paper is to illustrate the need for outcome-specific criteria in study evaluation or risk of bias evaluation, describe the process we used to develop the criteria, and summarize the resulting criteria.


We used a process of expert consultation to develop several sets of outcome-specific criteria to guide study reviewers, improve consistency, and ensure consideration of critical issues specific to the outcomes. The criteria were developed using the following domains: outcome assessment, exposure measurement (specifically timing of exposure in relation to outcome; other exposure measurement issues would be addressed in exposure-specific criteria), participant selection, confounding, analysis, and sensitivity (the study's ability to detect a true effect or hazard).


We discuss the application of this process to pregnancy-related outcomes (preterm birth, spontaneous abortion), other reproductive-related outcomes (male reproductive hormones, sperm parameters, time to pregnancy, pubertal development), chronic disease (diabetes, insulin resistance), and acute or episodic conditions (asthma, allergies), and provide examples of the criteria developed. For each outcome the most influential methodological considerations are highlighted including biological sample collection and quality control, sensitivity and specificity of ascertainment tools, optimal timing for recruitment into the study (e.g., preconception, specific trimesters), the etiologically relevant window for exposure assessments, and important potential confounders.


Outcome-specific criteria are an important part of a systematic review and will facilitate study evaluations by epidemiologists with experience in evaluating studies using systematic review methods who may not have extensive discipline-specific experience in the outcomes being reviewed.

Free full text

Supplemental Content

Full text links

Icon for Elsevier Science
Loading ...
Support Center