Appendix A Table 3Quality rating criteria

DesignU.S. Preventive Services Task Force quality rating criteria70National Institute for Health and Clinical Excellence methodology checklists132
Systematic reviews and meta-analyses
  • Comprehensiveness of sources considered/search strategy used
  • Standard appraisal of included studies
  • Validity of conclusions
  • Recency and relevance are especially important for systematic reviews
  • The study addresses an appropriate and clearly focused question
  • A description of the methodology used is included
  • The literature search is sufficiently rigorous to identify all the relevant studies
  • Study quality is assessed and taken into account
  • There are enough similarities between the studies selected to make combining them reasonable
Case-control studies
  • Accurate ascertainment of cases
  • Nonbiased selection of cases/controls with exclusion criteria applied equally to both
  • Response rate
  • Diagnostic testing procedures applied equally to each group
  • Measurement of exposure accurate and applied equally to each group
  • Appropriate attention to potential confounding variables
  • The study addresses an appropriate and clearly focused question
  • The cases and controls are taken from comparable populations
  • The same exclusion criteria are used for both cases and controls
  • What percentage of each group (cases and controls) participated in the study?
  • Comparison is made between participants and non-participants to establish their similarities or differences
  • Cases are clearly defined and differentiated from controls
  • Is it clearly established that controls are non-cases?
  • Measures have been taken to prevent knowledge of primary exposure influencing case ascertainment
  • Exposure status is measured in a standard, valid and reliable way
  • The main potential confounders are identified and taken into account in the design and analysis
  • Have confidence intervals been provided?
Randomized controlled trials (RCTs)
  • Initial assembly of comparable groups employs adequate randomization, including first concealment and whether potential confounders were distributed equally among groups.
  • Maintenance of comparable groups (includes attrition, crossovers, adherence, contamination)
  • Important differential loss to follow-up or overall high loss to follow-up
  • Measurements: equal, reliable, and valid (includes masking of outcome assessment)
  • Clear definition of the interventions
  • All important outcomes considered
  • The study addresses an appropriate and clearly focused question
  • The assignment of subjects to treatment groups is randomized
  • An adequate concealment method is used
  • Subjects and investigators are kept ‘blind’ about treatment allocation
  • The treatment and control groups are similar at the start of the trial
  • The only difference between groups is the treatment under investigation
  • All relevant outcomes are measured in a standard, valid and reliable way
  • What percentage of the individuals or clusters recruited into each treatment arm of the study dropped out before the study was completed?
  • All the subjects are analyzed in the groups to which they were randomly allocated (often referred to as intention-to-treat analysis)
  • Where the study is carried out at more than one site, results are comparable for all sites
Cohort studies
  • Initial assembly of comparable groups employs consideration of potential confounders with either restriction or measurement for adjustment in the analysis; consideration of inception cohorts
  • Maintenance of comparable groups (includes attrition, crossovers, adherence, contamination)
  • Important differential loss to follow-up or overall high loss to follow-up
  • Measurements: equal, reliable, and valid (includes masking of outcome assessment)
  • Clear definition of the interventions
  • All important outcomes considered
  • The study addresses an appropriate and clearly focused question
  • The two groups being studied are selected from source populations that are comparable in all respects other than the factor under investigation
  • The study indicates how many of the people asked to take part did so, in each of the groups being studied
  • The likelihood that some eligible subjects might have the outcome at the time of enrollment is assessed and taken into account in the analysis
  • What percentage of individuals or clusters recruited into each arm of the study dropped out before the study was completed?
  • Comparison is made between full participants and those lost to follow-up, by exposure status
  • The outcomes are clearly defined
  • The assessment of outcome is made blind to exposure status
  • Where blinding was not possible, there is some recognition that knowledge of exposure status could have influenced the assessment of outcome
  • The measure of assessment of exposure is reliable
  • Evidence from other sources is used to demonstrate that the method of outcome assessment is valid and reliable
  • Exposure level or prognostic factor is assessed more than once
  • The main potential confounders are identified and taken into account in the design and analysis
  • Have confidence intervals been provided?
Diagnostic accuracy studies
  • Screening test relevant, available for primary care, adequately described
  • Study uses a credible reference standard, performed regardless of test results
  • Reference standard interpreted independently of screening test
  • Handles indeterminate result in a reasonable manner
  • Spectrum of patients included in study
  • Sample size
  • Administration of reliable screening test
  • The nature of the test being studied is clearly specified
  • The test is compared with an appropriate gold standard
  • Where no gold standard exists, a validated reference standard is used as a comparator
  • Patients for testing are selected either as a consecutive series or randomly, from a clearly defined study population
  • The test and gold standard are measured independently (blind) of each other
  • The test and gold standard are applied as close together in time as possible
  • Results are reported for all patients that are entered into the study
  • A pre-diagnosis is made and reported

Hierarchy of research design

I Properly conducted randomized controlled trial (RCT)

II-1: Well-designed controlled trial without randomization

II-2: Well-designed cohort or case-control analytic study

II-3: Multiple time series with or without the intervention; dramatic results from uncontrolled experiments

III: Opinions of respected authorities, based on clinical experience; descriptive studies or case reports; reports of expert committees

From: Appendix A, Detailed Methods

Cover of Effectiveness of Primary Care Interventions for Weight Management in Children and Adolescents
Effectiveness of Primary Care Interventions for Weight Management in Children and Adolescents: An Updated, Targeted Systematic Review for the USPSTF [Internet].
Evidence Syntheses, No. 76.
Whitlock EP, O'Conner EA, Williams SB, et al.

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.