Appendix D. USPSTF Hierarchy of research design and quality rating criteria 1, 2

Publication Details

Hierarchy of Research Design

  • I Properly conducted randomized controlled trial (RCT)
  • II-1: Well-designed controlled trial without randomization
  • II-2: Well-designed cohort or case-control analytic study
  • II-3: Multiple time series with or without the intervention; dramatic results from uncontrolled experiments
  • III: Opinions of respected authorities, based on clinical experience; descriptive studies or case reports; reports of expert committees

Design-Specific Criteria

Systematic Reviews

Quality rating criteria:

  • Comprehensiveness of sources considered/search strategy used
  • Standard appraisal of included studies
  • Validity of conclusions
  • Recency and relevance are especially important for systematic reviews

Definition of ratings from above criteria:

  • Good: Recent, relevant review with comprehensive sources and search strategies; explicit and relevant selection criteria; standard appraisal of included studies; and valid conclusions
  • Fair: Recent, relevant review that is not clearly biased but lacks comprehensive sources and search strategies.
  • Poor: Outdated, irrelevant, or biased review without systematic search for studies, explicit selection criteria, or standard appraisal of studies.

Randomized Controlled Trials and Cohort Studies

Quality rating criteria:

  • Initial assembly of comparable groups
    • -for RCTs: adequate randomization, including first concealment and whether potential confounders were distributed equally among groups.
    • -for cohort studies: consideration of potential confounders with either restriction or measurement for adjustment in the analysis; consideration of inception cohorts
  • Maintenance of comparable groups (includes attrition, cross-over, adherence, contamination)
  • Important differential loss to follow-up or overall high loss to follow-up
  • Measurements: equal, reliable, and valid (includes masking of outcome assessment)
  • Clear definition of the interventions
  • All important outcomes considered
  • Analysis: adjustment for potential confounders for cohort studies, or intention-to-treat analysis for RCTs

Definition of ratings from above criteria:

  • Good: Meets all criteria: Comparable groups are assembled initially and maintained throughout the study (follow-up at least 80 percent); reliable and valid measurement instruments are used and applied equally to the groups; interventions are spelled out clearly; all important outcomes are considered; and appropriate attention to confounders in analysis. In addition, for RCT's, intention to treat analysis is used.
  • Fair: Studies will be graded “fair” if any or all of the following problems occur, without the fatal flaws noted in the “poor” category below: Generally comparable groups are assembled initially but some question remains whether some (although not major) differences occurred with follow-up; measurement instruments are acceptable (although not the best) and generally applied equally; some but not all important outcomes are considered; and some but not all potential confounders are accounted for. Intention to treat analysis is done for RCT's.
  • Poor: Studies will be graded “poor” if any of the following fatal flaws exist: Groups assembled initially are not close to being comparable or maintained throughout the study; unreliable or invalid measurement instruments are used or not applied at all equally among groups (including not masking outcome assessment); and key confounders are given little or no attention. For RCT's, intention to treat analysis is lacking.

Reference List for Appendix D

Harris R, Atkins D, Berg AO, Best D, Eden KB, Feightner JW et al. US Preventive Services Task Force Procedure Manual. Rockville, MD: Agency for Healthcare Research and Quality, 2001.
Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM. et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med. 2001;20(3 Suppl):21–35. [PubMed: 11306229]