Appendix D. USPSTF Hierarchy of research design and quality rating criteria 1, 2

Publication Details

Hierarchy of Research Design

  • I Properly conducted randomized controlled trial (RCT)
  • II-1: Well-designed controlled trial without randomization
  • II-2: Well-designed cohort or case-control analytic study
  • II-3: Multiple time series with or without the intervention; dramatic results from uncontrolled experiments
  • III: Opinions of respected authorities, based on clinical experience; descriptive studies or case reports; reports of expert committees

Design-Specific Criteria

Systematic Reviews

Quality rating criteria:

  • Comprehensiveness of sources considered/search strategy used
  • Standard appraisal of included studies
  • Validity of conclusions
  • Recency and relevance are especially important for systematic reviews

Definition of ratings from above criteria:

  • Good: Recent, relevant review with comprehensive sources and search strategies; explicit and relevant selection criteria; standard appraisal of included studies; and valid conclusions
  • Fair: Recent, relevant review that is not clearly biased but lacks comprehensive sources and search strategies.
  • Poor: Outdated, irrelevant, or biased review without systematic search for studies, explicit selection criteria, or standard appraisal of studies.

Randomized Controlled Trials and Cohort Studies

Quality rating criteria:

  • Initial assembly of comparable groups
    • -for RCTs: adequate randomization, including first concealment and whether potential confounders were distributed equally among groups.
    • -for cohort studies: consideration of potential confounders with either restriction or measurement for adjustment in the analysis; consideration of inception cohorts
  • Maintenance of comparable groups (includes attrition, cross-over, adherence, contamination)
  • Important differential loss to follow-up or overall high loss to follow-up
  • Measurements: equal, reliable, and valid (includes masking of outcome assessment)
  • Clear definition of the interventions
  • All important outcomes considered
  • Analysis: adjustment for potential confounders for cohort studies, or intention-to-treat analysis for RCTs

Definition of ratings from above criteria:

  • Good: Meets all criteria: Comparable groups are assembled initially and maintained throughout the study (follow-up at least 80 percent); reliable and valid measurement instruments are used and applied equally to the groups; interventions are spelled out clearly; all important outcomes are considered; and appropriate attention to confounders in analysis. In addition, for RCT's, intention to treat analysis is used.
  • Fair: Studies will be graded “fair” if any or all of the following problems occur, without the fatal flaws noted in the “poor” category below: Generally comparable groups are assembled initially but some question remains whether some (although not major) differences occurred with follow-up; measurement instruments are acceptable (although not the best) and generally applied equally; some but not all important outcomes are considered; and some but not all potential confounders are accounted for. Intention to treat analysis is done for RCT's.
  • Poor: Studies will be graded “poor” if any of the following fatal flaws exist: Groups assembled initially are not close to being comparable or maintained throughout the study; unreliable or invalid measurement instruments are used or not applied at all equally among groups (including not masking outcome assessment); and key confounders are given little or no attention. For RCT's, intention to treat analysis is lacking.

Reference List for Appendix D

1.
Harris R, Atkins D, Berg AO, Best D, Eden KB, Feightner JW et al. US Preventive Services Task Force Procedure Manual. Rockville, MD: Agency for Healthcare Research and Quality, 2001.
2.
Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM. et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med. 2001;20(3 Suppl):21–35. [PubMed: 11306229]