• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jepicomhJournal of Epidemiology and Community HealthVisit this articleSubmit a manuscriptReceive email alertsContact usBMJ
J Epidemiol Community Health. Jun 1998; 52(6): 377–384.
PMCID: PMC1756728

The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions

Abstract

OBJECTIVE: To test the feasibility of creating a valid and reliable checklist with the following features: appropriate for assessing both randomised and non-randomised studies; provision of both an overall score for study quality and a profile of scores not only for the quality of reporting, internal validity (bias and confounding) and power, but also for external validity. DESIGN: A pilot version was first developed, based on epidemiological principles, reviews, and existing checklists for randomised studies. Face and content validity were assessed by three experienced reviewers and reliability was determined using two raters assessing 10 randomised and 10 non- randomised studies. Using different raters, the checklist was revised and tested for internal consistency (Kuder-Richardson 20), test-retest and inter-rater reliability (Spearman correlation coefficient and sign rank test; kappa statistics), criterion validity, and respondent burden. MAIN RESULTS: The performance of the checklist improved considerably after revision of a pilot version. The Quality Index had high internal consistency (KR-20: 0.89) as did the subscales apart from external validity (KR-20: 0.54). Test-retest (r 0.88) and inter-rater (r 0.75) reliability of the Quality Index were good. Reliability of the subscales varied from good (bias) to poor (external validity). The Quality Index correlated highly with an existing, established instrument for assessing randomised studies (r 0.90). There was little difference between its performance with non-randomised and with randomised studies. Raters took about 20 minutes to assess each paper (range 10 to 45 minutes). CONCLUSIONS: This study has shown that it is feasible to develop a checklist that can be used to assess the methodological quality not only of randomised controlled trials but also non-randomised studies. It has also shown that it is possible to produce a checklist that provides a profile of the paper, alerting reviewers to its particular methodological strengths and weaknesses. Further work is required to improve the checklist and the training of raters in the assessment of external validity.

 

Full Text

The Full Text of this article is available as a PDF (150K).

Selected References

These references are in PubMed. This may not be the complete list of references from this article.
  • Rosenberg W, Donald A. Evidence based medicine: an approach to clinical problem-solving. BMJ. 1995 Apr 29;310(6987):1122–1126. [PMC free article] [PubMed]
  • Black NA, Downs SH. The effectiveness of surgery for stress incontinence in women: a systematic review. Br J Urol. 1996 Oct;78(4):497–510. [PubMed]
  • Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995 Feb;16(1):62–73. [PubMed]
  • Peto R. Clinical trial reporting. Lancet. 1996 Sep 28;348(9031):894–895. [PubMed]
  • Gardner MJ, Machin D, Campbell MJ. Use of check lists in assessing the statistical content of medical studies. Br Med J (Clin Res Ed) 1986 Mar 22;292(6523):810–812. [PMC free article] [PubMed]
  • Colditz GA, Miller JN, Mosteller F. How study design affects outcomes in comparisons of therapy. I: Medical. Stat Med. 1989 Apr;8(4):441–454. [PubMed]
  • Sackett DL. Bias in analytic research. J Chronic Dis. 1979;32(1-2):51–63. [PubMed]
  • Lionel ND, Herxheimer A. Assessing reports of therapeutic trials. Br Med J. 1970 Sep 12;3(5723):637–640. [PMC free article] [PubMed]
  • BADGLEY RF. An assessment of research methods reported in 103 scientific articles from two Canadian medical journals. Can Med Assoc J. 1961 Jul 29;85:246–250. [PMC free article] [PubMed]
  • Thomson ME, Kramer MS. Methodologic standards for controlled clinical trials of early contact and maternal-infant behavior. Pediatrics. 1984 Mar;73(3):294–300. [PubMed]
  • Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. N Engl J Med. 1987 Feb 19;316(8):450–455. [PubMed]
  • DerSimonian R, Charette LJ, McPeek B, Mosteller F. Reporting on methods in clinical trials. N Engl J Med. 1982 Jun 3;306(22):1332–1337. [PubMed]
  • Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159–174. [PubMed]
  • Davis CE. Generalizing from clinical trials. Control Clin Trials. 1994 Feb;15(1):11–14. [PubMed]
  • Marubini E, Mariani L, Salvadori B, Veronesi U, Saccozzi R, Merson M, Zucali R, Rilke F. Results of a breast-cancer-surgery trial compared with observational data from routine practice. Lancet. 1996 Apr 13;347(9007):1000–1003. [PubMed]
  • Bailey KR. Generalizing the results of randomized clinical trials. Control Clin Trials. 1994 Feb;15(1):15–23. [PubMed]
  • Harth SC, Thong YH. Sociodemographic and motivational characteristics of parents who volunteer their children for clinical research: a controlled study. BMJ. 1990 May 26;300(6736):1372–1375. [PMC free article] [PubMed]
  • Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995 Feb 1;273(5):408–412. [PubMed]

Articles from Journal of Epidemiology and Community Health are provided here courtesy of BMJ Group

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • Cited in Books
    Cited in Books
    PubMed Central articles cited in books
  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...