APPENDIX DCRITERIA USED IN QUALITY ASSESSMENT OF SYSTEMATIC REVIEWS

For reviews, first determine whether it is a systematic review. To be a systematic review, it must include a methods section that describes (1) a search strategy and (2) an a priori approach to synthesizing the data. For reviews determined to meet the systematic review criteria, assess methodological quality.*

General instructions: The purpose of this rating tool is to evaluate the scientific quality of systematic reviews. It is not intended to measure the literary quality, importance, relevance, originality, or other attributes of systematic reviews.

Step 1: Grade each criterion listed below as “Yes,” “No,” “Can’t tell” or “Not Applicable.” Factors to consider when making an assessment are listed under each criterion. Where appropriate (particularly when assigning a “No,” or “Can’t tell” score), please provide a brief rationale for your decision (in parentheses).

1. Is a focused clinical question clearly stated?
   At a minimum, the question should be developed a priori and should clearly identify population and outcomes. The study question does not have to be in PICO format (Population, Intervention, Comparisons, Outcomes.)
   [] Yes [] No [] Can’t tell [] N/A
2. Are the search methods used to identify relevant studies clearly described?
   Search methods described in enough detail to permit replication (The report must include search date, databases used, and search terms (Key words and/or MESH terms must be stated and where feasible the search strategy should be provided.)
   [] Yes [] No [] Can’t tell [] N/A
3. Was a comprehensive literature search performed?
   At least 2 electronic sources should be searched and electronic searches should be supplemented by consulting: reference lists from prior reviews, textbooks, or included studies; specialized registries (e.g., Cochrane registries); or queries to experts in the field.
   [] Yes [] No [] Can’t tell [] N/A
4. Was selection bias avoided?
   Study reports the number of studies identified through searches, the numbers excluded, and gives appropriate reasons for excluding – based on explicit inclusion/exclusion criteria.
   [] Yes [] No [] Can’t tell [] N/A
5. Was there duplicate study selection and data extraction?
   Did two or more raters make inclusion/exclusion decisions, abstract data, and assess study quality – either independently or with one rater over-reading the first raters result? Was an appropriate method used to resolve disagreements (e.g., a consensus procedure)?
   [] Yes [] No [] Can’t tell [] N/A
6. Were the characteristics of the included studies provided?
   In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in all the studies analyzed (e.g., age, race, sex, relevant socioeconomic data, disease status, duration, severity or other diseases) should be reported.
   [] Yes [] No [] Can’t tell [] N/A
7. Was the scientific quality of the included studies assessed and documented?
   A priori methods of assessment should be provided and criteria used to assess study quality specified in enough detail to permit replication.
   [] Yes [] No [] Can’t tell [] N/A
8. Were the methods used to combine the findings of studies appropriate?
   For pooled results, an accepted quantitative method of pooling should be used (i.e., more than simple addition; e.g., random-effects or fixed-effect model). For pooled results, a qualitative and quantitative assessment of homogeneity (Cochran’s Q and/or I²) should be performed. If only qualitative analyses are completed, the study should describe the reasons that quantitative analyses were not completed.
   [] Yes [] No [] Can’t tell [] N/A
9. Was the scientific quality of the included studies used appropriately in formulating conclusions?
   The results of the methodological rigor and scientific quality should be considered in the analysis (e.g. subgroup analyses) and the conclusions of the review, and explicitly stated in formulating recommendations.
   [] Yes [] No [] Can’t tell [] N/A
10. Was publication bias assessed?
    Publication bias tested using funnel plots, test statistics (e.g., Egger’s regression test), and/or search of trials registry for unpublished studies.
    [] Yes [] No [] Can’t tell [] N/A
11. Was the conflict of interest stated?
    Potential sources of support should be clearly acknowledged in both the systematic review and the included studies.
    [] Yes [] No [] Can’t tell [] N/A
12. Are the stated conclusions supported by the data presented?
    Were the conclusions made by the author(s) supported by the data and/or analyses reported in the systematic review?
    [] Yes [] No [] Can’t tell [] N/A

Step 2: Rate the overall quality of the SR as “Good,” “Fair,” or “Poor” using the guidance below.

Good = After considering items 1–12, item 12 is rated “Yes” with no important limitations. This means that few of the items 1–12 are rated “No,” and none of the limitations are thought to decrease the validity of the conclusions. If items 3, 4, 7, or 8 are rated “No,” then the review is likely to have major flaws

Fair = After considering items 1–12, item 12 is rated “Yes,” but with at least some important limitations. This means that enough of the items 1–12 are rated “No” to introduce some uncertainty about the validity of the conclusions.

Poor = After considering items 1–12, item 12 is rated “No.” This means that several of items 1–12 are rated “No,” introducing serious uncertainty about the validity of the conclusions.

*Adapted from:

1. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.
2. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354(9193):1896–900.
3. Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of continuing medical education. Evid Rep Technol Assess (Full Rep). 2007(149):1–69.

Table D-1 shows the quality ratings for the systematic reviews included in this evidence report.

Table D-1

Quality assessment for included systematic reviews.