Chapter  47:  Systems to Rate the Strength Of Scientific Evidence: Evidence Report/Technology Assessment Number 47

Data Collection

We reviewed the titles and abstracts for a total of 1,602 publications for this project. From this set, we retained 109 sources that dealt with systems (i.e., scales, checklists, or other types of instruments or guidance documents) pertinent to rating the quality of individual systematic reviews, RCTs, observational studies, or investigations of diagnostic tests, or with systems for grading the strength of bodies of evidence. In addition, we reviewed 12 reports from various AHRQ-supported EPCs. In all, we considered 121 systems as the basis for this report.

Specifically, we assessed 20 systems relating to systematic reviews, 49 systems for RCTs, 19 for observational studies, and 18 for diagnostic test studies. For final evaluative purposes, we focused on scales and checklists. In addition, we reviewed 40 systems that addressed grading the strength of a body of evidence (34 systems identified from our searches and prior research and 6 from various EPCs). The systems reviewed totals more than 121 because several were reviewed for more than one grid.

Systems for Rating the Quality of Individual Articles

Systems for Grading the Strength of a Body of Evidence

We reviewed 40 systems that addressed grading the strength of a body of evidence: 34 from sources other than AHRQ EPCs and 6 from the EPCs. Our evaluation criteria involved three domains -- quality, quantity, and consistency (Box C) -- that are well-established variables for characterizing how confidently we can conclude that a body of knowledge provides information on which clinicians or policymakers can act.

The 34 non-EPC systems incorporated quality, quantity, and consistency to varying degrees. Seven systems fully addressed the quality, quantity, and consistency domains.^11,81-86 Nine others incorporated the three domains at least in part.^{12,14,39,70,87-91}

Of the six EPC grading systems, only one incorporated quality, quantity, and consistency.⁹³ Four others included quality and quantity either fully or partially.^{59, 60,67,68} The one remaining EPC system included quantity; study quality is measured as part of its literature review process, but this domain appears not to be directly incorporated into the grading system.⁶⁶

Identification of Systems

We identified 1,602 articles, reports, and other materials from our literature searches, web searches, referrals from our technical expert advisory group, suggestions from independent peer reviewers of an earlier version of this report, and a previous project conducted by the RTI-UNC EPC. In the end, our formal literature searches were the least productive source of systems for this report. Of the more than 120 systems we eventually reviewed that dealt with either quality of individual articles or strength of bodies of evidence, the searches per se generated a total of 30 systems that we could review, describe, and evaluate. Many articles from the searches related to study quality were essentially reports of primary studies or reviews that discussed "the quality of the data"; few addressed evaluating study quality itself.

Our literature search was most problematic for identifying systems to grade the strength of a body of evidence. Medical Subject Headings (MeSH) terms were not very sensitive for identifying such systems or instruments. We attribute this phenomenon to the lag in development of MeSH terms specific for the evidence-based medicine field.

For those involved in evidence-based practice and research, we caution that they may not find it productive simply to search for quality rating or evidence grading schemes through standard (systematic) literature searches. This is one reason that we are comfortable with identifying a set of instruments or systems that meet reasonably rigorous standards for use in rating study quality and grading bodies of evidence. Little is to be gained by directing teams seeking to produce systematic reviews or technology assessments (or indeed clinical practice guidelines) to initiate wholly new literature searches in these areas.

At the moment, we cannot provide concrete suggestions for efficient search strategies on this topic. Some advances must await expanded options for coding the peer-reviewed literature. Meanwhile, investigators wishing to build on our efforts might well consider tactics involving citation analysis and extensive contact with researchers and guideline developers to identify the rating systems they are presently using. In this regard, the efforts of at least some AHRQ-supported EPCs will be instructive.

Factors Important in Developing and Using Rating Systems

Defining Quality

Critical to this discussion is the definition of quality, which authors of quality ratings often do not specify. In the previous AHRQ project, Lohr and Carey defined "methodologic quality" as "the extent to which all aspects of a study's design and conduct can be shown to protect against systematic bias, nonsystematic bias, and inferential error"; "nonmethodologic quality" refers to "the extent to which the information from a study has significant clinical or policy relevance." (Ref. 1, p. 472.)

We focus in this report on methodologic quality -- that is, the extent to which a study's design, conduct, and analysis has minimized selection, measurement, and confounding biases. Our definition of quality refers to the internal validity of a study, not its external validity or generalizability. Although not all experts in the evidence-based practice field would take this approach, we consider issues of generalizability more relevant for developing clinical practice guidelines than for producing rigorous systematic reviews per se, in that guideline development is a step that occurs after a body of evidence on a clinical topic has been assembled and the overall strength of that evidence assessed.⁹⁸

Developing Quality Assessments

Our project's first specific aim was to identify and compare tools to conduct quality assessment, which includes quality scales and checklists but also individual components of study quality. As part of the comparison among quality assessment instruments, it is important to understand proper scale development techniques.

Measurement scales, which in this context would include quality rating instruments, are developed in a stepwise fashion. The first steps involve defining the quality constructs or issues to be measured and the scope and purpose of the quality review, after which the actual questions are developed. The final steps require testing the instrument's reliability and validity and making such modifications as seem appropriate to meet conventional standards (for example, for internal consistency or test-retest reliability).⁹⁹

Typically, instrument developers examine three types of validity: face, content, and criterion validity. Asking whether the instrument appears to measure what it was intended to measure assesses face validity. The extent to which the quality domain of interest -- for example, randomization -- is comprehensively assessed measures content validity. Criterion validity is defined as the extent to which the measurement correlates with an external criterion variable (a "gold standard"),¹⁰⁰ preferably one that can be measured objectively and independently. Khan and colleagues suggest that the last step in this iterative process is to determine the measurement properties of the quality instrument.¹² For many types of instruments, researchers measure criterion validity if an acceptable gold standard is available. Quality assessment tools of the type under consideration in this report have no true, "objective" gold standard that lies outside the domain of subjective assessment of the "goodness" of the study or article at hand. Because criterion validity cannot be assessed, some scale developers assess the instrument's reliability -- that is, measuring whether a similar quality assessment score can be derived on the same study using either different scales or assessors (inter-rater reliability).¹² The rigorousness with which a scale is developed may influence its measurement properties.

Using Quality Ratings

No consensus exists on how study quality should be used in a systematic review. Moher, Jadad, and Tugwell (1996) describe four ways that quality assessment of RCTs may be used in systematic reviews and meta-analyses.¹⁰¹ The most basic approach is to use quality as an inclusion threshold. Many reviewers, for example, admit only RCTs into a systematic review and eliminate other study designs from further consideration. Others have used a numeric quality score as a statistical weight when conducting meta-analyses (i.e., quantitative systematic reviews) to calculate a summary estimate of effect. A third method involves conducting a cumulative meta-analysis that is initiated by including only the higher quality studies and then adding studies of lesser quality sequentially. Finally, some have recommended that quality be examined visually in a plot.

Experts also disagree about whether quality should be formally scored, used as a threshold for inclusion or exclusion, employed in sensitivity analysis, applied in some other analytic framework, simply described, or not considered at all. Each approach has some potential advantages or some serious problems. If quality is to be used as a threshold for inclusion or exclusion, how quality is determined matters.² In systematic reviews of treatment, for instance, including a very poor quality study, regardless of its size, can profoundly influence summary estimates of the effects of that treatment.^41,97

Complex statistical challenges arise when reviewers are attempting to arrive at a quantitative summary rather than attempting to conduct a more narrative review. Work by Detsky et al., Olkin, Moher et al., Sutton et al., and Tritchler is particularly helpful in guiding reviewers about the salient issues and statistical techniques involved.^{20,30,101-103}

Study Quality

Systematic reviews comprise evidence based on research papers from the published literature and, whenever possible, from the "gray" or unpublished literature as well. Although much of the material identified and used for systematic reviews is from the peer-reviewed literature, the process and thoroughness of review conducted by journal reviewers and by those doing systematic reviews may not be the same. Thus, the literature compiled for systematic reviews may be of varying quality, which can lead to conflicting summary estimates between systematic reviews on the same topic.

For example, Juni and colleagues evaluated study quality of 17 RCTs comparing low molecular weight heparin with standard heparin for preventing post-operative thrombosis using 25 different quality scales. Among the scales, both the indicators of study quality and their corresponding weights differed such that a study considered to be high quality on one scale was deemed low quality on another. Thus, summarizing the high quality articles according to one scale produced different relative risks than summarizing high quality studies using another scale.² Juni et al. found that an important predictor of summary relative risk was one particular component of study quality, whether the assessor of the outcome (risk of thrombosis) was masked to treatment allocation. This suggests that evaluating study quality is dependent on particular study design issues relevant to the topic under study. Their finding that a focus on methodologic components rather than summary scores to measure quality supports other work and editorial comment in the field.¹⁰⁴

As discussed in more detail below, several published studies provide empirical evidence that inadequate description of certain elements of experimental study design -- namely randomization procedures, allocation concealment (in which investigators do not know which drug will be assigned next), and outcome masking -- have been associated with biased results.^2,51,105,106Failure to mask the randomization procedures or outcome assessment was associated with elevated estimates of treatment effect compared with studies that reported using adequate masking procedures.

Whether potential design deficiencies in the published studies are the result of poor study design or poor reporting of study design is difficult to evaluate because reviewers typically see only the study report. Several collaborative efforts have put forth "statements" to standardize reporting; these include publishing guidelines for systematic reviews (QUOROM),²¹ RCTs (CONSORT),⁵⁷ and observational studies (MOOSE).²³ These guidelines appear as checklists that authors can use to ensure that they have adequately addressed all the necessary components of a systematic review or publication on a given clinical or health services research project. Because of the evidence that poor quality studies may bias summary estimates from systematic reviews, researchers have developed and incorporated study quality assessment into their procedures for abstracting information from the literature and then describing and evaluating that literature. Numerous quality rating checklists and scales exist for RCTs.^101,107 Few instruments have been developed specifically for systematic reviews, observational studies, or investigations of diagnostic tests; however, most of those pertinent for observational studies of treatment effects are general enough to evaluate RCTs. Among existing instruments, even fewer scales and checklists have been developed using rigorous scale development techniques.

In this project, we compared and contrasted quality rating approaches using the definition of quality offered above, which is based on study design characteristics indicative of methodologic rigor. As explained in Chapter 2, Methods, we developed the grids for evaluating study quality using domains or specific items from various sources that described study quality or that discussed epidemiologic design standards. Some domains include explicit case definition specification, treatment allocation, control of confounding, extensiveness of follow-up, standardized and reproducible outcome assessment methods, and appropriate statistical analysis. Because design standards differ by study types (e.g., RCTs, observational studies, systematic reviews, and diagnostic studies), we developed one grid for each of these design types.

Strength of a Body of Evidence

Figure 2. Continuum from Study Quality Through Strength (more...)

   Figure 2. Continuum from Study Quality Through Strength of Evidence to Guideline Development

The dashed line is the theoretical dividing line between summarizing the scientific literature and developing a clinical practice guideline. Below the dashed line, guideline developers would decide whether the evidence represents all the relevant subsets of the populations (or settings, or types of clinicians) for whom the guideline is being developed.

Figure 2

Since the appearance of the Surgeon General's Report on Smoking and Health in 1964,^148,108 epidemiologists have been using five criteria for assessing causal relationships.¹⁰⁹ Two criteria, consistency and strength, are of particular relevance. Consistency is the extent to which diverse approaches, such as different study designs or populations, for studying a relationship or link between a factor and an outcome will yield similar conclusions. Strength is the size of the estimated risk (of disease due to a factor) and its accompanying confidence intervals. Both of these concepts are directly related to grading the strength of a body of evidence.

Other epidemiologic criteria such as coherence, which examines whether the cause-and-effect interpretation for an association conflicts with what is known of the natural history and biology of the disease, are more relevant for developing clinical recommendations. The remaining two causality criteria typically used in epidemiology, specificity and temporality, are more appropriate for measuring risk than for conducting technology assessments.

Based on these epidemiologic principles, the literature, and prior RTI-UNC EPC work, we concluded that grading the strength of a body of evidence should take three domains into account: quality, quantity, and consistency. Quality is defined as above, but in this case we are concerned with the quality of all relevant studies for a given topic. Quantity encompasses several aspects such as the number of studies that have evaluated the question, the overall sample size across all of the studies, and the magnitude of the treatment effect. Quantity is along the lines of "strength" from causality assessment and is typically reported in a comparative sense as a mean difference, relative risk, or odds ratio. Consistency -- that is, whether investigations with both similar and different study designs report similar findings -- can be assessed only if numerous studies are done. Thus, consistency is an important consideration when comparing one study with many individuals to several smaller studies with few individuals. We contend that one needs to address all three factors -- quality, quantity, and consistency -- when grading the strength of the evidence.

Preliminary Steps

We carried out a multi-part effort to identify rating and grading systems and literature relevant to this question in several ways. First, we resurrected all documents acquired or generated in the original "grading project," including literature citations or other materials provided by the EPCs.¹ Second, as described in more detail below, we designed a supplemental literature search to identify articles that focused on generic instruments published in English (chiefly from 1995 through mid-2000). Third, we used information from the EPC directors documenting the rating scales and classification systems that they have used in evidence reports or other projects for AHRQ or other sponsors. Fourth, we examined rating schemes or similar materials forwarded by TEAG members.

In addition, we tracked activities of several other groups engaged in examining these same questions. These include The Cochrane Collaboration Methods Group (especially work on assessing observational studies), the third (current) U.S. Preventive Services Task Force, and the Scottish Intercollegiate Guidelines Network (SIGN).

Finally, we reviewed the following international web sites for groups involved in evidence-based medicine or guideline development:
Canadian Task Force on Preventive Health Care (Canada), http://www.ctfphc.org/.
Centre for Evidence Based Medicine, Oxford University (U.K.), http://cebm.jr2.ox.ac.uk/;
National Coordination Centre for Health Technology Assessment (U.K.), http://www.ncchta.org/main.htm;
National Health and Medical Research Council (Australia), http://www.nhmrc.health.gov.au/index.htm;
New Zealand Guidelines Group (New Zealand), http://www.nzgg.org.nz/; and
National Health Service (NHS) Centre for Reviews and Dissemination (U.K.), http://www.york.ac.uk/inst/crd/;
Scottish Intercollegiate Guidelines Network (SIGN) (U.K.), http://www.sign.ac.uk/;
The Cochrane Collaboration (international), http://www.cochrane.org/;

Searches

We searched the MEDLINE^® database for relevant articles published between 1995 and mid-2000 using the Medical Subject Heading (MeSH) terms shown in Tables 2 and 3 for Grids 1-4 (on rating the quality of individual studies) and Grid 5 (on grading a body of scientific evidence), respectively. For the Grid 5 search, we also had to use text words (indicated by an ".mp.") to make the search as inclusive as possible.

We compiled the results from all searches into a ProCite^® bibliographic database, removing all duplicate records. We also used this bibliographic software to tag eligible articles and, for articles determined to be ineligible, to note the reason for their exclusion.

Title and Abstract Review

Table 2. Systematic Search Strategy to Identify Instruments (more...)

Table 2. Systematic Search Strategy to Identify Instruments for Assessing Study Quality

	Search Strategy	Results
1	*Meta-analysis	895
2	*Randomized controlled trials/mt [Methods]	512
3	Systematic reviews.mp.	307
4	1 or 2 or 3	1,645
5	Limit 4 to (human and English language and year = 1995-2000)	858
6	Explode evidence-based medicine/ or explode quality control/ or explode reproducibility of results/ or explode data interpretation, statistical/ or explode "sensitivity and specificity"/ or explode research design/ or explode practice guidelines/	278,544
7	Explode guidelines/	13,969
8	Explode 8 (measurement scales or confidence profile or procedural methodology or study quality or study influence or effect measures).mp.	4,589
9	6 or 7 or 8	28,7281
10	5 and 9	704

*This term must be one of the four most important MeSH terms for the record.

Table 3. Systematic Search Strategy to Identify Systems (more...)

Table 3. Systematic Search Strategy to Identify Systems for Grading the Strength of a Body of Evidence

	Search Strategy	Results
1	Explode evidence-based medicine/ or evidence.mp.	374,101
2	Strength or rigor or standards or authority or validity.mp.	111,824
3	1 and 2	7,323
4	*Randomized controlled trials/st [Standards]	308
5	3 or 4	7,621
6	Limit 5 to (human and English language and year = 1995-2000)	2,586
7	Grading.mp.	9,238
8	Explode observer variation/	8,697
9	Explode reproducibility of results/	52,017
10	Explode sensitivity and specificity/	87,492
11	Ranking.mp.	3,241
12	7 or 8 or 9 or 10 or 11	144,265
13	6 and 12	679†

*This term must be one of the four most important MeSH terms for the record.

† The figure of 679 articles identified excludes two publications that had been identified and counted for both the Study Quality and Evidence Strength literature searches.

Table 4. Coding System Applied at the Abstract Stage (more...)

Table 4. Coding System Applied at the Abstract Stage for Articles Identified During the Focused Literature Search for Study Quality Grid and for Strength of Evidence Grid

Codes Used for Both Study Quality Grids and Strength of Evidence Grids	Definition
Include	Obtain the full paper to assess whether it contains useful information for either grid
Ref-back	Reference or background paper to be obtained
Exclusions
ECL	Editorial, comment, or letter
NR	Not relevant, requires specification of reason from below
NR-design/methods	Design or methodological issues, typically about clinical studies
NR-IA	Implementation/Application (e.g., described use of recommendations or guidelines in a clinical setting)
NR-OCD	Opinion/Commentary/Description (e.g., midway between ECL and review)
NR-ROS	Report of Study (e.g., report of a meta-analysis or clinical study)
NR-Review	Review/Overview (e.g., typically a narrative review of a clinical topic)
NR-Stat Meth	Statistical methodology (e.g., for conducting a meta-analysis)
NR-Other	Other reason for nonrelevance (e.g., continuing education, computer modeling systems)
Additional Code for Strength of Evidence Grid
NR-Text word only (TWO)	Studies identified by text word but not relevant for inclusion (e.g., title or abstract had "evidence" or "recommend" as part of the text)

We identified an additional 219 publications from various sources other than the formal searches, including the previous project,¹ bibliographies of seminal articles, suggestions from TEAG members, and searches of the web pages of groups working on similar issues (listed above). In all, we reviewed the abstracts for a total of 1,602 publications for the project; after review of all retained articles, we retained 109 that dealt with systems (i.e., scales, checklists, or other types of instruments or guidance documents) that were included in one or more of the grids and 12 EPC systems, for a total of 121 systems. The two-stage selection process that yielded these 121 systems is available from the authors on request.

Number and Structure of Grids

We developed the four Study Quality Grids (Appendix B) to account for four different study designs -- systematic reviews and meta-analyses, randomized controlled trials (RCTs), observational studies, and diagnostic studies.

Each Study Quality Grid has two parts. The first depicts the quality constructs and domains that each rated instrument covers; the other describes the instrument in various ways. For both Grids 1-4 (and Grid 5), columns denote evaluation domains of interest, and the rows are the individual systems, checklists, scales, or instruments. Taking these parts together, the grids form "evidence tables" that document the characteristics (strengths and weaknesses) of these different systems.

Overview of Grid Development

Preliminary Steps

Table 5. Study Constructs Believed to Affect Quality (more...)

Table 5. Study Constructs Believed to Affect Quality of Studies

Constructs	Definition
Selection of patients	Who was included and who was excluded Health, demographic, insurance, and other characteristics of these subjects Diagnostic and/or prognostic criteria used
Comparability of study groups	Randomization and allocation of patients to treatment and control/comparison groups Similarity at baseline of these groups
Blinding	Masking of patients, investigators, care providers, those who assessed outcomes to treatment groups or outcomes (or both)
Adequate sample size	Size of the study A priori justification of sample size Consequent power
Therapeutic regimen	Detailed information about the treatment, the settings in which the services were delivered, and the clinicians who delivered them Description of co-interventions Description of extra or unplanned treatments
Outcomes	Choice of primary and secondary endpoints or outcomes Ways the outcomes are measured
Availability of a study protocol	Study administration, including length of follow-up period
Handling of withdrawals after eligibility determination	Withdrawals, drop-outs, or other losses from the study, by patient group
Threats to validity	Confounders and bias and how they are accounted for
Statistical analyses	Appropriateness of statistical models Adequacy of description and reporting of statistical analyses Reporting levels of significance and/or confidence intervals Extent to which all analyses that should have been done were done "Intention-to-treat" analysis

1

Source: Adapted from Lohr and Carey (1999).

Previous work done by the RTI-UNC EPC had identified constructs believed to affect the quality of studies (Table 5).¹ Beginning with these constructs and an annotated bibliography of scales and checklists for assessing the quality of RCTs,^101,107 we examined several of the more comprehensive systems of assessing study quality to settle on appropriate domains to use in the grids. These included approaches from groups such as the New Zealand Guidelines Group,¹³ The Cochrane Collaboration,¹¹ the NHS Centre for Reviews and Dissemination,⁸⁵ and SIGN.¹⁴ After three rounds of design, review, and testing, we settled on the domains and elements outlined in tables discussed below.

Table 6. Items Used to Describe Instruments to Assess (more...)

Table 6. Items Used to Describe Instruments to Assess Study Quality

Descriptive Item*	Definitions of Descriptive Items
Generic or specific instrument	Generic: Instrument could be used to assess quality of any study of the type considered on that grid. Specific: Instrument is designed to be used to assess study quality for a particular type of outcome, intervention, exposure, test, etc.
Type of instrument	Scale: Instruments that contain several quality items that are scored numerically to provide a quantitative estimate of overall study quality. Checklist: Instruments that contain a number of quality items, none of which is scored numerically. Component: Individual aspect of study methodology (e.g., randomization, blinding, follow-up) that has a potential relation to bias in estimation of effect. Guidance Publication in which study quality is defined or Document: described, but does not provide an instrument that could be used for evaluative applications.
Quality concept discussion	Yes: Types or domains of quality that the instrument is designed to capture are discussed (e.g., biases that might affect the internal validity of the study). Partial: Quality concepts are discussed to some extent. No: Instrument itself or its documentation does not discuss the type or domains of study quality it assesses.
Method used to select items	Empiric: Items are based on criteria developed through empirical studies. Accepted: Items are based on accepted methodologic standards. Both: Items are of mixed empiric and accepted origin. Modification: The instrument represents a modification of another previously published instrument(s); original instrument is cited.
Rigor of development process	Yes: The use of standard scale development metrics in developing the instrument is explicitly described. Partial: The instrument was developed using an organized and reported consensus development process. No: No development process is reported or described.
Inter-rater reliability	Yes: Inter-rater reliability was assessed with appropriate statistical methods; results are reported in the grid. Partial: Issues concerning inter-rater reliability are discussed but the degree or range of reliability is not reported. No: Inter-rater reliability is not mentioned.
Instructions provided	Yes: Documentation of how to use and apply the instrument is adequate. Partial: Documentation of how to use the instrument is available in part (e.g., the questions on a checklist were clear and did not require substantial interpretation). No: Instrument did not provide instructions to guide its use.

*These items appear as column headings in the Study Quality and Evidence Strength Grids in Appendices B and C.

In addition to abstracting and assessing the content of quality rating instruments and systems, we gathered information on seven descriptive items for each article (Table 6). Definitions of key terms used in Table 6 appear in the glossary (Appendix G). These items, which were identical for all four study types, cover the following characteristics:

1. Whether the instrument was designed to be generic or specific to a given clinical topic;

2. The type of instrument (a scale, a checklist, or a guidance document);

3. Whether the instrument developers defined quality;

4. What method the instrument developers used to select items in the instrument;

5. The rigor of the development process for this instrument;

6. Inter-rater reliability; and

7. Whether the developers had provided instructions for use of the instrument.

Defining Domains and Elements For Study Quality Grids

Systematic Reviews and Meta-Analyses (Grid 1)

Table 7. Domains and Elements for Systematic Reviews (more...)

Table 7. Domains and Elements for Systematic Reviews

Domain	Elements*
Study Question	Question clearly specified and appropriate
Search Strategy	Sufficiently comprehensive and rigorous with attention to possible publication biases Search restrictions justified (e.g., language or country of origin) Documentation of search terms and databases used Sufficiently detailed to reproduce study
Inclusion and Exclusion Criteria	Selection methods specified and appropriate, with a priori criteria specified if possible
Interventions	Intervention(s) clearly detailed for all study groups
Outcomes	All potentially important harms and benefits considered
Data Extraction†	Rigor and consistency of process Number and types of reviewers Blinding of reviewers Measure of agreement or reproducibility Extraction of clearly defined interventions/exposures and outcomes for all relevant subjects and subgroups
Study Quality and Validity	Assessment method specified and appropriate Method of incorporation specified and appropriate
Data Synthesis and Analysis	Appropriate use of qualitative and/or quantitative synthesis, with consideration of the robustness of results and heterogeneity issues Presentation of key primary study elements sufficient for critical appraisal and replication
Results	Narrative summary and/or quantitative summary statistic and measure of precision, as appropriate
Discussion	Conclusions supported by results with possible biases and limitations taken into consideration
Funding or Sponsorship	Type and sources of support for study

*Elements appearing in italics are those with an empirical basis. Elements appearing in bold are those considered essential to give a system a Yes rating for the domain.

† Domain for which a Yes rating required that a majority of elements be considered.

Table 7 defines the 11 quality domains and elements appropriate for systematic reviews and meta-analyses; these domains constitute the columns for Grid 1 in Appendix B. The domains are study question, search strategy, inclusion and exclusion criteria, interventions, outcomes, data extraction, study quality and validity, data synthesis and analysis, results, discussion, and funding or sponsorship. Search strategy, study quality and validity, data synthesis and analysis, and funding or sponsorship have at least one empirically based element. The remaining domains are generally accepted criteria used by most experts in the field, and they apply most directly to systematic reviews of RCTs.

Randomized Controlled Trials (Grid 2)

Table 8. Domains and Elements for Randomized Controlled Trials (more...)

Table 8. Domains and Elements for Randomized Controlled Trials

Domain	Elements*
Study Question	Clearly focused and appropriate question
Study Population	Description of study population Specific inclusion and exclusion criteria Sample size justification
Randomization	Adequate approach to sequence generation Adequate concealment method used Similarity of groups at baseline
Blinding	Double-blinding (e.g., of investigators, caregivers, subjects, assessors, and other key study personnel as appropriate) to treatment allocation
Interventions	Intervention(s) clearly detailed for all study groups (e.g., dose, route, timing for drugs, and details sufficient for assessment and reproducibility for other types of interventions) Compliance with intervention Equal treatment of groups except for intervention
Outcomes	Primary and secondary outcome measures specified Assessment method standard, valid, and reliable
Statistical Analysis	Appropriate analytic techniques that address study withdrawals, loss to follow-up, missing data, and intention to treat Power calculation Assessment of confounding Assessment of heterogeneity, if applicable
Results	Measure of effect for outcomes and appropriate measure of precision Proportion of eligible subjects recruited into study and followed up at each assessment
Discussion	Conclusions supported by results with possible biases and limitations taken into consideration
Funding or Sponsorship	Type and sources of support for study

*Elements appearing in italics are those with an empirical basis. Elements appearing in bold are those considered essential to give a system a full Yes rating for the domain.

Table 8 presents the 10 quality domains for RCTs: study question, study population, randomization, blinding, interventions, outcomes, statistical analysis, results, discussion, and funding or sponsorship. Of these domains, four have one or more empirically supported elements: randomization, blinding, statistical analysis, and funding or sponsorship. Every domain has at least one essential element.

Observational Studies (Grid 3)

In observational studies, some factor other than randomization determines treatment assignment or exposure (see Figure 1 in Chapter 1 for clarification of the major types of observational studies). The two major types of observational studies are cohort and case-control studies. In a cohort study, a group is assembled and followed forward in time to evaluate an outcome of interest. The starting point for the follow-up may occur back in time (retrospective cohort) or at the present time (prospective cohort). In either situation, participants are followed to determine whether they develop the outcome of interest. Conversely, for a case-control study, the outcome itself is the basis for selection into the study. Previous interventions or exposures are then evaluated for possible association with the outcome of interest.

In all observational studies, selection of an appropriate comparison group of people without either the intervention/exposure or the outcome of interest is generally the most important and the most difficult design issue. Ensuring the comparability of the treatment groups in a study is what makes the RCT such a powerful research design. Observational studies are generally considered more liable to bias than RCTs, but certain questions can be answered only by using observational studies.

Table 9. Domains and Elements for Observational Studies (more...)

Table 9. Domains and Elements for Observational Studies

Domains	Elements
Study Question	Clearly focused and appropriate question
Study Population	Description of study populations Sample size justification
Comparability of Subjects†	For all observational studies: Specific inclusion/exclusion criteria for all groups Criteria applied equally to all groups Comparability of groups at baseline with regard to disease status and prognostic factors Study groups comparable to non-participants with regard to confounding factors Use of concurrent controls Comparability of follow-up among groups at each assessment Additional criteria for case-control studies: Explicit case definition Case ascertainment not influenced by exposure status Controls similar to cases except without condition of interest and with equal opportunity for exposure
Exposure or Intervention	Clear definition of exposure Measurement method standard, valid and reliable Exposure measured equally in all study groups
Outcome Measurement	Primary/secondary outcomes clearly defined Outcomes assessed blind to exposure or intervention status Method of outcome assessment standard, valid and reliable Length of follow-up adequate for question
Statistical Analysis	Statistical tests appropriate Multiple comparisons taken into consideration Modeling and multivariate techniques appropriate Power calculation provided Assessment of confounding Dose-response assessment, if appropriate
Results	Measure of effect for outcomes and appropriate measure of precision Adequacy of follow-up for each study group
Discussion	Conclusions supported by results with biases and limitations taken into consideration
Funding or Sponsorship	Type and sources of support for study

*Elements appearing in italics are those with an empirical basis. Elements appearing in bold are those considered essential to give a system a Yes rating for the domain.

† Domain for which a Yes rating required that a majority of elements be considered.

All nine domains and most of the elements for each domain apply generically to both cohort and case-control studies (Table 9). The domains are as follows: study question, study population, comparability of subjects, definition and measurement of the exposure or intervention, definition and measurement of outcomes, statistical analysis, results, discussion, and funding or sponsorship. Certain elements in the comparability-of-subjects domain are unique to case-control designs.

There are two empirically based elements for observational studies, use of concurrent controls and funding or sponsorship. However, a substantial body of accepted "best practices" exists with respect to design and conduct of observational studies, and we identified seven elements as essential.

Diagnostic Studies (Grid 4)

Table 10. Domains and Elements for Diagnostic Studies (more...)

Table 10. Domains and Elements for Diagnostic Studies

Domain	Elements*
Study Population	Subjects similar to populations in which the test would be used and with a similar spectrum of disease
Adequate Description of Test	Details of test and its administration sufficient to allow for replication of study
Appropriate Reference Standard	Appropriate reference standard ("gold standard") used for comparison Reference standard reproducible
Blinded Comparison of Test and Reference	Evaluation of test without knowledge of disease status, if possible Independent, blind interpretation of test and reference
Avoidance of Verification Bias	Decision to perform reference standard not dependent on results of test under study

*Elements appearing in italics are those with an empirical basis. Elements appearing in bold are those considered essential to give a system a Yes rating for the domain.

Assessment of diagnostic study quality is a topic of active current research.⁷⁸ We based the five domains in Table 10 for this grid on the work of the STARD (STAndards for Reporting Diagnostic Accuracy) group. The domains are study population, test description, appropriate reference standard, blinded comparison, and avoidance of verification bias. We designated five elements in Table 10 as essential, all of which are empirically derived.

The domains for diagnostic tests are designed to be used with the domains (and grids) for RCTs or observational studies because these are the basic study designs used to evaluate diagnostic tests. The domains for diagnostic tests can, in theory, also be applied to questions involving screening tests.

Assessing and Describing Quality Rating Instruments

Quality

Overall quality of a body of scientific studies is influenced by all the factors mentioned in our discussion of the quality of individual studies above. Grading systems that considered at least two of the following criteria -- study design, conduct, analysis, or methodologic rigor -- merited a Yes on quality. Systems that based their evidence grading on the hierarchy of research design without mention of methodologic rigor received a Partial rating.

Quantity

We use the construct "quantity" to refer to the extent to which there is a relationship between the technology (or exposure) being evaluated and outcome as well as to the amount of information supporting that relationship. Three main factors contribute to quantity:

• The magnitude of effect (i.e., estimated effects such as mean differences, odds ratio, relative risk, or other comparative measure);

• The number of studies performed on the topic in question (e.g., only a few versus perhaps a dozen or more); and

• The number of individuals studied, aggregated over all the relevant and comparable investigations, which provides the width of the confidence limits for the effect estimates.

The magnitude of effect is evaluated both within individual studies and across studies, with a larger effect indicative of a stronger relationship between the technology (or exposure) under consideration and the outcome. The finding that patients receiving a treatment are 5 times more likely to recover from an illness than those who do not receive the treatment is considered stronger evidence of efficacy than a finding that patients receiving a treatment are 1.3 times more likely to recover. However, absent any form of systematic bias or error in study design, and assuming equally narrow confidence intervals, there is no reason to consider this assertion (i.e., that the former is stronger evidence) to be the case. Rather, this illustrates the fact that one is simply measuring different sizes (magnitudes) of treatment effect. Nevertheless, no study is free from some element of potential unmeasured bias. The impact of such bias can overestimate or underestimate the treatment effect. Therefore, a large treatment effect partially protects an investigation against the threat that such bias will undermine the study's findings.

With respect to numbers of studies and individuals studied, common sense suggests that the greater the number of studies (assuming they are of good quality), the more confident analysts can be of the robustness of the body of evidence. Thus, we assume that systems for grading bodies of evidence ought to take account of the sheer size of that body of evidence.

Moreover, apart from the number of studies per se is the aggregate size of the samples included in those studies. All other things equal, a larger total number of patients studied can be expected to provide more solid evidence on the clinical or health technology question than a smaller number of patients. The line of reasoning is that hundreds (or thousands) of individuals included in numerous studies that evaluate the same issue give decisionmakers reason to believe that that the topic has been thoroughly researched. In technical terms, the power of the studies to detect both statistically and clinically significant differences is enhanced when the size of the patient populations studied is larger.

However, a small improvement or difference between study patients and controls or comparisons must still be considered in light of the potential public health implications of the association under study. A minimal net benefit for study patients relative to comparison may seem insignificant except if it applies to very large numbers of individuals or can be projected to yield meaningful savings in health care costs. Thus, when using magnitude of an effect for judging the strength of a body of evidence, one must consider the size of the population that may be affected by the finding in addition to the effect size and whether it is statistically significant. Magnitude of effect interacts with number and aggregate size of the study groups to affect the confidence analysts can have in how well a health technology or procedure will perform. In technical terms, summary effect measures calculated from studies with many individuals will have narrower confidence limits than effect measures developed from smaller studies. Narrower confidence limits are desirable because they indicate that relatively little uncertainty attends the computed effect measure. In other words: a 95-percent confidence interval indicates that decisionmakers and clinicians can, with comfort, believe that 95 percent of the time the confidence interval will include (or cover) the true effect size.

A Yes for quantity meant that the system incorporated at least two of the three elements listed above. For example, if a system considered both the magnitude of effect and a measure of its precision (i.e., the width of the confidence intervals around that effect, which as noted is related to size of the studies), we assigned it a Yes. Rating systems that considered only one of these three elements merited a grade of Partial.

Consistency

Consistency is the degree to which a body of scientific evidence is in agreement with itself and with outside information. More specifically, a body of evidence is said to be consistent when numerous studies done in different populations using different study designs to measure the same relationship produce essentially similar or compatible results. This essentially means that the studies have produced reasonably reproducible results. In addition, consistency addresses whether a body of evidence agrees with externally available information about the natural history of disease in patient populations or about the performance of other or related health interventions and technologies. For example, information about older drugs can predict reactions to newer entities that have related chemical structures, and animal studies of a new drug can be used to predict similar outcomes in humans.

Figure 2

Rating Study Quality

Our first task was to identify instruments ("systems" in the original legislation mandating this report for the Agency for Healthcare Research and Quality [AHRQ]) for rating study quality. During our search process, we identified scales, checklists, and evaluations of quality components. In addition, we identified publications that discussed the importance of assessing article quality and that included quality items for consideration; we refer to these publications as guidance documents. To be complete, we include the guidance documents in Grids 1-4 (Appendix B), but in their current state we do not believe such documents can or should be used to rate the quality of individual studies.

Overall, we reviewed 82 different quality rating instruments or guidance documents for all four grids. This number encompasses reference papers that describe a study quality rating scheme or a rating method that is specific to work from an AHRQ-supported Evidence-based Practice Center (EPC). Because several of these 82 systems could be used to rate quality for more than one study design, we included them on multiple grids. Some came from our literature search, but we identified most by reviewing the previous effort of the Research Triangle Institute-University of North Carolina EPC¹ and work from Moher et al.¹⁰¹ and by hand searching Internet sites and bibliographies.

Table 12. Number of Systems Reviewed for Four Types (more...)

Table 12. Number of Systems Reviewed for Four Types of Studies by Type of System, Instruments, or Document

Study Design (Grid)	Total	Scales, Checklists, and Component Evaluations	Guidance Documents	EPC Rating Systems
Systematic Reviews (Grid 1)	20	11	9	0
Randomized Controlled Trials (Grid 2)	49	32	7	10
Observational Studies (Grid 3)	19	12	5	2
Diagnostic Tests (Grid 4)	18	6	9	3

Grading the Strength of a Body of Evidence

We found it difficult to discern the most productive, yet specific, search terms for identifying literature that discussed grading a body of evidence. We approached our search from many different perspectives. In the end, although we identified numerous papers through the search, we found the majority of the relevant publications through hand searches and contacts with experts in the field. We suspect that, at present, the subject headings for coding the literature on this topic are not adequate to yield an appropriately thorough and productive search.

Thus, many of the 40 systems on which we provide information in Grid 5 (Appendix C) were identified through other sources or by reviewing bibliographies from papers retrieved by the search. Excluding the six evidence grading systems developed by the EPCs, approximately two-thirds (n = 22) of the remaining 34 systems arose from the guideline or clinical recommendations literature. Thus, only 12 of the evidence grading systems we reviewed were developed for nonguideline needs such as a literature synthesis or for purposes of evidence-based practice in general.

Background

Chapter 2 describes the four study quality grids in Appendix B, including both the domains and elements used to compare rating systems (see Tables 7-10) and the properties used to describe them.

Rating Systems for Systematic Reviews

Type of System or Instrument

Table 13. Evaluation of Scales and Checklists for Systematic Reviews, (more...)

Table 13. Evaluation of Scales and Checklists for Systematic Reviews, by Specific Instrument and 11 Domains

Instrument	Domains
	Study Question	Search Strategy*	Inclusion/Exclusion	Interventions	Outcomes	Data Extraction	Study Quality/Validity*	Data Synthesis and Analysis*	Results	Discussion	Funding*
Oxman et al., 1991;4 Oxman et al., 19915
Irwig et al., 19946
Sacks et al., 19967
Auperin et al., 19978
Beck, 19979
Smith, 199710
Barnes and Bero, 19983
Clarke and Oxman, 199911
Khan et al., 200012
New Zealand Guidelines Group, 200013
Harbour and Miller, 200114

*Domains with at least one element with an empirically demonstrated basis (see Table 7).

Twenty systems were concerned with systematic reviews or meta-analyses (Grid 1). Of these (Table 13), we categorized one as a scale³ and 10 as checklists.^4-14 The remainder are considered guidance documents.^15-23,59,68 In the presentation below, we group scales and checklists into one set of results and comment on guidance documents separately.

Evaluation of Systems According to Seven Domains Considered Informative for Study Quality

Apart from the four domains that contained empirical elements, we concluded that three additional domains provide important information on the quality of a systematic review -- study question, inclusion/exclusion criteria, and data extraction. The degree to which instruments concerned with systematic reviews covered these three domains is described just below, followed by a discussion of systems that appeared to deal with all seven domains.

Study Question

A clearly specified study question is important to define the search appropriately, determine which articles to exclude from the analysis, focus the interventions and outcomes, and conduct a meaningful data synthesis. Only two of the 20 systems omitted study question as a domain,^17,22 and an additional two received a Partial score for this domain.^8,10

Inclusion/Exclusion

After the search is completed, determination of article eligibility is based on clearly specified selection criteria with reasons for inclusion and exclusion. Developing and adhering to strict inclusion and exclusion criteria makes the systematic review more reproducible and less subject to selection bias. Of the 20 systems we reviewed, every one addressed the inclusion/exclusion domain, with only three systems receiving a Partial for this domain.^4,5,14,15

Data Extraction

How data had been extracted from single articles for purposes of systematic reviews is often overlooked in assessing the quality of a systematic review. Like the search strategy domain, the data extraction domain provides useful insight on the reproducibility of the systematic review. Reviews that do not use dual extraction may miss or misrepresent important concepts. Of the 20 systems we reviewed, six omitted data extraction altogether^{3,10,11,13,14,22} and three were given a Partial score for this domain.^4,5,15,19

Coverage of Seven Key Domains

Table 14. Evaluation of Scales and Checklists for Systematic (more...)

Table 14. Evaluation of Scales and Checklists for Systematic Reviews by Instrument and Seven Key Domains

Instrument	Domains
	Study Question	Search Strategy*	Inclusion/ Exclusion	Data Extraction	Study Quality*	Data Synthesis/ Analysis*	Funding*
Irwig et al., 19946
Sacks et al., 19967
Auperin et al., 19978
Barnes and Bero, 19983
Khan et al., 200012

*Domains with at least one element with an empirically demonstrated basis (see Table 7).

To arrive at a set of high-performing scales or checklists pertaining to systematic reviews, we took account of seven domains in all: study question, search strategy, inclusion/exclusion criteria, data extraction, study quality, data synthesis, and funding. We then used these seven domains as the criteria by which to identify a selected group of systems that could be said with some confidence to represent acceptable approaches that could be used today without major modifications. These are depicted in Table 14.

Five systems met most of the criteria for systematic reviews. One checklist fully addressed all seven domains.⁷ A second checklist also addressed all seven domains but merited only a Partial for study question and study quality.⁸ Two additional checklists^6,12 and the one scale³ addressed six of the domains. These latter two checklists excluded funding; the scale omitted data extraction and had a Partial score for search strategy.

Rating Systems for Randomized Controlled Trials

Evaluation of Systems According to Coverage of Empirical or Best Practices Domains

Empirical Domains

Table 15. Evaluation of Scales, Checklists, and Component Evaluations (more...)

Table 15. Evaluation of Scales, Checklists, and Component Evaluations for Randomized Controlled Trials, by Specific Instrument and 10 Domains

Instrument	Domains
	Study Question	Study Popu-lation	Randomization*	Blinding*	Interventions	Outcomes	Statistical Analysis*	Results	Discussion	Funding*
Chalmers et al., 198124
Liberati et al., 198626
Reisch et al., 198945
Schulz et al., 199551
van der Heijden et al., 199636
de Vet et al., 199718
Sindhu et al., 199738
van Tulder et al., 199739
Downs et al., 199840
Moher et al., 199841
Khan et al., 200012
NHMRC, 200049
Harbour and Miller, 200114
Turlik et al., 200042

*Domains with at least one element with an empirically demonstrated basis (see Table 8).

The 10 domains used for assessing these systems (Table 15 or Grid 2A) reflect characteristics specific to both RCTs and general study design (see Table 8). Of these domains, four contain elements that are derived from empirical research: randomization, blinding, statistical analysis, and funding or sponsorship. Both blinding and funding had only a single element (which was based on empirical research). The randomization domain comprised three elements, all of which were empirically derived. Statistical analysis had four elements, only one of which was empirically derived. In the results below, we focus on the extent to which the systems we reviewed covered these empirical domains.

Of the 32 scales, checklists, and component systems concerned with RCTs (Grid 2), only two fully addressed the four domains with empiric elements.^25,45 An additional 12 systems fully addressed randomization, blinding, and statistical analysis but not source of funding.^{12,14,18,26,36,38-42,49,51} If we consider the systems that addressed the first three domains (randomization, blinding, statistical analysis) either partially or fully, we would add another 14 to this count.^{13,25,27,28,29,31-35,37,43,44,47,48} Thus, only four of the RCT scales or checklists failed to address one or more of the three empirical domains, randomization, blinding, or statistical analysis.^29,30,46,50

Best Practices Domains

The remaining six domains -- study question, study population, interventions, outcomes, results, and discussion -- come from best practices criteria. We included these for comparison purposes and because many of the systems we evaluated included items addressing these domains.

Focusing on the 14 scales, checklists, and component evaluation (Table 15) that fully addressed the three empiric domains -- randomization, blinding, and statistical analysis -- few systems included either study question or discussion.^14,38,40,45 However, 11 systems did address three other domains -- study population, intervention, and results -- either partially or fully.^{12,14,18,24,26,36,38-40,42,45} Of these 11 systems, 10 also included outcomes as a domain; the exception is the work of the NHS Centre for Reviews and Dissemination.¹² Thus, these 11 systems included, either fully or in part, most of the domains that we selected to compare across systems.

Table 16. Evaluation of Guidance Documents for Randomized Controlled (more...)

Table 16. Evaluation of Guidance Documents for Randomized Controlled Trials, by Instrument and 10 Domains

Instrument	Domains
	Study Question	Study Population	Randomization*	Blinding*	Interventions	Outcomes	Statistical Analysis*	Results	Discussion	Funding*
Prendiville et al., 198852
Guyatt et al., 1993;54 Guyatt et al., 199453
Standards of Reporting Trials Group, 199455
Asilomar Working Group, 199656
Moher et al., 200157
Clarke and Oxman, 199911
Lohr and Carey, 19991

*Domains with at least one element with an empirically demonstrated basis (see Table 8).

Because guidance documents have not been developed as tools for assessing quality per se, we have examined them primarly for illustrative purposes (Table 16). The number of domains addressed in the guidance documents varied by system -- from as few as three to all 10 of the domains. The consensus statements typically include most of the 10 domains.^55-57 The earliest consensus statement fully addressed seven domains, partially addressed one other, and failed to address two domains.⁵⁵ The Asimolar Working Group included all 10 domains but received a Partial for the randomization, blinding, and statistical analysis domains.⁵⁶ The most recent CONSORT statement fully addressed nine domains, omitting funding.⁵⁷

Of the 10 EPC rating systems (see Grid 2A in Appendix B), all included both randomization and blinding at least partially. Statistical analysis was addressed either fully or partially by all but one system.⁶³ Study population, interventions, outcomes, and results were covered fully by five EPC systems.^{60,61,63,65,66} EPC quality systems for RCTs rarely included either study question or discussion.

Evaluation of Systems According to Seven Domains Considered Informative for Study Quality

As noted above, we identified four empirically based quality domains. To these we added three domains derived from best practices -- study population, interventions, and outcomes -- that we regarded as important for evaluating the quality of RCTs.

Study Population

The most important element in the study population domain is the specification of inclusion and exclusion criteria for entry of participants in the trial. Although such criteria constrain the population being studied (thereby making the study less generalizable), they reduce heterogeneity among the persons being studied. In addition, the criteria reduce variability, which improves our certainty of claiming a treatment effect if one truly exists.

Interventions

Intervention is another important quality domain mainly for one of its elements -- that the intervention be clearly defined. For reasons of reproducibility both within the study and for comparison with other studies, investigators ought to describe fully the intervention under study with respect to dose, timing, administration, or other factors. Paying careful attention to the details of an intervention also tends to reduce variability among the subjects, which also influences what can be said about the study outcome.

Outcomes

As important as it is to describe the intervention clearly, it is also critical to specify clearly the outcomes under study and how they are to be measured. Again, this is important for both reproducibility and to decrease variability.

Coverage of Seven Key Domains

Table 17. Evaluation of Scales and Checklists for Randomized (more...)

Table 17. Evaluation of Scales and Checklists for Randomized Controlled Trials, by Instrument and Seven Key Domains

Instrument	Domains
	Study Population	Random-ization*	Blinding*	Interventions	Outcomes	Statistical Analysis*	Funding*
Chalmers et al., 198124
Liberati et al., 198626
Reisch et al., 198945
van der Heijden and van der Windt, 199636
de Vet et al., 199718
Sindhu et al., 199738
Downs and Black, 199840
Harbour and Miller, 200114

*Domains with at least one element with an empirically demonstrated basis (see Table 8).

We designated a set of high-performing scales or checklists pertaining to RCTs by assessing their coverage of the following seven domains: study population, randomization, blinding, interventions, outcomes, statistical analysis, and funding. As with the five systems identified for systematic reviews, we concluded that these eight systems for RCTs represent acceptable approaches that could be used today without major modifications (Table 17).

Two systems fully addressed all seven domains,^24,45 and six others addressed all but funding.^{14,18,26,36,38,40} Two were rigorously developed.^38,40 We might assume that the rigorousness with which the instruments were developed is important for assessing quality, but this has not been tested. Users wishing to adopt a system for rating the quality of RCTs will need to do so on the basis of the topic under study, whether a scale or checklist is desired, and apparent ease of use.

Rating Systems for Observational Studies

Type of System or Instrument

Table 18. Evaluation of Scales and Checklists for Observational (more...)

Table 18. Evaluation of Scales and Checklists for Observational Studies, by Specific Instrument and Nine Domains

Instrument	Domains
	Study Question	Study Population	Comparability of Subjects*	Exposure/Intervention	Outcome Measure	Statistical Analysis	Results	Discussion	Funding*
Reisch et al., 198945
Spitzer et al., 199047
Cho and Bero, 199431
Goodman et al., 199432
Downs and Black, 199840
Corrao et al., 199969
Ariens et al., 200070
Khan et al., 200012
New Zealand Guidelines, 200013
NHMRC, 200049
Harbour and Miller, 200114
Zaza et al., 200050

*Domains with one element with an empirically demonstrated basis (see Table 9).

Seventeen systems concerned observational studies (Grid 3). Of these, we categorized four as scales^31,32,40,69 and eight as checklists (Table 18)^{12-14,45,47,49,50,70} We classified the remaining five as guidance documents.^1,71-74 Two EPCs used quality rating systems for evaluating observational studies -- these systems were identical to those used for RCTs. In the presentation below, we discuss scales and checklists separately from guidance documents and EPC rating systems.

Evaluation of Systems According to Domains Considered Informative for Study Quality

To arrive at a set of high-performing scales or checklists pertaining to observational studies, we considered the following five domains: comparability of subjects, exposure/intervention, outcomes, statistical analysis, and funding or sponsorship. As before, we concluded that systems that cover these domains represent acceptable approaches for assessing the quality of observational studies. The inclusion of the two empirical domains is self-explanatory (comparability of subjects and funding or sponsorship); we explain below why we considered the following as critical domains.

Exposure or Intervention

Unlike RCTs where treatment is administered in a controlled fashion, exposure or treatment in observational studies is based on the clinical situation and may be subject to unknown biases. These biases may result from provider, patient, or health care system differences. Thus, a clear description of how the exposure definition was derived is critical for understanding the effects of that exposure on outcome.

Outcomes

Investigators need to supply a specific definition of outcome that is independent of exposure. The presence or absence of an outcome should be based on standardized criteria to reduce bias and enhance reproducibility.

Statistics and Analysis

Of the six elements in the statistical analysis domain, confounding assessment was considered essential for a full Yes rating. Observational studies are particularly subject to several biases; these include measurement bias (usually addressed by specific exposure and outcome definitions) and selection bias (typically addressed by ensuring the comparability among subjects and confounding assessment). We did not consider any of the remaining five statistical analysis elements -- statistical tests, multiple comparisons, multivariate techniques, power calculations, and dose response assessments -- as more important than any other when evaluating systems on this domain.

Coverage of Five Key Domains

Table 19. Evaluation of Scales and Checklists for Observational (more...)

Table 19. Evaluation of Scales and Checklists for Observational Studies, by Instrument and Five Key Domains

Instrument	Domains
	Comparability of Subjects	Exposure/ Intervention	Outcome Measure	Statistical Analysis	Funding
Reisch et al., 198945
Spitzer et al, 199047
Goodman et al., 199432
Downs and Black, 199840
Harbour and Miller, 200114
Zaza et al., 200050

Of the 12 scales and checklists we reviewed, all included comparability of subjects either fully or in part. Only one included funding or sponsorship and the other four domains we considered critical for observational studies.⁴⁵ Five additional systems fully included all four domains without funding or sponsorship (Table 19).^{14,32,40,47,50} In choosing among these six systems for assessing study quality, one will have to evaluate which system is most appropriate for the task being undertaken, how long it takes to complete each system, and its ease of use. We were unable to evaluate these three instrument properties in the project.

Rating Systems for Diagnostic Studies

Evaluation of Systems According to Coverage of Empirical or Best Practices Domains

Empirical Domains

The five domains used for assessing these systems (Table 10 and Grid 4) reflect design issues specific to evaluating diagnostic tests. Three domains -- study population, adequate description of the test, and avoidance of verification bias -- have only a single, empirical element; the other two domains each contain two elements, one of which has an empirical base.

Table 20. Evaluation of Scales and Checklists for Diagnostic (more...)

Table 20. Evaluation of Scales and Checklists for Diagnostic Test Studies, by Specific Instrument and Five Domains

Instrument	Domains*
	Study Population	Adequate Description of Test	Appropriate Reference Standard	Blinded Comparison of Test and Reference	Avoidance of Verification Bias
Sheps and Schechter, 1984;75 Arroll et al., 198876
Cochrane Methods Working Group, 199677
Lijmer et al., 199978
Khan et al., 200012
NHMRC, 200049
Harbour and Miller, 200114

*All domains have at least one element based on empirical evidence (see Table 10).

Of the generic checklists we reviewed (Table 20), three fully addressed all six domains.^49,77,78 Two systems dealt with four of the five domains either fully or in part.^12,14 One checklist, the oldest of those we reviewed, addressed only one domain fully -- use of an appropriate reference standard -- and partially addressed the blinded reference comparison domain.^75,76

Almost all of the nine guidance documents included all these domains. One omitted the avoidance of verification bias domain;⁷¹ another omitted adequate description of the test.⁶ Of the three EPC scales, two addressed all five domains either fully⁸⁰ or in part.^59,68 We gave the remaining EPC system a No for adequate description of the test under study, although the information about the test was likely to have been captured apart from the quality rating system.⁷⁹

Background

We reviewed 40 systems that addressed grading the strength of a body of evidence. In discussing these approaches, we focus on 34 systems identified from our searches and prior research separately from those developed by six EPCs. The non-EPC systems came from numerous international sources, with the earliest systems coming from Canada. Based on the affiliation of the lead author, they originated as follows: Canada (11), United States (10), United Kingdom (6), Australia/New Zealand (3), the Netherlands (3), and a multi-national consensus group (1).

Evaluation According to Domains and Elements

Grid 5A distills the detailed information in Grid 5B. We use the same rating scheme as we did for the quality grids: Yes ( , the instrument fully addressed the domain); No ( , it did not address the domain at all); or Partial ( , it addressed the domain to some extent). Our findings for each system are discussed below.

Evaluation of Systems According to Three Domains That Address the Strength of the Evidence

Domains

Table 21. Extent to Which 34 Non-EPC Strength of Evidence Grading (more...)

Table 21. Extent to Which 34 Non-EPC Strength of Evidence Grading Systems Incorporated Three Domains of Quality, Quantity, and Consistency

Number of Domains Addressed and Extent of Coverage	Number of Systems
All three domains
Addressed fully	711,81-86
Addressed fully or partially	912,14,39,70,87-91
Two of three domains
Addressed fully	513,22,122,124,125
Addressed fully or partially	10112-116,118,121,123,126-128
One domain addressed fully or partially	3117,119,120

As indicated in Table 21, the 34 non-EPC systems incorporated quality, quantity, and consistency to varying degrees. Seven systems fully addressed the quality, quantity, and consistency domains.^11,81-86 Nine others incorporated the three domains at least in part.^{12,14,39,70,87-91}

Of the six EPC grading systems, only one incorporated quality, quantity, and consistency.⁶⁵ Four others included quality and quantity either fully or partially.^59,60,67,68 The one remaining EPC system included quantity; study quality is measured as part of their literature review process but this domain is apparently not directly incorporated into the grading system.⁶⁶

Domains, Publication Year, and Purpose of System

Whether the grading systems dealing with overall strength of evidence dealt with all three domains appeared to differ by year of publication. The more recent systems included, either fully or partially, all three domains more frequently than did the older systems. Of the 23 evidence grading systems that had been published before 2000, seven (30 percent) included quality, quantity, and consistency to some degree; the same was true for nine (82 percent) of the 11 systems published in 2000 or later. This wide disparity among the systems can be attributed to the consistency domain, which began to appear more frequently from 2000 onward.

Table 22. Number of Non-EPC Systems to Grade Strength (more...)

Table 22. Number of Non-EPC Systems to Grade Strength of Evidence, by Number of Domains Addressed, Primary Purpose for System Development, and Year of Publication

Number of Domains Addressed*	Guideline System		Non-Guideline System
	Before 2000	After 2000	Before 2000	After 2000
3 domains addressed either partially or fully	381,88,89	514,82,83,86,91	411,39,87,90	412,70,84,85
<3 domains addressed either partially or fully	13112-116,118-123,125,126,128	213,22	3117,121,126	0

*For systems to grade strength of evidence, domains are quality, quantity, and consistency.

As discussed above, many evidence grading systems came from the clinical practice guideline literature. Table 22 shows that, at least among the 34 non-EPC grading systems, whether the three domains were incorporated differed by year of publication and primary purpose (i.e., for guideline development per se or for evidence grading). The nonguideline systems seemingly tended to incorporate all three domains more than the guideline systems, and this trend appears to be increasing over time.

Evaluation of Systems According to Domains Considered Informative for Assessing the Strength of a Body of Evidence

Of the seven systems that fully addressed quality, quantity, and consistency,^11,81-86 four were used for developing guidelines or practice recommendations,^81-83,86 and the remaining three were used for promoting evidence-based health care.^11,84,85

Table 23. Characteristics of Seven Systems to Grade (more...)

Table 23. Characteristics of Seven Systems to Grade Strength of Evidence

Source	Domain
	Quality	Quantity	Consistency	Strength of Evidence Grading System	Comments
Gyorkos et al., 199481	Validity of studies	Strength of association and precision of estimate	Variability in findings from independent studies	Overall assessment of level of evidence based on four elements: Validity of individual studies Strength of association between intervention and outcomes of interest Precision of the estimate of strength of association Variability in findings from independent studies of the same or similar interventions For each element a qualitative assessment of whether there is strong, moderate, or weak support for a causal association.
Clarke and Oxman, 199911	Based on hierarchy of research design, validity, and risk of bias	Magnitude of effect	Consistency of effect across studies	Questions to consider regarding the strength of inference about the effectiveness of an intervention in the context of a systematic review of clinical trials: How good is the quality of the included trials? How large and significant are the observed effects? How consistent are the effects across trials? Is there a clear dose-response relationship? Is there indirect evidence that supports the inference? Have other plausible competing explanations of the observed effects (e.g., bias or cointervention) been ruled out?	Other domains: Dose-response relationship Supporting indirect evidence No other plausible explanation
Briss et al., 200082	Threats to Validity: - Study description - Sampling - Measurement - Data analysis - Interpretation of results - Other Quality of Execution: - Good (0-1 threats) - Fair (2-4 threats) - Limited (5+ threats) Design suitability: Greatest concurrent comparison groups and prospective measure-ment Moderate all retrospective designs or multiple pre or post measurements; no concurrent comparison group Least single pre and post measurements; no concurrent comparison group or exposure and outcome measured in a single group at the same point in time.	Effect size - Sufficient - Large - Small Larger effect sizes (absolute or relative risk) are considered to represent stronger evidence of effective-ness than smaller effect sizes with judgments made on an individual basis	Consistency as yes or no.	Evidence of effectiveness is based on execution, design suitability, number of studies, consistency, and effect size Strong: Good and greatest, at least 2 studies consistent, sufficient Good/fair and great/moderate, at least 5 studies, consistent, sufficient Good/fair and any design, at least 5 studies, consistent, sufficient Sufficient Good and greatest, one study, consistency unknown, sufficient Good/fair and great/moderate, at least 3 studies, consistent, sufficient Good/fair and any design, at least 5 studies consistent, sufficient Expert opinion: sufficient effect size Insufficient: insufficient design, too few studies, inconsistent, small effect size
Greer et al., 200083	Strong design not defined but includes issues of bias and research flaws	System incorporates number of studies and adequacy of sample size	Incorporates consistency	Grade    I: Evidence from studies of strong design; results are both clinically important and consistent with minor exceptions at most; results are free from serious doubts about generalizability, bias, and flaws in research design. Studies with negative results have sufficiently large samples to have adequate statistical power.    II: Evidence from studies of strong design but there is some uncertainty due to inconsistencies or concern about generalizability, bias, research design flaws, or adequate sample size. Or, evidence consistent from studies of weaker designs.    III: The evidence is from a limited number of studies of weaker design. Studies with strong design either haven't been done or are inconclusive.    IV: Support solely from informed medical commentators based on clinical experience without substantiation from the published literature.	Does not require a systematic review of the literature -- only six "important" research papers.
Guyatt et al., 200084	Based on hierarchy of research design, with some attention to size and consistency of effect	Multiplicity of studies, with some attention to magnitude of treatment effects	Consistency of effect considered	Hierarchy of vidence for application to patient care: N of 1 randomized trial Systematic reviews of randomized trials Single randomized trials Systematic review of observational studies addressing patient-important outcomes Single observational studies addressing patient-important outcomes Physiologic studies Unsystematic clinical observations Authors also discuss a hierarchy of preprocessed evidence that can be used to guide the care of patients: Primary studies -- by selecting studies that are both highly relevant and with study designs that minimize bias, permitting a high strength of inference Summaries -- systematic reviews Synopses -- of individual studies or systematic reviews Systems -- practice guidelines, clinical pathways, or evidence-based text book summaries	Evidence defined broadly as any empirical observation about the apparent relationship between events. "The hierarchy is not absolute. If treatment effects are sufficiently large and consistent, for instance, observational studies may provide more compelling evidence than most RCTs."
NHS Centre for Evidence Based Medicine, (http://cebm.jr2.ox.ac.uk) (Accessed 12-2001)85	Based on hierarchy of research design with some attention to risk of bias	Multiplicity of studies, and precision of estimate	Homogeneity of studies considered	Criteria to rate levels of evidence vary by one of four areas under consideration (Therapy/ Prevention or Etiology/Harm; Prognosis Diagnosis and Economic nalysis). For example, for the first area (Therapy/ Prevention or Etiology/Harm) the levels of evidence are as follows:    1a: SR with homogeneity of RCTs    1b: Individual RCT with narrow    1c: All or none (this criteri met when all patients died the treatment becm available and now some survive or some died previously and now none die)    2a: with homogeneity of cohort studies    2b: Individual cohort study (including low quality RCT; e.g. <80% follow-up)    2c: "Outcomes" research    3a: SR with homogeneity of case-control studies    3b: Individual case-control study    4: Case-series and poor quality cohort and case-control studies    5: Expert opinion without explicit critical appraisal or based on physiology, bench research or "first principles."
Harris et al., 200186 (for the U.S. Preventive Services Task Force)	Based on hierarchy of research design and methodologic quality (good, fair, poor) within research design	Number of studies, see Consistency	Consistency Consistency is not required by the Task Force but if present, contributes to both coherence and quality of the body of evidence	Levels of evidence:    I Evidence from at least one properly randomized controlled trial    II-1 Well-designed controlled trial without randomization    II-2 Well-designed cohort or case-control analytic studies, preferably from more than one center or group    II-3 Multiple time series with or without the intervention (also includes dramatic results in uncontrolled experiments):    III Opinions of respected authorities, based on clinical experience, descriptive studies, and case reports, or reports of expert committees    Aggregate internal validity is the degree to which the study(ies) provides valid evidence for the population and setting in which it was conducted. Aggregate external validity is the extent to which the evidence is relevant and generalizable to the population and conditions of typical primary care practice. Coherence/consistency	Other domains: Coherence Coherence implies that the evidence fits the underlying biologic model.

Quality of Individual Articles

Strength of a Body of Evidence

Similarly, our grid concerning systems for grading the strength of bodies of evidence (Appendix C) is tied directly to our conceptual framework. As discussed in Chapter 2, we focused on three domains -- quality, quantity, and consistency -- because they combine important aspects of the collective design, conduct, and analysis of studies that address a given topic. Quality here links back to the summation of the quality of individual articles. Quantity involves the magnitude of the estimated (observed) effect, the potential statistical power of the body of knowledge as reflected in the aggregate sizes of studies (i.e., their sample sizes), and the sheer number of studies bearing on the clinical or technology question under consideration. The accepted wisdom is that, all other things equal, a larger effect is better because a good deal of bias would have to be present to invalidate the likelihood of an association. Finally, consistency reflects the extent to which the results of included studies tell the same story and comport with known facts about the natural history of disease. These are well-established variables for characterizing how confidently we can conclude that a body of knowledge provides information on which clinicians or policymakers can act.

We did not include generalizability as a separate domain because we believed that our definition of consistency needed to focus only on concepts appropriate to grading the strength of a body of evidence. (In the evidence-based practice community, this idea is sometimes rendered as grading the strength of separate linkages in a comprehensive analytic framework or causal pathway.) In our view, generalizability (as it has typically been used in the clinical practice guideline arena) addresses whether the findings, aggregated across multiple studies, are relevant to particular populations, settings of care, types of clinicians, or other factors.

As we approached the tasks in this project, with the legislative mandate and AHRQ's history in mind, we concluded that our study ought to stop short of advising on the development or implementation of practice guidelines per se. Had we incorporated generalizability into our evaluative framework (as some peer reviewers suggested), our results and recommendations concerning systems for grading the strength of a body of evidence might have been very different.

Figure 2

Growth in Numbers of Systems

We identified at least three times as many scales and checklists for rating the quality of RCTs (n = 32) as we did for observational studies (n = 12), systematic reviews (n = 11), or diagnostic test studies (n = 6). We expect that ongoing methodological work addressing the quality of observational and diagnostic studies will over time affect both the number and the sophistication of these systems. Thus, our findings and conclusions with respect at least to observational and diagnostic studies may need to be readdressed once results from more methodological studies in these areas are available.

Development of Systems Appropriate for Observational Studies

As indicated in Appendix D, some empirical research is related to the design, conduct, and analysis of systematic reviews, RCTs, and studies evaluating diagnostic tests; much less information is presently available about the factors influencing the quality of observational studies. Many systems that we evaluated for observational studies (Grid 3) were ones that we also evaluated for RCTs (Grid 2). Reviewing the systems that apply to both types of study designs led us to conclude that the likely original intent of several of these systems was to evaluate the quality of RCTs and that the developers added questions to address observational studies as well.

Thus, abstracting information from and assessing these "one size fits all" systems against the two sets of relevant domains proved difficult (especially for the observational study grid). We see this as additional support for the view that a "single system" across all study types will not likely be achieved and, in fact, might be counterproductive.

Figure 1

By contrast, an observational study by its very nature "observes" what happens to individuals. Thus, to prevent selection bias, the comparison groups in an observation study are supposed to be as similar as possible except for the factors under study. For investigators to derive a valid result from their observational studies, they must achieve this comparability between study groups (and, for some types of prospective studies, maintain it by minimizing differential attrition). Because of the difficulty in ensuring adequate comparability between study groups in an observational study -- both when the project is being designed or upon review after the work has been published -- we wonder whether nonmethodologically trained researchers can identify when potential selection bias or other biases more common with observational studies have occurred.

Longer or Shorter Instruments

When comparing across all the quality rating scales and checklists that we evaluated, we noted that the older ones tended to be most inclusive for the quality domains we chose to assess.^24,45 However, these systems also tended to be very long and potentially cumbersome to complete. As factors critical to good study design have been identified -- that is, the empirical criteria we invoked in our assessments -- we saw that the more recent systems are shorter and focus mainly on these empirical criteria for rating study quality.

Shorter instruments have the obvious advantage of brevity, and some data suggest that they will provide sufficient information on study quality. Jadad and colleagues reported that simply asking about three domains (randomization, blinding, and withdrawals [a form of attrition]) serves to differentiate between higher- and lower-quality RCTs that evaluate drug efficacy.³⁴ However, the Jadad scale is not applicable to study designs other than RCTs of therapies, and it is not very useful for health services interventions where randomization or double blinding cannot occur. The Jadad team also omitted elements such as allocation concealment and use of intention-to-treat statistical analysis. We judged that these two elements have an empirical basis, but we acknowledge that the information supporting them has emerged since the publication of their scale.

The movement from longer, more inclusive instruments to shorter ones is a pattern observed throughout the health services research world for at least 25 years, particularly in areas relating to the assessment of health status and health-related quality of life. Thus, this model is not surprising in the field of evidence-based practice and measurement. However, the lesson to be drawn from efforts to derive shorter, but equivalently reliable and valid, instruments from longer ones (with proven reliability and validity) is that substantial empirical work is needed to ensure that the shorter forms operate as intended. More generally, we are not convinced that shorter instruments per se will always be better, unless demonstrated in future empirical studies.

Reporting Guidelines

Several authors of the QUOROM and CONSORT statements served on our technical expert panel.^21,57 They strongly emphasized that such reporting guidelines are not to be used for assessing the quality of either RCTs or systematic reviews, respectively. We believe this is an appropriate caution, and so we considered these consensus works only as guidance documents in our review.

We applaud these consensus guidelines for reporting RCTs and systematic reviews. If these guidelines are used (and they are currently required by certain journals) they will lead to better reporting and two downstream benefits. First, this may diminish the unavoidable tension (when assessing study quality) between the actual study design, conduct, and analysis and the reporting of these study characteristics. Second, if researchers follow these guidelines when designing their studies, they are likely to have better designed studies that will then be more transparent when published.

Interaction Among Domains

Our comparison of systems for assessing the strength of a body of evidence uses three domains (Grid 5). We did not try to unravel the interrelationships among quality, quantity, and consistency for this project. As the body of literature grows, additional studies (i.e., quantity) increase the likelihood of a large range of quality scores and heterogeneity with respect to population settings, outcomes measured, and results. When these factors are similar across studies, consistency (and thus, strength of evidence) is enhanced. When they are not, this heterogeneity will reduce consistency and presumably detract from the overall strength of the evidence. Alternatively, heterogeneity may provide clues that indicate important treatment differences across subpopulations under study.¹³⁰

Conflict Among Domains When Bodies of Evidence Contain Different Types of Studies

Adding to the complexities of evaluating interactive domains for one type of study design is the challenge of evaluating a body of knowledge comprising observational and RCT data. As our peer reviewers pointed out, a contemporary case in point is the association between hormone replacement therapy (HRT) and cardiovascular risk.

Several observational studies, but only one large trial and two small RCTs, have examined the association between HRT and secondary prevention of cardiovascular disease for older women with preexisting heart disease.^131-133 In terms of quality, much of the observational work is considered good and the RCTs are considered very good. In terms of quantity, both the numbers of reports and individuals evaluated in these reports are high for observational studies and modest for RCTs. Results are fairly consistent across the observational studies and across the RCTs, but between the two types of studies the results conflict. Observational studies show a treatment benefit. All three RCTs showed no evidence that hormone therapy was beneficial for women with established cardiovascular disease, and one RCT¹³³ found an increased risk of coronary events during the first year of HRT use.

Most experts would agree that RCTs minimize an important potential bias in the observational studies, namely selection bias. However, experts also prefer more studies with larger aggregate samples and/or with samples that address more diverse patient populations and practice settings -- often the hallmark of observational studies. The inherent tension between these factors is clear. The lesson we draw is that a system for grading strength of evidence, in and of itself and no matter how good it is, may not completely resolve the tension. Users, practitioners, and policymakers may need to consider these issues in light of the broader clinical or policy questions they are trying to solve.

Systems Related or Not Related to Development Of Clinical Practice Guidelines

Of the 34 non-EPC systems we evaluated for their performance in rating overall bodies of evidence, 23 addressed issues related to grading the strength of an evidence base for the development of clinical practice guidelines or treatment recommendations. The remaining 11 had not been derived directly from guideline development efforts per se. Interestingly, the first authors of all 11 of the non-guideline-derived systems are from outside the United States.^{11,12,39,70,84,85,87,90,117,124,126}

Based on the results of this project, it appears that the only U.S. investigators who currently grade the strength of the evidence, apart from those developing clinical practice guidelines or practice-related recommendations, are those affiliated with AHRQ's EPCs. We believe a useful follow-on to the present study might be to evaluate more directly all the strength-of-evidence approaches now being used in guideline development as well as non-guideline development activities. Such an effort might well entail review of considerable collections of gray literature -- for example, from the professional society's technical bulletins -- rather than purely peer-reviewed publications.

Emerging Uses of Grading Systems

Two of the 11 non-guideline-derived systems graded the strength of the evidence for a systematic review of risk factors for back and neck pain.^70,90 Narrative and quantitative systematic reviews are typically done for therapies, preventive services, or diagnostic technologies -- that is, to amass data that will inform clinical practice or reimbursement and coverage (policy) decisions. Traditional reviews are common for disease risk factors or health-related behaviors; evidence-based systematic reviews would be a likely next step as we move towards a greater reliance on evidence-based products for clinical or policy decisionmaking. Nonetheless, we are intrigued with this novel use of evidence grading for a systematic review on risk factors; it may foretell broader applications for systems of assessing study quality and evidence strength than has been seen to this point. Whether domains covered by extant rating and grading systems would need to be modified to take account of the types of research done to clarify risk factors is a matter of speculation and future research.

An example from the gray literature indicates that grading the strength of the evidence apart from the development of guidelines had been occurring even before the two risk evaluation studies^70,90 were published in the late 1990s. In 1994, the Institute of Medicine convened an expert panel to review the literature on the health effects of Agent Orange.¹³⁴ This team developed their own categorization system for grading the strength of this body of literature that also incorporated quality, quantity, and consistency.

As Guyatt and colleagues point out in their users' guides, summarizing the literature on treatment effects can (1) assist clinicians in treating their patients,^53,135 (2) help develop prevention strategies,¹³⁶ (3) resolve issues arising from conflicting studies of disease risk factors,⁹⁰ and (4) determine whether new treatments are worth their cost. Countries that have a national health service must identify ways to curb and prioritize health care spending, and many are turning to evidence-based practice to help them do so.

In the United States, we are beginning to see a rising emphasis on evidence-based practice and evidence-based policymaking. Like our foreign counterparts whose countries have national health plans, we may begin to see policymakers in public programs such as Medicare and Medicaid placing a greater reliance on systematic reviews -- and specifically systematic reviews that provide grades for the strength of evidence -- documenting the benefits (and harms) of preventive, diagnostic, and therapeutic interventions relevant to those beneficiary populations. The same may prove to be true for administrative leaders of integrated health systems and managed care organizations. Certainly, study quality and evidence grading will be important issues when comparisons need to be made of diagnostic or therapeutic options for a given disorder using cost-effectiveness methodologies.

Rating Article Quality

In reviewing Grids 1-4 (Appendix B), we can see that many systems cover many of the domains that we considered generally informative for assessing study quality. However, we did not believe this range of information provided sufficient practical guidance for users who want to know, today, where to start. Thus, we condensed the information to identify systems that fully or at least partially addressed what we regarded as key domains, and these systems -- largely scales and checklists -- are the ones that appear in the tables of Chapter 3.

More specifically, we identified five systems for evaluating the quality of systematic reviews, eight for RCTs, six for observational studies, and three for studies of diagnostic tests (see Tables 14, 17, 19, and 21, respectively). Summing across these sets, we arrived at a total of 19 unduplicated systems that fully address our critical quality domains (with the exception of funding or sponsorship for several systems).^{6-8,12,14,18,24,26,32,36,38,40,45,47,49,50,77,78}Three systems were used for both RCTs and observational studies.^14,40,45

Based on this iterative analysis, we feel comfortable recommending that those who plan to incorporate study quality into a systematic review or evidence report can use one or more of these 19 systems as a starting point, being sure to take into account the types of study designs occurring in the articles under review and the key methodological issues specific to the topic under study. We caution that systems ostensibly intended to be used to rate the quality of both RCTs and observational studies -- what we refer to as "one size fits all" quality assessments -- may prove to be difficult to use and, in the end, may measure study quality less precisely than desired.

We encourage those who will be incorporating study quality into a systematic review to examine many different quality instruments to determine which items will best suit their needs. We acknowledge that the resulting instrument will not be developed according to rigorous standards, but it will encompass domains that are important for the topic under evaluation. Other considerations for the selection and development of study quality systems include the topic to be reviewed, the available time for completing the review (some systems seem rather complex to complete), and whether the preference is for a scale or a checklist.

Rating Strength of Evidence

Systems for grading the strength of a body of evidence are much less uniform than those for rating study quality. This variability complicates the job of selecting one or more systems that might be put into use today. In addition, approaches for characterizing the strength of evidence seem to be getting longer or more complex with time. This trend stands in some contrast to that for systems related to assessing study quality, where the trend is for a reduction in the number of critical domains over time. This pattern may also reflect the fact that this effort is earlier on the development and diffusion curve.

Two other properties of these systems stand out. As discussed in Chapter 3, consistency has only recently become an integral part of the systems we reviewed in this area. We see this as a useful advance. Also continuing is the habit of using an older study design hierarchy to define study quality as an element of grading overall strength of evidence. As recently noted in methodologic work done for the U.S. Preventive Services Task Force, however, reliance on such a hierarchy without consideration of the domains we have discussed throughout this report is increasingly seen as unacceptable. We would expect, therefore, that systems for grading strength of bodies of evidence will increasingly call for quality rating approaches like those identified above.

As with the study quality systems, selecting among the evidence grading systems will depend on the reason for measuring evidence strength, the type of studies that are being summarized, and the structure of the review panel. Some systems appear to be rather cumbersome to use and may require sufficient staff, time, and financial resources. Again, for users, researchers, and policymakers uncertain about which among these seven might best suit their needs, we suggest also applying descriptive information from Grid 5B in the decisionmaking.

Introduction

An important element of this project was to summarize how the 12 evidence-based practice centers (EPCs) supported by the Agency for Healthcare Research and Quality (AHRQ) rated individual study quality and graded the strength of a body of evidence for their various evidence reports and technology assessments. The initial step in gathering information was for the AHRQ EPC Program Officer to ask the EPCs, on behalf of the team from the Research Triangle Institute-University of North Carolina (RTI-UNC) EPC, to identify the methods they used in these steps, assuming they did them at all. To assist in this process, RTI-UNC EPC staff reviewed the methods sections and appendices of all published evidence reports done by the EPCs for relevant information and then included this information with the form from the AHRQ Program Officer (Exhibit A-1) that asked the EPCs to specify how they handled quality ratings and evidence strength grading for their initial, subsequent, and current evidence reports and technology assessments. Several EPCs chose to summarize their procedures for us in a memorandum. We compiled the information (see Tables A-1 and A-2) and incorporated it into the appropriate grids (Appendices B and C).

Findings

Of the 12 EPCs, 10 did formally evaluate quality of articles in some fashion. Those that did applied numerous different techniques (Table A-1), and some based their quality assessments on study design only. Those that formally evaluated quality and developed a quality score employed several key study design components either as part of their inclusion/exclusion criteria or as components in their meta-analyses.

EPCs used quality ratings in several different ways:

1. As a factor in sensitivity or meta-analyses (Blue Cross and Blue Shield Association, Johns Hopkins University, New England Medical Center, Duke University, University of California at San Francisco-Stanford, University of Texas at San Antonio);

2. Descriptively in the evidence tables, results, and/or discussion section of their evidence reports (ECRI, McMaster University, Oregon Health Sciences University, RAND-Southern California, RTI-UNC); and

3. As inclusion/exclusion criteria for the literature searches of their evidence reports (MetaWorks, Inc.).

The data provided in Table A-2 are based on the completed surveys we received from each of the EPCs. Little changed over time with respect to whether and how the EPCs rated study quality. Five EPCs graded the strength of bodies of evidence in their first EPC projects, and the same five currently grade evidence strength.

Exhibit A-1. Information Requisition Letter to EPCs

July 7, 2000

Dear EPC Directors and staff:

Thank you so very much for participating in our phone call with all the EPCs on May 22. As was discussed during the call, the RTI-UNC EPC has a very exciting but somewhat daunting task ahead of them and they need as much help from their fellow EPCs as they can possibly get!

The RTI-UNC EPC has organized an absolutely wonderful expert panel for this project. They include:

• Doug Altman

• Lisa Bero

• Alan Garber

• Steven Goodman

• Jeremy Grimshaw

• Alejandro Jadad

• Joseph Lau

• David Moher

• Cynthia Mulrow

• Andy Oxman

• Paul Shekelle

As Sue West indicated on the call, she has already reviewed the published AHRQ evidence reports (ERs) to identify the rating scales and methodologies for grading the evidence that were used by each EPC for their first ER (please see attached spreadsheet indicating which reports were reviewed). If information was available on rating scales or grading classifications from your ER(s), we are including a copy of the specific pages from your report with this letter. Please review this attached information to make sure that it accurately reflects the procedures you used at that time. Also, several of you very graciously provided Kathleen Lohr with information for her earlier project on the issues involved in grading articles and evidence so you certainly do not need to re-send this to the RTI-UNC EPC!

As the spreadsheet indicates, all of the EPCs have been funded to develop additional ERs. Your procedures may have changed somewhat as you worked on subsequent ERs. We would appreciate if you would share your procedures and full documentation that indicates how you are currently rating the quality of studies and grading the evidence so that the RTI-UNC EPC can document this in their report to AHRQ.

This information can be sent directly to Sue West at the following address:
Suzanne L. West, Ph.D., M.P.H.
Cecil G. Sheps Center for Health Services Research
725 Airport Road CB# 7590
University of North Carolina
Chapel Hill, NC 27599-7590

Alternatively, you can email to Sue_West@med.unc.edu or fax to 919-966-5764.

If you have suggestions for other scales or grading schemes or people to contact for this information, please provide this to Sue as well.

In their first ER, Pharmacotherapy for Alcohol Dependence, the RTI-UNC EPC did grade the evidence for their 5 key questions. With this letter, we have included copied pages from their evidence report that give their procedures as an example of what is meant by "grading the evidence" in the context of the current project, "Systems to Rate the Strength of the Scientific Evidence."

If you have any questions or need further guidance regarding your contributions to the RTI-UNC project, please don't hesitate to call Sue at 919-843-7662. Because of the timeline for this project, it would be great if you could send your information to Sue West by Friday, July 21. In replying to Sue, please include this letter and check the appropriate box indicating which information you are sending (or not sending!) to UNC. We (AHRQ and the RTI-UNC EPC) really appreciate your assisting the RTI-UNC EPC with this project.

• 1997 ER (first ER) for AHRQ

Rating study quality

	Contained forms and description for rating the quality of individual studies		Rating and description is being sent to the RTI-UNC EPC   This info is not available to send
	Did not contain information on rating the quality of individual studies

Grading the evidence for key questions

	Contained information on grading the evidence		This info is being sent to the RTI-UNC EPC   This info is not available to send
	Did not contain information on grading the evidence
Subsequent ERs for AHRQ or other funding sources

Rating study quality

	Contained forms and description for rating the quality of individual studies		Rating and description is being sent to the RTI-UNC EPC   This info is not available to send
	Did not contain information on rating the quality of individual studies

Grading the evidence for key questions

	Contained information on grading the evidence		This info is being sent to the RTI-UNC EPC   This info is not available to send
	Did not contain information on grading the evidence
Are you currently:

Rating study quality?

Yes

No

Grading the evidence for key questions?

Yes

No

Legend:  Yes  Partial  No

Thank you, in advance, for your help!
Sincerely, Jacqueline Besteman

cc: Kathleen N. Lohr, PhD
Valerie King, MD
Suzanne L. West, PhD, MPH

encl: EPC-specific pages from first evidence report
Pages from RTI-UNC Alcohol Pharmacotherapies evidence report
Spreadsheet with all projects
4-page project summary

Topic and Nominators and Partners by EPC	How Was Quality Measured in the Report?	How Was Quality Used in the Report?
Blue Cross and Blue Shield Association Technology Evaluation Center (TEC)
Relative Effectiveness and Cost-Effectiveness of Methods of Androgen Suppression Treatment in the Treatment of Advanced Prostatic Cancer Health Care Financing Administration	Assessed the quality of methods and reporting to determine whether the studies could be grouped into categories by grade of methodologic quality. Factors assessed included:       Random sequence generation     Blinding of randomization process during recruitment     Blinding of investigator and patient to treatment     Study withdrawals     Intent to treat     Power     Compliance with treatment     Description of treatment protocols Formal quality rating was not given, component approach provided on evidence tables.	Quality used for sensitivity analyses. Meta-analysis combined hazard ratios for studies of "high" quality but "high" was not defined.
Duke University
Evaluation of cervical cytology American College of Obstetricians and Gynecologists	Quality criteria for diagnostic tests Test and reference measured independently Test compared to valid reference standard Choice of patients for reference standard independent of test results Sample selection addressed Location of publication Funding source Consensus on the seven quality points, weight determined by consensus and averaging	Evaluated the effect of study quality on summary effectiveness scores using individual components of the score, then using the total score both as a continuous and dichotomous (cutpoint 7).
ECRI
Diagnosis and treatment of dysphagia/swallowing problems in elderly Health Care Financing Administration	Quality measured by study design.	Study design reported in evidence tables and discussed in the results section of the evidence report.
Johns Hopkins University
Evaluation and treatment of new onset atrial fibrillation, in the elderly American Academy of Family Physicians	22 questions, major domains listed below:       Thoroughness of population description     Bias and confounding (description of randomization and blinding     Standard protocol, other therapies received     Outcomes and follow-up     Statistical quality and interpretation	The EPC noted that it would have used study quality in a sensitivity analysis but there were too few studies.
McMaster University
Treatment of attention deficit/hyperactivity disorder American Academy of Pediatrics, American Psychiatric Association	Quality was based on the Jadad scale for randomized controlled trials     Randomization     Blinding     Withdrawals     Industry support	Authors used quality to describe results and conclusions.
MetaWorks, Inc.
Diagnosis of sleep apnea Blue Cross/Blue Shield of Massachusetts, Sleep Disorder Center of Metro Toronto	Diagnostic studies rated by Irwig instrument before data abstraction       Random order of assignment       Use of a gold standard       Blinded reading of test and gold standard	Quality score ranged from 0-44; papers with a quality score of <16 were not abstracted.
New England Medical Center
Diagnosis and treatment of acute bacterial rhinosinusitis American Academy of Otolaryngology, American Academy of Family Practice, American Academy of Pediatrics, American College of Physicians	Quality was based on the Jadad scale for randomized controlled trials       Randomization     Blinding     Withdrawals	Quality was used for sensitivity measure in meta-analysis       Use of a composite quality score       Use of factor(s) that relate to systematic bias
Oregon Health Sciences University
Rehabilitation of persons with traumatic brain injury National Institute of Child Health and Human Development Brain Injury Association	Levels of quality   Class I: properly designed RCTs   Class II a: RCTs with design flaws or multicenter or population-based longitudinal (cohort) studies b: Controlled trials that were not randomized, case-control studies, case series with adequate description of population, intervention, outcomes   Class III: descriptive studies, expert opinion, case reports, clinical experience	Quality levels were used descriptively in the results and conclusions section of the report.
RAND-Southern California Evidence-based Practice Center
Prevention and management of urinary complications in paralyzed persons Paralyzed Veterans of America, American Association of Spinal Cord Injury Psychologists, American Congress of Rehabilitation Medicine, American Paraplegia Society, Association of Rehabilitation Nurses, Consortium for Spinal Cord Medicine	Quality was based on the Jadad scale for randomized controlled trials   Randomization   Blinding   Withdrawals Cohort studies:   Comparability at baseline or whether adjustments made during analysis   Masked measurement of outcomes and risk factors	Quality grades were reported in evidence tables.
Research Triangle Institute -- University of North Carolina, Chapel Hill
Pharmacotherapy for alcohol dependence American Society of Addiction Medicine	Quality rating score adapted from scoring for spinal cord clinical guideline.	Authors reported quality scores in evidence tables and used them descriptively for results and conclusions.
University of California at San Francisco/Stanford University
Management of stable angina American College of Cardiology/American Heart Association Task Force on Practice Guidelines/American College of Physicians	Four indicators:   Randomization   Blinding   Description of randomization method   Mention of exclusions	Authors used quality ratings in subgroup analyses.
University of Texas at San Antonio EPC
Depression treatment with new drugs National Institute of Mental Health, American Psychiatric Association, American Pharmaceutical AssociationVermont Department of Mental Health/Mental RetardationBlue Cross/Blue Shield of Massachusetts, American College of Physicians, Kaiser Permanente of Northern California	Internal validity used instead of quality   Randomization (method and concealment)   Blinding   Co-interventions   Dropouts	Authors used the dropout rate in meta-analysis looking at response rates.

EPC	Subsequent Evidence Reports for AHRQ or Others		Current Practice
	Rating Study Quality	Grading the Evidence for Key Questions	Rating Study Quality	Grading the Evidence for Key Questions
Blue Cross and Blue Shield
Duke University
ECRI
Johns Hopkins University
McMaster University
MetaWorks, Inc.
New England Medical Center
Oregon Health Sciences University
Southern California Evidence-based Practice Center-RAND
RTI-UNC
UCSF-Stanford
UT-San Antonio

Legend:  Yes  Partial  No

Refer to Table