Background

Evidence-based Practice Center (EPC) evidence reports and technology assessments aim to review the relevant scientific literature and to evaluate clinical and behavioral interventions, prognostic and diagnostic tools, health care utilization, and other health care organization and delivery issues. ¹ These reports are used for informing and developing coverage decisions, quality measures, educational materials and tools, guidelines, and research agendas. The reports are based on rigorous, comprehensive syntheses (systematic reviews) and analyses (meta-analyses) of the scientific literature² on topics relevant to clinical, social science/behavioral, economic, and other health care organization and delivery issues. In most circumstances, controlled trials, and especially randomized controlled trials (RCTs), are the study design least likely to produce biased estimates of the effect of an intervention;³ however, alternative study designs are often needed to capture information important to clinicians and other end-users of the reports, particularly when controlled trials are lacking, not appropriate for the outcomes or questions of interest (e.g., long-term outcomes, rare outcomes, adverse effects), or not generalizableto a broader population.⁴ In the context of such reviews, the appropriate classification of studies according to their design or design features is important in order to guide (1) decisions around inclusion, (2) the assessment of methodological quality or risk of bias, (3) the combining of study results in a narrative synthesis or by statistical pooling and (4) grading the body of evidence.

The Use of Nonrandomized Study Designs

RCTs are considered the gold standard for judging therapeutic efficacy and effectiveness;^5,6 however, RCTs are often unnecessary, inappropriate, impossible or inadequate to address particular research questions.^6–8 Given the wide range of topics addressed by EPC reports, researchers frequently need to include nonrandomized studies in order to provide “a more detailed picture of our current knowledge and its limitations for clinicians and policymakers.”⁹ This is especially true for outcomes that RCTs may not be adequately designed to address (e.g., adverse effects)¹⁰ and in areas in which few RCTs have been conducted, such as devices and surgical procedures (where RCTs account for less than 10 percent of the evidence base)¹¹ or educational interventions (e.g., medical education ¹²).

In the past, many EPC reports have restricted evidence to RCTs when addressing effectiveness questions in order to protect against sources of bias in other study designs even though the extent of bias associated with different nonrandomized designs varies and the direction and magnitude of bias can be unpredictable.^5,13–15 When there are few well-conducted RCTs, however, end-users are often dissatisfied if a report concludes that the evidence is insufficient without considering other study designs, especially since these other designs may be the very studies that are influencing current practice and policy debates.¹⁶ As a result, a past criticism of EPC reports is that they are too restrictive in their consideration of evidence.¹⁶

The fact that a high proportion of published intervention studies use nonrandomized designs is taken as evidence that this research is valued by clinicians.¹⁷ Moreover, some researchers believe that including nonrandomized designs lacking control groups may increase the evidence base concerning a health technology and strengthen the credibility of a review. Even if evidence to evaluate the effectiveness or harm is lacking, nonrandomized studies may provide evidence to guide the development of future research questions. There is also value in reviewing nonrandomized studies to clearly describe their limitations and to recommend the types of studies that would provide better evidence.^12,16

There is evidence that EPC reports that address the efficacy or effectiveness of a clinical intervention are becoming more inclusive in terms of designs other than RCTs.⁴ In addition, there is increasing attention and funding directed toward comparative effectiveness reviews (CERs) and an accompanying broadening of perspective;¹⁸ the topics identified by the AHRQ for CERs are broad and require practice-based evidence, which, if available, is often collected through nonrandomized studies. A particularly important and related principle is the move from efficacy to effectiveness.¹⁹ These changes in the focus of EPC work heighten the urgency for evaluating current practices in evidence synthesis for nonrandomized studies. The starting point in such a synthesis is the classification of study design.

Classification of Study Designs

The diversity of study designs and the similarities among them present challenges to reviewers who wish to clearly classify designs or design features for the purpose of assessing the strength of the evidence regarding a particular intervention. Textbooks in systematic reviews,⁵ epidemiology,²⁰ and social science research ²¹ as well as health technology assessment (HTA) reports⁷ provide detailed descriptions of study designs commonly used in those areas; however, none of these resources on its own provides a comprehensive treatment of all the design types encountered when evaluating studies within EPC reports, nor do any provide convenient summaries or decision rules for distinguishing among designs. Perhaps most troubling for systematic reviewers is the fact that the terminology and classification systems used to describe different nonrandomized studies are inconsistent.^9,14 This inconsistency is problematic because researchers often make decisions about research approaches and the interpretation of results on the basis of study design classification and labels. The use of a variety of similar and ambiguous study classifications (e.g., a “prospective study”) may lead to low sensitivity in study identification and inaccurate quality assessment of the conduct of studies.²² This results in uncertainty about the study designs that were used to address the research questions and how the evidence provided by these studies should be weighed.

An important goal of the EPC program is to advance the methods for conducting and reporting systematic reviews.¹⁶ Though much research has focused on systems for grading evidence,^23,24 and a variety of approaches have been developed for evaluating the quality of nonrandomized studies, less attention has been paid to the importance of developing a standard classification scheme or nomenclature system or an algorithm to correctly classify study designs. Research has demonstrated poor agreement among reviewers using a “traditional” study design classification scheme in epidemiology (called a “taxonomy” by the researchers) to classify a set of studies in the field of low back pain.¹⁴ The assignment of study design labels was also found to be unreliable, even when specific instructions and definitions were provided.¹⁴

The appropriate classification and assessment of a study’s design relies heavily, though not exclusively, on the adequacy of reporting by a study’s authors. Without an indication of the elements considered crucial for labelling studies, assessing the appropriateness of the study design, and assessing a study’s strengths and potential weaknesses, study authors are likely to omit these important features. For example, the Strengthening of Reporting of Observational Studies (STROBE) statement,⁶ a guideline that promotes transparency in reporting analytic observational studies, encourages authors to “indicate the study design with a commonly used term” and “to present key elements of study design”.⁶ However, the document provides limited guidance on the elements needed to adequately conduct the above processes. A standard classification tool may help authors to identify these crucial study design features, thus improving the transparency of reporting.

Finally, the development of a standard classification tool may help to streamline and facilitate the process of scoping the available scientific literature and of deciding what methodologies should be considered for any particular review. Accurate study design classifications are also important for the efficient and accurate communication of the results of a systematic review. A classification tool may also complement other approaches to defining study designs or design features that are important to consider when evaluating the strength of evidence.

Objectives of the Project

The University of Alberta Evidence-based Practice Center (UAEPC) undertook this project to identify a valid and reliable tool for the classification of randomized and nonrandomized studies of interventions and exposures that could be used in the conduct of systematic reviews.

The specific objectives were

1. To identify classification tools that are currently used by systematic reviewers and others to identify studies according to design.
2. To select a classification tool for modification and evaluation.
3. To develop instructions, including an algorithm and decision rules, for application of the modified tool to studies of interventions and exposures.
4. To test the tool and accompanying instructions for concurrent validity and inter-rater reliability.