Introduction

Emphasis on understanding the comparative effectiveness of health care interventions has increased attention on the applicability of research. As a result, systematic reviews aiming to inform clinical practice have expanded beyond randomized controlled trials, which have limited generalizability, to include nonrandomized studies. However, different nonrandomized study designs differ in their relative strengths and weaknesses. A standard nomenclature and taxonomy for categorizing these nonrandomized studies may help promote a common language and understanding among Evidence-based Practice Centers (EPCs) and other systematic reviewers regarding the inherent strengths and weaknesses of particular study designs.

The development of a standard classification tool may help to streamline and facilitate scoping of evidence and making decisions as to what kind of evidence should be considered for any particular review. Accurate classifications by study design are also important for the efficient and accurate communication of the results of a systematic review.

The primary objectives of this Methods Research Paper were:

1. To identify classification tools that are currently used by systematic reviewers and other researchers to identify studies according to design.
2. To select a classification tool for modification and evaluation.
3. To develop instructions, including an algorithm and decision rules, for application of the modified tool to studies of interventions and exposures.
4. To test the tool and accompanying instructions for concurrent validity and interrater reliability.

Methodology

A Steering Committee of seven members from three EPCs (University of Alberta, McMaster University, RTI International–University of North Carolina [UNC] at Chapel Hill) and the Agency for Healthcare Research and Quality was formed to guide the methods and direction of the project.

Results

Objective 4

Reference Standard

Three members of the Steering Committee developed the reference standard. Each member independently applied the flow diagram to assign design labels to the 30 studies. The initial agreement was fair (κ=0.33). Disagreements were resolved through discussion and consensus.

Disagreements occurred at most decision points in the algorithm. The area that created the greatest uncertainty and disagreements for the reference standard raters was the decision node “Was there a single cohort?” The initial decision node (“Was there a comparison?”) was also a source of disagreement.

Test 1

Tester characteristics. Six staff members at the UAEPC with varying levels of training and experience in systematic reviews tested the modified taxonomy.

Agreement. There were no studies for which all six testers agreed on the classification (Table A). Five of six testers agreed on the classification of seven studies, four agreed on five studies, three agreed on nine studies, two agreed on eight studies. The overall level of agreement was fair (κ=0.26). The levels of agreement for testers who had completed vs. those who were undertaking graduate-level training were fair (κ=0.38) and slight (κ=0.17), respectively.

Table A

Results of taxonomy testing.

Disagreements occurred at all decision points in the taxonomy; however, testers identified the determination of whether there was a single cohort as particularly problematic. The testers also said that certain terminology in the flow diagram was unclear (e.g., “group” vs. “cohort”) and that disagreements arose due to poor study reporting. There was some variation in the manner in which testers used the flow diagram (e.g., whether or not they used the glossary, working forward vs. backward through the algorithm).

Accuracy of testers compared to reference standard. There were no studies for which all six testers agreed with the reference standard, and there was wide variation in the testers’ accuracy of classification.

Test 2

Tester characteristics. Six staff members at the UAEPC with varying levels of training and experience in systematic reviews were involved in the second round of testing. Three of the testers had been involved in the first round of testing, and three of the testers had no previous involvement with the project or knowledge of the taxonomy being tested.

Agreement. There were three studies for which all six testers agreed on the classification (Table A). Five of six testers agreed on two studies, four agreed on six studies, three agreed on two studies, and two agreed on two studies. The overall level of agreement was considered moderate (κ=0.45). The levels of agreement for testers who had completed vs. those undertaking graduate-level training were moderate (κ=0.45) and fair (κ=0.39), respectively.

Accuracy of testers compared to reference standard. There were three studies for which all six testers agreed with the reference standard, but there was wide variation in the testers’ accuracy of classification.

The least common agreement occurred at four key decision nodes: whether the study was “experimental” (5/15 studies), whether there was a comparison (4/15 studies), whether the assessment of exposure and outcome was prospective or retrospective, and whether the intervention or exposure and outcome data were gathered concurrently (2/15 studies).

Discussion

We identified over 20 tools and selected 1 for modification and testing. The final testing of the modified tool showed moderate agreement among six testers and low accuracy against the predetermined reference standard. The moderate level of agreement is consistent with that observed in a previous study.

There are a variety of reasons for the moderate and low levels of agreement and accuracy observed in our study. The results likely reflect issues with the taxonomy itself, as well as attributes of the studies that were selected for testing. The studies used during testing were identified and selected because they had posed challenges for previous reviewers with respect to their design classification. Agreement might be better with a sample of studies that is more representative of the studies that would be included in a systematic review. Further, the sample of studies that we tested covered a wide range of topics. If the studies had been on the same topic, which would be the case in a systematic review, there might have been greater reliability. One of the main reasons that the selected studies were difficult to classify was poor reporting within the studies, which resulted in the need for testers to make assumptions (e.g., whether the timing of a study was prospective or retrospective). We also found classification challenging when there were discrepancies between the intent of the investigator and the conduct of the study, between the design and how data were analyzed, and between the investigators’ initial plan and study implementation.

Shortcomings of the taxonomy itself also resulted in moderate agreement. Many of the decision points were challenging. For example, in one-third of the studies, there were discrepancies as to whether or not the study was truly “experimental.” Identifying “quasi-experimental” studies is challenging, as the investigator has some control over certain aspects of design and study execution, and the study may not be considered either purely experimental (a “trial”) or purely observational. This area of study design needs to be more clearly reflected in the taxonomy, and clear guidelines are needed for interpreting the extent of control an investigator has. The practical repercussion of this uncertainty in classification is that some “quasi-experimental” studies (e.g., before-after or controlled before-after studies) may incorrectly be classified as trials; hence, their validity may be exaggerated and the results given too much weight in the context of a systematic review. One design that is particularly problematic has been variously referred to as an “uncontrolled trial” or “single-arm trial.” It is our opinion that this design should not be considered a “trial” because of the serious risk of bias associated with the lack of a control or comparison group. Consequently, studies with such a design should be considered “before-after” studies, and our taxonomy was designed to channel them toward this classification.

Other decision nodes that yielded inconsistent results concerned whether there was a comparison, whether the study was experimental, and whether the data collection was prospective or retrospective. Several factors may have contributed to this inconsistency, including a lack of clarity in the questions posed in the algorithm, the testers’ relevant background knowledge, the testers’ experience or training, and the inconsistent use of design terminology among the studies. While we provided a glossary in an attempt to offer standard definitions and clarity in terminology, there may have been shortcomings with the glossary that created confusion, including ambiguity and inconsistency in terms and definitions. Testers who had completed relevant graduate-level training had greater agreement than those who were undertaking graduate-level training.

We observed a fair level of agreement among the reference standard raters as well. The three reference standard raters had substantial expertise in research methods and systematic reviews. The low level of agreement among these raters may reflect the more general complexities of study designs and the challenge of including all design considerations in a single flow diagram.

Variability in classification of studies may also reflect differences in how individuals applied or worked through the taxonomy. For example, some testers worked backward or backtracked in order to classify the studies according to what they felt was the most appropriate description. The testers also used the glossary accompanying the tool to varying degrees.

The difficulties in interpreting study design labels and the consequent difficulties in reaching agreement in assigning these labels to individual studies are consistent with those of other researchers. These issues have led some authors to direct systematic reviewers to focus on features of designs rather than on design labels when assessing studies for inclusion and evaluating potential risk of bias. The use of a taxonomy may provide greater transparency and consistency to the process.

Implications for Practice

The appropriate classification of studies by design or by design features is important in a systematic review in order to guide the selection of studies, the assessment of the risk of bias, the analysis of study results, the interpretation of results, and the grading of the body of evidence. There is a clear need for consistent use of terminology and study design labels, as well as a clear understanding of the terminology used in a particular field by those undertaking a systematic review in that field. We believe that a tool such as the one developed and tested in this study would be useful to guide this process, although the application of the tool requires several considerations in order to optimize agreement and reliability among reviewers. First, training in research methods, as well as in the use of the tool, is essential. Pilot testing the tool in the context of each review is highly recommended. Second, decision rules are needed for different fields of research or review topics. Specifically, there need to be clear decisions about how to handle a lack of clarity in study reporting. We recommend that when the response to a question in the taxonomy is unclear, the reviewer assume that the condition was not met. Documentation of the decision rules will allow for consistency and transparency. Users of the algorithm need to use standardized definitions of study designs and design features.

Future Research

The tool developed and tested in this study serves as a basis for further research. Future research is needed to evaluate the tool within the context of a real systematic review, as well as to conduct more indepth testing for specific study designs that are difficult to classify. Additional critical review and refinement of the accompanying glossary are needed. We provide some preliminary data on factors that might create differences in reliability across individuals, including varied experience, training, and education. Further research is needed to provide more definitive results about these various factors and how they impact the performance of the tool. Our experiences also provide direction for methods to be employed in subsequent work, such as development of the reference standard (e.g., involving senior researchers with epidemiological training) and contacting authors for clarification when methods within the individual studies are unclear.

Conclusions

We developed and tested a taxonomy for the classification of study designs. The level of agreement among six testers was moderate and the accuracy against a reference standard was low. There are a number of explanations for the observed reliability and accuracy, including shortcomings of the taxonomy and accompanying glossary, inadequate reporting of the studies, and differences in tester characteristics. Application of such a tool in the context of a systematic review should be accompanied by adequate training, pilot testing, and documented decision rules. This study demonstrates that systematic testing and refinement enhance the reliability of the tool. At the study level, clear reporting, adherence to published reporting guidelines, and appropriate and consistent use of design terminology should be enforced. Table B is a summary of study findings, implications for practice, and directions for further research.

Table B

Summary of findings, implications for practice, and directions for future research.