The goal of this project was to identify a tool that could be used within the context of systematic reviews to assist with the classification of study designs. This study builds on previous work to test a “traditional taxonomy” in the area of interventions for low back pain.¹⁴ The previous study suggested a number of directions for further research including developing a more comprehensive taxonomy in terms of the scope of study designs, and testing the taxonomy in different fields of research. We identified over 20 tools and selected one for modification and testing. One of the critical criteria in the selection process was comprehensiveness of the tool in terms of study designs. The final testing of the modified tool showed moderate agreement among six testers and low accuracy against a predetermined reference standard. The moderate level of agreement is consistent with that observed in the previous study.¹⁴ The level of agreement observed in these two studies raises questions and concerns around the reliability, validity, and ultimately the utility of available classification tools. There are numerous tools in existence and, to our knowledge, few (if any) have undergone testing either during or after development. However, our findings also demonstrate that it is possible to systematically test and modify a tool in order to yield more reliable results.

There are various reasons for the moderate and low level of agreement and accuracy observed in our study. In general, it is difficult to determine the extent to which the results reflect issues with the taxonomy itself versus attributes of the studies that were selected for testing. The studies used during testing were identified and selected because they had posed challenges with respect to design classification within various systematic reviews. It might be expected that the agreement would be better among a more representative sample of all studies that would be included in a systematic review. The complexity of studies is partially reflected in the time taken to classify each study (16 minutes in Round 1 and 11 minutes in Round 2 compared to 3 to 8 minutes in a similar study that was restricted to one topic area¹⁴). Further, in the context of a systematic review, the challenges of study design classification will vary by topic depending on the designs included and the general state of the literature in the area. The sample of studies that we tested covered a wide range of topics. There may have been greater reliability if the studies had been on the same topic.

One of the main reasons that the selected studies were difficult to classify was poor reporting within the studies. This resulted in the need for testers to make assumptions or judgments in many cases (e.g., whether the timing of a study was prospective or retrospective). There is an urgent need for clearer reporting and stricter adherence to reporting guidelines, and these should be enforced at the journal level through the editorial and peer review process. However, these changes will have limited impact on systematic reviewers who typically review studies done in the past. In certain cases, we found that the classification could vary depending on the intent of the authors and this was not regularly clear from the written report. In addition, there were cases of a discrepancy between how the study was designed and how the data were analyzed. Consequently, the classification could vary depending on the focus and interpretation of the individual assigning the design. Further, studies may use one particular design for some outcomes and a different design for other outcomes (e.g., RCT for primary or short-term outcomes, subgroup analysis using a prospective cohort approach for safety or longer-term outcomes).

Clearly the observed level of agreement and accuracy may be due in part to shortcomings of the taxonomy itself. Many of the decision points were challenging. This may in part reflect some lack of clarity within the field of epidemiology. For example, one study found substantial variation in the interpretation and understanding of blinding across 25 textbooks and a sample of physicians.³⁹ It may also partly be explained by the evolution of study designs over time; designs are becoming more complex and incorporating mixed methods. We aimed to address Furlan’s observation that a useful taxonomy should be comprehensive in terms of study designs. One of the design categories that Furlan¹⁴ found lacking in the “traditional taxonomy” was “quasi -experimental” studies; this led to one of the main problems she describes in terms of difficulty choosing between the only two options of “experimental” and “observational” designs. Furlan further described three categories of studies: experimental (RCTs, CCTs), observational (cohort, case-control, cross-sectional), and descriptive (case series, case reports). This classification completely overlooks the area of “quasi-experimental” designs where the investigator has some control over certain aspects of design and study execution but the study may not be considered either purely experimental (a “trial”) or purely observational. While our intent was to capture these “quasi-experimental” designs using more precise methods (e.g., study design features) and terminology, this was not overtly apparent to the testers. This resulted in discrepancies in a third of the studies as to whether or not the study was truly “experimental.” Of note is the fact that two of the testers labelled several studies as “quasi-experimental” even though this was not a design label in the flow diagram. These study designs need to be more clearly reflected in the taxonomy, and clear guidelines for interpreting the extent of control that an investigator has are needed. The practical repercussion is that some “quasi-experimental” studies (e.g., before-after or controlled before-after studies) may incorrectly be classified as trials; hence, their validity may be exaggerated and the results given too much weight in the context of a systematic review. One design that is particularly problematic has been variously referred to as an “uncontrolled trial” or “single-arm trial.” It is our opinion that this design should not be considered a “trial” because of the high risk of bias associated with having no control or comparison group. These studies should be considered “before-after” studies, and our taxonomy was designed to channel them toward this classification.

Many of the questions or decision nodes that reviewers might consider relatively straightforward (e.g., was there a comparison, was the study experimental, was the data collection prospective or retrospective) did not yield consistent responses. There are several factors that may contribute to this inconsistency including a lack of clarity or definitions within the questions posed in the algorithm; variation in the level of background knowledge, experience or training of those classifying the studies; and, an inconsistent use of design terminology among studies. While we provided a glossary in an attempt to offer standard definitions and clarity in terminology, we acknowledge that there may have been shortcomings with the glossary that created confusion. In retrospect, we found there was some ambiguity and inconsistency in terms and definitions. For instance, the terms “group,” “cluster,” and “observation” required greater clarity and consistency across the definitions. Further, definitions for the different types of cohort studies could be revised for more consistency. These challenges with respect to terminology were discussed in the epidemiology literature as early as the 1950s and have yet to be resolved.^22,40

Inconsistent, inaccurate, or imprecise use of terminology can be confusing. For example, a sample may be randomly selected but not necessarily randomly assigned to treatment groups; therefore, the use of the word “random” does not necessarily mean that the study is an RCT. In addition, the term “cohort” was considered problematic, despite the fact that a definition had been provided in the accompanying glossary. This raises the issue that the same terms can refer to different things (e.g., “cohort” may describe a group of people or a study design) and clearly distinguishing between the common and technical meanings of terms may not always be sufficient to prevent potential ambiguity or confusion. The inappropriate use of terminology by the authors of the research studies also creates confusion for the reader. For example, one study was described by the authors as a “case-control”; however, the reference standard considered it a controlled before-after study. The terms “prospective” and “retrospective” are often used loosely or in an ill-defined manner in the literature⁴¹ and may refer to different constructs, including “directionality” (i.e., “temporal relationship between the observation of study factor level and the observation of disease status”⁴²) and “timing” (i.e., “chronological relationship between the onset of the study and the occurrence of the primary phenomena under study”⁴²). Reporting guidelines for observational studies recognize these different dimensions and recommend that authors refrain from using “prospective” and “retrospective” in favor of an explicit description of these dimensions of the study.⁴¹ In some cases, variation arises from terminology specific to different fields or similar terminology used variably across fields (e.g., social sciences, education, psychology, medicine). There is a clear need for consistent use of terminology and study design labels and/or an understanding of the terminology used in a particular field by those undertaking a systematic review in that field.

One practical reason for the low agreement is the relatively large number of testers we chose and the large number of potential response categories. Although the statistical tests accommodate for this to some extent, the more testers and response categories there are the greater the likelihood of disagreement. Restricting the testing to two or three individuals may have yielded better results and more closely replicated the systematic review process. Nevertheless, the fact that six testers with systematic review experience and relevant training showed moderate agreement is problematic. Moreover, the three investigators with doctoral training in epidemiology or research design who developed the reference standard showed fair agreement. These observations go beyond the studies and taxonomy used in this study and reflect the more general complexities of study designs. Perhaps it is unreasonable to distill the myriad design elements that populate textbooks into a single flow diagram. The fact that 15 people assigned eight different designs to the same study¹⁴ further highlights the complexity and variability of study design classification.

Variability in design classification and moderate agreement may also reflect differences in how individuals applied or worked through the taxonomy, or as Furlan described “the creativity of humans.”¹⁴ Testers commented on the presentation of the tool in terms of using letter codes rather than study design labels. The use of letter codes was done intentionally in an effort to increase the probability that testers would work through the flow diagram and answer each question sequentially, rather than place the studies into the categories they deemed appropriate based on study design label. Based on the feedback we received from the testers, they often worked backwards or back-tracked in any event in order to classify the studies according to what they felt was the most appropriate description. The reliance on study design labels, and the inappropriate or inaccurate use of study design labels or terminology by authors, has implications for the application of these types of tool. The glossary that we developed to accompany the tool should mitigate these inconsistencies to some extent; users of the algorithm need to use standardized definitions provided.

The difficulties in interpreting study design labels and the consequent difficulties in reaching agreement in assigning these labels to individual studies are consistent with those of other researchers. These issues have led some authors to direct systematic reviewers to focus on features of designs rather than on design labels when assessing studies for inclusion and evaluating potential risk of bias.³ We endorse this approach and recommend that reviewers should be as explicit as possible about the design features that are being considered. However, we do not think that this obviates the need for or the usefulness of some design labels in describing studies being considered for inclusion in systematic reviews. AHRQ reports tend to have broad inclusion criteria and, as a pragmatic issue, often require the consideration of design features as part of the inclusion/exclusion process. The use of a taxonomy is intended to provide greater transparency and consistency to the process by closely examining the design features. In addition, many reviewers still find it useful to be able to describe and categorize studies according to broad design rather than referring only to specific features.³ Moreover, groups that do not advocate the use of study design labels continue to recognize the importance of considering inherent weaknesses in design features. We believe that design labels allow reviewers to keep issues regarding inherent weaknesses in mind while retaining the ability to categorize studies both broadly and according to specific design features. Whether design labels and the differences between studies that they help to identify are also useful for assessing risk of bias has not been well investigated empirically. We consider the results of this study as a contribution to the developing evidence base that may help guide this larger discussion about the utility of design labels for assessing the quality of evidence.

Summary

We developed and tested a tool for the classification of study designs. The level of agreement among six testers was moderate and the accuracy against a reference standard was low. There are a number of explanations for the observed reliability and accuracy including possible shortcomings of the taxonomy (e.g., lack of clarity and comprehensiveness); inadequate study reporting (e.g., poorly described methods, inaccurate and inconsistent use of terminology); and variation in user characteristics (e.g., training/education/experience, preferences, application of the tool). Application of such a tool in the context of a systematic review should be accompanied by adequate training, pilot testing, and documented decision rules. This study demonstrates that systematic testing and refinement enhances the reliability of the tool. At the study level, clear reporting, adherence to published reporting guidelines, and appropriate and consistent use of design terminology should be enforced.

Table 10 is a summary of study findings, implications for practice, and directions for future research.

Table 10

Summary of findings, implications for practice, and directions for future research.