Types of reviews

We included all reviews that reported RCTs if they met four of the five methodological criteria outlined by the Centre for Reviews and Dissemination (CRD) at the University of York, as part of the Database of Abstracts of Reviews of Effects (DARE).¹⁷ DARE was consulted for its guidance in the application of the criteria:

1. Were inclusion/exclusion criteria reported?
2. Was the search adequate? (Databases stated, more than one database searched or one database plus checking references, hand-searching, contact with researchers, citation searching, internet searching.)
3. Were the included studies synthesised?
4. Was the quality of the included studies assessed?
5. Are sufficient details about the individual studies presented? (Details on the population/setting, intervention and a result for each included study.)

We included reviews of RCTs comparing CBT with an active or non-active comparator. Reviews containing randomised and non-randomised studies were included only if RCT data were summarised separately. We excluded reviews based on any other study designs (e.g. quasi-randomised, non-randomised).

Type of health condition

The ICD-11 classifies mental and physical diseases, disorders, injuries and other related health problems in a comprehensive and hierarchical fashion, and is used as a standard for both clinical and research purposes.¹³ The term condition will be used throughout this report to represent diseases, disorders and injuries. Participants with any conditions recognised by the ICD-11 or its nominal categorisation, and of any severity, were included. Non-health-related problems, such as procrastination, were excluded.

For physical conditions, we categorised reviews under the primary ICD-11 codes. For mental conditions, we used the secondary level of ICD-11 codes listed underneath the primary code of 06 mental, behavioural or neurodevelopmental disorders. A review was categorised according to its primary aims. For example, if a review examined the effectiveness of CBT to improve quality of life for people living with diabetes, then 05: Endocrine diseases was the condition category. However, if a review examined the effectiveness of CBT for improving depression in people living with diabetes, then the review was classified as 6A60-80 mood disorders, with comorbid 05 endocrine diseases. Box 1 shows all of the primary and secondary codes that could be considered in grouping reviews together.

BOX 1

Primary and secondary ICD-11 codes

Types of participants

We included participants of any age [children/adolescents (aged < 18 years), adults (aged 18–65 years) and older adults (aged > 65 years)], either sex and any ethnicity.

Types of health-care setting

We included reviews of RCTs that were conducted in any setting [e.g. primary care, secondary care, school/university, institutional (residential care)] and in any country.

Types of delivery timing

We included reviews of RCTs in which CBT was delivered preventatively, as a standard responsive care or as a relapse prevention.

Types of interventions

We included only reviews that evaluated CBT. Interventions were accepted as CBT when authors explicitly stated so in the title, abstract or keywords, or when the review defined the intervention as including at least one cognitive and one behavioural element.

If a trial intervention combined CBT with another therapy and the other therapy was used as a comparator condition (e.g. CBT plus pharmacotherapy compared with pharmacotherapy), then we included these trials. If a trial combined CBT with another therapy and this was compared with another type of comparator [e.g. CBT plus pharmacotherapy compared with wait-list control (WLC)], then we excluded these reviews because we could not extract the isolated effects of CBT.

All modes of CBT delivery were included and categorised into high or low intensity, based on the definitions by Roth and Pilling.¹ High-intensity CBT was defined as face-to-face, individual or group therapy, delivered by a trained CBT therapist. Low-intensity was CBT delivered via media (internet, written, telephone), or was when face-to-face, individual or group CBT was administered by a non-CBT therapist (paraprofessional or layperson). If the review did not report the intensity of the intervention, it was assumed to be high-intensity CBT. We excluded all non-CBTs: cognitive therapy, behavioural therapy, third-wave CBT (e.g. acceptance and commitment therapy, mindfulness therapy), motivational interviewing, stress inoculation therapy, problem-solving therapy and stress management therapy.

Types of comparators

We included reviews that compared CBT with one of the following: (1) a non-CBT-based active comparator (e.g. other psychological therapy, pharmacotherapy), (2) a non-active comparator [e.g. placebo, WLC, treatment as usual (TAU), standard care, no treatment] or (3) a CBT-based active comparator of different intensity [e.g. face-to-face CBT (high intensity) compared with internet-based CBT (low intensity)]. We excluded reviews that compared variations of high-intensity (e.g. group CBT compared with individual CBT) or low-intensity CBT (internet CBT compared with bibliotherapy CBT).

Types of outcomes

We included reviews if they reported data on at least one of the following outcomes: HRQoL, anxiety, depression or a condition-specific outcome (e.g. pain).

Length of follow-up

We included reviews with post-treatment, short-term (< 12 months) or long-term (≥ 12 months) follow-up data. If both short- and long-term follow-up data were reported, the synthesis of only the longest follow-up time point was included.

Information sources

The DARE (up to March 2015), Cochrane Database of Systematic Reviews, MEDLINE (via Ovid), EMBASE (via Ovid), PsycINFO (via Ovid), Cumulative Index to Nursing and Allied Health Literature (CINAHL) (via EBSCOhost), Child Development & Adolescent Studies (via EBSCOhost) and OpenGrey databases were searched on 25–27 April 2018 to identify relevant systematic reviews published between 1992 and 2018. An updated search was run on all the above databases on 30 January 2019, covering the period from April 2018 to 30 January 2019, excluding DARE, which is no longer updated. Owing to the volume of material being processed and the time constraints associated with this process, the reference lists of included reviews were not hand-searched for additional reviews. We did not contact authors for additional information to confirm inclusion/exclusion.

Search strategy

Comprehensive search strategies for each of the eight databases were designed by a senior research information specialist (SK). Each search strategy was developed iteratively, and a sensitivity check was performed in each database for the ability of each strategy to retrieve 36 key known papers (where indexed) that had been identified a priori (see Appendix 1). The included search terms were identified from these reviews and their associated database indexing terms, and with input from the expert consultation group (ECG). The search strategies utilised a combination of free text and controlled vocabulary search terms covering variations of ‘CBT’ searched in the title, abstract or keyword fields, and were combined with validated study-type filters for ‘systematic review’. The Scottish Intercollegiate Guidelines Network systematic review search filters available on the InterTASC (Technology Appraisal Support Collaboration) Information Specialists’ Sub-Group website¹⁸ was used to search the MEDLINE, EMBASE and CINAHL databases. The McMaster University Health Information Research Unit systematic review filter¹⁹ was modified and used in the PsycINFO search. The full search strategies for all the databases can be found in Appendix 2.

Restrictions

The scoping work identified that the earliest published review of CBT is from 1992.²⁰ Therefore, we restricted our search to reviews published since 1992. The search was not restricted in terms of language, although we subsequently excluded non-English-language reviews (see Protocol revisions).

Bubble chart

The evidence was grouped under the corresponding ICD-11 primary or secondary code. The volume of evidence, in terms of number of reviews, RCTs and participants, was then imported from Microsoft Excel into TIBCO Spotfire^® (TIBCO, Software Inc., Palo Alto, CA, USA) software²³ to produce a bubble chart. The axes of the bubble charts were very large, ranging from 0 to 45,000 participants. To help readability of the charts, we stratified reviews into those with < 1000 participants and those with ≥ 1000 participants.

Summary tables

The detailed description of each included review was represented in summary tables.

Gap maps

The condition and population and context characteristics extracted from the included reviews were populated in an Excel spreadsheet to identify any gaps in the evidence base, and were summarised.

Heterogeneity tests

We conducted a PMA per outcome measure. Review data were entered into an ICD-11 condition subgroup analysis and we tested the within-condition statistical heterogeneity across the reviews. In parallel, we tested the heterogeneity across the ICD-11 condition subgroup categories.

Assumptions for pooling across conditions

We developed three a priori conditions that must be met for us to pool the effect estimates across ICD-11 conditions categories:

1. Intervention homogeneity: the ECG and investigators (see Expert consultation group including patient and public involvement) agreed that, although investigators often use condition-, population- and context-specific protocols, the principles of CBT (see Figure 1) are the same across all conditions. This allows us to make a judgement of intervention homogeneity and, provided the other criteria are met, to pool estimates across conditions.
2. Design homogeneity: it is possible that meta-analytic estimates of effects would be moderated by differences between the review’s underlying design and methodologies. The review estimates would also be influenced by the quality of the included RCTs. However, a RCT-level quality assessment was beyond the scope of this overview. Therefore, we used the proxy of assuming that the highest-quality reviews would be more likely to be unbiased in their methods and would probably report from the best-available evidence. We minimised review design (but not RCT design) variation by including only reviews that adhered to the CRD review criteria and were graded as being of high or moderate quality on the AMSTAR-2 tool (higher-quality reviews); hence, we could claim design homogeneity. We ran a sensitivity analysis (which included higher- and lower-quality review data) to ascertain if the variation in review quality affected the homogeneity of effect estimates across conditions.
3. Statistical homogeneity: statistical heterogeneity was assessed using the I² statistic; this is expressed as a percentage. A higher percentage is indicative of greater heterogeneity. I² reflects the variation in effect estimates between reviews that is attributable to heterogeneity.²⁴ There is no guidance regarding acceptable heterogeneity for PMAs. We used the guidance for meta-analysis heterogeneity,²⁵ which suggests that I² of < 75% is acceptable for pooling across the categories.

Panoramic meta-analysis method

The PMA was undertaken using a two-step frequentist approach random-effects model using the ‘metan’ command in Stata (versions 13.1 and 16). The two-step analysis consists of performing a conventional meta-analysis of a series of meta-analyses. The first step is undertaken by the original reviewers in obtaining a pooled treatment effect based on their included trials. Many of these will have been estimated via a random-effects meta-analysis, but some will have been analysed using a fixed-effects approach. Nonetheless, we assume that within-review variability has been appropriately allowed for. In the second step, the pooled estimates (with CIs) from each of the systematic reviews are combined into an overall (over all reviews) pooled estimate. At this point, we use a random-effects approach using the DerSimonian–Laird²⁶ approach. We obtained a pooled estimate from within condition and also across conditions, if the across-condition heterogeneity was < 75%. In the few cases where data required for the meta-analysis, such as standard deviations or CIs, were missing, we referred to the individual RCT paper to extract this information.

Primary analysis

The primary analysis was conducted on continuous, end-point data extracted from higher-quality reviews (AMSTAR-2 rating of ‘moderate’ or ‘high’ quality) if there were more than two systematic review per comparison. The primary outcome was HRQoL and the secondary outcomes were depression, anxiety and pain.

We analysed the standardised mean differences (SMDs). When reviews reported values as mean differences, we converted the pooled estimate into a SMD using the standard deviation reported.²⁷ We reported the 95% CIs and the prediction intervals. These offer a prediction of the distribution of SMDs from future reviews, perhaps in other conditions that have not been included in our overview.

Secondary analysis

Some reviews reported change scores only; we pooled these separately because of concerns that these may be biased as a result of regression to the mean.²⁸ We performed separate PMAs for RRs and ORs.

We grouped reviews that directly compared high- with low-intensity CBT, irrespective of the condition, and analysed this group separately.

Transforming the standardised mean difference into a mean difference

To make meaningful interpretations, the overall pooled estimate (i.e. the SMD) for each outcome was transformed into a mean difference. The SMD was multiplied by the standard deviation of the most commonly used outcome measure (e.g. Beck Depression Inventory for depression) for each outcome.²⁹ To find a suitable standard deviation for the measurement tool, we identified a higher-quality review, which included a low risk-of-bias RCT that had used the most common outcome measure. From that trial, we extracted the standard deviation of the outcome measure at baseline.

Publication bias

When ≥ 10 reviews were included in the meta-analysis, publication bias and small-study effects were tested for using Egger’s regression intercept³⁰ and a visual assessment of funnel plot asymmetry. We used a conservative value of p < 0.1 at CIs of 95% to reflect asymmetry.

Subgroup analysis

Subgroup analyses were agreed a priori and were performed if four or more reviews were included in the meta-analysis across all conditions for each of the outcomes on the following: (1) CBT intensity (high/low intensity), (2) age (children and adolescents, adults, older adults), (3) duration of follow-up (short: < 12 months, long: > 12 months) and (4) comparator group (active, non-active). The subgroups were separated using the ‘by()’ command in Stata.

We ran interaction tests between the subgroups using an exploratory meta-regression. The meta-regression used the method of moments estimate of between-study variance and the ‘metareg’ command in Stata.

If we identified any reviews that directly compared high-intensity CBT with low-intensity CBT, we grouped these reviews together by outcomes (HRQoL, depression, anxiety and pain). If their estimates were homogeneous, then we pooled across the reviews as an example of direct comparison.

Sensitivity analysis

To test whether or not the quality of the reviews moderated the effect estimate and or heterogeneity, we ran a sensitivity analysis in which we included data from all reviews, irrespective of their AMSTAR-2 quality, for each outcome PMA. Then we compared the heterogeneity and pooled effect estimates between the sensitivity analyses and the primary analyses (which included only data from higher-quality reviews, i.e. those of ‘high’ and ‘moderate’ AMSTAR-2 quality).

We also conducted a sensitivity analysis in the HRQoL PMA for the two subscales of the Short Form questionnaire-12 items (SF-12)/Short Form questionnaire-36 items (SF-36) instruments. These instruments include a physical composite score and a mental composite score, but the tool does not pool them together. We prioritised the physical component scale (as recommended by the ECG; see Expert consultation group including patient and public involvement), then we re-ran the analyses using the mental component scale to determine if this changed the results.