4.2.1. Guideline Development Group meetings

Seventeen Group meetings were held between February 2003 and June 2004. During each day-long meeting, in a plenary session, clinical questions and clinical evidence were reviewed and assessed, statements developed and recommendations formulated. At each meeting all Group members declared any potential conflict of interest, and PTSD sufferer and carer concerns were routinely discussed as part of a standing agenda.

4.2.2. Topic leads

The Group divided its workload along clinically relevant lines to simplify the guideline development process, and individual members took responsibility for advising on guideline work for particular areas of clinical practice (psychological interventions, pharmacological interventions, early intervention, risk factors and screening, and children).

4.2.3. PTSD sufferers and carers

Individuals with direct experience of services gave an integral PTSD sufferer focus to the Group and to the guideline. The Group included two PTSD sufferers, both of whom had contact with other PTSD sufferers and carers. They contributed as full Group members to writing the clinical questions, helping to ensure that the evidence addressed their views and preferences, highlighting sensitive issues and terminology associated with PTSD, and bringing PTSD sufferer research to the attention of the Group. In drafting the guideline, they contributed to the editing of the introduction and Chapter 3, and identified good practice points from the PTSD sufferer and carer perspectives.

4.2.4. Special advisers

Special advisers who had specific expertise in one or more aspects of treatment and management relevant to the guideline assisted the Group, commenting on specific aspects of the developing guideline and making presentations to the Group. The Acknowledgements section at the beginning of this guideline lists those who agreed to act as special advisers.

4.2.5. National and international experts

National and international experts in the area under review were identified through the literature search and through the experience of the Group members. These experts were contacted to recommend unpublished or soon-to-be published studies in order to ensure up-to-date evidence was included in the development of the guideline. They informed the group about completed trials at the pre-publication stage, systematic reviews in the process of being published, studies relating to the cost-effectiveness of treatment, and trial data if the Group could be provided with full access to the complete trial report. Appendix 4 lists the researchers who were contacted.

4.4.1. Methodology

A stepwise, hierarchical approach was taken to locating and presenting evidence to the Group. The NCCMH developed this process based on advice from the NICE National Guidelines Support and Research Unit and after considering recommendations from a range of other sources. These included:

• the Centre for Clinical Policy and Practice of the New South Wales Health Department (Australia)
• Clinical Evidence Online
• Cochrane Collaboration
• New Zealand Guideline Group
• NHS Centre for Reviews and Dissemination
• Oxford Centre for Evidence-Based Medicine
• Scottish Intercollegiate Guidelines Network
• United States Agency for Health Research and Quality
• Oxford Systematic Review Development Programme.

4.4.2. The review process

A brief search of the major bibliographic databases for recent systematic reviews and existing guidelines was first conducted to help inform the development of the scope. After the scope was finalised, a more extensive search for systematic reviews was undertaken. At this point, the review team, in conjunction with the Group, developed an evidence map that detailed all comparisons necessary to answer the clinical questions. The initial approach taken to locating primary-level studies depended on the type of clinical question and availability of evidence.

After consulting the Group, the review team decided which questions were likely to have a good evidence base and which questions were likely to have little or no directly relevant evidence. For questions in the latter category, a brief descriptive review was initially undertaken by a member of the Group (see section 4.4.10). For questions with a good evidence base, the review process depended on the type of clinical question.

4.4.2.1. Search process for questions concerning interventions

For questions related to interventions, the initial evidence base was formed from well-conducted randomised controlled trials (RCTs) that addressed at least one of the clinical questions. Although there are a number of difficulties with the use of RCTs in the evaluation of interventions in mental health, the RCT remains the most important method for establishing treatment efficacy.

The initial search for RCTs involved searching the standard mental health bibliographic databases (EMBASE, Medline, PsycINFO, Cochrane Library) for all RCTs potentially relevant to the guideline.

After the initial search results were scanned liberally to exclude irrelevant papers, the review team used a purpose-built ‘study information’ database to manage both the included and the excluded studies (eligibility criteria were developed after consultation with the Group). For questions without good-quality evidence (after the initial search), a decision was made by the Group about whether to repeat the search using subject-specific databases, such as CINAHL, the Allied and Complementary Medicine Database (AMED), the System for Information on Grey Literature in Europe (SIGLE) and the Publishers International Literature on Traumatic Stress (PILOTS); conduct a new search for lower levels of evidence; or adopt a consensus process (see section 4.4.10.1). Future guidelines will be able to update and extend the usable evidence base starting from the evidence collected, synthesised and analysed for this guideline.

Data from unpublished pharmacological trials held by the Medical and Healthcare Products Regulatory Agency were routinely requested, and where these data were available and could be released they are considered within the review.

Recent high-quality English-language systematic reviews were used primarily as a source of RCTs (see Appendix 7 for quality criteria). However, where existing data-sets were available from appropriate reviews, they were cross-checked for accuracy before use. New RCTs meeting inclusion criteria set by the Group were incorporated into the existing reviews and fresh analyses performed. The review process is illustrated in Fig. 4.1.

Fig. 4.1

Guideline review process (GDG, Guideline Development Group).

Additional searches were made of the reference lists of all eligible systematic reviews and RCTs, and the list of evidence submitted by stakeholders. Known experts in the field (see Appendix 2), based both on the references identified in early steps and on advice from Group members, were sent letters requesting systematic reviews or RCTs that were in the process of being published (unpublished full trial reports were also accepted where sufficient information was available to judge eligibility and quality). In addition, the standard mental health bibliographic databases were periodically checked for relevant studies.

4.4.3. Search filters

Search filters developed by the review team consisted of a combination of subject heading and free-text phrases. Specific filters were developed for the guideline topic, and where necessary, for each clinical question. In addition, the review team used filters developed for systematic reviews, RCTs and other appropriate research designs (Appendix 6).

4.4.4. Study selection

All primary-level studies included after the first scan of citations were acquired in full and reevaluated for eligibility at the time they were being entered into the study information database. The inclusion criteria for RCTs are listed below (see section 4.4.8). For certain clinical questions these inclusion criteria were amended (see Chapter 9). All eligible papers were then critically appraised for methodological quality (see Appendix 8). The eligibility of each study was confirmed by at least one member of the Group.

For some clinical questions, it was necessary to prioritise the evidence with respect to the UK context. To make this process explicit, the Group took into account the following factors when assessing the evidence:

• participant factors (e.g. gender, age, ethnicity)
• provider factors (e.g. model fidelity, the conditions under which the intervention was performed, the availability of experienced staff to undertake the procedure)
• cultural factors (e.g. differences in standard care, differences in the welfare system).

The Group decided which prioritisation factors were relevant to each clinical question in light of the UK context, and then how they should modify the recommendations.

4.4.5. Synthesising the evidence

Where possible, outcome data were extracted directly from all eligible studies, which met the quality criteria, into Review Manager 4.2 (Cochrane Collaboration, 2003). Meta-analysis was then used, where appropriate, to synthesise the evidence using Review Manager. If necessary, re-analyses of the data or sensitivity analyses were used to answer clinical questions not addressed in the original studies or reviews. For continuous outcomes, where more than 50% of the total number randomised in a particular study were not accounted for, the data were excluded from the analysis because of the risk of bias (as outlined within the inclusion criteria in section 4.4.8).

Included/excluded studies tables, generated automatically from the study information database, were used to summarise general information about each study (see Appendix 14). Where meta-analysis was not appropriate and/or possible, the reported results from each primary-level study were also presented in the included studies table.

Consultation was used to overcome difficulties with coding. Data from studies included in existing systematic reviews were extracted independently by one reviewer directly into Review Manager and cross-checked with the existing data-set. Two independent reviewers extracted data from new studies, and disagreements were resolved with discussion. Where consensus could not be reached, a third reviewer resolved the disagreement. Masked assessment (i.e. masked to the journal from which the paper came, the authors, the institution and the magnitude of the effect) was not used, since it is unclear that doing so reduces bias (Jadad et al, 1996; Berlin, 1997).

4.4.6. Presenting the data to the Guideline Development Group

Where possible, meta-analysis was used to synthesise data. If necessary, sub-analyses were used to answer clinical questions not addressed in the original studies or reviews. The Group was given a graphical presentation of the results using forest plots generated with the Review Manager software. Each forest plot displayed the effect size and confidence interval (CI) for each study as well as the overall summary statistic. The graphs were organised so that the display of data in the area to the left of the ‘line of no effect’ indicated a ‘favourable’ outcome for the treatment in question. Dichotomous outcomes were presented as relative risks (RR) with the associated 95% CI (for an example, see Fig. 4.2). A relative risk (or risk ratio) is the ratio of the treatment event rate to the control event rate. An RR of 1 indicates no difference between treatment and control. In Figure 4.2, the overall RR of 0.73 indicates that the event rate (i.e. non-remission rate) associated with intervention A is about three-quarters of that with the control intervention, or in other words, the relative risk reduction is 27%.

Fig. 4.2

Example of a forest plot displaying dichotomous data (RR, relative risk).

The confidence interval shows with 95% certainty the range within which the true treatment effect should lie and can be used to determine statistical significance. If the CI does not cross the ‘line of no effect’, the effect is statistically significant.

All dichotomous outcomes were calculated on an intention-to-treat basis (i.e. a ‘once randomised always analyse’ basis). This assumes that participants who ceased to engage in the study – from whatever group – had an unfavourable outcome (with the exception of the outcomes of death and certain adverse events). Continuous outcomes were analysed as standardised mean differences (SMDs) to allow for ease of comparison across studies (Fig. 4.3). If provided, intention-to-treat data, using a method such as ‘last observation carried forward’, were preferred over data from completers.

Fig. 4.3

Example of a forest plot displaying continuous data (SMD, standardised mean difference).

To check for heterogeneity between studies, both the I² and χ² tests of heterogeneity (P<0.10), as well as visual inspection of the forest plots, were used. The I² statistic describes the proportion of total variation in study estimates that is due to heterogeneity (Higgins & Thompson, 2002). An I² of less than 30% was taken to indicate mild heterogeneity and a fixed effects model was used to synthesise the results. An I² of more than 50% was taken as notable heterogeneity. In this case, an attempt was made to explain the variation. If studies with heterogeneous results were found to be comparable, a random effects model was used to summarise the results (DerSimonian & Laird, 1986). In the random effects analysis, heterogeneity is accounted for both in the width of CIs and in the estimate of the treatment effect. With decreasing heterogeneity the random effects approach moves asymptotically towards a fixed effects model. An I² of 30–50% was taken to indicate moderate heterogeneity. In this case, both the χ² test of heterogeneity and a visual inspection of the forest plot were used to decide between a fixed and random effects model.

4.4.7. Forming and grading the statements and recommendations

The evidence tables and forest plots formed the basis for developing clinical statements and recommendations.

4.4.7.1. Developing statements

For each outcome a clinical statement describing the evidence found was developed. To assess clinical importance where a statistically significant summary was obtained (after controlling for heterogeneity) the Group set thresholds for determining clinical importance, in addition to taking into account the trial population and nature of the outcome.

Two separate thresholds for determining clinical importance were set. For comparisons of one active treatment against waiting list or non-active interventions, a higher threshold was applied than for comparisons of active treatments against one another.

For comparisons of one active treatment against another treatment the following thresholds were applied: for dichotomous outcomes an RR of 0.80 or less was considered clinically important and for continuous outcomes an effect size of approximately 0.5 (a ‘medium’ effect size; Cohen, 1988) or less was considered clinically important.

For comparisons of active treatment against waiting list the following thresholds were applied: for dichotomous outcomes a RR of 0.65 or less was considered clinically important and for continuous outcomes an effect size of approximately 0.8 (a ‘large’ effect size; Cohen, 1988) or less was considered clinically important.

In order to facilitate consistency in generating and drafting the clinical statements the Group used a statement decision tree (Fig. 4.4). This flow chart was designed to assist with decision-making, not to replace clinical judgement. Using this procedure, the Group classified each effect size as clinically important or not (i.e. whether or not the treatment is likely to benefit PTSD sufferers), taking into account both the comparison group and the outcome.

Fig. 4.4

Guideline statement decision tree.

Where heterogeneity between studies was judged problematic, in the first instance an attempt was made to explain the cause of the heterogeneity (e.g. outliers were removed from the analysis, or sub-analyses were conducted to examine the possibility of moderators). Where homogeneity could not be achieved, a random effects model was used.

In cases where the point estimate of the effect was judged clinically important, a further consideration was made about the precision of the evidence by examining the range of estimates defined by the CI. For level I evidence, where the effect size was judged clinically important for the full range of plausible estimates, the result was described as evidence favouring intervention x over intervention y (i.e. statement 1, or S1). For non-level-I evidence or in situations where the point estimate was clinically important but the CI included clinically unimportant effects, the result was described as limited evidence favouring intervention x over intervention y (i.e. S2). Where a point estimate was judged as not clinically important and the CI did not include any clinically important effects, the result was described as unlikely to be clinically important (i.e. S3). Alternatively, if the range of estimates defined by the CI included clinically important benefits as well as no effect or harmful effects, the result was described as inconclusive (i.e. S4).

Where for a particular review very few trials meet the threshold for clinical importance, further criteria are required to differentiate the relative efficacy of treatments considered. In this case treatments are evaluated according to whether they are both statistically significant and reasonably well tolerated. Specifically, the most effective treatments are identified as those for which, for the principal outcome measures, the effect sizes are statistically significant (95% CI to the left of the line of no effect).

4.4.7.2. Developing and grading the recommendations

Once all evidence statements relating to a particular clinical question were finalised and agreed by the Group, the associated recommendations were produced and graded. Grading allowed the Group to distinguish between the level of evidence and the strength of the associated recommendation. This allowed the Group to moderate recommendations based on factors other than the strength of evidence. Such considerations include the applicability of the evidence to the people in question, economic considerations, values of the development group and society, and the group’s awareness of practical issues (Eccles et al, 1998).

Each clinical evidence statement was classified according to a hierarchy. Recommendations were then graded A to C based on the level of associated evidence (Table 4.1), or as a good practice point (GPP). All evidence statements and associated forest plots are presented in Appendices 16 and 15 respectively, while a subset of the key evidence statements are presented in the relevant chapters for ease of reference.

Table 4.1

Hierarchy of evidence and recommendations grading scheme.

4.4.8. Inclusion criteria

The review used the following inclusion criteria:

• the study used a randomised controlled design
• at least 70% of participants needed to have a diagnosis of PTSD, other participants must have PTSD symptoms following a traumatic event
• the main target of treatment was PTSD
• PTSD symptoms were measured
• pre- and post-treatment data were reported
• for continuous data at least 50% of the intent-to-treat sample were assessed at the relevant time point
• double-blind administration of treatment (for pharmacological treatments only).

4.4.9. Measures of outcome

The main criterion for treatment effectiveness was its effect on PTSD symptoms. These were assessed either by independent assessors or by self-report. The instruments included in the analysis were as follows:

• assessor-rated PTSD symptoms: the Clinician-Administered PTSD Scale for DSM–IV (CAPS), the PTSD Symptom Scale – Interview Version (PSS–I), or the number of symptoms on the Structured Clinical Interview for DSM–IV (SCID)
• self-report instruments of PTSD symptoms: the Davidson Trauma Scale (DTS), or the Post-traumatic Diagnostic Scale (PDS), or the PTSD Checklist (PCL), or the Impact of Event Scale (IES) or Impact of Event Scale – Revised (IES–R).

If more than one self-report scale (for example, PCL and IES) was used, the instrument that mapped onto the DSM–IV criteria was included (in the example, PCL). Both continuous data (outcome measures scores and changes) and dichotomous data (PTSD remission) based on these scores were considered.

A number of scales have been developed for the measurement of PTSD and other outcomes for children and young people and these are discussed in Chapter 9.

4.4.10. Method used to answer a clinical question in the absence of appropriately designed, high-quality research

In the absence of level I evidence (or a level that is appropriate to the question), or where the Group members were of the opinion (on the basis of previous searches or their knowledge of the literature) that there was unlikely to be such evidence, either an informal or a formal consensus process was adopted. This process focused on questions that the Group considered a priority.

4.5.1. Search strategy

In January 2004, bibliographic electronic databases – Medline, PreMedline, EMBASE, CINAHL, PsycINFO, the Cochrane Database of Systematic Reviews (CDSR), Cochrane Controlled Trials Reports (CCTR), Database of Abstracts of Reviews of Effectiveness (DARE) and the NHS Health Technology Assessment (HTA) – and specific health economic databases, the NHS Economic Evaluation Database (NHS EED) and the Office of Health Economics Health Economic Evaluation Database (OHE HEED), were searched for economic studies. For Medline, PreMedline, EMBASE, CINAHL, PsycINFO, CDSR, CCTR and DARE, a combination of a specially developed health economics search filter already tested in earlier NCCMH guidelines and a general filter for post-traumatic stress disorder was used. A combination of subject headings and free-text searches was used. The HTA, NHS EED and OHE HEED databases were searched using shorter, database-specific strategies.

In addition to searches of electronic databases, reference lists of eligible studies and relevant reviews were searched by hand. Studies included in the clinical evidence review were also screened for economic evidence.

4.5.2. Review process

The database searches for general health economic evidence for PTSD resulted in a total of 345 references. Of these, 27 were identified as potentially relevant. Secondary searches for additional pharmaco-economic papers resulted in a further 46 references, of which 8 were initially considered relevant to criteria for health economic appraisal. A further 6 potentially eligible references were found by hand-searching. Full texts of all potentially eligible studies (including those for which relevance or eligibility was not clear from the abstract) were obtained: a total of 41 papers. (At this stage inclusion was not limited to papers only from the UK.) These publications were then assessed against a set of standard inclusion criteria by the health economist, and papers eligible for inclusion as economic evaluations were subsequently assessed for internal validity. The quality assessment was based on the 32-point checklist used by the British Medical Journal to assist referees in appraising economic analyses (Drummond & Jefferson, 1996) (Appendix 12).

4.5.3. Selection criteria

Cost-of-illness/economic burden studies:

• no restriction was placed on language or publication status of the papers
• studies published between 1980 and 2003 were included (this date restriction was imposed in order to obtain data relevant to current healthcare settings and costs)
• only studies from the UK were included, as the aim of the review was to identify economic burden information relevant to the national context
• selection criteria based on types of clinical conditions and patients were identical to the clinical literature review (see Appendix 7)
• studies were included provided sufficient details regarding methods and results were available to enable the methodological quality of the study to be assessed and provided the study’s data and results were extractable.

Economic evaluations:

• studies were included provided they had used cost-minimisation analysis, cost-effectiveness analysis, cost–utility analysis or cost–benefit analysis
• only clinical evidence from a meta-analysis, a randomised controlled trial, a quasi-experimental trial or a cohort study was used
• no restriction was placed on language or publication status of the papers
• studies published between 1980 and 2003 were included (this date restriction was imposed in order to obtain data relevant to current healthcare settings and costs)
• only studies from the UK were considered, as the aim of the review was to identify economic evaluation information relevant to the national context
• selection criteria based on types of clinical conditions, patients, treatments and settings were identical to the clinical literature review (see Appendix 7)
• studies were included provided sufficient details regarding methods and results were available to enable the methodological quality of the study to be assessed and provided the study’s data and results were extractable.

Health state utility studies:

• studies reporting health state utilities for PTSD were considered for inclusion
• no restriction was placed on language or publication status of the papers
• studies published between 1980 and 2003 were included
• only studies from Organization for Economic Cooperation and Development countries were considered, to assure the generalisability of the results to the UK context
• selection criteria based on types of clinical conditions, patients, treatments and settings were identical to the clinical literature review (see Appendix 7).

4.5.4. Data extraction

Data were extracted by the health economist. Masked assessment, whereby data extractors are masked to the details of journal, authors and so on was not undertaken, because the evidence does not support the claim that this minimises bias (Alderson et al, 2004).