Departing from the essential features of a high quality systematic review of psychotherapy: A response to Öst (2014) and recommendations for improvement

Öst's (2014) systematic review and meta-analysis of Acceptance and Commitment Therapy (ACT) has received wide attention. On the basis of his review, Öst argued that ACT research was not increasing in its quality and that, in contradiction to the views of Division 12 of the American Psychological Association (APA), ACT is "not yet well-established for any disorder" (2014, p. 105). We conducted a careful examination of the methods, approach, and data used in the meta-analysis. Based in part on examinations by the authors of the studies involved, which were then independently checked, 91 factual or interpretive errors were documented, touching upon 80% of the studies reviewed. Comparisons of Öst's quality ratings with independent teams rating the same studies with the same scale suggest that Ost's ratings were unreliable. In all of these areas (factual errors; interpretive errors; quality ratings) mistakes and differences were not random: Ost's data were dominantly more negative toward ACT. The seriousness, range, and distribution of errors, and a wider pattern of misinterpreting the purpose of studies and ignoring positive results, suggest that Öst's review should be set aside in future considerations of the evidence base for ACT. We argue that future published reviews and meta-analyses should rely upon diverse groups of scholars rather than a single individual; that resulting raw data should be made available for inspection and independent analysis; that well-crafted committees rather than individuals should design, apply and interpret quality criteria; that the intent of transdiagnostic studies need to be more seriously considered as the field shifts away from a purely syndromal approach; and that data that demonstrate theoretically consistent mediating processes should be given greater weight in evaluating specific interventions. Finally, in order to examine substantive progress since Öst's review, recent outcome and process evidence was briefly examined.