Format

Send to

Choose Destination
J Subst Abuse Treat. 2016 Jun;65:43-50. doi: 10.1016/j.jsat.2016.01.006. Epub 2016 Jan 28.

A Comparison of Natural Language Processing Methods for Automated Coding of Motivational Interviewing.

Author information

1
University of UT, Department of Educational Psychology and Social Research Institute, 395 South 1500 East #111, United States. Electronic address: Michael.Tanana@utah.edu.
2
University of Washington, Department of Psychiatry and Behavioral Sciences, Box 354944, Seattle, WA 98195-4944, United States. Electronic address: khallgre@uw.edu.
3
University of Utah, Department of Psychology, 1705 Campus Center Drive, Room 327, Salt Lake City, UT United States. Electronic address: zac.imel@utah.edu.
4
University of Washington, Department of Psychiatry and Behavioral Sciences, Box 354944, Seattle, WA 98195-4944, United States. Electronic address: datkins@uw.edu.
5
University of Utah, School of Computing, 50S. Central Campus Drive Room 3190, Salt Lake City, UT, United States. Electronic address: svivek@cs.utah.edu.

Abstract

Motivational interviewing (MI) is an efficacious treatment for substance use disorders and other problem behaviors. Studies on MI fidelity and mechanisms of change typically use human raters to code therapy sessions, which requires considerable time, training, and financial costs. Natural language processing techniques have recently been utilized for coding MI sessions using machine learning techniques, rather than human coders, and preliminary results have suggested these methods hold promise. The current study extends this previous work by introducing two natural language processing models for automatically coding MI sessions via computer. The two models differ in the way they semantically represent session content, utilizing either 1) simple discrete sentence features (DSF model) and 2) more complex recursive neural networks (RNN model). Utterance- and session-level predictions from these models were compared to ratings provided by human coders using a large sample of MI sessions (N=341 sessions; 78,977 clinician and client talk turns) from 6 MI studies. Results show that the DSF model generally had slightly better performance compared to the RNN model. The DSF model had "good" or higher utterance-level agreement with human coders (Cohen's kappa>0.60) for open and closed questions, affirm, giving information, and follow/neutral (all therapist codes); considerably higher agreement was obtained for session-level indices, and many estimates were competitive with human-to-human agreement. However, there was poor agreement for client change talk, client sustain talk, and therapist MI-inconsistent behaviors. Natural language processing methods provide accurate representations of human derived behavioral codes and could offer substantial improvements to the efficiency and scale in which MI mechanisms of change research and fidelity monitoring are conducted.

KEYWORDS:

Behavioral coding; Discrete sentence feature model; Machine learning; Motivational interviewing; Natural language processing; Recursive neural network; Treatment integrity

PMID:
26944234
PMCID:
PMC4842096
DOI:
10.1016/j.jsat.2016.01.006
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for Elsevier Science Icon for PubMed Central
Loading ...
Support Center