Background:

Collecting NHS patient experience data is critical to ensure the delivery of high-quality services. Data are obtained from multiple sources, including service-specific surveys and widely used generic surveys. There are concerns about the timeliness of feedback, that some groups of patients and carers do not give feedback and that free-text feedback may be useful but is difficult to analyse.

Objective:

To understand how to improve the collection and usefulness of patient experience data in services for people with long-term conditions using digital data capture and improved analysis of comments.

Design:

The DEPEND study is a mixed-methods study with four parts: qualitative research to explore the perspectives of patients, carers and staff; use of computer science text-analytics methods to analyse comments; co-design of new tools to improve data collection and usefulness; and implementation and process evaluation to assess use of the tools and any impacts.

Setting:

Services for people with severe mental illness and musculoskeletal conditions at four sites as exemplars to reflect both mental health and physical long-terms conditions: an acute trust (site A), a mental health trust (site B) and two general practices (sites C1 and C2).

Participants:

A total of 100 staff members with diverse roles in patient experience management, clinical practice and information technology; 59 patients and 21 carers participated in the qualitative research components.

Interventions:

The tools comprised a digital survey completed using a tablet device (kiosk) or a pen and paper/online version; guidance and information for patients, carers and staff; text-mining programs; reporting templates; and a process for eliciting and recording verbal feedback in community mental health services.

Results:

We found a lack of understanding and experience of the process of giving feedback. People wanted more meaningful and informal feedback to suit local contexts. Text mining enabled systematic analysis, although challenges remained, and qualitative analysis provided additional insights. All sites managed to collect feedback digitally; however, there was a perceived need for additional resources, and engagement varied. Observation indicated that patients were apprehensive about using kiosks but often would participate with support. The process for collecting and recording verbal feedback in mental health services made sense to participants, but was not successfully adopted, with staff workload and technical problems often highlighted as barriers. Staff thought that new methods were insightful, but observation did not reveal changes in services during the testing period.

Conclusions:

The use of digital methods can produce some improvements in the collection and usefulness of feedback. Context and flexibility are important, and digital methods need to be complemented with alternative methods. Text mining can provide useful analysis for reporting on large data sets within large organisations, but qualitative analysis may be more useful for small data sets and in small organisations.

Limitations:

New practices need time and support to be adopted and this study had limited resources and a limited testing time.

Future work:

Further research is needed to improve text-analysis methods for routine use in services and to evaluate the impact of methods (digital and non-digital) on service improvement in varied contexts and among diverse patients and carers.

Funding:

This project was funded by the NIHR Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 8, No. 28. See the NIHR Journals Library website for further project information.

Article history

The research reported in this issue of the journal was funded by the HS&DR programme or one of its preceding programmes as project number 14/156/16. The contractual start date was in April 2016. The final report began editorial review in April 2018 and was accepted for publication in June 2019. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Caroline Sanders was previously a Director (unpaid) for Affigo CIC (Altrincham, UK) (2016–17), a social enterprise providing digital health products for severe mental illness. Peter Bower reports grants from the National Institute for Health Research (NIHR) during the conduct of the study. Richard Hopkins reports that he is a current director of Affigo CIC, which promotes electronic monitoring of patient symptoms through the use of mobile application, outside the submitted work. Ruth Boaden reports that she was the Director of the NIHR Collaboration for Leadership in Applied Health Research and Care (CLAHRC) Greater Manchester (2013–19), which was hosted by Salford Royal NHS Foundation Trust where she held an honorary (unpaid) as an Associate Director to fulfil her role as Director of the CLAHRC. She was also a member of the NIHR Dissemination Centre Advisory Group (2015–19) and the Health Services and Delivery Research Funding Committee (2015–19). She was a member of the NIHR Knowledge Mobilisation Research Fellowships Panel (2013–15) and chaired the panel (2016–18). She is a member of the NIHR Advanced Fellowships Panel (2019–present). Azad Dehghan is the Managing Director of DeepCognito Ltd (Manchester, UK) and a Data Analytics Advisor for KMS Solutions Ltd (Manchester, UK). William Dixon receives consultancy fees from Bayer AG (Leverkusen, Germany) and Google Inc. (Mountain View, CA, USA). John Ainsworth reports that he is a Director of Affigo CIC. Shôn Lewis reports that he is a Director for Affigo CIC. Humayun Kayesh reports he is a contract engineer for DeepCognito Ltd. Goran Nenadic reports that he was previously a Scientific Advisor (Non-executive) of DeepCognito Ltd.