The study team set out to achieve the following, and has been successful in completing all stages within the original projected timelines.
This section begins by considering how well the study team met the original brief and discussing the more innovative aspects of the study’s work. This is followed by subsections that consider the processes and outputs from the different parts of the study, and their benefits and challenges, and strengths and limitations. The chapter concludes with a consideration of the work that remains to be done and the possibilities that the study’s work has foregrounded.
Responding to the brief
Improving the use of patient feedback data
The NIHR call was intended to improve the use of patient feedback data within the NHS. The study team looked at PES free-text comments specifically, with the National CPES as the study’s first case. There is no systematic analysis and use of CPES free-text comments (or indeed, any free-text patient feedback) either within the trusts from which they arise or at a national or regional level. This makes it impossible to respond systematically to emergent themes from the comments or to understand how they relate to variation in care quality across NHS trusts. This study has made significant strides towards the development and validation of an approach that could automate the analysis of summary free-text data and present summaries in a visually engaging and useful way for the 150 NHS trusts that participate in the CPES. It has been shown how various challenges have been met to develop a model of the features a NHS toolkit should have to optimise the use of patient experience free text. This is the first evidence-based attempt that the study team knows of to represent PES free-text comments in ways that are useful to health-care professionals and that the public can understand, and the team members are eager for this to be further developed. As one outcome of the study, proof of concept is thus provided.
Assurances of acceptability and value
With its embedded co-design (using surveys and group concept-mapping workshops) and validation with stakeholders using established preference-based usability and acceptability techniques, the study also provides substantial reassurance that the approach has value for NHS staff. Through these approaches, the study has also shown keen patient interest once governance issues are addressed; until then, the value to patients is indirect, from improvements in the patient experience. The toolkit generated considerable excitement when it was shown to health-care professionals, who know of nothing like it and welcome its implementation into practice. They have declared that they would use it regularly if it has tailored options to adapt the settings to their workflow and themes relevant to their specific work environment and teams. This has been enabled by the study team.
Commissioners have also declared their needs independently, in commissioner events run by Macmillan Cancer Support,13 which are a good fit with what the study team has provided, confirming that the approach meets current demands. These include:
Awareness of the lack of data for some marginalised groups – the data summary tab () helps to show where data are missing. Summary demographic details of people with relevant comments can be viewed on the main toolkit page in a different format ().
Annual and local comparisons as leverage at the board level.
Granularity, including access to raw CPES data sets.
A more systematic, easier and quicker way to analyse CPES free text, having had experience of its value when sorted into themes.
Online access rather than a report via e-mail.
Emphasis on priority areas [which the ranking of themes can achieve ()].
Part of a screenshot showing demographic details of respondents.
Part of a screenshot of the toolkit highlighting how demographic details of patients making comments on a particular theme may be determined through filters.
Overview of themes. As well as developing six high-priority default themes, a quick overview of all themes is provided. Note that the wording has changed slightly since this draft.
Commissioners in the Macmillan Cancer Support study13 also suggested an online forum for CCGs to compare and share examples of interventions for improvement, as well as peer support networks. The study’s own evidence confirms the value of such additions to the site [and a patient forum for the public-facing version (see Chapter 9)]. Neither of these has currently been included, as this would require some form of moderation or management, or a suitable filtering system to be set up, which needs further funding.
Patients were more likely to use the toolkit on a one-off basis to learn about treatment side effects and experiences of others suffering from the same condition, and to gain information on a site where they were to receive treatment.
Meaningful presentation of the data
The study has also addressed the point in the NIHR HSDR programme brief that there was still uncertainty as to how to present patient experience data in a meaningful and granular way that stimulates local action. An important result and advantage of the study’s approach is that it draws together very large and complex data sets into a thematically driven, simple visual display without loss of the nuances that other manually based methods can have, and it can still allow for exploration of the original text.
The study team is of the belief that:
Information is only useful if it is translated into knowledge and knowledge is only useful if it is used to improve the health of individual patients. One of the main reasons why big data have not fulfilled its[sic] full potential in health care is that small data are not adequately systematized to generate useful knowledge for future patients (research) and that big data are not used to improve health outcomes for individual patients (care).
Reproduced with permission from Sacristán and Dilla.
47 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See:
http://creativecommons.org/licenses/by/4.0/Sacristán et al.
The study team has achieved the potential systematisation of patient experience free-text comments in a way that has the potential to drive health-care improvements.
Good toolkit design has the following three requirements:
identification of what is being measured
connection to current strategic objectives
consideration of how the first can achieve the second, and how actionable outcomes can be achieved.
This third requirement was also the ethos of the study. The study team aimed to achieve this by establishing which dashboard or toolkit features were important from empirical evidence, and which were meaningful to individual users within the patient experience setting. This also indirectly satisfied the first and second requirements.
Transferability
The study’s approach is transferable with further work; it has been designed to enable this and has supporting documentation and code. The study’s testing on three very different health-care free-text data sets has elucidated some of the challenges with this and possible solutions.
Innovation
Using interdisciplinary approaches in innovative ways, the study successfully involved a range of salient stakeholders, whose feedback continued throughout the development process. This meant that the study team was accountable to them throughout for all of the team members’ design decisions. The study team considered the differences in perspective between them and facilitated their negotiation of these differences with each other. The evidence, in terms of the way the approach evolved in response to stakeholder input, makes it clear that novel interdisciplinary work such as ours is needed and has considerable benefits. Shared understandings, coherence and the facility to get meaning from data are required173 if feedback about care provision is to result in health-care improvements. Nonetheless, the differences between multidisciplinary data sources, such as the ones the study team has used, need consideration. In the study, these may have resulted from sampling bias/the participant mixtures (which users of the checklist should always consider), small numbers of study participants, biases caused by the methods chosen and real variation between the types of user. To determine which factors operate, more research is needed.
The dashboard-scoping survey and term and theme mindmapping survey enabled the different stakeholders in health care to contribute to the development of the work in a cheap, simple and efficient way. This approach could be readily replicated by teams with minimal effort during the design and development phase of complex interventions research as a way to engage the different stakeholders in iterative intervention development work more generally. However, it is prone to all of the standard limitations of surveys.
Concept mapping is usually used for the evaluation of interventions in health and social care and research and the use of this method in both co-design and intervention development work is innovative. The study showed how the process could be modified not only to help mixed groups of stakeholders to reach meaningful consensus face to face in a single session, but also to explore a topic (here patient experience) and representative themes in ways that considered and preserved the different stakeholder voices. Further consideration needs to be given to approaches to validation of the consensus themes.174 The study has also shown the feasibility of adapting the software-based approach to concept mapping to enable full participation that can include people who are not computer literate. This technique could be used by others, not least to obtain information to modify the PRESENT rule-based IR-toolkit system for different conditions.
The walk-through methods that were used to surface any usability issues are structured methods, and widely used in the industry. The successful use of these methods in a health-care context is a contribution to the practice of system design in health care. Incorporation of NPT, although achieved, needs explicit implementation models to have been developed to maximise its potential (see Use of Normalisation Process Theory themes).
Overall, the innovative use of surveys, concept mapping work, structured walk-throughs and a DCE has led to particularly rich data on the needs, trade-offs, prioritisations, implementation moderators and toolkit and patient experience data usage, which can be translated into more general recommendations and checklists, as well as further in-depth analyses to be undertaken after completion of this report.
Small studies
Several authors47,49 emphasise the importance of small studies in the use of big data, and findings from these were an important element of the work.
Toolkit
The dashboard-scoping survey and scoping review showed that the focus of any clinical toolkit designer should be on ensuring minimum user effort and making role-specific tasks simple and quick to achieve. The survey suffered from sampling bias (see Chapter 3, New surveys). The review also had limitations. Although it was rigorous and systematic, given that it was a scoping review, it had a broad research question. Most studies considered the health-care professional user and few evaluated toolkits designed for patients. Many had top-down designs, so it is perhaps no surprise that many of the features elucidated are no different to more generalised good toolkit design features.
Nonetheless, the study team was able to use the findings from the dashboard-scoping survey and scoping review as the foundation for the prototype toolkit development and exploration in the group concept-mapping workshops, DCE and walk-throughs. Overall, these confirmed the scoping review and dashboard-scoping survey data and added further features. For example, the toolkit had to make patient care needs and service failures clear, for leverage in funding bids, care planning and – if a patient were to use it – treatment considerations. In addition, the toolkit had to enable comparisons between sites and years for these purposes. Searches and filters were important, but the suggested form that these should take differed depending on the approach that was used to explore them, as did the colours to use in the charts (RAG or a more neutral scheme) and the use of maps to indicate patterns in the data. Contradicting the dashboard-scoping review, web-page scrolling was deemed to be acceptable by participants across the study; possibly, this was because users did not envisage checking the toolkit on anything except for a desktop PC.
Differences need further exploration (see Transferability to other applications). However, there were more similarities than differences across methods and groups, enabling us to develop a list of recommendations for the evidence-based clinical toolkit design. This can be checked by researchers and users within health care when developing new toolkits, for optimisation.
These recommendations should not replace gestalt principles of design, which are constructed from preattentive processing (i.e. subconscious cognitive processes) and apply regardless of the use to which a toolkit is put. In the preliminary and developmental parts of the study (stages 1 and 2), the study team effectively asked stakeholders for their considered impressions of what they believe they are doing in preattentive processing, and this was explored in the scoping review of clinical digital toolkit design (see Chapter 2). It could be argued that the DCE work attempted to get at preattentive processing more objectively, although the caveats were also discussed (see Chapter 7). The checklist does not aim to replace gestalt principles, but the criteria that were developed should make it easier for designers who use these principles to appreciate health-care stakeholder needs for a digital dashboard or toolkit.
Themes
The group concept mapping was undertaken to enable mixed stakeholder co-design of the theme names and scope (drawing on the term and theme mindmapping survey and rapid review), as well as the toolkit. These goals were achieved, resulting in a draft evidence-based taxonomy of themes suited to driving improvements in the patient experience.
The study team was also able to record the conceptualisations of the different stakeholders through the way in which they sorted statements into themes. However, these data have limitations. The team considered only a small selection of statements, smaller than the number usually used (≥ 100) when the process is undertaken online. A total of 60 statements were purposively selected from the several thousands of comments in the data set, which inevitably included a wider selection of themes. The choices were influenced by findings from the term and theme mindmapping survey and rapid review of themes, and so carried across their limitations. For example, the rapid review of themes was undertaken by one researcher and was limited in scope, the sample sizes for the surveys were small and there was considerable sampling bias across the study.
Nonetheless, it is also a strength of this output that it drew on several data sources across the study, involving multiple stakeholder groups, as it is likely to represent the most significant themes in the health-care experience. The 66 comments considered in the workshops represented 26 out of the 36 themes (72%) that had been determined deductively before the workshops or that became evident inductively through further discussion within the workshops. Participants did not consider there to be major gaps in the themes used. The data had good construct validity. Larger numbers of workshop participants might have enhanced external validity, but the minimum number recommended for robust results was exceeded.121
Overall, the study team is confident that the final taxonomy of themes had satisfactory external validity, as the stakeholders themselves determined theme names and rankings. Therefore, the taxonomy has meaning across the groups (being defined by them); moreover, themes were ranked by both importance and feasibility for health-care change. The taxonomy can be widely used within health-care studies and practice. The taxonomy was further refined in the last stage of the study and it will probably develop organically as it is used. It would be interesting to revisit it over time to see if priorities and needs change, in other words, to see if the patient experience changes and improves.
Information retrieval
Using a noun/verb-phrase approach to rule-based IR with gazetteer lookups informed by stakeholder mindmapped terms and phrases, the study team was able to group free-text data into themes with reasonable accuracy for the CPES. The free-text comments for the CPES almost exclusively refer to health-care services; data sets that included significant talk about home life or contained more expansive patient experience stories were less well analysed by the process. It is important to recognise that the study was exploring the ‘worst-case scenario’, in which no adaptations were made before analysing these further data sets. The study team was also conservative in scoring for the LAPCD data. In fact, the system can be easily modified to increase accuracy on the different sources of data; this is a work in progress. The ease with which rules can be tweaked was what led us to choose the approach we did in the first place. Nonetheless, it is important to acknowledge the inherent limitations with current IR and NLP technology (see Chapter 8, Results and discussion, and Realistic expectations).
The study team will need to explore whether or not modifications for transferability reduce the accuracy of the system for CPES data; the more complex rule-based approaches become, the more unstable they may be. Thus, it may be important to keep different uses packaged separately.
Opening up debates
Governance and big data
The study has served to open up or contribute to some significant debates, which was not anticipated. One of the most important is the current debate around the open use of big data and tensions with governance. This is explored further in Some remaining challenges.
Data misuse
A second debate is around data misuse. Each decision that is made from the moment a survey is designed, through the data collection phase, to its final display in digital form, influences the next. This means that users need to be aware of the different decisions that are made and the way in which they affect usage of the data. The study team has attempted to address this with a tab on the toolkit that describes the issues. For graphics from this tab, see and . This is good practice, as highlighted by the Government Statistical Service.175 In this regard, it is to be noted that many commercial data-mining companies do not follow this practice, but offer NHS commissioners dashboards that have not been co-designed, to show data sets, such as the Friends and Family Test, in ways that they claim can be used as quality assurance measures in their own right. In fact, these data should always be used to contribute to the wider picture – albeit in a particularly useful, fine-grained and practical way – rather than as hard data. There are sufficient studies to show that each survey has limitations.176–191 Potential users are often aware of these limitations already. A study by Macmillan Cancer Support13 showed that commissioners valued CPES free-text data, especially as a means of capturing broad information from patients and opening up conversations about patient experience between commissioners and providers. However, they would consider the qualitative data only alongside quantitative data and raised a number of issues with the CPES survey sampling and case mix, as well as the problem that transitory local difficulties, such as staff sickness, cannot be taken into account in national summaries.13 The study’s participants were of the same mind.
An example of how users are being alerted to consider the limitations as well as the strengths of all data sets, including the study’s own data set. Users can mine down to find more detail on each point. This graphic relates to PESs in general (more...)
A draft of how the limitations of qualitative data may be highlighted. Such highlights may need to be customised to different uses of the system.
Realistic expectations
A third area in which we wish to open up debate is around managing expectations in a thematic analysis of large data sets and the reification of manual analysis. It has been shown how the system missed some theme annotations that seem obvious to a human, but often picked up themes that humans missed. When qualitative researchers undertake inter-rater reliability checks for manual thematic analyses, they often come up with a concordance (similar to the accuracy metric) of around 76%, which is considered to be acceptable. The accuracy figures, which are comparable (indeed slightly higher), are therefore a good achievement, although the study team will continue to work on improving them. Whether human or computer, there will always be some themes that are not accurately categorised, and thus the expectations of computational methods should be realistic.
Economic outcomes, strengths and limitations
The DCE shows purchasing behaviour to be very much dependent on the toolkit features, going from a 10% to a 90% probability to purchase (at £1500 a year) when the study team moved from using a baseline toolkit to using a fully featured one. However, extrapolating actual market demand from the experimental findings should be treated with caution. The study had a moderate retention rate of around 20% and the team members have no knowledge about the reasons for non-completion. It is known, however, that completers found the task difficult or wearying, which can lead them to answer at random or without full engagement, reducing internal validity. Even with full engagement in the task, completers may behave differently in real life; although the DCE is a relatively objective forced-choice approach, it depends on rational decision-making, when in fact humans often make irrational decisions.192
In terms of the net benefit from the use of the toolkit, the DCE data provide only a lower-bound estimate, as non-monetary benefits are not included (such as improved health-care services, improved health outcomes, enhanced data availability and research, etc.). Such evaluation is outside the scope of this project and would need to be explored in future research.
Theory
The NPT was used throughout the study to structure data collection and design work and to drive the approach (i.e. to ensure that potential professional stakeholders in the patient experience could make sense of the free-text comments analysis in coherent ways that consider the contexts in which they work). However, there were limitations in what the study was able to explore with participants in the walk-through in terms of implementation, because they were not presented with an implementation model, but simply the toolkit to trial (see Chapter 9, Discussion).
The diffusion of innovations theory provided a more general overview of the cultural issues of introducing technologies into the NHS workplace. As self-assessments were used, this is, however, potentially open to bias, as people may have over-represented their innovativeness. Moreover, the study team considered only the aspects of this theory that did not overlap with NPT, as specified in the funding application.
Some remaining challenges
Notwithstanding the success of the project, some fine-grained aspects of what had been originally planned were not feasible.
A public-facing site
Although the study team developed HTML pages for a public-facing site, these are not currently useable as a result of governance limitations. These pages could be developed by the University of Southampton at a later date, with some more funding, once patient consent rules change. When this report was written, data regulations were in flux. Long-awaited results of the government consultation on the new health data security standards and consent/opt-out model for patient data had not been published. These were disseminated a month later in July 2017,193 and confirmed the recommendations of the independent review of data security, consent and opt-outs by the National Data Guardian, Dame Fiona Caldicott.194 This review recommended:
new data security standards
a method of testing compliance with these standards
a new consent model for data sharing in health and social care.
The NHS still needs to implement these recommendations, which will take time.
In parallel, in late December 2015 the European Council, Commission and Parliament reached agreement on new data protection rules. A final draft of the new European Union General Data Protection Regulation came into force in mid-2018, and the UK’s withdrawal from the EU (Brexit) is unlikely to change this. The British government has announced that it will opt out of Article 43a of the General Data Protection Regulation, which requires there to be a mutual legal assistance treaty in place with the relevant country before a transfer of personal data can be made to that country to comply with a court order.
Given these changes, and the tightening of governance rules following NHS data breaches (unconnected to the project), NHS England restricted use of the CPES 2015 data. This is appropriate, as patients who completed the survey did not consent to new use of their data on public display. Although most study participants found this frustrating and believed that patients would be happy for this new use, a few agreed with us that it should not be done. It is important to record here that patients often felt that this was tantamount to concealment and said that they would not trust the NHS until data were fully shared and they could follow the journey of their own comments in the survey. This is something that has been acknowledged as being important in a Cabinet Office White Paper of 2013.195
Use of Cancer Patient Experience Survey 2015 data
The same governance issues that precluded the launching of the public-facing site for CPES data also delayed the acquisition of these data, which before the study – and indeed until May 2016 – had been promised to us immediately on release. The study team did not get these data in time for the draft report. Although this did not compromise the study, it reduced the immediate benefits within the NHS. However, this should not be a problem in future years or with different data (it is also noted that these data can be obtained separately from each individual trust by local agreements, at the time of release).
Use of Normalisation Process Theory themes
In the first prototype, radar plots generated from completion of the NPT toolkit were included on the main dashboard page. It was felt that if professional users completed the NPT toolkit, this would enable them to reflect on the feasibility of implementing small-scale interventions as a response to the free-text data summaries. For example, if there were a great many negative comments about teamworking, and in the toolkit the score for interactional workability was poor, this might suggest both that the unit had an inherent problem with teamworking and that it was not practical to address this. Alternatively, it could be decided that the large number of negative comments relating to teamworking could be used as a lever to address deeper issues. However, the NPT toolkit was not designed or developed to drive change, but rather to understand the implementation, embedding and integration of new technology or complex interventions. NPT has, however, been shown to be very useful when interventions are developed,52,53 and it is likely, therefore, that the use of NPT was attempted too early in the potential intervention development process. This was, therefore, a high-risk application that did not work. Nonetheless, NPT has been retained as a link for users of the site to explore, given that the dashboard has now been developed into a broader toolkit. The toolkit explains that once health-care professionals have designed an intervention to improve the patient experience, they might explore its implementation using the NPT toolkit.
Final toolkit features
Some toolkit features were prepared by us, but could not be implemented either because of current governance restrictions or because the study team did not have > 1 year of data.
As an example, the current toolkit shows dials, even though there was consensus that these were not helpful, and currently additional data from other sources cannot be added, nor data reports exported or printed (although screens can be) – see Chapter 5, Back-end decisions and Software for a discussion. However, the study team has developed HTML pages to enable the chart style to be changed, and these other features to be included, once they can be used ().
Some alternative features that were developed for the dashboard. Note that as these were developed using WCPES data, health boards have been specified rather than trusts.
Recommendations for implementation into practice and the need for further research
Use for the Cancer Patient Experience Survey
Further steps are needed between this proof-of-concept study and real-world implementation into routine management.
At the start of the study, the study team discussed with Insight NHS England the possibility of embedding the approach within the management of the National CPES survey. However, discussions were frozen while governance issues are being attended to by Insight NHS England, and the full use of the system is not possible until CPES consent is changed. Implementation also requires more qualitative research and exploration of logistics, structured by implementation science theory, once an implementation model has been worked up. This should be the first research priority. If national roll-out is not embedded through Insight NHS England, and perhaps in any case as a preliminary to this, the first step is an exploration of local (small-scale) implementation into practice and potential sustainability of the rule-based IR toolkit approach for CPES data. One route forward could be to involve local strategic clinical networks13 in piloting the approach, once the necessary refinements to the work have been undertaken. It would be possible to use the rule-based IR process and toolkit for CPES in this way with only modest further financial input to undertake rule and toolkit refinement to suit the implementation model. The potential effect of the approach on practice itself could be examined, for example; a focused ethnography might be undertaken at a pilot site using actor–network theory. A multimodal evaluation of human–computer interaction would also be informative.
Any further study will need funding. Moreover, as indicated above, small refinements of the dashboard and IR for these studies require a small amount of funding for approximately 6 months of programmer and researcher time. This funding is being sought in 2018, with potential sources already identified. Further maintenance and updates, including the incorporation of an interactive forum as desired by stakeholders, are dependent on more funding or activation of a business model. A handover strategy has been developed, as the chief investigator has left the University of Southampton, and this is in the process of being worked through. See Transferability to other surveys.
Transferability to other surveys
The results have been less promising with non-CPES data than with CPES data, although the proof-of-concept work shows that transferability is possible with minor tweaking of the rules and the toolkit, which has been designed to be possible to do for someone with only basic technical know-how. Thus, a secondary priority is to further explore the use of the system with other health-care data sets to drive health-care improvements. More thorough testing of the technology on different data sets is critical if the approach is to be used extensively – as has already been shown, what works with some types of free text may work less well with other types, for example, with a more diffuse style of writing or with the conflation of clinical and personal experiences, as with the LAPCD data. If this further testing is not undertaken, the approach would still be useful for CPES data and similar free-text data, but it would be disappointing to limit it in this way.
On this basis, and with suitable documentation, the toolkit and the rule-based IR package will be placed on a new online repository targeted at health-care professionals, which the University of Southampton Faculty of Health Sciences is in the process of setting up. This will enable potential users to adopt and amend the toolkit freely, whereas the use of the rule-based IR package will probably need to incorporate a small maintenance fee depending on how it is intended for use; this is partly to ensure the integrity of the system, as rule-based approaches may cease to work well if they become overly complex. The processes in manuals and documentation have been operationalised to be accessed via this repository, whereby the system and the toolkit that have been developed could be transferred to other surveys or modified by other users. The study team also intends to use the system for further work of this nature with other data, subject to funding.
Transferability to other applications
The rule-based IR process as a stand-alone feature has considerable application in the processing of data for research and education use. The approach results in literal themes, the first stage of a thematic analysis, freeing up analyst time to focus on more conceptual theme development. Transferability has been built into the system, but further research needs to be done to explore how this works in the different settings of health care, research and teaching.
Engagement activities
More work needs to be done to determine how to engage members of the public with such approaches and how to enable single trusts or commissioners to use them. This might be facilitated through the involvement of charities and advocacy groups or the setting up of a dedicated group. Such a group could, for example, hold regular national training and discussion events developed locally within Wessex and then spread to other areas.
Methodological research
The co-design processes that were used could be further developed and refined to improve their usefulness, with a further exploration of the degree of concordance between the different approaches.
Information retrieval is a rapidly developing area, but for uses such as ours, more sophisticated processes are not needed. However, a scoping review of what is being undertaken and its underlying theoretical frameworks might be a useful way forward, so that guidelines on good practice can be developed. This might also inform the refinement of the analytical approach for better transferability.
Cost–benefit analysis
A detailed cost–benefit analysis is dependent on further research of the type detailed above.
Final conclusion
This study provides proof of concept for an approach that could automate the aggregation of patient feedback free-text data into themes and present summaries in a visually engaging and useful way. Novel application was made of existing multidisciplinary research approaches in ‘small studies’ (literature reviews, ‘mindmapping’ and scoping the design of outputs through surveys, group concept-mapping workshops, a DCE and structured walk-throughs) that complemented the more computational ‘big data’-style IR work. The use of these methods has provided us with rich data on the various perspectives of the different stakeholders and points of concordance and discordance. It has also provided the study team with a comparison of the types of evidence that these different methods provide, which the team hopes to explore further. These small studies ensured that the approach and outputs had meaning for diverse stakeholders in health care, including service users, so that they should be able to drive improvements in the patient experience.
The study team believes that the proof of concept is the first attempt of its kind in health care. Importantly, a modular approach has been used, which can be easily adapted for other surveys and disciplines, with supporting documentation. Nonetheless, further work is required to develop this work beyond the current prototype or pilot design and to embed the approach routinely within health-care processes. This is also true of the empirically derived taxonomy of themes and checklist of recommendations for the health-care dashboard and toolkit design, which need uptake, organic refinement and validation in use. Some further research is therefore needed. The approach has shown that routine adoption within health care of carefully automated analyses of free text and a move from annual to real-time feedback models are possible, once the process is refined and barriers to its use (such as governance issues) become resolved.
