Having undertaken the groundwork for developing the free-text analysis and display approach, in this chapter the processes involved in rule-based IR are considered in more detail. This is followed by a detailed explanation of the solutions to problems with the analysis of survey free-text comments. In addition, the chapter describes problems that became apparent a posteriori; for example, it was realised that real-name redaction by the data suppliers [QualityWatch; URL: www.qualitywatch.org.uk (accessed 9 October 2018)] was incomplete, and a new secondary aim was to provide a solution to this.
Introduction
GATE (a General Architecture for Text Engineering) is a software framework and collection of resources that can be used for various natural language processing (NLP) tasks.44,116 NLP is a field of computer science that involves getting computers to process and interpret human language. This covers a range of possibilities, such as:
parsing (i.e. sequentially ‘reading’ or analysing) natural language and ‘annotating’ it for grammatical features and other syntactic elements
extracting information from segments of natural language
actually ‘understanding’ natural language.
Each of these possibilities usually begins by ‘tokenising’ text, that is, breaking it up into words, punctuation marks, numbers and other discrete features. This is a very basic level of analysis, but provides a lot of information and packages the text into units for further processing. The next stage is usually to classify words into their grammatical categories or POSs and label them accordingly, which is known as POS tagging or tagging. Each token is associated with a POS. Tokens and tags are forms of linguistic annotation of the text.
One branch of NLP that is relevant and of interest to us is best described as information extraction (IE) from the text.117 IE is concerned with the extraction of structured data from unstructured or semistructured data. Free-text answers on a survey are an example of unstructured text data. An example of semistructured text data is the ingredient lines in recipes for cooking food. Data that are grouped into themes provide an example of structured data.
Rules-based NLP approaches, such as the one used in the study, are based on an expert system of rules hand-coded by humans (see Alternative approaches to the analysis of free-text comments and Information retrieval for the rationale for choosing this approach). As the system becomes more complex, the interactions of these rules can also become more and more complex. Other non-rule-based approaches include statistical NLP approaches: NLP using statistics, probabilities, machine learning and similar techniques. Machine learning can be supervised (annotated corpora as learning sets), unsupervised or semisupervised. Different approaches can be combined.
GATE is being used in PRESENT for a gazetteer lookup and rule-based approach to IE from the free-text responses to survey questions. The rule-based approach is an approach that is suitable for the type of IE the study is interested in (i.e. theme extraction), whereas more complex information understanding would require more complex approaches. Tweaking gazetteers and rules to work with new data is far less time-consuming for a small team than annotating large numbers of new training data to create a sufficiently varied training set for every new set of survey questions for the template-based machine learning approach that members of the current team have used previously. This was a key factor in the choice of GATE for the study.
Individual GATE resources perform different elements of the work of parsing a piece of text and annotating it to allow extraction of information from it. These resources are assembled together into pipelines (series of resources), allowing the results of one resource’s processing to be used by other resources further along the pipeline. The result is a set of annotations to the original text.
The gazetteer component of the approach allows us to make use of resources, such as WordNet, and lists of words and phrases specifically related to the health-care domain. The use of gazetteers allows us to improve the ability of the system to specifically recognise the topics of interest for a given survey and tweak what is being looked for to suit differing questions in new surveys.
The rules portion consists of sets of rules written in Java Annotation Patterns Engine (JAPE). This is the core of the rule-based approach within GATE: a set of rules for matching patterns in the text and performing actions on the text. It provides regular expression-based pattern-matching; this simply means that it can ‘recognise’ or match elements of grammar, such as POSs, and the text annotations created by prior JAPE rules, gazetteer lookup or any of the other prior steps in the processing flow. These JAPE rules can add new annotations when a positive match is made. The rules can also undertake complex processing once potential patterns have been matched, to further determine if a new annotation should be made, and better disambiguate which type to make an annotation in cases in which a possible match can be made for more than one annotation type or entity type (i.e. a single unique object, such as a person, object or organisation).
Challenges in analysing survey free-text comments and key foci
Free-text comments raise a number of challenges for NLP programming:
‘messy’ language that does not neatly follow formal language rules: unusual punctuation, typos, capitalisation, spelling, use of slang, regional variations and figures of speech, sarcasm, garden path phrases (e.g. ‘the horse jumped over the gate fell’)
terse structure with messy syntax (e.g. short partial sentences, abrupt topic switches and odd lists of unrelated items, such as ‘the food, bed and staff’
lack of context, giving rise to ambiguities (e.g. comment boxes containing only the words ‘excellent’ or ‘nothing’, which might be responses to the original survey questions – ‘Was there anything particularly good about your NHS cancer care?’, ‘Was there anything that could have been improved?’, ‘Any other comments?’)
word use – emotive word use (e.g. ‘soooo not good’), words with multiple meanings (e.g. ‘may’), new words or acronyms [e.g. TBH (to be honest)]
co-reference – for example, using people’s names (e.g. ‘Bob’) to refer to the condition or body part affected and then starting the next sentence with ‘He’ might lead the process to annotate ‘He’ as if it were Bob (see
Chapter 3).
These features of free-text comments are a challenge to language identification tools and can lead to linguistic pre-processing failure that may decrease the performance of standard IE tools considerably. Errors early in the pipeline can have a knock-on effect. For example, the GATE developers at the University of Sheffield report a decrease from ≈90% to ≈40% accuracy when the standard GATE program is run on tweets rather than on news articles.118 A similar reduced accuracy was found when WCPES free-text comments were run through the standard GATE program.
The initial key foci, or research problems, were therefore to:
determine figures of speech that might confuse the system [through survey consultation and experimentation (see
Chapter 3)]
cope with the syntactic irregularities and terse style of the texts (through programming)
develop gazetteers that could cope with word-use issues (including programming)
associate answers with the questions to reduce ambiguity (through programming).
A sentiment analysis was also incorporated that could work with the peculiarities of these data (through programming). Comments for a theme could be positive or negative, and this was indicated rather than separating comments into positive and negative themes per se.
As data were worked on, further problems were determined, which are described with their solutions in the section Additional issues realised during the study and their possible solutions.
Data used to develop and test the approach
A list of potential target themes from previous work (see Chapter 3) was developed to enable domain-specific gazetteers and possible rules to be developed, which mapped to existing themes, concepts and structures referred to within the NHS. The study team was required to go through a rigorous NHS permissions and advanced data management training process before it was possible to obtain the most current national England CPES data. Hence, half of the data from the 2013 WCPES were used to determine the challenges that the team was required to meet and to test the solutions and debug them [i.e. identify and remove coding errors; the other 50% was used in the validation stage (see Chapter 8)]. (The study team had permission to use WCES data as a result of approvals gained during the earlier work, although not to disseminate them via the new system.) As is usual in text analytics, this was an iterative process, with sensitivity and specificity improved through refinement of the gazetteers and rules. Less commonly in text analytics, the original target themes were modified through the co-design work (see Chapter 6), meaning that gazetteers and rules were added to at an advanced stage of the process, and further debugging was needed.
The main task: coping with the syntactic irregularities and terse style
The nature of survey free-text comment data compared with more formal text meant that some difficulties needed to be overcome to provide an accurate automatic thematic analysis. The sentences are highly complex and often diverge greatly from standard syntactic structure, as described in the section Challenges in analysing survey free-text comments and key foci. This rules out ‘Word_A is within n words of Word_B’ rules and means that standard computational syntax cascades cannot be applied without further modules to pre-process the data (notably this also means that the training set approach would require more templates than for more formal text – further reducing its usefulness compared with the solution that has been developed – and without necessarily effectively resolving conflicting or ambiguous annotations).
Approach rationale
The way the target themes were named meant that rules could be written that did not require the subject, verb and object of the comment to be determined, although the code that Laurence Antao wrote is easily adaptable to do so. Nonetheless, syntax was still important for accurate rule-writing and for sentiment analysis. The study team wished to identify subclauses to break the comments down into meaning units, but with standard GATE parsing this was often inaccurate. Laurence Antao determined that using noun phrases, verb phrases (including object noun phrases) and adjectival phrases, and also prepositional phrases along with subclauses, made it possible to develop an effective rule-based approach to create syntactically viable lists of topics using domain-specific gazetteers.
Syntactic patterns
Apache OpenNLP, version 1.8.4 (The Apache Software Foundation, Forest Hill, MD, USA), a GATE parser plugin (i.e. a software component that adds to an existing computer program), is good at determining noun phrases and verb phrases, and annotates them hierarchically. This has two important (non-mutually exclusive) features that are of great use. First, noun phrases may contain multiple noun phrases within them (likewise, verb phrases may contain multiple verb phrases – as in the garden path example in Challenges in analysing survey free-text comments and key foci). Second, verb phrases may contain noun phrases. Whatever the structure of a sentence, a clause will always have reference to a verb phrase and a subject phrase, whether that be within the clause itself, in another clause within the sentence or in another sentence completely (although this is not true of single-word sentences or very short sentence fragments that stand alone, this study is considering surveys, in which there is always a question being asked). By pairing up verb phrases to noun phrases, with phrases that are able to be paired multiple times, syntactically correct semantic meaning can be drawn from highly complex sentences and non-sentences (). The full meaning of a sentence can then be gleaned by using rules to interpret conjunctions and verb tenses (e.g. a conditional tense implies the negative of that action in the present).
How a complex sentence can be broken down into its constituent noun phrases and verb phrases (prepositional phrases were also noted). These can be parsed in turn by the system so that it picks up each and all variations. NP, noun phrase; PP, prepositional (more...)
The parser that was used outputs subclause category SyntaxTreeNodes (tree representations of the abstract syntactic structure of the programming source code), which means that rules can be created to differentiate between noun and verb phrases that are next to each other but not necessarily directly connected.
GATE pipeline
In , the ‘processing resources’ (PRs) highlighted in yellow are those that interpret the clause structures and produce the syntactically feasible lists of topics contained in the verb phrase and noun phrase ‘pairs’. The PRs take, as input, the keywords produced by the blue elements, some of which contain a polarity (‘negative’ Boolean) and ‘degree’ (0 = average, 1 = very, 2 = extremely). The ‘themes’ JAPE transducer PR (highlighted in red), then assigns the different ‘pairs’ to their corresponding themes. The green highlighted PR merely annotates hospital names or trust names with their corresponding trust. The non-highlighted PRs are the default PRs of GATE known as ANNIE (‘A Nearly-New IE’ system) PRs (although the sentence splitter has been swapped with a more efficient one). The JAPE transducer works out whether or not the phrase is negative, taking into account modal verbs and multiple negatives, as well as ‘if’ statements; there is not a necessity to factor in negative conjunctions. For convenience (at the theme stage), if a keyword is determined to be negative, its majorTypes or minorTypes list entry will be preceded by ‘not’.
The PRESENT GATE pipeline.
The GATE sentiment analysis JAPE was used and extended by adding relevant domain-specific entities to the JAPE so that the sentiment analysis can recognise specific entities from the domain-specific work to apply sentiment to.
Rule logic
Rules depended on a hierarchical approach upstream and an ‘if–then’ format and Boolean logic downstream and for gazetteer lookup.
An example of a rule that draws on gazetteers using Boolean logic is given below, where | means ‘or’. Rules can be easily tweaked by adapting or changing the gazetteer chosen.
Word use issues and WordNet
Although a rule-based system is potentially better able to cope with the unfamiliar than memory-based systems, it is not foolproof. The system has been built to take this into account and accommodate a wider possibility of previously unseen input words in a parsimonious way, to allow for the interpretation of unexpected vocabulary and reduce the reliance on human prediction. This has been achieved using the WordNet dictionary to tag words with synonyms and hypernyms and then annotating the most specific gazetteer entry with the flexible gazetteer PR. A flexible gazetteer was used rather than JAPE rules to make for easier thematic decisions, as it essentially presents the features to be added to the terms in a spreadsheet. This should be better in terms of legacy and accessibility, and should also make adding terms quicker.
One drawback of using WordNet is that it could lead to words being included in lookup that are inappropriate for the task and, therefore, lead to inaccuracies. Thus, it might become useful to put in delimiters. Users will need to balance the sensitivity and the specificity.
Output granularity
Through the use of metadata annotations, theme data were generated:
at the individual, local and national levels
considering demographics (ethnicity, gender and age of respondents)
considering tumour type, year or region
by sentiment.
The text and tags that were produced by the GATE pipeline were used to populate the dashboard at these different levels. Programming code was used to enable a user to run the GATE application via GATE Developer and then update and use the dashboard directly from a computer.
Additional issues realised during the study and their possible solutions
Issues of providing full comments publicly
One issue is that patients did not consent to have their comments publicly displayed when they completed the CPES. Therefore, the full comments cannot be shown on an open site.
Trust disambiguation
Respondents to a survey may be referred from the trust to which their survey is attributed to a centre in another trust for some aspects of their care. Previous work by some of the research team suggests that this occurs in 9% of cases and has the potential to cause significant problems.14,21
Rules can be created that compare the trust name in the metadata for each comment with the name of organisations mentioned in the comment to see whether or not they correspond. The system includes such a rule, and also, in case the trust files need to be updated or edited for specificity of trust (names have several shortened forms), there is a Python program (version 3.6.5; The Python Software Foundation, Wilmington, DE, USA) to automatically create the gazetteer files from the Extensible Markup Language (XML) data file available on the NHS website containing trust names.
Redactions
Personal names were identified for redaction to ensure the privacy of patients and other individuals and confidentiality of patient information. This required disambiguation between the names of persons and of hospitals that happen to contain personal names, such as Jimmy’s for St James’s. Real names were replaced with [health-care professional name], [patient name], [hospital name] or any relevant alternative.
HealthWatch redactions, such as [xxxx] or [name removed], were replaced by the use of programming to indicate the entity type, such as [hospital].
Words with multiple meanings
Rules were developed to identify words that might be confused with a person’s name, such as X-ray or hospital ward.
Data volume
Loading large batches of data into the GATE graphical user interface (GUI), to play with visually, may cause problems for the GATE GUI. The developers of GATE advocate the use of their cloud system as an alternative. ‘GATE Embedded’ (a version of GATE usually used by developers) can handle much bigger data sets than the GUI can, but the GUI is more useful for looking at things when developing new rules. This means that refinements in development should be made on subsets of any data, but for the final outputs, large batch-loading is not a problem. Therefore, the option of choosing to load into a corpus up to 500 or 1000 rows only was included.
Further possible refinements
The development of systems such as this is always iterative, and other adaptations could be made to it; this is currently being considered. A couple of such possible new features are specified here.
Themes extracted
The themes for the process were determined through co-design work with the different stakeholders in health care, but this work was started with a finite pool of themes determined from previous research (see Chapter 3), an initial survey (see Chapter 3) and the literature (see Chapter 2). This is a deductive approach. Using Python, a latent Dirichlet allocation (LDA) option was developed. This is an unsupervised clustering technique that can be used to group texts by discovering apparent topics that they share through looking at words that occur together in some of the texts but not all (words that occur in all of the texts are expected to be common words rather than words indicating a topic). LDA can be used as an inductive topic-modelling/topic-discovery approach; the PyLDAVis library (The Python Software Foundation, Wilmington, DE, USA) (a Python library for interactive topic model visualisation) was used to better visualise the results of the LDA. This is potentially useful to:
see how close results come to what humans with domain knowledge have worked out the topics of interest are
see if the LDA finds topics that the humans had not thought of, but that are of interest.
For future data sets, topic discovery could make a useful addition to the process of humans mindmapping which topics are of interest. This option is particularly useful for free text for which the topics have not been predetermined or when there is insufficient prior information for a useful list to be drawn up deductively. It could also be used for discovery within the existing data set.
Latent Dirichlet allocation/topic-modelling does have drawbacks that need to be considered with the data:119
Topic discovery is a black-box approach and is prone to overfitting.
Efficiency decreases when a very large number of topics are used to fit the LDA.
The LDA performs better when the underlying topics are well separated (in the sense of Euclidean distance).
‘Topics’ discovered in an unsupervised way may not match the true topics in the data.
It is theoretically impossible to guarantee the identification of topics from a small number of documents/comments.
Short documents, such as free-text comments, may result in poor performance of the LDA.
In addition to LDA, the system could incorporate further unsupervised machine learning techniques in similar supporting roles to improve the automation of the overall system.
Case sensitivity
Existing PRESENT gazetteer entries in the pipeline are case insensitive (case sensitive = false). The assumption was made that words would not mean something else entirely if they were uppercase versus lowercase. This eliminates the need to make a duplicate of every lowercase word that might be at the start of a sentence within the gazetteers. It also helps to prevent errors arising when the case is used wrongly within a comment. This means that if there are any words that need to be case sensitive in the future, a second case sensitive = true gazetteer needs to be added to the pipeline for these words. This might be required, for instance, if acronyms are used that would otherwise be taken as normal words.
Transferability to other surveys and data sets
There are two key aspects to transferability of a domain-specific system to other surveys and health-care data sets. First, and most importantly, a system needs to be easily adapted. The power of the PRESENT system is derived from its modular nature, which also allows it to be quickly adapted and refined. Modification need only occur in:
The lower-level rules and gazetteers, hence it is relatively uncomplicated to make modifications.
Relating the responses to the questions, which are likely to differ across surveys. This may be incorporated into the automatic analysis.
Second, the system needs to be able to use alternative sources of data. For PRESENT, an approach was developed to batch-load text from a comma-separated values (CSV) file that could be varied in accordance with the number of header rows in the data file and the column of the spreadsheet in which the data occurred, as well as an identifier tag for the comment.
General maintenance needs
The system could run as is for a few years on the CPES, but is likely to become increasingly in need of maintenance because of:
Implications for research and health care
Using the approach, a structure can be systematically applied to PES free text to make it more accessible to health-care providers and more quickly summarised than by existing approaches. The approach produces literal themes in a deductive process that researchers could use as the first stage in more conceptual analyses.
For the sensitivity, specificity and transferability tests, see Chapter 8.
Summary
Rule-based parsing was applied to patient answers to the free-text questions in the CPES via gazetteers and custom JAPE rules, using the GATE framework augmented with Python programming. Responses were categorised for their use in improving feedback to cancer care trusts on the patient experience. Rule-writing has sought to solve the following, in addition to producing a basic thematic analysis:
Accurate interpretation of disparate and syntactically errant free-text comments. The modular hierarchical nature of the rules allows this to be done by operating the lower-level (e.g. verb phrases, noun phrases) ‘best-match’ tags from the SUPPLE parser (University of Sheffield Prolog Parser for Language Engineering; University of Sheffield, Sheffield, UK). Thus, as with word recognition, interpretation is not reliant on the development corpus, and disparately structured entries will be interpretable.
Incorporating a WordNet feature. This means that the study’s gazetteers do not have to be exhaustive, but rather they need to have a sufficient range of words for WordNet to augment with synonyms and antonyms to provide a vastly expanded lookup vocabulary.
Relating answers to the survey question so that responses that would otherwise have been incomprehensible, such as ‘good’ or ‘nothing’, can be contextually analysed.
Reducing the ‘misattribution’ problem to accommodate situations in which a patient associated with one trust/site mentions receiving treatment at a different site. Names of hospitals and trusts were identified and disambiguated to enable feedback to be directed to the correct institution.
Sentiment analysis of the responses was performed by adapting a GATE tool that was originally designed for use on Twitter (Twitter, Inc., San Francisco, CA, USA; www.twitter.com) to the health-care domain. This was made even more sensitive by using WordNet to identify metonyms and programming to assign one of five levels of sentiment (‘very good’, ‘good’, ‘neutral’, ‘needs improvement’ and ‘needs a lot of improvement’).
A system was developed for batch-loading new data sets.
Rules were developed to identify and mark people’s names for redaction, which takes account of hospital names containing personal names. The study team also aimed to replace the redactions throughout with entity types.
Rules were written to identify words that might be confused with a person’s name, such as X-ray or hospital ward.
Transferability and maintenance were explored, and key issues were dealt with. The modular nature of the system and the need to alter only lower-level rules and gazetteers make this relatively unproblematic. This included the potential deployment of LDA to identify or discover possible topics of interest in the corpus of patient responses.
Through the IE work, theme data can be:
generated at the individual, local and national levels
generated considering demographics (ethnicity, gender and age of respondents)
generated considering tumour type, year or region
scored using sentiment analysis to measure the polarity and strength of mood expressed in a comment.
