A Qualitative Analysis of EHR Clinical Document Synthesis by Clinicians
Oladimeji Farri, MBBS, David S. Pieckiewicz, PhD, [...], and Genevieve B. Melton, MD, MA
Abstract
Clinicians utilize electronic health record (EHR) systems during time-constrained patient encounters where large amounts of clinical text must be synthesized at the point of care. Qualitative methods may be an effective approach for uncovering cognitive processes associated with the synthesis of clinical documents within EHR systems. We utilized a think-aloud protocol and content analysis with the goal of understanding cognitive processes and barriers involved as medical interns synthesized patient clinical documents in an EHR system to accomplish routine clinical tasks. Overall, interns established correlations of significance and meaning between problem, symptom and treatment concepts to inform hypotheses generation and clinical decision-making. Barriers identified with synthesizing EHR documents include difficulty searching for patient data, poor readability, redundancy, and unfamiliar specialized terms. Our study can inform recommendations for future designs of EHR clinical document user interfaces to aid clinicians in providing improved patient care.
1. Introduction
The transition from paper-based media to electronic health record (EHR) systems, supported by recent national mandates for the implementation of health information technology (HIT), provides unprecedented access to vast amounts of diverse clinical data at the point of care. However, clinicians are often challenged by the ‘disconnect’ between current implementations of EHR systems and the complexities of clinical decision-making, including the organization of text-based clinical information within these systems.
Medical cognitive science emphasizes the complex nature of clinical reasoning and the significance of knowledge representation in medical decision-making. An ongoing range of cognitive processes are utilized by clinicians in constructing mental models that aptly reflect clinical scenarios and assist in making effective clinical decisions (1).
Over the last decade, an important focus of informatics research has been the development and evaluation of EHR user interfaces such that they are equipped to adequately satisfy clinicians’ information needs to effectively reduce the cognitive load of information retrieval and improve the learning process involved in using these systems (2–4). Improving our understanding of clinicians’ cognitive processes, specifically those surrounding the use of text-based EHR clinical documents, could improve user-centered cognitive models and aid the design of clinical document user interfaces (5,6). The objective of this study was to gain insight into cognitive processes of clinicians as they synthesize information from an EHR prototype, specifically concentrating on the use of text-based clinical documents as primary data sources.
2. Background
2.1. Information Overload within EHR Systems
In clinical practice, complex data processing remains an integral aspect of problem-solving strategies utilized by experts, sub-experts and novices alike (6)(7). Timely access to patient information relevant to routine and emergency clinical processes determines the clinician’s familiarity with clinical concepts and the context of clinical situations. EHR implementations provide clinicians with rich and extensive patient-specific information from a large number of sources in multiple and different formats (8)(9). Narratives (free text) recorded in the EHR by clinicians (physicians, nurses, and other healthcare providers) as they care for patients are contained in documents that have significant information over and above the structured nature of data such as vital signs and laboratory results. However, the quantity of information within these documents can be overwhelming, thereby posing cognitive challenges to clinicians reading and using these documents. Therefore, the issue with information accessibility at the point of care is transforming the balance of information from ‘having too little’ to ‘having too much’ (10).
One factor responsible for excessive information within EHR clinical documents is the frequent and often indiscriminate ‘copying and pasting’ of redundant patient information in an attempt to accurately capture pertinent details from previous clinical encounters and provide sufficient information for billing purposes. Despite the time-savings and administrative benefits facilitated by the ‘stand-alone’ clinical encounter documents, transferring information from one clinical document to another may propagate unidentified errors which could have adverse effects on patient management (11).
2.2. Cognitive Demands at the Point of Care
Assuming a cognitive network of humans and computers as the fundamental unit of analysis, some experts draw attention to the dynamic and radical changes in a given professional or social environment following the introduction of new computer technology (12). To mitigate possible increases in cognitive demands associated with learning HIT, it is important to consider information processing techniques at the point of clinical care and the extent to which these activities can be influenced by the implementation of EHR systems.
Within the time-constrained and considerably stressful patient encounter, the clinician’s cognitive efforts are devoted to consuming relevant information from previously documented clinical encounters in order to construct a mental model representative of the patient’s situation. Constructing this mental model becomes even more taxing when an unfamiliar patient’s medical record is to be reviewed during a first-time clinical encounter.
Synthesizing EHR clinical documents requires allocation of cognitive resources to processing both novel and familiar information. According to experts, no more than two or three novel information elements can be processed adequately at any one time by the working memory (WM) - a division of the human cognitive architecture where all conscious information processing takes place. Therefore, when clinicians review multiple clinical documents associated with unfamiliar clinical scenarios, the WM likely experiences a cognitive burden that has detrimental effects on professional motivation and productivity.
Leveraging on the cognitive load theory, the cognitive load associated with reviewing large amounts of EHR clinical documents may depend on how information is presented to the clinicians and the range of actions required to access the information in a format that is easy to consume (Figure 1) (13,14). If patient information within an EHR clinical document user interface is presented in a poorly organized fashion that warrants laborious ‘browsing’ to derive critical data, system users may experience frustrations and have reduced motivation for thoroughness, resulting in a increased propensity for erroneous clinical judgment.
2.3. Think-Aloud Protocol
Critical thinking can be represented as sequences of thoughts or cognitive states separated by processing activities (15). The think aloud (TA) protocol, as a scientific method through which human cognitive activities can be made verbal, was first highlighted in the mid 1940s (16,17).The principle of the TA protocol is to obtain data in the form of verbalized statements in order to investigate cognitive processes relative to certain human activities.
Based on principles of information processing theory, the TA protocol uses simulations of problem-solving tasks to elicit verbal reports that potentially reveal and describe which information is being analyzed and how the information is structured or reconfigured within the WM during a problem-solving activity (18,19). Evidences that support the use of the TA protocol include the fact that (a) human cognition refers to a sequence of internal states typically transformed by information processing, (b) these sequences of internal states can be externalized through verbalizations, and (c) recently acquired information which has become the focus of an individual’s concentration can be accessed directly as verbal data (18,20).
3. Methods
Clinicians were observed as they interacted with clinical documents within a prototype EHR system. The prototype EHR system was designed based on the user interface framework of the Veterans Affairs’ computerized patient record system (VistA CPRS) and provided basic functionalities available in most EHR systems for reviewing clinical documents e.g. a read-only document viewer and lists of authored clinical documents that can be sorted by date. Clinicians were asked to verbalize their thought processes (TA protocol) while reviewing clinical documents in the context of accomplishing a set of routine clinical tasks. The think-aloud protocol audio provides qualitative data that was synchronized with the screen display and navigation on the EHR system screen captured by a video camera in a controlled environment (21) (Figure 2). Also, a content analysis of the clinicians’ verbalizations while accomplishing the clinical tasks was performed. Approval for this study was obtained from the University of Minnesota Institutional Review Board.
3.1. Study Sample
A purposive sample of clinical interns was recruited for our study based on similar sample sizes in studies with qualitative analysis of medical cognition and clinical decision-making (21–23). We restricted participation in the research to the intern level physicians in order to control for differences in cognitive processes and medical decision-making techniques due to varying clinical expertise.
3.2. Experimental Design
Each intern reviewed nine patient records from the Fairview Health Services at the University of Minnesota Medical Center. These records contained free-text documentations of eight to nine office visits related to the management of chronic medical diagnoses such as type 2 diabetes mellitus and essential hypertension. The interns reviewed the records while performing routine clinical tasks within a simulated clinical setting.
With the assistance of two experienced clinicians (GBM, TJA), we developed clinical practice scenarios requiring ongoing assessments of clinical documents within the EHR system. An example of a clinical practice scenario is given below:
As the interns performed the clinical tasks using the patient records within the EHR system, the observing researcher would only interrupt if there is a short (15 – 20 seconds) period of silence in order to prompt the intern to continue ‘thinking aloud’.
4. Results
Six clinical interns were observed as they utilized the text-based clinical documents for these controlled patient scenarios. The average length of a scenario observation was 18.96 minutes. The technical expertise of the interns, in terms of EHR system use, ranged from intermediate to professional; each intern was familiar with at least three different vendor-based EHR systems in their clinical rotations. There were 2 male and 4 female subjects in our sample of interns between 26 and 30 years of age. Overall, 853 minutes of observations were transcribed and analyzed using the QSR NVIVO (version 9) qualitative analysis software.
4.1. Protocol Analysis
We reviewed all study transcripts to enable familiarity and to identify general impressions from the observational data. Consideration of our study objectives and literature on medical decision-making research and the use of think aloud protocols resulted in the use of a three-step coding scheme for the analysis of the study transcripts based on recognized frameworks for protocol analysis (18,24–26).
4.1.1. Referral Phrase Analysis
As a first step in the protocol analysis, the interns’ verbalizations were organized according to various concepts referred to by the nouns and noun phrases contained in the transcripts. The referral phrases identified were used in defining the concepts that constituted the main focus of intern reasoning as they performed the clinical tasks using the EHR clinical documents. The universe of concepts derived from the referral phrase analysis (RPA) constitutes an ontology for the virtual domain of information synthesis from EHR clinical documents (27). In order to ensure the validity of this coding procedure, the researcher continued with the RPA until all concepts within the transcribed data were adequately defined and coded (Table 1). During the RPA, when a transcribed statement contained several nouns and/or noun phrases referring to multiple concepts, the statement was coded under all appropriate concepts in order to ensure completeness in the data analysis and to retain the contextual information within the statement. For instance, in the following statement:
“She has had headaches since last fall. So why does it improve with Levaquin? That’s an antibiotic!”
There are words and phrases that refer to the Symptom (She has had headaches…), Time (…since last fall…), and Treatment (…So why does it improve with Levaquin? That’s an antibiotic!) concepts.
4.1.2. Assertional Analysis
In the second coding step, assertions made by the interns were coded based on how they determined relationships between verbalized nouns and noun phrases as they performed stated clinical tasks using the EHR clinical documents. The assertional analysis (AA) facilitates the combination of the concepts identified in the RPA and the existing relationships between these concepts in order to understand the epistemology (the nature, validity and limitations) of information synthesis from EHR clinical documents as reflected by the study participants (18,27). Each statement under the RPA concepts were exclusively coded based on the whether the intern established any significative, implicative or causal relationship between concepts in the statement (Table 2). In contrast to the RPA, the AA did not involve multiple coding of the same statement as each statement was assessed for the dominant relationship/assertions between concepts. For example, in this statement:
“I like that I see some of his past medical history like substance abuse.”
Despite indicating that a past medical history of substance abuse is present in the patient’s record (implicative assertion), the highpoint of the statement is that the intern asserts the relevance of the past medical history to information processing; thus there is a relationship of significance (significative) between the past medical history (Problem) and the intern’s access to clinical information (Format).
4.1.3. Script Analysis
Script Analysis (SA), the final step in the protocol analysis, was carried out in order to determine the overall configuration of the interns’ cognitive activities during the experiments; the transcribed data were collectively reviewed and analyzed based on a reference frame of cognitive operators (24). These operators were defined based on the results of preceding analytic steps (RPA and AA). The SA identified predominant reasoning and decision-making processes involved as the EHR clinical documents were synthesized by the interns (Table 3).
To determine interrater reliability, a second researcher with recognized expertise in qualitative analysis (DSP), and who was familiar with the coding scheme, analyzed a subset representing 16% of the transcripts. Overall, the mean % agreement between the investigators was 82. Coding discrepancies between the investigators were discussed and addressed for potential overlaps.
4.1.4. Cognitive Pathway
There was considerable variation in the concepts and assertions identified as each intern reviewed and synthesized EHR clinical notes within the patient records. The three most frequently occurring RPA concepts were Problem (24%), Treatment (17%), and Symptom (13%), and relationships established between these concepts were mostly those of significance (Significative, 56%) and meaning (Implicative, 29%) (Table 4). Based on these findings, in conjunction with operators observed during the SA (Review, Assume, Explain and Decide), we constructed a common cognitive pathway associated with the synthesis of EHR clinical documents by the interns (Figure 3).
The pathway begins with attentive consideration of presenting complaints/symptoms and generation of hypotheses on etiologies and disease processes responsible for these complaints (A). This is followed by a thorough review of patient-specific facts regarding previous diagnoses (medical and surgical), familial medical conditions, and medically-relevant social habits, towards providing evidence to support the clinician’s hypotheses. This process facilitates the establishment of new connections between disease processes and presenting symptoms and distinguishing between exacerbations of previous complaints and the onset of new problems (B). In further clarifying and establishing the clinician’s hypotheses, deductive analysis of medications and other treatment regimen is carried out to determine their correlation with past and ongoing complaints and to ascertain the extent to which these interventions alleviate existing problems (C and D). Finally, based on knowledge acquired from previous clinical experience and evidences gathered via information synthesis, the clinician constructs a mental model that summarizes the presenting clinical scenario, narrows the range of possible diagnoses, and decides on specific clinical interventions to address these diagnoses (E).
4.2. Content Analysis
In order to identify potential barriers to information synthesis from EHR documents, we performed a content analysis of study transcripts and concentrated on themes related to the consumption of EHR documents (Table 5).
The main themes from our content analysis included:
Difficulty with Searching for Information: While synthesizing the EHR documents to provide care in line with stated clinical scenarios, clinicians experienced difficulties with searching out vital patient-specific details due to information overload and reduced motivation to find ‘the needle in the hay stack’. Inability to identify pertinent clinical data within EHR documents towards satisfying clinician information demands at the point of care can significantly reduce provider efficiency and the likelihood of them delivering quality healthcare. Some comments related to the difficulty in searching include:
“So it’s not really too obvious what the result was. Let’s see… still trying to find out what the pathology said.”
“Am I missing something that is in here and I’m just not seeing it? I still don’t see a surgical history.”
Poor Document Readability: The general formatting of the EHR documents, including the layout of the sections within the document, largely determined the quantity and quality of information synthesized from these documents. Trending of past medical diagnoses, medications, and laboratory values were particularly difficult due to poor alignment of dates or incongruent organization of relevant patient information (e.g. interns’ comments) on reviewing medications and problem lists include:
“She has a lot of medications. I think it would be even better if they were listed in alphabetical order or some other way that would make them a little bit easier to read.”
“This is kind of messy to read. I think this is better than the other list because it has some start and end dates.”
Good versus Bad Redundancy: In most instances, the interns thoroughly reviewed only the most recent document in the electronic patient record and browsed through the rest in search of new information that may be relevant to the clinical task being performed. As highlighted in similar studies (9,11) and suggested by the interns’ verbalizations, the redundant information contained in the older documents constituted a significant cognitive burden and resulted in an increase in time and mental efforts required to review the patient records during the TA protocol. However, valuable insights about the overall clinical picture documented in the patient records often depended on the interns’ review of the redundant information as noted in statements like:
“A lot of redundancy in this note. It doesn’t flow and make the most sense but it had lots of good information.”
“A lot of these are kind of carried over from the last one, which doesn’t always change like social history and stuff like that. So, it’s good just to have it in there. But it’s not giving me any new information.”
Unfamiliar Specialized Terms: Due to the sub-expert clinical experience of the interns, and the diverse medical specialties (e.g., pulmonology, cardiology) represented in the EHR documents reviewed during the TA protocol, some terms and abbreviations specific to these specialties were incomprehensible and could not be synthesized along with other relevant patient information. Although the inability to interpret these terms did not result in misdirected clinical decisions, there was likely an increased cognitive burden associated with processing these unfamiliar terms in addition to other patient-specific information. Statements that revealed the interns attempt at interpreting specialized terms and abbreviations include:
“So, now she’s had two weeks of diarrhea. But it’s improved with a BRAT diet. I don’t know what that is.”
“Fusion of neck… fusion… neck…I don’t know what that is.”
5. Discussion
To improve the impact of EHR clinical documents on patient care, the organization and presentation of patient information should be in sync with the mental models and expectations of clinicians. Our study provides insights on the cognitive processes associated with synthesis of lengthy text-based EHR clinical documents during patient care. We utilized a think-aloud protocol to explicate the cognitive processes of six medical interns as they synthesized EHR clinical documents towards accomplishing routine clinical tasks within a simulated clinical setting. Our findings reveal that, in creating concise conceptualizations of the clinical scenarios, clinicians often synthesized information related to the concepts of problems, symptoms and treatment, thus corroborating evidence that clinicians screen and prioritize clinical information while managing information overload and redundancy as they review electronic clinical documents during patient care (29,30). The clinicians established mostly correlations of significance and meaning between these concepts, and these correlations informed hypotheses generation on etiology and disease processes, and decisions on the most appropriate treatment regimens. These insights also informed the construction of a common cognitive pathway for clinicians and provided a platform for content analysis of the clinicians’ cognitive processes to identify barriers to information synthesis from EHR documents.
In addition, knowledge from our research informed the development of recommendations for the design of EHR document user interfaces that can support clinicians’ information synthesis in order to reduce existing cognitive burden and generate effective action sequences while performing clinical tasks. These recommendations include:
Cues for Improved Visualization of Sections: Display and organization of information within EHR clinical document user interfaces can effectively reduce the likelihood of missing data necessary for appropriate diagnosis and treatment of clinical conditions. Knowledge from this research suggests that EHR document sections containing information related to the concepts of problem, symptom, and treatment are among the most critical to clinical reasoning and decision-making. Therefore, we recommended that software development efforts and HIT research be devoted to developing and implementing solutions towards visually emphasizing these sections in order to support the critical cognitive activities dependent on access to patient information related to the aforementioned concepts. Examples of possible data visualization aids include, but are not limited to, (a) distinct manipulation of fonts in sections or section headers related to problem, symptoms, and treatment; (b) line-spacing and paragraphing to better organize and distinguish these sections in the EHR document; and (c) color-coded highlighting of section headers within the EHR clinical document user interface.
Highlighting Status Changes in Patient Information: As noted above, excessive redundancy arising from ‘copying and pasting’ unchanged patient data can make it difficult to find information of interest within EHR documents, promote the propagation of data inconsistencies (11), and make the process of reviewing these documents error-prone and time consuming. However, redundant information can contribute to creating a contextual framework of clinical scenarios represented by narratives within the EHR. Therefore, to minimize the difficulty in navigation associated with duplication of clinical information and to leverage the contextual benefits of access to patient information, we recommend the implementation of methods to distinguish the most recent changes in patient information within the EHR clinical document as compared to details provided during a previous clinician-patient encounter. One of these methods involves highlighting these changes such that inductive cues are provided to aid clinicians in tracking and interpreting changes in the patient’s healthcare status over time. Further research in natural language processing (NLP) may be necessary to develop applications that identify these changes and possibly extract them for effective disease risk and patient outcome assessment. We are currently evaluating a prototype of a visualization tool to test the effect of this for clinicians in using clinical notes.
Glossary or Infolinks to Specialized Terms: Due to the continuum in clinical expertise and distinct nomenclature in several clinical specialties, demand for clinical decision support towards improved clinical expertise development may require ready access to tools that can aid the interpretation of terms and abbreviations commonly encountered while synthesizing documentations of patient care specific to certain specialties (28). Therefore, we recommend the development of customizable electronic glossaries of specialty-biased terms and abbreviations that can be edited by local and/or national clinical specialty organizations. Implementation of text-based infolinks to these glossaries within the EHR clinical document user can facilitate interpretation and synthesis of specialized terms at the point of care.
Limitations in this study include our sampling of clinicians with expertise at the intern level only, which meant we did not explore the potential influence of differences in clinical expertise and specialties on cognitive processes employed in synthesizing EHR clinical documents. Since only medical interns at the University of Minnesota participated in this study, our findings may not adequately reflect the cognitive processes or barriers experienced by other inter-disciplinary healthcare providers (e.g. nurses, pharmacists) and interns in other institutions as they routinely utilize EHR documents in caring for patients. Verbal protocols obtained while the interns synthesized electronic clinical text during the TA experiments were not controlled for quantity of speech and the possibility of additional cognitive processes directly related to ‘speaking one’s thoughts’ during task performance. Also, the design of the prototype EHR system used in the study may have influenced the cognitive strategies employed while synthesizing the EHR documents. Therefore, further studies are required to validate the observed cognitive activities as clinicians review electronic text within current vendor-based EHR applications. Finally, because the research was conducted in a simulated ambulatory setting using hypothetical clinical scenarios, TA experiments were void of any workflow interruptions and direct clinician-patient interaction (e.g. during physical examination) that are typical in realistic clinical settings. Therefore, the results of this study will need validation in the “in situ” clinical environment and among other groups of providers.
6. Conclusion
A scientific approach towards improving clinicians’ synthesis of text-based EHR clinical documents during patient care requires studying clinicians’ cognitive processes while performing routine clinical tasks using these documents. This work supports and informs the design of future EHR clinical document user interfaces. Qualitative methodologies utilized in this study were effective at revealing a range of cognitive processes and barriers associated with EHR document synthesis and helped to highlight how these processes can inform the design of EHR clinical document user interfaces. Given the limitations in our study, directions for future research include the design of appropriate validation studies to analyze cognitive processes associated with the synthesis of EHR clinical documents by physicians and other healthcare providers in various specialties and at different levels of clinical expertise within realistic patient care settings.
7. Acknowledgement
This study was supported by the University of Minnesota Institute for Health Informatics Research Support Grant. We would like to thank Fairview Health Services and the University of Minnesota interns participating in the study.
