Inter-Annotator Reliability of Medical Events, Coreferences and Temporal Relations in Clinical Narratives by Annotators with Varying Levels of Clinical Expertise

Abstract

Introduction

The clinical community creates and uses a variety of semi-structured and unstructured clinical narratives including medical reports such as progress notes, radiology reports, social work assessments and hospital discharge summaries. Longitudinal patient narratives account for various tests, procedures, and diagnoses in a patient’s medical history. Patient narratives are often written in a medical sub-language with semantic categorization of words, domain specific terminology, incomplete phrases and omission of information. They are also very temporal in nature with multiple implicit and explicit temporal expressions co-occurring with medical events. However, the medical events in clinical narratives do not occur in a temporally coherent manner. The ability to automatically extract and induce temporal order in medical events found in clinical text has many applications in tasks such as document summarization, temporal information retrieval, and automatically matching patients to temporal clinical trials eligibility criteria.

High quality annotated narratives are essential for establishing training sets and gold standards for research. We build a gold standard corpus to facilitate development of methods to automatically learn the Temporal Constraint Structure¹ and to facilitate automatic temporal ordering of medical events in clinical text. In order to do this task efficiently, we need to solve other related problems such as medical event coreference resolution and learning to anchor temporal expressions to medical events. The idea is to create a corpus with high quality annotations, which will serve as a gold standard that can be leveraged for a number of different applications. These annotations are intended to eventually lead to the development of a system that can perform automatic temporal ordering of medical events in clinical narratives.

However creating annotated clinical corpora with such detailed features is tedious, expensive and requires experts with domain knowledge. Many clinical narrative annotation efforts have used physicians, which can potentially be cost-prohibitive. Within this population, it can be difficult to find individuals willing to devote the time and effort to doing manual annotations. The main focus of this paper is to describe and evaluate an annotation effort that leverages annotators that are current students or graduates from diverse clinical backgrounds with varying levels of clinical experience. We demonstrate that in spite of this diversity, the annotation agreement across the team of annotators is reasonably high.

The main contributions of this work are as follows:

1. Measurement of inter-annotator reliability by determining inconsistencies across annotators by various annotators at various levels.

   1. We measure the consistency of identifying a word or phrase as a medical event and applying the same concept code to the event.
   2. We also measure inconsistencies in noting that medical events corefer and in noting temporal relations between events.
2. Demonstration of high agreement across annotators with diverse clinical expertise. In this study, the annotators were current students and recent graduates from diverse medical and nursing backgrounds with varying levels of clinical experience. In spite of this diversity, we demonstrate that the annotation consistency across the team of annotators is high. We also describe the patterns of agreement between annotators from these different backgrounds.

Motivation

Time is an important aspect of longitudinal clinical narratives. Frequently, temporal expressions co-occur with medical events giving the clinician some intuition about when the event occurred relative to other events in the patient’s history. However, the nature of temporal expressions tends to be varied and complicated. Moreover, the clinical narrative is usually temporally incoherent. Thus, there have been very few past efforts at tackling the hard problem of time in longitudinal clinical narratives.

In this paper, we describe annotation efforts in clinical narratives for marking medical events, related temporal expressions, temporal relations and coreference information in order to develop automatic methods to order medical events in clinical text by time. A step in this direction is to automatically induce the Temporal Constraint Structure (TCS) for clinical narratives. The TCS, proposed by Zhou et al.¹, models the time over which a medical event occurs as an interval. Each interval has “start” and “finish” time points, each of which may be constrained by temporal expressions. Based on the type of medical event and the context surrounding it, it may be possible to capture either the start time, finish time or both these times from the clinical narrative. Given medical events represented in this manner, automatically learning to order unique events on a timeline requires extensive linguistic analysis along with domain specific knowledge. An important step towards creating a timeline of unique medical events is coreference resolution. Multiple mentions of medical events may resolve to the same instance of a medical event. For instance, mentions of “heart attack,” “acute myocardial infarction”, and “spontaneous myocardial infarction related to ischaemia” may all refer to the same instance of “myocardial infarction” in the patient’s history. Biomedical domain specific information such as the semantic group of the medical event, UMLS concept identifier, related concepts in the UMLS along with some information of when the medical event occurred may help the process of coreference resolution.

In this annotation effort, we intend to capture all such linguistic and domain-specific details that will allow development of methods to enable automatic induction of the TCS and facilitate temporal reasoning. Zhou et al. describes the TCS for medical events and temporal relations in the same discharge summary. However, we consider all clinical narratives such as admission notes, radiology and pathology reports, history and physical report, social work assessment report and discharge summaries in defining our annotation format. We demonstrate our efforts on a small subset of these clinical narratives. Importantly, we annotate the dataset of clinical narratives using annotators with diverse clinical backgrounds and demonstrate high agreement. This is valuable as getting expert physicians to annotate large datasets of clinical narrative is often infeasible and expensive.

Related Work

The TimeBank corpus³ annotated using the TimeML specification is widely used for temporal relation learning in numerous natural language processing applications. A number of temporal relation annotation schemes have been developed for annotating clinical text.^{2, 4, 5} In the medical domain, TimeText⁶ defines annotations for temporal expressions found in discharge summaries. The annotated temporal expressions are extracted and represented using a Temporal Constraint Structure (TCS), which is then used for temporal reasoning.

There have been some efforts to annotate clinical text with coreference temporal relation information. Mowery et al.⁷ annotated 24 clinical reports of different types and annotate temporal expressions using the format defined by Zhou et al.² They additionally annotate Trigger Terms that are explicit signals (words and phrases) in text other than temporal expressions whether a condition is recent or historical. However, they only examine temporal expressions annotation and the annotation format does not fully capture the requirements for temporal reasoning and coreference resolution within and across clinical narratives of a patient. We attempt to define an annotation format that supports these tasks and in turn enables creation of a longitudinal record of events over a patient’s medical history.

Savova et al.⁵ propose how they are going to work towards temporal relation discovery with the long-term goal of integrating temporal reasoning into cTAKES.⁸ Their objective is to generate a timeline of events from clinical text. They observe how off the shelf parsers don’t work well with medical data as most of the parsers are trained on The Wall Street Journal. They note for example that rash is typically an adjective in newswire, but is a noun in clinical notes; erythema is not identified as a noun. They think a sufficient amount of domain training should help solve this. They also observe that in case of PropBank (used for semantic role labeling), the misidentification is due to inaccurate POS tagging. They propose using TimeML for tagging clinical narratives. They do not explain why or how these tags are the right choice.

Our annotations build on TimeML, but define new tags and tag attributes. Our contributions include the introduction of a new class of events called Medical. This is motivated by the observation that medical events do not have properties similar to events described in TimeML. Further, attributes like TENSE and ASPECT are not applicable to medical events. They may not be useful for other types of events either, as most clinical narratives are written in the PAST tense. Since our primary interest is in ordering medical events, we argue that there is a need to define special, domain-specific attributes for medical events that help identify coreferring medical events and help relate medical events to each other as well as other types of events, in order to generate a timeline of unique medical events.

Annotations

Medical Event Coreference

Another important annotation label is the coreference label. Annotators are assigned the task of marking co-referential events within and across clinical narratives for each patient. This involves identifying all medical events across the narratives for a patient and determining which mentions of events across these clinical narratives are referring to the same instance of an event in time. This enables us to create a non-redundant list of medical events that have occurred in a patient’s medical history.

We can see examples of medical events that corefer within and across clinical notes in Figure 1. It also illustrates that multiple mentions of the same medical event may or may not resolve to the same instance of the medical event, i.e. corefer. The mentions of myocardial infarction/ chemotherapy indicated by the same color corefer whereas the ones that differ in color do not corefer.

Open in a separate window

Figure 1:

Medical event coreferences within and across clinical notes of a patient. The medical events connected by arrow with label “SAME” corefer whereas the ones connected with the “DIFFERS” arrow do not corefer.

We study the following coreference relations between medical event pairs, “same,” “part of,” and “type of.” In case of “same” case, the medical event mentions are mostly synonymous and refer to the same instance. We also count abbreviations as part of this category of coreference relations. Examples of such coreference pairs include (acute myocardial infarction, heart attack), (B-cell CLL, chronic lymphocytic leukemia). In some cases, a medical concept may be “part of” or “type of” the other medical concept, as in the case of (mass, specimen) or (B-cell CLL, evidence).

Consider the following excerpt from a clinical narrative. “Endoscopic ultrasonographic examination of the upper gastrointestinal tract to the second part of the duodenum, performed 13 months before admission, revealed a large, submucosal, noncircumferential mass. The mass was well defined and did not appear to be locally invasive.

Examination of a specimen obtained by fine-needle aspiration showed evidence of an extranodal B-cell lymphoma of the MALT type.”

In this example, the medical events specimen and evidence corefer with the medical event mass. Here, specimen and mass are linked by a “part of” relation, i.e. specimen “part of” mass; whereas specimen and evidence are linked by “type of” relation i.e. specimen “type of” evidence. We intend to capture all such examples of coreference described above in our gold standard that will help us build and evaluate automatic methods to identify pairs of medical events that corefer.

Data

The corpus used in this study consists of three clinic notes from a chronic lymphocytic leukemia (CLL) patient’s record. This patient was one of approximately 2060 CLL patient records we have collected over the last 10 years at The Ohio State University Wexner Medical Center’ (OSUWMC) and are continuing to annotate. The notes consisted of a discharge summary, radiology report and a history and physical report with an average of 600 words per narrative.

Methods

A team of 5 annotators with diverse backgrounds, but with some experience in understanding medical terminology was hired to annotate the corpus. The background profiles of our annotators were as follows. Our team had one medical student with a degree biomedical engineering, but no clinical experience (medstud); three recently graduated nurse practitioners with clinical experience gathered through the process of receiving their nurse practitioner degrees (np); and one graduate entry nurse practitioner student with some clinical experience and experience in working with biomedical documents (nstud). In order to achieve the level of detail we wanted in our annotations, each annotator required approximately one month’s effort. This included clinical informatics IRB training, getting them familiar with the UMLS, explaining the motivation behind the task, having them read and understand the annotation guidelines and annotating a sample clinical narrative. The annotations efforts described in this paper were coded in Excel sheets.

An important aspect of annotating a large corpus is consistency. We measure consistency in terms of inter-annotator reliability. Inter-annotator agreement measures the consistency in annotating a particular concept across annotators. We measure inter-annotator reliability using Cohen’s kappa statistic¹¹. Kappa is interpreted as the proportion of agreement among raters after chance agreement has been removed. It can be expressed as follows:

Chance agreement is estimated by the proportion of agreements that would be expected if the observer’s ratings were completely random. Chance agreement increases as the variability of observed ratings decreases. The use of Kappa requires minimal assumptions about the underlying nature of the data. Three data collection conditions should be met: 1) The subjects to be rated are independent of each other, 2) the raters score the subjects in an independent fashion, and 3) the rating categories are mutually exclusive and exhaustive. The flexibility of different forms of kappa is also a major advantage. Kappa is appropriate for nominal and ordinal data, where there are two or more raters per subject. Kappa can be calculated for each scale point or averaged into a generalized Kappa across the entire set of ratings.

We use the methods proposed by Conger¹² to calculate agreement between multiple annotators. Conger suggests a multiple-rater agreement statistic obtained by averaging all pairwise overall and chance-corrected probabilities proposed by Cohen (1960)¹¹.

For annotation of medical events, if the annotators marked a partially overlapping section of text as a medical event, we considered it to be an agreement. For medical event coreference, we considered agreement in a pairwise fashion. For example, for events A, B, and C, if annotator #1 identified events A and B corefer, and events B and C corefer, but annotator #2 only identified events B and C corefer, we count that as having 1 annotation in agreement. We also did not consider transitive closure. In the example given, if annotator #1 also identified events A and C coreferring and annotator #2 also identified events A and B as coreferring, which would count as having 2 annotations in agreement, and not 3. For temporal relations, we considered whether or not the same pairwise temporal relationship between events was identified. With medical events, we further analyzed whether or not the coders identified the medical events with the same UMLS CUIs.

Results

Error Analysis

While the inter-annotator agreement for medical event CUIs was lower than for medical events, coreference, and temporal relations, agreement was still very high. Figure 1 indicates that the pattern of agreement across various annotations by different annotators with varying clinical expertise is more or less uniform.

In an analysis of the reason behind the discrepancies we discovered that in many cases there was either a discrepancy in the granularity to which the medical events were coded or whether or not clinical judgment was used in selecting the CUI. For example, all of our annotators marked “B-Cell CLL” as an event. The three NPs coded this term as “C0023434: Chronic Lymphocytic Leukemia.” Both medstud and nstud coded this event as “C0475774: B-cell chronic lymphocytic leukemia variant.” While both could be considered correct annotations for “B-Cell CLL,” C0475774 is the more specific term. In another example, all of the annotators marked the phrase “white blood cell count of 10,000.” For this situation, medstud selected “C0750426: white blood cell count increased,” while nstud selected “C0023508: White Blood Cell count procedure.” In contrast, all three NPs selected different CUIs, applying clinical judgment to the medical events. Np2 selected “C0860797: differential white blood cell count normal.” Overall we found the medical student’s (who did not have any real life clinic experience) annotations remained true to what was observed and could be inferred based on the data. However, the nursing student and the nurse practitioners often used clinical judgment to infer certain annotations that were not directly observed in the data. For instance, classifying something as an acute condition based on certain readings or values in the text.

Discussion

The temporal constraint structure (TCS) proposed by Zhou et al.¹ was restricted to individual discharge summaries. Our annotation scheme is intended to capture medical event attributes in longitudinal patient data found in the form of clinical text in admission and progress notes, radiology and pathology reports, discharge summaries and other types of clinical notes. Given these attributes, we can build systems to automatically induce the TCS. The TCS represents medical events related temporally and constrained by time. These can be learned with the help of the medical event annotations and the temporal relation annotations respectively. We can further improve this structure by resolving coreferences between medical events that are the same. This leads to a more accurate and effective TCS representation for temporal reasoning between medical events. Moreover, since our annotations capture attributes in longitudinal patient data, the induced TCS can be used to temporally reason across medical events found in various clinical narratives of a patient. As a step in this direction, our initial efforts towards learning a coarse temporal order for medical events within a clinical narrative is explained in Raghavan et.al.¹⁴ Further, our efforts towards automatically learning pairs of medical events that corefer is illustrated in Raghavan et al.^{15, 16}.

Consider the following excerpt from a patient case report in the New England Journal of Medicine:

“Twenty-two months after the implantation procedure, the level of PSA was 0.6 ng per milliliter; the PSA was less than 0.1 ng per milliliter 5 years after the procedure. Approximately 6 months before this evaluation, routine follow-up urinalysis revealed microscopic hematuria. The level of PSA was reportedly less than 0.2 ng per milliliter.”

There are 3 mentions of PSA in the excerpt. A PSA is a prostate specific antigen test that measures a protein produced by the cells of the prostate. As these were 3 separate tests involving 3 separate blood draws at different times (and having 3 different values), they do not corefer. But there may be other mentions of PSA in the narrative that do corefer. We can learn to automatically identify which mentions of PSA corefer with the help of the annotated temporal relations. The temporal relations clearly indicate the temporal order of the 3 PSA tests. This in turn helps us differentiate that the mentions of the PSA tests are indeed 3 different instances of a PSA test.

However, we do not annotate for pronoun anaphora resolution. Savova et al.¹⁷ focused on the annotation of anaphoric coreference.

One limitation of our study is the small number of narratives. The main reason for this limitation is due to the large amount of effort required to annotate the narratives to the detail that we desired. Given that the 3 narratives used in the study required a month of effort for each annotator, we needed to begin having the annotators annotate non-overlapping narratives in order to increase the overall size of our gold standard. Another limitation of our study is the lack of a physician annotator to compare their annotations. Given the amount of time required for our existing annotators to complete the annotations, having an additional physician annotator was not feasible. One reason for the time required to generate these annotations may be the lack of sophisticated annotation tools. In future, we plan to create the annotation schema using Knowtator¹⁸ and train our annotators to use this tool.

Conclusion

We have demonstrated that across our diverse group of annotators, ranging from current students to recent graduates from diverse medical and nursing backgrounds with varying levels of clinical experience, inter-annotator agreement is high for medical event identification, medical event coreference, temporal relation annotation, and medical event concept coding, inter-annotator reliability is high. The average inter-annotator kappa statistic for medical events, coreferences, temporal relations, and medical event concept unique identifiers was 0.843, 0.859, 0.833, and 0.806, respectively. The agreement was calculated on a small subset of chronic lymphocytic leukemia narratives.

Many other clinical narrative annotation efforts have used physicians, which can potentially be cost-prohibitive. In addition, identifying physicians who are willing to devote the time and effort to performing annotations is difficult. By using current students and recent graduates of clinical degree programs, we are able to capitalize on a readily available and valuable resource for annotation intensive research Considering the high inter-annotator agreement amongst this diverse cohort of annotators, we believe that it is unnecessary to utilize physicians for performing the annotations described in this study. However, we intend to further evaluate the quality of the annotations that we have generated by having a group of physicians review the annotations from the notes in this study. We believe that this study provides the foundations upon which to build upon for doing a larger study regarding the level of clinical expertise needed for developing a gold standard.

Acknowledgments

References