Who Said It? Establishing Professional Attribution among Authors of Veterans’ Electronic Health Records

Abstract

1. Background

The narrative clinical text within an electronic health record (EHR) presents users with complex and sometimes conflicting clinical information about a patient. Such document variability results from a wide variety of training levels and domain expertise among the clinical providers who generate documentation. Subsequent use of these records for patient care requires an assessment by the reader as to which documents should reviewed for their clinical purpose, and among those, which are preferentially utilized in case of information conflicts. This requires an evaluation of document characteristics such as relevance, veracity, completeness, and an appropriate degree of granularity. Document quality is critical at the point of care, where time is often of essence. The clinician’s search experience in patient charts overstuffed with notes that have little information value can be a needle in the haystack exercise.

Knowledge of the professional characteristics of an EHR’s author, such as the type and duration of clinical training, can be particularly informative for document relevance and veracity. The inability to determine this information among clinical narrative text results in uncertainty by the reader and limits the utility and usability of the data for future clinical application (1) (2). In clinical care, some of this uncertainty can be resolved through the readers’ personal knowledge of the author or by reading the narrative text for further inference clues. However, for automated clinical decision support and research, these options are not available. Operationally, what is required in supplying clinical decision support systems with profession information is to make this data computer interpretable.

An author’s professional attributes would allow humans or computer applications to rank the assertions encountered in health care documents for veracity, relevance or efficacy of expression. With a reliable author attribution mechanism, tracing errors introduced into a medical record would be more tractable than it is presently. Medical errors often fall into categories of profession and specialty, thus tracing documenting practices according to professional information would benefit quality of care investigation and reporting (3). In the context of healthcare payment, charges are billed relative to the profession of the provider documenting a given service. A consult document which is not authored by a physician or qualified non-physician practitioner, for example, represents a potential an instance of fraud (4). In addition, since the text of notes authored by medical students are not legally part of the record, the training status of EHR authors must be independently verified by other healthcare staff as well as billing and legal personnel (5). An audit trail with the author’s name and professional attributes would make the link between the act of documentation and the documenter explicit. An EHR with such audit trail functionality would require computer interpretable data pertinent to profession, specialty and training level of the EHR author. The implementation of the Health Information Technology for Economic and Clinical Health Act (HITECH) will likely generate a massive expansion in the exchange of electronic protected health information, and widen the scope of privacy and security protections under the Health Insurance Portability and Accountability Act (HIPAA) including providing audit trails of all electronic record disclosure.(6)

Many EHR systems attempt to maintain a structured registry for all clinical providers of their professional training category and training level, but there are a number of limitations within this data because they are not populated for clinical purposes. The training level of a provider changes over time, and these changes are poorly maintained within most systems, and many providers change clinical specialties or their official clinical training does not represent expertise for certain types of patients. In addition, these types of databases are administrative, clinical privilege, or payroll-based, and are rarely accessible by point-of-care clinical providers or health service researchers. In most current EHR systems, computer interpretable profession and training data are not available. Neither the Veterans Administration’s EHR, Computerized Patient Record System (CPRS), Epic, the EHR used at the Vanderbilt Medical Center, StarPanel nor the Longitudinal Medical Record (LMR) at Partner’s Healthcare facilities contain computer interpretable data accessible at the point of care within the EHR indicating the profession of EHR authors. But without a mechanism for easily retrieving documents by the professional attributes of EHR authors valid at the time of documentation, clinicians may waste critical time searching for documents relevant to the clinical situation

An alternative approach to delivering an EHR author’s professional characteristics to the general EHR user has been to embed that information within the document title. This approach has the advantage of conveying information that is valid at the point of documentation. The clinical note title is prominently displayed to all clinical providers, and is represented as structured data within most EHRs. Elements of the clinical note titles proposed in 2001 by Brown et al as well as the design of Logical Observation Identifiers Names and Codes (LOINC) document titles are able to convey very broad profession information about clinical note authors (7), (8).

Under the LOINC system, each note title is composed of at least 1 of the 5 distinct information types: Subject Matter Domain, Role, Setting, Service, and Document Type, where one element (or axis) must be Document Type. The information organization within the clinical title presents opportunities for parsing algorithms to interpret the profession, specialty and organizational level of an author. While these structured frameworks for document title naming are in wide EHR use, there have been a number of studies showing that the majority of document titles using the LOINC system are poorly specified, sometimes with only one of the 5 recommended elements included. (9). VA standardization EHR document titles to the LOINC system was deployed nationally in 2007. This included all locally defined document titles being mapped to a LOINC compliant national standard document title set, but both local and national document titles remained under-specified for many information types. It has been shown in a variety of electronic health records systems using LOINC compliant titles that under-specification is a significant problem (10). This has not yet been systematically evaluated in the VA.

The inaccessibility and absence of time-sensitive validation in privileged databases and the imprecision of LOINC note titles in determining EHR authors’ professional characteristics indicate that additional data sources are needed for the task. One such source is the author’s signature line text generated at the moment of electronically signing the EHR. Presently the VA uses free text electronic author signature lines with informal recommendations for including professional credentials in the signature line. This information is inserted into the EHR document upon execution of the electronic signature. Thus, like EHR document titles, the information is generated at the point of documentation. EHR document titles differ from electronic signature-line text blocks, however, in two respects: their informational function and their data structure. EHR document titles, both LOINC compliant and locally generated, are intended to communicate a variety of information types concerning the content of the note. In contrast, the electronic signature line has a singular purpose: to bear the name and professional credential of the document’s signer. Structurally they differ in that electronic signature lines are author selected free text fields while EHR document titles are semi-structured data whose content is administratively controlled. Among popular home grown and commercial EHRs, electronic signatures are widely used. The Veterans Administration’s EHR, CPRS, Star Panel which is used with the Vanderbilt Medical Center and Epic are all examples of EHR systems that make use of electronic signatures and associated signature line text blocks.

The HL7 EHR functional model would, if populated, represent an idealized EHR. The implemented idealized EHR presupposes a codified meta-data element detailing the professional credentials of its author. (11) We investigate and report here not the ideal EHR, but the existent EHR. In an effort to evaluate the data elements that come closest to providing professional attributes of EHR authors, we assessed the ability to computationally determine the professional details of a medical document author through the use of both structured and unstructured data elements present within the VA CPRS system. The primary hypothesis was that a significant improvement in professional classification could be achieved by combining the structured LOINC EHR document title information with the unstructured and non-standardized electronic signature line field of information.

3. Results

We present the results of the quantitative analysis first, followed by the results of the qualitative analysis. Overall, the automated algorithms achieved a final sensitivity and specificity of 87.7% and 99.4% on the training set for document titles only and 99.7% and 99.3% for the signature line –note title aggregate, respectively. The testing set sensitivity was 76.1% and the specificity was 69.4% for the document titles only and 95.7% and 98.5% for the signature line – note titles aggregate, respectively. The inter-rater agreement across the 12 attested profession classes over the entire data set of 1000 documents was 92.7% and the unweighted Kappa Co-efficient Score was 89.9%, with a 95% confidence interval of (87.8% — 92.1%). The most common types of professions in the testing sample of notes were nursing (235), physician (159), and respiratory therapy and rehabilitation services (49). There were no chiropractic, emergency medical service, or podiatry service notes in this sample. A comparison of the performances of note titles alone vs. the aggregate of note titles with signature lines by profession type is shown in Table III.

Among specific professions, the detection of Physicians, Nurses, Physician Assistants & Nurse Practitioners, and Technologists were noted to be statistically significantly improved with the inclusion of the electronic signatures, as can be observed from the non-overlapping confidence intervals of the sensitivity scores shown in Table 3.

The qualitative analysis concerns the informational content of the document titles. From among the 1000 randomly selected documents of our study, the contributions to informativity that each of the five elements (or axes) of the National Standardized (LOINC compliant) EHR document title make can be observed. Table IV displays the document count for each possible title axes combination among the 1000 documents of our study cohort.

Among the well-formed National Standardized LOINC Titles in our study sample (964 documents), we found the following rates of axes occurrence: 1) subject Matter axis was present in 56% of the titles, 2) role axis was present in 47% of the titles, 3) setting axis was present in 29% of the titles, 4) service axis was present in 23% of the titles, 5) document type axis was present in 100 % of the titles. The most frequently occurring combination of axes was titles with the 2 axes Subject Matter and Document Type represented. There were no titles in which all 5 axes were present. The least frequently occurring axes combination was titles with the 4 axis Subject Matter, Role, Service and Document Type.

In our qualitative analysis of LOINC document title axes we also found that indications of professional category are distributed across all five axes; as are training level and specialty. Some axis-to-profession mappings are unique, while others are ambiguous (i.e., an axis text that maps to more than one profession). We point out the ambiguous mappings because an automated retrieval of the profession of EHR authors that uses LOINC document titles would achieve low specificity in these cases.

Table V provides some examples of each LOINC document title axis in isolation and the professional categories to which it maps, along with the evidence type that justifies the mapping. Some axes indications are direct and textually represented, (Textually Present in Table V). The information in these axes would provide unambiguous mappings to author profession. Other axes texts require inferences based on knowledge of healthcare directives, (Point of Care Directive Inference in Table V) such as knowing that a note whose service is designated as “Evaluation and Management” is likely to have been authored by a physician of some type or a nurse practitioner or a physician assistant. Other types of inferencing rules draw from a professional classification hierarchy (Professional Classification Inference in Table V).

In contrast to the gaps in direct evidence in LOINC EHR document titles seen in the above qualitative analysis, the electronic signature block provides direct textually represented information which unambiguously identifies the profession of the signer. The only inference required is that the signer of the EHR document and the author of the EHR document are the same. The electronic signature in CPRS and in most electronic documentation systems is set up to contractually ensure that the author and the person making use of the electronic signature are the same.(16) There were however 0.4% of the study documents in which the electronic signature represented a person other than the author of the document.

Among the 1000 documents in our study, 97.6 % of the study documents had textually present profession information in the electronic signature line. 2.4% of the study documents had signature lines from which no professional information was extracted. Of the twenty-four documents occurring without profession information in the electronic signature line, most were instances in which the signer provided no information apart from the name. Four cases were instances in which a substitute signer was authorized to sign the document. Substitution contexts presented a challenge to the text retrieval process in that the situations governing such instances do not render uniform textual patterns.

4. Discussion

We found that the use of electronic signature lines in addition to document titles significantly improved professional classification detection, and the performance of the algorithms composed of both data types were quite accurate. In addition, review of the national standardized (LOINC compliant) EHR title information revealed frequent under-specification.

The codified EHR meta-data was an important component to the accuracy of the rule algorithms, particularly when only the EHR title information was used. With respect to accessing the author’s profession, the LOINC document title axes combination of Subject Matter and Role or Subject Matter and Service could provide the profession and training level, and for some combinations, the specialty as well. However, the procedures governing which personnel may use which combinations of LOINC document title axis values are highly unconstrained. The result is that many document titles are underspecified, and an uncertain correlation exists between the actual profession, specialty and training level of the author with the terms appearing in the LOINC document title axes. Even given the most fully specified document titles possible, i.e., all five axes in use and appropriately designated, there is no one strategy employable to discern all of the professions of the potential EHR authors from such ideally specified LOINC documents titles, and even combining various LOINC title axes still yield ambiguous mappings to the profession.

The ROLE axis of the LOINC EHR title is potentially the most pertinent to identifying the profession of the author. However the function of the ROLE axis is not precisely to identify the EHR author’s profession. The ROLE axis is a hybrid indicator of profession, training level, and clinical activity. Based on reports and manuscripts from the Document Ontology Task Force and LOINC Committee, it is clear that the ROLE axis was not intended as a static indicator of the EHR author’s profession. Instead, it was meant to function dynamically, pointing either to the author’s level within the healthcare institution (training or hierarchical), or to the performance of one of a set of particular healthcare activities. The training/hierarchical categories cut across professions and specialties. Additionally, there are situations in which a provider’s medical role is distinct from his or her actual profession. The ROLE axis of the LOINC EHR title is meant to document situations, for example, when a cardiology resident is on duty in the emergency room and he or she delivers physician services. These and other scenarios where the EHR author’s role is explicitly distinct from his or her professional designation are precisely the kinds of cases that demonstrate that that the functional design of EHR document titles cannot and should not be expected to deliver the provider’s professional profile.

Although there is interplay between the LOINC ROLE axis and the profession of the EHR author, our efforts to use the LOINC ROLE axis as a predictor for profession indicate that they are not inter-substitutable. Of the 1000 study documents 53% do not contain the ROLE axis. Further, relying on ROLE would increase the false negative rate of the profession detection algorithms where the profession-to-ROLE axis correspondence is either ambiguous or nonexistent. Specifically, Dental Service Providers, Eye and Vision Services Providers are not represented in the ROLE axis, and there is no specific ROLE axis term dedicated to Behavioral Health and Social Service Providers, or Dietary and Nutritional Service Providers, or Respiratory, Rehabilitative and Restorative Service Providers, or Speech, Language and Hearing Service Providers. In addition, the governance of EHR title usage does not constrain title construction vis-à-vis the profession of the document author. Such cases tended to increase the false positive rate of the profession detection algorithms.

We have distinguished performances of the EHR titles algorithms from that of the aggregate algorithms. We can also distinguish between error types somewhat by separating the performance errors brought about by aberrant EHR title usage from those introduced by the terminology in the EHR titles themselves. Errors of some kinds were introduced by the behavior of healthcare workers in selecting the EHR titles that are designed to be used by some other professional category.

The electronic signature line is a free text field in the EHR with no special meta-data indexing. While offering rich detailed profession information, the electronic signature line comes with enormous variety in both form and informational specificity. Further, there were a small number of records whose electronic signature lines yielded no profession information. Through a normalization process, however the electronic signature lines containing profession information offer a means of supplying data that the EHR document title often fails to provide. Unlike EHR note titles, however, retrieving the text of electronic signature lines presently requires parsing the EHR text because this portion of the document is not coded or indexed as a meta-data item. The differences in the performance of EHR note titles as against that of electronic signature lines in determining the profession of the EHR author indicate that EHR titles are an imperfect means for retrieving this information, and that a document field like the electronic signature line offers a significantly better choice for automating profession determination.

There were a number of limitations in this study. First, it was carried out in the context of Electronic Health Records in use in the VA. In addition, no systematic investigation was made for the local institutional variations in usage and construction of the LOINC National Standardized Titles, and this type of variation may decrease performance of the rule algorithms when applied to a VA national dataset. Another limitation of the study is the profession taxonomy. We chose the National Uniform Claims Committee for our profession taxonomy because we knew it to be in wide use and because we found no industry-wide taxonomy of health workers. The NUCC taxonomy was drawn up for the purposes of billing, as opposed to distinguishing healthcare personnel according to function. A taxonomy that is oriented toward healthcare functions may provide a more meaningful evaluation of the capacity to predict EHR author professions.

5. Conclusion

In this study of clinical documentation within the Veterans Health Administration, we were able to accurately detect professional classification of authors using both local and national (LOINC compliant) EHR document titles augmented by self-determined electronic signature line information. There was significant information gain through the use of electronic signature lines, indicating that EHR document titles are under-specified for the purposes of professional classification determination. We have suggested some uses that professional attribution as an EHR data-point could have within the Veteran’s Health Administration, such as speeding relevant document identification during a clinic visit or for quality measurement and improvement activities. We advocate here that consideration be given to the inclusion of professional attributes as a discrete codified meta-data item in future VA Electronic Health Records.

Acknowledgments

Footnotes

References