Applying an Organizational Framework for Health Information Technology to Alerts
Colene M. Byrne, PhD, Eric C. Pan, MD, MSc, [...], and Helga E. Rippen, MD, PhD, MPH, FACPM
Abstract
We are far from understanding how best to design, implement, and use health information technology (IT). A comprehensive framework, developed by Rippen et al1 to capture and organize knowledge on the implementation, use, and optimization of health IT, may guide and inform more effective health IT deployment. This study applied Rippen’s framework to a focused type of health IT – alerts – through clinical decision support (CDS), an area with a substantial evidence base around many facets of implementation, including the technology, use, and outcomes. We report results from applying this framework for capturing, organizing and standardizing knowledge and related measures around alerts. It is clear there are gaps in information shared and that measures across studies vary significantly. Insights identified using the framework highlight areas for further study and development, directed toward a shared conceptualization and representation of knowledge, and ultimately, a more comprehensive and deeper understanding of health IT.
Introduction
Knowledge capture in any field is improved by measurement based on standard concepts, terminologies, and taxonomies. This applies to health IT, which is complex and poorly understood.2–4 Kaushal conducted a systematic review of health IT and concluded that uniform standards for the reporting of research on health IT implementation are a high priority.5 The authors of this study strongly concur, and in this spirit, have leveraged the Organizational Framework developed by Rippen et al1, to continue moving the goal of uniform standards for reporting health IT research.
A framework can serve as a basis for organizing and reporting research on implementation of health IT, and reduce heterogeneity in reporting by ensuring that all important ‘facets’ (a term the framework developers used to capture important aspects of health IT) and their key components (e.g., characteristics and dimensions) are identified using common terminology, concepts and measures. The authors applied Rippen’s organizational framework to studies relating to a specific type of CDS, alerts, to assess what was being described within the literature relating to facets, components and measures. The purpose of this analysis is to pilot the use of the organizational framework by: (1) apply the framework to support the specific health IT application, alerts; (2) identify gaps in the literature around a specific component; and (3) explore the measures used to describe the facets relevant to the application.
Background
The organizational framework developed by Rippen et al. is based on existing health IT and related theories and models including technology diffusion, change management, and sociotechnical theories.1 The framework recognizes that health IT actions such as alerts are “sociotechnical” interactions between the IT and the user—including the organization’s existing social and technical systems, such as their workflows, culture, and social interactions.6–8 By cross-walking elements of current theories and models, Rippen identified five major, interrelated facets of an organizational framework that provide a structure to organize and capture information on the implementation and use of health IT. These are: (1) Technology—elements relevant to the specific health IT itself; (2) Use—elements related to the actual use of the technology; (3) Environment—elements related to the context in which the technology is used; (4) Outcomes—elements capturing the end results of the technology in use in its environment; and (5) Temporality—time and the developmental trajectory of other elements such as implementation and clinical disease processes.
Alerts are CDS tools that notify health professionals and patients with general and person-specific information, intelligently filtered and organized, at appropriate times of decision-making, to enhance health and health care. Drug-allergy checking and alerting is one of the simplest yet most important CDS tools used in electronic order entry systems.9 Alerts supporting the delivery of preventive services are another CDS application that enables action.10 As one of the most recognizable and common types of CDS, it is important to more fully understand this health IT application and assess the state of knowledge for all of its facets.
Methods
The authors selected a convenience sample of 17 published studies on alerts from peer reviewed journals in order to apply the Rippen organizational framework to the alert study description and findings.2,3,5,9–22 These 17 papers include original research and review papers (systematic review and meta-analyses) for the most common types of alerts: (1) basic medication alerts such as drug-drug interaction (DDI), drug-allergy interaction, and drug duplication alerts; (2) basic medication order guidance such as advanced drug alerts, drug-laboratory, drug-condition, drug-disease, drug-age and appropriate prescribing alerts; (3) drug formulary and dosing alerts; (4) condition-specific dosing guidelines (e.g., renal function); and (5) preventive services alerts or reminders, in both inpatient and outpatient settings. All of these studies were based on health IT applications such as order entry systems that used active alerts.
Each paper was reviewed by one member of a team of four health IT researchers, and important study descriptions and findings were mapped to the appropriate framework facets and associated characteristics from the original Rippen paper (for example, type of outcome or feature of technology). Themes for measures were identified, and examples of measures used in the study were identified. One paper was initially selected by the authors for internal discussion to ensure agreement on definitions and application of the framework. Facets, characteristics, themes, and examples were identified, captured, and conceptualized, and standardized terminology was assigned as needed.
Results
The results of the review, summarized by facet in Tables 1–5, lists the facet categories, related characteristics, themes, and the measures used to describe them in the literature. The technology facet, highlighted in Table 1, has six categories associated with it: cost, data and interoperability, functionality, non-functional requirements, product and user-based design. For alerts, data is a critical component as it has a direct effect on the validity of the alerts to the clinician, and hence the outcome of the patient. Not surprisingly, there are many types and sources for the data such as knowledge database derived data, patient specific data, system generated data (e.g., alert, reminder, order set) and clinician entered data (e.g., the order, override). The attributes of the data such as currency and validity are especially important but not often discussed. Also, data from other systems are also important but often not available (e.g., laboratory or diagnosis data). In general, details around the data itself were often not included in these papers. Not mentioned in any of the studies were the non-functional requirements (shaded in grey).
Table 2 presents the use facet organized by the associated categories of user characteristics and attitudes, knowledge, ownership/buy-in, and usability and workflow. The category user attitudes in the Organizational Framework1 was expanded to include user characteristics and attitudes to clearly identify the concept of user, which was described in each study and is an important concept when addressing use. The organizational framework category of knowledge was not a characteristic mentioned in any of the studies reviewed (shaded in grey).
The environmental factors have been shown to be important to successful implementation of health IT systems.23 In the papers reviewed, attributes relating to this facet were rarely identified and discussed with the exception of setting, as is highlighted in Table 3. The papers were also scant on measures related to leadership (shaded in gray).
Table 4 summarizes the information around outcomes and organized by the categories adoption, business/financial, clinical, and methodology. There were a large number of themes and measures around adoption and use. This variation makes it more difficult to compare studies. Clinical outcomes were usually not associated with adoption and user characteristic measures such as user acceptance and level of alert use. Many adoption studies focused on clinician acceptance and compliance with alerts as a function of number, type of alerts, interface, and ability to control alerts, and the most comprehensive studies provided information around the technology and use facet categories to aid in interpreting outcomes.
Table 5 presents the temporality facet and associated subcategories of time, implementation cycle, and outcome lifecycle as extracted from the seventeen studies reviewed using Rippen’s Organizational framework for health information technology.1 The measure around study duration or time was at the month/year or year level. However, this information was often missing in the papers reviewed.
In the temporal facet, measures used include objective time (e.g. dates and durations), subjective time (e.g. enhancement period), mapping of subjective time to objective time (e.g. the enhancement period started on <date>) and change of measures from other facets over time. Not surprisingly, there’s good agreement in the use of standard time units in describing objective time, with the only variation being in the precision or granularity of the time units. Subjective time is not used significantly in the sample studies. This may be a reflection of the relative “youth” of the CDS subject domain, yet to develop a common accepted terminology for implementation phases. The immaturity of the CDS domain is also reflected in the lack of outcome lifecycle measures, largely due to the scarcity of longitudinal studies that would associate and define outcome over time.
Discussion
The Rippen framework facilitated identifying and categorizing research findings and context about alerts. Although all of the facets were touched upon, technology, use and outcomes were most often addressed. The framework facets not generally addressed in the studies were environmental and temporality. When reviewing each facet it is clear that the components addressed across facets varied as did the measures used to describe or quantify them.
This analysis accomplished its threefold purpose:
- Application of the organizational framework to alerts. Rippen’s organizational framework provided a robust framework to capture the concepts, themes, and measures presented in the papers reviewed. No key characteristics and measures were identified that were not addressed by the framework. For clarity, the category ‘user attitude’ in the framework was changed to ‘user characteristics and attitude’ for this paper.
- Identify gaps in the literature around alerts. Health IT implementation is complex in that it is difficult to adequately “control” for context under which an application is implemented. Researchers often focus on the details of the technology that they are studying. Not surprisingly, the most detail in the studies was around the technology facet, specifically the application itself. However, even in this facet there were gaps. Non-functional requirements were not addressed despite the fact that they can have a significant effect on the outcome. For example, a response time of 300 milliseconds vs. 15 seconds for the time between when an order is entered and when the alert is seen is important as to whether and how the alert is used. Failure to provide a robust description of the context and technology is a significant barrier to understanding the field. Furthermore, user and environment facets and characteristics such as user acceptance, training, clinical leadership, and presence of quality improvement initiatives in settings also hindered an in-depth understanding of alert adoption and related outcomes.
- Explore the granularities and consistencies of measure definitions for alerts. One of the most compelling findings was the lack of clear measures across the studies. This includes not having measures that account for alert “burden.” For example, a physician with 90% of his population prescribed drug A compared to a physician who will prescribe drug A less than 1% of the time will have a different experience when “nuisance alerts” are fired for drug A.
Focusing on a specific health IT application – CDS alerts, and including more than one type of application of alerts (medication related and prevention) provided additional insights. The concept of user acceptance of the alert recommendation in conjunction with workflow (easy action) is highlighted in the comparison of a medication-related versus a preventive service order recommendation and likelihood of use.
Without a common framework to provide context, meaning and measures to describe them, our ability to build on different studies will be limited. If the same technology is implemented in similar settings, how will it be known why one fails and another succeeds? Broader adoption of the framework would encourage more standard, comparable, and comprehensive collection and reporting of important implementation information.
Case studies and more focused literature reviews applying the framework to specific health IT implementation projects will help to more clearly identify the benefits and limitations of the framework, in order to more effectively apply the framework to real life health IT projects and to studies of the implementations.
Conclusion
The framework provides a strong starting point for identifying the important facets and characteristics associated with implementation of alerts. The available studies did not provide detailed information on many important implementation facets and characteristics. As health IT applications become more widespread, future studies, guided by a framework such as the Rippen one, can and should examine more facets in greater depth. Besides the technical facets, which are generally well-reported, other facets and characteristics of the environment, setting, culture, intended users, etc. can be better identified and investigated. Only by examining the individual facets and characteristics in detail will it be possible to fully understand which of them are critical success factors (or critical impediments) and how to optimize those success factors (or avoid impediments).