Applying an Organizational Framework for Health Information Technology to Alerts
Abstract
We are far from understanding how best to design, implement, and use health information technology (IT). A comprehensive framework, developed by Rippen et al1 to capture and organize knowledge on the implementation, use, and optimization of health IT, may guide and inform more effective health IT deployment. This study applied Rippen’s framework to a focused type of health IT – alerts – through clinical decision support (CDS), an area with a substantial evidence base around many facets of implementation, including the technology, use, and outcomes. We report results from applying this framework for capturing, organizing and standardizing knowledge and related measures around alerts. It is clear there are gaps in information shared and that measures across studies vary significantly. Insights identified using the framework highlight areas for further study and development, directed toward a shared conceptualization and representation of knowledge, and ultimately, a more comprehensive and deeper understanding of health IT.
Introduction
Knowledge capture in any field is improved by measurement based on standard concepts, terminologies, and taxonomies. This applies to health IT, which is complex and poorly understood.2–4 Kaushal conducted a systematic review of health IT and concluded that uniform standards for the reporting of research on health IT implementation are a high priority.5 The authors of this study strongly concur, and in this spirit, have leveraged the Organizational Framework developed by Rippen et al1, to continue moving the goal of uniform standards for reporting health IT research.
A framework can serve as a basis for organizing and reporting research on implementation of health IT, and reduce heterogeneity in reporting by ensuring that all important ‘facets’ (a term the framework developers used to capture important aspects of health IT) and their key components (e.g., characteristics and dimensions) are identified using common terminology, concepts and measures. The authors applied Rippen’s organizational framework to studies relating to a specific type of CDS, alerts, to assess what was being described within the literature relating to facets, components and measures. The purpose of this analysis is to pilot the use of the organizational framework by: (1) apply the framework to support the specific health IT application, alerts; (2) identify gaps in the literature around a specific component; and (3) explore the measures used to describe the facets relevant to the application.
Background
The organizational framework developed by Rippen et al. is based on existing health IT and related theories and models including technology diffusion, change management, and sociotechnical theories.1 The framework recognizes that health IT actions such as alerts are “sociotechnical” interactions between the IT and the user—including the organization’s existing social and technical systems, such as their workflows, culture, and social interactions.6–8 By cross-walking elements of current theories and models, Rippen identified five major, interrelated facets of an organizational framework that provide a structure to organize and capture information on the implementation and use of health IT. These are: (1) Technology—elements relevant to the specific health IT itself; (2) Use—elements related to the actual use of the technology; (3) Environment—elements related to the context in which the technology is used; (4) Outcomes—elements capturing the end results of the technology in use in its environment; and (5) Temporality—time and the developmental trajectory of other elements such as implementation and clinical disease processes.
Alerts are CDS tools that notify health professionals and patients with general and person-specific information, intelligently filtered and organized, at appropriate times of decision-making, to enhance health and health care. Drug-allergy checking and alerting is one of the simplest yet most important CDS tools used in electronic order entry systems.9 Alerts supporting the delivery of preventive services are another CDS application that enables action.10 As one of the most recognizable and common types of CDS, it is important to more fully understand this health IT application and assess the state of knowledge for all of its facets.
Methods
The authors selected a convenience sample of 17 published studies on alerts from peer reviewed journals in order to apply the Rippen organizational framework to the alert study description and findings.2,3,5,9–22 These 17 papers include original research and review papers (systematic review and meta-analyses) for the most common types of alerts: (1) basic medication alerts such as drug-drug interaction (DDI), drug-allergy interaction, and drug duplication alerts; (2) basic medication order guidance such as advanced drug alerts, drug-laboratory, drug-condition, drug-disease, drug-age and appropriate prescribing alerts; (3) drug formulary and dosing alerts; (4) condition-specific dosing guidelines (e.g., renal function); and (5) preventive services alerts or reminders, in both inpatient and outpatient settings. All of these studies were based on health IT applications such as order entry systems that used active alerts.
Each paper was reviewed by one member of a team of four health IT researchers, and important study descriptions and findings were mapped to the appropriate framework facets and associated characteristics from the original Rippen paper (for example, type of outcome or feature of technology). Themes for measures were identified, and examples of measures used in the study were identified. One paper was initially selected by the authors for internal discussion to ensure agreement on definitions and application of the framework. Facets, characteristics, themes, and examples were identified, captured, and conceptualized, and standardized terminology was assigned as needed.
Results
The results of the review, summarized by facet in Tables 1–5, lists the facet categories, related characteristics, themes, and the measures used to describe them in the literature. The technology facet, highlighted in Table 1, has six categories associated with it: cost, data and interoperability, functionality, non-functional requirements, product and user-based design. For alerts, data is a critical component as it has a direct effect on the validity of the alerts to the clinician, and hence the outcome of the patient. Not surprisingly, there are many types and sources for the data such as knowledge database derived data, patient specific data, system generated data (e.g., alert, reminder, order set) and clinician entered data (e.g., the order, override). The attributes of the data such as currency and validity are especially important but not often discussed. Also, data from other systems are also important but often not available (e.g., laboratory or diagnosis data). In general, details around the data itself were often not included in these papers. Not mentioned in any of the studies were the non-functional requirements (shaded in grey).
Table 1.
Technology facet categories and measures for alerts.
| Category | Characteristics | Example Themes | Measures Used |
|---|---|---|---|
| Cost | Cost relating to and specific to the technology | Development cost (when built in-house); implementation cost; hardware and software costs, operations and maintenance costs | Dollars2; Dollars/year2; Dollars/patient/year2 |
| Data and interoperability | Attributes related to the knowledge databases | Knowledge databases, type, fields, and source name | Type: Medication database, Source: pharmacy database listing stock; national medication database22 Type: Allergy tables; medication ingredients11,16 Type: Contraindications Database, Source: internally developed11,16 Type: Drug-Disease database11,16 |
| Attributes around validation, integrity, quality, currency | Quality Comprehensiveness Currency: Knowledge database Update frequency Data standard Magnitude of change | Vague; unhelpful16 Incomplete labs (e.g., pregnancy test results)22 Updates monthly22 ICD 10 code16 Changes in alerts secondary to changes in drug dictionary over time11 | |
| Attributes related to patient specific data including that which is entered by the user (prescription) | Clinician entered data: medication orders, allergy data, diagnosis data Registration data: sex, age Laboratory data: test/procedure result; drug level Pharmacy system | Data type Availability of data22 | |
| Functionality | Describes the functionality and design purpose of the technical application | Application type: Computerized Physician Order System (CPOE) Clinical Decision Support System (CDSS) Knowledge database | Application type/definition3,11 |
| Alert level | High, Medium, Low22 Seriousness Index, F,E,D,C,B,A22 Level 1, Level 2, Level 320 | ||
| Alert target/reach | All/None (hospital wide)22 Physician office | ||
| Alert recipient | Clinician Patient10 | ||
| Alert trigger | Drug-drug interaction Overdose Duplicate orders Drug allergy Drug contraindications (lab, disease, pregnancy) Reminders (corollary orders) Formulary2,16,22 Age, sex, diagnosis specific Preventive Services14 | ||
| Trigger event | Per order11 Per patient14 | ||
| Rule threshold | True allergy Possible allergy Medication intolerance11 | ||
| Alert delivery | Printed Electronic Phone Letter Postcard10,14 | ||
| Alert action requirements | Eliminate contraindication; override reason; read and confirm; none 20 Stop current order, override, respond, cancel new order22 Approval only (yes/no) Approval3 Reminder21 | ||
| Physician order entry | Order set Free text prescription22 | ||
| Required data entry fields (CPOE) | Drug name, dosage form, strength, drug dose, frequency, start date, start time11,20,22 | ||
| Alert trending | Availability, # alerts/month11 | ||
| Non-functional requirements | Indicates how well the system performs | ||
| Product | Describes the specific technology product (i.e., hardware, software) | Product name Version number | Name and version2,11,20,22 |
| Vendor | Name of Company | ||
| Location | City, State, Country2,11,20,22 | ||
| User-based design | Includes user interface design but also the workflow that the health IT was designed to support | Alert response | Intrusive vs non-intrusive11,20,22 |
| Prompt | Unambiguous action Promotes an action3 Yes/No10 Patient specific or generalized10 | ||
| Timing of prompt | Yearly 1 month in advance Night before clinic visit Before visit at reception desk During routine intake procedures10 | ||
| Communication content feature: Justification | Reasoning Research evidence Citation of authority Institution-specific guidelines3,16 | ||
| Prioritization | Clinically significant warnings differentiated from other warnings16 | ||
| User-based design | Local user involvement – yes/no 3 |
Table 5:
Temporality facet categories and measures relating to alerts
| Category | Characteristics | Example Themes | Measures Used |
|---|---|---|---|
| Time | Independent variable; external anchor; construct for understanding duration | Start of study; End of study; Start of data collection; End of data collection; Electronic availability of data; Duration of study | Date as Month, Day, Year14,20 Date as Month, Year22 Date as Year14 Duration as months22 |
| Implementation cycle | Characterizes a step in a lifecycle of health IT implementation; not dependent on time; supports comparison across different implementations independent of time | Enhancements | Period of enhancement to CDS11 |
| Outcome lifecycle | Characterizes the lifecycle of when an intervention can be expected to generate a given outcome; provides the ability to account for findings when a study has not provided for sufficient time for evaluation | Frequency of alerts; Clinician reaction to alerts | Change in frequency of alerts over time11 Change in clinician compliance to alerts over time11 |
Table 2 presents the use facet organized by the associated categories of user characteristics and attitudes, knowledge, ownership/buy-in, and usability and workflow. The category user attitudes in the Organizational Framework1 was expanded to include user characteristics and attitudes to clearly identify the concept of user, which was described in each study and is an important concept when addressing use. The organizational framework category of knowledge was not a characteristic mentioned in any of the studies reviewed (shaded in grey).
Table 2:
Use facet categories and measures relating to alerts.
| Category | Characteristics | Example Themes | Measures Used |
|---|---|---|---|
| User characteristics and attitudes | Cover a wide range of concepts such as user characteristics, satisfaction, perceived usefulness and usability, and user acceptance | End users characteristics | Prescribers, pharmacist, age, specialty, percent faculty10,14,19,21 |
| Studies evaluated questions such as: Value of DDI information context is perceived to be helpful | Likert scale of agreement (1–5) Percent positive in the Likert scale of statement agreement.15 | ||
| DDI alerts improved their ability to prescribe safely | Likert scale of agreement (1–5) Percent positive in the Likert scale of statement agreement.13 | ||
| Satisfaction with CPOE | Likert scale of agreement related to satisfaction15 | ||
| Provider belief about medication | Likert scale of agreement16 | ||
| Clinicians reaction to DDI alerts | Relevance of DDI alerts as helpful (surveyed at the time of warning)16 | ||
| Usability & workflow | Covers usability and actual workflow of the user | How the alert system is integrated into the prescribers workflow and at what point influence the success of CPOE alerts. | Interventions assessed as succeeded when the CDSS was provide to clinicians automatically5,22 |
| Other reported barriers (e.g. poor visual presentation) | Poor signal to noise ratio13 | ||
| Alert text readability | Not lengthy, clear and concise to be helpful with links to supporting evidence22 | ||
| Automatic provision of decision support as part of the clinician workflow and no need for additional clinician data entry | |||
| Ownership/Buy-in | Captures the amount of user involvement and participation in the implementation process | Advise having clinician-users review the alert text prior to introducing them into practice | Degree of involvement; ability to turning alerts on or off22 |
| Knowledge | Includes concepts around adult learning, training, capability |
The environmental factors have been shown to be important to successful implementation of health IT systems.23 In the papers reviewed, attributes relating to this facet were rarely identified and discussed with the exception of setting, as is highlighted in Table 3. The papers were also scant on measures related to leadership (shaded in gray).
Table 3:
Environment facet categories and measures relating to alerts.
| Category | Characteristics | Example Themes | Measures Used |
|---|---|---|---|
| Business drivers | This includes governmental policies and regulations that influence the organization and business factors, e.g., competition | Patients | Number of patients |
| Cultural/organizational | Captures teamwork climate, values, culture | Cultural views regarding data retention Physician acceptance of guidance | Hesitance to remove allergy data from an EMR even if dubious16 Physician acceptance10 |
| Leadership | Senior leaders and champions fit into this category | ||
| Resources | This includes not only the resources available to support the implementation of the health IT, but also the IT infrastructure that can enable it | Availability | Number of terminals in given location14 |
| Setting | Which environment the health IT is being used | Facility type; affiliation | Inpatient (# of beds); outpatient (# physicians, staff)11,14,20,22 University affiliated, public clinic10 |
| Name and location of facility | Name; city, state, country11,14,20,22 | ||
| Staffing mix | Physicians (specialties), residents, physician assistants, nurses, medical assistants, receptionists14,20 | ||
| Support | Many aspects of implementation management fit under this category, including training | Customization | Ability to support local customization16 |
Table 4 summarizes the information around outcomes and organized by the categories adoption, business/financial, clinical, and methodology. There were a large number of themes and measures around adoption and use. This variation makes it more difficult to compare studies. Clinical outcomes were usually not associated with adoption and user characteristic measures such as user acceptance and level of alert use. Many adoption studies focused on clinician acceptance and compliance with alerts as a function of number, type of alerts, interface, and ability to control alerts, and the most comprehensive studies provided information around the technology and use facet categories to aid in interpreting outcomes.
Table 4:
Outcome facet categories and measures relating to alerts.
| Category | Characteristics | Example Themes | Measures Used* |
|---|---|---|---|
| Adoption | The number of alert users, the rate of use, depth of their use | Compliance with alerts (acceptance of alert recommendation) Override of alerts (non acceptance of alert recommendation) Number of unique alerts and total alerts fired Alert fatigue Turning on/off alerts Documentation of exceptions to alerts which impacts number of patients eligible for performance measure | Number of alerts accepted over total alerts fired22 Number of patients eligible for performance measures with alert functional allowing documentation of exceptions to alert rule18 Acceptance of alerts and overrides20,22 Clinician decision to turn off/on alert based on information available with alerts22 Number of unique alerts, and overridden22 |
| Drug prescribing according to alert rules | Drug prescribing according to alert rules Ordering corollary medications | Proportion of all physicians with perfect or near perfect (90%–99%) for 7 drug prescribing guidelines in alerts18 25% improvement in ordering of corollary medications by faculty and residents2 | |
| Attributes related to alert type, severity, and tiring | Basic or Advanced Alerts: Classification system adapted to derive three main categories of alerts and prompts: 1) basic alerts consisting of drug allergy, drug-drug interaction and duplication warnings, and basic medication order guidance; 2) advanced alerts including drug-lab, drug-condition, formulary alerts; and 3) complex alert systems with basic and advanced alerts. | Alert acceptance by level and severity of alert17 Differences in alert acceptance by tiering17 Number of DDI alerts fired, accepted, rejected by type of alert10,21,22 | |
| Assessment of alert severity by clinicians | Clinician drug severity versus that from drug databases | Agreement between clinicians and drug database on alert severity22 | |
| Ability to tailor alerts | Tailor-ability according to specific user | Number of frequently overridden alerts turned off, suppressed alerts22 | |
| Business/Financial | Healthcare utilization and costs | Reduced utilization and associated costs Reduced costs of care Formulary compliance | Length of stay3 Laboratory test orders3 Cost per hospitalization2 Medication costs9 |
| Clinical | Prescribing behavior | Adverse drug events Prescribing errors Duplicate orders Other adverse outcomes related to prescribing | Prescribing errors22 -related renal impairment9 Number and type of serious and life-threatening adverse drug events9 Number of duplicate orders16,20,22 Falls in elderly9 Hospital length of stay9 |
| Quality of care according to guidelines | Acceptance of CDS alerts for chronic and preventive care according to guidelines | Quality measure performance related to CDS alerts, for selected chronic care and preventive quality measures (e.g., lipid lowering drugs; ACE Inhibitor or ARB; ACE Inhibitor or ARB in LVSD; beta blocker in LVSD; anticoagulant for atrial fib; LDL-C control is persons with diabetes; aspirin for primary prevention - diabetes; neuropathy mgmt. diabetes3,12,14,18 No change in mammography screening 14 | |
| Methodology | Study Design | Study type (e.g., pre-post, observational field study, case–control, prospective, retrospective, time series)
| Number of patients, alerts fired, alerts accepted, alert overrides17,22 |
| Study Participants | Health IT with alert users Type of user | Number of active users of the CPOE system, Specialty types - internal medicine, cardiology (specialists), registered hospital pharmacists. Internal med & cardiology : 4 specialists, 2 residents; 4 registered pharmacists and 2 residents in pharmacy; surgery - 2 specialists, 4 final year residents) - 576 assessments22 Clinicians, nurses, pharmacies | |
| Limitations | Ability to control for alert intervention variable Study population | An intervention was multifaceted, and study unable to determine which components were most responsible for the observed results18 Results only generalizable to experienced EHR users18 |
Table 5 presents the temporality facet and associated subcategories of time, implementation cycle, and outcome lifecycle as extracted from the seventeen studies reviewed using Rippen’s Organizational framework for health information technology.1 The measure around study duration or time was at the month/year or year level. However, this information was often missing in the papers reviewed.
In the temporal facet, measures used include objective time (e.g. dates and durations), subjective time (e.g. enhancement period), mapping of subjective time to objective time (e.g. the enhancement period started on <date>) and change of measures from other facets over time. Not surprisingly, there’s good agreement in the use of standard time units in describing objective time, with the only variation being in the precision or granularity of the time units. Subjective time is not used significantly in the sample studies. This may be a reflection of the relative “youth” of the CDS subject domain, yet to develop a common accepted terminology for implementation phases. The immaturity of the CDS domain is also reflected in the lack of outcome lifecycle measures, largely due to the scarcity of longitudinal studies that would associate and define outcome over time.
Discussion
The Rippen framework facilitated identifying and categorizing research findings and context about alerts. Although all of the facets were touched upon, technology, use and outcomes were most often addressed. The framework facets not generally addressed in the studies were environmental and temporality. When reviewing each facet it is clear that the components addressed across facets varied as did the measures used to describe or quantify them.
This analysis accomplished its threefold purpose:
- Application of the organizational framework to alerts. Rippen’s organizational framework provided a robust framework to capture the concepts, themes, and measures presented in the papers reviewed. No key characteristics and measures were identified that were not addressed by the framework. For clarity, the category ‘user attitude’ in the framework was changed to ‘user characteristics and attitude’ for this paper.
- Identify gaps in the literature around alerts. Health IT implementation is complex in that it is difficult to adequately “control” for context under which an application is implemented. Researchers often focus on the details of the technology that they are studying. Not surprisingly, the most detail in the studies was around the technology facet, specifically the application itself. However, even in this facet there were gaps. Non-functional requirements were not addressed despite the fact that they can have a significant effect on the outcome. For example, a response time of 300 milliseconds vs. 15 seconds for the time between when an order is entered and when the alert is seen is important as to whether and how the alert is used. Failure to provide a robust description of the context and technology is a significant barrier to understanding the field. Furthermore, user and environment facets and characteristics such as user acceptance, training, clinical leadership, and presence of quality improvement initiatives in settings also hindered an in-depth understanding of alert adoption and related outcomes.
- Explore the granularities and consistencies of measure definitions for alerts. One of the most compelling findings was the lack of clear measures across the studies. This includes not having measures that account for alert “burden.” For example, a physician with 90% of his population prescribed drug A compared to a physician who will prescribe drug A less than 1% of the time will have a different experience when “nuisance alerts” are fired for drug A.
Focusing on a specific health IT application – CDS alerts, and including more than one type of application of alerts (medication related and prevention) provided additional insights. The concept of user acceptance of the alert recommendation in conjunction with workflow (easy action) is highlighted in the comparison of a medication-related versus a preventive service order recommendation and likelihood of use.
Without a common framework to provide context, meaning and measures to describe them, our ability to build on different studies will be limited. If the same technology is implemented in similar settings, how will it be known why one fails and another succeeds? Broader adoption of the framework would encourage more standard, comparable, and comprehensive collection and reporting of important implementation information.
Case studies and more focused literature reviews applying the framework to specific health IT implementation projects will help to more clearly identify the benefits and limitations of the framework, in order to more effectively apply the framework to real life health IT projects and to studies of the implementations.
Conclusion
The framework provides a strong starting point for identifying the important facets and characteristics associated with implementation of alerts. The available studies did not provide detailed information on many important implementation facets and characteristics. As health IT applications become more widespread, future studies, guided by a framework such as the Rippen one, can and should examine more facets in greater depth. Besides the technical facets, which are generally well-reported, other facets and characteristics of the environment, setting, culture, intended users, etc. can be better identified and investigated. Only by examining the individual facets and characteristics in detail will it be possible to fully understand which of them are critical success factors (or critical impediments) and how to optimize those success factors (or avoid impediments).