• We are sorry, but NCBI web applications do not support your browser and may not function properly. More information
Logo of jamiaJAMIA - The Journal of the American Medical Informatics AssociationInstructions for authorsCurrent TOC
J Am Med Inform Assoc. 2007 Jul-Aug; 14(4): 415–423.
PMCID: PMC2244906

The Extent and Importance of Unintended Consequences Related to Computerized Provider Order Entry

Joan S. Ash, PhD, MBA, a , * Dean F. Sittig, PhD, a , b Eric G. Poon, MD, MPH, c Kenneth Guappone, MD, a Emily Campbell, RN, MS, a and Richard H. Dykstra, MD, MS a

Abstract

Background

Computerized provider order entry (CPOE) systems can help hospitals improve health care quality, but they can also introduce new problems. The extent to which hospitals experience unintended consequences of CPOE, which include more than errors, has not been quantified in prior research.

Objective

To discover the extent and importance of unintended adverse consequences related to CPOE implementation in U.S. hospitals.

Design, Setting, and Participants

Building on a prior qualitative study involving fieldwork at five hospitals, we developed and then administered a telephone survey concerning the extent and importance of CPOE-related unintended adverse consequences to representatives from 176 hospitals in the U.S. that have CPOE.

Measurements

Self report by key informants of the extent and level of importance to the overall function of the hospital of eight types of unintended adverse consequences experienced by sites with inpatient CPOE.

Results

We found that hospitals experienced all eight types of unintended adverse consequences, although respondents identified several they considered more important than others. Those related to new work/more work, workflow, system demands, communication, emotions, and dependence on the technology were ranked as most severe, with at least 72% of respondents ranking them as moderately to very important. Hospital representatives are less sure about shifts in the power structure and CPOE as a new source of errors. There is no relation between kinds of unintended consequences and number of years CPOE has been used. Despite the relatively short length of time most hospitals have had CPOE (median five years), it is highly infused, or embedded, within work practice at most of these sites.

Conclusions

The unintended consequences of CPOE are widespread and important to those knowledgeable about CPOE in hospitals. They can be positive, negative, or both, depending on one’s perspective, and they continue to exist over the duration of use. Aggressive detection and management of adverse unintended consequences is vital for CPOE success.

Introduction

Implementation of computerized provider order entry (CPOE), defined as a computer-based system that allows a physician or other ordering authority to directly enter medical orders, can be an effective mechanism for improving patient safety and practitioner performance. 1–14 There can be unintended consequences as well, however, 15–24 and it is likely that fear of adverse unintended consequences is impeding the diffusion of CPOE throughout hospitals in the U.S. 25–29

The unintended consequences of CPOE are especially intriguing because they can surprise implementers. Adverse unintended consequences unfortunately can become impediments to health care quality. The terms “unintended consequences” and “unanticipated consequences” are not synonymous. The “unintended” implies lack of purposeful action or causation, while the “unanticipated” means an inability to forecast what eventually occurred. Either kind of consequence can be adverse or beneficial. Unanticipated beneficial consequences are actually happy surprises. Unanticipated, unintended adverse consequences capture news headlines and are often what people imagine when they hear the term “unintended consequences.” An important goal of the current work is to identify the types of unintended consequences, so that they can be monitored and managed: once unintended consequences are predicted or detected, their management can knowingly involve tradeoffs. An example CPOE-related tradeoff is the degree to which the extra time physicians must exert to use CPOE (an undesirable consequence), is offset by the increased information physicians find available at the point of care via CPOE—making patient visits more effective. 30 The ability to maintain control over consequences may give hospital decision makers more confidence when making the determination to implement CPOE.

Recent descriptions in the literature of the unintended consequences of patient care information systems include a taxonomy of kinds of errors, 31 a list of kinds of errors related to CPOE, 18 a hierarchical model of unintended consequences of CPOE, 30 and a more general summary of the unintended consequences of patient care information systems. 19 In addition, the authors of the current study previously described in detail a typology of unintended adverse consequences of CPOE. 32 However, none of the previous studies provided a comprehensive, semi-quantitative description of the extent and importance of unintended consequences in hospitals across the nation. We therefore transformed our descriptions of types of unintended consequences into survey questions to discover how widespread and important the consequences are. We tested and, using short telephone interviews, administered the resulting instrument to representatives of 561 U.S. hospitals that have reported that they have installed CPOE systems, out of a total of 5,759 accredited hospitals. 33

Preliminary Work

The current study protocol for this three year-project, as proposed to the granting agency and approved by appropriate human subjects committees, included an extensive qualitative study to be followed by a national survey. In the initial qualitative study, we used semi-structured interviews and observations at five sites with experience in using CPOE to find out what types of unintended consequences exist. 32 Study sites for qualitative data gathering included: Wishard Memorial Hospital, a county hospital in Indianapolis which uses a system developed by the Regenstrief Institute; Brigham and Women’s Hospital in Boston, which developed its own system; Massachusetts General Hospital in Boston, which uses a modified version of that same system; The Faulkner Hospital in Jamaica Plain, MA, which uses a commercially available system; and Alamance Regional Medical Center in Burlington, NC, which also uses a commercial product. Institutional review boards at Oregon Health & Science University, Kaiser Permanente Northwest, and each of the individual study sites approved the qualitative study.

An experienced interviewer (JSA) conducted all of the taped interviews following an interview guide of questions. Thirty-two individuals, representing clinical, administrative, and technology roles, were formally interviewed. A multidisciplinary team of Ph.D.-level informatics researchers, physicians, a nurse, and a pharmacist conducted 390 hours of observation with a purposive sample of 142 individuals representing different clinical disciplines. The team performed 120 person-hours of observation at Wishard, 144 at the three Boston sites, and 126 at Alamance. For each visit, five to six researchers spent four days at each site in the period between August 2004 and April 2005. Observations took place in medicine, surgery, emergency, and intensive care areas in addition to the pharmacy, outpatient offices, and clinics.

Qualitative data analysis followed accepted principles for assuring trustworthy results. 34–38 All field notes and transcripts were entered into QSR N6, a qualitative analysis software program, and were individually coded by team members. The multidisciplinary team met 36 times to review transcripts and agree that what was seen or discussed was an unintended consequence and of what type.

Results of this preliminary study included identification of nine types of unintended consequences. Each of these categories can include desirable as well as undesirable consequences, depending on one’s point of view. However, our focus has been on those that are undesirable (adverse consequences), because institutions must manage them. The nine types are:

  • “More/New Work Issues: Physicians find that CPOE adds to their workload by forcing them to enter required information, respond to alerts, deal with multiple passwords, and expend extra time.
  • Workflow Issues: Many unintended consequences result from mismatches between the clinical information system (CIS) and workflow and include workflow process issues, workflow and policy/procedure issues, workflow and human computer interaction issues, workflow and clinical personnel issues, and workflow and situation awareness issues.
  • Never Ending Demands: Because CPOE requires hardware technically advanced enough to support the clinical software, there is a continuous need for new hardware, more space in which to put this hardware, and more space on the screen to display information. In addition, maintenance of the knowledge base for decision support and training demands are ongoing.
  • Paper Persistence: It has long been hoped that CIS will reduce the amount of paper used to communicate and store information, but we found that this is not necessarily the case since it is useful as a temporary display interface.
  • Communication Issues: The CIS changes communication patterns among care providers and departments, creating an “illusion of communication,” meaning that people think that just because the information went into the computer the right person will see it and act on it appropriately.
  • Emotions: These systems cause intense emotions in users. Unfortunately, many of these emotions are negative and often result in reduced efficacy of system use, at least in the beginning.
  • New Kinds of Errors: CPOE tends to generate new kinds of errors such as juxtaposition errors, in which clinicians click on the adjacent patient name or medication from a list and inadvertently enter the wrong order.
  • Changes in the Power Structure: The presence of a system that enforces specific clinical practices through mandatory data entry fields changes the power structure of organizations. Often the power or autonomy of physicians is reduced, while the power of the nursing staff, information technology specialists, and administration is increased.
  • Overdependence on Technology: As hospitals become more dependent on these systems, system failures can wreak havoc when paper backup systems are not readily available.”39

Methods

Survey Development

We developed an interview script, which included a short description of the purpose of the survey, five questions about the kind of CPOE system in place, and eight questions about possible unintended consequences. The five questions about the system were designed to collect descriptive information about the degree to which the system is infused into the institution. Infusion of technology is defined as the extent to which one uses an innovation in a complete and sophisticated manner, 40–41 and it occurs as information technology applications become more deeply embedded within the organization’s work systems. 42

Though we had previously discovered nine types of unintended adverse consequences, we decided not to ask interviewees about one of them, paper persistence, because as the role of paper has changed from a long-term storage medium to a temporary memory aid and disposable display device, its persistence should not now be considered adverse. The unintended consequences typology developed from the qualitative study was therefore transformed into eight survey questions, which were pilot tested (see [triangle]). The questions were designed to be as neutral as possible to avoid bias. We prefaced the questions by stating that the study concerned “what organizations have learned about CPOE.” We asked about surprises, “things that happened that you didn’t expect,” so the focus was on both unintended and unanticipated consequences; responses might then involve either desirable (beneficial) or undesirable (adverse) effects. The questions were asked first as yes/no queries to determine if the respondent thought the unintended consequence existed at that site (0 = No). If the answer was yes, the importance was determined on a 1 to 5-point Likert scale ranging from 1 for “not at all important” to 5 for “very important.” We received institutional review board approval from Oregon Health & Science University and Kaiser Permanente Northwest to conduct the survey.

Table 1
Table 1 Unintended Consequences Survey Questions *

Survey Administration

We attempted to contact the entire population of U.S. hospitals using CPOE, instead of a sub-population from which we would make statistical inferences, because we wanted to gain insight into the nature of unintended consequences related to CPOE from organizations beyond those we visited. We selected acute care hospitals from the 2004 HIMSS Analytics™ Database that reported having CPOE in place (N= 448). Since that database did not include U.S. Veterans Affairs hospitals, which the authors believed to constitute important models of CPOE use, we also surveyed them (N= 113). We contacted hospital staff listed as appropriate contacts via electronic mail and then made phone appointments for the 10 to 20 minute survey. Follow-up included multiple phone calls until a knowledgeable informant was found. Each question asked the interviewee to respond on a 5-point Likert scale about the extent of the problem; researchers took notes on anything that was volunteered beyond that. Numerical data were analyzed with descriptive statistics and the text of comments was analyzed qualitatively, again with the aid of N6. Whenever a respondent spontaneously commented while answering a question, the interviewer wrote down the comment. At the end of the interview, we asked if the interviewee had anything else they would like to share, and these comments were also recorded.

Survey Data Analysis

Quantitative data were analyzed using SPSS software (V14.0, SPSS Inc. Chicago IL). We used descriptive statistics to understand the degree of CPOE infusion in the institutions. We explored the correlation between the length of time CPOE has been in place and the extent to which unintended consequences were present. Using bed size and ownership as established proxies for “other important organizational characteristics in studies of hospital organizational issues,” 43 as well as geographic location and teaching status, we compared respondents to non-respondents using logistic regression.

Qualitative data in the form of notes taken during the interviews were analyzed using QSR N6. This analysis simply entailed grouping comments with the questions to which they related, each representing a type of unintended consequence.

Results

We were able to establish contact with 265 of the 561 hospitals and discovered that 89 of those listed in the HIMSS Analytics™ database did not actually have CPOE, so they were dropped from further consideration. We therefore conducted 176 full interviews. Thirty-four additional institutions had policies against responding to surveys, so they were added to the list of non-respondents. Using the Institute for Social and Economic Research (ISER) calculation for response rate, 44 which takes into consideration the ineligibility of some sites (e.g., listed as having CPOE when they actually do not), our response rate was 47%. To estimate how representative our responding sites were of all hospitals that have CPOE, we categorized the sites in four ways: bed size, management type, teaching hospital status, and geographic location 45 because these measures are considered important differentiators. 43 Logistic regression (forward, stepwise, likelihood ratio) comparing respondents and non-respondents showed no difference with respect to hospital bed-size and teaching hospital status. Management type differed between respondents and non-respondents, in that the VA hospitals were most likely to respond, followed by other governmental hospitals, private institutions and non-profits (compared to VA, governmental LR = 0.63, private not for-profit LR = 0.38 and private for-profit LR = 0.35; p < 0.001). There was also a difference in geographic location with a site in the Northeast region most likely to be a responder, followed by the West, South and the Midwest, in that order (compared to South, Northeast LR = 2.11, West LR= 1.65 and Midwest LR = 0.93; p = 0.007).

Infusion Levels

The length of time that CPOE had been in place ranged from 6 months to 25 years (median = five years). The percentage of orders entered electronically ranged from 1–100% (median = 91%). Greater than 96% of the sites used CPOE to enter pharmacy, laboratory, and imaging orders; 82% were able to access all aspects of the clinical information system with a single sign-on; 86% of the respondents had at least three types of clinical decision support (order sets, drug-drug interaction warnings, and pop-up alerts); and 90% had a CPOE committee in place. The overall infusion levels were therefore at the high end along all measures. 40

Extent of Unintended Consequences

[triangle] summarizes the responses to the eight questions about unintended consequences. For six of the questions, the responses indicated that the unintended consequences were important in those institutions. Workflow was rated as a 3, 4, or 5 by 88%, communication 84%, dependence on technology 83%, system demands 82%, emotions 80%, and more work or new work 72%. For two of the questions, many respondents answered that the unintended consequence had not been seen or was less important than others. Changes in the power structure was rated as 0 by 36% of respondents, and either 1 or 2 by 29%. New kinds of errors were rated 0 by 9% and either 1 or 2 by 44%. We considered that the number of years that CPOE had been in place might affect the occurrence of these consequences to either a lesser or greater extent. The response rating for these questions was evaluated for a correlation between it and duration of use in years and no correlation was found for any of the eight questions (Spearman’s rho statistic). 46

Table 2
Table 2 Responses to the Survey Questions Regarding Unintended Consequences *

Types of Unintended Consequences

Sixty-four pages of qualitative data were analyzed from the free text responses, with representative quotes from respondents shown in [triangle]. Comments are summarized below:

  • More/New Work: Since large unanticipated changes in the amount and nature of work seem to be a regular consequence of CPOE, we asked how great a problem it is hospital-wide and discovered there are many kinds of new work generated. Over 72% said it is a moderate to very important issue. Work increases because 1) sometimes the systems are slow, 2) non-standard cases call for more steps in ordering, 3) there are training issues, 4) some tasks become more difficult, 5) with more data available more is done, 6) nurses have to be diligent about taking orders off regularly, 7) the computer “forces the user to complete all steps,” and 8) information technology staff take on many new responsibilities. Respondents often noted that there were also marked shifts in work, so that the physician must do tasks formerly done by others.
  • Workflow: We asked if CPOE altered workflow and to what extent. 87% of respondents said that it is a moderate to very important issue. Many commented that CPOE increased efficiency, which is generally an expectation and therefore not an unintended consequence. Often, however, interviewees noted that while it improved workflow for some workers, it also negatively affected the workflow of others, especially physicians, who must put extra time into ordering. Some commented that it is hard to identify the effect of CPOE vs. other technological advances in communications such as cell phones, and others noted that the workflow was negatively affected in the beginning but positively affected later on.
  • System Demands: We asked about the maintenance, training, and support efforts that are ongoing and the extent of the problem. 82% of informants said it is a moderate to very important problem. Demands are greatest during the first few months, but even after that “it takes an army to build and maintain” a system, even a commercial one. There is need for continuous equipment upgrades. A positive yet unintended consequence described often is “demand from the organization exceeding our supply,” meaning that expectations escalate or, as another interviewee said, “there is such an appetite for continued growth.” One informant strongly stated “do not do this unless you can support it.”
  • Communication: The next question asked about alterations in communication patterns among clinical staff, with 84% of respondents noting that this was an important outcome of CPOE, but not necessarily for the better. Beliefs were strong one way or the other, with about half of the comments indicating that CPOE has cut down on the need for clarifying orders and the other half saying that people no longer communicate adequately. Many respondents commented that their organizations had to make a concerted effort to educate clinicians about using face-to-face communication when it is appropriate and not to rely solely on the system.
  • Emotions: We asked if users tend to express strong feelings about CPOE and 80% said it is a moderate to very important issue. Many interviewees noted that reactions go “in both directions.” One informant said “pick your favorite terms of praise or profanity. They are all used.” Physicians get upset because it takes more of their time, but most tend to accept CPOE over time. The strength of the emotions is often unanticipated, along with the change over time from strongly negative to strongly positive.
  • New Kinds of Errors: Although CPOE has been proposed as a solution to the medical errors problem, there are also new kinds of errors generated as a result of computerization. We asked if respondents had seen new patient safety issues with CPOE. 9% said it was not a problem and only 47% said it is moderately to very important. The respondents cited entering orders on the wrong patient, errors of omission, nurses not knowing an order had been generated, desensitization to alerts, loss of information during care transitions, wrong medication dosing, and overlapping medication orders. There were many comments that safeguards have been built into the systems, that there are many near misses but few actual errors and they tend to be minor, and that there is “a potential for new issues [errors] so awareness is vital.”
  • Power Shifts: We asked how the balance of power shifts when CPOE is used. Although shifts in the power structure were clearly evident during our fieldwork, our interviewees did not feel they were very important. 36% said it is not a problem, 29% said it is a less important issue, and only 36% thought it was a moderate to very important issue. They noted that power flows away from physicians to pharmacists, nurses, information technology staff, and administration. Some said it is a phenomenon that occurs early during the implementation effort, that CPOE is used as a way of forcing or monitoring physician behavior. Others said that there is “not so much a change in the balance of power, but it does shine a light on how decisions are made in a very public way.”
  • Dependence on the System: In fieldwork, we found that dependence on CPOE and surrounding systems is expected, but the extent of this dependence was an unintended outcome. Clinicians eventually become extremely dependent on the system, so downtime creates problems. We therefore asked about consequences when the CPOE becomes unavailable so that we could discover the extent of the problem. This is indeed an issue for over 82% of respondents and comments indicated that it gets worse over time. Most interviewees said that backup processes go into effect if there is downtime, but that “it is basically managed chaos with a very negative impact on productivity.”

Table 3
Table 3 Comments from Respondents

Discussion

Our preliminary qualitative study identified major types of unintended adverse consequences of CPOE. This survey study goes beyond the prior study describing these types by advancing our knowledge about how widespread the problems are. The survey results verified the existence of the eight unintended adverse consequences types we asked about at most of the 176 hospitals with CPOE, and analysis of comments has offered insight into the perceived importance of the consequences. All types of consequences are indeed widespread although two of them, power shifts and new kinds of errors, were not considered as important as the others. While the current study was based on a categorization of adverse unanticipated consequences of CPOE from the prior study, we found that there are positive as well as negative unintended consequences. Often, the same consequence can be viewed different ways by different people depending on their perspectives.

Our mixed methods approach, using extensive fieldwork to gather data about types of unintended consequences at excellent sites, followed by a phone survey to verify the existence of these same types across the country and to gain further insight, yielded rich data.

The median length of time organizations have had CPOE in place is five years, which is relatively recent since systems have been available since the early 1970’s, 47 although only a few pioneering hospitals had it prior to ten years ago. Interestingly, there appears to be no correlation between the importance of each type of unintended consequence and the length of time that a system has been in place. Despite the recent timing of many implementations, the systems are highly infused, indicating that they are deeply embedded into the workflow, heavily used, and sophisticated.

One limitation of this study is the difficulty of asking for a simple score in response to a complex question. This difficulty was offset by allowing respondents to provide comments after each question—a distinct advantage of telephone surveys. The ability of the researcher to give examples and explain questions was helpful, but it is possible that some respondents may have misinterpreted some questions. There was no attempt to test the reliability and validity of the instrument. We asked to speak to someone in the organization who is most knowledgeable about CPOE, and we usually spoke to someone on the information technology staff with a clinical background. While these individuals seemed to offer balanced answers, they might have given answers biased in a positive way because they believe in CPOE’s benefits. This might help explain why the two categories of power shifts and new kinds of errors did not rank highly: often the power shifted to administration or information technology staff, so informants may not recognize their own gain in power. New kinds of errors may seem minor to these interviewees who seem confident that these errors are most often caught before harm is done. In addition, as representatives of these hospitals, informants may have been reluctant to discuss the sometimes-sensitive subject of errors in heath care. As previously noted, there was a bias toward eliciting adverse unintended consequences, based on the survey taxonomy being derived from a prior study of adverse consequences.

Responding sites tended to differ from non-respondents by management type, with VA hospitals the most likely to respond and for-profit centers, the least likely. Perhaps VA centers are proud of their systems and/or staff members are simply more willing to take the time to talk about them. Unfortunately, many organizations have policies against doing surveys, an increasingly common problem. 48 The reason for geographic differences in the likelihood of response is unclear. As stated, there was no difference in the level of response with respect to bed size or whether the site was a teaching institution.

Once unintended consequences are revealed, hospitals can work to either try to avoid them or to accept them as tradeoffs. For example, the negative impact of information technology on clinical workflow noted by others 49–52 is especially true of CPOE because it takes more of the physician’s time to enter orders than in the manual process, 53–54 but this might be accepted as a tradeoff for the added safety benefits of CPOE.

Our chief recommendation is that developers and implementers of CPOE systems consider each of the eight types of unintended adverse consequences carefully during their planning. The workflow issues are immense, and they affect nearly every hospital staff member. Communication changes, 55 and implementers must take steps to prevent or offset the altered communications created by electronic systems in this respect. As people become more and more dependent on such systems, 56 preparations for downtime must be more elaborate and serious. More work and new work are inevitable, as are increasing system demands, so ample resources must be made available on an ongoing basis. Finally, emotions run high for and against CPOE, 57 and efforts must be made to understand and manage them. Expectation management is therefore paramount. Informants indicated that power shifts and new kinds of errors were less important kinds of unintended consequences, but we must nevertheless be diligent about identifying them. 18,58–63 Unless we make a concerted effort to avoid, manage, and/or overcome unintended consequences, the implementation of clinical information systems may lead to detrimental results. 64–67

The eight kinds of unintended adverse consequences of CPOE we investigated are experienced by most CPOE sites, regardless of length of use of CPOE. CPOE is highly infused, or deeply embedded, at most of these sites, which is surprising considering that most implementations of CPOE are relatively recent. The telephone survey has given us insight into the extent and importance of unintended consequences. Management of possible unintended consequences is vital for maintaining control and avoiding problems. Implementation success depends on managing the unintended, as well as the intended, consequences, of CPOE.

Footnotes

This research was supported by grant LM06942 from the National Library of Medicine, National Institutes of Health, titled Overcoming the Unintended Consequences of Computerized Physician Order Entry Implementation. Emily Campbell and Dr. Guappone were also supported by National Library of Medicine training grant ASMMI0031. The authors extend special thanks to Caroline Wethern, Cody Curtis, and Ashley Jones for their assistance with data collection and organization.

References

1. Rind DM, Safran C, Phillips RS, et al. Effect of computer-based alerts on the treatment and outcomes of hospitalized patients Arch Intern Med 1994;154:1511-1517. [PubMed]
2. King WJ, Paice N, Rangrej J, Forestell GJ, Swartz R. The effect of computerized physician order entry on medication errors and adverse drug events in pediatric inpatients Pediatrics 2003;112(3 Pt 1):506-509. [PubMed]
3. Bates DW, Boyle DL, Vander Vliet MB, Schneider J, Leape L. Relationship between medication errors and adverse drug events J Gen Intern Med 1995;10:199-205. [PubMed]
4. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients JAMA 2001;285(16):2114-2120. [PubMed]
5. Chertow GM, Lee J, Kuperman GJ, et al. Guided medication dosing for inpatients with renal insufficiency JAMA 2001;286(22):2839-2844. [PubMed]
6. Dexter PR, Perkins S, Overhage JM, Haharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients N Engl J Med 2001;345(13):965-970. [PubMed]
7. Nightingale PG, Adu D, Richards NT, Peters M. Implementation of rules based computerized bedside prescribing and administration: intervention study BMJ 2001;320(7237):750-753. [PMC free article] [PubMed]
8. Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes JAMA 2005;293(10):1223-1238. [PubMed]
9. Teich JM, Merchia PR, Schmiz JL, Kuperman GJ, Spurr CD, Bates DW. Effects of computerized physician order entry on prescribing practices Arch Intern Med 2000;160:2741-2747. [PubMed]
10. Bates DW, Kuperman G, Teich JM. Computerized physician order entry and quality of care Qual Man Health Care 1994;2:18-27. [PubMed]
11. Bates DW, Kuperman GJ, Rittenberg E, et al. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests Am J Med 1999;106(2):144-150. [PubMed]
12. Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention J Am Med Inform Assoc 1999;6(4):313-321. [PMC free article] [PubMed]
13. Bobb A, Gleason K, Husch M, Feinglass J, Yarnold PR, Noskin GA. The epidemiology of prescribing errors: the potential impact of computerized prescriber order entry Arch Int Med 2004;164(7):785-792. [PubMed]
14. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review[see comment] Arch Int Med 2003;163(12):1409-1416. [PubMed]
15. Massaro TA. Introducing physician order entry at a major academic medical center: I. Impact on organizational culture and behavior Acad Med 1993;68:20-25. [PubMed]
16. Massaro TA. Introducing physician order entry at a major academic medical center: II. Impact on medical education Acad Med 1993;68:25-30. [PubMed]
17. Southon FC, Sauer C, Grant CN. Information technology in complex health services: organizational impediments to successful technology transfer and diffusion J Am Med Inform Assoc 1997;4:112-124. [PMC free article] [PubMed]
18. Koppel R, Metlay JP, Cohen A, et al. Role of computerized physician order entry systems in facilitating medication errors JAMA 2005;293(10):1197-1203. [PubMed]
19. Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors J Am Med Inform Assoc 2004;11(2):104-112. [PMC free article] [PubMed]
20. Berger RG, Kichak JP. Computerized physician order entry: helpful or harmful? J Am Med Inform Assoc 2004;11(2):100-103. [PMC free article] [PubMed]
21. Nebeker JR, Hoffman JM, Weir CR, Bennett CL, Hurdle JF. High rates of adverse drug events in a highly computerized hospital Arch Int Med 2005;165(10):1111-1116. [PubMed]
22. Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot JAMA 2005;293(10):1261-1263. [PubMed]
23. Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system Pediatrics 2005;116(6):1506-1512. [PubMed]
24. Wachter RM. Expected and unanticipated consequences of the quality and information technology revolutions JAMA 2006;295(23):2780-2783. [PubMed]
25. Ash JS, Gorman PN, Seshadri V, Hersh WR. Computerized physician order entry in U.S. hospitals: Results of a 2002 survey J Am Med Inform Assoc 2004;11:95-99. [PMC free article] [PubMed]
26. Ash JS, Gorman PN, Lavelle M, Lyman J, Fournier L, Carpenter J, Stavri PZ. Perceptions of physician order entry: results of a cross-site qualitative study Meth Inform Med 2003;42:313-323. [PubMed]
27. Poon EG, Blumenthal D, Jaggi T, Honour MM, Bates DW, Kaushal R. Overcoming barriers to adopting and implementing computerized physician order entry systems in U.S. hospitals Health Aff 2004;23(4):184-190. [PubMed]
28. Cutler DM, Feldman NE, Horwitz JRUS. adoption of computerized physician order entry systems Health Aff 2005;24(6):1654-1663. [PubMed]
29. Hillman JM, Given RS. Hospital implementation of computerized provider order entry systems: results from the 2003 leapfrog group quality and safety survey J Healthc Inf Manag 2005;19(4):55-65. [PubMed]
30. Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended sociotechnical consequences of computerized provider order entry Int J Med Inform 2007;76(suppl 1):21-27. [PubMed]
31. Zhang J, Patel VL, Johnson TR, Shortliffe EH. A cognitive taxonomy of medical errors J Biomed Inform 2004;37:193-204. [PubMed]
32. Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry J Am Med Inform Assoc 2006;13:547-556. [PMC free article] [PubMed]
33. American Hospital Association Resource Center. Available at: www.aha.org/aha/resource_center/fastfacts/fast_facts_US_hospitals.html. Accessed June 30, 2006.
34. Lincoln Y, Guba E. Naturalistic Inquiry. Newbury Park, CA: Sage; 1985.
35. Crabtree BF, Miller WL. Doing Qualitative Research. Newbury Park: Sage; 1992.
36. Glaser B, Strauss A. The Discovery of Grounded Theory. New York: Aldine; 1967.
37. Miles MB, Huberman AM. Qualitative Data Analysis. 2nd ed.. Thousand Oaks, CA: Sage; 1994.
38. Ash JS, Smith AC, Stavri PZ. In: Friedman CP, Wyatt JC, editors. Performing subjectivist studies in the qualitative traditions responsive to users, chapter in Evaluation Methods in Medical Informatics. 2nd ed.. New York: Springer; 2006.
39. Ash JS, Sittig DF, Dykstra RH, Campbell EM, Guappone K. Exploring the unintended consequences of computerized physician order entry MedInfo Proc 2007[in press]. [PubMed]
40. Sittig DF, Guappone K, Campbell EM, Dykstra RH, Ash JS. A survey of U.S.A. acute care hospitals’ computer-based provider order entry system infusion levels MedInfo Proc 2007[in press]. [PubMed]
41. Boudreau MC, Seligman L. A learning-based model of use: insights from a case study of ERP implementation Diffusion Interest Group in Information Technology Workshop, ICIS Conference. 2003. Seattle, WA, December 13.
42. Saga V, Zmud RW. The nature and determinants of IT acceptance, routinization, and infusion Proc IFIP TC8 Working Conf Diff, Trans Implement Inform Technol 1993:67-86.
43. Longo DR, Hewett JE, Ge B, Schubert S. The long road to patient safety: a status report on patient safety systems JAMA 2005;294:2858-2865. [PubMed]
44. Lynch P, Beerten R, Laiho J, Marin J. Recommended standards for final outcome categories and standard definitions of response rate for social surveys. Institute for Social and Economic Research, University of Essex ISER Working Papers No. 2001-21, Available at: www.iser.essex.ac.uk/pubs/workpaps/pdf/2001-23.pdf. Accessed July 12, 2006.
45. U.S. Census Bureau. Available at: www.census.gov/geo/www/us_regdiv.pdf. Accessed March 12, 2006.
46. Conover WJ. Practical Nonparametric Statistics. 2nd ed.. New York: John Wiley; 1980.
47. Sittig DF, Stead WW. Computer-based physician order entry: the state of the art J Am Med Inform Assoc 1994;1:108-123. [PMC free article] [PubMed]
48. Dillman DA. Mail and Internet Surveys: The Tailored Design Method. 2nd ed.. New York: John Wiley; 2000.
49. Berg M. Medical work and the computer-based patient record: a sociological perspective Meth Inform Med 1998;37:294-301. [PubMed]
50. Berg M. Patient care information systems and health care work: a sociotechnical approach Int J Med Inform 1999;55(2):87-101. [PubMed]
51. Cheng CH, Goldstein MK, Geller E, Levitt RE. The effects of CPOE on ICU workflow: an observational study AMIA Annu Symp Proc 2003:150-154. [PMC free article] [PubMed]
52. Cordero L, Kuehn L, Kumar RR, Mekhjian HS. Impact of computerized physician order entry on clinical practice in a newborn intensive care unit J Perinatology 2004;24(2):88-93. [PubMed]
53. Overhage JM, Perkins S, Tierney WM, McDonald CJ. Controlled trial of direct physician order entry: effects on physicians’ time utilization in ambulatory primary care internal medicine practices J Am Med Inform Assoc 2001;8:361-371. [PMC free article] [PubMed]
54. Aarts J, Ash J, Berg M. Extending the understanding of computerized physician order entry: implications for professional collaboration, workflow and quality of care Int J Med Inform 2007;76(suppl 1):4-13. [PubMed]
55. Dykstra R. Computerized physician order entry and communication: reciprocal impacts AMIA Annu Symp Proc 2002:230-234. [PMC free article] [PubMed]
56. Kilbridge P. Computer crash–lessons from a system failure N Engl J Med 2003;348(10):881-882. [PubMed]
57. Sittig DF, Krall M, Kaalaas-Sittig J, Ash JS. Emotional aspects of computer-based provider order entry: a qualitative study J Am Med Inform Assoc 2005;12(5):561-567. [PMC free article] [PubMed]
58. French JRP, Raven B. The bases of social powerIn: Cartwright D, editor. Group Dynamics: Research and Theory. Evanston, IL: Row, Peterson; 1962.
59. Block P. The Empowered Manager: Positive Political Skills at Work. San Francisco, CA: Jossey-Bass; 1988.
60. Stein LI. The doctor-nurse game Arch Gen Psychiat 1967;16:699-703. [PubMed]
61. Mechanic D. Physician discontent: challenges and opportunities JAMA 2003;290:941-946. [PubMed]
62. DiPalma C. Power at work: navigating hierarchies, teamwork and webs J Med Human 2004;25:291-307. [PubMed]
63. Ash JS, Sittig DF, Campbell E, Guappone K, Dykstra R. An unintended consequence of CPOE implementation: shifts in power, control, and autonomy AMIA Annu Symp Proc 2006:11-15. [PMC free article] [PubMed]
64. Chiles JR. Inviting Disaster: Lessons from the Edge of Technology. New York: HarperBusiness; 2002.
65. Doerner D. The Logic of Failure: Recognizing and Avoiding Error in Complex Situations. New York: Basic Books; 1996.
66. Perrow C. Normal Accidents: Living with High-Risk Technologies. Princeton NJ: Princeton U. Press; 1999.
67. Tenner E. Why Things Bite Back: Technology and Revenge of Unintended Consequences. New York: Vintage; 1997.

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association
PubReader format: click here to try

Formats: