 |
|
|
The Agency for Healthcare Research and Quality (AHRQ), through its Evidence-Based Practice Centers (EPCs), sponsors the development of evidence reports and technology assessments to assist public- and private-sector organizations in their efforts to improve the quality of health care in the United States. This report was requested and funded by the Office of the Assistant Secretary for Planning and Evaluation (ASPE) with additional funding from the Office of Disease Prevention and Health Promotion (ODPHP), U.S. Department of Health and Human Services. In addition, the report was requested by the Leap Frog Group and the Centers for Medicare & Medicaid Services (CMS). The reports and assessments provide organizations with comprehensive, science-based information on common, costly medical conditions and new health care technologies. The EPCs systematically review the relevant scientific literature on topics assigned to them by AHRQ and conduct additional analyses when appropriate prior to developing their reports and assessments.
To bring the broadest range of experts into the development of evidence reports and health technology assessments, AHRQ encourages the EPCs to form partnerships and enter into collaborations with other medical and research organizations. The EPCs work with these partner organizations to ensure that the evidence reports and technology assessments they produce will become building blocks for health care quality improvement projects throughout the nation. The reports undergo peer review prior to their release.
AHRQ expects that the EPC evidence reports and technology assessments will inform individual health plans, providers, and purchasers, as well as the health care system as a whole, by providing important information to help improve health care quality.
We welcome written comments on this evidence report. They may be sent by mail to the Task Order Officer named below at: Agency for Healthcare Research and Quality, 540 Gaither Road, Rockville, MD 20850, or by e-mail to epc@ahrq.gov.
Carolyn M. Clancy, M.D.
Director
Agency for Healthcare Research and Quality
Michael J. O'Grady, Ph.D.
Assistant Secretary for Planning and Evaluation
U.S. Department of Health and Human Services
CAPT Penelope Royall, P.T., M.S.W.
Director
Office of Disease Prevention and Health Promotion
Office of Public Health and Science
Office of the Secretary
Jean Slutsky, P.A., M.S.P.H.
Director, Center for Outcomes and Evidence
Agency for Healthcare Research and Quality
Beth A. Collins Sharp, R.N., Ph.D.
Acting Director, EPC Program
Agency for Healthcare Research and Quality
Margaret Coopey, R.N., M.G.A., M.P.S.
EPC Program Task Order Officer
Agency for Healthcare Research and Quality
Objectives: An evidence report was prepared to assess the evidence base regarding benefits and costs of health information technology (HIT) systems, that is, the value of discrete HIT functions and systems in various healthcare settings, particularly those providing pediatric care.
Data Sources: PubMed®, the Cochrane Controlled Clinical Trials Register, and the Cochrane Database of Reviews of Effectiveness (DARE) were electronically searched for articles published since 1995. Several reports prepared by private industry were also reviewed.
Review Methods: Of 855 studies screened, 256 were included in the final analyses. These included systematic reviews, meta-analyses, studies that tested a hypothesis, and predictive analyses. Each article was reviewed independently by two reviewers; disagreement was resolved by consensus.
Results: Of the 256 studies, 156 concerned decision support, 84 assessed the electronic medical record, and 30 were about computerized physician order entry (categories are not mutually exclusive). One hundred twenty four of the studies assessed the effect of the HIT system in the outpatient or ambulatory setting; 82 assessed its use in the hospital or inpatient setting. Ninety-seven studies used a randomized design. There were 11 other controlled clinical trials, 33 studies using a pre-post design, and 20 studies using a time series. Another 17 were case studies with a concurrent control. Of the 211 hypothesis-testing studies, 82 contained at least some cost data. We identified no study or collection of studies, outside of those from a handful of HIT leaders, that would allow a reader to make a determination about the generalizable knowledge of the study's reported benefit. Beside these studies from HIT leaders, no other research assessed HIT systems that had comprehensive functionality and included data on costs, relevant information on organizational context and process change, and data on implementation.
A small body of literature supports a role for HIT in improving the quality of pediatric care. Insufficient data were available on the costs or cost-effectiveness of implementing such systems.
The ability of Electronic Health Records (EHRs) to improve the quality of care in ambulatory care settings was demonstrated in a small series of studies conducted at four sites (three U.S. medical centers and one in the Netherlands). The studies demonstrated improvements in provider performance when clinical information management and decision support tools were made available within an EHR system, particularly when the EHRs had the capacity to store data with high fidelity, to make those data readily accessible, and to help translate them into context-specific information that can empower providers in their work.
Despite the heterogeneity in the analytic methods used, all cost-benefit analyses predicted substantial savings from EHR (and health care information exchange and interoperability) implementation: The quantifiable benefits are projected to outweigh the investment costs. However, the predicted time needed to break even varied from three to as many as 13 years.
Conclusions: HIT has the potential to enable a dramatic transformation in the delivery of health care, making it safer, more effective, and more efficient. Some organizations have already realized major gains through the implementation of multifunctional, interoperable HIT systems built around an EHR. However, widespread implementation of HIT has been limited by a lack of generalizable knowledge about what types of HIT and implementation methods will improve care and manage costs for specific health organizations. The reporting of HIT development and implementation requires fuller descriptions of both the intervention and the organizational/economic environment in which it is implemented.
The United States health care system is at risk due to increasing demand, spiraling costs, inconsistent and poor quality of care, and inefficient, poorly coordinated care systems. Some evidence suggests that health information technology (HIT) can improve the efficiency, cost-effectiveness, quality, and safety of medical care delivery by making best practice guidelines and evidence databases immediately available to clinicians, and by making computerized patient records available throughout a health care network. However, much of the evidence is based on a small number of systems developed at academic medical centers, and little is known about the organizational changes, costs, and time required for community practices to successfully implement off-the-shelf systems.
An analysis of the usefulness of implementing HIT must take into consideration several factors:
The potential of this technology to improve health care quality, safety, and patient satisfaction—and how this potential has been demonstrated.
The cost-effectiveness of the technology—the business case for adoption of the technology—including the total costs of implementation (both financial and in terms of resources) and any cost savings that accrue. Concerns exist that those who bear the greatest share of such costs are not able to recoup those costs.
The ability to generalize the effects of an HIT intervention on costs and benefits in existing systems (using published experience with or research on these systems) to the technology's use by other health care organizations.
The Leap Frog Group and a number of components of the U.S. Department of Health and Human Services (HHS)—the Centers for Medicare & Medicaid Services (CMS), the Office of the Assistant Secretary for Planning and Evaluation (ASPE), the Office of Disease Prevention and Health Promotion (ODPHP), and the Agency for Healthcare Research and Quality (AHRQ)—requested a review of the research on HIT to compile and evaluate the evidence regarding the value of discrete HIT functions and systems in various health care settings. This Evidence-based Practice Report on the costs and benefits of health information technology systems, along with an accompanying interactive database that catalogs and assesses the existing evidence was prepared by the Southern California Evidence-based Practice Center (EPC). This report systematically reviews the literature on the implementation of HIT systems in all care settings and assesses the evidence in four specific circumstances:
1. The costs and benefits of HIT for pediatric care.
2. The ability of one aspect of HIT, the electronic health record (EHR), to improve the quality of care in ambulatory care settings.
3. The costs and cost-effectiveness of implementing HER.
4. The effect of HIT on making care more patient-centered.
An electronic search of PubMed, the Cochrane Controlled Clinical Trials Register, and the Cochrane Database of Reviews of Effectiveness (DARE) was conducted for articles published from 1995 to January 2004. Additional references were obtained by reviewing the references in several major reports prepared by private industry and by RAND Health. Two reviewers, each trained in the critical analysis of scientific literature, independently reviewed each study and resolved disagreements by consensus. The principal investigator resolved any disagreements that remained unresolved after discussions between the reviewers.
Studies selected for review had to be either:
A meta-analysis.
A systematic review.
Original research that tested a hypothesis (that is, a report that compared data between groups or across time periods, assessing a specific question and using statistical tests to assess differences).
Original research that conducted predictive analyses (a report that used modeling techniques and simulations to predict the effects of an HIT implementation).
Of 855 articles screened, 256 were accepted for review. Descriptive studies of HIT implementations were identified and classified according to the categories listed below, but were not reviewed in more detail.
The contents of each selected article or report were abstracted using electronic data-abstraction forms prepared especially for this analysis. Abstracted data included the system's capabilities, interventions used, study design, implementation processes, evaluation methods, outcomes, costs, and barriers to implementation. A structured abstract was created for each report; these abstracts can be accessed in an online, interactive database created for this evidence report. (This database can be accessed at http://healthit.ahrq.gov/tools/rand.)
Of the 256 studies reviewed, 156 were about decision support, 84 assessed the electronic medical record, and 30 were about computerized physician order entry (CPOE). One hundred twenty-four of the studies assessed the effect of the HIT system in the outpatient or ambulatory setting, while 82 assessed its use in the hospital or inpatient setting. Ninety-seven studies used a randomized design. There were 11 controlled clinical trials, 33 studies that used a pre/post design, 20 studies that did a time series, and another 17 that were case studies with a concurrent control. Among the 211 hypothesis-testing studies, 81 contained at least some cost data.
Many of the studies concerned HIT systems developed and evaluated by academic and institutional leaders in HIT.
Regenstrief Institute in Indianapolis, IN (18 studies)
Partners/Brigham and Women's Hospital in Boston, MA (19 studies)
Intermountain Health in Salt Lake City, UT (11 studies)
Kaiser Permanente health care system (5 studies)
Vanderbilt University in Nashville, TN (2 studies)
U.S. Department of Veterans Affairs (VA) health care system (15 studies)
Studies from these institutions have contributed greatly to our knowledge about the usefulness of particular HIT functionalities (such as CPOE or computerized electronic alerts), and are examples of what can be realized by the implementation of broadly functional HIT at these specific institutions. But these studies also have limitations, in terms of their usefulness to inform decisions about the adoption of HIT elsewhere. The primary limitation is that these HIT systems were developed over the course of many years by technology champions at these institutions and, in a process of co-evolution, were adapted particularly to the working environment and culture of their respective institutions. Consequently, the “intervention” at these sites consists not only of the HIT system but also the local champions, who were often also the evaluators in published studies. Furthermore, it is challenging to calculate the cost of the development of the HIT system as a whole, since this process occurred over many years at each institution. In addition, these systems are not commercially available from a vendor—and vendors supply most HIT systems in use in the U.S.
We were able to identify only 15 studies that used a randomized or controlled clinical (RCT or CCT) design, included cost data, and assessed HIT systems that were not from one of the leading academic and institutional HIT institutions or the United Kingdom (another setting that has limited generalizability to U.S. health care institutions). When these 15 studies were examined for their HIT functionality using the classification system developed by the Institute of Medicine, 4 of them concerned only decision support and 4 assessed HIT systems with decision support and administrative processes. The remaining seven studies addressed other single functionalities or combinations of up to three functionalities. We were not able to find a single study that used a randomized or controlled clinical trial design, that did not report data from one of the leading academic or institutional HIT systems or the U.K., that reported cost outcomes and that assessed an HIT system including at least four of the eight IOM categories of functionality.
For the 103 hypothesis-testing studies that used a design other than a randomized or controlled clinical trial, 45 reported cost data. Of these 45 studies, 23 assessed systems that were not one of the leading academic or institutional HIT systems or that came from the U.K. An examination of these 23 studies for their functionalities showed, as in the studies using an RCT or CCT design, that most studies did not evaluate systems with a broad level of functionality. Five studies assessed only decision support, and three studies each assessed only administrative processes or order entry management. Three studies assessed HIT systems with two functionalities, order entry management and decision support. The remaining nine studies assessed various combinations of two or three functionalities. No study evaluated an HIT system with at least four of the eight categories of functionality.
The literature is even sparser regarding information about the organizational context of an HIT implementation. Of the hypothesis-testing studies, we identified only 3 studies that provided information about the financial context of the organization, such as the degree of managed care/capitation penetration; 6 studies with information about system penetration; 2 studies about facilitators to implementation; 1 study explicitly discussing sustainability of the HIT intervention; 12 studies reporting extrinsic factors in valuing costs and benefits such as the health care market competitiveness; and 6 and 9 studies, respectively, reporting the initial costs of the HIT system and costs of implementation.
In summary, we identified no study or collection of studies—outside of those from a handful of HIT leadership institutions—that would allow a reader to make a determination whether the study's reported benefit was generalizable. Besides these studies from HIT leaders, no other research assessed HIT systems that had comprehensive functionality while including data on costs, relevant information on organizational context and process change, and data on implementation. This limitation in generalizable knowledge is not only a matter of study design and internal validity. Even if further randomized, controlled trials are performed, the generalizability of the evidence would remain low unless additional systematic, comprehensive, and relevant descriptions and measurements are made regarding how the technology is utilized, the individuals using it, and the environment it is used in.
Early evidence shows that stand-alone clinical decision-support systems (CDSS) (such as drug dosing calculators) can reduce medication dosing errors, and CPOE plus CDSS can reduce the incidence of harmful medication errors in the inpatient pediatric and neonatal intensive care settings. Other HIT systems, such as electronic medication administration records, pharmacy-based robots, smart infusion pumps/devices, and medication bar-coding, are predicted to reduce medication errors, but need further study.
The use of CPOE plus CDSS has been demonstrated, in separate studies, to (1) reduce the frequency or duration of antibiotic use for common pediatric illnesses such as pharyngitis and otitis media, and (2) improve completeness and reduce variation in clinical documentation. In the ambulatory setting, a single study showed that an appointment reminder system is cost-effective and significantly reduces missed appointments, while in the neonatal intensive care unit, another study showed that CPOE can reduce medication and radiology turnaround times. Therefore, the evidence for HIT cost-savings in pediatrics is limited, but appears optimistic.
Adoption of EHR systems is widely believed to be critical to the delivery of consistent, high-quality health care, although the current use of EHRs is limited. Seven studies were identified on the use of EHR in four ambulatory care settings (three in the United States and one in the Netherlands). The findings reported in all of these studies were primarily related to the implementation processes and to changes in clinical processes.
With the exception of one study that examined the effects of incorporating HIV care guidelines and alerts on quality of care for HIV-positive patients, all the studies assessed the effects of adding various types of information related to laboratory test and prescription ordering to EHR ordering screens. In general, these studies showed that providing laboratory test guidelines and related information on test-ordering screens was associated with a decrease in orders for overused tests and an increase in orders for underused tests; provision of formulary guidance was associated with increased adherence to a formulary for at least one class of medication; and addition of HIV care guidelines and alerts was associated with improved quality of care.
While EHR systems may be essential for improving efficiency and quality of health care, implementation of an EHR system requires substantial capital investments and organizational change. Consequently, many health care organizations are seeking evidence from previously implemented systems about the costs and benefits of EHR adoption in order to better inform decisions about the optimal timing and strategy for implementation.
Not all of the costs and benefits reported when implementing new systems or making changes to existing systems were financial. EHRs were associated with improvements in service and other resource utilization, provider productivity, care efficiency, documentation quality, clinical decisionmaking, guideline compliance, and costs of care.
Despite considerable variation among the few studies that modeled financial costs and benefits, all predicted substantial cost savings from EHR implementation. However, these studies each made a number of assumptions, and the predicted break-even points ranged from as short a time as 3 years to as long as 13 years.
The evidence is sparse for the ability of HIT systems to make health care more patient-centered. The best evidence of such a change is the beneficial effect on preventive care of using computerized reminders to patients. Telemedicine and consumer health informatics also have limited evidence of benefit in specific contexts. The evidence is much more limited about the health effects of more general, interactive health information technologies such as the Internet or e-mail, or the effect on patient trust and satisfaction of implementing HIT systems such as the electronic health record.
Studies identified a large number of barriers to the implementation of HIT. These barriers can be classified as situational barriers (including time and financial concerns), cognitive and/or physical barriers (including users' physical disabilities and insufficient computer skills), liability barriers (including confidentiality concerns), and knowledge and attitudinal barriers. Cutting across all of these categories, however, may be the need for a major structural and ideological reorganization of clinical medicine as it is now practiced in the majority of settings to be able to integrate itself with and enjoy the benefits of HIT.
The primary limitation of this review is the quality, quantity, and generalizability of the available (published) studies. Substantially more information regarding implementation may have been obtained by contacting leading HIT implementers and conducting structured interviews with them.
Many of the costs and financial benefits of EHR will change over the years, because they depend on the changing price of such factors as hardware, software, labor, and pharmaceuticals and medical devices. Consequently, the costs reported in some of the older articles are of limited relevance.
Predictions based on statistical models suggest that HIT has the potential to assist in dramatically transforming the delivery of health care, making it safer, more effective, and more efficient. However, the experimental evidence supporting benefits from HIT is more limited.
A number of large health care organizations have realized some of these major gains through the implementation of multifunctional, interoperable HIT systems built around an electronic health record.
The impact of HIT implementation on the cost and quality of care is not going to be consistent across institutions, independent of context. However, the specific context within which HIT is implemented, including the setting, the clinical issues, and the patient populations, greatly influences its use and effects.
More widespread implementation of HIT is limited by the lack of generalizable knowledge about what types of HIT and methods for its implementation will prove most useful for specific health organizations, especially for small practices and small hospitals.
The reporting of HIT developments and implementations needs to be improved, with greater attention given to descriptions of both the intervention and the organizational/economic environment in which the technology is implemented.
A high priority must be placed on establishing standards for the information that needs to be measured and reported in studies of HIT implementation, similar to the CONSORT standards developed for reporting clinical trials of therapeutics.
Using existing published evidence, it is not possible to draw firm conclusions about which HIT functionalities are most likely to achieve certain health benefits—and the assessment of costs is even more uncertain.
Existing evidence is not sufficient to clearly define “who pays for” and “who benefits from” HIT implementation in any health care organization—except those, such as Kaiser and the VA, that are responsible for paying for and delivering all the care for the defined population.
Statistical models can be built to estimate the costs and benefits of interoperable HIT systems within and across health care provider settings, payers/purchasers, and cumulatively across the health care continuum, but these models are based on many untested assumptions.
Implementation of HIT faces many barriers, including institutional barriers, cognitive and/or physical barriers, liability barriers, and knowledge and attitudinal barriers.
The use of health information technology (HIT) has been promoted as having tremendous promise in improving the efficiency, cost-effectiveness, quality, and safety of medical care delivery in our nation's healthcare system. The realization of these benefits is especially important in the context of reports that show five years of consecutive annual double-digit increases in healthcare costs and increases in the numbers of adverse health events.1, 2 At the same time, reports have suggested that 50 percent of all healthcare dollars are wasted on inefficient processes. Legislators and organizational leaders at the federal and state levels have emphasized the need for healthcare to follow the example of many non-healthcare industries, in which implementation of computer information technology has been critical in increasing the accessibility of mission-critical information, automating labor-intensive and inefficient processes, and minimizing human error.
The most important use for HIT may be to help reduce medical errors. This technology-based strategy has proven effective in reducing the effects of human error in industries such as banking and aviation. Clinical HIT systems may make a substantial impact on medical quality and safety by integrating relevant automated decision making and knowledge acquisition tools into the practices of medical providers, thereby reducing errors of omission that result from gaps in provider knowledge or the failure to synthesize and apply that knowledge in clinical practice. These systems, when integrated within larger HIT systems, may improve medical decision making and appropriate use of diagnostic tests and therapeutic agents.
In the ambulatory healthcare environment, the use of HIT offers a variety of benefits. First, it can improve the efficiency and financial health of the practice. For years, many offices have used computerized scheduling and financial systems to streamline office processes by tracking practice productivity and automating reimbursement processes. Second, the use of ambulatory electronic health records (EHRs) also offers an opportunity to monitor and improve clinical quality by improving information access and reducing duplicative documentation. And technology-based “e-prescribing” tools may improve the efficiency and safety of prescribing practices in the outpatient setting just as they have done in the hospital setting. Finally, the widespread adoption of HIT will allow the achievement of system connectivity and information exchange among providers of the same organization, among organizations, and ultimately regionally and nationwide.
However, the majority of medical organizations and providers have been slow to adopt HIT. Recent surveys of computerized physician order entry (CPOE) use show that only 9.6 percent of hospitals have CPOE completely available for use, and only half of these hospitals require use of CPOE.3 In the ambulatory setting, recent estimates place the use of electronic health records at 6 to15 percent of office-based physicians.4, 5 The potential advantages of widespread adoption of HIT in our nation's healthcare system make it vital to examine the scientific evidence that currently supports the relative costs and benefits of HIT, and the barriers to implementing various types of HIT systems across the spectrum of healthcare environments.
Private organizations deciding whether to invest in HIT must weigh the costs and benefits of doing so. Although the primary goal of nonprofit healthcare organizations may be to provide high-quality care, these organizations still need to watch the bottom line to survive, which includes understanding the costs of measures designed to improve quality. Such private return-on-investment (ROI) calculations can provide results that are quite different from those of societal cost-benefit analysis, which are often reported in clinical journals.
For example, one study showed that a hospital that installed a computerized reminder system to alert providers when patients were not up-to-date on their immunizations increased pneumococcal vaccine orders by 8 percent.6 Another study showed that, among the elderly, each $12 vaccination averts $20.27 in hospital costs and increases life expectancy an average of 1.2 days.7 From society's point of view, the reminder system saves money and improves health, so it is a win-win program. However, from a financial perspective, the hospital has spent money on a system that had no effect on the costs or revenues of current stays because the pneumococcal vaccine is not delivered in the hospital. To benefit from this intervention, the hospital must make a reputation for higher quality and convert it into profits. This is one example of the potential for a mismatch between who pays for and who accrues cost savings from HIT use. A more extreme example would be a hospital's implementation of a HIT intervention that averts future hospitalization. In this case, HIT implementation both costs the hospital money and decreases hospital revenues, even if the HIT implementation has a net cost-savings from a societal (or Medicare) perspective.
The business case for investing in HIT must consider both financial and nonmonetized1 consequences. The financial aspect deals with the effect on the organization's bottom line. Any HIT investment has immediate costs in purchase, adaptation to the local organization, and staff training. So the business case for HIT depends on the downstream financial benefits exceeding the immediate costs. Because profits = revenue - costs = (revenue per patient - costs per patient) × (number of patients), long-term profits can come from increases in (profitable) patients, increases in revenues per patient, or decreases in cost per patient. The easiest of these to understand is costs per patient. All organizations benefit from becoming more efficient and reducing the costs of providing particular services. HIT can reduce the waste involved in collecting information and getting it to where it is needed for better decisionmaking. This increase in efficiency can streamline health care and billing processes, and avoid the costs of unnecessary services and of dealing with errors, both in patient care and in billing. Also, working in high quality organizations has some intangible benefits to staff, which may lead to better retention and productivity at equal levels of pay.
However, if the HIT is used to raise the quality of care or change the mix of services provided, the resulting financial costs and benefits depend on how the organization is paid and what expenses it bears. These factors can greatly affect what kind of return on investment is likely and when it will be realized. The next three paragraphs provide some examples.
A reputation for higher quality should increase the demand for an organization's services in a competitive market, but it is difficult to prove that you are better than your competition or better than you used to be. HIT can raise quality and can also generate the statistics to prove you have done so. Perceived higher quality allows organizations to increase market share and to negotiate higher prices from payers whose members demand access to those organizations, even if they have to pay slightly higher premiums to get it. In a competitive fee-for-service environment, greater market share increases revenues and may also permit some economies of scale.
HIT can also be used to increase reimbursable services per patient, such as covered immunizations and exams. HIT pays if it reduces waste, but it reduces profit if it reduces current or downstream services. Hospitals whose payments are set by DRGs (a fixed payment that depends on the diagnosis of the patient but does not vary with actual costs) benefit somewhat from shorter length of stay (although the last days of a hospitalization are the cheapest), but not from reduced readmissions (except those where a Medicare patient bounces back into the hospital before sufficient time lapses post-discharge to qualify the readmission for reimbursement as a “new” episode of care). A hospital also will not benefit financially from interventions that shift care to physicians' practices.
The biggest gains from quality and HIT come when providers are paid by means of a capitated fee system. Under such a system, any investment that reduces the total costs of care for these patients can be recouped, so it pays to reduce unnecessary services and to provide care in the most efficient setting. HIT may help to share the information needed to do so. Such reasoning was behind the Department of Veterans Affairs' (VA's) decision to develop its HIT system. Most published examples of cost-saving quality projects come from health maintenance organizations (HMOs)—for example, better diabetes or heart failure care that keeps patients out of the hospital. Also for HMOs, high quality can offset other undesirable features—such as poor access or amenities—or can justify higher premiums. The gains to HMOs of better care will be more certain when capitation payments are adequately risk adjusted. Without risk adjustment, providing high quality chronic illness care, an area where HIT is particularly useful, may have the unprofitable side effect of attracting more-expensive patients.
Because some of the financial gains from high quality may go to purchasers (employers) rather than providers, particularly in noncapitated, fee-for-service environments, some purchasers have started to pay directly for quality. If the case for HIT were strong enough, insurers might want to subsidize it in part (i.e., based on the insurer's share of the provider's caseload). However, unless an insurer covers most of the patients in a particular health care organization or insurers agree to collaborate, it does not pay one insurer to subsidize HIT for an entire provider or organization because a substantial portion of the cost savings accrue to other payers (the “free rider” problem).
Non-healthcare businesses that are selecting investments might consider only financial return on investment (ROI), but providing health care is a business with an unusual emphasis on nonmonetized goals. The nonmonetized part of the business case includes all nonmonetary arguments that the organization feels will influence the decision to adopt or reject the intervention. Examples include the following:
Maintaining credentials
Satisfying reporting requirements
Satisfying a requirement to do a quality improvement project
Avoiding exposure to liability
Building goodwill or reputation
“Because it's the right thing to do.”
Many of these nonmonetized items have financial aspects. For example, the intervention may reduce the cost of meeting a preexisting reporting requirement. Also, many organizations, particularly nonprofits, have nonfinancial goals—such as providing high quality care—in addition to financial goals.2
In this report, we use the term generalizable knowledge to mean published evidence of the effects of a HIT intervention on costs and benefits that other health care organizations can use to implement HIT and reasonably expect benefits similar to those reported in the original study. Therefore, generalizable knowledge from a study has two components: (1) the internal validity of the study and (2) the utility of the information to others considering implementing HIT. We can illustrate differences in generalizable knowledge by considering some examples.
The simplest example is that of a particular pharmaceutical therapy for patients with a certain condition. In this case, a randomized, placebo-controlled trial of the new pharmaceutical agent would be a study with good internal validity. Because pharmaceuticals are manufactured for consistency in strength and are given according to specified dosing schedules, another health care organization examining the results of such a study could reasonably assume that administration of the new pharmaceutical in the same doses and to patients with similar characteristics would result in benefits similar to those reported in the original study.
A second example would be the assessment of a new surgical therapy. In such a case, the evidence would not come from a randomized, double blind, placebo-controlled trial, since this design is not generally feasible for tests of surgical therapy. Data may come from studies comparing patients randomly assigned to surgical therapy or to an alternate therapy or nonrandomized studies comparing surgically treated patients with historical controls or even case series. As the confidence in the equivalence of the comparison groups at baseline diminishes, the difference in benefit must become greater for the reader to conclude that beneficial effects on outcomes are due to differences in therapy and not other differences between groups at baseline.
Even after accepting that a particular study reports a real difference in outcomes between groups, the healthcare organization or practitioner contemplating offering surgery must consider more factors than when contemplating the prescription of a new pharmaceutical agent. Surgical therapies are not as standardized as pharmaceutical agents, and outcomes depend upon such factors as the skill of the surgical team and hospital. There is no reason to expect that every surgeon and hospital delivers equivalent care the way physicians and patients can expect a standard dose of a pharmaceutical to have equivalent potency. Hence, a study describing the effects of a surgical therapy needs to give more detail than a study describing the effects of a pharmaceutical drug, namely, enough description of the surgeon and hospital that other healthcare organizations or providers can determine whether the reported outcomes are likely to be achieved in their own clinical situation.
When considering HIT evaluation, the situation becomes even more complex. Both the intervention and the subjects of the intervention are qualitatively different in a study of HIT than in a study of a pharmaceutical or surgical intervention. HIT implementation consists of a complex organizational change undertaken to promote quality and efficiency. Studies of organizational change are fundamentally different from studies of medical therapies. Organizational interventions interact with a wide range of organizational system components. To be successful, they must address these components in a locally effective way. Thus, in a sense, these interventions are by nature not widely generalizable, in contrast to studies of narrow interventions such as pharmaceuticals, which aim to identify treatment effectiveness that is operator-independent, or generalizable across settings or providers. This difference has several important consequences. First, randomized controlled trials are not always feasible for assessing organizational change. The risks and benefits of reliance on controlled trials for evidence about interventions involving organizational change has been debated.8, 9 However, reliance only on randomized clinical trials for evidence of the effect of HIT on costs and outcomes risks restricting the focus to narrow and tightly defined elements of HIT. In many real-world applications, complex organizational change interventions are implemented as a series of steps, with each step dependent on the organizational response to the previous step. Therefore, we judge that generalizable knowledge must and can come from many types of studies. However, we also judge that these studies must report details of the intervention and the organizational characteristics of where the intervention was implemented to allow other organizations to make judgments about the applicability of the results.
We consider the intervention in HIT studies to have at least four components:
Technical—including the system components being tested (which may consist of CPOE, clinical charting, or electronic prescribing); the preexisting technology infrastructure (e.g., clinical and financial systems, network); and the existing electronic interfaces and integration.
Human factors (machine-person interface)—system usability (e.g., “user-friendliness,” system response time, intuitive user interface, support for workflow processes), support for specialty or context-specific actions (e.g., clinical content, order-sets, and level and acceptability of clinical decision-support).
Project management—effecting complex sociotechnical process change around HIT implementation, aligning IT and organizational resources to achieve project milestones, and controllership of IT budgets.
Organizational and cultural change, which may include a partnership of medical staff and administrative leadership to govern, align incentives, and mobilize organizational inertia to achieve desired outcomes through process change.
Cutting across all four of these components is effective communication. Most organizational change and IT projects have a strong but unrecognized communication component, which encompasses, among other things, the sharing of vision, values, and information about the components of HIT system selection, as well as its implementation and use.
Without an adequate description of all of these components in a study of HIT costs and benefits, it is difficult for others to be able to infer how, or even whether, they can reproduce the results. Omitting such information would be analogous to omitting the strength or dosing schedule from the report of a study of a pharmaceutical intervention.3
Similarly, the analogue of the patient in a study of HIT is the organization. No consensus exists regarding what aspects of the organization are most important to report, but some aspects are clearly important. Aspects that have been proposed as important include size, staffing, the organization's prior experience with quality improvement initiatives, processes expected to be influenced by the intervention and how these work currently, and the financial context of the organization. These characteristics may well determine which types of HIT interventions work in a given setting. For this review, we assessed (a) whether studies measured some key organizational characteristics and (b) what those characteristics were. Such characteristics might be considered key organizational demographics, just as gender, age, and illness severity would be considered key demographic characteristics for an efficacy and safety study of a new pharmaceutical.
However, knowing even these characteristics may not be enough to understand why a HIT intervention did or did not work. An organization has to do more than simply buy the software to be successful. It must also invest in adapting the software to the organization, developing new policies and procedures, and training staff. The extent to which the organization is willing and prepared to perform these and other critical additional functions to embed the HIT into all relevant systems determines organizational readiness for change. There is unfortunately little scientific knowledge about which organizational characteristics are essential, and which, like the color of the patient's eyes when assessing the effect of taking a new pill, are unimportant. Thus, even if the description of a successful intervention includes many of the details described above, without information about organizational readiness, readers cannot know whether or not the same intervention is likely to work in their own organization and how long and expensive the transitional process might be.
An evidence report on the costs and benefits of HIT systems was requested by the Leap Frog Group, the Centers for Medicare and Medicaid Services (CMS), the Office of the Assistant Secretary for Planning and Evaluation (ASPE), the Office of Disease Prevention and Health Promotion (ODPHP), and the Agency for Healthcare Research and Quality (AHRQ). The purpose of the report was to develop an evidence base regarding the value of discrete HIT functions and systems in various healthcare settings.
Original key questions for the report were:
What does the evidence show with respect to the costs and benefits of inter-operable electronic HIT data exchange for providers and payers/purchasers?
What is a framework that could be used in this study to describe levels/bundles of EHR functionality and to estimate the costs and benefits by such levels/bundles of functionality by payer/purchaser and percentage of provider penetration?
What knowledge or evidence deficits exist regarding needed information to support estimates of cost, benefit and net value with regard to HIT systems? Discuss gaps in research, including specific areas that should be addressed, and suggest possible public and private organizational types to perform the research and/or analysis.
What critical cost/benefit information is required by decision makers (at various levels) in order to give a clear understanding of HIT Systems value proposition particular to them?
What analytic methods (e.g., sources of data, algorithms, etc.) could be used to produce evidence of the costs and benefits within and across health care provider settings, payers/purchasers, and cumulatively across the health care delivery continuum and payers, of deploying electronic health information technology functions examined in this study?
What are the barriers that health care providers and health care systems encounter that limit implementation of electronic health information systems?
Each AHRQ evidence report is guided by a Technical Expert Panel (TEP). We invited a distinguished group of scientists, clinicians, and information technology experts, including individuals with expertise in medical informatics, Internet health, and telecommunications to participate in the TEP for this report. A list of panel members is included asAppendix A *
The TEP's participation in the preparation of the report began with a meeting that was conducted via conference call at the start of the project; the purpose of this meeting was to get TEP input on the scope of the project, especially the specific information technology applications to address. We were also seeking input on what constitutes evidence because most of the data on HIT implementation derive from interventions that are not RCTs, which are the usual backbone of EPC evidence reports. This particular meeting was held at two separate times in order to accommodate scheduling conflicts; TEP members were asked to participate on the date that was more convenient for them. The meetings were held on March 19 and March 26, 2004.
At this meeting, we also discussed the framework for how to conduct our research. Many TEP members were interested in HIT implementation issues, for example, what can be learned from others who have implemented HIT in various settings, including both community and academic settings. They also emphasized that HIT is often implemented through multicomponent interventions, of which IT is just one aspect.
Based on the comments received during the TEP conference calls and numerous discussions with AHRQ, it was determined that the report would focus on reviewing the evidence from existing published articles regarding the costs, benefits, and barriers to implementing HIT. Many other excellent suggestions were received during the conference calls, such as performing new cost-benefit analyses or collecting unpublished information on barriers, but the decision was made that a review of existing published evidence should precede any other analyses.
At the time this report was undertaken, another team at RAND was working on a project entitled “Leveraging Modern Information Technology to Transform Medical Care Delivery.” This project, funded by private industry, aimed to suggest policy changes that are likely to increase the rate of adoption of HIT in the United States. One part of the project involved assessing the effects of information technology on costs, health outcomes, and adverse events. We were given the list of titles from the team's November 2003 search of PubMed, which sought systematic reviews published in English from 1995 to 2003. PubMed, which is maintained by the U.S. National Library of Medicine, is widely recognized as the premier source for bibliographic coverage of biomedical literature. It encompasses information from Index Medicus, the Index to Dental Literature, and the Cumulative Index to Nursing and Allied Health Literature (allied health includes occupational therapy, speech therapy, and rehabilitation), as well as other sources of coverage in the areas of health care organization, biological and physical sciences, humanities, and information science as they relate to medicine and health care.
Our own search for studies of HIT began with an electronic search of PubMed on January 6, 2004 for reports of original research as well as any additional articles about HIT published since 1995. We ordered all articles on the HIT topics, regardless of study design or language. Appendix B shows our specific search strategies. We also searched the Cochrane Controlled Clinical Trials Register Database and the Cochrane Database of Reviews of Effectiveness (DARE). The Cochrane Collaboration is an international organization that helps people make well-informed decisions about health care by preparing, maintaining, and promoting the accessibility of systematic reviews on the effects of heath care interventions. In December 2004, we also conducted a specific search of the journal Health Affairs, developing a list of all articles with “information technology” or “information systems” as keywords. Health Affairs has published special editions on this topic in recent years.
Several other sources of evidence were considered, based on the recommendations of the TEP. Advanced Technologies to Lower Health Care Costs and Improve Quality was published in fall 2003 by the Massachusetts Technology Collaborative in partnership with the New England Healthcare Institute. Research was conducted by the First Consulting Group and was sponsored by several Massachusetts companies involved in healthcare and health insurance. The report focuses on seven advanced technologies (including examples of HIT, such as computerized physician order entry and electronic prescribing in the inpatient and ambulatory care setting) that have demonstrated both financial benefits and improved quality of care. It also includes discussions of barriers to implementation.
The Value of Computerized Provider Order Entry (CPOE) in Ambulatory Settings was published in 2003 by the Center for Information Technology, also located in the Boston area. This group conducted an international search for both academic and commercial sources of literature and also contacted 35 vendors regarding their currently available health information technology packages. The report found that CPOE can significantly improve quality while lowering costs.
Meta-Analysis on Computer-Based Clinical Reminder Systems reports on a 1996 meta-analysis of 16 trials by Shea, DuMouchel, and Bahamonde published in the Journal of the American Medical Informatics Association (JAMIA). The authors found that computer reminders in the ambulatory care setting improved utilization of vaccinations, breast cancer screenings, and colorectal cancer screenings, but not pap smears or other preventive care. Personal files were contributed by project staff, consultants, and technical expert panel members in response to a request for any applicable unpublished literature on the costs and benefits of HIT.
Articles could have been identified in more than one way (for example, the PubMed search and personal files might contain some of the same articles).
We reviewed the articles retrieved from the various sources against our exclusion criteria to determine whether to include them in the evidence synthesis and in the special interactive database tool we created to accompany this report (see below). A screening review form that contains a series of categorization questions was created to track the articles (see Appendix C *). Two reviewers, each trained in the critical analysis of scientific literature, independently reviewed each study, and resolved disagreements by consensus. The principal investigator resolved any disagreements that remained unresolved after discussions between the reviewers.
As previously indicated, this report includes evidence from articles with many different study designs. Our initial search was unrestricted by study design. The resulting articles were divided into four categories: reviews, descriptive reports, hypothesis testing-studies, and predictive analysis studies.
Review articles identified by the search were classified as either systematic (including meta-analyses) or nonsystematic. The determination of systematic versus nonsystematic was made by reading the methods section of the article to see whether an acceptable method was employed to identify evidence. This assessment was made by the Center directors working independently with consensus resolution. Only systematic reviews were considered for further inclusion.
Articles were classified as descriptive if they primarily described the workings or implementation of a HIT system. We further classified these as qualitative or quantitative, based on the presentation of information regarding such factors as number of tests ordered and costs of implementation.
A third category of articles was classified as hypothesis-testing studies, indicating that researchers attempted to answer a study question by comparing data between groups or across time periods and using statistical tests to assess differences. Hypothesis testing studies were further classified as (1) those containing an intervention with a concurrent comparison group, which included randomized and nonrandomized controlled trials and controlled before-after studies; and (2) studies with an intervention but without a concurrent comparison group, which included pre-post studies, time-series studies with more than two measurement points, and studies that used a historical control group. Additional classifications of hypothesis testing studies included those without an intervention, which were cross-sectional in nature, and “other” hypothesis testing studies.
The fourth category of studies was predictive analyses, which included studies that used modeling techniques to predict what might happen with a HIT implementation rather than what did happen. Predictive analyses include cost-effectiveness and cost-benefit analyses. They typically use data from multiple studies and depend upon several assumptions, some of which are not always explicitly stated.
Articles that were classified as systematic reviews, meta-analyses, hypothesis-testing, or predictive analyses went on to more detailed review. For reasons discussed below, we created structured abstracts for these articles and placed them in an interactive database of HIT studies (http://healthit.ahrq.gov/tools/rand).
We looked for the following data in each article: a description of the HIT system; the purpose of the study; the year or years the study was performed; the study design; the outcomes reported; a description of the study settings; the intervention and control arm; the evaluation method; a description of the HIT system, including how the system was acquired, the year the system was installed, the capability and comprehensiveness of the system; the integration of guidelines or decision support; the interoperability; the HIT implementation strategy; the financial context, such as whether this is a managed care or capitation environment, pay for performance, or area of public accountability; the system penetration; facilitators and barriers; evidence of the HIT system sustainability; extrinsic factors in valuing costs and benefits; the cost of the HIT system or systems, including initial costs of the hardware and the software; the cost of implementation, including planning, hiring, training, temporary productivity loss, data entry, and other organizational resources; anything about long-term cost; and outcomes, in terms of changes in healthcare utilization, changes in quality of care and patient safety, changes in healthcare costs, changes in efficiency and productivity, changes in revenue, and time needed to accrue the benefit. These data were judged to be important—and, in some cases, vital—to an understanding of the study's results as generalizable knowledge.
Based on considerations about a framework for considering costs and benefits of HIT and what constitutes generalizable knowledge, we determined that a synthesis of the results of the included studies could not be meaningfully accomplished using conventional EPC methods for such syntheses. In other words, because the interpretation of the results of HIT studies is quite context-specific, meta-analysis would not be appropriate. No studies were really homogeneous or similar enough to consider together.
Similarly, a narrative review needs an organizing construct, such as “studies about CPOE,” or “studies of HIT in rural hospitals,” or even “studies of HIT that incorporate decision support and report benefits and costs for patient safety in the capitated ambulatory environment.” However, the possible combinations of key variables is so vast that any limited number of narrative syntheses we might produce for this evidence report would inevitably not meet the needs of many potential users. Therefore, we decided that the most useful synthesis of this evidence would be in the form of structured abstracts of the included studies, presented in the interactive searchable database, which can be used by interested readers of this report to identify those HIT studies that meet their own particular contextual requirements. We also present four narrative reviews of studies in particular contexts, to illustrate the uses of the interactive database and also as a mechanism to discuss the strengths and limitations of the evidence regarding HIT.
A draft of this report was prepared in April 2005 and sent to the TEP members and others for review. We received comments from the persons listed in Appendix D *. Each comment received was tracked in an electronic spreadsheet and addressed in preparing the final report. Peer review comments and our responses to them are listed in Appendix E. Service as a reviewer of this report should not in any way be construed as agreeing with or endorsing the content of the report.
We screened 855 articles, of which 599 were rejected: 124 did not have HIT as the subject; 4 did note report relevant outcomes; 288 were descriptive qualitative studies; and 183 were categorized as descriptive quantitative studies. A total of 256 articles was included in the HIT interactive database. (Figure 1 presents this information pictorially.)
Of the 256 studies included in the database, 156 pertained to decision support, 84 assessed the electronic medical record, and 30 were about CPOE (categories are not mutually exclusive). One hundred twenty four of the studies assessed the effect of the HIT system in the outpatient or ambulatory setting, while 82 assessed its use in the hospital or inpatient setting. Ninety-seven studies used a randomized design; 11 were other controlled clinical trials, 33 used a pre-post design, 20 used a time series, and another 17 were case studies with a concurrent control. Among the 211 hypothesis-testing studies, 82 contained at least some cost data (or data on utilization or efficiency, that could be converted to costs).
Many of the studies concerned HIT systems developed and evaluated by academic and institutional leaders in HIT: the Regenstrief Institute, Partners/Brigham and Women's Hospital, Intermountain Health, Kaiser, Vanderbilt, and the VA health care system. The HIT systems at the Regenstrief Institute and Partners were each assessed in 18 and 19 separate studies, respectively; 15 assessed the VA health information system; 11 studied Intermountain Health; 5 studied Kaiser; and 2 assessed the HIT system at Vanderbilt. Studies from these institutions have contributed greatly to our knowledge about the usefulness of particular HIT functionalities (such as CPOE or computerized electronic alerts) and are examples of what can be realized by the implementation of broadly functional HIT at these specific institutions. But these studies also have limitations in terms of their usefulness to inform decisions about the adoption of HIT in other locations. The primary concern is that these HIT systems were developed over the course of many years by champions at these institutions, and, in a process of coevolution, were specially adapted to the working environment and culture of their respective institutions. Consequently, the “intervention” consists of not only the HIT system but also its local champions, who were often also the evaluators in published studies. Furthermore, it is challenging to calculate the cost of the development of the HIT system as a whole, since this process has occurred over many years. Finally, these systems are not commercially available from vendors, whereas most HIT systems in the United States are commercial systems.
We were able to identify only 15 studies that used a randomized or controlled clinical design, included cost data, and assessed HIT systems that were not located at one of the leading academic and institutional HIT institutions or in the United Kingdom (UK), another setting that has limited generalizability to U.S. health care institutions. When these 15 studies were examined for their HIT functionality using the classification system developed by the Institute of Medicine, 4 four of them concerned only decision support; four assessed HIT systems with decision support and administrative processes; and one study each assessed HIT systems with health information and data storage; health information and data storage with decision support; order entry management alone; order entry management with reporting and population health management; decision support with patient support and administrative processes; and health information with data storage decision support and administrative processes. In other words, we were unable to find a single study that used a randomized or controlled clinical trial design, reported data from a site other than one of the leading academic or institutional HIT systems or the UK, reported cost outcomes, and assessed a HIT system that included at least four of the eight IOM categories of functionality.
Of 103 hypothesis-testing studies that used a design other than a randomized or controlled clinical trial, 45 reported cost data. Of the 45 studies that reported cost data, 23 assessed systems that were not one of the leading academic or institutional HIT systems or UK systems. An examination of these 23 studies for their functionalities showed, as in the studies using an RCT or CCT design, that most did not evaluate systems with a broad level of functionality. Five studies assessed only decision support, and three studies each assessed only administrative processes or order entry management. Three studies assessed HIT systems with two functionalities: order entry management and decision support. The remaining nine studies assessed various combinations of two or three functionalities. No study evaluated a HIT system with at least four of the eight categories of functionality.
Regarding information about the organizational context of a HIT implementation, the literature is even more sparse. Of the hypothesis-testing studies, we identified only three studies that provided information about the financial context of the organization, such as the degree of managed care/capitation penetration; six studies with information about system penetration; one study about facilitators to implementation; one studies explicitly discussing sustainability of the HIT intervention; twelve studies reporting extrinsic factors in valuing costs and benefits, such as the healthcare market competitiveness; and six studies and nine studies, respectively, reporting on the initial costs of the HIT system and costs of implementation. No studies explicitly discussed sustainability of the HIT intervention.
In summary, we identified no study or collection of studies, outside of those from a handful of HIT leaders, that would allow a reader to make a determination about the generalizable knowledge of the system's reported benefit. Besides these studies from HIT leaders, no other research assessed HIT systems with comprehensive functionality while also including data on costs, relevant information on organizational context and process change, and data on implementation. This limitation in generalizable knowledge is not simply a matter of study design and internal validity: Even if more randomized controlled trials are performed, the generalizability of evidence will remain low unless more systematic, comprehensive, and relevant descriptions and measurements are made regarding how the technology is utilized, the individuals using it, and the environment it is used in.
As is apparent from the preceding discussion, the interpretation of studies of HIT is highly context-specific and is not amenable to the techniques of meta-analysis frequently used in other evidence reports to summarize results across studies. Certain functionalities of HIT systems have been the subject of recent reviews, such as CPOE, 10 computer-based clinical decision support systems, 11–13 and the use of computer-based guideline implementation systems.14 We will not summarize these reviews here. Readers are referred to the interactive database of HIT studies to select those studies that are most relevant to their own situation in terms of functionalities, clinical settings, outcomes reported, and other factors. The remainder of this chapter presents four examples of syntheses of the literature for specific situations: the effect of HIT in the field of pediatrics; evidence regarding the effect of the electronic health record on quality of ambulatory care; studies that report and predict the potential benefits and costs of implementation of the electronic health record; and health information technology and patient-centered care.
A decision to implement health information technology should carefully weigh the costs and benefits of incorporating it into the clinical environment. This is especially true in settings involved in the healthcare of infants and children, where patterns of practice and the needs of clinicians are unique. A recent report issued by the medical informatics taskforce of the American Academy of Pediatrics (AAP) cited a number of special requirements for the effective use of electronic medical record (EMR) systems in pediatrics.15 The practice of primary care and subspecialty pediatrics requires specialized collection of growth data, immunization history, longitudinal developmental inventories, parent education, age- and weight-based norms and dosing of therapeutics, specialized terminologies, and unique school-based forms and reports.
In the area of pediatric patient-safety, a growing number of studies have described the frequency of medication errors and adverse drug events (ADEs) in both the inpatient and ambulatory settings.16–19 For a number of reasons—including weight- and age-based medication dosing, medication unit-doses designed for adult patients, and the limited ability of children to communicate or self-check medications before they are administered, 20, 21—infants and children are at higher risk for serious medication errors and resultant ADEs than are adults. HIT is believed to be a vital component in the quest to improve medication safety in pediatrics.
These special requirements, combined with a small commercial market for pediatric HIT systems relative to the adult population, make the implementation of HIT in the pediatric setting challenging and perhaps costly. Clearly, more must be known about the relative costs and benefits of HIT implementation and use in pediatrics and evidence of its impact on the six quality aims identified in the IOM report, Crossing the Quality Chasm, 22 to deliver safe, effective, efficient, patient-centered, timely, and equitable healthcare.
Of the 256 articles included in the database, 14 articles were determined to contain quantitative data on the costs and/or benefits of HIT use in the pediatric healthcare setting. Because of a paucity of evidence, we also included descriptive quantitative studies in this section.
Medication Use and Patient Safety. Given recent insight into the prevalence of medication errors in the pediatric population, health information technology is believed by most to be an important tool in reducing the rate of medication errors that occur in the care of infants and children.
Mullett et al.23 enhanced an existing adult antiinfective computerized decision-support system for use in an academic pediatric intensive care unit (PICU) and measured its impact on medication-related outcomes. The study reported a 59-percent decrease in pharmacist interventions for erroneous drug doses and a decreased number of patient days of subtherapeutic (p<0.001) or excessive (p<0.001) antiinfective doses. In addition, the surveyed physicians reported that the use of the system improved their antiinfective choices and perhaps reduced the likelihood of ADEs. The authors also reported a decreased number of orders per patient-antiinfective course as well as decreased robust estimated costs of antiinfective use by 9 percent in the intervention group vs. control ($86.60 vs. 78.43).
A study by Fortescue and colleagues24 examined and characterized 616 medication errors occurring in the pediatric inpatient units of two academic tertiary referral medical centers. In a hypothetical experiment, physician experts determined what percentage of these errors could potentially have been prevented by the implementation of safety systems. Specifically, this hypothetical experiment determined that basic CPOE would avert 60 percent of potentially harmful errors, while CPOE with clinical decision-support systems (CPOE +CDSS) would increase the prevention of harmful errors to 75.8 percent. Other HIT systems identified by the report as being important for averting medication errors in pediatrics settings included computerized/electronic medication administration record (e-MAR) (19.2 percent of potentially harmful errors), robots in pharmacy (2.5 percent), smart intravenous infusion devices (4.2 percent), medication and patient and staff bar-coding (4.2 percent), and an automated bedside medication dispensing device (5.8 percent).
A number of studies have directly measured the benefit of CPOE using a variety of error-capture methodologies and study designs in different pediatric clinical environments. In a prospective cohort study, the authors documented medication prescribing errors (MPEs) and potential adverse drug events (PADEs) in a pediatric intensive care unit before and after implementation of a “home-grown” CPOE system.25 The data showed a significant reduction of both MPEs (30.1 to 90.2 percent, p< 0.001) and PADEs (2.2 to 1.3 percent, p<0.001). A study by Cordero and colleagues in the neonatal intensive care setting (NICU) showed that CPOE could eliminate gentamicin prescribing errors as well.26 The sum of this early evidence indicates that CPOE +CDSS has significant potential to reduce harmful medication errors, but the relative costs and complexities of achieving these beneficial outcomes need to be examined further.
Immunizations. Although a growing body of literature suggests that the use of HIT in pediatrics may be an important ingredient in reducing medication errors, a key challenge for pediatric providers lies in the area of maximizing adherence to vaccination recommendations. Paper-based immunization records do not allow for rigorous population-based monitoring or quality control. Therefore, computerized immunization registries, as separate or integrated systems and with clinical decision-support or reporting capabilities, offer tremendous potential in tracking and improving the rates of adherence to recommended immunization guidelines.
Ornstein et al.27 evaluated a computer-based preventive services alerting system integrated into an electronic medical record system in an academically affiliated family practice clinic. In addition to surveying patient and physicians regarding their perceptions of the reminder system, the researchers performed before-and-after audits of adherence to recommended preventative services including childhood immunizations. Of the five immunization services tracked, only the administration of diphtheria and tetanus booster showed a small but significant improvement (48.8 to 50.6 percent, p=0.02). Adherence to the other recommended vaccinations did not show a significant improvement.
Szilagyi, and colleagues, 28 in an academically affiliated pediatric urban clinic, used a computerized database system to generate reminder letter for influenza vaccination to patients identified with moderate to severe asthma. Eligible patients were randomized into an intervention group, which received the reminders, and a control group. After four months, a review of the medical chart revealed a significant difference in influenza vaccination rates (30 percent intervention vs. 7 percent control, p<0.01). This study demonstrated that computerized disease registry systems could serve as an important tool in improving vaccination rates in pediatrics.
Effective Disease Management. In addition to providing a potential means to influence prescribing and immunization practices in pediatrics, HIT systems also hold tremendous promise in improving clinical decisionmaking and disease management.
Medication Dosage and Delivery. Chiarelli et al.29 evaluated a microprocessor device with computerized algorithms for insulin dose adjustment for pediatric patients with insulin-dependent diabetes mellitus, based on self-monitored blood glucose (SMBG) levels. This prospective randomized on-off-on study revealed that although the mean glycosylated hemoglobin levels and pre-meal SMBG levels did not improve, control patients were more likely to experience episodes of hypoglycemia than were patients using the device, and patients using the device used less insulin than during their corresponding baseline phase (p<0.0001) and less insulin than the control group.
Disease-Based Clinical Decision Support. Schriger and colleagues30 implemented an electronic medical record in a university hospital emergency department that provided documentation advice and recommendations for laboratory testing and treatment. Using an on-off-on interrupted study design, the authors measured appropriateness of care for febrile children less than three years of age, when measured against an evidence-based guideline. No evidence was found for improvements (or worsening) in appropriateness of care during the intervention phase compared to the baseline phase. However, use of the system was found to increase documentation of essential elements of the history and physical examination by 13 percent (95% confidence interval, 10 to 15 percent) as well as documentation of after-care instructions by 33 percent (95% confidence interval, 28 to 38 percent).
The appropriate course of antibiotic treatment for acute otitis media (AOM) is an area of concern in pediatrics. A study by Christakis et al.31 measure the impact of HIT on the antibiotic prescribing behavior of pediatric providers in an academic pediatric residency training clinic and compared cohorts during the pre-intervention and post-intervention phases. During the post-intervention phase, providers were randomized to receive point-of-care advice recommending a course of antibiotics of less than 10 days duration (primary outcome) or delayed initiation of antibiotics (secondary outcome) for the treatment of AOM. Measurement of adherence to this computerized alert showed that providers in the intervention group had a 34-percent increase compared to the control group in the proportion of antibiotic prescriptions that were for less than 10 days (p<0.01). However, during the intervention period, both the intervention and control groups became more likely to prescribe antibiotics, with the intervention group deteriorating less than the control group (p<0.095). The results demonstrate that the prescribing practices of pediatricians for treatment of a common pediatric illness can be affected by a computerized reminder system.
Using a similar study design, Margolis et al.32 developed a computerized algorithm system that mandated structured input of data by providers for common pediatric problems. In return, the system provided recommendations for disease management and correct use of antibiotics. The investigators demonstrated decreased use of antibiotics for OM (p<0.001) and pharyngitis (p<0.01) as well as increased adherence to protocol recommendations for these two disease processes in the intervention group compared with the control group. However, the use of antibiotics for upper respiratory infections (URIs) did not change. The authors noted that the structured algorithms in the HIT system did improve the documentation of clinical elements important to ideal clinical care of pharyngitis, otitis media, and upper respiratory infections. It must be noted however, that this system's rigid requirements for physician documentation also made the HIT system unusable, and the physicians refused to use the system after five weeks.
Improved Documentation. Because many studies have reported an impact of HIT use on the quality and completeness of medical documentation, a study by Carroll and colleagues focused on the impact of a personal digital assistant (PDA) on documentation discrepancies in a NICU.33 In this before-and-after study, all the NICU resident physicians used a PDA-based charting system during the intervention phase, comparing their progress notes against a predefined reference standard during both phases. The authors demonstrated that after adjustment for covariates, PDA-based charting did reduce discrepancies in patient weights in the charts but did not affect the number of medication or vascular line discrepancies.
Timeliness, Efficiency and Cost-Effectiveness of Care. Quattlebaum et al.34 studied a scheduling/practice management system that automatically generated reminder postcards for appointment the following week. In this randomized controlled trial, the authors demonstrated a reduction of the no-show rate in their pediatric ambulatory practice from 19 to 10 percent. A cost-benefit analysis of the HIT system and its impact on missed appointments revealed that for each $1 spent on reminders, an additional $7.50 of revenue was captured.
In the inpatient setting, the previously discussed study of CPOE in the NICU by Cordero and colleagues26 measured not only CPOE's effect on gentamicin dosing errors but also the time from medication prescription to administration for initial doses of a single medication and radiology tests during the pre- and post-CPOE phases. The authors documented significant reductions in the average turnaround time for both medications (10.5 to 2.8 hours, p<0.01) and radiology tests (42 to 32 minutes, p<0.001).
Early evidence shows that stand-alone CDSS can reduce medication dosing errors, and CPOE + CDSS can reduce the incidence of harmful medication errors in the inpatient pediatric and neonatal intensive care settings. However, other HIT systems, such as electronic MAR, pharmacy-based robots, smart infusion pumps/devices, and medication bar coding, are predicted to reduce medication errors but need to be studied further.
HIT also has tremendous potential to improve vaccination rates and disease management in pediatric outpatients. CDSS and registries have been shown to be effective in increasing vaccination rates in targeted populations, but only a limited HIT impact on general pediatric immunization rates has been demonstrated. Similarly, a patient clinical decision-support device that assists insulin dosing in children with diabetes reduces episodes of hypoglycemia and overall insulin requirements, but does not affect traditional measurements of glycemic control. And the use of computerized documentation systems with integrated CDSS has been demonstrated, in separate studies, to 1) reduce the frequency or duration of antibiotic use for common pediatric illness such as pharyngitis and otitis media, and 2) improve completeness and somewhat reduce variation in clinical documentation.
In the ambulatory setting, a single study showed that an appointment reminder system is cost-effective and significantly reduces missed appointments. Another study showed that CPOE in the NICU can reduce medication and radiology turnaround times. Therefore, the evidence for HIT cost-savings in pediatrics is limited but deserving of optimism.
A small body of literature supports the assertion that HIT use in pediatrics is beneficial in the areas of medication safety, adherence to immunization and disease-based guidelines, patient decision-support in diabetes management, clinical documentation, patient appointments, and in-hospital order processing. No data on the costs or cost-effectiveness of implementing these systems were found, except in one case. In addition, because many of these HIT systems were tested and/or developed in academic settings, the ability to generalize these findings to other organizations is uncertain.
Despite rapid advances in the biomedical sciences, a growing body of evidence shows serious shortfalls in the quality of care Americans receive, 35, 36 and significant longstanding shortfalls in performance have persisted despite recent increases in attention to quality.37–40 International comparisons have demonstrated similar problems in quality.41–43 If the United States is to realize the full value of biomedical knowledge and of financial investments made in healthcare, the mechanisms through which that knowledge is operationalized and care is delivered must be radically redesigned.
Although the content of healthcare continues to change dramatically, the methods of health care delivery have not. In particular, a vast majority of the healthcare industry continues to deliver care, manage information, and conduct clinical transactions through the use of paper records.
Although the use of electronic health records (EHRs) is limited in healthcare, there is a renewed conviction by the government, provider groups, and healthcare purchasers that widespread adoption is critical to the delivery of consistent, high-quality care. However, EHR implementation, without other important changes in the way healthcare services are provided, is unlikely to improve quality. Such process redesign and reengineering is difficult and resource-intensive and is also hampered by the complexity and fragmentation of our current healthcare system. Therefore, despite the potential benefits of widespread EHR use, better empirical evidence is needed to confirm that EHR use does in fact improve quality and—perhaps more fundamentally—to understand what capabilities EHRs need to have for quality to be improved. At present, the depth and breadth of the empirical evidence regarding EHR use and its attributable impact on the quality of care remains unclear.
The purpose of this review is to examine and synthesize the available research evidence for the impact of EHR on quality of care in the outpatient setting. The review will also attempt to differentiate the direct impact of EHRs as point-of-care and workflow tools from how EHRs have been used to indirectly achieve those results, by measuring clinical and process outcomes. We elected to focus on ambulatory care because of the large volume of health services delivered in this arena. In addition, because the vast majority of outpatient practices comprise fewer than ten providers—many of whom lack technical infrastructure and resources—it is unclear whether widespread implementation of EHRs will be feasible in this environment.
From our database of 256 articles, we selected all 84 papers that related to EHRs. We then screened these articles against the following inclusion criteria: (1) the study reported quality-of-care data as study outcomes, (2) the EHR was documented to have the following minimal functionality—electronic documentation (viewing, entry, or both), results management, CPOE, and some form of decision support, (3) the study was conducted in the ambulatory setting. The criteria for functionality were chosen based on the IOM's “White Paper on Key Capabilities of an Electronic Health Record.” Given the rapid technical advances in EHR systems, we reviewed additional functional criteria to provide decisionmakers with the most relevant and forward-looking information available.
The Donabedian “Structure—Process—Outcome” model for quality was used as a framework for this review.44 In this model, structure is defined as the resources and factors involved in producing care and the manner in which those resources and factors are organized. Examples of structural quality include the number of beds in a hospital, the number of physicians in an emergency room per shift, the budget for a clinic, and the presence of disease management program for diabetes. Process of care is defined as the activities that constitute health care. Examples include screening for breast cancer, ordering laboratory tests, and prescribing a medication. Outcomes are the end results of healthcare delivery processes. They are the consequences of health services or can be logically attributed to the act of providing those services. Whereas structure relates to the environment in which healthcare is delivered and process relates to the provisions of care, outcomes are events that occur with patients and consumers—as individuals, groups, or populations.
Two aspects of this model are particularly relevant to EHRs. First, to fully assess quality of care, there need to be links from structure to process to outcomes. The technical and functional capabilities of EHRs form a structure for care. In order to derive value from the EHR structure, new clinical processes need to be designed to utilize the EHR functional structure. These EHR-mediated processes should in turn lead to a specific set of better outcomes. Second, the distinctions among structure, process, and outcome are somewhat arbitrary in the model. Health care delivery is viewed as an interconnected series of structure—process—outcome relationships. For example, in a primary care clinic of three physicians (structure 1), a patient may have an electrocardiogram performed (process 1), which shows an abnormality (outcome 1). This abnormal result necessitates a referral to a cardiologist (structure 1) who orders a stress test (process 2), which comes back suggestive of coronary artery disease (outcome 2) and so forth. This structure—process—outcome chain is central to the role of EHRs in quality, because an EHR is a tool that explicitly links the three. An EHR with decision support for diabetes management (structure) allows a physician to order a hemoglobin A1C (process) and check the results (outcome). Because this outcome is stored in the EHR database, it in turn becomes part of the structure of care. The EHR can allow or even remind the physicians to act on that result (process 2), e.g., modify the patient's insulin dose, which will, in turn, lead to a lowering of the patient's blood sugar (outcome 3) or a reduction in the likelihood of a long-term diabetic complication.
In addition to imposing the Donabedian Structure—Process—Outcome model, we organized deficits in quality by means of a conceptual framework that divides quality problems into three types: (1) the underuse of appropriate health services, (2) the misuse or inappropriate use of health services, and (3) the overuse of health services.36
Seven research studies were identified using the search criteria outlined earlier.45–51 Four of the seven were conducted at academic medical centers, 45–47, 51 and three of those four were conducted at a single institution, Regenstrief Institute (the fourth was conducted at Beth Israel Hospital in Boston).45–47 Two studies were conducted at a large, integrated healthcare delivery network, Kaiser, 48, 49 and one was conducted in the Netherlands.50 All four studies from the academic medical centers assessed internally developed HIT systems, rather than a commercially available system. One study48 assessed two different systems at two sites, one of which was internally developed by the organization and the other a commercially developed product.
All studies included data on structural quality. These varied highly and were largely qualitative in nature. In particular, reporting on the organizational and workflow changes needed to implement an EHR or a new EHR functionality was limited. All seven studies analyzed quality with respect to process of care. Six of the seven45–49, 51 assessed quality with respect to some type of outcome.
In terms of the types of problems the interventions were trying to address, six of the seven45–50 included data on the effects of EHRs on decreasing overused or redundantly used healthcare services. Two48, 51 included measures of the effects of EHRs on appropriate but underused care. None used explicit methods to evaluate the impact of EHRs on inappropriate use of care.
Structure. Three studies that met our criteria were conducted at the Regenstrief Institute, which includes a research institute and an ambulatory care practice affiliated with both a university medical school and a large public hospital.45–47
The development of the EHR at the Institute began in the mid-1970s. Subsequent system enhancement and implementation broadened its functionality and scope of use. In 1984, CPOE was added to the EHR capabilities and became uniformly used in the outpatient setting. In the three studies covered in this analysis, the system included electronic documentation, results management, CPOE, and decision support.
All three studies examined the effects of incorporating new information elements into the process used by physicians to order diagnostic tests. Each involved the integration of EHR-stored data into physician decisionmaking at the point of care.
The first paper45 reported on the effect of a structural change in care delivery: the addition of diagnostic test cost data to the EHR order function. Thus, physician workflow was altered through the inclusion of cost data in the order entry process, to be shown at the point of care. After physicians ordered tests, the charges for each test and the total charges for all tests were provided automatically in a new window. Physicians were then offered the option to cancel any or all tests.
This study used a complex randomized design to test the effect of the intervention. First, baseline utilization data were collected during a 14-week observation period. Second, physicians were randomized either to receive the cost data during order entry or to use the usual EHR functional interface where no cost data were provided. Finally, data collection continued for a 19-week post-intervention period.
The second paper reported on the effect of an intervention47 in which the investigators created statistical models to predict the likelihood of abnormal results for commonly ordered diagnostic tests. These pretest probabilities were displayed to physicians immediately prior to test ordering. Data needed for the models were obtained through EHR mediated prompts to physicians and from patient-specific data already electronically stored. Physician workflow was altered by the need to enter data during the test-ordering process and by the incorporation of the pretest probability into their decisionmaking process. The number of data prompts given to physicians was not reported.
The study used a randomized design in which patients were the unit of randomization. The EHR sorted patients automatically by the predetermined allocation. When physicians cared for intervention patients, the pretest probability function was activated during the electronic ordering process. When physicians ordered tests for the control patients, no additional decision support was provided.
The third paper reported on an intervention47 in which care delivery was modified through an intervention in which past diagnostic test results were automatically displayed as physicians ordered new tests. The last three results for a test, the time interval between tests, and the total number of times the test had been ordered for the patient were displayed at the point of care. No additional data entry was required of physicians. Physician workflow was altered by the need to incorporate past test results during decisionmaking.
The study used a complex multiphased, randomized design. First, during a 13-week pre-intervention period, baseline data were collected regarding physician test ordering patterns. During the 16-week intervention period, patients were the unit of randomization. Finally, test ordering was monitored during an 8-week post-intervention period.
Process. Each of these studies examined the impact of the EHR-related structural improvements described previously on physician diagnostic test ordering practices. In each study, the EHR-based intervention decreased the number of diagnostic tests ordered by physicians, suggesting that quality of care was improved through the decrease in overused health services.
The first study, 45 in which test charge data was displayed, showed an overall 14-percent decrease in the number of tests ordered by physicians per visit in the intervention group. Decreases were observed for both scheduled and for unscheduled visits. The multiphase design allowed additional conclusions to be made regarding the importance of maintaining the decision-support element as part of the structure of care. In the pre-intervention period, no differences in test ordering were noted. During the intervention period, physicians randomized to the decision support tool ordered 17 percent fewer tests. In the post-intervention period, after the decision support was removed from the EHR, physicians who had been in the intervention group ordered only 7 percent fewer tests than during baseline. This effect decrement suggests that the knowledge of costs the test physicians gained during the intervention was not sufficient to alter practice over time. Instead, it was the presence of the additional cost information within the structure of care that most affected performance.
The second study, 46 in which abnormal test result probabilities were displayed to providers, showed a 9 percent overall decrease in the number of tests ordered by physicians. Two tests, urinalysis and urine culture, which had been underused prior to the intervention, showed increases in ordering frequency (+14 percent and +27). However, for the other six, overused tests examined, decreases ranged from 4 percent (electrocardiogram) to 14 percent (chest x-ray). These findings suggest that providing point-of-care pretest probabilities via an EHR improves the quality of care processes by decreasing overused testing and by increasing the use of previously underused care.
The third study, 47 in which past abnormal test results were displayed, showed an overall 9-decrease in the number of tests ordered by physicians. As in the first study, data were analyzed in the post-intervention period to assess the persistence of the effect. After the EHR intervention was discontinued, the researchers observed a non-statistically significant 11-percent increase in the number of tests ordered (the post-intervention period time frame was not long enough for this trend to reach statistical significance).
Outcomes. In the first study, 45 the primary outcome was diagnostic test-related charges. Charges were 13 percent ($6.68 in 1988 dollars) lower per visit for the intervention group physicians than for the control group. Decreases in charges were directly due to the decrease in the number of tests ordered, i.e., the improvements in quality of care processes. Given the likelihood that the intervention reduced overused care, this outcome increases the efficiency of care delivery.
In the second study, 46 which used statistical models to predict whether any of eight commonly ordered diagnostic tests were likely to be abnormal, the primary outcome was financial charges for tests. Overall, charges decreased 9 percent ($1.09 in 1986 dollars). A technical outcome of the study was the operating characteristics of the statistical models used to predict lab test abnormalities, which were based on data collected and stored in the EHR. All predictive models for the study tests performed well, with receiver operating curve areas generally over 0.75 (range 0.66 to 0.92).
In the third study, 47 efficiency outcomes were also observed due to the decrease in the number of tests ordered. Charges for tests in the intervention group were 13 percent lower per visit (approximately $1.82 per visit in 1986 dollars).
Structure. Two studies came from regional medical centers in the Kaiser Permanente network, an integrated, not-for-profit, nonacademic healthcare delivery system.
In the first of these two studies, 48 comprehensive EHRs were implemented in two regions (Northwest and Colorado) of the enterprise. One EHR was internally developed (Colorado) and the other externally developed by a commercial vendor (Northwest). Although the EHR systems were different, both were reported to have comparable functionality, including the following: documentation, clinical results management, CPOE for both diagnostics and medications, administrative data management, and decision support. Specific decision support functions varied between the two sites in both content and scope.
The second of these two studies provided a brief qualitative description of structural changes associated with and supporting EHR implementation. Implementation was carried out gradually, in phases, beginning in discrete areas of the ambulatory care clinics. The majority of system implementation was completed within one year of initiation. The authors note that because of phased implementation, it was “some time” before changes in health care delivery were noted. (Data related to time course of impact will be discussed in the Outcomes section of this analysis.) No data on the specific organizational drivers for adoption were included.
Structural reorganization and improvements in workflow were described briefly. Prior to the implementation of EHR in each region, the presence of multiple ambulatory sites required paper records to be physically delivered. Paper charts were warehoused and had to be delivered “several miles.” For same-day and unscheduled ambulatory visits, availability of paper records was “unreliable.” After EHR implementation, use of paper charts was “essentially eliminated” and electronic patient charts became available for emergency room visits, unscheduled visits, and same-day appointments. Charts also became available for telephone contacts, and the resulting improvement in clinical workflow led to more effective utilization of telephone-based care, with physicians reporting that they were better able to address patient health issues over the phone when provided with access to electronic records. The authors cite this outcome as a primary reason for decreased office visits, one of the primary outcomes of the study.
In terms of time frame of impacts, little difference in services was noted during the first year of implementation (the authors reported system implementation was mostly completed one year after implementation began).
Evaluating the effects of EHR adoption was itself a form of structural change in this study. In order to determine appropriate utilization measures to assess the effect of EHR implementation, interviews were conducted with 100 individuals with a broad array of organizational roles. Interviews led the investigators to hypothesize that ambulatory care delivery had become more efficient by making needed information available during the initial episode of care, thus decreasing the need for follow-up visits and redundant services. Interviews also suggested that quality had improved. These hypotheses formed the basis for the selection of metrics and the quantitative evaluation done in the study (discussed in the Process and Outcome section of this analysis). No further details were provided regarding the data acquired from these interviews or the methodology used to conduct them.
The study design was a retrospective time-series analysis with data analyzed at one-year intervals before and after implementation. Baseline data were used from the three years prior to implementation. For the Kaiser Northwest site, four years of post-implementation data were available, whereas for Colorado, only two years of post-implementation data were available because of later implementation.
The second of the two studies, conducted in the Kaiser Northwest system, 49 examined the incorporation of guidelines through the EHR to support the decisionmaking process for ordering radiology tests and medications. The two-phase implementation process was described briefly. In the first, a read-only results reporting system that integrated data from departmental systems was implemented. In the second phase, the commercially developed EHR described above for the first Kaiser study was implemented. Together, both phases took approximately three years to complete. Per the authors, attempts were made to present guidelines to providers as efficiency aids that would streamline their workflow; the electronic guidelines were kept simple and integrated smoothly into existing procedures. Provider adherence to guidelines was not mandated; however, the electronic ordering system was designed to make adherence simple, and the guidelines were presented in text form without requiring explicit interaction. No further specific implementation-related information was provided in the paper for either the EHR or the guideline tools.
The study design was a time-series analysis that examined utilization patterns at multiple time points before and after guideline implementation.
Process. The first Kaiser study, 48 which examined the effects of EHR implementation at two sites in the Kaiser network, examined multiple processes of care.
Three quality indicators from the Health Plan Employer and Data Information Set (HEDIS) were chosen to assess quality. These items were chosen in part because their definitions remained consistent over the time period of the analysis. Each was a process-of-care measure: advice on smoking cessation, cervical cancer screening, and retinal eye examination. No statistically significant differences were found in performance on these process measures from the pre- to the post-implementation period.
However, multiple utilization-related processes were examined and showed considerable change after EHR implementation. In general, they suggest improvements in quality of care through a decrease in redundant health services. Age-adjusted rates of radiology test utilization decreased overall by 4 percent after EHR implementation. The authors note that over this same period, radiology service use increased within the Kaiser system as a whole and nationally as well (quantitative data not provided for either increase). Laboratory test utilization in one site decreased 18 percent four years post-implementation. However, utilization rates subsequently increased 5 to 7 percent annually. In the other site, the rate of laboratory test utilization had risen 14 percent prior to EHR implementation, but decreased by 3 percent over the two post-implementation years included in this study. Comparisons of laboratory utilization with other non-EHR sites in the network were not included in the analysis. The number of telephone encounters physicians scheduled with patients increased substantially after EHR implementation, rising from 1.3 telephone encounters per member per year to 2.1 telephone encounters per year. Per the authors, physicians qualitatively reported that telephone encounters were more effective because their capacity to resolve patient issues was enhanced by accessing the EHR.
The second Kaiser study49 focused on processes of care-related adherence for two radiology tests and on formulary adherence for one medication, after guidelines were incorporated into the EHR.
Use of upper gastro intestinal (UGI) radiology testing decreased from 11 UGI per thousand members to 6 UGI per thousand members after guideline implementation (40-percent relative decrease). The number of chest x-rays ordered also decreased 20 percent. Prescription of a nonformulary medication for depression decreased from 4.7 percent of all selective serotonin reuptake inhibitors (SSRIs) to 2.4 percent (SSRIs are the most widely used class of medications for depression and multiple agents are available for prescription in this class). Noted effects were sustained over time. The analysis made no attempt to control for other factors that may have affected utilization of radiology testing or formulary adherence.
Outcome. Outcomes in the first Kaiser study related to efficiency and utilization.48 The age-adjusted total office visits decreased by 9 percent in year 2 after initial implementation. Primary care visits decreased by 11 percent and specialty care visits decreased by approximately 5 percent, both of which were statistically significant. Reductions in visits held across patient cohorts, including those with the greatest baseline rates of visits. The number of patients making three or more visits decreased by approximately 10 percent between year 1 and year 2 post-implementation in the Northwest region and by 11 percent in Colorado. In year 4, a further decrease of 2 percent was noted in the Northwest region. No comment was made on whether these decreases in the high-volume use category were statistically significant. Direct comparisons with utilization at non-EHR sites were not possible because of inconsistent definitions of office visits. However, in three other network regions (all of which used independent definitions of a visit) for which visit utilization data were available for the same time period, no similar decreases were noted.
In terms of the statistical analysis, no strict control variables were included in the analysis. The following structural measures were reviewed separately to examine possible confounding: rates of ER visits, ratio of primary care providers to members, ratio of referrals to outside providers. Per the authors report, none changed significantly over the study time frame.
Appointments made for patients after doctor-managed telephone encounters decreased by 7 percent after the EHRs were implemented. However, when telephone contacts reverted to nurses, these appointments “rose” (no quantitative data provided).
Structure. A single report details a large multisite study in the Netherlands in which the effects of two different types of EHR laboratory test order interfaces were examined.50 Both sought to decrease the number of laboratory tests ordered by providers by presenting a limited set of tests on the primary laboratory order screen in the EHR. While all available tests in a laboratory system cannot usually be presented at once on a computer screen, these interventions did not allow screen size or human factor constraints to dictate which test options were initially made available to providers. Instead, they presented considerably smaller sets of choices. Thus, both interfaces changed provider workflow considerably when compared to paper or to nonrestrictive EHR order interfaces. Although providers could order any tests they wanted, any test not explicitly present on the EHR laboratory screen required additional search time to find and call up.
In one experimental condition, statistical probability was used to select the fifteen most commonly ordered tests overall to present to a provider on the initial order interface. In the other condition, the tests presented to providers depended on the patient's specific diagnosis. Diagnosis-specific tests were presented electronically, based on recommendations from existing guidelines. This intervention altered provider workflow to a greater extent than did the first intervention. First, a menu of guidelines/indications was presented, from which the provider had to select those most relevant to the patient's conditions. Based on the indications for testing entered by the provider, the EHR picked the most relevant to present as possible options for ordering. The guideline set was not comprehensive for all possible tests, and all possible indications for a test were not included in the electronic guidelines. Physicians could override recommended tests and order nonrecommended tests at their discretion by entering, “other indication.”
All physician practices in the sample were already using EHRs at the time of the proposed laboratory-ordering intervention. However, prior to the intervention, lab tests were ordered through structured paper-based order forms. All practices in the region using EHRs were offered the opportunity to participate in the experiment and add one of the electronic lab ordering functionalities to their systems. Of 64 practices, 46 (72 percent) agreed. Sixty-two general practitioners worked at those 46 sites. A three-month implementation period was included to familiarize the physicians with the software. Over the course of the study, four practitioners withdrew: one solo practitioner withdrew because the software decreased the performance of his computer, another withdrew because of dissatisfaction with the system, and two other physicians in the same practice withdrew for unspecified reasons. Thus, complete data were available for 44 practices, representing 60 physicians.
Physicians were still left with the option of using paper order forms during the study. In the non-guideline specified cohort, 88 percent of all orders were entered through the software. In the guideline-based electronic order cohort, 71 percent of all tests were ordered through the software. Final data analysis included total lab tests ordered both electronically and through paper forms.
Process. This study50 focused primarily on process change: examining the effect of changes in information presentation on test ordering. Physicians randomized to the guideline-based interface ordered 1.4 percent fewer lab tests (5.5 vs. 6.9) than did physicians presented with the list of most commonly ordered tests. This difference translated into a relative decrease of 20 percent in tests ordered. The 20 most commonly ordered tests accounted for 80 percent of all tests. No data on human factors issues or usability were reported. Such data may have been informative, given the different workflows created by each intervention. Further supporting the potential utility of such data are the different rates of use for each software package (in the guideline cohort, 71 percent of all tests were ordered through the software and 29 percent through paper; in the other cohort, 88 percent of all tests were ordered through the software and the remaining 12 percent through paper).
Outcomes. No outcomes were reported for this study.50
Structure. The last EHR study was conducted at the ambulatory care medicine practice at the Beth Israel Deaconess Medical Center in Boston, an academic medical center.51 Development of their clinical computing system began in the 1970s and was internal. System functionality at the time of the study included documentation, results management, order entry, decision support administrative data management, and electronic communication through email. Electronic documentation and results management capabilities were available through the Internet.
The goal of the study, which began in 1990, was to improve quality of outpatient HIV care by incorporating guideline-based alerts and alarms into the system. At the time of the study, no national consensus guideline on HIV care existed. Thus, as a first step, a set of guidelines was developed internally by a panel of local experts. The guidelines were then automated and incorporated into the EHR.
The alerts and alarms created new provider workflows when compared to a paper-based system. Decision support was given to providers on-line and without provider prompting. Clinicians were given the opportunity to act on the alerts and reminders as they appeared, by sending electronic messages for orders to be executed. The system also allowed providers to decline recommendations; and space was included in the EHR to document the reason. The workflow for each of those options differed. Alerts popped up each time a provider logged on, regardless of the patient being seen or reason for accessing the system. Reminders were shown only at the time of the patient visit. This study, which was conducted over 18 months (from 1992 to 1993), used a controlled clinical trial design. Five practice sites were involved. Coin flips were used to assign practices to the intervention or the control condition. All clinicians at a site were assigned to the same condition over the course of the study. The total sample included 22 providers.
Process. The purpose of this study51 was to assess the effects on processes of care of incorporating electronic guidelines for outpatient HIV care into the EHR. One year after implementation of the EHR guidelines, the number of eligible patients receiving recommended HIV care in the alerts intervention group was 85 percent vs. 64 percent in the control group. At three months post-implementation enhanced utilization of appropriate services was noted for all measures, including ordering CD4 counts (82 percent vs. 60 percent), starting AZT or DDI when appropriate (86 percent vs. 65 percent), modifying AZT dose (76 percent vs. 62 percent), PCP prophylaxis (88 percent vs. 42 percent), and complete blood counts (89 percent vs. 65 percent). All findings were statistically significant to a p value of 0.05 except for starting AZT/DDI or changing the AZT dose. The median response time for a provider to order appropriate services in response to new clinical information was 11 days in the intervention group and 52 days in the control group.
At one year, the number of eligible patients receiving recommended HIV care in the reminders intervention group was significantly greater than in the control group (68 percent versus 46 percent). Processes of care examined included pneumovax receipt (82 percent vs. 38 percent), TB skin testing (78 percent vs. 62 percent), H. influenza vaccination (41 percent vs. 25 percent), tetanus vaccination (31 percent vs. 17 percent), and referrals to ophthalmology (75 percent vs. 46 percent) (p values were less than 0.05 for all results except TB testing, for which p=0.07 and tetanus vaccination, for which p=0.1). At one year, toxoplasmosis titers were drawn on an equivalent percent of patients (82 percent vs. 81 percent). However, the median response time in the intervention group was 8 days vs. 168 days for the control group. No differences in cervical cancer screening were noted. In the intervention group, the median time for a provider to act on clinical information to order appropriate services was 114 days vs. more than 500 days in the control group.
In the intervention patient cohort, 303 alerts and 432 reminders were generated and sent to clinicians. In the control group, 388 alerts and 360 reminders would have been sent to providers.
Outcome. This study51 examined rates of visits to primary care, rates of hospitalizations, visits to emergency rooms or walk-in clinics. No statistically significant differences were observed. Rates of pneumocystic disease and one-year mortality also showed no differences.
The studies reviewed in this analysis illustrate a range of ways in which ambulatory EHRs can serve to improve quality of care. In particular, they demonstrate how provider performance can be improved when the clinical information management and decision support tools are available within an EHR system. A recurrent theme in these studies was the capacity of EHRs to store data with high fidelity, to make those data readily accessible, and to help translate them into context-specific information that can empower providers in their work.
This analysis is limited by a number of factors. The small number of studies included in the sample was largely a function of the search criteria. In particular, few systems with the core EHR functionalities of documentation, results management, provider order entry, and decision support have been examined, particularly for commercially developed products. These functional criteria were chosen to make the analysis more pertinent to decisionmakers currently considering EHR adoption. Because of the rapid technical advances in EHR, many of the studies of EHR systems are out of date. This review has focused on EHRs with these core functionalities in order to provide decisionmakers with an overview of the evidence that is most likely to be pertinent to the choices they are making now. Another major limitation is the lack of description (and data) pertaining to the workflow reengineering and organizational change that were required for EHR use. As discussed earlier, the “intervention” in these studies is not only the EHR system but also the manner in which these systems change the way healthcare professionals work, organizations function, and consumers receive care. This information is highly context-specific, and for the findings of research on EHRs to be more widely generalizable, this part of the “intervention” needs to be characterized, described, and measured more accurately and comprehensibly. Without such process implementation data, the applicability of findings from one context to another will be a barrier to informed decisionmaking.
Consumers of the healthcare system, including government in the United States, employers, and patients, are demanding higher quality, safety, consistency, efficiency, and value. In order to meet these demands, interoperable computerized health information technology, especially an EHR system that documents patient care processes and outcomes across the continuum of care, is widely believed to be a critical tool. Ideal use of an EHR system enables improved capture and integration of patient information from diverse sources and allows clinicians to access longitudinal patient-specific information for clinical decisionmaking and disease management. Other commonly used terms referring to aspects of an EHR system include personal health record and electronic medical records. In this review, EHR refers to a HIT element that performs the functions of electronic recording, storage, accessing, and viewing of patient medical information.52–55 An EHR system is a computer application with EHR functionality at minimum. Often, financial data are also included in such a system. Since the system is designed to be used institution-wide to replace paper-based medical records and to aid the efficiency of healthcare processes, many EHR applications also contain other system functions, including prescription and test ordering, care management reminders, and other clinical decision-support capabilities. While the EHR is considered essential technology for improving efficiency and quality of health care, implementation of an EHR system requires substantial capital investments and organizational change. Consequently, many health care organizations are seeking evidence and lessons learned about the costs and benefits of EHR adoption in order to better inform decisions about the timing and strategy for implementation.
EHR is the second most common HIT element among the articles identified that contain economic data. Our literature search identified 92 hypothesis-testing or predictive analysis articles containing information on costs, utilization, or efficiency. Of these, 32 studies assessed a HIT system in which EHR was one of the major system elements. However, only nine articles quantitatively assessed the economic value of an EHR system as a whole. We discuss these articles in further detail below. Most of the remaining studies were tests of certain nonfinancial hypotheses or examination of a subset of functionality, such as decision support, instead of the entire EHR system. Although these studies do not assess the costs and benefits of the entire system, they provide indirect, often empirical, evidence that can support the economic appraisal of the value of an EHR system. Before we begin the review of the nine articles, we first summarize the main findings of the remaining studies. Interested readers are referred to our interactive evidence database to learn more about these studies (http://healthit.ahrq.gov/tools/rand).
Among its other functions, an EHR system can facilitate automatic generation of patient reminders for preventive services, screening, and disease management. Five Canadian studies used the same EHR system to generate patient reminders and compared the effectiveness and cost-effectiveness of three strategies—physician reminders, telephone reminders, or letter reminders—to remind patients to get preventive services.56–60 All forms of reminders were effective, with reminders delivered directly to patients being somewhat more effective than reminders to physicians. Another study used computerized pharmacy records to generate patient feedback and compared the effectiveness and cost-effectiveness of two depression care programs.61 Feedback with care management was significantly better than feedback alone.
Electronic charting is a feature of EHR that has been reported to affect provider productivity.62–65 These studies found that the time needed for development of care plans and documentation initially increased, but preparation time decreased subsequently. The initial loss of productivity was associated with the baseline computer skills of the users (clinicians). Two studies assessed computerized documentation systems used for the ICU.66, 67 The authors of the first study asserted that addition of their computer-based nursing documentation required no specific ICU software or bedside workstations because it was implemented in a well-networked information technology environment.66 Thus, they found that compared with paper charting, their electronic charting system was relatively inexpensive (although the figures were not provided). In addition, the documentation was more complete because of the presence of reminders for missing entries, and data quality remarkably improved. The other study used work sampling and cost analysis methodologies to show net savings of a vendor-developed bedside documentation system specifically designed for an ICU.67
Another potential benefit of EHR systems is avoidance of morbidity because of improved patient safety. One study of ADEs found that building ADE detection and reporting capability into EHR can improve detection and potential reduction of ADE in hospital settings because of the ability of the EHR system to easily identify and confirm patients experiencing ADEs and thus enabled early intervention.68 Several studies have shown that severe ADEs were associated with longer hospitalizations and higher hospitalization costs (over $2,000 in 2005 dollars).68–70
Several studies investigated the impact of point-of-care alerts and reminders imbedded within an EHR system during the process of documentation or entry of orders into the system.45, 47, 71–76 These decision-support functions within an EHR system, when accompanied by the required changes in process and communication, altered physicians' ordering behaviors by facilitating appropriate resource utilization and reducing unnecessary charges.45, 47, 71–76 For example, one study estimated that reducing the ordering of redundant clinical laboratory tests could produce an annual savings of $35,000 in laboratory charges.72 A randomized controlled trial that tested the effect of immediately printed summaries of a computerized medical record on physician test ordering rates in an emergency room setting77 showed significant improvement in the cost-effectiveness of internists' ordering behaviors. The impact on surgeons' ordering patterns was positive but not significant.
Some EHR systems offer sophisticated CPOE and decision-support functionality. A randomized controlled trial found positive effects of an EHR with integrated CPOE on resource utilization, provider productivity, and care efficiency.78 Two additional studies showed that an EHR with integrated decision support helped providers improve the quality of documentation, clinical decisionmaking, and guideline compliance, and resulted in reduced utilization of services and costs of care.76, 79 However, another study found that implementation of clinical guidelines via an EHR had no significant effect on clinical outcomes or healthcare costs.30 The benefits of information technology seem to depend greatly on the quality of the implementation and the level and type of decision-support technology.
| Reference | HIT elements | IOM categories | Organization | Known system | Sources of data | Cost variables | Benefit variables | Year(s) of study | Findings |
|---|---|---|---|---|---|---|---|---|---|
| Wang, 200352 | EHR, Electronic prescribing | HIDS, RM, OEM, DS, AP | Outpatient | Part | Primary data from EMR, literature review, expert group consensus | Software license, implementation, support, hardware, productivity loss | Quantified: Chart pull, transcription, prevention of ADE, drug, lab, radiology, charge capture, billing, & net value of AEMR during a five-year period | 2002 | Predicted to reduce healthcare costs, improve efficiency and productivity, and outweigh the costs from year 2 of the EMR implementation |
| Arias-Vimarlund, 199680 | EHR | HIDS | Outpatient | Work place observations and key informant interviews | Direct: Training, hardware & software, project manager system supplier, maintenance; | Quantified: time saved from chart pulling; | 1996 | One-year comparative case studies showed improved quality of care but the quantifiable benefits were less than HIT costs from societal perspective | |
| Unexpected: self-training at work, loss of leisure hours, increase in administrative workload, extra service, summarizing medical records; | Unquantified: increase in knowledge capital for care team, easier and quicker communication for GPs during telephone consultations, clearer information to patients | ||||||||
| Indirect: unquantified | |||||||||
| Agrawal, 200283 | EHR | HIDS, AP | Outpatient | Part | Hypothetical scenario, published studies | Unquantified: hardware, software, network, maintenance, installation and training, opportunity cost | Quantified: chart pull, transcription, time to document diagnostic codes; | 2002 | Illustrated a framework for return-on-investment calculation for EHR systems |
| Unquantified: charge capture, cash flow, prescribing, malpractice premium, health resource waste, ADE and injuries, delivery of preventive and health maintenance procedures, provider and patient satisfaction | |||||||||
| Khaoury, 199853 | EHR, Decision Support | HIDS, RM, DS, AP | Outpatient | KP | Not described | Personnel, printing, network, hardware, license | Quantified: Medical record room and support staff, automated clinical forms, automated billing data; | 1992-1997 | Expected substantial savings |
| Unquantified: support of a quality program | |||||||||
| Schmitt, 200281 | EHR | HIDS, OEM, AP | IDN | Opinion from clinical advisory team, vendor, site visit observation | Annual cost over a seven-year period, with no detail breakdowns and assuming no cost associated with the temporary reduction in physician productivity | ADE, capitated drug benefits, inpatient medication cost, laboratory, radiology, pharmacy, documentation, information at the point of care, charge capture, & net benefit over the 7 years period | 2000 | Predicted to reduce healthcare costs, improve efficiency and productivity, see benefit in Year 2, and outweigh the costs from year 3 of the EMR implementation | |
| Valancy, 200284 | EHR | N/A | Outpatient | Illustrative | 5-year annual cost with detail on hardware, software, vendor support, and data entry cost | Not reported | 2002 | Presented a tool to estimate the costs of implementing an EMR system, no organizational adaptation cost | |
| Khoury, 199754 | EHR | HIDS, DS, AP | Outpatient | KP | Not described | Personnel, printing, network, hardware, license | Quantified: Medical record room and support staff, automated clinical forms, automated billing data; | 1997 | Predicted the system to pay for itself 13 years from its initiation |
| Unquantified: patient communications, guideline compliance, disease management and prevention | |||||||||
| Kian, 199555 | EHR | HIDS, AP | IDN | Literature review, surveying 10 institutions, interviewing vendors, process mapping sessions | 10 years annual cost including hard and software, integration, network infrastructure, vertical integration, I/S infrastructure, institutional issues | Quantified: Data capture and access, decision support, business management, streamlining patient flow; net impact over 10 years period | 1994 | Predicted to begin to see benefits in the 3rd year after implementation and substantial savings over 10 years | |
| Unquantified: patient readmission, staff reduction | |||||||||
| Walker, 200582 | EHR and health care information exchange and interoperability | DS | Organizations, nationwide | Literature review, published data, expert interviews, expert panel | Interface development, electronic health record acquisition, maintenance | Quantified: Lab, radiology, pharmacy, chart request and referral, reporting, provider-payer transaction, & net value of health care information exchange and interoperability | 2005 | Predicted fully standardized and implemented health care information exchange and interoperability could yield a net value of $77.8 billion per year; nonstandardized health care information exchange and interoperability is less but still positive | |
NOTE: HIDS = health information and data storage; RM = results management; OEM = order entry management; DS = decision support; electronic communication and connectivity; AP = administrative processes
To assess the value of EHR, one methodological paper described a return-on-investment framework to evaluate the costs and benefits of implementing an ambulatory EHR system, and another presented a spreadsheet tool to help family physicians estimate the costs of implementing an EHR system. Both reports provided illustrative examples. Of the remaining six studies, five used cost-benefit analysis52–55, 81, 82 and one used cost-consequence analysis.80 All the cost-benefit analyses adopted the ROI framework, which assesses the difference between the costs of an EHR investment and the benefits reaped from it. In these studies, both costs and benefits are quantified in monetary terms to the extent the authors determined was feasible. For example, one study aimed to justify the cost of EHR by first identifying the goals of the system in order to determine benefits.55 To quantify benefits, the authors then performed an extensive literature review, surveyed other institutions that were implementing EHR, interviewed EHR vendors, and conducted process-mapping sessions to identify potential cost savings on work processes affected by EHR. However, cost estimation was based largely on vendor response to a request for information or on current information system costs in the healthcare organization.52, 80 The cost-consequence study showed costs in monetary terms but did not quantify the benefits of EHR except for time saved from chart pulling.80 All studies except two used the perspective of an organization, either outpatient settings52–54, 83, 84 or IDN.55, 81 Of the other two studies, one adopted a societal perspective, 80 and the other used multiple perspectives, from organization level to national level.82
Six studies reported their data sources.52, 55, 80–83 All used multiple data sources, including primary data collected from an existing EHR system, 52 published data, 52, 55, 82, 83 workplace and demonstration site observations, 80, 81 and surveys or interviews of key informants or EHR users. 55, 80 Experts were a primary source of data for some studies, 52, 81, 82 as were vendors.55, 81 One study also used process-mapping sessions.55
The benefit variables included the following: savings from chart pull and transcription;52, 80–83 time saved to document diagnostic codes;81, 83 prevention of ADEs;52 reduction in drug52, 81 laboratory, 52, 81 or radiology costs;52, 81 improvement of charge capture;52, 81 decreased billing errors;52 personnel and space savings from reduced existing and future medical record storage requirements;53, 54 as well as automated generation of clinical forms, 53, 54 pharmacy information, 81 and billing data generation.53, 54 One study provided a comprehensive list of potential benefits grouped into four categories: data capture and access, decision support, business management, and streamlining patient flow.55 The health care information exchange and interoperability study reported quantifiable benefits from health information connectivity for providers, payers, and other stakeholders, including radiology centers, laboratories, pharmacies, and public health departments.82 Three studies showed the net ROI.55, 82
Various analytic designs were used, although the reference strategy was always the traditional paper-based medical record. One study assessed the economic value of an EHR system concurrently with its implementation and reported the first-year economic consequences of the system.80 Another study predicted a fixed annual savings and expenses based on seven years of EHR implementation experiences in a HMO.53, 54 Yet another study constructed a hypothetical primary care provider patient panel using average statistics from one of the nation's leading organizations in EHR implementation.52 Both studies of EHR systems in IDN settings are analytic predictions because EHR had not been implemented in the studied organizations.55, 81 The health care information exchange and interoperability study for a fully standardized nationwide system is also an analytic prediction based on a conceptual framework describing how health care entities share information and a functional taxonomy reflecting the amount of human involvement required, the sophistication of IT, and the level of standardization.82 Sensitivity analyses were performed in three studies.52, 80, 82
Costs of Implementing an EHR System. Five studies quantitatively assessed the costs of implementing an EHR system.52–55, 80, 81 The costs can be divided into two categories: (1) cost of the system itself (hardware, software, license, maintenance, and support) and (2) implementation cost (training, temporary loss of productivity, etc.). The costs vary significantly by the scale of the healthcare organization and the functionality of the EHR system.
(1) Cost of an EHR System. One study estimated the system costs for an ambulatory EHR to be $9,700 per provider (in 2002 dollars), which included $1,600 for the annual software license, $1,500 for annual support and maintenance, and $6,600 for hardware (three computers and network, refreshed every three years).52 The estimate was for a hypothetical primary care provider office and was modeled after a well-developed and widely used EHR system at a leading IDN health care system. The component parts of the EHR system include online patient charts, electronic prescribing, laboratory order entry, radiology order entry, and electronic charge capture.
In a Swedish primary health care setting with 50 staff, the system cost of a vendor-developed EHR system was estimated to be $240,000 in the first year (in 1995 U.S. dollars).80 The EHR system supported full-text patient records and included a controlled medical terminology, a structured patient database, and tools for the analysis and reporting of patient data. The hardware at the sites comprised one server supporting approximately 40 workstations and 20 printers.
In a large HMO with 13 outpatient care locations in Ohio, a homegrown ambulatory EHR was estimated to have had a system development cost of $10 million (in 1996 dollars) and additional annual expenses of $630,000 (in 1996 dollars) for printing, network expenses, memory, and license renewals.53, 54 The EHR system was used routinely by 220 physicians and 110 allied health professionals. It implemented an encounter system that collected and presented such medical data elements as diagnoses, allergies, medications prescribed, immunizations, vital signs, and smoking status at the time of an encounter. The system also generated physician reminders for guideline compliance and patient reminders for preventive services and was linked to centralized clinical data on the mainframe, such as laboratory results, radiology reports, emergency department notes, and hospital discharge summaries.
An academic cancer center with a staff of about 8000 and facilities that included a hospital, outpatient clinics, and remote patient-care sites was interested in implementing an EHR for its IDN as both a clinical and financial information management tool in 1994. The cost estimates for vendor-developed EHR systems to meet the center's organizational needs ranged from $15.8 million to $21 million (in 1994 dollars), which included costs of hardware, software, interface development, network cost, data conversion, training, and annual maintenance.55 Additionally, the annual support costs were estimated to range from $3.8 million to $5.3 million.
A 2002 study estimated the costs of implementing an EHR81 for an IDN that included a medical center with a 280-bed acute care hospital, 16 hospital-based and satellite outpatient clinics, a research institute, and a network of about 400 employed physicians. A vendor-developed EHR was estimated to cost approximately $19 million (in 2001 dollars) for the seven-year implementation period. This included costs for the various software products, server hardware, professional services related to installation and training, as well as desktop devices, monitors, biometric security devices, imaging hardware and software, additional technical-support staff, and other associated costs.
(2) Cost of the Implementation Process. Only two studies provided an estimate of costs associated with the EHR implementation process. Both were for ambulatory settings.52, 80 One estimated an implementation cost of $3,400 per provider (in 2002 dollars) in the first year associated with workflow process redesign, training, and historical paper chart abstracting.52 It also estimated a revenue loss of $11,200 in the first year due to temporary loss of productivity. The total implementation process cost, $14,600 per provider, is 1.5 times the estimated EHR system cost.
The Swedish study used a societal perspective and included costs of training and unexpected costs pertained to self-training during working hours, loss of normal activities in leisure hours, increase in administrative work load, extra service, and medical records summary.80 These costs were estimated at $75,000 (in 1995 U.S. dollars), approximately 30 percent of the EHR system cost.
A third study projected the costs of implementing health care information exchange and interoperability where EHR is a requirement for Levels 3 and 4 implementation, 82 which, according to the authors' taxonomy, refers to the ability to handle machine-organizable data and machine-interpretable data, respectively. The authors projected costs for multiple stakeholders of the healthcare system for Levels 3 and 4 health care information exchange and interoperability. The national ten-year rollout cost of Levels 3 and 4 were estimated to be $320 billion and $276 billion, respectively. Additionally, the national ten-year annual costs were estimated at $20.2 billion and $16.5 billion, respectively.
Quantified Benefits from an EHR System. Benefits of an EHR system or health care information exchange and interoperability were also quantified in the six studies that assessed the cost of implementing such a system.
One study was based on the Partners HealthCare ambulatory EHR, which not only provided health information and data storage capability but also possessed results management, order entry management, point-of-care decision support, and administrative information management functionalities. Therefore, many benefits were expected. The study divided the benefits into three categories: (1) payer-independent benefits, including savings from chart pulls and transcription; (2) benefits under capitated reimbursement, including averted costs from ADEs, drug utilization, laboratory utilization, and radiology utilization; and (3) benefits under fee-for-service reimbursement, including improvement in charge capture and decreased billing errors.52 The authors predicted that savings from chart pulls and transcription would be seen immediately after the EHR implementation, and costs associated with ADEs and drug utilization could be averted from second year on, but other potential savings would not be realized until the fourth year. Five-year total benefits of an EHR implementation were estimated to be $129,300 per provider (in 2002 dollars), or a net savings of $86,400 per provider (in 2002 dollars). Sensitivity analyses showed that the estimates were sensitive to the assumption of the proportion of patients whose care was capitated. The net financial value could range from a $2,300 net cost to a $330,900 net benefit per provider.
The Swedish study examined an EHR system with functionality limited only to health information and data storage. Therefore, the expected benefits were limited and included increase in knowledge capital for the primary health care team, easier and quicker communication for general practitioners during telephone consultations, clearer information to patients, and time saved in retrieving paper-based medical records.80 Only the value of the last item was estimated, at a total of approximately $10,500 (in U.S. 1995 dollars) for the first year of EHR implementation.
Despite potential savings from ADE prevention and reduced resource utilization under capitated reimbursement, the study of the ambulatory EHR system in a HMO (13 outpatient clinics) did not quantify this aspect of benefits. Instead, it quantified only the averted costs associated with improved efficiency.53, 54 The study estimated an annual savings of $3,700,000 (in 1996 dollars) from reduced medical record room and support staff, elimination of clinical forms, and automatic collection of billing data.
The cancer center study projected the benefits of an EHR over ten years.55 This projection made several key assumptions, including no benefit until the third year after implementation, benefits to phase-in as the EHR system became functional, physician acceptance and use of the system, a link between business management benefits and managed care, and productivity changes. The authors divided the benefits into capture and access, decision support, optimization of clinical practice, business management, and streamlining of patient flow. The estimated total quantified benefits were $129.69 million over ten years. Adjusted by the total implementation and system costs, the authors' assigned confidence factor, and 9.5-percent discount rate, the net value was predicted to be $24.9 million (in 1994 dollars).
The other IDN expected even greater benefits from an EHR implementation. The authors predicted approximately $68.5 million in gross quantifiable benefits over a seven-year period; subtracting the cost of the EHR system implementation, the net benefit would be $31.4 million (in 2001 dollars), using a 10-percent discount rate.81 The authors predicted and quantified benefits from savings in laboratory and radiology order entry, pharmacy order entry, documentation availability of information, and charge capture.
Another study estimated that implementation of a standardized interoperable EHR system by all healthcare organizations in the United States would yield substantial financial benefits. The health care information exchange and interoperability study predicted that investment on a fully standardized, Level-4 nationwide system will have the most financial return, a net value of $77.8 billion per year once fully implemented.82 Non-standardized health care information exchange and interoperability also can have positive financial returns, but the returns are smaller compared to the Level-4 implementation.
In summary, despite the heterogeneity in the analytic methods used, all five cost-benefit analyses predicted substantial savings from EHR (and health care information exchange and interoperability) implementation.52–55, 81, 82 In other words, the quantifiable benefits are projected to outweigh the investment costs. However, the predicted time needed to break even varied from three52, 81 to six55 to perhaps as long as 13 years.54
Conclusion. Our evidence review found consistent predictions from five cost-benefit studies that implementation of an EHR system can be financially viable at the individual organization level or through a nationwide implementation with high levels of health care information exchange and interoperability.52–55, 81, 82 However, there are several caveats.
All studies are predictive analyses that are based on many analytical assumptions and limited empirical data. The strength of evidence is considered weak.
In all studies, the EHR system was assumed to have multiple functionalities that include, at minimum, health information and data storage, administrative processes, decision support, and results management, as well as information exchange capabilities. The functional capability of an EHR system is critical to the benefit accrued.
The individual organizations that were the subjects of four studies were all large organizations. Large organizations involve many people, units, and subsystems and have complicated processes and interactions. They can benefit greatly from automated, transparent information processing through HIT, and substantial economies of scale. The literature review did not identify cost-benefit studies for EHR implementation in small organizations.
The costs of implementing an EHR system may be underestimated. Only one of the five cost-benefit analyses included the cost of the implementation process, 52 and it found that this cost was 1.5 times the cost of the EHR system. Implementing an EHR system requires extensive changes in the organizational processes, individual behaviors, and the interactions between the two. These resulting costs are often omitted or not reported from studies but can be substantial.
The financial benefits depend on the financing system. As shown in the sensitivity analysis of one study, 52 the benefit estimates are most sensitive to the assumption of the proportion of capitated patients. Realizing all quantifiable benefits of EHR implementation would require changes to the current health care financing system.
Both the cost and the benefit of attaining interoperability among EHR systems are directly proportional to the level of data exchange achieved. For example, the cost of achieving machine-organizable (Level 3) or machine-interpretable (Level 4) interoperability is greatest, but it offers the most potential for increased efficiency, improved healthcare utilization, and reduced costs.
In conclusion, there is some empirical evidence to support the positive economic value of an EHR system and the component parts of EHRs. However, realizing the projected benefits will require proper alignment of the healthcare financing system, strong leadership, effective implementation strategies, and focused efforts to successfully adapt the EHR system.
Many advocates of HIT believe that one of its primary goals is to increase the extent to which the patient is at the center of his or her health care. For this report, studies of HIT and “patient centeredness” were defined as those that assessed HIT systems that included the element of patient decision support/consumer health informatics, telemedicine, or data-exchange/community health information networks or that reported patient satisfaction as an outcome. From the database of 256 articles there were 34 unique studies or systematic reviews meeting these criteria.
Ten studies assessed computer-generated reminders. Of these, seven assessed the use of reminder programs to improve the delivery of preventive care such as mammography and immunizations.27, 85–90 All studies reported greater use of preventive services by patients—or the physicians of patients—who received computer-generated reminders. Two other studies assessed the effect of computer-supported or -generated reminder systems for refilling medications, 91, 92 neither of which reported statistically significant improvements in compliance. A third study assessed the effect of a computer-generated reminder chart on patients' compliance with drug regimens, which reported significant improvements in mean compliance score for patients receiving an automatically generated reminder chart.93
Seven original studies evaluated various aspects of telemedicine, and a review article assessed the role of telemedicine in surgery.94 In the context of surgery, telemedicine included tele-mentoring, tele-proctoring, tele-conferencing, and tele-presence surgery, all of which are designed to allow physicians to communicate and improve the technical delivery of remote surgical procedures. Thus, this article was not considered relevant to patient centeredness. Five studies assessed telemedicine in particular contexts, including the intensive care unit, 95 the control of essential hypertension, 96 the evaluation of patients in an outpatient pulmonary clinic, 97 the role of telemedicine as one component of HIT and its importance to child safety, 98 and the use of a remote video system that allowed nurses and patients to interact in real time for patients with a variety of health conditions.99 All of these articles reported benefits from the use of telemedicine technologies. The last article in this group was an assessment of ComputerLink, which was conceived as an alternative to traditional caregiver support services such as support groups and health education programs. It was tested in a 12-month randomized trial in family caregivers of people with Alzheimer's disease. Compared with the control group that did not have access to ComputerLink, caregivers in the experimental group reported more improvement in caregiver strain.100
Three studies assessed the effect on patient trust and satisfaction of some aspects of HIT that are used during a patient consultation. In a pre-post study of general surgical patients, 96 percent of patients stated that their contact with a doctor was as easy and as personal after installation of a computer in the office as before the installation.101 In a controlled trial, the use of an automatic voice recognition system for transcribing progress notes was not associated with any significant negative effect on patient satisfaction, and was associated with some positive effects in terms of preventive maintenance and patient education.102 We also identified a study that assessed the effects of a nursing module used at the point of care. In this time-series study, an integrated, menu-driven electronic health record was associated with marked increases in nursing documentation and no effect on patient satisfaction.103 One study assessed physician satisfaction, and was excluded.104
Two studies assessed the effect of computer-guided management of patients. The first study compared the effect of computer-controlled administration of analgesic for post-operative pain to that of patient-controlled administration. The computer-controlled infusion used custom-written software designed to rapidly attain and maintain a theoretical target plasma concentration of the analgesic. In a double-blind randomized trial, the study found that computer-controlled analgesia conferred a more rapid onset of pain relief and was as effective as patient-controlled administration in providing post-operative analgesia.105 Which of these interventions is more “patient-centric” may be debatable, however. In the other study, computer-guided behavioral therapy that allowed patients to progress through a self-paced workbook was compared with clinician-directed behavioral therapy and with relaxation therapy in a randomized trial of patients with obsessive-compulsive disorder. At ten weeks, the Yale-Brown obsessive-compulsive scale showed significantly greater improvement in the patients receiving clinician-guided behavior therapy than in the group receiving computer-guided behavior therapy, and both of these were significantly greater than the improvement attained with relaxation therapy, which was found to be essentially ineffective. This study concluded that computer-guided behavior therapy was effective and might be a helpful first step in treating patients with obsessive-compulsive disorder when clinical-guided behavior therapy was unavailable.106
Five studies assessed various aspects of consumer health informatics. The interventions included a clinical trial of an interactive computerized patient education system in family practice;107 and assessments of the effects of computer tailored smoking cessation in family practice, 108 the effectiveness of a computer-generated patient health summary in changing patients' knowledge, attitudes, and behavior concerning health promotion, 109 and the use of self-administered computerized assessments for psychiatric disorders in patients in primary care.110 All of these studies reported benefits of the computerized health informatics system. A review of 37 studies of computer-generated health behavior interventions intended to motivate individuals to adopt various treatment regimens concluded that such systems are effective.111
Two review articles assessed the effect of various HIT systems that are directly accessed by the consumer or patient. One of the reviews assessed ten comparative studies of consumers using the Internet to access health information and services. This review included controlled studies, before-and-after studies, and interrupted time-series analyses of Internet users versus nonusers, or of the use of the Internet versus other communication media. The authors concluded that rigorous research regarding the effects of consumer Internet use on health outcomes is lacking. In the ten studies they assessed, all showed some beneficial effects on health outcomes, although the authors note the methodological quality of many studies was poor.112 A second review article assessed comparative studies evaluating the health or social outcomes of virtual peer-to-peer communities, which they characterized as a type of electronic support group. Among 45 publications describing 38 studies (of which 20 were randomized trials), only six evaluated “pure” peer-to-peer communities without other interventions. The other studies assessed complex interventions, of which a peer-to-peer community was only one component. The authors concluded that no good evidence exists on the effects of consumer-led peer-to-peer communities, partly because most such interventions have been evaluated only as part of a complex intervention or interventions involving health professionals.113
One review article assessed what is known about email consultations in healthcare. The authors noted that a rapidly expanding proportion of the population has access to email and that, while email consultations have the potential to play an important role in the delivery of preventive health care and facilitation of self-management of chronic disorders, there is little evidence from controlled trials that this potential benefit can be translated to routine clinical care.114
Another review article assessed studies of “electronic communication with patients,” which was defined to include studies of computerized communication, telephone follow-up and counseling, telephone reminders, interactive telephone systems, telephone access, and telephone screening. This article concluded that distance medicine technology has benefits in the areas of preventive care, management of osteoarthritis, cardiac rehabilitation, and diabetes care and that “distance medicine technology enables greater continuity of care by improving access and supporting the coordination of activities by a clinician.”115
Last, two studies dealt with data exchange networks or community health information networks. One study described the experience of developers of an electronic laboratory reporting system.116 In the second study, researchers developed a cost-benefit model and used published evidence and expert opinion to assess the ten-year rollout and annual cost of healthcare information exchange and interoperability, a development that would allow providers to access patient health care information in any clinical setting. The researchers concluded that a fully standardized interoperable system could save $77.8 billion a year, once fully implemented.82
In summary, evidence for an effect of HIT on patient-centeredness in health care is sparse. The best evidence is the beneficial effect of using computerized patient reminders for preventive care. The evidence for benefits of telemedicine and consumer health informatics is also limited to specific contexts. Finally, the evidence is much more limited for effects of more general interactive HIT (such as the internet or email on health) or the effect of implementing HIT systems (such as the electronic health record) on patient trust and satisfaction.
All studies initially reviewed were screened for data on barriers to adoption and implementation. For this analysis, qualitative studies that were primarily focused on barriers and studies that collected quantitative data on barriers were included. Studies in which barriers were briefly discussed, but were not a primary focus, were excluded. A primary focus on barriers was identified through reviewer consensus.
We identified 20 publications that focused on the barriers to implementing HIT. Of these, 8 reported the actual or potential barriers encountered with specific HIT implementations, 55, 62, 116–124 usually as part of an article discussing the implementation. Two articles were short opinion pieces about potential barriers from the physician perspective.125, 126 Two studies assessed the physician time for order entry using CPOE compared to paper methods;118, 127 both demonstrated that CPOE took more physician time, although the study by Overhage and colleagues found this additional time to be modest. A third study assessed the effect on primary care physicians' time before and after implementation of an EHR system and reported that the time for a patient visit actually fell by half a minute with EHR use.128 Last, one study compared physician user satisfaction with two HIT systems: the VA CPRS system and the Mt. Sinai hospital physician order entry system. This study demonstrated CPRS users to be much more satisfied than Mt. Sinai hospital users on many dimensions and also demonstrated that satisfaction was correlated most strongly with the ability of the HIT system to perform tasks in a “straightforward” manner.117 Finally, one article was a systematic review of physician use of electronic retrieval systems such as Medline.129
The other five articles focused more broadly on barriers to HIT implementation. One systematic review130 summarized barriers mentioned in the medical and pediatric literature that are significant for pediatric practices. These barriers were divided into four categories. Situational barriers included time and financial pressures, unproven return on investment, insufficient access to the internet or to computer technology in the office setting, the prohibitive cost of information technology for small practices, and software not being supportive of pediatric practice needs. Cognitive and or physical barriers include physical disabilities and insufficient computer skills. Liability barriers included confidentiality concerns. Finally, knowledge and attitudinal barriers included insufficient research about information technology in pediatrics, insufficient knowledge about benefits afforded by information technology, apprehension about change, and philosophical opposition to information technology.
Two studies used surveys to identify barriers in the use of electronic medical records131 and barriers to implementing CPOE systems in U.S. hospitals.132 In the first of these studies, the authors conducted 90 interviews with electronic medical record managers and physician champions in 30 physicians' organizations between 2000 and 2002. Key barriers to electronic medical record use were high initial financial costs, slow and uncertain financial payoffs, and high initial costs in terms of physician time. Additional barriers included difficulties with technology, complementary changes in support, electronic data exchange, financial incentives, and physician attitudes. The authors note that these barriers were most acute for physicians in solo/small group practice, which account for a large proportion of U.S. physicians. The second article132 reported the results of 52 interviews at 26 hospitals in various stages of implementation of CPOE—from not considering implementation to fully implemented. Most respondents were Chief Information Officers; the remainder consisted of Chief Financial Officers, Chief Medical Officers, and other management officials. Three main barriers to CPOE adoption were identified. The first was physician and organizational resistance due to the perceived negative impact on the physician's workflow. The authors noted that resistance from physicians could escalate to the point of a “physician rebellion,” which could derail the entire implementation process. The second barrier identified was the high cost, with estimates from prior studies for the cost of CPOE ranging from $3 million to $10 million, depending on the hospital's size and the level of existing information technology infrastructure. The third major barrier identified was product/vendor immaturity. Survey respondents reported that many current vendor products did not fit the needs of their hospital, and extensive software modifications were required to accommodate established workflow in the hospital.
We also identified two recent prominent editorials about barriers to HIT implementation that summarized the issues succinctly.133, 134 The first of these133 identified several challenges for adoption of electronic health records. These included cost, technical issues, system interoperability, concerns about privacy and confidentiality, and a lack of a well trained clinical informatics work force to lead the process. This author identified financing as the biggest impediment, which he attributed to a misalignment of costs and benefits. He noted that while some studies have suggested a substantially positive return on HIT investment for the health care system as a whole, those who are expected to pay for the systems (physicians and other practice organizations) see only about 11 percent of that return on investment. The rest of the savings go to those who typically do not pay directly for the electronic health record. Another major challenge he identified was system and data interoperability, noting that most health care data (whether on paper or electronic) are trapped in “silos.” A third concern was privacy and confidentiality: the author stated that physicians, other health care professionals, and healthcare organizations must be vigilant in protecting patient privacy. The last major barrier identified was the need for a workforce capable of leading the implementation of information technology.
The second editorial134 stated that, despite predictions of a “bright and near future” for the use of HIT, this future never seems to be realized. The authors attributed the lack of progress in HIT implementation to a lack of attention to the social component, citing the need to view the clinical workplace as a complex system in which technologies, people, and organizational routines dynamically interact, which leads to the following observations:
“(1) Organizations are simultaneously social (e.g. consisting of people, values, norms and culture) and technical (i.e., without tools, equipment, procedures, technology and facilities the people could not work and the organization would not exist). (2) These social and technical elements are deeply inter-dependent and inter-related—hence the term socio-technical systems. Every change in one element affects the other. (3) Accordingly, good design and implementation is not a technical problem but rather one of jointly optimizing the combined socio-technical system.”
The authors also note, “...an information technology in and of itself cannot do anything, and when the patterns of its use are not tailored to the workers and their environment to yield high quality care, the technological interventions will not be productive. This implies that any IT acquisitions or implementation trajectory should, first and foremost, be an organization change trajectory.”
In summary, studies have identified a large number of barriers to the implementation of HIT. These barriers can be classified as situational barriers (including time and financial concerns), cognitive and or physical barriers (include physical disabilities and insufficient computer skills), liability barriers (including confidentiality concerns), and knowledge and attitudinal barriers. Cutting across all these categories, however, may be the need for clinical medicine as it is now practiced in the majority of settings to undergo a major structural and ideological reorganization, so it can be integrated with and enjoy the benefits of HIT.
The primary limitation of this review is the quality and quantity of the available studies. As we have noted, understanding the benefits and costs of implementing a HIT system requires an understanding of the intervention in terms of its technical components as well as its human factors, project management, and organizational component; and understanding the organization implementing the HIT system requires understanding many things about it, including its past and current culture of change and the economic situation in which it operates. Most of this information is absent for most of the published studies of HIT. Past limits on word count in published articles may have prevented some authors from including such information in their published reports, but recognition of the information that is needed and the recent practice of allowing supplementary methodological information to be posted online should obviate the problem.
A second limitation is that not all relevant published studies may have been identified. While our search efforts were comprehensive, it is inevitable that we did not find some relevant studies. An advantage of our interactive database of evidence (http://healthit.ahrq.gov/tools/rand) is that it can be updated easily, so we invite readers to send us the citations for relevant articles we may have missed.
We selected only articles that were classified as systematic reviews, hypothesis testing, or predictive analysis for more detailed review and structured abstracts in our interactive database. These articles tend to have less description about how the HIT actually operated and its implementation processes than do qualitative, descriptive articles. Although in general we did not find good evidence of such critical information in the literature during our review processes, we provide citations of qualitative articles in our interactive database for interested readers. However, it should be noted that while these qualitative articles might contain more contextual information about the HIT systems, they are completely lacking in any generalizable knowledge about the benefits of HIT, such as reduction in errors or quality improvement. Any such studies that compared outcomes (such as error rates) with and without a HIT system would have been classified as hypothesis-testing studies and would have been included in our analyses.
A third limitation is that we considered only published studies. As noted by our TEP, substantially more information regarding implementation may have been obtained by contacting leading HIT implementers and conducting structured interviews with them. An additional limitation of reliance on published studies is that less successful, or failed, HIT attempts may be less likely to be published than successful ones (a result of publication bias).
A final limitation is that many of the costs and financial benefits of EHR will change over the years, because they depend on the changing price of such factors as hardware, software, labor costs, and medical prices. Consequently, it is not easy to translate costs as they were reported in the original article into today's costs.
Predictive analyses, based on statistical modeling techniques, suggest that HIT has the potential to enable a dramatic transformation in the delivery of health care, making it safer, more effective, and more efficient. The empirical research evidence base supporting HIT benefits is more limited.
Organizations that have realized some of these major gains through the implementation of multi-functional, interoperable HIT systems built around an electronic health record include the VA, Partners, and Regenstrief Institute.
The impact of HIT implementation on cost and quality will not be consistent across institutions, independent of context. The specific context within which HIT is implemented, including the setting, the clinical issues, and the patient populations, greatly influences its use and effects.
More widespread implementation of HIT is limited by the lack of generalizable knowledge about what types of HIT and methods for its implementation will result in changes in benefits and costs that are specific for specific health organizations, especially for small practices and small hospitals.
The reporting of HIT developments and implementations needs to be improved with greater attention to descriptions of both the intervention and the organizational/economic environment in which it is implemented.
A high priority must be placed on establishing standards for the information that needs to be measured and reported in HIT implementation studies, similar to the CONSORT standards for clinical trials of therapeutics.
Using existing published evidence, it is not possible to draw firm conclusions about which HIT functionalities are most likely to achieve certain health benefits. The assessment of costs is even more uncertain.
Existing evidence is not sufficient to clearly define “who pays for” and “who benefits from” HIT implementation in any healthcare organization except those, such as Kaiser and the VA, that are responsible for paying for and delivering all the care for the defined population.
Models can be built to estimate the costs and benefits of interoperable HIT systems within and across health care provider settings, payers/purchasers and cumulatively across the health care continuum, but these models are based on many assumptions.
Implementation of HIT faces many barriers.
Limited empiric evidence exists to support a benefit for HIT use in pediatrics in the areas of medication safety, clinical decision-support, process improvement, and cost reduction.
Only one scientific study weighed these benefits against the costs or cost-effectiveness of implementing HIT systems in pediatric healthcare settings.
A majority of HIT systems for use in pediatric practices were tested and/or developed in academic settings, and the ability to generalize these findings to commercially available systems used in nonacademic settings is limited.
A small set of high quality studies shows that implementation of a comprehensive ambulatory EHR improves quality of care. Available evidence focuses primarily on the impact of ambulatory EHRs on decreasing overused health services by enhancing access to data, providing capabilities for real-time analysis of clinical data, and acting as platforms for decision support.
Ambulatory EHRs improve the structure of care delivery, improve clinical processes, and enhance outcomes. Most available evidence shows the effects of ambulatory EHRs on processes of care.
Interpreting the precise causal effects of ambulatory EHRs on quality is difficult due to lack of systematic and detailed descriptions of system capabilities, limited data (either qualitative or quantitative) on the workflow redesign and organizational changes that accompanied implementation of an ambulatory EHR (or implementation of a new function in an existing EHR package), use of ad hoc measures to assess quality, and use of study designs that do not explicitly take into account sources of bias and confounding. Thus, while existing evidence may have high internal validity, the generalizability of findings is limited.
Although substantial potential exists, evidence for the ability of ambulatory care EHRs to improve quality by making healthcare more consumer- and patient-centered is scant.
The main quantifiable benefits of an EHR system were savings from data capture and access; decision support to improve efficiency, quality, and safety of care; business management related to staffing, billing, and overheads; and streamlining patient flow.
Few studies quantitatively assessed the costs to implement an EHR system and the financial benefits reaped from it.
All the cost-benefit analyses of an EHR system predicted that the financial benefits would significantly outweigh the costs, in a timeframe that varied from three to thirteen years, but this evidence is limited to large organizations and multi-functional EHR systems.
The positive economic estimates for EHR system implementation are encouraging but are based on limited evidence at this time. Only limited empirical evidence supports the assumptions made in the predictive analyses. Most studies omitted the costs of implementing an EHR system that were associated with the temporary loss of productivity and the cost of process redesigns. Moreover, realization of the financial return is highly sensitive to the organization's financial incentives.
There is some evidence regarding the positive economic value of implementing component parts of an EHR system, with models suggesting that many of the benefits do not accrue unless a broadly functional system is implemented.
This section is divided into two parts: (1) research recommendations and (2) recommendations regarding the appropriate types of researchers to carry out the recommended work.
High on the list of future research is the need for agreed-upon standards for reporting HIT implementation studies, similar in purpose to the CONSORT standards for the reporting of clinical therapeutics trials.
The organizational change and workflow redesign required by and accompanying HIT implementation (or implementation of a new HIT function) need to be described and measured with greater validity, reliability, and precision in order to understand the impact of HIT on care delivery. Without such information, the true “intervention” remains unclear, and the generalizability of results will remain limited. This kind of reporting will require the development and dissemination of publishing standards.
While HIT implementation does not easily lend itself to randomized trials, better use of quasi-experimental study designs and other study designs of high internal validity could greatly enhance the clinical relevance of results, reduce bias and confounding, and increase the generalizability of findings. Currently, the published literature is dominated by simple pre-post implementation designs.
Creative, alternative research methodology should be considered to estimate costs and benefits of HIT as a supplement to traditional hypothesis-testing studies. Traditional experimental or quasi-experimental approaches may be impractical because they are expensive, time-consuming, and interfere with HIT implementation. Qualitative studies are often subjective, descriptive, and lack generalizability. Simulation modeling is a promising alternative to generate knowledge and evidence; it is different from analytical modeling where the result functionally depends on the input (a number of parameters). Simulation, or dynamic, modeling uses a set of rules that define how the system being modeled will change in the future, given its present state, existing knowledge, and foreseeable uncertainties. For complex problems like HIT implementation, where time dynamics is important and experimenting with the real system is expensive or impossible, simulation modeling can support estimates of cost, benefit, and net value of HIT systems.
The costs and benefits of HIT depend not only on the internal system (the practice environment) but also on the interactions with the external system, including consumers (patients and potential users of the healthcare system), medical service suppliers (laboratories, radiology centers, other healthcare organizations), technology suppliers, and the regulatory and financing systems an organization operates. Multi-perspective studies are needed to investigate the flow of costs and benefits in order to maximize the benefits of HIT in the larger healthcare delivery system. Again, simulation modeling may be the best methodology for this type of research.
The conceptual foundation for the impact of EHRs on improving care is strong. More research concerning the efficacy and effectiveness of EHRs across health care settings, providers, and patient populations needs to be carried out. Such research will require focusing on how EHR tools are implemented and utilized in day-to-day practice, a broadening of environments to include nonacademic/nonintegrated network practices, the development of methods and instruments directed at evaluation of externally developed systems, and a broader understanding of the human factors issues relevant to healthcare.
More research is needed on which specific components of an EHR are beneficial and also on evaluating new specific components—for example, clinical decision-support. Much of the existing decision support relies on simple rules, and it should be possible to provide substantially better assistance with the use of more-complex rules and models.
More research is needed to evaluate the effects of EHRs on improving quality by making care more consumer-centered.
Process and outcome benefits of HIT that are important and unique to pediatrics must be better quantified, given the unique workflow and information needs of pediatric organizations and practice settings. A growing body of epidemiologic studies has demonstrated the frequency of medication errors in the pediatric healthcare setting. Well-designed studies are needed to demonstrate empirically the benefit of HIT in improving patient safety, not only in the hospital environment, but also in ambulatory and other settings.
Well-designed studies measuring the costs of HIT implementation and resultant benefits in pediatrics and other vulnerable populations (e.g., chronically ill, disabled, etc.) are needed, especially in nonacademic settings and with commercially available HIT systems.
The assessment of HIT implementations of greatest relevance to most U.S. healthcare institutions will occur in nonacademic settings. Most nonacademic settings have limited research expertise or infrastructure to design and support a research project on HIT. If extramural funds are desired for an evaluation of HIT implementation, the ability to secure funding coincident with the project plan is difficult, if not impossible, especially given the funding cycle of grants. Also, to use a pre-and post-implementation design, the researcher needs funding for an extended period of time to collect enough data to adequately power the study before the HIT system is in place. For financial and pragmatic reasons, this pre-implementation data collection cannot delay the HIT implementation process.
Therefore, we would suggest that for HIT research to be feasible in nonacademic settings with commercial systems, some important steps should be taken:
Create incentives (e.g., matching funds) for nonacademic medical centers and provider organizations to perform high-quality evaluations of vendor-based HIT implementation. These projects should be funded by organizational dollars and support should be provided for academic investigators to partner with such organizations. These measures would help organizations that lack a built-in research infrastructure to conduct rigorous research.
Provide a number of extramural funding mechanisms (government, state, foundation, or even vendor) to evaluate HIT with limited-funding cycles, allowing for adequate pre-implementation measurements and/or rigorous study design. The investigators typically do not determine the timing of implementation, which is often delayed, and funders much be cognizant of this and not penalize the investigators, by disallowing no-cost extensions.
Devise a standard means to adequately assess and describe the “socio-technical” milieu of an organization relevant to HIT implementation.
| David Bates, MD, MSc | John Byrnes, MD |
| Chief, General Medicine | Chief Medical Officer |
| Brigham and Women's Hospital | Spectrum Health |
| 620 Tremont Street, Room BC3-002M | 100 Michigan Ave. NE |
| Boston, MA 02120-1613 | Grand Rapids, MI 49503 |
| 617-732-5650 | 616-391-2774 |
| 617-732-7072 (fax) | john.byrnes@spectrum-health.org |
| dbates@partners.org | |
| David Classen, MD | Simon Cohn, MD, MPH |
| Vice President | National Director for Health Information Policy |
| First Consulting Group | Kaiser Permanente Medical Care Program |
| 561 East Northmont Way | One Kaiser Plaza, 27L |
| Salt Lake City, UT 84103 | Oakland, CA 94612 |
| 801-532-3633 | 510-271-6461 |
| 801-596-3443 (fax) | 510-267-2194 (fax) |
| dclassen@fcg.com | simon.cohn@kp.org |
| Mary Jo Deering, PhD | Jennie Harvell |
| Director | Senior Policy Analyst |
| Health Communication and Telehealth Office of Disease Prevention and Health Promotion | U.S. Department of Health and Human Services |
| U.S. Department of Health and Human Services | Humphrey Building |
| 200 Independence Avenue SW | 200 Independence Avenue SW |
| Room 738-G | Washington, DC 20201 |
| Washington, DC 20201 | 202-690-6443 |
| 202-260-2652 | 202-401-7733 (fax) |
| 202-205-0463 (fax) | jennie.harvell@hhs.gov |
| mdeering@osophs.dhhs.gov | |
| Mary Jo Deering, Ph.D.* | |
| Director for Informatics Dissemination | |
| NCI Center for Bioinformatics | |
| National Cancer Institute | |
| National Institutes of Health, USDHHS | |
| 6116 Executive Blvd. - #400 | |
| Rockville, MD 20852 | |
| Phone: 301-594-1273 | |
| Fax: 301-480-3441 | |
| E-mail: deeringm@mail.nih.gov | |
| Robert Kolodner, MD | David Lansky, Ph.D. |
| Acting Deputy CIO for Healthcare Department of Veterans Affairs | Foundation for Accountability |
| 810 Vermont Avenue NW | 1200 NW Naito Parkway, Suite 470 |
| Washington, DC 20420 | Portland, OR 97209 |
| 202-273-8663 | 503-223-2228 |
| robkolodner@hq.med.va.gov | 503-223-4336 (fax) |
| dlansky@facct.org | |
| Janet M. Marchibroda | Blackford Middleton, MD, MPH, MSc |
| Chief Executive Officer eHealth Initiative | Clinical Informatics R&D Partners Healthcare |
| 1500 K Street, N.W., Suite 900 | 93 Worcester St., Box 81905 |
| Washington, D.C. 20005 | Wellesley, MA 02481 |
| 202-624-3270 | 781-416-8528 |
| 202-624-3263 (direct) | 781-416-2286 (fax) |
| Arnold Milstein, MD, MPH | Elliott Sternberg, MD |
| Principal and Consultant | Senior Vice President and Chief Medical Officer |
| William M. Mercer, Inc. | St. Joseph Health System |
| 3 Embarcadero Center | 500 S. Main Street, Suite 900 |
| Suite 1500 | Orange, CA 92868 |
| San Francisco, CA 94111 | 714-347-7841 |
| 415-743-8803 | esternbe@corp.stjoe.org |
| 415-743-8950 (fax) | |
| arnold.milstein@mercer.com | |
| Paul C. Tang, MD | Scott Weingarten, MD, MPH |
| Chief Medical Officer | Chief Medical Officer Zynx Health, Inc. |
| Palo Alto Medical Foundation | 9100 Wilshire Blvd., Suite 655-E |
| 795 El Camino Real | Beverly Hills, CA 90210 |
| Palo Alto, CA 94301 | 310-247-7700 |
| 650-853-5775 | 310-247-710 (fax) |
| 650-853-6050 (fax) | weingarten@zynx.com |
| tang@smi.stanford.edu | |
| Scott Young, M.D. | David Cutler, PhD |
| Director, Health Information Technology Programs and Research | Economist, |
| Agency for Healthcare Research and Quality | Department of Economics |
| 540 Gaither Road | Littauer Center |
| Rockville, MD 20850 | 1875 Cambridge Street |
| 301-427-1580 | Harvard University |
| 301-427-1595 (fax) | Cambridge, MA 02138 |
| syoung@ahrq.gov | |
| Marc Berg | Margaret Coopey RN, MGA, MPS |
| Prof. Dr. Marc Berg, MA, MD, PhD | Senior Health Policy Analyst Agency for Healthcare Research and Quality |
| Chair of Dept. of Social Medical Sciences Institute of Health Policy and Management | 540 Gaither Road |
| Erasmus University Rotterdam | Rockville, Maryland 20850 |
| P.O. Box 1738 | 301-427-1618 |
| 3000 DR Rotterdam | 301-427-1520 (fax) |
| The Netherlands | mcoopey@ahrq.gov |
| Phone: ++31-10-4088531 / 4088525 | |
| Fax: ++31-10-4089094 | |
| M.Berg@bmg.eur.nl | |
| David Atkins | |
| Agency for Healthcare Research and Quality | |
| 540 Gaither Road | |
| Rockville, Maryland 20850 | |
| 301-427-1618 | |
| 301-427-1520 (fax) | |
| datkins@ahrq.gov |
SEARCH METHODOLOGY - INFORMATION TECHNOLOGY + EFFECTS/OUTCOMES
DATABASE SEARCHED: PUBMED
YEARS OF COVERAGE: 1995-2003
OTHER LIMITERS:
English
SEARCH STRATEGY:
automatic data processing[majr] OR medical informatics[majr] OR medical informatics applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR information technolog* OR information infrastructure* OR ehealth OR e-health
AND
adverse effects[sh] OR outcome and process assessment health care[mh] OR costs and cost analysis OR efficiency, organizational OR risk assessment OR outcome*[ti] OR outcome*[ab] OR cost[ti] OR cost[ab] OR costs[ti] OR costs[ab] OR efficien*[ti] OR efficien*[ab] OR risk*[ti] OR risk*[ab] OR adverse[ti] OR adverse[ab]
AND
systematic[sb] - NOTE: This is the pre-formulated search developed by Medline to include systematic reviews, meta analyses, etc.
NUMBER OF ITEMS RETRIEVED: 516
================================================================
SEARCH METHODOLOGY - INFORMATION TECHNOLOGY + IMPROVEMENT
DATABASE SEARCHED: PUBMED
YEARS OF COVERAGE: 2002-2004
OTHER LIMITERS:
English
SEARCH STRATEGY:
automatic data processing[majr] OR medical informatics[majr] OR medical informatics applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR information technolog* OR information infrastructure* OR ehealth OR e-health
AND
outcome assessment health care OR process assessment health care OR workplace OR workflow* OR work flow* OR quality indicators, health care
AND
improve*[tiab] OR chang*[tiab]
NUMBER OF ITEMS RETRIEVED: 244
==================================================================
SEARCH METHODOLOGY - PRELIMINARY PUBMED SEARCH ON INFORMATION TECHNOLOGY BARRIERS
Database Searched and Time Period Covered:
PubMed - 2000-2003
Other Limiters:
English only
Search Strategy:
automatic data processing[majr] OR medical informatics[majr] OR medical informatics applications[majr] OR public health informatics[majr] OR electronics, medical[majr] OR attitude to computers OR information technolog*
AND
barrier* OR challeng* OR difficult* OR resist OR resisting OR resistance
NUMBER OF ITEMS RETRIEVED: 1662
===================================================================
INFORMATION TECHNOLOGY & HEALTH - POPULAR LITERATURE
Database: Periodical Abstracts
Query: ((de: information and de: technology) or (de: information and de: systems) or de: computer+ or de: digital or de: electronic or de: internet) and (de: health or de: healthcare or de: medical or de: medicine or de: physician+ or de: doctor+ or de: hospital+) and yr: 2002-2003
NUMBER OF ITEMS RETRIEVED: 433
==================================================================
SEARCH METHODOLOGY
Health Affairs Journal
SEARCH #1 (PERFORMED 12/23/04)
DATABASE SEARCHED AND TIME PERIOD COVERED:
PUBMED 1966-2004
SEARCH STRATEGY:
information technology
AND
information systems
NUMBER OF ITEMS RETRIEVED: 98
==================================================================
SEARCH METHODOLOGY
COMPUTERIZED MEDICAL RECORDS - COST BENEFITS
SEARCH #1 (PERFORMED 1/6/04)
DATABASE SEARCHED AND TIME PERIOD COVERED:
PUBMED 1995-2004
SEARCH STRATEGY:
medical records systems, computerized[majr]
AND
costs and cost analysis OR economics OR financ* OR economics(subheading)
AND
Review
NUMBER OF ITEMS RETRIEVED: 26
==================================================================
SEARCH #2 (PERFORMED 1/6/04)
DATABASE SEARCHED AND TIME PERIOD COVERED:
PUBMED 1995-2004
SEARCH STRATEGY:
medical records systems, computerized[majr]
AND
costs and cost analysis OR economics OR financ* OR economics(subheading)
NOT
Review
NUMBER OF ITEMS RETRIEVED: 871
NUMBER OF ITEMS SENT TO RESEARCHER (after deleting irrelevant items): 186 (COVERING 2000-2003)
Article ID: 0000
Reviewer: ______________
Reviewers:
Assigned on:
Check all that apply on each question. To change pre-screener data from “checked” to “unchecked” please write “uncheck” next to the appropriate box.
| 1. What are the one or two main HIT elements being tested? | ||
| Computerized Provider Order Entry | □ | |
| Electronic Health Record | □ | |
| Decision Support | □ | |
| Results Reporting/Viewing Systems | □ | |
| Electronic Prescribing (incl. Barcoding) | □ | |
| Mobile Computing | □ | |
| Data Exchange Networks/ Community Health Information Network | □ | |
| Patient Decision Support/ Consumer Health Informatics | □ | |
| Communication Systems | □ | |
| Administrative | □ | |
| Knowledge/ Information Retrieval Systems | □ | |
| Data Collection/ Data Summary Systems | □ | |
| Telemedicine | □ | |
| HIT in general | □ | |
| Other (specify: ________________________) | □ | |
| Not HIT | □ | |
| N/A, N/R | □ | |
| 2. Which IOM categories does the HIT address? | ||
| Health information and data storage | □ | |
| Results management | □ | |
 Order entry management | □ | |
| Decision support | □ | |
| Electronic communication and connectivity | □ | |
| Patient support | □ | |
| Administrative processes | □ | |
| Reporting and population health management | □ | |
| N/A, N/R | □ | |
| 3. What are the types of healthcare organization settings? | ||
| Hospital/inpatient | □ | |
| Outpatient/ambulatory | □ | |
| Integrated delivery Network (IDN) | □ | |
| Emergency room | □ | |
| Nursing home | □ | |
| Patient home | □ | |
| Pediatrics | □ | |
| Pharmacy | □ | |
| Internet | □ | |
| Other setting (specify: __________________) | □ | |
| N/A, N/R | □ | |
| 4. What is the article's purpose? | [Circle one] | |
| Descriptive | ||
| 1. Qualitative | 2. Quantitative | 3. Other descriptive |
| Hypothesis testing: | ||
| With Intervention, with concurrent comparison group: | ||
| 4. RCT | 5. CCT | 6. Cntrl. Before/After |
| With intervention, without concurrent comparison group: | ||
| 7. Pre-Post | 8. Time series | 9. Historical control |
| No intervention | ||
| 10. Cross-sectional | 19. Case study with concurrent control | |
| Other hypothesis testing: | ||
| 11. Other hypothesis testing | ||
| Predictive analysis | ||
| 12. Cost-effect. | 13. Cost-benefit | 14. Other pred.analysis |
| Review | ||
| 15. Non-systematic | 16.Systematic/MA | |
| Other Purpose | ||
| 18. Other (specify: _________________) | ||
| 5. Does this article report data from any of the following systems? | ||
| Intermountain | □ | |
| Partners | □ | |
| Regenstrief | □ | |
| VA | □ | |
| UK's NHS | □ | |
| Kaiser | □ | |
| Vanderbilt University | □ | |
| N/A, N/R | □ | |
| 6. Which outcomes are measured (numerically reported) in the article? | ||
| Impact on patient safety | □ | |
| Impact on patient satisfaction | □ | |
| Impact on health care effectiveness and quality | □ | |
| Impact on efficiency, utilization, and costs | □ | |
| Time: Admin □; Nurs. □; Phys. □; Pt □; NOS □ | ||
| Impact on healthcare access | □ | |
| Other (specify:_____________________) | □ | |
| N/A, N/R | □ | |
| 7. Are barriers or facilitators the main focus of the paper, and/or are numerical results given? | ||
| [Circle One] | ||
| Yes | 1 | |
| No | 2 | |
| N/A, N/R | 8 | |
| 8. What years did the research take place? | ||
| (Enter 4-digit years. N/A, N/R: enter 9999) | ||
| Year began | Year ended | |
| NOTES: | ||
| □ Check here if this article should be a Star Article Candidate | ||
| David Bates, MD, MSc | David Atkins |
| Chief, General Medicine | Agency for Healthcare Research and Quality |
| Brigham and Women's Hospital | 540 Gaither Road |
| 620 Tremont Street, Room BC3-002M | Rockville, Maryland 20850 |
| Boston, MA 02120-1613 | 301-427-1618 |
| 617-732-5650 | 301-427-1520 (fax) |
| 617-732-7072 (fax) | datkins@ahrq.gov |
| dbates@partners.org | |
| Marc Berg | Mary Jo Deering, Ph.D.* |
| Prof. Dr. Marc Berg, MA, MD, PhD | Director for Informatics Dissemination |
| Chair of Dept. of Social Medical Sciences | NCI Center for Bioinformatics |
| Institute of Health Policy and Management | National Cancer Institute |
| Erasmus University Rotterdam | National Institutes of Health, USDHHS |
| P.O. Box 1738 | 6116 Executive Blvd. - #400 |
| 3000 DR Rotterdam | Rockville, MD 20852 |
| The Netherlands | Phone: 301-594-1273 |
| Phone: ++31-10-4088531 / 4088525 | Fax: 301-480-3441 |
| Fax: ++31-10-4089094 | E-mail: deeringm@mail.nih.gov |
| M.Berg@bmg.eur.nl | |
| Jennie Harvell | Scott Weingarten, MD, MPH |
| Senior Policy Analyst | Chief Medical Officer |
| U.S. Department of Health and Human Services | Zynx Health, Inc. |
| Humphrey Building | 9100 Wilshire Blvd., Suite 655-E |
| 200 Independence Avenue SW | Beverly Hills, CA 90210 |
| Washington, DC 20201 | 310-247-7700 |
| 202-690-6443 | 310-247-710 (fax) |
| 202-401-7733 (fax) | weingarten@zynx.com |
| jennie.harvell@hhs.gov | |
| Arnold Milstein, MD, MPH | Elliott Sternberg, MD |
| Principal and Consultant | Senior Vice President and |
| William M. Mercer, Inc. | Chief Medical Officer |
| 3 Embarcadero Center | St. Joseph Health System |
| Suite 1500 | 500 S. Main Street, Suite 900 |
| San Francisco, CA 94111 | Orange, CA 92868 |
| 415-743-8803 | 714-347-7841 |
| 415-743-8950 (fax) | esternbe@corp.stjoe.org |
| arnold.milstein@mercer.com | |
| Cynthia Baur | |
| Office of Public Health and Science, Office of Disease Prevention and Health Promotion | |
| HHS/OS/OPHS/ODPHP | |
| 738G Humphrey Building | |
| telephone: (202) 205-2311 | |
| telefax: (22) 205-0463 | |
| e-mail:cbaur@osophs.dhhs.gov | |
| Section | Page | Comment | Response |
|---|---|---|---|
| 4. Conclusions | The overall cautious tone of the conclusions is appropriate given the limitations of the data. | No response needed. | |
| 4. Conclusions | Use IOM framework of HIT functions to outline conclusions about which are most likely to produce benefits. | This is a desirable goal, but one we do not think is achievable with published data, because the benefit of functionalities are context specific, dependent on other functionalities of the HIT system, and the current state of the organization. | |
| Report could do more to make its findings useful in the current context of decisions re. HIT - which systems should be implemented to support which functions. | |||
| Overall | What savings will derive from better quality, to whom do those savings accrue, and over what time frame? | We have indicated that existing data do not answer this question. | |
| 1. Introduction | 1 | Few data about stand-alones making a difference. Second paragraph, after “These systems,” please remove “used as stand-alone clinical decision support tools, or” | Done. |
| 1. Introduction | 3 | Fourth par, after “Unless they have most of the business for that organization,” add “or agree to collaborate” Also, remove definition of “free rider” problem. | Done. |
| 1. Introduction | 5 | Not fair to impugn prior studies without specifying all? in advance. Moreover, most journals wouldn't publish the articles if you added more description of components. Please add caveats. | We are not meaning to impugn studies, only specifying what we and others consider important components of an HIT “intervention.” We added a caveat. |
| 1. Introduction | 6 | Regarding HIT articles, would need some standards about how to do what is expected. Standard is to provide bed size, academic/ not. I don't disagree that more info would be helpful. | No response needed. |
| We all agree we need a CONSORT like statement for HIT implementation research. | |||
| 3. Results | 13 | Second paragraph “..furthermore, it is not possible to calculate the cost of the development of an HIT system, since this process has occurred over many years.” Too gross a generalization. Sometimes you add a module quickly. Replace “not possible” with “challenging.” Change to “since this process often has occurred over..” | We added “as a whole” to indicate we are referring to the main HIT system, not the addition of a single component. |
| 3. Results | 13 | paragraph 2, before last sentence, add “They are not commercially available from a vendor, and vendors supply most HIT systems in the U.S.” | Done. |
| 3. Results | 14 | “No study evaluated an HIT system with at least 4 of the 8 categories of functionality.” You might list the categories here or explain. | We added this information. |
| 3. Results - EHR and ambulatory Care | 21 | Do you just mean ambulatory EHRs? You have left out lots of inpatient studies. | Yes, this section focuses only on ambulatory EHRs. |
| 3. Results - EHR and ambulatory care | 24 | “Evaluating the effects of EHR adoption was itself a form of structural change in this study.” Please reword, unclear. | This section has been entirely rewritten in an attempt to make this more clear. |
| What about the studies on preventive care? Should explain why omitted. | |||
| 3. Results - Economic benefits of EHR | 36 | 2nd par - can you repeat size to provide perspective? | Done. |
| 3. Results - Economic benefits of HER | 37 | Should make separate point about importance of interoperability and that it needs to be machine level for high value. | We have included a sixth point to highlight the importance of interoperability. |
| 3. Results -barriers | 41 | Add cites: Lee, F, Journal of American Medical Informatics Assoc Shu K, in MEDINFO proceedings, 2005 Pizziferri, et al 2005 These may be too recent for your entry criteria. | We were unable to obtain (Lee, F, Journal of American Medical Informatics Assoc) & (Shu K, in MEDINFO proceedings, 2005). We added the Pizziferri article to our section on barriers. |
| 3. Results -barriers | 42 | Last paragraph - I don't think you should close with this. This editorial overall is about as evidence free as such things get. | We deleted this text. |
| 4. Limitations | 45 | 1st paragraph “Most of this information is absent for most published studies of HIT.” You should say why journals have word limits, and standards for publication. | We added a caveat for this. |
| 4. Future research | 47 | “This will require the development of high quality data collection instruments that will be feasible to use before, during and after the implementation process.” I don't think this will help. | We changed this to “published standards for such reports.” |
| 4. Future research | 48 | #6 “Well-designed studies are needed to empirically demonstrate the benefit of HIT in improving patient safety not only in the hospital environment, but also in ambulatory settings.” These will be published soon. | We look forward to it. |
| 4. Future research | 49 | Add “support for academic investigators to partner with such organizations" (i.e. non-academic centers, provider orgs) | Done. |
| 4. Future research | 50 | “Provide research support and expertise to grantee organizations lacking a built-in research infrastructure.” Not realistic - won't happen. Suggest omitting. | We changed the wording of this, to more strongly emphasize a public-private partnership. |
| 4. Conclusions | vi | “We identified no hypothesis testing study of HIT that assessed a system of broad functionality which included sufficient cost data and organizational context information to allow generalization to other health care settings.” - What about the Regenstrief CPOE study? Suggest dropping this statement. | We meant cost data in terms of acquisition and implementation costs of the entire system, not a single component. We have added this information to this sentence. |
| 4. Conclusions | I think you are too critical of the published literature on HIT re. inclusions of administrative info, given lack of a standard and word limits on published papers, though I agree it is a good idea. | We added caveats to this. | |
| Overall | This is truly a stellar piece of work. | No response needed | |
| 1. Introduction | 1 | “The use of health information technology (HIT) holds tremendous promise in improving the efficiency, cost-effectiveness, quality and safety of medical care delivery in our nation's healthcare system.” see comment on conclusions. Unless tied to a source, such as an IOM report, this statement is more appropriate to a policy paper than an evidence report. | We changed this to “has been promoted as having...” |
| 1. Introduction | 5 | HIT interventions have a fifth component: communication (see in text comment) | We have added communication to the four existing components as a cross cutting component. |
| 3. Results - Pediatrics | 21 | Tone and content of the 3 section summaries (within peds) are very different. Make more consistent. | In this revision, we have used a medical editor to try and make the report's tone more consistent. |
| 3. Results - EHR and ambulatory | 22 | “studies that were accepted after screening, review, and reconciliation” unclear | This was reworded. |
| 3. Results - EHR and ambulatory | 32 | need to harmonize content and tone. Need to acknowledge extremely small number of studies, almost 50% are at same institution | Done. |
| 3. Results - Economic benefits of EHR | 33 | use of “we” is not consistent across report | We tried to make this more consistent. |
| 3. Results - Economic benefits of HER | 34 | health care information exchange and interoperability - HIEI - this seems like an unhelpful acronym | Done. |
| 4. Future research | 50 | This section seems unfinished in terms of development of ideas. | This particular reviewer received an early, unedited version of report. The future research section has been more fully developed in this draft. |
| Preface | iii | Add that ODPHR provided additional funding | We have included all funding sources. |
| 4. Conclusions | viii | “HIT has the potential to enable a dramatic transformation in the delivery of health care, making it safer, more effective, and more efficient.” I think this sentence should be reconsidered in light of the conclusion sections of the case studies. As an evidence report and not a policy paper, the sentence should more closely reflect the findings, which are that the evidence is very limited, from a very small number of studies and organizations, and is inconclusive. The findings are suggestive of future research more than anything else and of the need to systematize data collection about organizations and contexts in order to be able to make comparisons across studies and settings, per the final sentence of this section. Perhaps another point to make is that cost/benefit analysis may be a relatively recent way of looking at the decision-making. The analysis could suggest the types of information required to collect during evaluation so this type of comparative analysis can be done in the future | We revised this statement to state that predictive analyses suggest that HIT has this potential. |
| Overall | 1 | I read the report with pleasure. The database is excellent and useful. | No response needed. |
| 2. Methods | 2 | Most reviews and ‘searches for evidence’ on benefits (either financial or otherwise) of HIT in the health care literature fall into the trap of treating a HIT as just another ‘new intervention’ that has to be evaluated using RCTs or its somewhat lesser equals. The important contribution of this report is that it has picked up the message... that HIT cannot be treated that way. | No response needed. |
| 2. Methods | 3 | ...any ‘IT innovation’ is highly context-specific. There are so many key variables at stake, the authors argue, that no overall review is possible. Therefore, the authors revert to creating an interactive database, and giving three narrative reviews of substudies in particular contexts, as ‘illustrative of the uses of the interactive database’. Given the job-definition the authors have chosen, this is a wise and defendable decision. Yet I would like to raise the question whether the authors have not tied their hands behind their back by learning only half the lesson of how one should approach the question of the ‘costs and benefits of HIT’ when HIT implementations are truly seen as organizational change events. | No response needed. |
| Overall | 4 | This report candidly and powerfully lays out the dilemmas involved in the questions it sets out to answer. Simultaneously, however, I feel that it could have made a stronger step forward in arguing for a REPHRASING of the question for ‘proof’ for ‘benefits’ than it currently does. | We have added that the organizational context and organizational readiness to change are important components for HIT implementation. |
| ..the important question would be whether the technology technically does what it promises to do (which is a question dependent on the specific application in question) and whether the organization is subsequently ‘ready’ to use this technology in such a way that it may reap these benefits. The latter question is about proper organizational change, about creating a ‘fit’ between the technology and the organization which will generate these potential benefits. Again, nobody in that world would consider asking for RCTs or other such designs. The potential benefit, after all, is plain common sense; the question is whether these specific instances (technology and organization) will be able to yield these benefits. That is a substantially different question from interventions such a novel drug where it is NOT plain common sense whether it will yield improved outcomes compared to a placebo or its competitor drugs. The whole idea of a RCT is brilliantly designed to answer this particular question, and to avoid all other potential confounding variables from interfering with this core question. Whether a particular organization will realize the benefits that such ‘common sense’ innovations may yield is ultimately a managerial question, just like some industries succeeded in strategically reinventing their businesses through IT, while others mainly introduced it as a somewhat smarter way of doing the same thing. | |||
| General | I see absolutely no discussion in the main text to support the statement about lack of evidence for ambulatory EHRs and improving quality by making it more consumer and patient-centric. [I have] the desire to see more research on the costs and benefits to patients of HIT. | We have added a new section, “Health Information Technology and Patient Centeredness” | |
| The dearth of research on the value of HIT to the patient and the consumer, and the value of HIT to healthcare quality through more patient-centric focus, should be elevated in the discussion section of the report | |||
| 1. Introduction | 2 | 2nd para - isn't there some impact on costs, ie. increasing revenue by providing more vaccinations? Would this revenue be offset by reduced hospitalizations? | We have re-done this example to make it more clear. |
| 1. Introduction | 3 | 4th para - add: The gains to HMOs of better care will be sure when capitation payments are risk adjusted. | We have added the sentence: “The gains to HMOs of better care will be more certain when capitation payments are adequately risk adjusted.” |
| 1. Introduction | 3 | Remove comment on “free rider” problem. | We have explained this in more detail. |
| 1. Introduction | 3 | After “insurers might want to subsidize it in part” add: i.e. based on the payer's share of the provider's case mix. | We have added, “i.e. based on the insurer's share of the provider's case-load” |
| 1. Introduction | 4 | “There are financial aspects to many of these non-monetized items. For example, the intervention may reduce the cost of meeting a preexisting reporting requirement.” Not sure of the point, given that even nonprofits ..must stay in business. | Agreed, but many things are done in health care that have non-monetary benefits. We are simply making that point. |
| 2. Methods | 9 | Report “Advanced Technologies to Low Health Care Costs and Improve Quality” - what settings were discussed? | Settings of care have been added to the sentence, (inpatient and ambulatory care). |
| 2. Methods | 9 | JAMIA meta-analysis on computer based clinical reminder systems - what settings were discussed? | Settings of care have been added to the sentence, (ambulatory care). |
| 3. Results - EHR and ambulatory | 20 | Structure-process-outcome framework did not fit with types of outcomes discussed. Framework also interrupted the flow of discussion. Please rewrite | We have restructured the framework by study and structure-process-outcome framework to allow for more fluid discussion. |
| 3. Results - EHR and ambulatory | 20 | 3rd par - “Therefore, the roles of EHRs in improving quality is unclear at this time.” However, later on the authors state that there is evidence of improvements in quality. “5 of the 7 assessed quality through an analysis of some type of outcome.” This conflicts with previous paragraph “6 of the 7 assessed quality with respect to some type of outcome.” | We have entirely rewritten this section to satisfy these comments. |
| Par 3 - What is meant by “patient charts became available”? Electronic charts? | |||
| 3. Results - Economic benefits of HER | 31 | Somewhere early you need to define what constitutes an EHR, minimum functionality, etc. | This is now done in the introductory paragraph. |
| 3. Results - Economic benefits of HER | 31 | Do you mean to imply that decision support is not part of EHR? | This is better explained in the introductory paragraph. |
| 3. Results - Economic benefits of HER | 31 | Strengthen the discussion of evidence that emerges from each of the following studies that support an “economic appraisal” of the EHR component. | We have expanded this discussion. |
| 3. Results - Economic benefits of HER | 33 | 1st Par - Provide an example or two about how these studies quantified / monetized costs and benefits. | How these variables were defined is now indicated in the text. |
| 3. Results - Economic benefits of HER | 33 | Be consistent - either refer to the study location or don't. You mention Kaiser and Partners but not the other study sites. | We have changed this for consistency. |
| 3. Results - Economic benefits of HER | 34 | First study - ($9700 per provider) - what were the component parts of the EHR system? | We now provide the component parts of the EHR system. |
| 3. Results - Economic benefits of HER | 37 | Conclusion 1 - What about information exchange capabilities? | We added this as point 6 under Conclusion. |
| 3. Results - Economic benefits of HER | 37 | Conclusion 5 - Revise last sentence to: Changes to the current health care financing system would be required to realize all quantifiable benefits of EHR implementation. | We have changed the last sentence to, “Realizing all quantifiable benefits of EHR implementation would require changes to the current health care financing system.” |
| 4. Conclusions | 47 | Change “Economic value” heading to “Economic value of HIT and EHR Implementation” | This has been changed to “Economic Value of an EHR System” |
| 4. Conclusions | 47 | Conclusion 4. Under economic value - change 1st sentence to “The positive economic estimates for EHR system implementation are encouraging but at this time are based on limited evidence.” | Done. |
| 4. Conclusions | 47 | Add conclusion “There is some evidence regarding the positive economic value of implementing component parts of EHRs.” | We have added this sentence to this section. |
| 4. Future research | 47 | Change 1 to "Available evidence highlights the potential for EHRs to impact quality of care. However, a more systematic, evidence based understanding of the role of EHRs in quality improvement in ambulatory care and other settings is needed. | We have changed this sentence to “Available evidence highlights the potential for ambulatory EHRs to affect quality of care. However, a more systematic, evidence-based understanding of the role of ambulatory EHRs in quality improvement is needed. This information is critical not only to facilitate adoption, but also to enhance the benefits adopters realize from EHR implementation.” |
| 4. Future research | 48 | Remove “ambulatory” before EHRs in all the sentences. | Done. |
| Preface | iv | Health Communication and Telehealth provided funding - who is this? | This was an incorrect reference to our funders. |
| assorted | typos, language usage | We have made the suggested revisions. | |
| Overall | This is a wonderful review and a very intuitive organization of key findings. | No response needed. | |
| Overall | Given our national predilections for leaping before we look, what you have compiled and made accessible will nonetheless provide a more valuable guide for pioneers than pure intuition. | No response needed. | |
| 1. Introduction | Intro focuses the report on costs, benefits, and barriers to HIT. Organization of the report and conclusions should specifically focus on those areas, or intro should be changed. | We have restructured the organization of the report. | |
| 4. Conclusions | Are there any recommendations for effective metrics to be tracked for organizations implementing HIT? | We have added a conclusion that the field needs standards in this area. | |
| Overall | Add more on potential disconnect between who is funding and who is benefiting, i.e. physician adoption at a community based medical center is distinctly different than at an academic medical center based on financial relationships. | We have added to the text and the conclusion that “who pays” and “who benefits” is a central question and that, except for systems like Kaiser and the VA, it is not possible with published data to reach definitive conclusions, other than to state the potential for a mismatch exists. | |
| Overall | There is a problem knowing when to “jump in” i.e. when will interfaces be optimized and products ready. | We agree this is a problem, and one there is scant evidence regarding | |
| Overall | High quality is less of an issue when patients do not have the ability to choose / access all providers. | No response needed. | |
| Overall | A simple projected case study might be illustrative, i.e.. adding Fine Pneumonia Protocols for pneumonia admission into CPOE or EHR. | We discussed this idea but decided not to include this, there are dozens of such case studies published, which are included in our database. Interested readers may find them there. | |
| Overall | Report is terrific. | No response needed. | |
| Overall | Comment on how this report extends or compliments recent JAMA review. | It extends the recent JAMA review by nothing that the HIT literature, in general, and not just the decision support literature, is dominated by a few centers evaluating their own products. | |
| The generalizability of the results of these evaluations is questionable, for the reasons given. | |||
Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Free Full text in PMC]Nonmonetized consequences are merely costs and benefits that are not expressed in dollar terms. It may be easy to express some of them in dollars but difficult to realize the corresponding cash flows. (For example, the time you spend in traffic may be worth $100/hour, but who is going to pay you for it?) Others may resist expression in dollars.
Nonprofits may explicitly have commitments to provide the highest quality care, but for-profits also share medical ethics and culture to do the best they can for their patients.
However, we recognize that there are barriers to providing this level of specification: For example, prior to that advent of the internet, journals might have been reticent to devote limited space to such descriptions, and the knowledge of what variables need to be included changes over time.
Appendixes and Evidence Tables for this report are provided electronically at http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf.
The eight functionalities are documentation (health information and data storage); results management; order entry management; decision support; electronic communication and connectivity; patient support; administrative processes; and reporting and population health management.
