NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Shekelle P, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006 Apr. (Evidence Reports/Technology Assessments, No. 132.)

Cover of Costs and Benefits of Health Information Technology

Costs and Benefits of Health Information Technology.

Show details


Original Proposed Key Questions

An evidence report on the costs and benefits of HIT systems was requested by the Leap Frog Group, the Centers for Medicare and Medicaid Services (CMS), the Office of the Assistant Secretary for Planning and Evaluation (ASPE), the Office of Disease Prevention and Health Promotion (ODPHP), and the Agency for Healthcare Research and Quality (AHRQ). The purpose of the report was to develop an evidence base regarding the value of discrete HIT functions and systems in various healthcare settings.

Original key questions for the report were:

  1. What does the evidence show with respect to the costs and benefits of inter-operable electronic HIT data exchange for providers and payers/purchasers?
  2. What is a framework that could be used in this study to describe levels/bundles of EHR functionality and to estimate the costs and benefits by such levels/bundles of functionality by payer/purchaser and percentage of provider penetration?
  3. What knowledge or evidence deficits exist regarding needed information to support estimates of cost, benefit and net value with regard to HIT systems? Discuss gaps in research, including specific areas that should be addressed, and suggest possible public and private organizational types to perform the research and/or analysis.
  4. What critical cost/benefit information is required by decision makers (at various levels) in order to give a clear understanding of HIT Systems value proposition particular to them?
  5. What analytic methods (e.g., sources of data, algorithms, etc.) could be used to produce evidence of the costs and benefits within and across health care provider settings, payers/purchasers, and cumulatively across the health care delivery continuum and payers, of deploying electronic health information technology functions examined in this study?
  6. What are the barriers that health care providers and health care systems encounter that limit implementation of electronic health information systems?

Technical Expert Panel

Each AHRQ evidence report is guided by a Technical Expert Panel (TEP). We invited a distinguished group of scientists, clinicians, and information technology experts, including individuals with expertise in medical informatics, Internet health, and telecommunications to participate in the TEP for this report. A list of panel members is included asAppendix A *

The TEP's participation in the preparation of the report began with a meeting that was conducted via conference call at the start of the project; the purpose of this meeting was to get TEP input on the scope of the project, especially the specific information technology applications to address. We were also seeking input on what constitutes evidence because most of the data on HIT implementation derive from interventions that are not RCTs, which are the usual backbone of EPC evidence reports. This particular meeting was held at two separate times in order to accommodate scheduling conflicts; TEP members were asked to participate on the date that was more convenient for them. The meetings were held on March 19 and March 26, 2004.

At this meeting, we also discussed the framework for how to conduct our research. Many TEP members were interested in HIT implementation issues, for example, what can be learned from others who have implemented HIT in various settings, including both community and academic settings. They also emphasized that HIT is often implemented through multicomponent interventions, of which IT is just one aspect.

Based on the comments received during the TEP conference calls and numerous discussions with AHRQ, it was determined that the report would focus on reviewing the evidence from existing published articles regarding the costs, benefits, and barriers to implementing HIT. Many other excellent suggestions were received during the conference calls, such as performing new cost-benefit analyses or collecting unpublished information on barriers, but the decision was made that a review of existing published evidence should precede any other analyses.

Literature Search

At the time this report was undertaken, another team at RAND was working on a project entitled “Leveraging Modern Information Technology to Transform Medical Care Delivery.” This project, funded by private industry, aimed to suggest policy changes that are likely to increase the rate of adoption of HIT in the United States. One part of the project involved assessing the effects of information technology on costs, health outcomes, and adverse events. We were given the list of titles from the team's November 2003 search of PubMed, which sought systematic reviews published in English from 1995 to 2003. PubMed, which is maintained by the U.S. National Library of Medicine, is widely recognized as the premier source for bibliographic coverage of biomedical literature. It encompasses information from Index Medicus, the Index to Dental Literature, and the Cumulative Index to Nursing and Allied Health Literature (allied health includes occupational therapy, speech therapy, and rehabilitation), as well as other sources of coverage in the areas of health care organization, biological and physical sciences, humanities, and information science as they relate to medicine and health care.

Our own search for studies of HIT began with an electronic search of PubMed on January 6, 2004 for reports of original research as well as any additional articles about HIT published since 1995. We ordered all articles on the HIT topics, regardless of study design or language. Appendix B shows our specific search strategies. We also searched the Cochrane Controlled Clinical Trials Register Database and the Cochrane Database of Reviews of Effectiveness (DARE). The Cochrane Collaboration is an international organization that helps people make well-informed decisions about health care by preparing, maintaining, and promoting the accessibility of systematic reviews on the effects of heath care interventions. In December 2004, we also conducted a specific search of the journal Health Affairs, developing a list of all articles with “information technology” or “information systems” as keywords. Health Affairs has published special editions on this topic in recent years.

Additional Sources of Evidence

Several other sources of evidence were considered, based on the recommendations of the TEP. Advanced Technologies to Lower Health Care Costs and Improve Quality was published in fall 2003 by the Massachusetts Technology Collaborative in partnership with the New England Healthcare Institute. Research was conducted by the First Consulting Group and was sponsored by several Massachusetts companies involved in healthcare and health insurance. The report focuses on seven advanced technologies (including examples of HIT, such as computerized physician order entry and electronic prescribing in the inpatient and ambulatory care setting) that have demonstrated both financial benefits and improved quality of care. It also includes discussions of barriers to implementation.

The Value of Computerized Provider Order Entry (CPOE) in Ambulatory Settings was published in 2003 by the Center for Information Technology, also located in the Boston area. This group conducted an international search for both academic and commercial sources of literature and also contacted 35 vendors regarding their currently available health information technology packages. The report found that CPOE can significantly improve quality while lowering costs.

Meta-Analysis on Computer-Based Clinical Reminder Systems reports on a 1996 meta-analysis of 16 trials by Shea, DuMouchel, and Bahamonde published in the Journal of the American Medical Informatics Association (JAMIA). The authors found that computer reminders in the ambulatory care setting improved utilization of vaccinations, breast cancer screenings, and colorectal cancer screenings, but not pap smears or other preventive care. Personal files were contributed by project staff, consultants, and technical expert panel members in response to a request for any applicable unpublished literature on the costs and benefits of HIT.

Articles could have been identified in more than one way (for example, the PubMed search and personal files might contain some of the same articles).

Article Review

We reviewed the articles retrieved from the various sources against our exclusion criteria to determine whether to include them in the evidence synthesis and in the special interactive database tool we created to accompany this report (see below). A screening review form that contains a series of categorization questions was created to track the articles (see Appendix C *). Two reviewers, each trained in the critical analysis of scientific literature, independently reviewed each study, and resolved disagreements by consensus. The principal investigator resolved any disagreements that remained unresolved after discussions between the reviewers.

As previously indicated, this report includes evidence from articles with many different study designs. Our initial search was unrestricted by study design. The resulting articles were divided into four categories: reviews, descriptive reports, hypothesis testing-studies, and predictive analysis studies.

Review articles identified by the search were classified as either systematic (including meta-analyses) or nonsystematic. The determination of systematic versus nonsystematic was made by reading the methods section of the article to see whether an acceptable method was employed to identify evidence. This assessment was made by the Center directors working independently with consensus resolution. Only systematic reviews were considered for further inclusion.

Articles were classified as descriptive if they primarily described the workings or implementation of a HIT system. We further classified these as qualitative or quantitative, based on the presentation of information regarding such factors as number of tests ordered and costs of implementation.

A third category of articles was classified as hypothesis-testing studies, indicating that researchers attempted to answer a study question by comparing data between groups or across time periods and using statistical tests to assess differences. Hypothesis testing studies were further classified as (1) those containing an intervention with a concurrent comparison group, which included randomized and nonrandomized controlled trials and controlled before-after studies; and (2) studies with an intervention but without a concurrent comparison group, which included pre-post studies, time-series studies with more than two measurement points, and studies that used a historical control group. Additional classifications of hypothesis testing studies included those without an intervention, which were cross-sectional in nature, and “other” hypothesis testing studies.

The fourth category of studies was predictive analyses, which included studies that used modeling techniques to predict what might happen with a HIT implementation rather than what did happen. Predictive analyses include cost-effectiveness and cost-benefit analyses. They typically use data from multiple studies and depend upon several assumptions, some of which are not always explicitly stated.

Selection of Articles and Data Elements for Interactive Database

Articles that were classified as systematic reviews, meta-analyses, hypothesis-testing, or predictive analyses went on to more detailed review. For reasons discussed below, we created structured abstracts for these articles and placed them in an interactive database of HIT studies (

We looked for the following data in each article: a description of the HIT system; the purpose of the study; the year or years the study was performed; the study design; the outcomes reported; a description of the study settings; the intervention and control arm; the evaluation method; a description of the HIT system, including how the system was acquired, the year the system was installed, the capability and comprehensiveness of the system; the integration of guidelines or decision support; the interoperability; the HIT implementation strategy; the financial context, such as whether this is a managed care or capitation environment, pay for performance, or area of public accountability; the system penetration; facilitators and barriers; evidence of the HIT system sustainability; extrinsic factors in valuing costs and benefits; the cost of the HIT system or systems, including initial costs of the hardware and the software; the cost of implementation, including planning, hiring, training, temporary productivity loss, data entry, and other organizational resources; anything about long-term cost; and outcomes, in terms of changes in healthcare utilization, changes in quality of care and patient safety, changes in healthcare costs, changes in efficiency and productivity, changes in revenue, and time needed to accrue the benefit. These data were judged to be important—and, in some cases, vital—to an understanding of the study's results as generalizable knowledge.

Synthesis of Results

Based on considerations about a framework for considering costs and benefits of HIT and what constitutes generalizable knowledge, we determined that a synthesis of the results of the included studies could not be meaningfully accomplished using conventional EPC methods for such syntheses. In other words, because the interpretation of the results of HIT studies is quite context-specific, meta-analysis would not be appropriate. No studies were really homogeneous or similar enough to consider together.

Similarly, a narrative review needs an organizing construct, such as “studies about CPOE,” or “studies of HIT in rural hospitals,” or even “studies of HIT that incorporate decision support and report benefits and costs for patient safety in the capitated ambulatory environment.” However, the possible combinations of key variables is so vast that any limited number of narrative syntheses we might produce for this evidence report would inevitably not meet the needs of many potential users. Therefore, we decided that the most useful synthesis of this evidence would be in the form of structured abstracts of the included studies, presented in the interactive searchable database, which can be used by interested readers of this report to identify those HIT studies that meet their own particular contextual requirements. We also present four narrative reviews of studies in particular contexts, to illustrate the uses of the interactive database and also as a mechanism to discuss the strengths and limitations of the evidence regarding HIT.

Peer Review

A draft of this report was prepared in April 2005 and sent to the TEP members and others for review. We received comments from the persons listed in Appendix D *. Each comment received was tracked in an electronic spreadsheet and addressed in preparing the final report. Peer review comments and our responses to them are listed in Appendix E. Service as a reviewer of this report should not in any way be construed as agreeing with or endorsing the content of the report.


Appendixes and Evidence Tables for this report are provided electronically at http://www​​/pub/evidence​/pdf/hitsyscosts/hitsys.pdf.


  • PubReader
  • Print View
  • Cite this Page

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...