NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Shekelle P, Morton SC, Keeler EB. Costs and Benefits of Health Information Technology. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006 Apr. (Evidence Reports/Technology Assessments, No. 132.)

Cover of Costs and Benefits of Health Information Technology

Costs and Benefits of Health Information Technology.

Show details

4Discussion

Limitations

The primary limitation of this review is the quality and quantity of the available studies. As we have noted, understanding the benefits and costs of implementing a HIT system requires an understanding of the intervention in terms of its technical components as well as its human factors, project management, and organizational component; and understanding the organization implementing the HIT system requires understanding many things about it, including its past and current culture of change and the economic situation in which it operates. Most of this information is absent for most of the published studies of HIT. Past limits on word count in published articles may have prevented some authors from including such information in their published reports, but recognition of the information that is needed and the recent practice of allowing supplementary methodological information to be posted online should obviate the problem.

A second limitation is that not all relevant published studies may have been identified. While our search efforts were comprehensive, it is inevitable that we did not find some relevant studies. An advantage of our interactive database of evidence (http://healthit.ahrq.gov/tools/rand) is that it can be updated easily, so we invite readers to send us the citations for relevant articles we may have missed.

We selected only articles that were classified as systematic reviews, hypothesis testing, or predictive analysis for more detailed review and structured abstracts in our interactive database. These articles tend to have less description about how the HIT actually operated and its implementation processes than do qualitative, descriptive articles. Although in general we did not find good evidence of such critical information in the literature during our review processes, we provide citations of qualitative articles in our interactive database for interested readers. However, it should be noted that while these qualitative articles might contain more contextual information about the HIT systems, they are completely lacking in any generalizable knowledge about the benefits of HIT, such as reduction in errors or quality improvement. Any such studies that compared outcomes (such as error rates) with and without a HIT system would have been classified as hypothesis-testing studies and would have been included in our analyses.

A third limitation is that we considered only published studies. As noted by our TEP, substantially more information regarding implementation may have been obtained by contacting leading HIT implementers and conducting structured interviews with them. An additional limitation of reliance on published studies is that less successful, or failed, HIT attempts may be less likely to be published than successful ones (a result of publication bias).

A final limitation is that many of the costs and financial benefits of EHR will change over the years, because they depend on the changing price of such factors as hardware, software, labor costs, and medical prices. Consequently, it is not easy to translate costs as they were reported in the original article into today's costs.

Conclusions

General Conclusions

  1. Predictive analyses, based on statistical modeling techniques, suggest that HIT has the potential to enable a dramatic transformation in the delivery of health care, making it safer, more effective, and more efficient. The empirical research evidence base supporting HIT benefits is more limited.
  2. Organizations that have realized some of these major gains through the implementation of multi-functional, interoperable HIT systems built around an electronic health record include the VA, Partners, and Regenstrief Institute.
  3. The impact of HIT implementation on cost and quality will not be consistent across institutions, independent of context. The specific context within which HIT is implemented, including the setting, the clinical issues, and the patient populations, greatly influences its use and effects.
  4. More widespread implementation of HIT is limited by the lack of generalizable knowledge about what types of HIT and methods for its implementation will result in changes in benefits and costs that are specific for specific health organizations, especially for small practices and small hospitals.
  5. The reporting of HIT developments and implementations needs to be improved with greater attention to descriptions of both the intervention and the organizational/economic environment in which it is implemented.
  6. A high priority must be placed on establishing standards for the information that needs to be measured and reported in HIT implementation studies, similar to the CONSORT standards for clinical trials of therapeutics.
  7. Using existing published evidence, it is not possible to draw firm conclusions about which HIT functionalities are most likely to achieve certain health benefits. The assessment of costs is even more uncertain.
  8. Existing evidence is not sufficient to clearly define “who pays for” and “who benefits from” HIT implementation in any healthcare organization except those, such as Kaiser and the VA, that are responsible for paying for and delivering all the care for the defined population.
  9. Models can be built to estimate the costs and benefits of interoperable HIT systems within and across health care provider settings, payers/purchasers and cumulatively across the health care continuum, but these models are based on many assumptions.
  10. Implementation of HIT faces many barriers.

Pediatrics

  1. Limited empiric evidence exists to support a benefit for HIT use in pediatrics in the areas of medication safety, clinical decision-support, process improvement, and cost reduction.
  2. Only one scientific study weighed these benefits against the costs or cost-effectiveness of implementing HIT systems in pediatric healthcare settings.
  3. A majority of HIT systems for use in pediatric practices were tested and/or developed in academic settings, and the ability to generalize these findings to commercially available systems used in nonacademic settings is limited.

EHRs and the Quality of Ambulatory Care

  1. A small set of high quality studies shows that implementation of a comprehensive ambulatory EHR improves quality of care. Available evidence focuses primarily on the impact of ambulatory EHRs on decreasing overused health services by enhancing access to data, providing capabilities for real-time analysis of clinical data, and acting as platforms for decision support.
  2. Ambulatory EHRs improve the structure of care delivery, improve clinical processes, and enhance outcomes. Most available evidence shows the effects of ambulatory EHRs on processes of care.
  3. Interpreting the precise causal effects of ambulatory EHRs on quality is difficult due to lack of systematic and detailed descriptions of system capabilities, limited data (either qualitative or quantitative) on the workflow redesign and organizational changes that accompanied implementation of an ambulatory EHR (or implementation of a new function in an existing EHR package), use of ad hoc measures to assess quality, and use of study designs that do not explicitly take into account sources of bias and confounding. Thus, while existing evidence may have high internal validity, the generalizability of findings is limited.
  4. Although substantial potential exists, evidence for the ability of ambulatory care EHRs to improve quality by making healthcare more consumer- and patient-centered is scant.

Economic Value of an HIT and EHR System

  1. The main quantifiable benefits of an EHR system were savings from data capture and access; decision support to improve efficiency, quality, and safety of care; business management related to staffing, billing, and overheads; and streamlining patient flow.
  2. Few studies quantitatively assessed the costs to implement an EHR system and the financial benefits reaped from it.
  3. All the cost-benefit analyses of an EHR system predicted that the financial benefits would significantly outweigh the costs, in a timeframe that varied from three to thirteen years, but this evidence is limited to large organizations and multi-functional EHR systems.
  4. The positive economic estimates for EHR system implementation are encouraging but are based on limited evidence at this time. Only limited empirical evidence supports the assumptions made in the predictive analyses. Most studies omitted the costs of implementing an EHR system that were associated with the temporary loss of productivity and the cost of process redesigns. Moreover, realization of the financial return is highly sensitive to the organization's financial incentives.
  5. There is some evidence regarding the positive economic value of implementing component parts of an EHR system, with models suggesting that many of the benefits do not accrue unless a broadly functional system is implemented.

Recommendations for Future Research

This section is divided into two parts: (1) research recommendations and (2) recommendations regarding the appropriate types of researchers to carry out the recommended work.

Recommendations

  1. High on the list of future research is the need for agreed-upon standards for reporting HIT implementation studies, similar in purpose to the CONSORT standards for the reporting of clinical therapeutics trials.
  2. The organizational change and workflow redesign required by and accompanying HIT implementation (or implementation of a new HIT function) need to be described and measured with greater validity, reliability, and precision in order to understand the impact of HIT on care delivery. Without such information, the true “intervention” remains unclear, and the generalizability of results will remain limited. This kind of reporting will require the development and dissemination of publishing standards.
  3. While HIT implementation does not easily lend itself to randomized trials, better use of quasi-experimental study designs and other study designs of high internal validity could greatly enhance the clinical relevance of results, reduce bias and confounding, and increase the generalizability of findings. Currently, the published literature is dominated by simple pre-post implementation designs.
  4. Creative, alternative research methodology should be considered to estimate costs and benefits of HIT as a supplement to traditional hypothesis-testing studies. Traditional experimental or quasi-experimental approaches may be impractical because they are expensive, time-consuming, and interfere with HIT implementation. Qualitative studies are often subjective, descriptive, and lack generalizability. Simulation modeling is a promising alternative to generate knowledge and evidence; it is different from analytical modeling where the result functionally depends on the input (a number of parameters). Simulation, or dynamic, modeling uses a set of rules that define how the system being modeled will change in the future, given its present state, existing knowledge, and foreseeable uncertainties. For complex problems like HIT implementation, where time dynamics is important and experimenting with the real system is expensive or impossible, simulation modeling can support estimates of cost, benefit, and net value of HIT systems.
  5. The costs and benefits of HIT depend not only on the internal system (the practice environment) but also on the interactions with the external system, including consumers (patients and potential users of the healthcare system), medical service suppliers (laboratories, radiology centers, other healthcare organizations), technology suppliers, and the regulatory and financing systems an organization operates. Multi-perspective studies are needed to investigate the flow of costs and benefits in order to maximize the benefits of HIT in the larger healthcare delivery system. Again, simulation modeling may be the best methodology for this type of research.
  6. The conceptual foundation for the impact of EHRs on improving care is strong. More research concerning the efficacy and effectiveness of EHRs across health care settings, providers, and patient populations needs to be carried out. Such research will require focusing on how EHR tools are implemented and utilized in day-to-day practice, a broadening of environments to include nonacademic/nonintegrated network practices, the development of methods and instruments directed at evaluation of externally developed systems, and a broader understanding of the human factors issues relevant to healthcare.
  7. More research is needed on which specific components of an EHR are beneficial and also on evaluating new specific components—for example, clinical decision-support. Much of the existing decision support relies on simple rules, and it should be possible to provide substantially better assistance with the use of more-complex rules and models.
  8. More research is needed to evaluate the effects of EHRs on improving quality by making care more consumer-centered.
  9. Process and outcome benefits of HIT that are important and unique to pediatrics must be better quantified, given the unique workflow and information needs of pediatric organizations and practice settings. A growing body of epidemiologic studies has demonstrated the frequency of medication errors in the pediatric healthcare setting. Well-designed studies are needed to demonstrate empirically the benefit of HIT in improving patient safety, not only in the hospital environment, but also in ambulatory and other settings.
  10. Well-designed studies measuring the costs of HIT implementation and resultant benefits in pediatrics and other vulnerable populations (e.g., chronically ill, disabled, etc.) are needed, especially in nonacademic settings and with commercially available HIT systems.

Recommendations Regarding Public and Private Types of Organizations to Perform the Proposed Research and/or Analysis

The assessment of HIT implementations of greatest relevance to most U.S. healthcare institutions will occur in nonacademic settings. Most nonacademic settings have limited research expertise or infrastructure to design and support a research project on HIT. If extramural funds are desired for an evaluation of HIT implementation, the ability to secure funding coincident with the project plan is difficult, if not impossible, especially given the funding cycle of grants. Also, to use a pre-and post-implementation design, the researcher needs funding for an extended period of time to collect enough data to adequately power the study before the HIT system is in place. For financial and pragmatic reasons, this pre-implementation data collection cannot delay the HIT implementation process.

Therefore, we would suggest that for HIT research to be feasible in nonacademic settings with commercial systems, some important steps should be taken:

  1. Create incentives (e.g., matching funds) for nonacademic medical centers and provider organizations to perform high-quality evaluations of vendor-based HIT implementation. These projects should be funded by organizational dollars and support should be provided for academic investigators to partner with such organizations. These measures would help organizations that lack a built-in research infrastructure to conduct rigorous research.
  2. Provide a number of extramural funding mechanisms (government, state, foundation, or even vendor) to evaluate HIT with limited-funding cycles, allowing for adequate pre-implementation measurements and/or rigorous study design. The investigators typically do not determine the timing of implementation, which is often delayed, and funders much be cognizant of this and not penalize the investigators, by disallowing no-cost extensions.
  3. Devise a standard means to adequately assess and describe the “socio-technical” milieu of an organization relevant to HIT implementation.
PubReader format: click here to try

Views

  • PubReader
  • Print View
  • Cite this Page

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...