Logo of nihpaAbout Author manuscriptsSubmit a manuscriptNIH Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Biomed Inform. Author manuscript; available in PMC 2008 Dec 1.
Published in final edited form as:
J Biomed Inform. 2007 Dec; 40(6 Suppl): S46–S49.
Published online 2007 Aug 31. doi:  10.1016/j.jbi.2007.08.003
PMCID: PMC2137930

Evaluating Public Health Uses of Health Information Exchange


Health information exchange (HIE) initiatives are in various stages of development across the United States. They aim to bring previously unavailable clinical data from patients’ disparate health records, which may be spread over multiple provider and payer networks, to the point of care where clinicians and their patients need it most. The implications of these initiatives on public health are numerous. This article provides general evaluation methods for measuring the impact of HIE on public health in six use cases: 1) mandated reporting of laboratory diagnoses, 2) mandated reporting of physician-based diagnoses, 3) public health investigation, 4) disease-based non-reportable laboratory data, 5) antibiotic-resistant organism surveillance, and 6) population-level quality monitoring.

Keywords: interoperability, health information exchange, electronic health records, public health, evaluation studies


The potential of health information exchange (HIE) to improve the public health and public health activities may seem obvious, but current literature provides little evidence proving these effects. More than 165 regional health information organizations (RHIOs) in 49 states nationwide are developing HIE in one form or another [1] and, therefore, the volume of HIE data that could be used for public health purposes is substantial. There are a number of public health use cases for HIE [2;3], but only a few have been implemented [46], and even fewer formally evaluated. Those that have been implemented mainly use proprietary data feeds directly from individual hospitals and other data providers instead of data directed to them through a clinical HIE implementation, often because a functional HIE network is not yet available.

This paper will: (1) describe some of the potential public health use cases for HIE as described by Mostashari et al. [2;3] and (2) discuss potential approaches to the evaluation of these use cases. Because RHIOs and HIE vary widely in their own use cases and implementations, it is difficult to create specific evaluation recommendations. The approaches to evaluation of public health uses of HIE described here will be of a general nature; this chapter will discuss methods and metrics when they may be more easily generalized across varying public health HIE scenarios.

Public Health Use Cases

1) Mandated Reporting of Laboratory Diagnoses

Description of Use Case

This use case describes the situation where a reportable disease may be identified through laboratory results. States specify diseases requiring mandated reporting, which usually includes such diseases as tuberculosis, gonorrhea and viral hepatitis [7]. Electronic laboratory reporting directly to health departments has been shown to improve the timeliness and completeness of reporting [8],[9]. Despite this demonstrated benefit, laboratories and health departments have been slow to adopt electronic lab reporting because of the difficulty of mapping disparate systems and codes to a standard vocabulary [10]. Because many HIE projects create these mappings for clinical exchange, HIE could enable or greatly enhance a region’s ability to electronically transmit reportable disease data to the appropriate agency and may increase the efficiency and quality of public health surveillance, particularly for high-volume diseases.

Approaches to Evaluation

Measuring the volume of relevant data transmitted would be a primary measure of success in this use case (i.e. recall):


If feasible, a before-after study could be conducted to demonstrate any change in the frequency of mandated reporting of laboratory diagnoses using HIE laboratory data. The study could be limited to a particular disease that might be easy to measure in the pre-implementation phase, such as tuberculosis (TB). The study would consider the number of positive TB cultures from the participating HIE data providers (i.e. hospitals), the number of these cases that were reported to the health department both pre- and post-implementation, and the length of time it took these cases to be reported. In this case the metric would be:

{AllTBposculturesfromstakeholder}{TB cases reported to DOH}{AllTBposculturesfromstakeholder}

Additional metrics of interest could look at changes in efficiency, completeness and timeliness of reporting.

2) Mandated Reporting of Physician-based Diagnoses

Description of Use Case

Physicians are obligated to independently report to the health department the same disease described above in the first use case so that additional clinical (e.g. date of onset) or risk factor (e.g. occupation, travel) information can be gained. Physician compliance with this requirement is highly variable (10–85%) [11]. HIE systems may use diagnosis codes (i.e. ICD9-CM), procedure codes (CPT), or medications to identify cases that would otherwise have gone unreported. This automatic screening for certain data through the HIE would likely require manual review to filter out erroneous or non-relevant cases (i.e. diagnoses of rule-out TB with results that are negative for TB) before being reported to the health department. As HIE systems mature and begin to incorporate more information (e.g. complete clinical notes from an electronic health record (EHR)), this use case may employ more advanced informatics functions, such as natural language processing, data mining, or use of knowledge bases, to screen the entire record for relevant cases and improve system recall.

Approaches to Evaluation

Recall would be difficult to measure in this use case since measuring all relevant cases for the denominator would be time consuming and expensive, likely requiring a chart review instrument and expert reviewers to serve as a gold standard. Another measure of interest is to evaluate the performance of the system (i.e. the precision):


Precision gives an idea of how well the system is performing in terms of the number of erroneous or non-relevant cases that are included. Measurement of precision would require a log file capturing all potential clinician-reportable diseases detected by the system, followed by manual review to determine which cases are relevant. An example of this metric using the TB example would be:


Another useful evaluative measure might be to determine the volume of reporting pre- and post-implementation in much the same way that was suggested above for Mandated Reporting of Laboratory Diagnoses, as well as the efficiency, completeness and timeliness of reporting.

3) Public Health Investigation

Description of Use Case

In this instance, the department of health is already aware of a reportable disease or other case requiring additional investigation and in possession of full patient identifiers. Traditionally these additional investigations required multiple telephone calls, travel, and requests for paper records in order to conduct the investigation. With the advent of HIE, the health department investigator simply becomes another user of the HIE system and queries the system for additional information on the patient in question, potentially decreasing the amount of time and effort necessary to complete the required investigation.

Approaches to Evaluation

Because implementation of this use case will vary depending on the extent to which the HIE incorporates clinical data, and the scope and granularity of that data (e.g. does the HIE give access to the full electronic health record, or does it only provide results retrieval?), recommendation of a specific evaluation is difficult. One possibility is to do a qualitative or semi-qualitative study of the public health investigator’s experience through surveys, semi-structured interviews or observational techniques. This would allow the evaluators to determine if the HIE system, usually designed primarily for clinician use, satisfies health department needs when conducting investigations. On the surface, this is more of an implementation-level evaluation to determine if an individual project is successful, but if this were done across multiple HIE projects, analysis might permit the development of a set of standard practices for HIE implementations to help identify the data elements and user interface features that are most essential to this use case.

4) Disease-based Non-reportable Laboratory Data

Description of Use Case

Other diseases are of interest for biosurveillance but are not often on the list for mandated reporting (e.g. influenza, respiratory syncitial virus, norovirus, rotavirus). Some laboratories are already gathering data on these pathogens [12], but the use of HIE systems would help consolidate these data and make them more accessible to health departments. Although these cases will not require public health action on an individual basis, knowledge of the disease patterns in the community can help guide public health messages and rule out less innocuous epidemiological causes.

Approaches to Evaluation

Because this use case will rely on the development of faster, less expensive, and more accessible assays to clinicians, and these are being developed and deployed in parallel with HIE systems, they will likely act as a confounder, making it difficult to construct an evaluation plan that incorporates a pre-implementation phase. Of most interest here is the gathering and analysis of empirical data to make new discoveries regarding the epidemiology of these common pathogens.

5) Antibiotic-resistant Organism Surveillance

Description of Use Case

In this use case, microbiology culture resistance patterns from would be sent through the HIE system directly to the health department, and could be used to construct community-wide antibiograms that help focus antibiotic selection based on local resistance patterns. Additionally, this system could be used to notify clinicians when a patient with a previously-diagnosed antibiotic-resistant organism (ARO) presents [13], which can decrease the spread of AROs among hospitalized patients as much as sixteen-fold [14]. This also potentially affects the financial evaluation of an HIE implementation since the cost of hospital-acquired antibiotic-resistant infections may be as high as $27,083 [15].

Approaches to Evaluation

A study evaluating the rates of local antibiotic resistance patterns before and after the implementation of a community-wide antibiogram would be of interest. Since the intervention here is the dissemination of a community-wide antibiogram, measurement could be done electronically both pre- and post-intervention as long as the HIE system is in place and can provide the required data. Similarly, studies could be done pre- and post-implementation on a hospital level to see if an ARO notification system leads to earlier identification and isolation of ARO-infected patients, and to see if hospital rates of nosocomial ARO infections decrease.

6) Population-level Quality Monitoring

Description of Use Case

There has been mounting concern over the growing epidemics of chronic diseases (diabetes, heart disease), and public health has become increasingly involved in campaigns aimed at their prevention (secondary prevention) and the prevention of their sequelae (tertiary prevention). Although the quality of preventative care is known to be poor [16], monitoring on a community-wide level across different providers and payers has proved very difficult. To the extent that an HIE system transcends the barriers of institutionally “siloed” data, the HIE might be able to provide an enhanced ability to monitor quality metrics across an entire community for these diseases (e.g. rates of colonoscopy or mammograms for cancer screening, or hemoglobin A1C levels for diabetes control).

Approaches to Evaluation

Again, this use case would require a pre- post-implementation study. A number of organizations have developed, or are developing, standard sets of quality measures that could be used for evaluation [17]. Quality measures in an ambulatory care setting are typically organized by disease category, often following the twenty priority areas defined by the Institute of Medicine [18] and reflected in Healthy People 2010 (U.S. Department of Health and Human Services, 2000) and Take Care New York [19]. Any of these quality measures would need to be evaluated for each HIE implementation (i.e. are the data available) and target diseases that the local health department wants to monitor.


This paper describes preliminary suggestions for measuring the impact of HIE on public health in specific use cases. There are other secondary and tertiary benefits to improved public health that would be much more difficult to measure. If these HIE systems do improve quality and safety, while reducing costs and improving patient and provider satisfaction as many believe, then the overall health of the population will improve. In turn, this will lead to fewer lost wages and decreases in productivity due to illness or disability, resources will be freed for use in other areas, and an overall improved health should follow. These kinds of global improvements are difficult or impossible to measure.

During the early phases of development of regional HIE, while projects are amassing stakeholders and building infrastructure, the projects will be limited in scope due to only partial penetration of local markets and therefore only partial data capture (recall). Measures likely to be affected during this early phase are efficiency measures (decreased duplicate testing, decreased admissions and decreased length of stay) and costs (monetization of the efficiency measures listed above and monetization of safety or quality measures such as decreased that may be detected early such as decreased adverse drug events or decreased nosocomial infections). These early measures may be used to calculate a return on investment of the initial implementation costs, and to make the argument to fund ongoing operational support for these HIE systems.

As these HIE systems mature and begin to share data with one another, and a truly interoperable nationwide health information network (NHIN) begins to coalesce, quality and safety effects will begin to accrue and be measurable. As the safety and quality of care improves, there will likely be synergies between the increased levels of information available and improved services that can be layered on top of the NHIN backbone (e.g. clinical decision support that employs knowledge of a patient’s comprehensive medical history, medication lists based on claims and prescription fill data from payers and pharmacies, or pharmacovigilance that allows surveillance for adverse affects of new medications that were not detected in late-phase clinical trials). This will lead to a general improvement in population health and will likely be very difficult to accurately measure or monetize.


Work for this article was supported by: National Library of Medicine T15 NLM 007079 (07/04-08/06)

Pfizer Inc.: Biomedical Informatics Training Grant (08/06-07/07)

National Library of Medicine 1K99LM009556-01 (07/07-07/09)


Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Reference List

1. Marchibroda J, Bordenick JC. Emerging Trends and Issues in Health Information Exchange: Second annual survey of state, regional and community-based health information exchange initiatives and organizations. eHealth Initiative 2005 [cited 2005 Oct 5];Available from: URL: http://ccbh.ehealthinitiative.org/communities/register_download.mspx.
3. Mostashari. Population health monitoring in regional health information networks. S57 Panel; AMIA Annu Symp Proc. 2006
4. Grannis SJ, Wade M, Gibson J, Overhage JM. The Indiana public health emergency surveillance system: ongoing progress, early findings, and future directions. AMIA Annu Symp Proc. 2006:304–8. [PMC free article] [PubMed]
5. Barthell EN, Aronsky D, Cochrane DG, Cable G, Stair T. The Frontlines of Medicine Project progress report: standardized communication of emergency department triage data for syndromic surveillance. Ann Emerg Med. 2004 Sep;44(3):247–52. [PubMed]
6. Richards J, LaVenture M, Snow LA, Hanrahan LP, Ross DA. Panel - Planning for public/private health data exchange: a national and state-based perspective. AMIA Annu Symp Proc AMIA Annu Symp Proc. 2006
7. United States Department of Health and Human Services. Centers for Disease Control and Prevention; Atlanta, GA: 2007. March 23, Nationally Notifiable Infectious Diseases United States 2007. [cited 2007 Jun 20]; Available from: URL: http://www.cdc.gov/epo/dphsi/phs/infdis.htm.
8. Hripcsak G, Knirsch CA, Jain NL, Stazesky RC, Jr, Pablos-Mendez A, Fulmer T. A health information network for managing innercity tuberculosis: bridging clinical care, public health, and home care. Comput Biomed Res. 1999 Feb;32(1):67–76. [PubMed]
9. Silk BJ, Berkelman RL. A review of strategies for enhancing the completeness of notifiable disease reporting. J Public Health Manag Pract. 2005 May;11(3):191–200. [PubMed]
10. Overhage JM, Suico J, McDonald CJ. Electronic laboratory reporting: barriers, solutions and findings. J Public Health Manag Pract. 2001 Nov;7(6):60–6. [PubMed]
11. Progress in improving state and local disease surveillance--United States, 2000–2005. MMWR Morb Mortal Wkly Rep. 2005 Aug 26;54(33):822–5. [PubMed]
12. Laboratory-based surveillance for rotavirus--United States, July 1997–June 1998. MMWR Morb Mortal Wkly Rep. 1998 Nov 20;47(45):978–80. [PubMed]
13. Evans RS, Lloyd JF, Abouzelof RH, Taylor CW, Anderson VR, Samore MH. System-wide surveillance for clinical encounters by patients previously identified with MRSA and VRE. Medinfo. 2004;11(Pt 1):212–6. [PubMed]
14. Jernigan JA, Titus MG, Groschel DH, Getchell-White S, Farr BM. Effectiveness of contact isolation during a hospital outbreak of methicillin-resistant Staphylococcus aureus. Am J Epidemiol. 1996 Mar 1;143(5):496–504. [PubMed]
15. Abramson MA, Sexton DJ. Nosocomial methicillin-resistant and methicillin-susceptible Staphylococcus aureus primary bacteremia: at what costs? Infect Control Hosp Epidemiol. 1999 Jun;20(6):408–11. [PubMed]
16. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003 Jun 26;348(26):2635–45. [PubMed]
17. Institute of Medicine. Performance Measurement: Accelerating Improvement. 2006
18. Institute of Medicine. Priority Areas for National Action: Transforming Health Care Quality. 2003
19. Frieden TR. Take Care New York: a focused health policy. J Urban Health. 2004 Sep;81(3):314–6. [PMC free article] [PubMed]
PubReader format: click here to try


Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...


Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...