NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Henriksen K, Battles JB, Marks ES, et al., editors. Advances in Patient Safety: From Research to Implementation (Volume 3: Implementation Issues). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005 Feb.

Cover of Advances in Patient Safety: From Research to Implementation (Volume 3: Implementation Issues)

Advances in Patient Safety: From Research to Implementation (Volume 3: Implementation Issues).

Show details

Implementation of a Data-based Medical Event Reporting System in the U.S. Department of Defense

and .

Author Information

Abstract

Objective: As a result of the Institute of Medicine (IOM) report, To Err Is Human, the Department of Defense (DoD), per the direction of President Clinton, developed an action plan to reduce medical errors. A system to track and trend near misses and adverse safety events for all the medical treatment facilities in the DoD was developed. Process: A standard spreadsheet for data collection through e-mail was created and implemented. A relational database management system was utilized for data analysis. Conclusion: The data reporting system is a qualified success. There are limitations with statistical analysis and variations in the data submitted. The system has helped to identify patterns of patient safety errors and areas where patient safety events warrant further investigation.

Introduction

Department of Defense Instruction 6025.17 established the Military Health System Patient Safety Registry (MHSPSR) on August 16, 2001. MHSPSR is required to “review reports of adverse events, close calls, and root cause analyses; analyze the data; [and] develop and execute action plans for addressing patterns of patient care errors….” 1 The original system to collect event data utilized a paper form that was mailed or faxed to the MHSPSR. This system was not ideal for data collection, analysis, or storage. A standardized spreadsheet for data collection using e-mail was created, and the data were placed in a relational database management system. Multiple challenges presented themselves at various stages of development and implementation. This paper explores the challenges of implementing a new medical event reporting system, the lessons learned, and other issues related to data submission, collection, reporting, and analysis.

Background

The Institute of Medicine (IOM) released To Err Is Human, Building A Safe Health System 2 to the public in November 1999. On December 7, 1999, President Clinton directed the Quality Interagency Coordination Task Force (QuIC) to evaluate the IOM report and to respond with a strategy to increase patient safety and reduce medical errors. 3 The QuIC report, the response to President Clinton's directive, proposed clear measures to promote safer health care. One of the measures was to implement a mandatory reporting system in the hospital and clinics operated by the Department of Defense (DoD). 3 The Floyd D. Spence National Defense Authorization Act of Fiscal Year 2001(NDAA 01), sections 742 and 754, established a centralized patient care error reporting and management system within the DoD. 4

DoD responded to President Clinton's direction in January 2000, creating the DoD Patient Safety Work Group (PSWG). Using the recommendations of the IOM report and the NDAA 01 directive, the PSWG developed both mandatory and voluntary reporting systems similar to the Veteran Health Administration (VHA) Patient Safety Reporting Program for error tracking and reporting within all military health care facilities. 5 On August 16, 2001, DoD Instruction number 6025.17 “established a Military Health System Patient Safety Center (MHSPSC), including a MHS Patient Safety Registry (MHSPSR)”. 1 The DoD Instruction authorized the registry to review reports of adverse events, close calls, and root cause analyses. 1

In 2001, the Patient Safety Registry (PSR) was established as part of the Patient Safety Center (PSC), a Division of the Armed Forces Institute of Pathology. Prior to the establishment of the registry, a safety event-reporting tool was developed at the Air Force Major Command (MAJCOM) level for their utilization. The PSWG adopted the Air Force tool and utilized the Safety Assessment Code (SAC) matrix, a severity and probability category event rating produced by the VHA National Center for Patient Safety. 5 Prior to the establishment of the PSR, this form was used for over a year for data collection submitted to the PSC.

Organizational structure: military health system

Military treatment facilities (MTFs)—144 facilities, including 71 clinics and 73 hospitals—have been instructed to report incident related data to the PSR. The Army has 37 MTFs (9 clinics and 28 hospitals) reporting, the Navy has 31 MTFs (8 clinics and 23 hospitals) reporting and the Air Force has 76 MTFs (54 clinics and 22 hospitals) reporting patient safety events to the PSR monthly. The Army and Navy have hospitals reporting from onsite and offsite clinics. The number of beds per hospital ranges from 10 to over 300. The hospitals and clinics have various medical and dental services and specialties.

Prior to 2002, Patient Safety was a shared function within an MTF. After 2002, a Patient Safety Officer/Manager (PSO/M) position was created within each MTF. A military officer, government employee, or contract employee with a background in the health profession, data analysis, or in administration fills this position. In a number of smaller facilities (free standing and branch clinics), the PSO/M position remains an additional duty.

Each military service has a different reporting structure for their facility's PSO/M. The Air Force has a MAJCOM level with several PSO/Ms reporting to one central MAJCOM headquarter. The ten MAJCOMs report to the service level. In the Army and the Navy, the MTF PSO/M reports to the service level.

Spreadsheet design and implementation

Prior to the creation of the PSR, the MTFs sent event reporting forms to the PSC sporadically. A pilot reporting project was initiated in 2000 and the first report was received with data from October 2000. The report was a two-page form listing Joint Commission on Accreditation of Healthcare Organizations (JCAHO) reportable categories with an “other/miscellaneous” section for events not included in the categories. Several subcategories with causal descriptions of the event were listed under the JCAHO categories. A checklist to the right of the categories and subcategories was used to fill in the number of safety near misses or events, noting the SAC level of the event. The MTF demographics included service type, facility type, bed size, and teaching or non-teaching facility. No specific identifier of facility, patient, or provider was submitted. This paper form was faxed or mailed to the PSC monthly. In 2002, a third page was added requesting a description of near misses that occurred in the facilities and a description of safety actions.

When the PSR was established several weaknesses were found in the existing paper reporting system. Facilities were modifying their reports to meet their individual facility requirements, and the submissions to the PSR were sporadic, inconsistent, and time-insensitive. A decision was made to change the data collection system. Since no comparable national medical error incident reporting system was in place, a new system needed to be developed. To improve the paper-based system, the PSR decided to develop an electronic system. An e-mail system for data collection and reporting was implemented in November 2002 and will be used until a Web-based database is established. The interim solution, a spreadsheet, needed to be user friendly and to allow data to be imported into a relational database management system. The reporting spreadsheet, called the Monthly Summary Report (MSR), was created using the Microsoft® Excel program because it fit the requirements and was accessible to PSO/Ms in the MTFs.

The data collection process begins in the MTF. Each safety event incident report is reviewed by the PSO/M, who decides on the appropriate category and subcategory for placement in the MSR spreadsheet. The data is aggregated. All incident information not included in the spreadsheet stays at the MTF.

When designing the MSR spreadsheet, relevant categories for reported incidents had to be selected for data submission. The data collected from the original form was reviewed and analyzed. Several safety events submitted under the “miscellaneous” section were found to be areas that warranted categorization. The following categories were added to the MSR spreadsheet: Documentation, Identification, Nosocomial infections, Radiology, Staff injuries, and Visitor injuries. The Army customized their form to include additional categories for its data collection. Some of the Army's categories were incorporated into the MSR spreadsheet. The Army categories or subcategories include Environment of care, Operative/Procedure related, Laboratory, Obstetrics related, and Exposure to body fluid. These events were added to the MSR along with an updated JCAHO event listing. Twenty-five categories and 134 subcategories were selected for the MSR spreadsheet. These categories and subcategories were based on those used by the reporting MTFs and not necessarily based on other emerging national or State standards.

Since the “miscellaneous” section of the original form was a catch-all section for any safety event that did not fit into any single category or subcategory, an “other/miscellaneous” subcategory section was added to each category on the MSR spreadsheet, and one additional category for “other/miscellaneous” events with three subcategories for narrative descriptions also was included in the MSR spreadsheet. The narrative section for “near miss” events and “safety actions” provides the opportunity to submit a plethora of information. Since this is important data that needs to be analyzed, it was kept on the MSR spreadsheet. An additional narrative text was added to encourage the PSO/M to make comments, give suggestions, or ask questions.

Once the reporting categories were determined, the spreadsheet had to be created with definitions and instructions. The SAC matrix, a VA tool, was used for the severity and probability categorization of the event. The PSWG developed definitions for a “near miss” and an “actual event,” and the JCAHO definition of a “sentinel event” was used.

The heading of the spreadsheet is “Patient Safety Center Monthly Summary Report.” The e-mail address and phone number of the PSC are clearly noted at the right and left of the heading. There is a space for the MTF's name, and there are drop-down boxes for the year, month, bed size, and service. In addition, there is a question to denote whether the MTF is a teaching facility. Most safety event categories have two or more subcategories for data input. The number of events is inserted into the boxes to the right of the categories, which indicate whether the event was a near miss or an actual event. Each actual event is categorized with a SAC score. The SAC scores are divided into “no harm” and “harm” events and the SAC 3 rating, the most severe, is divided into non-sentinel and sentinel events.

The last page has definitions as well as instructions on the use of the form. The form is color coded to help identify areas and to simplify data collection. The JCAHO sentinel event categories are in red to indicate their importance. The color code also makes the various areas easier to identify, saving the user time. Colors identify areas that will not accept an entry, areas for numeric data, areas that are available for brief text narratives, and areas that accept extensive text descriptions. The MSR spreadsheet designers were concerned about the definition of the categories and subcategories. The JCAHO event categories are easy to interpret since JCAHO has clearly defined their events. Most categories are self-explanatory. However, since the subcategories were selected from data submitted by facilities or from references related to the event type, some subcategories are not clearly defined.

The spreadsheet format is protected to prevent alterations. On an unprotected spreadsheet, the user can alter the design or the wording, potentially compromising the data collected. A spreadsheet with an altered format can freeze the importation of data into the database. The programmer should uniquely identify each version of the spreadsheet created thus enabling importation of each spreadsheet version submitted.

When the new MSR spreadsheet form was completed, it was reviewed and approved by the DoD PSWG. After its approval, the various military service representatives informed their MTF's of the change and rolled out the MSR spreadsheet. Instruction on the use of the MSR was given during DoD Patient Safety Training, which is available quarterly for all personnel interested in the Patient Safety Program. The MSR form is available on the DoD PSC Web site (https://patientsafety.satx.disa.mil/) and can be downloaded for use at the MTF sites. A section for “Frequently Asked Questions” (FAQ) was added to the Web site for questions concerning the spreadsheet, and examples of events were given to help with the selection of subcategories in the MSR. Within two months of the roll out, the PSR began receiving data submitted on the new MSR spreadsheet. The length of time from development to receipt of the first MSR at the PSC was 4 months.

The PSO/Ms are instructed by their service representatives to send data to the PSR monthly. The MSR is to be received at the PSR by the 15th of the following month. Timely access to the data is needed to analyze and provide quarterly reports to the Assistant Secretary of Defense for Health Affairs (ASD (HA)). 1 The number of submissions is counted and the service representative is kept appraised of the status of the MSRs submitted from each MTF.

The Army continues to use a service-specific reporting form, which is formatted in Excel. The PSR converts the data from the Army report into the PSR MSR spreadsheet. The Army event categories that are not collected on the MSR spreadsheet (e.g., patient complaints) are removed during the conversion. The remaining Army event subcategories that are equivalent to the subcategories on the MSR spreadsheet are converted “one to one”—one Army subcategory to one MSR subcategory. The Army subcategories that are not equivalent are converted “many to one”—multiple Army subcategories are summed together into one MSR Army category total. The narrative information is sent in an additional word document and pasted into each PSR MSR spreadsheet. The programmer has embedded a version number into the Army spreadsheet to help facilitate the importation of the data into the database.

Patient safety database

Microsoft Access, a relational database management system that works with Excel, was chosen as the data management application. Data received is sorted, validated, updated, and stored in the PSR. Multiple queries, standard, and ad hoc reports are created. The database is flexible in this respect as analyzing the data often requires reviewing several different categories, one category with several subcategories, or potentially utilizing data from multiple facilities or time periods. The current database represents data from 144 military facilities collected over the past 18 months.

After the database application was selected, a system to import the Excel spreadsheet data into the Access database needed to be implemented. A data entry person was designated to import the data from the Excel spreadsheets into the database, and a procedure for reviewing and importing the data was created to ensure consistent and reliable data importing. After importing the data into the database, an analysis of the data was initiated.

As with any database, the programmer needed to understand why the database was being created and what type of data was being analyzed. This required several sessions of brainstorming prior to the initial design of the database in order to obtain a sense of the type of reports and queries needed. Anyone who was familiar with the data or anyone who was going to utilize the database was included in the brainstorming session. As Access is a relational database, the data was divided into several tables, and relationships established between the tables. 6 To develop an efficient system, the programmer had to understand how the data was to be analyzed and grouped. The MSR database queries and reports are given descriptions to help the user locate the data quickly, and report names are standardized to the extent possible to facilitate easy retrieval. As the database is used, additional areas of inquiry are identified, and new queries and reports are created.

A system for backing up the database should be implemented as soon as data are imported. The PSR performs a daily backup using a removable magnetic storage drive. Daily data backup protects and saves the data in the event the database is damaged. An uninterruptible power supply should be used to avoid power loss.

Patient safety data collection

When the MSR is completed, it is e-mailed to the PSR. Since all information received at the PSC is de-identified prior to being entered into the PSR, a unique code is created for each facility. As the spreadsheets are received, they are coded by month, year, and MTF. The data are imported into the database with a unique identifier. The specific MTFs, patients, or providers are de-identified. 1 Although the data are anonymous, each military service can request its data for service-specific analysis.

Each spreadsheet is checked for the correct date and format. It also is inspected for any gross data input errors or format alterations. If incorrect, the spreadsheet is returned to the facility. The data is reviewed and any “other/ miscellaneous” subcategory events that can be placed in an existing subcategory are recategorized. The database gives every event an identifying number. If there are no subcategories for an event, the number given to the event or the event description can be used for further analysis.

The spreadsheets are retrieved for importing daily. The procedure for importing a spreadsheet has multiple checks to verify the integrity of the format and to check for duplicate submissions. During the importing of the spreadsheet, if the MSR is found to be a duplicate, the data that have already been imported are compared line by line. If the MTF has submitted an updated spreadsheet, the data will be updated within the database. If there is no explanation for the duplicate submission, the MTF is sent an e-mail requesting an explanation. A time restriction is given for the reply. Manual procedures are time consuming and tedious, but they need to be performed to validate the data being submitted.

Data that have been imported into the database remain there. If a duplicate is found after importation or data are recategorized, revised, or changed, the original data remains in the database and is available for review. All non-duplicate data are counted in the database. Once the data are imported, the information is placed into tables where it is available for review and analysis.

Patient safety data analysis

One of the principle functions of the database is to track and trend events. The 2001 National Defense Authorization Act, Patient Safety Section states, “The Secretary of Defense shall implement a centralized process for reporting, compilation, and analysis of errors” and “identify the systemic factors that are associated with such occurrences.” 4 The database is utilized to identify these systematic factors by tracking events over time.

Initially, the standard database reports were developed from the reporting form's general areas such as the number of facilities reporting, number of events per MTF, or number of events per category. The various database reports included counts of total events, events per categories and subcategories, and SAC levels. They also included events by month, service, size and type of facility. All of the areas can be sorted and reviewed when an ad hoc report is needed. As the database matures, new reporting needs are identified. After a year of experience analyzing the data and submitting quarterly reports to ASD (HA), database reports are being tailored to specific information related to areas of increased safety concern such as the number of post-operative infections or type of transfusion errors reported.

The database reports focus on frequency and severity levels. The descriptive information is limited to the narrative submission in the comment section or “other/miscellaneous” section. The narrative information gives an abundance of specific information regarding an event or action, but is usually a condensed version of the event and gives no causal factors related to the event. The “other/miscellaneous” section in specific categories and in the general “other/ miscellaneous” category has several lines available for data to be inserted and then analyzed at the PSC. The information submitted is sometimes brief or vague, but because it is related to an individual event or action it gives the PSC a more detailed view from the MTF level.

The “other/miscellaneous” categories are used as a catch-all area for any event that is not easily categorized. Two database forms were developed to review and recategorize the data that are submitted in the “other/miscellaneous” category. Each event reported in this category is reviewed and recategorized into a relevant section of the MSR. If the event can't be recategorized, the data are left in the “other/miscellaneous” section. New subcategories are added within the database when a specific “other/miscellaneous” safety event is repeated several times.

Two database forms analyze the narrative information that is submitted with the monthly reports. One form assists in the identification of near misses and the other captures corrective actions taken by the MTF. The narrative submissions are rich in detail and represent a large source of data, but the information has to be structured for use in the database. The PSWG defines a near miss as: “any process variation or error that could have resulted in harm to a patient, a visitor, or staff, but through chance or timely intervention did not reach the individual.” Validation rules ensure a consistent assessment of the data received. The corrective action form allows the reviewer to categorize the safety action narratives into 34 safety action categories. This list of action categories was compiled from appropriate safety actions submitted by the facilities in the MSRs. Examples of action categories include: revision of policies or procedures, education or training, utilization of a checklist, and technology implementation.

The submission of reports and the number of reported events have increased over the past 18 months. E-mail problems due to virus assaults on various DoD and other Internet servers occasionally complicated the receipt of MSRs. The PSC worked with service representatives to identify and update reports that were incomplete or contained corrupted files, were lost in transmission, or were otherwise missing. In fiscal year 2003 (FY03) MSR data was submitted for 11 months. In that time, over 64,000 events were reported, and slightly over 50 percent of the events were near misses (events that did not reach the patient). Throughout FY03, a number of MTFs reported that no patient safety events were identified within their facility (“zero event” reports). While low volume reporting may be due, at least in part, to reliance on an incident-report based system for identifying events, other factors including the assignment of dedicated patient safety officers, reporting difficulties, and even operational impacts likely played a role. Also, it is essential to note that largely due to the reporting variability both between MTFs and within MTFs, the PSC was unable to use the database to establish normative data for DoD MTF event reporting. MSR narrative descriptions of near misses and patient safety actions, while highly variable between MTFs, highlighted active programs and identified unique solutions that were then shared across DoD through PSC publications.

Conclusion

The data reporting system is a qualified success. A system of event reporting has been initiated, and the data received, although limited, can be used for reporting purposes. Patient safety issues identified from the data collected are disseminated to the services by quarterly patient safety newsletters (available on the DoD patient safety Web site), patient safety hot topic articles, quarterly reports to ASD (HA) and patient safety training sessions. As quality-assurance documents, PSC reports and other publications containing quality assurance-derived information are protected from release under 10 United States Code Section 1102. As such, they are released only to those within the military health system on a need to know basis.

All 144 MTFs report monthly to the PSC, and the system has increased the awareness of patient safety in the MTFs. The MSR spreadsheet described in this paper is an interim system for collecting and analyzing data. It will be used until a comprehensive Web-based application is developed. There are limitations in the use and analysis of the current system. Statistical analysis can be performed on the percentage of events and total numbers submitted, but rates can't be calculated at this time due to the inability to receive the total discharge or patient visit data for each facility per month. The data received are aggregated, and no specific detailed information concerning an event is available. This is a simple system employed to collect data on patient safety events. The system has helped to identify patterns of error and areas that need to be addressed in the forthcoming Web-based application. It also had identified areas where patient safety events have increased and areas that warrant further investigation. This system is a useful tool in decreasing adverse safety events within the health care system.

Acknowledgments

The views in this paper are those of the authors and do not necessarily represent the official position of the Armed Forces Institute of Pathology, the Department of the Army, or the Department of Defense.

Glossary

ASD (HA): Assistant Secretary of Defense for Health Affairs

DoD: Department of Defense

FAQ: Frequently Asked Questions

IOM: Institute of Medicine

JCAHO: Joint Commission on Accreditation of Healthcare Organizations

MAJCOM: Air Force Major Command

MHSPSC: Military Health System Patient Safety Center

MHSPSR: Military Health System Patient Safety Registry

MSR: Monthly Summary Report

MTF: Military Treatment Facility

NDAA 01: National Defense Authorization Act of Fiscal Year 2001

PSC: Patient Safety Center

PSO/M: Patient Safety Officer /Manager

PSR: Patient Safety Registry

PSWG: Department of Defense Patient Safety Work Group

QuIC: Quality Interagency Coordination Task Force

SAC: Safety Assessment Code

VHA: Veteran Health Administration

References

1.
Department of Defense Instruction 6025.17. Military Health System Patient Safety Program (MHSPSP). January 16, 2001.
2.
Kohn LT, Corrigan JM, Donaldson MS, editors. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. Washington, DC: National Academy Press; 2000.
3.
Quality Interagency Coordination Task Force (QuIC). Doing what counts for patient safety: Federal actions to reduce medical errors and their impact. Report of the Quality Interagency Coordination Task Force to President Clinton. February 2000. Available at: http://www​.quic.gov/report/index.htm. Accessed February 12, 2004.
4.
Public Law 106–398. Floyd D. Spence National Defense Authorization Act of Fiscal Year 2001. October 30, 2000. Available at: http://www​.fas.org/man​/congress/2000/hr616.htm. Accessed February 12, 2000.
5.
Tackett S, Birk C. The patient safety mandate—rebuilding the trust and creating a reporting system. Legal Medicine 2001:17–23. Silver Spring, MD: Armed Forces Institute of Pathology, Department of Legal Medicine.
6.
Roman S. Access database design and programming, Published by O'Reilly & Associates, Inc., Sebastopol, CA.

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this page (139K)

Other titles in this collection

Similar articles in PubMed

See reviews...See all...

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...