Chapter  51:  Training of Clinicians for Public Health Events Relevant to Bioterrorism Preparedness: Evidence Report/Technology Assessment Number 51

Objectives

Recent terrorist attacks have increased concerns about the Nation's vulnerability to terrorism, including the potential use of biological weapons. The purpose of this evidence report is to identify and review data on the most effective ways to train clinicians to respond to a bioterrorist attack or other public health events that may pose similar threats to the health care system, including infectious disease outbreaks, toxidromes or mass poisonings, catastrophic events that incite public fear, and events that call for use of hospital disaster plans.

Search strategy

The Johns Hopkins University Evidence-based Practice Center (EPC) searched electronic literature databases, including MEDLINE^® and the Educational Research Information Clearinghouse (ERIC^®), using separate strategies for each database. The EPC also searched Internet Web sites and conducted a hand search of references and selected journals. The search covered articles published through May 2001. Search terms included biological warfare, bioterrorism, communicable disease, disease outbreaks, epidemic, disaster planning, catastrophe, toxins, toxidromes, poison, disease notification, surveillance, education, and evaluation.

Selection criteria

Paired investigators independently reviewed the titles and abstracts of citations identified by the search to exclude articles that were not written in English, did not include human data, had no original data, had only a meeting abstract, did not include health care professionals, did not address bioterrorism or a relevant public health model, did not include training or education, or did not include evaluation data.

Data collection and analysis

The paired reviewers evaluated study quality in terms of representativeness of study population, methodologic bias, description of the educational intervention, outcomes, and statistical analysis. The reviewers also extracted information on learning objectives, targeted learners, educational methods, results, and conclusions. The EPC team synthesized information qualitatively because the studies were too heterogenous to support quantitative synthesis.

Main results

The search identified 1,942 unique citations, of which 60 were eligible for complete review. Of these, 53 evaluated the training of clinicians for detection and management of an infectious disease outbreak, 1 evaluated training of clinicians in how to detect and manage toxidromes or mass poisonings, 5 addressed training in how to respond to events that call for the use of hospital disaster plans, and 1 evaluated training of clinicians to report infectious diseases to a central agency. None of the studies evaluated the training of clinicians in how to use Web- or telephone-based central information resources or to communicate with other health professionals during a public health event. Many of the studies had low study quality scores. However, several pertinent findings emerged from some of the studies: 1) use of standardized patients was an acceptable and effective way to train physicians in detection and management of infectious disease outbreaks; 2) satellite broadcasting was an effective way to train large numbers of clinicians and to standardize training across geographically separated groups; 3) a tabletop exercise may be useful for training health care professionals about management of a bioterrorist attack; 4) disaster drill training improved clinicians' knowledge of hospital disaster plans and allowed identification of problems; and 5) a didactic program can help train infection control nurses to report certain infectious disease symptom complexes to a central agency.

Conclusions

Modest evidence exists about effective ways to train clinicians to detect and manage an infectious disease outbreak. Very little evidence exists about how to effectively train clinicians to respond to other types of public health events deemed relevant to bioterrorism preparedness. Almost no evidence exists on training clinicians in aspects of response such as using central information resources, communicating with other professionals, and reporting events to a central agency. This gap in evidence warrants an increased commitment to developing and evaluating educational programs relevant to bioterrorism preparedness, infectious disease outbreaks, and other public health events.

Overview

Recent terrorist attacks against the United States have increased awareness of the Nation's vulnerability to terrorism. One particularly serious form of terrorism involves the use of biological weapons that could cause devastating epidemics.

To minimize the risks of bioterrorism, the United States has made bioterrorism preparedness a priority for government and military agencies, public health advocates, law enforcement, first responders, and health care professionals. Based on the recommendation of a working group led by the Centers for Disease Control and Prevention (CDC), preparation efforts are concentrating on smallpox, anthrax, plague, botulism, tularemia, and the viral hemorrhagic fevers. These agents have been chosen as areas of focus due to their ease of dissemination and transmission, high mortality rates, ability to cause public panic, and need for special public health preparedness.

Until recently, the public and private health care sectors had been largely excluded from the Nation's bioterrorism preparatory efforts. The very group that would handle the consequences of an attack has yet to receive widespread education on the topic. Fortunately, the value of bioterrorism education has been recently recognized, leading to a significant question: How does one effectively train clinicians for such an unusual public health crisis? The purpose of this evidence report is to identify and review data on the most effective ways to train clinicians to respond to a bioterrorist attack or other public health event posing similar challenges to the health care system.

Reporting the Evidence

The target population addressed in the studies reviewed in this evidence report consists of clinicians including physicians, physician assistants, nurses, nurse practitioners, and community health workers. The target audience for the report consists of policymakers and others developing educational strategies for health care professionals that could be involved in the assessment and management of victims of a bioterrorist attack.

To identify the most effective methods to train clinicians to respond to a bioterrorist attack, the Johns Hopkins University Evidence-based Practice Center (EPC) addressed the following key questions:

Q1a. What are effective methods for the initial training of clinicians for detection and management of a bioterrorist attack or other public health event?

Q1b. What are effective methods for updating and reinforcing the training of clinicians for detection and management of a bioterrorist attack or other public health event?

Q2. What are effective methods for training clinicians to use Web- or telephone-based central information resources in response to a bioterrorist attack or other public health event?

Q3. What are effective methods for training clinicians to report events to a central agency in response to a bioterrorist attack or other public health event?

Q4. What are effective methods for training clinicians to communicate with other health care professionals in response to a bioterrorist attack or other public health event?

Due to the paucity of literature pertaining specifically to the education of health professionals in bioterrorism preparedness, the EPC sought to include evidence on the effectiveness of training clinicians for other types of public health events with similar training requirements. Distinctive requirements include the ability to rapidly identify unusual disease syndromes, to contact public health officials, and to communicate with disease control agencies as well as other health professionals. The relevant public health events considered in this report were infectious disease outbreaks, toxidromes or mass poisonings, catastrophic events that incite public fear, and events that call for use of hospital disaster plans.

Methodology

To identify all studies potentially relevant to the key questions, the EPC team searched electronic databases and Web sites and conducted hand searching of references. The databases searched were: MEDLINE^®; the Educational Research Information Clearinghouse (ERIC^®); HealthSTAR^®; the Specialized Register of Effective Practice and Organization of Care Cochrane Review Group (EPOC); the Research and Development Resource Base in Continuing Medical Education (RDRB/CME^®); the Social, Psychological, Educational and Criminological Trials Register (SPECTR); and PsychINFO^®. The team also searched the database of the National Technical Information Service of the United States Government. Hand searching focused on journals that were most likely to have eligible studies, as well as reference lists in key articles. The Internet was searched using the metasearch engine Copernic 2000^®. The search covered articles published through June 2001.

For the first step in the review process, two members of the EPC team independently reviewed the titles identified by the search for relevance to the project. All titles deemed irrelevant by both reviewers were excluded from the abstract review process.

Each potentially relevant abstract was circulated to two members of the study team who independently reviewed the abstract and indicated which, if any, of the key questions the article addressed. For articles found not relevant, the reviewers indicated a reason for exclusion. The exclusion criteria were: not written in English; did not include human data; no original data; meeting abstract only; did not include health care professionals; did not address bioterrorism or a relevant public health model; and did not include training or education.

Each relevant article was read by a pair of reviewers using a form to assess study quality and a form to extract information from the article. At least one reviewer had advanced training in research methods and at least one had relevant advanced clinical training. The reviewers evaluated study quality in terms of educational methods, reporting of representativeness, bias and confounding, description of outcomes, and statistical quality. Study quality scores were calculated for each of the five categories based on the percentage of study quality items that were adequately addressed. An overall quality score was calculated as an average of the five category scores. On the content form, the reviewers abstracted the following types of information from each eligible study: learning objectives, characteristics of targeted health care professionals, educational methods, results for each type of learning objective, and conclusions. Data from the article review process were entered into a relational database.

Findings

• Of the 1,942 unique studies identified by the literature search, 60 met eligibility criteria for the final evidence report.

• Web site searching failed to identify any additional reports of training programs that met the eligibility criteria for this evidence report.

• Fifty-three (88 percent) of the 60 eligible studies addressed the training of clinicians in how to detect and manage an infectious disease outbreak.

• For these 53 studies, the scores for the categories of study quality ranged from the minimum possible score of zero percent to the maximum possible score of 100 percent, with a mean representativeness score of 56 percent, mean bias score of 15 percent, mean description of educational methods score of 55 percent, mean outcome reporting score of 72 percent, and mean statistical analysis score of 30 percent.

• Seven of these studies described an educational intervention using standardized patients and indicated that use of standardized patients was an acceptable and effective way to train physicians to detect and manage an infectious disease. Physicians tended to prepare for a standardized patient visit ahead of time through self-study materials. Costs for the standardized patients were not discussed.

• In three articles, the use of satellite broadcasting for training on management of infectious disease outbreaks was an effective way to train large numbers of people and to standardize training across geographically separated groups. Satellite conferences improved knowledge, enhanced print-based materials, and appeared to be as effective as classroom training.

• One study described a tabletop exercise in which a theoretical plague release was used to test the medical and public health infrastructure. This was the only article directly pertaining to bioterrorism. Tabletop exercises may be a useful, albeit expensive, tool for training clinicians for bioterrorism preparedness.

• One study addressed training in how to detect and manage toxidromes or mass poisonings. This study had a weak design with a quality score of 21 percent, and it did not fully support the authors' conclusion that teleconferencing was an efficient method for educating clinicians about toxidromes.

• Five studies addressed training in how to respond to events that call for use of a hospital disaster plan. Four of these studies used disaster drills and had total study quality scores ranging from 18 to 34 percent. These studies suggested that disaster drill training may improve knowledge of the disaster plan and allow for identification of problems that may then be addressed.

• One study described the use of computer simulation to train clinicians on use of hospital disaster plans and had a study quality score of 9 percent. The simulation allowed identification of deficiencies in staffing, equipment, medications, electromechanical systems, crowd control, and security.

• One study evaluated a program designed to train clinicians to report a public health event to a central agency. This study had a quality score of 50 percent and indicated that didactic methods can help train infection control nurses to report infectious disease symptom complexes to a central agency.

• None of the studies specifically addressed how to update and reinforce the training of clinicians in how to respond to a public health event.

• No studies evaluated educational programs designed to train clinicians to use Web- or telephone-based central sources of information in response to a bioterrorist attack or other public health event.

• No studies evaluated educational programs designed to train clinicians to communicate with other health care professionals during a public health event.

• None of the studies addressed training in how to respond to events that incite anxiety, fear, or mass hysteria.

• The most common educational techniques used in the studies were lectures (31 studies), discussion (19 studies), audiovisual aids (18 studies), and written material (14 studies).

• Over half the studies (42) employed more than one educational technique in the intervention.

• The eligible studies were extremely heterogeneous in terms of learning objectives, setting, targeted clinicians, and methods, thereby limiting the ability to synthesize results across studies.

Future Research

This evidence report highlights the lack of strong published evidence about how to train clinicians for bioterrorism preparedness. Furthermore, there is a paucity of well-designed studies pertaining to the training of clinicians in management of public health events relevant to bioterrorism preparedness. This has significant implications for future research in training health professionals in this area.

To determine the most effective way to train clinicians on how to respond to a bioterrorist attack or other serious public health event, future work will need to give more attention to evaluating the effectiveness of educational programs. Evaluation methods should include pretesting and posttesting, as well as at least one comparison group. The use of measurable outcomes will be critical to ensure unbiased determination of the efficacy of educational strategies. Furthermore, targeted outcomes should be linked to well-defined learning objectives.

The following specific questions are areas for future research. They are vitally important questions to answer and are currently without published evidence.

• What are the most effective and efficient educational methods to impart knowledge and skills to physicians, nurses, and other health care professionals about how to respond to bioterrorism or other public health events?

• How often does clinicians' knowledge about preparedness for bioterrorism or other public health events need to be reinforced?

• What is the most effective technique to train clinicians to use Web- or telephone-based central information resources in response to a public health event?

• What is the most effective approach to training clinicians to report possible public health events to a central agency?

• What is the most effective way to train clinicians to communicate with other health care professionals during a public health event?

• How can information technology (i.e., Web-based educational programs, teleconferencing, and computer simulations) enhance training of clinicians for bioterrorism preparedness and other public health events?

• Are disaster drills and tabletop exercises cost-effective educational methods for training clinicians in how to respond to a bioterrorist attack or other public health event?

• To what extent will clinicians' preparedness for bioterrorism be strengthened by training in how to manage public health events such as infectious disease outbreaks?

Biological Agents: Background

Risk assessment expert J.D. Simon, M.D., wrote in August 1997, "Since biological weapon terrorist attacks could have catastrophic effects in terms of lives lost, and create a medical, political, and social crisis unparalleled in our history, it is important to prepare now for this new age of terrorism."¹

Preparation for the new age of terrorism is a daunting task for our Nation as government and military agencies, public health advocates, and health care professionals all are seeking to determine the best ways to minimize the risk and potential impact of a bioterrorist attack.² To help focus the monumental effort required to prepare the United States for a biological weapons attack, a working group led by the Centers for Disease Control and Prevention (CDC) has identified six pathogens (Category A agents) that are most likely to be used in such an attack. Category A agents pose the most significant threat due to ease of dissemination and transmission, projected high mortality rates, ability to cause public panic, and need for special public health preparedness.³ We will briefly describe the Category A agents to provide background information.

The Threat of Bioterrorism

Despite ratification of the Biological and Toxin Weapons Convention in 1972 prohibiting the development, production, and stockpiling of biological weapons, many countries have continued to covertly develop such weapons.¹² For example, following the Gulf War, investigators discovered that Iraq had weaponized and deployed anthrax, botulinum toxin, and aflatoxin.¹³ Recently, a Russian defector described an extensive and sophisticated offensive biological weapons program in the former Soviet Union. The program supported the mass production of smallpox and the development of genetically engineered biological agents.¹⁴

Why have biological agents become a potential weapon for terrorists? The bacteria, viruses, and toxins have been chosen for several reasons. They have the potential to produce devastating illness even in small amounts and could incapacitate cities or even nations. Many of the agents are easy to procure as one could obtain recipes from the Internet, consult a microbiology book, hire a scientist, or order an agent from a biological supply house. These agents can be disseminated fairly easily, and aerosolization (the most likely method) could be achieved with such vehicles as crop dusters, fumigating trucks, or aerosol cans. Furthermore, the agents are difficult to detect, most being colorless and odorless. Finally, these agents have the ability to produce utter terror and panic. Even a perceived attack could incapacitate a city and its health care system with "worried well" (i.e., persons who were potentially exposed but are without clinical illness) seeking information or reassurance.¹⁰

The actual risk of a biological terrorist event is difficult to quantify and controversial. In the last 25 years, there have been only three successful terrorist attacks with biological or chemical agents. In 1984, a religious commune in Oregon inoculated salad bars with Salmonella, sickening 751 people and hospitalizing 45 people.¹⁵ In 1994, the Aum Shinrikyo, a Japanese doomsday cult, used the nerve gas sarin in Matsumoto, Japan, exposing 250 people and killing 7 people.¹⁶ A year later, the same cult released sarin in a Tokyo subway, killing 11 people and sending more than 5,000 to area hospitals.¹⁷

Some health experts argue that given the rarity of relevant events, bioterrorism should not be considered a major public health threat in comparison to other public health issues such as tuberculosis, pollution, and increasing strain on the health care system.^{18, 19} However, most experts believe that given the potentially high consequences (albeit low probability) of a bioterrorist attack, bioterrorism does indeed pose a significant security and public health threat. The last two decades have seen an accelerated pattern of terrorist attacks on United States property around the world, including some on domestic soil. Examples include: the New York City World Trade Center bombing in 1993; the bombing of the Murrah Federal Building in Oklahoma City in 1995; the attacks on the United States embassies in Kenya and Tanzania in 1998; and the attack on the U.S.S. Cole, an American warship refueling in Yemen, in 2000. In his statement before the Senate Armed Services Committee, the Director of Central Intelligence, George Tenet, stated, "On terrorism, Mr. Chairman, I must be frank in saying that Americans increasingly are the favored targets. United States citizens and facilities suffered more than 35 percent of the total number of international terrorist attacks in 1998."²⁰ His comment was prophetic as on September 11, 2001, the United States experienced the most devastating attack yet recorded. Over 3,000 people died as a result of a far reaching and well coordinated attack in New York City and Washington, DC. This attack revealed the vulnerability of the United States and its citizens and heightened concerns regarding bioterrorism.

Education: Key to Preparation

With the growing number of terrorist attacks, the United States has come to realize it is no longer possible to anticipate or thwart all terrorist activity. However, it is possible and prudent to undertake a campaign to improve bioterrorism response in order to minimize the consequences of any attack. Until recently, preparatory efforts focused on the military, government personnel, and first responders. Ironically, the health care system, which would bear the brunt of an attack, had been excluded from bioterrorism response-planning activities. It is well recognized that in a covert attack, victims will appear at hospitals, doctors' offices, urgent care facilities, and health departments rather than being located at one disaster scene. If health care professionals are unprepared to be first responders and lack knowledge of the syndromes caused by biological agents, the attack could go unrecognized until it is too late, with disastrous results.^{21, 22}

The CDC Strategic Planning Workgroup has identified the major components of domestic preparedness, including improved ability to identify and detect agents, enhanced communication systems, stockpiled antibiotics and vaccines, enhanced epidemiologic surveillance, and continued research in and development of diagnostic tests, vaccines, and drugs. The CDC Workgroup states that "fundamental to these efforts is comprehensive, integrated training designed to ensure core competency in public health preparedness and the highest levels of scientific expertise among local, state, and federal partners."³ A physician with the Institute for Environmental Studies at the University of Pennsylvania states that we can "mitigate considerably the severity of such a catastrophe, and certainly its spread in the context of contagious agents, by careful, cost-effective training and consciousness-raising in the emergency medical services and medical communities." ²³ Clearly, education of health care professionals is a key component of our Nation's response efforts.^{21, 22, 23}

Yet there appears to be a gap between the recognition of the role of education and the actual implementation of educational programs. Emergency medicine clinicians, who are likely to be first responders in a biological attack, have reported a lack of relevant training in bioterrorism. In a 1999 study report, 86 percent of the responding emergency physicians felt training was necessary, but only 53 percent indicated that formal training on biological weapons was included in their residency program.²⁴

So why has there been delay or hesitation in educating this nation's clinicians? The American College of Emergency Physicians (ACEP) has formed a Nuclear, Biological and Chemical Task Force to evaluate the status of bioterrorism training in the United States, identify barriers to this training, and offer recommendations for effective education.²⁵ The Task Force has identified and described six major training programs that are available nationally. These programs are sponsored by such agencies as the U.S. Army Medical Research Institute of Infectious Diseases, Centers for Disease Control and Prevention, Food and Drug Administration, Federal Emergency Management Agency, Department of Defense, Federal Bureau of Investigation, and Bureau of Justice Assistance. The educational techniques are mixed formats with combinations of written material, case studies, audiovisual aids, group discussions, and lectures. The target audiences include military and civilian medical care providers, public health professionals, and first responders.

In addition to identifying the major weapons of mass destruction (WMD) educational programs in the United States, the CDC Task Force has identified three major groups of barriers to training on response to bioterrorism. First, it acknowledges the difficulty of integrating WMD-specific content into medical school and residency curricula. Existing programs are busy and time-restricted; and no approved body of information exists to integrate. Second, it is difficult to develop and maintain effective training materials for curricula. This is exacerbated by the lack of standardized content and a lack of funding and expertise for material development. Third, there is a lack of funding for continuing education and lack of priority given to such education, resulting in substantial barriers to educating clinicians on bioterrorism.

It is reassuring to know that education of clinicians in bioterrorism has been initiated in the United States despite the barriers to training discussed above. However, it is unclear if these programs have been effective. Nor is it known if there are other relevant and effective educational programs available. Finally, the availability of effective programs specifically designed to update and reinforce training for clinicians to prepare for bioterrorism events is unknown.

Purpose of This Evidence Report

To this end, the Johns Hopkins University (JHU) Evidence-based Practice Center (EPC) has prepared this evidence report, the purpose of which is to summarize existing evidence on the effectiveness of training clinicians for public health events relevant to bioterrorism preparedness. In 2000, the Agency for Healthcare Research and Quality (AHRQ) was allocated $5 million for bioterrorism research. The AHRQ proposed a pipeline of research on issues related to bioterrorism. The proposed focus included research on a variety of related health issues, development of tools and procedures for identifying and managing a bioterrorist incident, and ultimately translating this research into practice. The development the JHU EPC evidence report is one aspect of AHRQ's Bioterrorism Initiative.

Unfortunately, little information exists about preparing clinicians for such an unusual public health emergency as bioterrorism. To prepare the evidence report, public health models were needed as proxies from which the JHU EPC team could extrapolate useful information on how to train clinicians. For this reason, the team chose to examine the following types of public health events that would pose challenges to the health care system similar to a bioterrorist event: infectious disease outbreaks, toxidromes or mass poisoning events, catastrophic events that incite fear and mass hysteria, and incidents that require use of hospital disaster plans (see Chapter 2, Public Health Models for Training Clinicians).

Although a myriad of agencies and personnel groups must be trained in bioterrorism preparedness, this evidence report focuses on the training of clinicians. In this report, we define "clinicians" as all clinical health care professionals, including physicians, physician assistants, nurse practitioners, nurses, and community health workers.

Recruitment of Experts

The EPC team identified a core group of nine technical experts to provide input at key points during the project (Appendix A). These included four representatives of relevant professional organizations, two experts in government, and three from academic settings.

The professional organizations were the American College of Physicians-American Society of Internal Medicine, the American Academy of Family Physicians, the American College of Emergency Physicians, and the Infectious Diseases Society of America.

The Government experts were drawn from the Bureau of Health Professions in the Health Resources and Services Administration and the U.S. Army Medical Research Institute of Infectious Diseases.

The university-based experts included: Associate Director for the Johns Hopkins Center for Civilian Biodefense Studies; Program Manager for Research and Development in Medical Training, Medical Modeling, and Patient Simulation at the Research Triangle Institute; and Director of the Center for Disaster Preparedness, University of Alabama at Birmingham.

The JHU EPC team also identified representatives from a range of other stakeholder organizations to serve as peer reviewers of the draft evidence report. The reviewers included representatives of the Association of Practitioners of Infection Control, the Maryland and Delaware State Departments of Health, the Maryland Emergency Management Agency, and the American Red Cross (Appendix A).

The JHU EPC team involved the core experts in refining the key questions and asked both experts and peer reviewers to review the draft report. (See Peer Review Process, below.)

Target Population

The target population addressed in the studies in this evidence report consisted of clinicians that participated in an educational intervention related to the selected public health events. For the purpose of this report, "clinicians" included all clinical health providers, such as physicians, physician assistants, nurse practitioners, nurses, first responders, and community health workers. The targeted clinicians were in various stages of learning, including undergraduate and graduate students, postgraduate trainees, and practicing professionals.

Identifying the Specific Questions

The JHU EPC team developed an initial list of potential questions about training in bioterrorism preparedness. The preliminary list of questions was reviewed by the core technical experts. The core technical experts rated the relevance and importance of proposed questions. They also commented on the clarity of each of the proposed questions and gave their opinion on the ability to answer each key question. The EPC team reviewed the core experts' ratings and comments and established the final list of key questions that would be addressed in the evidence report.

Public Health Models for Training Clinicians

Based on input from the core experts, the following types of public health events or problems were identified as the most useful models to study to gain information on training clinicians to detect and manage a bioterrorist attack:

• Model 1 -- Infectious disease outbreaks (including reportable diseases).

• Model 2 -- Toxidromes (symptom complexes related to toxin exposures) or mass poisonings.

• Model 3 -- Fear and anxiety in the general population or among health care providers in response to catastrophic events or other events inducing mass hysteria.

• Model 4 -- Events that require proper use of hospital disaster plans.

Key Questions

The JHU EPC team sought to address the following key questions as they pertained to preparedness for a bioterrorist attack or other public health event:

Q1a. What are effective methods for the initial training of clinicians for detection and management of a bioterrorist attack or other public health event? ("Effective methods" were defined as those which achieve learning objectives).

Q1b. What are effective methods for updating and reinforcing the training of clinicians in detecting and managing a bioterrorist attack or other public health event?

Q2. What are effective methods for training clinicians to use Web- or telephone-based central information resources in the event of a bioterrorist attack or other public health event?

Q3. What are effective methods for training clinicians to report events to a central agency in the event of a bioterrorist attack or other public health event?

Q4. What are effective methods for training clinicians to communicate with other health care professionals (within and across institutions) in the event of a bioterrorist attack or other public health event?

Causal Pathway

Figure 1: Causal pathway for bioterrorism preparedness (more...)

   Figure 1: Causal pathway for bioterrorism preparedness with key questions on training of clinicians

Q1: What are effective methods for the training of clinicians for detection and management of a bioterrorist attack?

Q2: What are effective methods for training clinicians to use web or telephone based central information resources in the event of a bioterrorist attack?

Q3: What are effective methods for training clinicians to report events to a central agency in the event of a bioterrorist attack?

Q4: What are effective methods for training clinicians to communicate with other health care professionals in the event of a bioterrorist attack?

Figure 1

Literature Search Methods

The literature search consisted of several steps, including identifying sources, formulating a search strategy for each source, and executing and documenting each search.

Web Site Search Methods

The process of searching the Internet for relevant studies required selecting an appropriate search engine, developing search terms, and reviewing the retrieved Web sites for eligible studies.

Qualitative and Quantitative Data Abstraction

The study team developed two article review forms which were reviewed, pilot tested, and revised prior to use. The quality assessment form is shown in Appendix G. The content abstraction form is in Appendix H.

To make sure that all articles met eligibility criteria, the study quality form began with a check of the exclusion criteria (see Abstract Review, above). On the quality assessment form, the reviewers also indicated which of the key questions were addressed in the article. The form then asked 30 questions about study quality in the following categories: representativeness of clinician learners/trainees, bias and confounding, description of the educational intervention, outcomes of the educational intervention, and statistical techniques and quality of interpretation. The items in these categories were derived from study quality forms used in previous JHU EPC projects. They were modified to fit a focus on educational interventions based on published criteria for evaluating an educational program.²⁷ The study team assigned each response level a score of zero (criteria not met), 1 (criteria partially met), or 2 (criteria fully met). The score for each category of study quality was the percentage of the total points available in each category and therefore could range from zero to 100 percent. The overall quality score was the average of the five categorical scores.

The content abstraction form included 22 items that described the participants, the geographic location, the type of participation in the study (mandatory, voluntary, etc.), the setting for the educational intervention, goals and objectives of the study, study design, educational methods, duration of the intervention, extent to which the learning objectives were met, and the study conclusions. Learning objectives were classified as measurable or not and categorized as addressing knowledge, attitudes, skills, behaviors, or clinical outcomes. Information about the extent to which objectives were met was categorized in the same way.

Article Review Process

The team conducted the article review in a serial fashion, reviewing each eligible article identified by the abstract review process. At least one reviewer had clinical training in infectious diseases or emergency medicine, and at least one reviewer had training in clinical research methods. A primary reviewer was responsible for initially completing the quality assessment form and the content abstraction form. The secondary reviewer independently completed a separate quality assessment form. The secondary reviewer then reviewed and checked the primary reviewer's content abstraction form. Any differences between the two reviewers in either form had to be resolved by a consensus of the two reviewers. Reviewers were not masked to author or journal names because previous work has shown that masking is unlikely to make a significant difference in the results of the data abstraction²⁸ and would have complicated the review process.

The team developed a Microsoft^® Access 2000 (Copyright © 1992-9 Microsoft Corporation) database to collect, maintain and analyze the quality assessment and content abstraction data. This database was also used to produce the evidence tables.

Evidence Grading

The EPC team planned to grade the level of evidence on each key question if the review revealed sufficient evidence to merit formal grading. The evidence grades were based on an established grading scheme with well defined levels of evidence. The grading scheme, used in previous systematic reviews,^{29, 30} assigns grades as follows:

• Grade A (strong) -- Appropriate data are available for evaluating the outcomes of the training program, including at least one well done randomized controlled trial; the population of learners is sufficiently large and well described, and adequate controls have been used; data are consistent; and the educational intervention is well described, and one intervention is clearly superior, equivalent or inferior to another at outcome.

• Grade B (moderate) -- Appropriate data available for evaluating the outcomes of the training program; the population of learners is sufficiently large and well described, and adequate controls have been used; data are reasonably but not entirely consistent; and the educational intervention is well described and one intervention is superior, or equivalent for well-defined outcomes, but there is insufficient evidence to make a definite conclusion of superiority of one approach over another.

• Grade C (weak) -- Some data for evaluating the educational interventions are available; the population is adequately large but poorly defined; there may be a trend indicated for preference of one intervention over another for well-defined outcomes, but there is insufficient evidence to draw firm conclusions of superiority.

• Grade I (insufficient) -- Appropriate data are not available, or there is an insufficient number of trainees to assess the intervention either alone or in comparison to alternatives.

Construction of Evidence Tables

Peer Review Process

Upon completion of the draft report, a copy was sent to the core technical experts as well as to the peer reviewers, as listed in Appendix A. Each expert was asked to comment on the form and content of specific sections of the report according to his or her areas of expertise and interest and invited to comment on all other parts as well. The EPC team incorporated the reviewers' comments into the final report.

Literature Search and Abstract Review

The literature search process identified 1,942 unique potentially relevant citations. A summary of the literature search results through electronic databases and handsearching is provided in Appendix I. The bulk of the searching was completed in March and April 2001, with final searches of PubMed completed in June of 2001.

Through the review of titles of the identified citations, 357 citations were determined eligible for abstract review. Of these 75.9 percent (271 articles) did not meet the criteria for article review. Abstracts were excluded for the following reasons: the article did not report on health care professionals (55); the article did not include training and education (204); the article did not include any of the defined public health or bioterrorist attack models (111); or the article did not meet our criteria, including no evaluation information (76). The total number of reasons for exclusion exceeded the number of abstracts reviewed because the paired reviewers did not have to agree on the reason for exclusion, only that the abstract was excluded.

Web Site Search and Review of Eligibility

The Web site search identified 39 unique addresses for review (see Appendix J). Each Web site was reviewed independently by investigators using the eligibility form (see Appendix F). None of the 39 Web sites revealed new information that met the eligibility criteria for the evidence report.

Article Review

On review of the 86 articles, 60 (69.8 percent) were determined to be eligible for data abstraction. The other 26 articles were excluded for the following reasons: did not include health care professionals (1), did not include training or education (12), did not include evaluation of training or education (12), did not include one of the relevant public health models (12), and did not contain original information (1). The total number of reasons for exclusion exceeded the number of articles reviewed because reviewers did not have to agree on the main reason for exclusion, only that the article should be excluded.

Focus and Design of the Reviewed Studies

As shown in the evidence tables, almost all of the studies (54 of 60; 90 percent) focused on infectious disease outbreaks. Of these 54 articles, the most common infectious diseases addressed were human immunodeficiency virus/acquired immune deficiency syndrome (HIV/AIDS), sexually transmitted diseases (STDs), malaria, leprosy, and tuberculosis. Only three studies specifically addressed one of the CDC Category A agents (Ebola hemorrhagic fever, smallpox, and plague).^{60, 62,71} Six studies focused on the other public health event models: one focused on toxidromes and mass poisonings,⁸⁴ and five focused on events that call for use of a hospital disaster plan.^85-89 No studies addressed training for catastrophic events that incite fear. Only two studies addressed more than one question or more than one type of public health event model.^{43, 88}

For the five studies that addressed training of clinicians on use of hospital disaster plans, the total quality scores were all low, ranging from 9.2 to 33.5 percent.^85-89 The bias and statistical scores for these studies were all zero percent.

The one study on training of clinicians to detect and manage toxidromes had a low total quality score of 21.2 percent.⁸⁴ The study on training clinicians to report events to a central agency had a total quality score of 50 percent.⁹⁰

Evidence Grades

After reviewing the studies, the EPC team decided that the evidence was too heterogeneous and the quality of studies generally too poor to justify formal grading of the level of evidence on each key question. The team felt that it was more important to simply identify the questions for which no evidence exists in the literature and identify the limitations in study quality found in the literature on the other key questions.

Results of Key Questions

Each key question is individually addressed in this section. The evidence is reported for each question based on the public health event model represented.

Articles Pertaining to CDC Category A Biological Agents

Of note, three articles described educational programs focusing specifically on one of the CDC Category A biological agents.^{60, 62, 71} The first article described a 3-day workshop in which clinicians in the Congo learned to diagnose Ebola hemorrhagic fever, institute barrier precautions, make post-mortem diagnoses, and initiate contact tracing.⁷¹ The educational program, which was given in order to establish a long-term surveillance system for Ebola hemorrhagic fever in Bandundu, was considered cost effective and successful. Unfortunately, the article did not provide details about the educational techniques used in the workshop.

The second article pertaining to one of the Category A agents described the efforts to eradicate smallpox in Guinea, West Africa in 1967.⁶⁰ Clinicians underwent an 11-day training course on clinical recognition of smallpox, administration of vaccinations, logistics of running a vaccination station, disease surveillance, and epidemiologic investigation. The training sessions included a written program guide and a practical field exercise. Within 13 months of implementation of the program, smallpox was eradicated from Guinea.

The final article was an evaluation of a tabletop exercise involving a theoretical plague release in Denver, Colorado.⁶² This is the only article that described and evaluated an intervention that specifically pertained to bioterrorism. The TOPOFF drill was designed to test the medical and public health infrastructure and was the largest exercise of its kind to date, costing $3 million. The exercise took place over 4 days and involved participants from State and county health departments and Federal health agencies as well as experts in emergency medicine, emergency management, and hospital infection control. Participants were given new information verbally or by memo, and all media communication was transmitted through a virtual news agency. Communication between participants was accomplished largely via conference calls. Evaluation of the educational intervention occurred through interviews with drill participants, controllers, and observers following the exercise. The drill revealed many important vulnerabilities and challenges in bioterrorism response that the medical and public health care system must overcome if it is to effectively care for victims of a bioterrorist attack.

Principal Findings

Despite an extensive literature and Web site search that identified over 1,900 articles, only 60 articles were found that described and evaluated an educational intervention involving one of the key questions and a relevant pubic health event model. The majority of identified studies pertained to the training of clinicians for detection and management of an infectious disease outbreak (key question 1, public health model 1).

These studies combined a variety of educational techniques, such as lectures, discussion, audiovisual aids, and written material. Drawing comparisons among the studies was challenging because of differences in the educational topics, educational techniques, audiences, contact times, evaluation methods, and outcomes. Hence, it was difficult to form definitive conclusions regarding training of clinicians for bioterrorism preparedness from this disparate body of literature.

Nonetheless, common themes were identified in some of the articles. For example, standardized patients were employed as part of an educational intervention in seven studies.^{31, 33, 38, 39, 41, 44, 53} The literature suggested that this technique was acceptable to physicians, who usually prepared ahead of time using self-study materials. Because this technique was generally used for one-on-one educational encounters, it is unclear how useful this educational method would be for widespread training of clinicians about bioterrorism preparedness. In addition, costs for standardized patients were not discussed in the studies reviewed. This educational method may be cost prohibitive for widespread training.

Teleconferencing, or satellite broadcasting, was used in four studies.^{36, 37, 64, 84} This technique appeared to be an effective way to reach large numbers of learners and may be as effective as classroom learning.

A tabletop exercise was used in one article.⁶² Interestingly, this was the only article to directly address the training of clinicians in bioterrorism preparedness. The article suggested that tabletop exercises or theoretical drills may be useful in educating clinicians about bioterrorism response.

Only one article studied training of clinicians using the public health model of toxidromes or mass poisonings.⁸⁴ Unfortunately, this article had a weak study design and did not adequately support the conclusion that teleconferencing is an acceptable, economic, and efficient method for educating clinicians on toxicology. No studies were identified that applied to the training of clinicians using the public health model of events that incite fear and anxiety. Five articles were identified that described and evaluated educational interventions pertaining to the use of hospital disaster plans.^85-89 The literature suggested that disaster drill training improved knowledge of the disaster plan and allowed identification of problems in plan execution that may then be addressed. One of these articles described the use of a computer simulation in disaster drill training. The article concluded that simulation can replace expensive large scale drills and allow identification of deficiencies in staffing, equipment, medications, electromechanical systems, crowd control, and security.

Only one study evaluated methods for training clinicians to report events to a central agency.⁹⁰ In this study, an educational intervention involving monthly structured education, networking, and feedback was effective in training infection control nurses to report certain infectious disease symptom complexes to a central agency.

Three of the key questions went unanswered. No literature specifically addressed the updating and reinforcing of clinicians' training. No literature was identified that addressed the training of clinicians to use Web- or telephone-based central information sources. Also, no information was found on the topic of training clinicians to communicate with other health care professionals during a public health event. Thus, the lack of these important bodies of literature highlights the need for future research into the most effective way to train clinicians in areas that will improve clinicians' ability to respond to a bioterrorist attack or other public health event.

Limitations

This EPC report has several important limitations. The most obvious limitation is the paucity of studies that directly evaluated the effectiveness of training clinicians to detect and manage a bioterrorist attack. We recognize that our search was limited to published English language articles and Web sites; there may be classified or other unpublished material on bioterrorism training that is not included in this report. For example, theU.S. Army Medical Research Institute of Infectious Diseases offers both an in-house training course called the Medical Management of Chemical and Biological Casualties, as well as satellite distance learning courses. In 4 years, these courses trained over 58,000 personnel. Posttesting occurs for both courses, but the data have yet to be published (personal communication, Col. Edward Eitzen) (see Appendix K).

Of the selected public health event models that are relevant to the training of clinicians in bioterrorism preparedness, the infectious disease outbreak model is the only one that had a substantial body of literature evaluating educational methods. However, even the literature on this model provided very little information on three of the key questions. Thus, little is known about effective methods for training clinicians to use Web- or telephone-based central information resources, to report events to a central agency, or to communicate with other health care professionals in the event of a bioterrorist attack.

Another important limitation of this report stems from the marked heterogeneity of the studies on this topic. Because of the differences in the educational interventions, objectives, targeted learners, and evaluation methods, it is difficult to draw conclusions about the overall effectiveness of clinical training programs relevant to bioterrorism preparedness. This problem is aggravated by the small size of the studies and their methodologic limitations. For example, few studies used comparative evaluation methods, and some of the studies did not link well-defined objectives and specific measurable outcomes.

Other significant limitations in the literature have been identified. For example, we were unable to locate any studies that specifically addressed how to update and reinforce knowledge and skills related to the public health models most relevant to bioterrorism preparedness. Clinicians frequently need to retain rarely used information; yet there is little evidence base for understanding how to efficiently update and sustain such knowledge. Furthermore, our literature search did not reveal any studies that evaluated how the effectiveness of educational methods might differ among different types of clinicians. Clinicians in different fields with different responsibilities (e.g., public health nurses versus hospital-based infectious disease specialists) most likely will need different types of training in how to respond to public health events such as a bioterrorist attack (e.g., symptom recognition and reporting versus diagnosis and treatment). Finally, the studies provided very little information about the costs of the educational interventions.

Implications

The recent terrorist attacks in New York City, Washington DC, and Pennsylvania have increased concerns about the risk of future terrorist attacks, including use of biological agents. Policymakers throughout the United States are in the midst of reassessing priorities for efforts needed to protect the country from further terrorist attacks, such as strengthening the public health infrastructure, enhancing disease surveillance systems, and stockpiling vaccines and drugs. While it is likely that the government will assign high priority to these vitally important components of bioterrorism preparedness, it will also be necessary to better prepare the Nation's clinicians for the roles they will play in responding to a bioterrorist attack. This evidence report clearly indicates that the lack of strong evidence on how to train clinicians for public health events represents an important gap in bioterrorism preparedness. Since Federal and State governmental agencies may not be able to fully fund all of the work that is needed to prevent and/or minimize the effects of a bioterrorist attack, other entities such as professional societies, foundations, and academic medical centers, may need to take responsibility for developing and evaluating appropriate educational programs for clinicians. A recent report by the American Medical Association (AMA) stated that, "local, state, and specialty medical societies and the AMA can play crucial roles in preparing the medical community to deal with the challenges presented by such disasters."⁹²

Because of the limitations discussed in the previous section of this report, professional groups and governmental agencies should exercise caution in applying the existing evidence to specific guidelines on how to train clinicians to respond to public health events such as a bioterrorist attack. Instead of developing specific guidelines, it may be more important to call for development of new training programs and encourage appropriate evaluation of each educational intervention so that policymakers can identify the types of programs that are most effective. This evidence report should help to provide direction for such efforts by highlighting weaknesses and gaps in previous work.

As a final aspect of its comprehensive report (discussed in Chapter 1), the ACEP Nuclear, Biological and Chemical Task Force offered general recommendations for the education of clinicians in bioterrorism preparedness.²⁵ Its recommendations are summarized below:

• Recruitment of related organizations to assure integration of education on the response to biological weapons into curricula.

• Development of free, technology-based self-study materials.

• Development of free instructor/learner teaching materials.

• Development of self-study programs for continuing medical education credit.

• Development of materials to be presented at conferences and meetings.

The American Medical Association echoed these requests at the 2000 AMA Interim Meeting.⁹² The AMA called for:

• Development of audience-specific medical education curricula on bioterrorism.

• Development of information resources on the medical response to bioterrorism.

• Encouragement of cooperation between State and specialty societies and Federal agencies to develop model plans for community medical responses to terrorism.

• Encouragement of timely, reliable, and adequate reporting of dangerous diseases to public health authorities.

The comprehensive summary provided by this evidence report may serve as a basis for responding to these recommendations by generating ideas for development of educational methods and stimulating efforts to better evaluate the programs that are being developed. These educational endeavors will be particularly important for those clinicians most likely to be involved in responding to a bioterrorist attack or other public health event, including emergency physicians, specialists in intensive care, and infectious disease and primary care practitioners.

Appendix A. Core Technical Experts and Peer Reviewers

Appendix B. Priority Journals for Handsearching

Journals published monthly, bimonthly, and quarterly (searched Nov. 1, 2000 to May 31, 2001)

Journals published weekly and semimonthly (searched Nov. 1, 2000 to May 31, 2001)

Appendix C. Literature Search Strategy

Appendix D. Literature Abstract Review Form

Record Number:	EPC Bioterrorism Training Project	Reviewer: __< >____
First Abstract Review:	Abstract review Form	Data Entry: _________
Title:

Do not review, because article (check 1 or more) not in English does not include human data no original data meeting abstract (no full article for review) does not include healthcare professionals does not include public health, bioterrorist attack models does not include training or education no evaluation other: (specify)___________ unclear: get article to decide Do not continue if any item above is checked. Otherwise, continue to next column and check at least one box in each in each section

Article relates to Key Questions(check all that apply) initial training (Q1a) updating and reinforcing training (Q1b) training for use of central information (Q2) training for reporting events to central agency (Q3) training to communicate with other healthcare professionals (Q4) does not apply _________________________________________ infectious disease outbreaks (includes reportable diseases) toxidromes or mass poisonings catastrophic events that incite anxiety, fear, and mass hysteria events that use hospital disaster plan does not apply

Article for reference only

Appendix E. Search Terms and Details of Web Site Search

Appendix F. Web Site Eligibility Form

<print date>	EPC Bioterrorism Training Project Website Screening Form	Reviewer: __< >____Data Entry: _________
<Record #>
<title>
<address/URL>

Exclude website from further consideration because (check one):
no material in English personal page unable to identify source (website author or organization) no relevant content: no description or evaluation of educational programs or models Do not continue if any item above is checked	purpose is to promote or advocate violence no original material (i.e., all links are to other sites) this URL/address is not home page specify home page URL, if known: ______________________________ Uncertain/ other: (specify) _____________________________________ _____________________________________

Publications or reports to be considered for review are located at: Home page Links to (please list links to review; write in the name of each link you select as it appears on the web page):
Homepage address: www.________________________ www.________________________ www.________________________ www.________________________ www.________________________ www.________________________	Specific Reports (links form home page): _______________________________ _______________________________ _______________________________ _______________________________ _______________________________ _______________________________
Other comments:__________________________________________________________________ ______________________________________________________________________________________ ______________________________________________________________________________________ Useful as reference only

Appendix G. Quality Assessment Form

EPC Bioterrorism Team Article Quality Review Form

• 4

  Basic Features (check one if applicable)

  Not in English    Does not include human data    No original data    Meetings abstract (no full article to review)    Does not include healthcare professionals    Does not include public health, bioterrorist attack models    Does not include training or education    Does not include evaluation of training or education    Other (specify) _______________________________________________

IF ANY OF THE ABOVE ITEMS IS CHECKED, STOP: DO NOT COMPLETE FORM

• 5

  Does article relate to Key Questions? (check all that apply)
  

  The initial training of clinicians (Q1a)    Methods for updating and reinforcing the training of clinicians (Q1b)    Training clinicians to use web or telephone based central information (Q2)    Training clinicians to report events to a central agency (Q3)    Training clinicians to communicate with other health care professionals (Q4)    Does not apply →stop review here; do not continue

• 6

  Which public health event model corresponds to topic discussed in article?
  

  Infectious disease outbreaks (includes reportable diseases)    Toxidromes and mass poisonings    Catastrophic events that incite fear    Events that call for use of hospital disaster plan    Does not apply →stop review here; do not continue

Representativeness of Clinician Learners/Trainees: For each question, indicate your response by circling one response

• 7

  Were the setting and population from which the learners drawn described and calendar dates of training reported?
  

  a.	adequate (setting and population described and start and end date specified)	2
  b.	fair (one or more of these NOT reported OR poor descriptions)	1
  c.	inadequate (not specified)	0
  d.	not applicable	N/A

• 8

  Were detailed inclusion/exclusion criteria provided (so that groups could be reassembled)?
  

  a.	adequate (Detailed description of specific inclusion and exclusion criteria OR statement that all potential trainees enrolled)	2
  b.	fair (Some description of inclusion OR exclusion criteria)	1
  c.	inadequate (Minimal description or none at all)	0
  d.	not applicable	N/A

• 9

  Was there information about excluded trainees or learners?
  

  a.	adequate (all reasons for exclusion and # excluded OR no exclusions)	2
  b.	fair (only one of above criteria specified OR information not sufficient to allow replication)	1
  c.	inadequate (none of the above criteria specified)	0
  d.	not applicable	N/A

• 10

  Does the article adequately describe the clinician participants (e.g., Specific profession, specialty and education level)?
  

  a.	adequate (Specific profession and speciality within profession; education level specified)	2
  b.	fair (One or more of these not reported OR poor descriptions)	1
  c.	inadequate (not specified)	0
  d.	not applicable	N/A

Bias and Confounding: For each question, circle one response

• 11

  Was there an appropriate comparison group?
  

  a.	adequate (concurrent and similar group)	2
  b.	fair (non-concurrent OR non-similar)	1
  c.	inadequate (non-concurrent and non-similar)	0
  d.	none →Skip to item 16	0
  e.	not applicable	N/A

• 12

  Was assignment of study groups randomized?
  

  a.	yes	2
  b.	no	0
  c.	unclear	0
  d.	not applicable	N/A

• 13

  Did the groups have any important differences on key factors at baseline?
  

  Key Characteristics

  Educational level (e.g., undergraduate, graduate/professional school, post-graduate level)

  Profession (e.g., nurses, EMT, MD)

  Specialty (e.g., Emergency Medicine, Internal Medicine, Pediatrics)

  a.	Groups equivalent in all key factors	2
  b.	Groups have minor difference in 1 factor	1.5
  c.	Groups have major difference in 1 factor or minor differences in more than 1 factor	1
  d.	No information about groups' characteristics or inadequate to compare	0
  e.	Not applicable	N/A

• 14

  Were the key people measuring the educational outcomes appropriately masked?
  

  a.	yes	2
  b.	no	0
  c.	can't tell	0
  d.	not applicable	N/A

• 15

  Was there any intervention other than the educational intervention of interest that differed between groups?
  

  a.	yes	0
  b.	no	2
  c.	can't tell	0
  d.	not applicable	N/A

Description of Intervention: For each question, circle one letter

• 16

  Are the objectives of the intervention clearly stated in specific measurable terms?
  

  a.	adequate (objectives clearly stated in measurable terms)	2
  b.	fair (objectives stated but not stated in specific measurable terms)	1
  c.	inadequate (objectives not stated)	0
  d.	not applicable	N/A

• 17

  Did the objectives of the intervention specifically take into consideration knowledge, beliefs/attitudes, skills, behaviors, or clinical outcomes?
  

  a.	adequate (considers any 3 of 5)	2
  b.	fair (considers 1 or 2 of 5)	1
  c.	inadequate (considers none of the above)	0
  d.	not applicable	N/A

• 18

  Was there a complete description of the educational methods, eg, audio-visuals, lectures, discussion, etc, used in the study?
  

  a.	adequate (intervention could be replicated given the completeness and detail of educational methods)	2
  b.	fair (some detail but insufficient to ensure replication)	1
  c.	inadequate (no detail)	0
  d.	not applicable	N/A

• 19

  Are the required resources adequately described in sufficient detail (e.g. faculty, facility, equipment)?
  

  a.	adequate (resources clearly described in detail sufficient to be replicated)	2
  b.	fair (some detail but insufficient to ensure replication)	1
  c.	inadequate (no detail)	0
  d.	not applicable	N/A

• 20

  Was the curriculum content described in sufficient detail to be replicated?
  

  a.	adequate (curriculum materials described in sufficient detail to be replicated	2
  b.	fair (some detail but insufficient to ensure replication)	1
  c.	inadequate (no detail)	0
  d.	not applicable	N/A

• 21

  Was the organization and administration of the educational intervention described in sufficient detail?
  

  a.	adequate (administrative structure and operational details described in detail sufficient to be replicated)	2
  b.	fair (some detail but insufficient to ensure replication)	1
  c.	inadequate (no detail)	0
  d.	not applicable	N/A

Outcomes of the Educational Intervention: For each question, circle one response

• 22

  Outcomes of the educational intervention were based upon
  

  a.	pre- and post-intervention evaluation	2
  b.	post-intervention evaluation	1
  c.	neither pre- nor post-intervention evaluation	0
  d.	not applicable	N/A

• 23

  Are the evaluation methods described in sufficient detail?
  

  a.	adequate (evaluation methods described in detail sufficient to be replicated)	2
  b.	fair (some detail but insufficient to ensure replication)	1
  c.	inadequate (no detail)	0
  d.	not applicable	N/A

• 24

  Are the evaluation methods congruent with the stated objectives of the curriculum?
  

  a.	adequate (evaluation methods congruent with the stated objectives of the curriculum)	2
  b.	fair (evaluation methods related to but not completely congruent with the stated objectives of the curriculum)	1
  c.	inadequate (evaluation methods either poorly defined OR incongruous with program objectives)	0
  d.	not applicable	N/A

• 25

  Were objective methods used to evaluate outcomes?
  

  a.	adequate (objective methods were used to evaluate outcomes)	2
  b.	fair (methods to evaluate outcomes defined but objectivity of methods is questionable)	1
  c.	inadequate (no objective methods used to evaluate outcomes)	0
  d.	not applicable	N/A

• 26

  Was there any follow-up after completion of the intervention?
  

  a.	yes	2
  b.	no	0
  c.	not applicable	N/A

Statistical Quality and Interpretation: Please circle one response for each question

• 27

  For primary endpoints, is the magnitude of difference between groups AND an index of variability (e.g test statistic, p value, standard error, confidence interval) stated?
  

  a.	adequate (both reported with index of variability using standard error or confidence intervals)	2
  b.	fair (both reported with index of variability using only test statistic or p value)	1
  c.	inadequate (one or both not reported)	0
  d.	no comparison group	0

• 28

  Were the appropriate analyses and statistical tests performed?
  

  a.	adequate (yes for all analyses)	2
  b.	fair (yes for only some of the analyses)	1
  c.	inadequate (not for any of the analyses or can't tell)	0
  d.	not applicable	N/A

• 29

  If groups were not comparable at study onset, was there adjustment of potential confounders with multi-variate or stratified analyses AND were confounders coded in a way to make such control adequate?
  

  a.	adequate (adjustment done AND confounders appropriately coded)	2
  b.	fair (adjustment done BUT confounders not coded appropriately OR coding unclear OR can't tell)	1
  c.	inadequate (adjustment not done OR comparability not previously reported)	0
  d.	not applicable	N/A

• 30

  Were crossovers and/or dropouts handled appropriately in analysis?
  

  a.	sensitivity analysis	2
  b.	by intention to teach (analogous to intention to treat)	2
  c.	by 'intervention received' analysis only	1
  d.	by none of the above	0
  e.	unknown	0
  f	not applicable	N/A

Any comments for us: __________________________________________________________________________

_____________________________________________________________________________________________

_____________________________________________________________________________________________

THANK YOU! For completing this form. Please return to Mollie Jenckes

Appendix H. Content Review and Abstraction Form

EPC Bioterrorism Team Article Content Review Form

• 4

  Type of health care Professional (Check all that apply)
  

  Community health worker Dentist First Responder (e.g EMT) Nurse Pharmacist Physician Other (specify): ___________________ Not specified

• 5

  Clinician specialty (Check all that apply)
  

  Emergency Medicine Family Medicine Internal Medicine Military Medicine Pediatrics Psychiatry Surgery Toxicology Other (specify):_________________________________________________________________ Not specified N/A (for EMTs, pharmacists, dentists, nurses)

• 6

  Training level (Check all that apply)
  

  Undergraduate student Graduate/medical/nursing student Resident Post-doctoral fellow Practicing professional (i.e., completed training) Other (specify):_________________________________________________________________ Not specified

• 7

  Please check the relevant intervention groups and provide a brief description of each group (NS=not specified).
  

  Group	Intervention	Control	# in group	Description of groups
  A
  B
  C
  D
  Total				Total in study, N=

• 8

  Participation of clinician learners in this intervention was (check all that apply):
  

  Incentive-based (monetary gain, licensure) Mandatory (job requirement) Voluntary Other (specify): ___________________________________________________________ Not specified

• 9

  In what part of the world was the intervention mainly performed (check all that apply):
  

  Africa Asia Australia Canada Europe Mexico, South or Central America U.S Other, (specify): ___________________________ Not specified

• 10

  Mark the setting(s) where the educational intervention took place. Check all that apply
  

  Classroom, eg, any enclosed space: lecture hall, community center Clinical office Field Hospital Laboratory Military Pharmacy Web Other (specify)_______________________________________ Not specified

Complete EITHER item 11 OR item 12 below

• 11

  If the overall goals of the educational intervention are stated in this article, write the broad goals verbatim in the spaces provided. If the overall goals are not explicitly stated go to question 12.
  

  1._______________________________________________________________________________ ________________________________________________________________________________ 2._______________________________________________________________________________ _________________________________________________________________________________ 3._______________________________________________________________________________ _________________________________________________________________________________ 4._______________________________________________________________________________ _________________________________________________________________________________

• 12

  If the overall goals were alluded to but not explicitly stated, briefly summarize the broad educational goals:
  

  1._______________________________________________________________________________ ________________________________________________________________________________ 2._______________________________________________________________________________ ________________________________________________________________________________ 3._______________________________________________________________________________ ________________________________________________________________________________ 4._______________________________________________________________________________ ________________________________________________________________________________

• 13

  What are the measurable objectives of the educational intervention in this article:
  

  Measurable Objectives all that apply	Briefly describe the measurable objectives for each item checked in the spaces below here if no measurable objective stated
  Knowledge
  Attitudes/Beliefs
  Skills
  Behaviors
  Clinical Outcomes

• 14

  What educational methods were used in the intervention? (Check all that apply)
  

  Audio/Visual (eg. videotapes) Clinical experiences Computer models Demonstration Discussion Drill/Practice exercise Other (specify): ______________________________ Not specified

  Lectures Programmed learning (sequential, controlled instruction) Readings Role play Standardized patient Problem based learning

• 15

  Was the intervention designed for individuals or groups? Check all that apply.
  

  Individuals Groups Other (specify): Not specified

• 16

  Average total amount of contact time spent by learners in actual training
  

  <2 hours 2-10 hours 11-20 hours >20 hours Not specified

• 17

  Average duration of educational intervention (over what length of time administered) including initial education and feedback period?
  

  <1 day 1-29 days 1 - 12 months >1 year Not specified

• 18

  Length of time to last follow up
  

  <1 day (includes immediate post test) 1-29 days 1-12 months >1 year Not specified

• 19

  Measurement methods used in evaluation of this intervention (check all that apply):
  

  Essays Group interviews Individual interviews Oral exams Performance audits Participant Questionnaires Observer/Rater questionnaire

  Self Assessment forms Written exams Computer interactive tests Other (Specify): __________________________ Not specified

• 20

  How well were the learning objectives met?     No measurable objectives
  

  Outcomes Measures (check all that apply)	Were the learning objectives met?	Briefly describe the measured outcomes in the space provided.
  Knowledge	yes no unclear N/A
  Attitudes/Beliefs	yes no unclear N/A
  Skills	yes no unclear N/A
  Behaviors	yes no unclear N/A
  Clinical Outcomes	yes no unclear N/A

• 21

  Author conclusion/summary:
  

  Overall improvement after educational intervention Partial improvement or mixed results No improvement after educational intervention Unclear Other ____________________________________________________________________

• 22

  Briefly summarize the main conclusions:
  

  1______________________________________________________________________ ______________________________________________________________________ 2______________________________________________________________________ ______________________________________________________________________ 3______________________________________________________________________ ______________________________________________________________________ 4______________________________________________________________________ ______________________________________________________________________

Appendix I. Literature Search Summary

Appendix J. Web Sites Reviewed and Reasons for Exclusion

Appendix K. U.S. Army Medical Research Institute of Infectious Diseases Bioterrorism Training Courses