Figure 1. Organizational framework of the Brighton Collaboration concerning adverse events following immunization (AEFIs)
 |
|
|
Immunizations are extremely powerful public health interventions due to their high cost-effectiveness, safety, and efficacy. Few other medical interventions have such long-lasting positive impact, given the relatively minor acute costs associated with immunizations. However, unlike most drugs, immunizations are usually given to healthy, young individuals for preventive purposes and thus merit special attention to their safety. In 1974, the World Health Organization (WHO) launched its Expanded Programme on Immunization (EPI). Since then the proportion of children immunized against routine vaccine-preventable diseases has increased from 5 percent to around 80 percent during their first year of life, with a corresponding decrease in disease rates (http://www.who.int/). The immunization programs in the United States and in many other countries are among the most successful public health programs to date.1, 2
Unfortunately, like all medical interventions, no immunization is perfectly safe. With the increase in vaccine coverage in both developed and developing countries, and the reduction in target vaccine-preventable diseases (VPDs), has also come a growing concern for the safety of immunizations.3, 4This is due to an increase in the absolute number of adverse events following immunizations (AEFIs) as well as their increased prominence relative to the decline in VPDs. For vaccines targeted against diseases where herd immunity exists, individual risk-benefit assessment diverges from that of the society at high vaccine coverages.5With many VPDs having become so rare that most parents (and, increasingly, providers) are no longer familiar with their risks and complications, the actual or perceived risk of experiencing any AEFI may outweigh the actual or perceived benefit of immunization to a given individual.6
However, decreases in immunizations have been shown to lead to higher incidence of VPDs in individuals or to community-wide outbreaks. Additionally, few VPDs are eradicable, and even for those that are, stopping immunizations may be unwise in an era of bioterrorism.7Therefore, most immunizations will have to continue indefinitely, with their associated risks and the need to maintain the highest safety standards possible. As we look ahead, a credible and effective immunization safety system in which such risks can be studied and minimized when possible is essential to maintaining the success of immunizations8and to assuring the success of new vaccines made possible by advances in biotechnology.
Unfortunately, unlike efficacy, the “safety” of a vaccine cannot be measured directly. Safety can only be inferred indirectly from the relative absence of multiple, likely adverse events following immunization. To then best address concerns about real or perceived risks of immunization in a scientific manner, several components need to be in place. For example, regulatory agencies need to ensure that adequate trials for safety and efficacy are conducted prior to licensure of new vaccines (and that good manufacturing practices are in place and maintained). However, due to practical limitations of prelicensure trials, such as limited sample size and study duration, the principal focuses for collection of data on rare events are postlicensure studies by various stakeholders (e.g., the public health and clinical care communities, regulators, and manufacturers). Additionally, aggregation of data across study settings would be essential to maximize our understanding of vaccine safety. For such aggregated data to be meaningful, standardized case definitions, i.e., a globally harmonized set of criteria for the identification and assessment of a given AEFI, including guidelines for data collection, analysis, and presentation, are needed.
Unfortunately, only limited standardization has occurred in the past.9, 10The Brighton Collaboration—named after the city in England where the collaboration was conceived, and modeled loosely after the Cochrane Collaboration (http://www.cochrane.org/)—was formed in fall 2001 to meet this urgent need for globally implemented, standardized case definitions and guidelines for data collection, analysis, and presentation of AEFIs. Global standardization would enable comparability of vaccine safety data collected from clinical trials, surveillance systems, individual case reports, and retrospective epidemiologic studies. Creation of a common vocabulary for vaccine safety research will, in turn, allow the development of the best scientific knowledge on the risks of immunization, and ultimately is aimed at enhancing and ensuring trust in immunization programs.
It is well recognized that uniformly accepted case definitions of clinical entities under surveillance (e.g., syphilis, HIV/AIDS, severe acute respiratory syndrome) greatly facilitate the meaningful assessment of the surveillance data. Such definitions allow comparability of numbers of affected patients across different surveillance systems in different countries.11–15Equally, in clinical trials, the impact of case definitions has been well established on which patients are included in a given study16, 17and the necessity of evaluating patients in a standardized format to allow for comparability of study results.18Additionally, the standardized reporting of data from randomized, controlled, clinical trials has reached wide acceptance among publishers and editors of scientific journals because of the apparent improvement of the quality and interpretability of data.19However, the standardized collection, assessment, and reporting of data are largely lacking across different trials and surveillance systems for the safety of medical interventions.20, 21One comparison of standard case definitions for severe adverse cutaneous reactions reported after the use of specific drugs in five countries showed low rates of agreement within and between countries regarding the standardized assessment and the initial diagnosis made for those patients on the initial reports. Lack of such standard coding forms (i.e., definitions) makes interpretation of the data difficult.21
Usefulness and importance of case definitions in public health are well established, and a change in a public health case definition can significantly alter the reporting rate of the disease under surveillance, just as different case definitions in clinical trials can impact the population studied and the overall study results.12–14, 16, 22–26For example, the 2003 case definition for West Nile illness in Colorado and Nebraska included febrile illness as a criterion and yielded 2,947 and 1,942 reported cases, respectively; however, the case definition in Kansas (which did not include febrile illness) yielded 91 reported cases (personal communication from James Sejvar). On the basis of geography, there is no reason to believe that the epidemic did not occur in Kansas; the difference can more likely be explained by a higher specificity and lower sensitivity of the Kansas definition. The balance between sensitivity and specificity chosen for a given definition depends on the respective purpose—e.g., case finding or case diagnosis. In general, standardized case definitions or classifications are viewed as a core of public health surveillance27and are increasingly recognized together with standardized terminology in their importance in enabling meaningful scientific exchange.22, 28–31In vaccine safety specifically, standardized case definitions and guidelines regarding adverse events are also needed to further enhance the quality of vaccines or of vaccine safety management.
Medical dictionaries for regulatory affairs32–34—as well as case definitions for adverse drug reactions35—have been developed and implemented previously. However, relatively little work to develop case definitions for use in immunization safety has occurred to date.10, 36, 37Those developed are not widely implemented, nor do they represent an exhaustive set of definitions or provide guidelines for the standardized collection, analysis, and presentation of data that are needed for data comparability. This leads to the current situation in which, for example, the respective Brighton working groups for fever and local reactions found 9 different cut-off temperatures for fever in 120 vaccine safety studies, and 11 different cut-off diameters for local reaction size in 102 vaccine studies.
Ideally, newly developed case definitions would be evaluated for their applicability, reliability, sensitivity, and specificity in their settings of future use. However, this has largely not been done; case definitions for most clinical entities as well as for adverse drug events are usually implemented without formal evaluation,22, 35, 36and revisions may occur based on experience with their use. Lack of formal evaluation occurs for most case definitions developed for reporting purposes in public health surveillance designed to capture comparable information for a given population. This may be acceptable depending on the objective of the use of a definition. For example, data comparability will be achieved as long as all investigators use the same definition in the same fashion, even if some patients were to be falsely included or excluded from the data analysis. Where evaluation was done for case definitions of clinical entities, researchers found that small changes in the definition can have a large impact on its sensitivity (i.e., who is classified as having the condition), the completeness and timeliness of reporting data to surveillance systems, and the definition's appropriateness in different cultural settings.16, 17, 23, 38–40
Finally, to achieve the stated purpose, recommendations, products (e.g., analytical software programs), or documents (including case definitions and guidelines) need to be shared with the intended target groups and implemented in the respective research, surveillance, and geographic settings. Use of documents can be enhanced via various routes, e.g., through the reputation or high-profile character of the group responsible for their development, through regulatory mandates, extensive training and education about the benefit of their use, or merely on the basis of their scientific integrity and benefit.
The Brighton Collaboration has created the infrastructure and process to improve comparability of vaccine safety data through the development, evaluation, and implementation of case definitions and guidelines designed for studies of AEFIs.
At its first in-person meeting in September 2000, the Brighton Collaboration steering committee concluded that to set a new global standard of understanding of AEFIs, it would be imperative to have a transparent process with global input at all stages by various scientific stakeholders interested in immunization safety (e.g., scientific, regulatory, clinical care, and vaccine-producing groups). As part of strategic planning for present and future tasks of the Collaboration, an organizational structure was developed (Figure 1
) to best achieve our goal
of enhancing the safety of immunization.2
Through publications, presentations, informational letters to members of professional organizations, as well as direct contact of experienced scientists, as of April 2004 we recruited more than 500 volunteers. These volunteers in varying capacities, participate in (a) working groups developing case definition and guidelines (more than 130 volunteers); (b) the “reference group” as experts recruited via the Brighton e-mail list to comment on draft documents (more than 80 volunteers); (c) evaluating or implementing finalized documents (more than 190 volunteers); and (d) staying informed about our progress and serving as steering committee members, consultants to working groups, or representatives of their respective organizations (includes all 500 volunteers). The volunteers come from 57 countries; however, almost half come from the United States, and efforts are being made to increase participation by non-U.S. volunteers.
The progress of the Collaboration, including its participants and the finalized documents (Figure 2
),
can be followed through its Web site at: http://brightoncollaboration.org/.We are currently funded largely by the U.S. Centers for Disease Control and Prevention (CDC), with additional funding from WHO and, for the first 3 years, through a Research Grant for Improved Vaccine Safety Surveillance (EUSAFEVAC) from the European Commission. Although funding has been sufficient to support basic coordination and Web-programming work of the Collaboration, full use of its capacity requires additional resources from other sources. These additional resources would allow the Collaboration to address all aspects of global networking, development of case definitions and guidelines (e.g., formation of additional working groups), formal evaluation of finalized documents (e.g., on a routine basis, and in clinical trials as well as in the one surveillance system study formally conducted within the Collaboration presently), and coordinated implementation (e.g., in-depth assistance provided to researchers interested in implementing the definitions and guidelines in existing research settings), including training on use of the definitions. To this end, we are partnering with the CDC Foundation, a U.S.-based nonprofit organization that is dedicated to “fostering the attainment of CDC's vision of healthy people in a healthy world through prevention” (http://www.cdcfoundation.org/about/mission.html). In addition, in December 2003 a Switzerland-based nonprofit/nongovernmental organization was established, the Brighton Collaboration Foundation, which aims to protect and preserve public health by promoting immunization safety through support of the Brighton Collaboration.
It needs to be emphasized that we could not exist nor fulfill our principles of globality and a multidisciplinary approach without our many uncompensated volunteers.
As of April 2004, 6 finalized case definition and guidelines documents have been published,41–46and 18 additional documents are at various stages of development (Table 1). Documents are being developed by working groups of 5–20 volunteers per AEFI. The documents are based on a review of systematic literature searches on the respective AEFI and existing case definitions from other groups or surveillance systems, as well as on consensus-building activities.
Working groups use a Web-based platform for file sharing to facilitate international participation, to accommodate volunteers with varying e-mail capacities, and to permit online revision of documents. Draft documents developed by the respective working groups are first reviewed by representatives of the Brighton steering committee and then posted on the Brighton Web site, together with a Web-based survey asking detailed questions about all aspects of the case definitions and guidelines. The surveys are completed by the “reference group,” which is drawn from an e-mail list of all Brighton Collaboration volunteers and announcements to numerous professional organizations. Typically, 20–40 scientists comment on draft documents. Scientists come from all aspects of immunization safety (i.e., clinical care, regulatory, public health, academic, professional, and manufacturing organizations). Comments received are reviewed and incorporated into the final documents, as deemed appropriate by the respective working groups.
The case definition and guidelines documents are structured into (a) a preamble, which explains essential decisions made on the case definition; (b) the case definition itself; and (c) guidelines for data collection, analysis, and presentation. (Additional information about the structure of these documents can be found on the Brighton Collaborative Web site.) It is understood that levels of reporting detail may vary depending upon available resources, geographic region, and whether the source of information is a prospectively designed clinical trial, postmarketing surveillance, an epidemiologic study, or an individual report of an AEFI. Therefore, the definitions and guidelines—which are developed for worldwide use—need to accommodate the quality of information a country, surveillance, or study setting can provide, given the varying availability of tools to explore an adverse event and give a diagnosis. For example, surveillance for intussusception (invagination of one segment of intestine in a segment of distal intestine) in developing countries has to be able to capture clinical events that did not involve autopsy, surgery, or sophisticated imaging techniques. Additionally, clinical field trials for vaccines, including studies of vaccine safety, are often conducted in developing countries with the highest anticipated impact of future implementation of the vaccine (e.g., rotavirus vaccine or malaria vaccine).
Most public health case definitions for outcomes follow well-established, clinical definition formats as “definite,” “probable,” and “possible” cases. However, we deliberately avoided using this terminology with AEFIs, because this grading can apply equally to the degree of certainty about (1) the causal relation between immunization and the adverse event (e.g., is a seizure experienced after immunization definitely, probably, or possibly caused by a given vaccine?), or (2) the clinical adverse event outcome itself (e.g., does the clinical entity reported definitely, probably, or possibly constitute a seizure?). For the latter, if a gold standard is lacking, it can at most be defined with a higher “level of diagnostic certainty” if based on the best scientifically proven information available, in contrast to reports with less specific information or that lack objective scientific proof. To ensure that definitions are not used as a filter for reporting of events, events that are missing the information needed to meet even the lowest level of diagnostic certainty can be analyzed as an additional category.
Reconsideration and possible revision of selected case definitions and guidelines is envisioned every 3–5 years, as new evidence becomes available or on the basis of feedback during implementation.
It has become good common practice to evaluate public health programs or new interventions, including clinical criteria for diagnosis.47–53However, evaluation studies are often not performed of public health case definitions. The most notable exception in public health is for the HIV/AIDS surveillance case definitions put forth by CDC in the United States and WHO. These definitions were tested for their reliability, sensitivity, specificity, and positive predictive value in different socioeconomic settings, in different age groups, and also at the level of economic development.12–14, 54In vaccinology, vaccine studies showed that results, including the degree of vaccine efficacy, depended largely on the case definition for the VPD being targeted (e.g., laboratory-confirmed B. pertussis versus clinical diagnosis alone).55, 56Those studies were facilitated by the presence of a gold standard laboratory test against which any clinical criteria could be measured. Such a true gold standard does not exist for most AEFIs, necessitating different methodologies to evaluate Brighton Collaboration case definitions.
One study, by the Finnish National Public Health Institute, uses hospital discharge data to compare active (actively solicited) and passive (reporter initiated) surveillance to evaluate Brighton Collaboration case definitions for fever, generalized convulsive seizure, and hypotonic-hyporesponsive episodes. The study is being conducted in the framework of the European Program of Improved Vaccine Safety Surveillance, and preliminary results suggest a high interrater reliability of the case definitions and between passively received and actively collected information (greater than 95 percent) (personal communication Jan Bonhoeffer, April 2004).
| Level 1 of diagnostic certainty | Level 1 |
| In the presence of: | - Highest level of specificity |
| criterion 1:_______________ AND/OR | - Sensitive (least) for the respective AEFI |
| criterion 2:_______________ AND/OR | - Applicable primarily in clinical trials and |
| criterion n:_______________ | settings of active follow-up and settings with |
| more resources | |
| In the absence of: | |
| criterion a:_______________ AND/OR | |
| criterion b:_______________ AND/OR | |
| criterion n:_______________ | |
| Level 2 of diagnostic certainty | Level 2 |
| In the presence of: | - Intermediate level of specificity |
| criterion 1:_______________ AND/OR | - Sensitive (lower) for the respective AEFI |
| criterion 2:_______________ AND/OR | - Applicable in clinical trials, postmarketing |
| criterion n:_______________ | surveillance |
| In the absence of: | |
| criterion a:_______________ AND/OR | |
| criterion b:_______________ AND/OR | |
| criterion n:_______________ | |
| Level 3 of diagnostic certainty | Level 3 |
| In the presence of: | - Lower level of specificity |
| criterion 1:_______________ AND/OR | - Highly sensitive for the respective AEFI |
| criterion 2:_______________ AND/OR | - Applicable primarily in settings with less |
| criterion n:_______________ | resources in clinical trials and postmarketing |
| surveillance | |
| In the absence of: | |
| criterion a:_______________ AND/OR | |
| criterion b:_______________ AND/OR | |
| criterion n:_______________ | |
Finally, another study currently underway is designed to evaluate the reliability, applicability, and agreement between manual clinical reviews of reports submitted to the U.S. passive surveillance system for vaccine adverse events (the Vaccine Adverse Event Reporting System or VAERS) and automatic application of criteria in the respective Brighton Collaboration definitions to the same reports in the database. This study is similar in design to a previous study of a case definition for encephalitis37tailored for the needs of the National Vaccine Injury Compensation Program (http://www.hrsa.gov/osp/vicp/). Reviews of VAERS reports by independent clinical experts who are unaware of the respective standardized Brighton case definition are viewed as the gold standard for classifying a report as a “definite,” “probable,” or “possible” case, and are measured against automatic application of the criteria of the case definitions to the same VAERS reports. Complete study results will include a manual for the evaluation of case definitions in surveillance systems for use in other countries. First results show that the criteria of the case definitions may facilitate database searches to select true cases of a given adverse event while circumventing the need for time-consuming manual reviews of reports (personal communication Manya Magnus, August 2003). A similar evaluation of medical records from occupational health clinics, for a case definition for occupational asthma that was developed by the National Institute of Occupational Safety and Health, found that 46 percent of the initial physician's diagnosis of occupational asthma did not meet the proposed case definition after additional independent review of clinical charts by another clinician, and the definition was revised accordingly.57
In general, it needs to be kept in mind that our goal of comparability of data requires widespread use of standardized definitions, and evaluation is a secondary, albeit important, objective to achieve this goal.
Public health investigations and studies typically result in recommendations, which may or may not be followed. Often the implementation of a recommendation is and has to be voluntary (e.g., condom use for the prevention of sexually transmitted infections). Its success then needs to be supported by creation of an environment that facilitates implementation (e.g., making condoms available at no or low charge in high-risk bars and other public places). At other times, a recommendation can lead to a public health policy that is reinforced by provision of control laws (e.g., prohibition of smoking in public places and airplanes). Whether a recommendation receives widespread acceptance depends at least in part on its value to the intended users, environmental and legal circumstances, and the ease of implementation. In the field of drug safety, the most widespread acceptance of new reporting or analysis tools is achieved by regulation, while traditionally public health case definitions—particularly for infectious diseases—have been adopted based on need and on the merit of having been developed by well-respected public health organizations like CDC or WHO.
The same concept applies to implementation of case definitions for immunization safety as well as guidelines for data collection, analysis, and presentation developed as public health and research tools. First of all, easy access to the definition and guidelines documents has to be guaranteed, to make implementation possible. Brighton documents are available for download from the Brighton Collaboration Web site and also upon request from the Brighton secretariat. Potential users also need to be informed about the availability of documents, which can be done via scientific publications, e-mail announcements, newsletters of professional organizations, and other means. Finally, sponsors of clinical vaccine safety research and regulatory organizations need to be encouraged to consider implementation of the definitions and guidelines in the respective study protocols.
Development of training material for education on why and how to use the documents is anticipated. In clinical trials and for individual case reports in pre- and postlicensure settings, the case definition and guidelines can be used to develop data collection forms and assure that prospectively collected information on a given adverse event will reach a sufficient level of certainty of its diagnosis. In addition, use of these case definitions and guidelines will enable meaningful interpretation of the event and future comparability with reports of the same adverse event in other studies. In surveillance systems and epidemiologic studies, the documents can be used to guide followup on reported cases, thus ensuring collection of complete information, or to develop surveillance forms in systems that incorporate case definitions at the level of reporting, i.e., when the reporter is asked to report events that comply with a given case definition, often printed directly on the reporting form. In the United States, definitions or guidelines will likely be used to verify coded events in VAERS and to follow up on reports of AEFIs. Another envisioned possibility for Web-based reporting is to have a pop-up window with the respective items from the definition and data collection guidelines prompting the reporter for the information if key words (e.g., fever) are mentioned in the report.
As of April 2004, spontaneous requests of documents to the secretariat or downloads from the Brighton Web site have been made by 199 investigators from more than 30 countries, covering the range of study settings envisioned for use of the definitions (i.e., clinical trials, surveillance, and epidemiologic studies). Interested users are asked to use the download mechanism from the Brighton Web site (http://brightoncollaboration.org/en/index/aefi.html) to retrieve documents after a short registration process. This will enable the secretariat to provide registered users with updates on available and new case definitions and guidelines. It will also enable future followup for feedback on the applicability and usefulness of the documents in the respective settings. A more structured implementation strategy has been initiated based on interactions with key groups from regulatory authorities (e.g., the U.S. Food and Drug Administration) and global public health (e.g., WHO), professional (e.g., the American Academy of Pediatrics), clinical care and research (e.g., Walter Reed Medical Army Medical Center or the U.S. National Institutes of Health), and manufacturing organizations (e.g., European Vaccine Manufacturers).
We have shown that the current organizational infrastructure of the Brighton Collaboration and support of a large group of volunteers allow the development of new global standards for immunization safety in a short time. Participation of scientific stakeholders from all aspects of immunization safety and at all steps in the process, in a transparent and consensus-based manner, has proved to be a scientifically stimulating and (thus far) fruitful undertaking. And while vaccines are generally safe, the Brighton Collaboration provides tools that can be used to facilitate detection of any potential problems and, ideally, to prevent the spread of rumors and misinformation based on the lack of available definitions or the use of different and sometimes discrepant definitions. Future goals will be the continued development of case definition and guidelines documents, and enhanced efforts toward standardized evaluation and structured implementation of finalized documents in various geographic and research settings to enhance the quality of the available vaccine safety information. We anticipate that our concept of global collaboration and a scientific approach toward best-evidence-based standardization of public health tools such as case definitions and guidelines can serve as a model for other areas of medical interventions, especially other domains in patient safety, such as drug safety.58
We are grateful for the substantial commitment and input into our work by all of our voluntary working group and reference group participants, and particularly the team leaders of the respective working groups. We would also like to thank everybody using or evaluating our definitions and guidelines in their respective settings, as well as our project manager, Dr. Bakary Drammeh.
Disclaimer:This paper does not necessarily represent the official view of the U.S. Department of Health and Human Services.
Centers for Disease Control and Prevention, Atlanta, GA (KSK, RTC). University Children's Hospital Basel, Switzerland (JB, UH). Food and Drug Administration, Rockville, MD (MMB). World Health Organization, Geneva, Switzerland (PD). Swedish Institute for Infectious Disease Control, Stockholm, Sweden (HH). Aventis Pasteur, S.A, Lyon, France (EL). UCLA Center for Vaccine Research, Kaiser Foundation Hospital, Los Angeles California (SMM).
Address correspondence to: Katrin S. Kohl; CDC, National Immunization Program, 1600 Clifton Road, Mailstop E-61, Atlanta, GA 30333, USA. Phone: +1-404-639 8073; e-mail: kkohl@cdc.gov.
Free Full text in PMC]Free Full text in PMC]The current structure was chosen to (a) reflect oversight by representatives of the various stakeholders in immunization safety, i.e., clinical care professionals, academicians, and professionals from regulatory agencies, public organizations, and vaccine manufacturers; (b) have coordinators in at least two geographic regions, i.e., Europe and the United States; (c) separate visually and largely functionally the three steps in the process of global acceptability of newly developed definitions and guidelines, i.e., development, evaluation, and implementation of case definitions and guidelines; and (d) have most participants actively volunteer in working groups or the reference group developing or reviewing case definition and guidelines documents.
