Review Approach

The methods for the evidence review upon which this report is based follow the Methods Guide for Effectiveness and Comparative Effectiveness Reviews of the Evidence-based Practice Center (EPC) Program of the Agency for Healthcare Research and Quality (AHRQ), and reporting follows the Preferred Items for Reporting in Systematic Reviews and Meta-Analyses (PRISMA) guideline.^{39, 40}

Throughout the project, the evidence review team was supported by a Technical Expert Panel (TEP), a diverse panel of relevant stakeholders that included vaccine experts with particular clinical expertise in key populations (children, adults, older adults, and pregnant women), vaccine safety methodologists, and consumers. The TEP members were not responsible for the content of the evidence report, but they provided the review team with important perspectives and advice on key components of the systematic review.

This update builds on the prior 2014 AHRQ report,²⁸ which itself built upon the 2011 Institute of Medicine (IOM) consensus report.²⁹ The IOM report assessed the causality between specified adverse events and vaccines, based on both epidemiologic evidence and mechanistic evidence (from animal and individual human case studies). The prior 2014 report did not search for or include studies on vaccines that were reviewed in the IOM report and published prior to 2011. Similarly, in this update only for vaccines for which there were new indications (or for new vaccines) did we perform targeted searches for research published prior to 2014.

We searched MEDLINE^® (including TOXLINE), Embase^®, CINAHL^®, Cochrane CENTRAL (including International Clinical Trials Registry Platform registry), Web of Science, and Scopus, through November 9, 2020 for randomized controlled trials (RCTs), observational studies with a comparator, and studies that employed a self-controlled design. The searches built on the literature searches undertaken for the prior 2014 report. In addition, we reference-mined existing systematic reviews and Food and Drug Administration (FDA) reports; screened the trial registry, Clinicaltrials.gov; reviewed supplemental material submitted to AHRQ (following a Federal Register Notice posted regarding the availability of a portal for submission of unpublished studies); and consulted with content experts. The search strategy was not limited to specific adverse events and is documented in full in Appendix A.

The Key Questions were publicly posted on the AHRQ Effective Health Care website (https://effectivehealthcare.ahrq.gov/) to allow additional input. The final protocol was posted on the Effective Health Care website at https://effectivehealthcare.ahrq.gov/products/safety-vaccines/protocol and is registered with PROSPERO.⁴¹ The draft report has been similarly posted for public comment on the Effective Health Care website.

Analytic Framework

This report was limited to a safety assessment and focuses on reported adverse events associated with vaccines; the effectiveness of vaccines is outside the scope of this report. The analytic framework (see Figure 1) outlines the populations and corresponding KQ, the interventions (vaccines routinely recommended for use in the United States), and the outcomes (key adverse events for the population and vaccines) that were addressed in the evidence synthesis. The analytic framework indicates patient and vaccine factors that could affect the type, severity, and frequency of adverse events associated with different vaccines in different populations. While we analyze the results of studies that report race or ethnicity as a risk factor for adverse events in this report, we recognize that race or ethnicity themselves are not risk factors because they are not biologically distinct entities. Rather, racism and social determinants of health equity are a risk factor that may manifest as differences in outcomes based on race or ethnicity.

Figure 1

Analytic framework for safety of vaccines used for routine immunization in the United States. Abbreviations: KQ—Key Question

Study Selection

Studies of vaccines included in the Centers for Disease and Prevention (CDC’s) routine immunization schedules that reported presence or absence of adverse events in adults, children, and pregnant women were eligible for inclusion. To enable true assessments of risk, only studies with a comparator were included. We included comparisons to placebo, unvaccinated groups or time periods, or pre-vaccination status. We also included studies that compared new vaccines to the prior standard of care, meaning the previously available or closest vaccine formulation. Studies were excluded if they were non-English language or were published in abbreviated form (e.g., conference abstracts). No studies were excluded based on setting, timing of follow-up, or risk of bias. English language studies conducted outside of the United States were included if the vaccines studied were part of the current CDC immunization schedules and the formulations were approved for use in the United States. We included trial records from sources such as the Clinicaltrials.gov database, even in the absence of a corresponding publication, from which we extracted all serious adverse events and all adverse events graded as 3 or higher in severity according to Common Terminology Criteria for Adverse Events (CTCAE) criteria or described by study authors as severe, as well as deaths. Detailed inclusion and exclusion criteria are documented in Appendix A.

Titles and abstracts of studies identified by the searches described in Appendix A as well as those suggested by the TEP or public commenters were screened by two independent members of the research team using DistillerSR, an online data extraction program. Full-text versions were obtained for all studies identified for potential inclusion by either team member. Two reviewers independently screened full text publications; disagreements were resolved through discussion. Publications reporting on the same participants were consolidated into a single record for analysis. Excluded studies are listed in Appendix B.

Data Extraction and Risk of Bias Assessment

Study-level and outcome data were extracted for each study using a detailed data extraction form in online software for systematic reviews. Data were extracted by an experienced subject matter expert and were checked by methodologists and literature reviewers for accuracy and consistency across studies. Data extraction followed a specific algorithm detailed in Appendix A. We extracted some adverse events by outcome category (e.g., cardiovascular events). Because some patients may have experienced multiple events within an outcome category, if studies listed more than one relevant adverse event in a given outcome category then we selected the most commonly reported adverse event in the category. Given that some studies reported a large number of outcomes, for each Key Question and vaccine, we prioritized key adverse events to allow for synthesis across studies; these are included in summary of findings tables to assess the strength of the evidence. These key adverse events were identified a priori with the help of the TEP, with content expert input, and informed by published literature. The selection process and the included key adverse events are listed in Appendix A.

All included studies were assessed for sources of bias that could have influenced the reported results.⁴² We used the McHarm scale, a tool for structured critical appraisal of adverse event data reported in research studies, for the assessment. Adverse event assessment and reporting are often lacking in rigor; thus, we applied critical appraisal criteria assessing two main domains:^{42, 43} data collection of adverse events and reporting of adverse events.

Study details, findings of all included studies, the risk of bias domains, and the results of the risk of bias assessment are documented in Appendix C. All studies included in the prior 2014 report on which this report builds on are documented in detail in Appendix C in the 2014 report.²⁸

The evidence tables (included in Appendix D) report all extracted outcomes in the individual studies, but the strength of evidence (SoE) assessment considered only key adverse events to evaluate the overall safety of the vaccines across studies. We reported on the longest follow-up available for each study to capture all relevant events.

Data Synthesis and Analysis

Data synthesis and analysis were conducted according to the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews.⁴⁴ The synthesis is organized by KQ, then by type of vaccine, then by type of adverse event. Summary tables document evidence across studies. The synthesis documents the presence as well as the absence of adverse events. We report the relative frequency and severity of the adverse events and the SoE for the presence or absence of specific adverse events.

To answer KQ1a, KQ2a, and KQ3a, we documented the adverse events that have been assessed in the included studies.

To address KQ1b, KQ2b, and KQ3b, we documented the adverse events that were reported in the included studies.

To address KQ1c1, KQ2c1, and KQ3c1, where possible, we described the severity of the adverse events for which there was increased risk across included studies.⁴⁵ We extracted data using the CTCAE rating system, which includes 26 system categories and five severity grades ranging from 1 (mild) to 5 (death due to the event), as described in Appendix A.

To address KQ1c2, KQ2c2, and KQ3c2, we documented how many times the event occurred in a vaccinated group compared to a control group, and the range of possible effects based on the confidence interval surrounding the point estimate, where available, across studies. In addition, we characterized the severity and frequency of the events associated with the vaccines wherever the information was available.

To address KQ1c3, KQ2c3, and KQ3c3, we documented potential risk factors for adverse events reported in included studies; we had planned to explore subgroups and meta-regressions, but insufficient data were available.

We computed the effect size and 95 percent confidence interval (CI) for all individual studies and key adverse events. In a first step, we calculated the RR of an adverse event comparing a vaccinated group to a control group. Many adverse events of interest are not vaccine-specific, such as allergies and infections, and we first determined whether there are statistically more incidences of these events in the vaccinated group compared to a control group. Where studies did not report sufficient detail, we reported the metrics authors provided, which ranged from verbal statements of no difference, to p-values, to effect measures such as odds ratios.

For each key adverse event, we combined estimates across studies in random effects meta-analyses using Hartung-Knapp correction of standard errors where appropriate. For studies with zero events in one group, we added a constant to the empty cell to enable computation. We determined the most appropriate meta-analysis model as outlined in Appendix A, given that for many adverse events only a small number of studies was available, studies reported on rare events, and several studies reported zero events.^46–49

For the synthesis of whether there is increased risk of adverse events, we reviewed the RR estimates across studies. In addition to the relative effect, we also documented the actual incidence rates, the sample sizes, and the resulting rates of adverse events in the vaccinated and the control groups for all individual studies. All findings of more adverse events in the vaccinated group were reviewed further and described in detail in the narrative synthesis. We also reviewed the risks reported in individual studies and highlighted all instances where individual studies had reported a statistically significant risk, both in the summary of findings tables as well as the narrative synthesis. Finally, since many estimates were very imprecise (given the small number of reported events and the small number of samples from which conclusions for the true risk could be estimated) the narrative synthesis also reported observed rates to transparently document the available evidence.

The results section describes in detail all new evidence identified since the 2014 report, followed by a synthesis across all available evidence.

Grading the Strength of the Body of Evidence

The strength of the body of evidence (SoE) was assessed based on the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach and the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews.⁵⁰ For each KQ, we selected key adverse events that are documented in summary of findings tables to assess the SoE. The evidence tables in the appendix report all outcomes addressed in the individual studies, but the SoE assessment used the a priori determined list of adverse events to evaluate the overall safety of the vaccines across studies. We used four criteria to grade the SoE: study limitations, consistency, precision, and reporting bias. The outcomes and criteria for grading the SoE are described in detail in Appendix A. We differentiated high, moderate, low, and insufficient evidence to communicate the confidence in the summary estimates for the findings across studies. Briefly, high indicates high confidence that the evidence reflects the true effect, moderate indicates moderate confidence that the evidence reflects the true effect. Low indicates low confidence that the evidence reflects the true effect. Insufficient indicates that evidence either is unavailable due to a lack of studies reporting on the outcomes, or the evidence does not permit a conclusion (e.g., due to conflicting results across studies or methodologic flaws).

The summary of findings tables document the results across studies grouped by vaccines. For each key adverse event, we report the RR and CI, where available, of the adverse event among those who received the vaccine of interest compared to controls who did not receive the vaccine. The summary of findings tables include the relative effect across studies as well as the absolute rate of adverse events—number of events and assessed participants in the vaccine and control groups—reported in the individual studies. In addition, the summary of findings tables summarize any other studies that could contribute to the SoE but that could not be combined into the pooled risk estimates. We integrated data for vaccines still in use from studies identified in the 2014 report for which we could calculate the RR of an adverse event comparing a vaccinated group to a control group. In addition, we assessed all findings for which there was a graded SoE in the prior 2014 report and integrated any new evidence. The summary of findings tables document the reasons for downgrading the SoE where applicable; and the findings for the outcomes of interest together with a grading of the SoE. The summary of findings tables are followed by summary tables comparing the evidence identified in the 2014 report and the update, followed by a synthesis of SoE and findings.