Understanding COVID-19 Vaccine Effectiveness Against Death Using a Novel Measure: COVID Excess Mortality Percentage

COVID-19 vaccines have saved millions of lives and prevented countless adverse patient disease outcomes. Understanding the long-term effectiveness of these vaccines is imperative to developing recommendations for precautions and booster doses. Comparisons between more and less vaccinated groups may be misleading due to selection bias, as these groups may differ in underlying health status and thus risk of adverse COVID-19 outcomes. We study all adult deaths over April 1, 2021-June 30, 2022 in Milwaukee County, Wisconsin, linked to vaccination records, use mortality from other natural causes to proxy for underlying health, and report relative COVID-19 mortality risk (RMR) for vaccinees versus the unvaccinated, using a novel outcome measure that controls for selection effects. This measure, COVID Excess Mortality Percentage (CEMP) uses the non-COVID natural mortality rate (Non-Covid-NMR) as a measure of population risk of COVID mortality without vaccination. We validate this measure during the pre-vaccine period (r = 0.97) and demonstrate that selection effects are large, with Non-Covid-NMRs for two-dose vaccinees less than half those for the unvaccinated, and Non-COVID NMRs still lower for three dose (booster) recipients. Progressive waning of two-dose effectiveness is observed, with relative mortality risk (RMR) for two-dose vaccinees aged 60+ versus the unvaccinated of 11% during April-June 2021, rising steadily to 36% during the Omicron period (January-June, 2022). Notably, a booster dose reduced RMR to 10–11% for ages 60+. Boosters thus provide important additional protection against mortality.


Introduction
COVID-19 vaccines have saved millions of lives, and mortality rates have fallen in 2022. However, U.S. mortality alone is still running at an annual rate of over 100,000. It is important to understand the mortality risk faced by the vaccinated, how protection varies with age, time since vaccination and other factors, and the bene t of booster doses.
Many studies have reported evidence on vaccine effectiveness (VE) against infection, hospitalization, and death (for succinctness, we cite primarily reviews). 1,2 However, most of these studies face potential confounding due to selection bias. Most studies of effectiveness against mortality have limited controls for individual characteristics, often only age and gender, [3][4][5][6][7][8][9] or study short time periods. 5,10-12 U.S.-based studies typically lack population-level data, 7,[13][14][15] and datasets with information about vaccination status often have limited information about health status. Some use a test-negative design, 10,14,15 which is prone to selection bias. 16 If healthier people are more likely to be vaccinated, lower vaccinee mortality rates for vaccinees may in part re ect their (unobserved) better health and thus lower inherent risk.
In this study, we propose a novel method for evaluating COVID-19 vaccine effectiveness against mortality, which uses non-COVID natural mortality rates as a surrogate for underlying health. This surrogate is attractive as non-COVID mortality, COVID mortality, and vaccination status are often available for the same population. We propose, as an outcome measure, the COVID Excess Mortality Percentage (CEMP), de ned as COVID-19 deaths divided by non-COVID natural deaths, converted to a percentage. The CEMP denominator controls for differences in population health between two groups, such as vaccinated versus unvaccinated.
We provide evidence on selection bias, measure the magnitude of this bias, and validate the CEMP measure as a means of reducing selection bias. We then use CEMP as an outcome to measure VE against death for the entire adult population of a large Midwestern city. We compare relative mortality risk (RMR = 1-VE) for vaccinated versus unvaccinated, using this measure, to previous approaches that do not include robust adjustment for underlying health status.

Data And Methods
We obtain linked, de-identi ed mortality and vaccination records for all adults aged 18 + in Milwaukee County, Wisconsin (adult population 722,000), for January 1, 2021, through June 30, 2022, including residence zip code, age at death, gender, race/ethnicity, education, income, marital status, veteran status, manner of death, and text elds for cause of death and conditions leading to death. We use text analysis to identify deaths due to COVID-19 versus other natural causes; this counts more COVID-19 deaths than relying on ICD-10 cause-of-death codes prepared by the National Center for Health Statistics (NCHS), see Appendix for details. We treat vaccination as effective against mortality beginning 30 days after receipt; this time period allows for a vaccine dose to become fully effective, as well as the typical several-week lag between infection and death. We exclude immune-compromised decedents.
The mRNA vaccines (Moderna, P zer) use two initial does; J&J uses one dose. We report results based on number of doses, thus treating one J&J dose as equivalent to one mRNA dose, but obtain similar results if we exclude J&J vaccinees.
We de ne CEMP, VE versus the unvaccinated, and relative mortality risk after vaccination (RMR) in each time period, within a population group, as: RMR can be obtained by comparing mortality rates for both groups, or as an odds ratio from logistic regression for a population containing both groups. We also compute RMR and VE for two-versus-one-dose and three-versus-two-dose vaccinees. CEMP, VE and RMR involve ratios, so could be unde ned if the denominator is zero; we did not have this issue with our data and population groups.
CEMP represents the odds, for natural-cause decedents, of dying from COVID-19 versus other natural causes. The ratio of CEMPs for two groups, such as vaccinated versus unvaccinated, is an odds ratio, obtainable from logistic regression. We both conduct simple comparisons of CEMP for two groups, de ned by age, gender, and vaccination status, and conduct multivariate logistic regression analysis of the association between vaccination and RMR, in which we adjust for other variables, available from death certi cates, that may be associated with mortality risk. The predictors in this analysis are age, age 2 , zip-code-level socio-economic status (zip-SES), gender, race/ethnicity, education level, marital status, and military veteran status.
We measure race/ethnicity as non-Hispanic White ("White"), Black, non-Black Hispanic ("Hispanic") and Other (including Asian, Native American, and mixed race). We measure zip-SES using the Graham Social Deprivation Index, which we have found to perform well in other work. 17 We estimate population for 2020 from the American Community Survey, measure the number of vaccinees; and assume other persons are unvaccinated. We measure the Non-COVID-19 natural mortality rate (Non-Covid-NMR or NCNMR) for a group as natural deaths divided by population.
CEMP treats the non-COVID natural mortality rate as a proxy for overall health of a given group, and thus the likelihood of mortality if not vaccinated. We assessed the validity of this approach out-of-sample, by studying the correlation in Indiana (a nearby state where we have mortality data), between natural mortality in April-December 2019 (pre-COVID) and COVID-19 mortality over April-December 2020 (same months during the pre-vaccine period) Using 2019 natural mortality (rather than 2020 non-COVID natural mortality) to predict 2020 COVID mortality avoids the mechanical correlation which could arise if COVID deaths were undercounted, or prior COVID infection leads to higher non-COVID mortality.
Our RMR estimates will be biased only if both: (i) COVID-19 mortality is undercounted, and (ii) the degree of undercounting differs systematically between vaccinated and unvaccinated persons. We assess condition (i) --whether signi cant undercounting existsas follows. We use natural mortality in Wisconsin over 2017-2019 to predict non-COVID natural mortality in the same month in 2020, using linear extrapolation, and compare predicted to measured non-COVID natural mortality in 2020. We assess whether measured non-COVID natural mortality exceeds predicted mortality, either overall, or during periods of high COVID mortality. Even if undercounting exists, we have no reason to expect that condition (ii) holds, but cannot provide evidence on this with our data.

Results
Validating the CEMP Measure Figure 1 shows the correlation in Indiana between natural mortality in April-December 2019 (pre-COVID period) and COVID-19 mortality in April-December 2020 (COVID period, but pre-vaccine), for population groups de ned by age (groups are 18-39, 40-49, 50-59, 60-69, 70-79, 80-89, and 90+, gender, race/ethnicity, and zip-SES. The Pearson correlation coe cient is 0.97, consistent with non-COVID natural mortality rates strongly predicting COVID mortality rates for unvaccinated persons. Further validation comes from the multivariate regression analysis discussed below, in which RMR estimates within groups de ned solely by age are similar to multivariate estimates that adjust for other factors that are associated with COVID-19 mortality. This suggests that the CEMP measure already controls well for population health.  Evidence for Selection Effects Table 1 provides evidence on differences in baseline health, using Non-Covid-NMR as a surrogate for health, between three-dose vaccinees, two-dose vaccinees, and the unvaccinated. The table reports Non-Covid-NMRs by age group, vaccination status, and time period, and the ratios of Non-COVID-NMR for 2-dose and 3-dose vaccinees to that for the unvaccinated. The table uses *'s to report statistical signi cance for the ratios, see Table App-13 for 95% con dence intervals (CIs). Vaccinees have substantially lower Non-Covid-NMRs, consistent with vaccinees being healthier on average, and thus likely facing lower baseline COVID mortality risk.
In the booster-available periods, there is a further separation, in which some persons who previously received two doses chose to obtain a third, booster dose, while others did not. This choice also involves strong selection effects. Three-dose recipients have lower Non-Covid-NMRs than two-dose recipients. This difference is largest for ages 60-79, for whom Non-COVID-NMR is 1.632% for twodose recipients versus 0.797% for 3-dose recipients.
Appendix Figure App-2 provides information on Milwaukee County vaccination rates, which are broadly in line with national averages. Overall, around 74% of the adult population received at least one dose, 70% were fully vaccinated (one J&J dose or two mRNA doses), and of two-dose recipients, and 56% received a third dose with higher rates for older persons.
CEMP and RMR by Time Period and Age Range: Overview Table 2

RMR for One Dose
One-dose RMR versus the unvaccinated has been rarely studied. RMR relative to unvaccinated is substantial, at 57%, 58%, 29%, and 43% across our four time periods. One-dose RMR was similar in older and younger individuals. One-dose RMR, unlike two-dose RMR, did not exhibit waning. Thus, over time, as the bene t from a second dose, in reducing RMR has been shrinking.

Multivariate Estimates
In

Discussion
A central need, when estimating how COVID vaccination affects mortality, is to estimate the counterfactual: What would COVID-19 mortality have been for the vaccinated, if they had not been vaccinated? We use Non-Covid-NMR as a proxy for background mortality risk, and nd important differences in background mortality risk between vaccinated and unvaccinated and between two-dose and three-dose recipients. By this measure, two-dose recipients are healthier (have lower non-COVID-NMR) than the unvaccinated, and three-dose vaccinees are healthier than two-dose vaccinees. These selection effects, unless controlled for (through our CEMP measure or in another way) can produce large biases in VE estimates.
Non-Covid-NMR does an excellent job of predicting COVID-19 mortality during the pre-vaccine period, out-of-sample in Indiana (Fig. 1). It performs similarly well in sample, both for Wisconsin as a whole, and for Milwaukee County (Appendix Figure App-1). This suggests that using CEMP as the outcome when measuring RMR provides a good estimate of the protective effects of vaccination relative to the counterfactual. The similarity between unadjusted RMR estimates ( Table 2) and multivariate estimates (Table 3) provides further evidence that the CEMP denominator does a good job of controlling for underlying health and mortality risk.
The data on Non-Covid-NMR ratios in Table 1 can be used to assess the extent of selection bias: Assume counterfactually that vaccination was useless against Covid mortality. What RMRs would one estimate, controlling only for age group? Given the high correlation between Non-Covid-NMR and COVID-19 mortality for the unvaccinated, the RMR ratios of vaccinated to unvaccinated in Advantages of the CEMP Measure CEMP, as a measure of COVID-19 mortality, has attractive features relative to other measures. It relies only on death certi cates, which are available for all decedents, but addresses selection effects, by using non-COVID NMR to proxy for population health, which is otherwise di cult to observe. An alternative approach, controlling for comorbidities captured in electronic health records, faces important limitations: comorbidity data may not be fully reported, and, in the U.S., population-level data on comorbidities is not available. Even studies that control for comorbidities often examine only people who seek medical care for COVID-19 infection. [8,20] This will miss the association among underlying health, who becomes infected, and infection severity.
Using data only on decedents also avoids the challenges in estimating the population at risk. Population statistics may undercount some groups because of non-participation in the Census or the American Community Survey, or inaccurate data. While race/ethnicity can be inaccurately captured in death-certi cate data, it is unlikely that inaccuracies will differ systematically between those who die of COVID-19 versus other natural causes.

Overview of Results: Substantial RMRs, Large Value for Boosters
Our analysis provides a number of insights for vaccine effectiveness against mortality and selection effects in who gets vaccinated. First, our two-dose RMRs versus unvaccinated are substantially higher (VE is lower) than in other studies. The higher RMRs re ect our use of CEMP to address selection bias, as well as continued vaccine waning in the Omicron period.
The studies covered by the available systematic reviews report lower two-dose RMRs, from 6-17%, as compared to this study. 1,2 We found only one study that reports similar two-dose RMRs for the pre-Omicron, pre-booster period. This study nds an 18% RMR for fully vaccinated U.S. veterans (two mRNA or one J&J) for ages < 65 and 28% for ages 65 + . 14 Likely not coincidentally, this study uses the rich VA data to control for an extensive set of comorbidities; it also nds that vaccinees have lower all-cause mortality. We found only one other U.S. study that reports RMR from linked, population-wide mortality and vaccination data. A study of Puerto Rico through mid-October 2021 (thus pre-Omicron and pre-booster), reports two-dose RMR after 144 days (longest period considered) of 14% for P zer and 7% for Moderna, versus 3% and 1% soon after vaccination. 8 This study does not control for selection effects. We did not nd studies of VE against mortality covering the Omicron period.
Second, we nd substantial waning of two-dose protection against mortality, with two-dose RMR versus unvaccinated for ages 60 + increasing from 11% in 2Q-2021 to 34% in 1H-2022. This contrasts with prior studies, which typically report limited waning against severe disease and death. 1,2 Note that we cannot separate the effects of waning over time from differences in protection against different virus variants.
Third, we nd that two-dose RMR increases with age, but boosters provide substantial additional protection. This makes boosters especially important for the elderly, particularly those age 80+. For ages 60 + three-dose RMR is 10% in 4Q-2021 and 11% in 1H-2022, versus 28% and 34% for two-doses. At the same time, three-dose RMRs are higher than reported in prior booster studies, 10 again showing the importance of controlling for selection effects. Nonetheless, three-versus-two-dose differences in RMRs for ages 60 + are large, at 18% in 4Q-2021 and 23% in 1H-2022. The reduction in RMR is even higher for ages 80+, at 32% for 1H-2022. In effect, the higher two-dose RMRs that we nd leave more room for boosters to reduce mortality, even though we also nd higher three-dose RMRs than prior research. Our evidence supports public health messaging and policy that encourages boosters for the elderly.
Fourth, we nd stronger relative two-dose protection for ages 18-59 in 2021, compared to older persons in the pre-Omicron period, but not in the Omicron period. Absolute COVID mortality risk after two doses is smaller for younger persons, but boosters are highly effective in reducing that risk: We nd zero deaths among younger three-dose recipients. Our results for the Omicron period contrast to the perception among many younger persons that two doses provide su cient protection.
Fifth, we nd that a single dose provides only moderate protection, with RMR versus the unvaccinated around 50%. However, this protection appears to be long-lived. Limited waning has been reported before for the single-dose J&J vaccine. 7, 21 We nd similar results for one-dose mRNA recipients.

An Opportunity for Targeted Booster Messaging
Evidence of vaccine waning rst appeared in mid-2021, initially from Israel. Based on this evidence, Israel launched a booster campaign in late July 2021, which reached the whole population by the end of August. 22 Other countries soon followed, relying in part in Israeli evidence that boosters added important value. In the U.S., however, FDA scientists publicly questioned the need for boosters. 23 An advisory committee to the Food and Drug Administration (FDA) in September 2021 approved only a limited rollout to the elderly and persons at risk due to occupational exposure; 24 similarly, an advisory committee to the CDC endorsed boosters only for the elderly. 25 Two months later, the FDA and CDC approved boosters for all adults; although a CDC recommendation came only at the end of November, 2021. 26 Public health messaging remained muddled, with the value of boosters "lost in the sea of changing recommendations and guidance," 27 Even today, U.S. booster percentages lag many other countries, 28,29 and public knowledge of booster recommendations is limited. 30 Our study provides strong evidence on booster value for ages 60+, which account for the vast majority of COVID-19 deaths, and nearly all vaccinee deaths.

Toward Enhanced Public Reporting of COVID-19 Mortality
Although many public sources report data on COVID-19 deaths. None reports a comparison to other natural deaths. Reporting both COVID-19 and non-COVID natural deaths, and ideally CEMP (the ratio of the two) would provide valuable information on the risk of death from COVID-19 versus other natural causes. Reporting CEMP would show that the unvaccinated face substantial COVID-19 mortality risk, even at younger ages. This might make more salient the large reductions in mortality risk from vaccination, and the value of a booster dose. Reporting both COVID-19 mortality and other natural mortality could also focus attention to selection effects, and their importance when estimating vaccine effectiveness.

Limitations
This study has important limitations. We study only mortality. COVID-19 mortality is uncommon for younger persons, which limits statistical power. Our data is only for Milwaukee County, which is racially, ethnically, and economically diverse, but its COVID-19 experience may not be representative of other areas.
We do not observe, and thus cannot control for, prior COVID-19 infection. Especially in the Omicron era, many people, both vaccinated and unvaccinated, have already been infected. For them, VE can be understood as measuring the extra protection from hybrid immunity (from prior infection plus vaccination) versus natural immunity alone (from prior infection).
We do not observe individual health characteristics, except through the limited lens of death certi cates. There could be vaccinatedvs-unvaccinated differences that affect COVID-19 mortality, not re ected in Non-Covid-NMR.
The CEMP measure has several inherent limitations. Though likely a reasonable proxy for overall health, it does not consider behavioral or other differences between more and less vaccinated groups. Behavioral differences are likely to exist. For example. the unvaccinated or less-vaccinated, may believe COVID is less severe than the maximally vaccinated, and therefore take fewer precautions. Conversely, vaccinated persons may accept greater risks of becoming infected, because they believe they are protected against serious illness. Neither ours nor other VE studies can control for behavioral differences.
The CEMP measure implicitly assumes that COVID-19 infection does not meaningfully affect non-COVID mortality. Yet COVID-19 infection is known to predict higher post-infection mortality from other causes, at least in the near term. 31 This will cause downward bias in CEMP values. If this bias is similar for the vaccinated and unvaccinated, RMR estimates should still be unbiased. The downward bias in CEMP could be larger for the unvaccinated, who will on average face more severe COVID-19. If so, then RMR estimates based on CEMP will be somewhat below those we would estimate if we could attribute to COVID-19 these extra natural deaths. However, any bias is too small to be visible in Fig. 2.
COVID-19 deaths could be underreported, but we coded COVID-19 deaths based on reading death certi cates; this produced signi cantly larger counts than ICD-10 codes from the NCHS. Any remaining undercount appears small (Fig. 2).
This study assesses vaccine effectiveness only against mortality, not against other important adverse outcomes including hospitalization and long-term symptoms. A full understanding of vaccine effectiveness must include a broader range of outcomes.

Conclusion
We use a novel outcome measure, CEMP, to study how vaccination affects COVID-19 mortality risk. This measure uses mortality from other natural causes to control for selection effects in who gets vaccinated. We nd substantially lower non-COVID natural mortality risk for vaccinated than for unvaccinated persons. Thus, though vaccination provides very substantial protection against mortality, the vaccinated would likely face lower COVID-19 risk even if not vaccinated. After controlling for these selection effects, we nd increasing two-dose RMR over time, and large differences in RMR after two doses between younger (age 18-59) and older (age 60+) people. These ndings imply that boosters are highly important in reducing mortality, especially for ages 60+. The RMRs after twodose vaccination, and the meaningful although smaller three-dose RMRs for ages 60+, imply that non-vaccine mitigation strategies remain an important tool in reducing mortality in vaccinated populations, particularly among the elderly.

Supplementary Files
This is a list of supplementary les associated with this preprint. Click to download. VEMKEappendix20221208bbsubmit.docx