Ascertainment of vaccination status by self‐report versus source documentation: Impact on measuring COVID‐19 vaccine effectiveness

Abstract Background During the COVID‐19 pandemic, self‐reported COVID‐19 vaccination might facilitate rapid evaluations of vaccine effectiveness (VE) when source documentation (e.g., immunization information systems [IIS]) is not readily available. We evaluated the concordance of COVID‐19 vaccination status ascertained by self‐report versus source documentation and its impact on VE estimates. Methods Hospitalized adults (≥18 years) admitted to 18 U.S. medical centers March–June 2021 were enrolled, including COVID‐19 cases and SARS‐CoV‐2 negative controls. Patients were interviewed about COVID‐19 vaccination. Abstractors simultaneously searched IIS, medical records, and other sources for vaccination information. To compare vaccination status by self‐report and documentation, we estimated percent agreement and unweighted kappa with 95% confidence intervals (CIs). We then calculated VE in preventing COVID‐19 hospitalization of full vaccination (2 doses of mRNA product ≥14 days prior to illness onset) independently using data from self‐report or source documentation. Results Of 2520 patients, 594 (24%) did not have self‐reported vaccination information to assign vaccination group; these patients tended to be more severely ill. Among 1924 patients with both self‐report and source documentation information, 95.0% (95% CI: 93.9–95.9%) agreement was observed, with a kappa of 0.9127 (95% CI: 0.9109–0.9145). VE was 86% (95% CI: 81–90%) by self‐report data only and 85% (95% CI: 81‐89%) by source documentation data only. Conclusions Approximately one‐quarter of hospitalized patients could not provide self‐report COVID‐19 vaccination status. Among patients with self‐report information, there was high concordance with source documented status. Self‐report may be a reasonable source of COVID‐19 vaccination information for timely VE assessment for public health action.

To ascertain COVID-19 vaccination status, post-marketing evaluations have used various data sources typically categorized as self-report or source documentation of vaccination. In the United States, documented sources of vaccination typically include computerized immunization information systems (IIS) (also known as vaccine registries), electronic medical records, pharmacy records, occupational health records, and vaccination record cards. [5][6][7][8] For COVID-19 vaccination providers, reporting of vaccination to IIS is generally required. 9,10 However, the compilation and subsequent analysis of VE data can be compromised by receipt of vaccines in a different jurisdiction than SARS-CoV-2 testing or hospitalization, loss of vaccination cards, timeliness of reporting to electronic systems, and lack of access to IIS and other documented sources among study investigators.
Therefore, self-report of COVID-19 vaccination, defined as the patient or a proxy verbally providing a history of prior vaccination, might serve as a useful source of COVID-19 vaccination data during the pandemic and facilitate timely evaluations of VE. However, relying on self-report of COVID-19 vaccination could also result in misclassification of vaccination status through recall bias, social desirability bias, 11 or other mechanisms. To inform the use of self-report versus source documentation for estimating VE against COVID-19-related hospitalization, we evaluated concordance of COVID-19 vaccination status ascertained from these sources among hospitalized adults in a multistate VE surveillance network during the first few months of the COVID-19 vaccine roll-out.

| Participants and setting
To evaluate ascertainment of COVID-19 vaccination status by selfreport versus source documentation, we used data from the Influenza and Other Viruses in the Acutely Ill (IVY) network, a multistate network that conducts analyses of real-world VE against COVID-19 hospitalizations among adults. 12 In brief, this analysis included hospitalized adults (aged ≥18 years) from 18 academic medical centers in 16 states with admission dates from March 11 to June 6, 2021.
Trained medical abstractors reviewed hospital admission logs or medical records daily at participating sites to identify hospitalized patients who had received clinical testing for acute SARS-CoV-2 infection.
Patients who tested positive for SARS-CoV-2 by reverse transcription (RT)-PCR and met criteria for COVID-19-like illness were included as COVID-19 cases. We used two control groups, which were combined in this analysis. Test-negative controls included adults hospitalized with COVID-19-like illness who tested negative for SARS-CoV-2 by RT-PCR. We also included a secondary control group without COVID-19-like illness who tested negative for SARS-CoV-2 by RT-PCR (syndrome-negative controls).

| COVID-19 vaccination status comparison by data source
We compared COVID-19 vaccination status determined through self-report with vaccination status determined through source documentation. To mirror the vaccination groups used in previous VE analyses using these data, 12  To compare COVID-19 vaccination status by self-report and source documentation, we estimated the percent agreement and unweighted kappa with 95% confidence intervals (CIs). To be considered vaccinated by self-report, the patient or proxy needed to provide both a date and location of vaccination. For patients or proxies who reported vaccination but did not know the vaccine product, we assumed the patient received an mRNA vaccine as these vaccine products accounted for most vaccines administered in the United States during the surveillance period. 1 We performed unadjusted logistic regression to identify any associations between COVID-19 vaccination status discordance (self-report versus source documentation) and any of the following: age, sex, race/ethnicity, U.S. Census region, health insurance, interview of patient versus other (proxy, mix of patient or proxy, or unspecified), education (some college or more versus less than college), or employment status.
Syndrome-negative controls did not have illness onset dates because they were admitted to the hospital for reasons other than an acute respiratory illness; for these patients, we used the date of hospital admission as the reference date of illness onset for determining vaccination group.
Patients who had unknown COVID-19 vaccination status or could not provide self-reported dates of vaccination or could not be assigned to a vaccination group were excluded from concordance analyses but were compared by demographic characteristics with patients whose vaccination status and date could be classified by either self-report or source documentation. Additionally, patients who participated in blinded COVID-19 vaccine randomized clinical trials were excluded.

| COVID-19 VE comparison
In prior analyses of mRNA COVID-19 VE, 5,12-14 we primarily considered a patient vaccinated based on source documentation but accepted self-report of vaccination when vaccination status from documented sources was missing. In the case of vaccination dates or products that differed between self-report and documented sources, data from documented sources were used. For this analysis, we compared VE estimates obtained using (1) self-reported data only or estimates, we excluded patients tested for SARS-CoV-2 >10 days after illness onset, hospitalized >14 days after illness onset, who received a non-mRNA vaccine product (for which sample size was insufficient for analysis), for whom the vaccine product was not known, and who were enrolled in the syndrome-negative control group but later tested positive for SARS-CoV-2 infection.

| Patient characteristics of those included in vaccination concordance analysis
Of 2520 hospitalized cases and controls, 596 (24%) were excluded from the COVID-19 vaccination status concordance analysis because self-report data were missing or incomplete (n = 594) or the patient was previously enrolled in a blinded COVID-19 vaccine trial (n = 2) ( Figure 1A). Compared with patients who were able to provide a self-reported vaccination history, patients with missing self-reported vaccination data were more likely to be older (median age of 61 vs.

| Comparison of vaccination status by data source
High agreement in COVID-19 vaccination status by vaccination group was observed between the self-report group and source documenta-  Figure 1B). VE against COVID-19-associated hospitalization for full mRNA vaccination was 85% (95% CI: 81-89%) using vaccination status from source documentation only and 86% (95% CI: 81-90%) using vaccination status from self-report only, with overlapping CIs ( Among patients with discordant vaccination status, patients who were female, had a high-school education or less, or were unemployed were more likely to have discordance between self-report and documented COVID-19 vaccination status. Lower education and unemployment have previously been associated with discordance between vaccination sources. 28 There may be other factors that could explain discordance between sources such as recall bias, misclassification of vaccines, and interviewer biases. However, due to the timing of this analysis where the median time between full vaccination and study interview was 45 days, and given the global attention of COVID-19, recall biases may be less relevant to this analysis, but analyses examining further time since vaccination and after the introduction of booster doses may be subject to these biases. 11,18 Further, interviews were conducted consistently by trained staff across sites with regular network-wide and site-specific meetings to maintain consistency in interviews and data entry across the network to prevent misclassification of vaccines and interview biases.
This analysis is subject to limitations. First, this analysis was limited to hospitalized patients, and results may be different in outpatients. In addition, the concordance analysis could produce different results based on the distribution of excluded patients across case and control groups. Second, the patients in the group with missing selfreport data were different than the patients included in our analysis.
Among those missing self-report data, there was a higher percentage of patients for whom documented vaccination was available, so the VE estimate for this group may be different than patients included in our self-report VE analysis. 11

DATA AVAILABILITY STATEMENT
The data that support the findings of this study are available from the corresponding author upon reasonable request. The data are not publicly available due to privacy or ethical restrictions.