Abstract
Background
The accuracy of self-reported vaccination status is important to guide real-world vaccine effectiveness studies and policy making in jurisdictions where access to electronic vaccine registries is restricted.
Objective
This study aimed to determine the accuracy of self-reported vaccination status and reliability of the self-reported number of doses, brand, and time of vaccine administration.
Methods
This diagnostic accuracy study was completed by the Canadian COVID-19 Emergency Department Rapid Response Network. We enrolled consecutive patients presenting to 4 emergency departments (EDs) in Québec between March 24, 2020, and December 25, 2021. We included adult patients who were able to consent, could speak English or French, and had a proven COVID-19 infection. We compared the self-reported vaccination status of the patients with their vaccination status in the electronic Québec Vaccination Registry. Our primary outcome was the accuracy of the self-reported vaccination status (index test) ascertained during telephone follow-up compared with the Québec Vaccination Registry (reference standard). The accuracy was calculated by dividing all correctly self-reported vaccinated and unvaccinated participants by the sum of all correctly and incorrectly self-reported vaccinated and unvaccinated participants. We also reported interrater agreement with the reference standard as measured by unweighted Cohen κ for self-reported vaccination status at telephone follow-up and at the time of their index ED visit, number of vaccine doses, and brand.
Results
During the study period, we included 1361 participants. At the time of the follow-up interview, 932 participants reported at least 1 dose of a COVID-19 vaccine. The accuracy of the self-reported vaccination status was 96% (95% CI 95%-97%). Cohen κ for self-reported vaccination status at phone follow-up was 0.91 (95% CI 0.89-0.93) and 0.85 (95% CI 0.77-0.92) at the time of their index ED visit. Cohen κ was 0.89 (95% CI 0.87-0.91) for the number of doses, 0.80 (95% CI 0.75-0.84) for the brand of the first dose, 0.76 (95% CI 0.70-0.83) for the brand of the second dose, and 0.59 (95% CI 0.34-0.83) for the brand of the third dose.
Conclusions
We reported a high accuracy of self-reported vaccination status for adult patients without cognitive disorders who can express themselves in English or French. Researchers can use self-reported COVID-19 vaccination data on the number of doses received, vaccine brand name, and timing of vaccination to guide future research with patients who are capable of self-reporting their vaccination data. However, access to official electronic vaccine registries is still needed to determine the vaccination status in certain susceptible populations where self-reported vaccination data remain missing or impossible to obtain.
Trial Registration
Clinicaltrials.gov NCT04702945; https://clinicaltrials.gov/ct2/show/NCT04702945
Keywords: electronic vaccination registry, self-reported vaccination status, COVID-19, accuracy, diagnostic study, interrater agreement
Introduction
Background
Since the beginning of the COVID-19 pandemic, vaccination campaigns have been instrumental in preventing mortality and morbidity related to COVID-19 worldwide [1]. With over 13 billion doses of vaccines administered worldwide [2], public health policy makers around the world have been relying on evidence from clinical trials to guide decision-making and develop vaccination strategies to prevent the spread and consequences of this pandemic [3]. In addition to evidence from controlled clinical trials, real-world evidence regarding vaccine effectiveness is important to guide policy making [4,5] and patient decision-making [6]. In particular, real-world evidence can help understand vaccine effectiveness against different disease severities in different susceptible populations often excluded from clinical trials [7], inform policy makers about the length of vaccine protection over time [8], and document rarer side effects [9]. Real-world data can also help understand the evolving prevalence and underlying causes of vaccine hesitancy over time [10] and shape future vaccination campaigns [11,12]. Understanding the role of herd immunity and how to promote vaccination are important questions that need reliable vaccination data [13-16]. Studying COVID-19 occupational hazards for professionals working with susceptible populations also requires timely and reliable access to vaccination data [17,18]. As new variants emerge, real-world evidence informs current and future strategies to protect against COVID-19 infection [19]. To measure real-world vaccine effectiveness and inform evidence-based health promotion strategies, researchers and policy makers alike need access to reliable vaccination data.
Unfortunately, not all researchers and policy makers have access to official electronic public health vaccine registries [20-23], while others have substantial challenges in securing access to individual-level vaccination data even when they exist [24]. Many researchers and policy makers must then rely on patients’ self-reported status to measure real-world vaccine effectiveness, and if this is inaccurate, it may bias vaccine effectiveness estimates in test-negative designs [25]. For these studies, not only are accurate data about vaccination status key, but, most importantly, reliable data about when vaccine doses were administered are essential to determine the effectiveness in preventing infections and health service use such as emergency department (ED) visits [26].
Objective
To inform the conduct of real-world vaccine effectiveness studies using patient self-reported vaccine information and to support the development of evidence-based public health promotion policies, this study aimed to compare the accuracy of self-reported vaccination status for COVID-19 with data from an official electronic vaccination registry in the province of Québec, Canada [27].
Methods
Study Design and Setting
The Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN; pronounced “SED-rin”) is a national collaboration created to harmonize data collection related to COVID-19 in over 50 EDs across 8 provinces in Canada (British Columbia, Alberta, Manitoba, Saskatchewan, Ontario, Québec, New-Brunswick, and Nova Scotia) [28]. The CCEDRRN has created a national observational database of suspected and confirmed patients with COVID-19 who presented to acute care EDs. Data from this network are used in national studies to inform decision-making, emerging treatment guidelines, resource allocation, and planning in response to the pandemic. As of April 2023, CCEDRRN had captured data on over 208,000 patients across Canada [29]. In December 2020, CCEDRRN initiated data collection on vaccination status that supports this study.
This CCEDRRN study is a nested diagnostic accuracy study within the larger CCEDRRN cohort that was reported using the 2015 Standards for Reporting of Diagnostic Accuracy studies [30]. This study only used CCEDRRN data collected from 4 sites in the province of Québec because access to government electronic vaccine registries in the other provinces of Canada was unavailable. These 4 Québec CCEDRRN sites only collected data on patients with a positive COVID-19 test. The study enrolled consecutive consenting patients presenting to these 4 participating EDs between March 24, 2020, and December 25, 2021.
Vaccination in Québec started on December 14, 2020, with long-term care residents and health care workers [31]. On March 1, 2021, vaccination in the general population started in sequence based on their age group. On April 14, 2021, more than 2 million Québecers had received at least 1 dose, representing a quarter of the population. On May 18, 2021, more than half of the population had received at least 1 dose, and on June 6, more than 75% of the population aged >12 years had received at least 1 dose. On July 5, 2021, Québec reduced the dosing interval to 4 weeks to maximize 2-dose coverage, such that by September 30, 2021, a total of 75% of adults were considered fully vaccinated. On September 28, 2021, a third booster dose was recommended for residents in long-term care and assisted living. Between November 16, 2021, and December 29, 2021, the booster dose was progressively authorized for specific high-risk subgroups and then in sequence for the general population based on age groups. After each dose was administered in Québec, vaccinators were required to document the vaccine brand and the time of administration directly within the provincial electronic registry [32]. The identities of the vaccinated patients were validated using at least 2 personal identification questions. Another key public health measure to consider for this study was the mandate on September 1, 2021, to implement a mandatory vaccination passport that contained information about the number of doses, vaccine brand names, and vaccination dates to access certain public services in Québec [31].
Ethics Approval and Informed Consent
The ethics committee at the Centre intégré de santé et de services sociaux de Chaudière-Appalaches approved this project in February 2021 (MP-23-2021-766) with a waiver of informed consent for retrospective data collected; however, informed consent was obtained during telephone follow-up for all participants. The participants did not receive any financial compensation.
Participants
Patients were included if they had a confirmed COVID-19 infection (≥1 nucleic acid amplification tests positive for SARS-CoV-2 from specimens collected during the index ED visit, 14 days before the ED visit in the community, or within 14 days after admission) [28]. Included patients were aged ≥18 years, were able to communicate in French or English, and consented independently to telephone follow-up and the use of their health data. All included patients sought care at any of the 4 participating Québec sites (Hôtel-Dieu de Lévis, Hôpital Sacré-Coeur de Montréal, Royal Victoria Hospital, and the Montreal General Hospital). Patients were excluded if they were unable to provide consent (eg, had died since the ED encounter), were unable to communicate in French or English, could not be reached after 5 attempts, or did not have a record in the Québec Vaccination Registry (eg, out-of-province patients).
Data Collection and Outcome Measures
Trained research assistants collected data for our study from different sources and at different time periods. First, starting on July 9, 2020, we used a retrospective chart review to extract demographics, the time and date of ED visits, chief complaint, laboratory tests, treatments, admission status, and comorbidities. We retrospectively collected data for the index visits of all patients and any subsequent ED visits at least 30 days after the index visit. Second, starting on March 24, 2021, we initiated data collection from the Québec Vaccination Registry and phone follow-ups. A single research assistant at each site was designated to access the Québec Vaccination Registry to determine patients’ vaccination status, vaccine brands received, the total number of doses, and dates of vaccination. Separate research assistants at each site were trained to only conduct telephone follow-ups. They were blinded to the reason for collecting self-reported vaccination data and did not access the information collected from the Québec Vaccination Registry. These research assistants contacted patients by telephone to obtain consent to use their data; to obtain additional sociodemographic data not available in their medical records (eg, income, self-identified race, and education level); and to collect their self-reported vaccination data (vaccination status, vaccine brands received, the total number of doses, and dates of vaccination). The median (IQR) time to contact the patients was 202 (108-352) days after the ED index visit. As patients were called at most 30 days after their data were consulted from the Québec Vaccination Registry, we only used self-reported vaccination data from before the date we consulted the Québec Vaccination Registry.
In accordance with CCEDRRN’s national data management plan, after assigning a unique study identifier for each participant, deidentified data were collected and stored in REDCap (Research Electronic Data Capture; Vanderbilt University) at the University of British Columbia. After data verification, registry data were uploaded into CaraSpace, a secure private cloud for the storage and analysis of privacy-sensitive data [28]. Analysts accessed the cloud space via an encrypted virtual private network through a firewall and 2-factor authentication.
The primary outcome of this study was the accuracy of self-reported COVID-19 vaccination status at the time of the telephone follow-up compared with their vaccination status in the Québec Vaccination Registry. We considered the self-reporting of vaccination status via telephone as the index test and the Québec Vaccination Registry as the reference standard. The index test had 2 possible results: “no vaccine dose” (self-reported as not vaccinated) or “at least one vaccine dose” (self-reported as vaccinated). We calculated the accuracy of the self-reported vaccination status as follows:
| Accuracy = (correctly self-reported as vaccinated + correctly self-reported as not vaccinated) / (correctly self-reported as vaccinated + correctly self-reported as not vaccinated + incorrectly self-reported as vaccinated + incorrectly self-reported as not vaccinated) (1) |
We also compared the clinical and demographic characteristics of patients with concordant and discordant self-reported vaccination status compared with the Québec Vaccination Registry.
Secondary outcomes were the sensitivity, calculated as
| (correctly self-reported as vaccinated / [correctly self-reported as vaccinated + incorrectly self-reported as not vaccinated]) (2) |
and specificity, calculated as
| (correctly self-reported as not vaccinated / [correctly self-reported as not vaccinated + incorrectly self-reported as vaccinated]) (3) |
of the self-reported vaccination status and vaccination status in the Québec Vaccination Registry. Relaxing the reference standard assumption about the Québec Vaccination Registry, we also measured the interrater agreement between self-reported vaccination status and vaccination status in the Québec Vaccination Registry, interrater agreement between self-reported COVID-19 vaccine brand received and the brand recorded in the Québec Vaccination Registry, and interrater agreement between self-reported number of COVID-19 vaccine doses received compared with the number of doses recorded in the vaccine registry.
An additional secondary outcome was the interrater agreement between vaccination status at the time of the index ED visit determined retrospectively by using the self-reported vaccination date compared with the vaccination status at the time of the index ED visit determined by the vaccination date in the Québec Vaccination Registry (Multimedia Appendix 1). Vaccination status was defined as fully, partially, and not vaccinated at the time of the index ED visit based on the number of doses and time since the last dose received to account for likely waning protection [33,34] (Multimedia Appendix 1). We defined fully vaccinated patients at the time of the index ED visit as patients who received ≥2 doses ≥14 days before the index ED visit but less than 6 months since the last dose. We defined partially vaccinated patients as having received only 1 dose before the index ED visit or having received their second or third dose <14 days before the index ED visit or having received their second or third dose >6 months before the index ED visit. Patients were considered not vaccinated if no doses had been received before their index ED visit. Patients who could not recall the date of their last vaccination were excluded from this analysis.
Analysis
The vaccination data in the Québec Vaccination Registry were considered the reference standard. As the interest was to compare the same information collected by telephone as in the Québec Vaccination Registry, the sample size calculations were based on a 1-sided binomial test for sensitivity [35,36]. A sample size of 613 patients was needed to achieve a 95% CI for sensitivity of half width 0.02 assuming a sensitivity of 0.98 and prevalence of at least 0.40.
Data were summarized by using descriptive statistics (eg, mean [SD], counts, and percentage). Accuracy, sensitivity, and specificity for COVID-19 vaccination status reported by patients compared with the reference standard (ie, the Québec Vaccination Registry) were calculated with associated 95% CIs. Secondary analyses relaxed the reference standard assumption and provided interrater agreement using the unweighted Cohen κ coefficient [37]. We chose to use the unweighted Cohen κ because we had 2 raters (ie, self-report by participant and report from the Québec Vaccination Registry), and we wanted disagreements to carry the same weight. All analyses were conducted in R (R Foundation for Statistical Computing) using the rms and psych packages [38-40]. To ensure patient privacy, a cell-size restriction policy prohibited us from reporting counts of <5.
Results
Participant Characteristics
During the study period, there were 5912 adult patients who presented to the Québec study sites, and 1482 met the inclusion criteria; of them, 1361 (23.02%) patients had the required data from follow-up and available in the Québec Vaccination Registry (Figure 1). Among the 4430 excluded patients, 1596 (36.03%) did not consent to participate or were not able to consent. There were also 62 (1.05%) patients who did not have any vaccination record in the Québec Vaccination Registry because they were patients from outside the province or did not have a provincial health insurance number. The baseline participant characteristics are shown in Table 1. Patients who did not or could not consent were slightly older than patients who participated (age: mean 58.6, SD 21.3 years vs mean 55.1, SD 17.4 years, respectively) and were more likely to be male individuals (nonconsenting male individuals: 871/1596, 54.57%, vs consenting male individuals: 686/1361, 50.4%). Three-quarters (1010/1361, 74.21%) of the patients presented to the ED during the second and third waves of the pandemic (August 23, 2020, to July 17, 2021).
Figure 1.
Flow diagram of patients in the study.
Table 1.
Sociodemographic and clinical characteristics of study participants whose index visit occurred in a Québec emergency department between March 24, 2020, and December 25, 2021 (n=1361).
|
|
Study participants | ||
| Age (years), mean (SD) | 55.1 (17.4) | ||
| Age group (years), n (%) | |||
|
|
<24 | 45 (3.3) | |
|
|
25-39 | 230 (16.9) | |
|
|
40-64 | 685 (50.3) | |
|
|
65-79 | 272 (20) | |
|
|
≥80 | 129 (9.5) | |
| Sex, n (%) | |||
|
|
Male | 686 (50.4) | |
|
|
Female | 675 (49.6) | |
|
|
Intersex | 0 (0) | |
| Site, n (%) | |||
|
|
Hôtel-Dieu de Lévis | 293 (21.5) | |
|
|
Royal Victoria | 501 (36.8) | |
|
|
Montreal General | 132 (9.7) | |
|
|
Sacré-Coeur de Montréal | 435 (32) | |
| Index visit occurred during, n (%) | |||
|
|
Wave 1 (March 1, 2020, to August 22, 2020) | 207 (15.2) | |
|
|
Wave 2 (August 23, 2020, to March 20, 2021) | 725 (53.3) | |
|
|
Wave 3 (March 21, 2021, to July 17, 2021) | 285 (20.9) | |
|
|
Wave 4 to early 5 (July 18, 2021, to December 31, 2021) | 144 (10.6) | |
| Index visit occurred before the start of vaccination campaign in Québec (December 14, 2020), n (%) | |||
|
|
Prevaccination campaign | 490 (36) | |
|
|
Postvaccination campaign | 871 (64) | |
| Time elapsed from index visit to telephone follow-up (days), median (IQR) | 202 (108-352) | ||
| Top 10 comorbid conditions, n (%) | |||
|
|
Hypertension | 440 (32.3) | |
|
|
Dyslipidemia | 314 (23.1) | |
|
|
Diabetes | 241 (17.7) | |
|
|
Hypothyroidism | 160 (11.8) | |
|
|
Asthma | 144 (10.6) | |
|
|
Psychiatric condition or mental health diagnosis | 113 (8.3) | |
|
|
Coronary artery disease | 99 (7.3) | |
|
|
Past malignancy | 73 (5.4) | |
|
|
Chronic neurological conditions | 66 (4.8) | |
|
|
Rheumatologic disorder | 66 (4.8) | |
| EDa disposition, n (%) | |||
|
|
Discharged home | 740 (54.4) | |
|
|
Admitted | 576 (42.3) | |
|
|
Transferred to other hospital | 38 (2.8) | |
|
|
Transfer to long-term care or rehab | <5b (<0.4) | |
|
|
Other | 6 (0.4) | |
aED: emergency department.
bCells with fewer than 5 individuals could not be presented as requested by the ethics committee.
Vaccination Status
Over two-thirds (932/1361, 68.55%) of the participants reported at least 1 dose of a COVID-19 vaccine at the time of their follow-up interview (Table 2). The most administered vaccine type was Pfizer-BioNTech (689/1361, 50.6%) followed by Moderna (158/1361, 11.6%).
Table 2.
Québec Vaccination Registry information and self-reported vaccination status during the initial telephone follow-up of participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=1361).
|
|
Québec Vaccination Registry, n (%) | Self-reported vaccination status, n (%) | |||
| Vaccination status | |||||
|
|
Yes (at least 1 dose) | 913 (67.1) | 932 (68.5) | ||
|
|
No (no dose given) | 448 (32.9) | 429 (31.5) | ||
|
|
Unknown or not assessed | 0 (0) | 0 (0) | ||
| First vaccine brand | |||||
|
|
Pfizer-BioNTech | 707 (51.9) | 689 (50.6) | ||
|
|
Moderna | 162 (11.9) | 158 (11.6) | ||
|
|
AstraZeneca or Covishield | 42 (3.1) | 42 (3.1) | ||
|
|
Johnson & Johnson | <5 (<0.4) | <5a (<0.4) | ||
|
|
Unknown, but vaccinated | 0 (0) | 37 (2.7) | ||
|
|
Unknown or not vaccinated | 448 (32.9) | 433 (31.8) | ||
| Number of vaccines doses | |||||
|
|
0 | 448 (32.9) | 429 (31.5) | ||
|
|
1 | 366 (26.9) | 368 (27) | ||
|
|
2 | 501 (36.8) | 511 (37.5) | ||
|
|
3 | 46 (3.4) | 53 (3.9) | ||
aCells with fewer than 5 individuals could not be presented as requested by the ethics committee.
Self-Reported Vaccination Status Accuracy and Interrater Agreement Between Self-Reported Vaccination Data and Québec Vaccination Registry Data
We reported high measures of accuracy (96%), sensitivity (98%), and specificity (92%) for self-reported vaccination status compared with the Québec Vaccination Registry as the reference standard (Tables 3 and 4). Moreover, Cohen κ was 0.91 (95% CI 0.89-0.93), indicating excellent agreement between patient self-report and the Québec Vaccination Registry.
Table 3.
Confusion matrix comparing self-reported vaccination status with the Québec Vaccination Registry data for all participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=1361).
| Self-reported vaccination status (index test) | Vaccination status in the Québec Vaccination Registry (reference standard), n | |
|
|
Yes | No |
| Yes | 896 | 36 |
| No | 17 | 412 |
Table 4.
Sensitivity, specificity and accuracy of self-reported vaccination status and interrater agreement of self-reported vaccination status compared with the Québec Vaccination Registry for all participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=1361).
|
|
Measure (95% CI) |
| Sensitivity | 0.98 (0.97-0.99) |
| Specificity | 0.92 (0.89-0.94) |
| Accuracy | 0.96 (0.95-0.97) |
| Cohen κ | 0.91 (0.89-0.93) |
With respect to vaccine brand type, Cohen κ for dose 1 brand was 0.80 (95% CI 0.75-0.84) indicating strong agreement (Table 5). However, this estimate decreased with subsequent doses: Cohen κ was 0.76 (95% CI 0.70-0.83) and 0.59 (95% CI 0.34-0.83) for dose 2 and dose 3 brands, respectively.
Table 5.
Interrater agreement for self-reported brand by dose compared with the Québec Vaccination Registry for all participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021.
| Dose and self-reported vaccine brand | Québec Vaccination Registry vaccine brand, n | Cohen κa (95% CI) | |||||
|
|
Pfizer | Moderna | Astra Zeneca or Covishield | Johnson & Johnson |
|
||
| 1 | 0.80 (0.75-0.84) | ||||||
|
|
Pfizer | 645 | 19 | <5 | 0 |
|
|
|
|
Moderna | 11 | 137 | 0 | 0 |
|
|
|
|
AstraZeneca or Covishield | <5 | 0 | 38 | 0 |
|
|
|
|
Johnson & Johnson | 0 | 0 | 0 | <5 |
|
|
|
|
Unknown, but vaccinated | 29 | 6 | <5 | 0 |
|
|
| 2 | 0.76 (0.70-0.83) | ||||||
|
|
Pfizer | 386 | 19 | <5 | 0 |
|
|
|
|
Moderna | 5 | 84 | 0 | 0 |
|
|
|
|
AstraZeneca or Covishield | 0 | 0 | 9 | 0 |
|
|
|
|
Unknown, but vaccinated | 16 | 5 | 0 | 0 |
|
|
| 3 | 0.59 (0.34-0.83) | ||||||
|
|
Pfizer | 24 | <5 | 0 | 0 |
|
|
|
|
Moderna | <5 | 7 | 0 | 0 |
|
|
|
|
Unknown, but vaccinated | <5 | <5 | 0 | 0 |
|
|
aCohen κ calculated without participants who answered “Unknown, but vaccinated” were 0.89 (95% CI 0.86-0.93) for first dose, 0.85 (95% CI 0.80-0.91) for second dose, and 0.70 (95% CI 0.43-0.97) for third dose.
Most patients reported receiving 1 dose (368/1361, 27.04%) or 2 doses (511/1361, 37.56%) of COVID-19 vaccines at the time of the follow-up interview (Table 6). Only 3.89% (53/1361) of patients reported receiving 3 doses. One-third (429/1361, 31.52%) of the patients reported not having received the COVID-19 vaccine at the time of follow-up. Cohen κ for the number of self-reported doses compared with the Québec Vaccination Registry was very strong, at 0.89 (95% CI 0.87-0.91).
Table 6.
Interrater agreement for self-reported number of doses compared with the Québec Vaccination Registry for participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=1361).a
| Self-reported number of doses | Number of doses in the Québec Vaccination Registry, n | |||
|
|
0 | 1 | 2 | 3 |
| 0 | 412 | 7 | 8 | <5 |
| 1 | 20 | 336 | 12 | 0 |
| 2 | 13 | 23 | 469 | 6 |
| 3 | <5 | 0 | 12 | 38 |
aCohen κ: 0.89 (95% CI 0.87-0.91).
Cohen κ for self-reported vaccination status at the time of ED index visit was 0.85 (95% CI 0.77-0.92). However, many patients (774/1361, 56.87%) could not recall the exact date of their latest vaccine dose and had to be excluded from this analysis. Of those who did recall their vaccination dates, most (538/587, 91.7%) reported not being fully vaccinated at the time of the ED visit, with only 16 (2.7%) patients being fully vaccinated (Table 7).
Table 7.
Interrater agreement for self-reported vaccination status at emergency department index visit date versus the Québec Vaccination Registry for participants whose emergency department index visit occurred between March 24, 2020, and December 25, 2021 (n=587).a
| Self-reported vaccination status determined using the date of latest vaccination dose in relation to the EDb visit datec | Québec Vaccination Registry vaccination status determined in relation to the ED index visit dated, n | ||
|
|
Not vaccinated | Partially vaccinated | Fully vaccinated |
| Not vaccinated | 527 | 5 | 6 |
| Partially vaccinated | <5 | 29 | 0 |
| Fully vaccinated | 0 | 0 | 16 |
aCohen κ: 0.85 (95% CI 0.77-0.92).
bED: emergency department.
cCategories of vaccination status at the time of ED index visit (fully, partially, and not vaccinated) for this analysis are defined in Multimedia Appendix 1.
dA total of 774 (56.9%) participants were excluded from this analysis because they could not recall the date of their last dose.
The demographic, clinical, and sociocultural characteristics of patients with discordant (n=53) and concordant (n=1308) vaccination statuses are shown in Multimedia Appendix 2. Participants with discordant self-reported vaccination status compared to those with concordant self-reported vaccination status were younger (age: mean 51.1, SD 20.5 years, vs mean 55.3, SD 17.3 years, respectively), had lower self-reported income (5/53, 9% reported current income >CAD $73,701 (US $56,693) per year vs 294/1308, 22.48%), had lower education level (5/53, 9% with a university degree vs 444/1308, 33.94%), and had a lower vaccination rate according to the Québec Vaccination Registry (17/53, 32% vs 896/1308, 68.5%). The median time elapsed between the ED index visit and the telephone follow-up to determine the patient’s self-reported vaccination status was similar for discordant (median 94, IQR 53-191 days) and concordant cases (median 126, 62-196 days). Immigration status, race, and ethnicity did not influence the accuracy of our results.
Discussion
Principal Findings
We found that the participants’ self-reported vaccination status was highly accurate at the time of the initial follow-up telephone call. The interrater agreement between the self-reported vaccine number of doses and the number of doses recorded in the vaccine registry was also strong. The agreement between self-reported vaccination status at the index visit and the patient’s vaccination status as determined by the registry was strong when patients could provide vaccination dates. However, many patients could not recall their vaccination dates. Although there was a strong and moderate agreement for the self-reported vaccine brand for the first and second doses, respectively, the agreement dropped to weak for the vaccine brand of the third dose. A few discordant cases concerning the accuracy of patients’ self-report of receiving at least 1 dose were found. The discordant cases involved patients of younger age groups, lower self-reported pre– and post–COVID-19 income, lower education levels, and lower vaccination rates in the Québec Vaccination Registry.
Our results are similar to those of a study conducted in the United States by Stephenson et al [41] that showed high levels of agreement between self-reported vaccination status and official computerized vaccination registry data (unweighted κ=0.9127, 95% CI 0.9109-0.9145). Similar to our study, they also found that discordant self-reported vaccination data were associated with lower education levels and unemployment. It could be that both studies found that lower education is associated with lower accuracy of self-reported vaccination status because information provided about vaccination status tends to be complex [42]. Comprehending and remembering this information requires certain health literacy skills [43]. Rolnick et al [44] also found that education and income level could influence the accuracy of self-reported vaccination status for 8 different vaccines (pneumococcal, influenza, tetanus diphtheria, tetanus diphtheria pertussis, human papillomavirus, hepatitis A, hepatitis B, and herpes zoster).
Contrary to our study, the study by Stephenson et al [41] found that the female sex increased the risk of discordant self-reported vaccination status. Sex and gender are important factors in understanding the real-world effectiveness of vaccines and postmarketing safety monitoring [45,46]. Sex and gender differences in vaccine hesitancy and immunization outcomes have been observed for COVID-19 and other vaccine-preventable diseases, with women typically reporting more vaccine hesitancy and higher adverse reactions than men [47-49]. Our results support the accuracy of self-reported vaccination data from both men and women.
Ulrich et al [50] studied the accuracy of the self-reported vaccination status by university students in a 2-dose meningococcal vaccine campaign. This study showed good self-reported vaccination status accuracy at 2 months (86%) and poor self-reported vaccination status at 20 months (69%). King et al [51] also reported a decrease in the agreement of patient self-reported vaccination status compared with a registry for prior season influenza vaccination (κ=0.86) versus the current season (κ=0.95). Although our study did not observe a time-dependent increase in discordant self-reported vaccination status, we did find that patients reporting a higher number of doses were more likely to forget the vaccine brand received and did not remember the dates of their latest vaccination dose. In patients who did remember the vaccine brand for each dose, the agreement with the registry was high. This limits the use of self-reported vaccination brand names for studies comparing the effectiveness of multiple doses of different types of vaccines [52].
The reliability of data on the timing of vaccination before being tested for a viral infection such as SARS-CoV-2 is essential for test-negative design vaccine effectiveness studies [53]. Our study was interested in determining the reliability of the self-reported vaccination data (ie, the date of their last vaccination and the total number of doses) used to retrospectively determine their vaccination status (not vaccinated, partially vaccinated, or fully vaccinated) at the time of their index ED visit. For patients who could remember their last vaccination date, our study found high levels of agreement between the self-reported vaccination status determined at the ED index visit date and the Québec Vaccination Registry. Unfortunately, more than half of our participants could not recall the date of their last dose, making this measure subject to an important selection bias when used in test-negative designs. This is an argument to support wider implementation and easy access to rigorous vaccination registries in all jurisdictions [21]. This is especially important to improve the care, research, and support of vulnerable populations presenting to the ED. In Canada, access to these registries is currently restricted by data custodians and administrative barriers that hinder timely knowledge creation and decision-making for a rapid-learning health system [54,55]. Although a study on the accuracy of self-reported influenza and pneumococcal vaccination status in a high-risk inflammatory bowel disease population showed high accuracy for self-reported influenza (97%) and pneumococcal (84%) vaccination status [56], our study did not find any difference between high-risk groups with different comorbidities. In concordance with Laurence et al [57], who found that older patients accurately self-reported their influenza (96%) and pneumococcal (87%) vaccine status, we also found that older adults adequately self-reported their COVID-19 vaccine status. Surprisingly, we found a higher proportion of discordant self-reported COVID-19 vaccination statuses among patients aged ≤34 years. This could be associated with the fact that COVID-19 presents a milder course in younger patients [57,58] and increased vaccination hesitancy in this age group [59-61].
There are several potential explanations for the high accuracy of self-reported vaccination status at telephone follow-up and moderate to strong agreement between the self-reported number of doses and the self-reported vaccination status determined at the time of the ED index visit. First, this high accuracy and strong agreement could have been caused by the implementation of Québec’s mandatory vaccine passport, which registered dates, brands, and doses as a requirement to access the workplace and most commercial services. Although we did not ask participants to consult their passports before answering our phone questionnaire, our study personnel did not prevent participants from consulting their passports during our telephone follow-ups. Participants’ recall of their own vaccination data could also have been improved by having to consult their passport frequently, as both federal and provincial governments in Canada granted substantial benefits to fully vaccinated citizens, such as lifting travel restrictions and access to event venues, fitness centers, and restaurants via the use of a vaccination passport [62]. Second, the impact of COVID-19 on the Canadian population has been substantial, and it may have improved the recall and the accuracy of the self-reported vaccination status. Third, the COVID-19 vaccination campaign; the omnipresence of the COVID-19 pandemic in the lay press; the many frequently reported statistics portraying the evolution of this pandemic (cases, hospitalizations, deaths, and population vaccination coverage); and the vigorous public debate about vaccination public health mandates may have also raised citizens’ awareness of their own vaccination status. Fourth, this study only included patients with confirmed COVID-19 infections. This could have improved the recall of participants compared with patients who were not infected with COVID-19.
Our results indicate that when access to official electronic vaccination registries is not available, the reliance on patients’ self-reported COVID-19 vaccination status, vaccine brand received, number of doses, and vaccination dates (when provided) can be used for research purposes. This is a substantial finding for researchers in jurisdictions that do not have access to electronic vaccine registries [20,63] and for others that have substantial challenges in securing timely access to individual-level vaccination data even when they exist [24]. Vaccine registries can also suffer from missing data for patients who have received their vaccines in different jurisdictions (as shown in our results), misclassification bias because of human error, and the timeliness of data entry into electronic vaccination registries [64]. Relying on timely and highly accurate self-reported vaccination status reduces the potential underestimation of real-world test-negative vaccine effectiveness studies introduced by vaccination-reporting bias [25]. Knowing the sensitivity and specificity of the self-reported vaccination status is also important to correct the estimation of vaccine effectiveness based on the observed risks for vaccination status misclassification [65].
This study has limitations. First, although we assumed that the Québec Vaccination Registry was the reference standard, we cannot rule out the possibility that some vaccination data were entered incorrectly. Second, our results do not apply to patients with cognitive disorders who cannot provide consent. This represents a selection bias that we could not avoid because of ethical constraints. Patients who consented to participate in our study were also more likely to be younger and female than those who did not consent or who could not consent. Understanding the impact of the pandemic on patients who cannot consent to provide their vaccination status because of cognitive issues will still necessitate access to linked vaccination registry data. Access to a vaccination registry also avoids systematic differences in the characteristics of consenting patients versus nonconsenting patients in studies that rely on self-reporting. Third, although we had many landed immigrants constituting an ethnically diverse sample, our results are potentially not applicable to patients who do not speak English or French. Other studies have shown that the self-reported vaccination status is inaccurate in certain susceptible and hard-to-follow-up racial and ethnic groups [44,50]. Although we had to exclude patients who could not speak English or French, immigration and race and ethnicity did not influence the accuracy of our results. Fourth, our study did not verify the accuracy of vaccination status provided at the time of ED care. Future prospective studies at the point of care would be needed to validate the accuracy of self-reported data for use in guiding clinical decision-making.
Conclusions
Our study found that relying on self-reported COVID-19 vaccination status is accurate for adult patients without cognitive disorders who can express themselves in English or French. Researchers can use self-reported COVID-19 vaccination data on the number of doses received, vaccine brand name, and timing of vaccination to guide future research with certain groups of patients who are capable of self-reporting their vaccination data. However, access to official electronic vaccine registries is still needed to determine the vaccination status in certain susceptible populations where self-reported vaccination data remain missing or impossible to obtain.
Acknowledgments
The authors would like to acknowledge the assistance of Ms Amber Cragg and Ms Josie Kanu in the preparation of this manuscript. The authors would like to thank the University of British Columbia clinical coordinating center staff; legal, ethics, privacy, and contract staff; and the research staff at each of the participating institutions in the network outlined in the attached supplement (Multimedia Appendix 3). In particular, the authors would like to thank Chantal Lanthier and David Iannuzzi for their support in collecting the data for this study. The network would not exist today without the dedication of these professionals. The authors would also like to thank all the patient partners who shared their lived experiences and perspectives to ensure that the knowledge we cocreate addresses the concerns of patients and the public. Creating the largest network of collaboration across Canadian emergency departments would not have been possible without the tireless efforts of emergency department chiefs, research coordinators, and research assistants at participating sites. Finally, the authors would like to express their most humble and sincere gratitude to all their colleagues in medicine, nursing, and the allied health professions who have been on the frontlines of this pandemic from day 1, staffing ambulances, emergency departments, intensive care units, and hospitals bravely facing the risks of COVID-19 to look after fellow citizens and after one another. The authors would like to dedicate this network to them. This network was funded by the Canadian Institutes of Health Research (447679, 464947, and 466880); Ontario Ministry of Colleges and Universities (C-655-2129); Saskatchewan Health Research Foundation (5357); Genome BC (COV024 and VAC007); Fondation du CHU de Québec (Octroi No. 4007); and Sero-Surveillance and Research (COVID-19 Immunity Task Force Initiative), who provided peer-reviewed funding. The BC Academic Health Science Network and BioTalent Canada provided non–peer-reviewed funding. These organizations are not for profit and had no role in conducting the study, analysis, or manuscript preparation.
The Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) consisted of the following members: Patrick Fok, Hana Wiemer, Samuel Campbell, Kory Arsenault, Tara Dahn, Kavish Chandra, Patrick Archambault, Joel Turner, Lars Grant, Greg Clark, Raoul Daoust, Éric Mercier, Sébastien Robert, Laurie Morrison, Steven Brooks, Ivy Cheng, Jeffrey Perry, Rob Ohle, Justin Yan, Rohit Mohindra, Megan Landes, Michelle Welsford, Tomislav Jelic, Phil Davis, Andrew McRae, Brian Rowe, Katie Lin, Stephanie VandenBerg, Jake Hayward, Jaspreet Khangura, Corinne Hohl, Daniel Ting, Maja Stachura, Frank Scheuermeyer, Balijeet Braar, John Taylor, Ian Martin, Sean Wormsbecker and Lee Graham.
Abbreviations
- CCEDRRN
Canadian COVID-19 Emergency Department Rapid Response Network
- ED
emergency department
- REDCap
Research Electronic Data Capture
Definitions used to determine the interrater agreement between the self-reported vaccination status and the vaccination status determined by the Québec Vaccination Registry at the time of emergency department index visit.
Characteristics of all participants by concordant or discordant vaccination status.
Contributors to the Canadian COVID-19 Emergency Department Rapid Response Network.
Data Availability
Data are available on reasonable request. For investigators who wish to access Canadian COVID-19 Emergency Rapid Response Network data, proposals may be submitted to the network for review and approval by the network’s peer review publication committee, the data access and management committee, and the executive committee as per the network’s governance. Information regarding submitting proposals and accessing data may be found on the Canadian COVID-19 Emergency Department Rapid Response Network website [9].
Footnotes
Conflicts of Interest: None declared.
References
- 1.Coccia M. Optimal levels of vaccination to reduce COVID-19 infected individuals and deaths: a global analysis. Environ Res. 2022 Mar;204(Pt C):112314. doi: 10.1016/j.envres.2021.112314. https://europepmc.org/abstract/MED/34736923 .S0013-9351(21)01615-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.COVID-19 map. Johns Hopkins Coronavirus Resource Center. [2023-04-06]. https://coronavirus.jhu.edu/map.html .
- 3.Coccia M. COVID-19 vaccination is not a sufficient public policy to face crisis management of next pandemic threats. Public Organ Rev (forthcoming) 2022 Oct 17; doi: 10.1007/s11115-022-00661-6. https://link.springer.com/article/10.1007/s11115-022-00661-6#citeas . [DOI] [Google Scholar]
- 4.Patel MK, Bergeri I, Bresee JS, Cowling BJ, Crowcroft NS, Fahmy K, Hirve S, Kang G, Katz MA, Lanata CF, L'Azou Jackson M, Joshi S, Lipsitch M, Mwenda JM, Nogareda F, Orenstein WA, Ortiz JR, Pebody R, Schrag SJ, Smith PG, Srikantiah P, Subissi L, Valenciano M, Vaughn DW, Verani JR, Wilder-Smith A, Feikin DR. Evaluation of post-introduction COVID-19 vaccine effectiveness: summary of interim guidance of the World Health Organization. Vaccine. 2021 Jul 05;39(30):4013–24. doi: 10.1016/j.vaccine.2021.05.099. https://europepmc.org/abstract/MED/34119350 .S0264-410X(21)00707-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Patel MM, Jackson ML, Ferdinands J. Postlicensure evaluation of COVID-19 vaccines. JAMA. 2020 Nov 17;324(19):1939–40. doi: 10.1001/jama.2020.19328.2772136 [DOI] [PubMed] [Google Scholar]
- 6.Stacey D, Ludwig C, Archambault P, Smith M, Taljaard M, Carley M, Plourde K, Boland L, Gogovor A, Graham I, Kobewka D, McLean RK, Nelson ML, Vanderspank-Wright B, Légaré F. Decisions and decisional needs of Canadians from all provinces and territories during the COVID-19 pandemic: population-based cross-sectional surveys. JMIR Public Health Surveill. 2023 Mar 21;9:e43652. doi: 10.2196/43652. https://publichealth.jmir.org/2023//e43652/ v9i1e43652 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Shea LL, Becker A, Lee BK, Miller KK, Cooper D, Anderson K, Salzer MS, Vanness DJ. Self-reported COVID-19 vaccination acceptance and hesitancy among autistic adults. Vaccine. 2022 May 26;40(24):3288–93. doi: 10.1016/j.vaccine.2022.04.060. https://linkinghub.elsevier.com/retrieve/pii/S0264-410X(22)00500-X .S0264-410X(22)00500-X [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Sasanami M, Fujimoto M, Kayano T, Hayashi K, Nishiura H. Projecting the COVID-19 immune landscape in Japan in the presence of waning immunity and booster vaccination. J Theor Biol. 2023 Feb 21;559:111384. doi: 10.1016/j.jtbi.2022.111384. https://linkinghub.elsevier.com/retrieve/pii/S0022-5193(22)00375-7 .S0022-5193(22)00375-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wesselink AK, Hatch EE, Rothman KJ, Wang TR, Willis MD, Yland J, Crowe HM, Geller RJ, Willis SK, Perkins RB, Regan AK, Levinson J, Mikkelsen EM, Wise LA. A prospective cohort study of COVID-19 vaccination, SARS-CoV-2 infection, and fertility. Am J Epidemiol. 2022 Jul 23;191(8):1383–95. doi: 10.1093/aje/kwac011. https://europepmc.org/abstract/MED/35051292 .6511811 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Dubé E, Laberge C, Guay M, Bramadat P, Roy R, Bettinger J. Vaccine hesitancy: an overview. Hum Vaccin Immunother. 2013 Aug;9(8):1763–73. doi: 10.4161/hv.24657. https://europepmc.org/abstract/MED/23584253 .24657 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Yoon H, Choi BY, Seong WJ, Cho GJ, Na S, Jung YM, Jo JH, Ko HS, Park JS. COVID-19 vaccine acceptance during pregnancy and influencing factors in South Korea. J Clin Med. 2022 Sep 28;11(19):5733. doi: 10.3390/jcm11195733. https://www.mdpi.com/resolver?pii=jcm11195733 .jcm11195733 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Coccia M. Improving preparedness for next pandemics: max level of COVID-19 vaccinations without social impositions to design effective health policy and avoid flawed democracies. Environ Res. 2022 Oct;213:113566. doi: 10.1016/j.envres.2022.113566. https://europepmc.org/abstract/MED/35660409 .S0013-9351(22)00893-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Etienne D, Archambault P, Aziaka D, Chipenda-Dansokho S, Dubé E, Fallon CS, Hakim H, Kindrachuk J, Krecoum D, MacDonald SE, Ndjaboue R, Noubi M, Paquette JS, Parent E, Witteman HO. A personalized avatar-based web application to help people understand how social distancing can reduce the spread of COVID-19: cross-sectional, observational, pre-post study. JMIR Form Res. 2023 Apr 25;7:e38430. doi: 10.2196/38430. https://formative.jmir.org/2023//e38430/ v7i1e38430 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Hakim H, Bettinger JA, Chambers CT, Driedger SM, Dubé E, Gavaruzzi T, Giguere AM, Kavanagh É, Leask J, MacDonald SE, Orji R, Parent E, Paquette JS, Roberge J, Sander B, Scherer AM, Tremblay-Breault M, Wilson K, Reinharz D, Witteman HO. A web application about herd immunity using personalized avatars: development study. J Med Internet Res. 2020 Oct 30;22(10):e20113. doi: 10.2196/20113. https://www.jmir.org/2020/10/e20113/ v22i10e20113 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Akinbami LJ, Kruszon-Moran D, Wang CY, Storandt RJ, Clark J, Riddles MK, Mohadjer LK. SARS-CoV-2 serology and self-reported infection among adults - national health and nutrition examination survey, United States, August 2021-May 2022. MMWR Morb Mortal Wkly Rep. 2022 Dec 02;71(48):1522–5. doi: 10.15585/mmwr.mm7148a4. doi: 10.15585/mmwr.mm7148a4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Miao Y, Zhang W, Li Y, Wu J, Xu D, Gu J, Wang M, Wei W, Ye B, Miao C, Tarimo CS, Dong W. Association between lifestyle and COVID-19 vaccination: a national cross-sectional study. Front Public Health. 2022 Oct 11;10:918743. doi: 10.3389/fpubh.2022.918743. https://europepmc.org/abstract/MED/36304239 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Kronfli N, Dussault C, Maheu-Giroux M, Halavrezos A, Chalifoux S, Park H, Balso LD, Cheng MP, Cox J. Importance of occupation for SARS-CoV-2 seroprevalence and COVID-19 vaccination among correctional workers in Quebec, Canada: a cross-sectional study. Front Public Health. 2022 Nov 09;10:1021871. doi: 10.3389/fpubh.2022.1021871. https://europepmc.org/abstract/MED/36438247 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Gormley MA, Nolan MS, Heo M, Litwin AH, Alier A, Daguise V. Differences in motivating factors for SARS-CoV-2 vaccination and perceptions of infection risk among healthcare and EMS personnel in South Carolina. South Med J. 2022 Jun;115(6):381–7. doi: 10.14423/SMJ.0000000000001406. https://europepmc.org/abstract/MED/35649524 .SMJ_220205 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Coccia M. Sources, diffusion and prediction in COVID-19 pandemic: lessons learned to face next health emergency. AIMS Public Health. 2023 Mar 02;10(1):145–68. doi: 10.3934/publichealth.2023012. https://europepmc.org/abstract/MED/37063362 .publichealth-10-01-012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Namageyo-Funa A, Samuel A, Bloland P, Macneil A. Considerations for the development and implementation of electronic immunization registries in Africa. Pan Afr Med J. 2018 May 29;30:81. doi: 10.11604/pamj.2018.30.81.11951. https://europepmc.org/abstract/MED/30344865 .PAMJ-30-81 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Benati I, Coccia M. Global analysis of timely COVID-19 vaccinations: improving governance to reinforce response policies for pandemic crises. Int J Health Gov. 2022 May 31;27(3):240–53. doi: 10.1108/ijhg-07-2021-0072. https://www.emerald.com/insight/content/doi/10.1108/IJHG-07-2021-0072/full/html . [DOI] [Google Scholar]
- 22.Coccia M. Pandemic prevention: lessons from COVID-19. Encycl. 2021 May 31;1(2):433–44. doi: 10.3390/encyclopedia1020036. https://www.mdpi.com/2673-8392/1/2/36 . [DOI] [Google Scholar]
- 23.González MJ, Rodríguez FS. Self-reported COVID-19 vaccination coverage in Venezuela: results of national sample surveys (June-December 2021) Gac Med Caracas. 2022 Jul 15;130(2S):S459–69. doi: 10.47307/gmc.2022.130.s2.17. http://saber.ucv.ve/ojs/index.php/rev_gmc/article/view/24127 . [DOI] [Google Scholar]
- 24.McRae AD, Archambault P, Fok P, Wiemer H, Morrison LJ, Herder M, Canadian COVID-19 Emergency Department Rapid Response Network Investigators Reducing barriers to accessing administrative data on SARS-CoV-2 vaccination for research. CMAJ. 2022 Jul 18;194(27):E943–7. doi: 10.1503/cmaj.211712. https://www.cmaj.ca/content/cmaj/194/27/E943.full.pdf .194/27/E943 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Jackson ML. Use of self-reported vaccination status can bias vaccine effectiveness estimates from test-negative studies. Vaccine X. 2018 Dec 29;1:100003. doi: 10.1016/j.jvacx.2018.100003. https://linkinghub.elsevier.com/retrieve/pii/S2590-1362(18)30003-2 .S2590-1362(18)30003-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Feikin DR, Higdon MM, Abu-Raddad LJ, Andrews N, Araos R, Goldberg Y, Groome MJ, Huppert A, O'Brien KL, Smith PG, Wilder-Smith A, Zeger S, Deloria Knoll M, Patel MK. Duration of effectiveness of vaccines against SARS-CoV-2 infection and COVID-19 disease: results of a systematic review and meta-regression. Lancet. 2022 Mar 05;399(10328):924–44. doi: 10.1016/S0140-6736(22)00152-0. https://linkinghub.elsevier.com/retrieve/pii/S0140-6736(22)00152-0 .S0140-6736(22)00152-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Québec vaccination registry. Gouvernement du Québec. [2023-04-06]. https://www.quebec.ca/en/health/your-health-information/quebec-vaccination-registry .
- 28.Hohl CM, Rosychuk RJ, McRae AD, Brooks SC, Archambault P, Fok PT, Davis P, Jelic T, Turner JP, Rowe BH, Mercier É, Cheng I, Taylor J, Daoust R, Ohle R, Andolfatto G, Atzema C, Hayward J, Khangura JK, Landes M, Lang E, Martin I, Mohindra R, Ting DK, Vaillancourt S, Welsford M, Brar B, Dahn T, Wiemer H, Yadav K, Yan JW, Stachura M, McGavin C, Perry JJ, Morrison LJ, Canadian COVID-19 Emergency Department Rapid Response Network investigators and for the Network of Canadian Emergency Researchers and the Canadian Critical Care Trials Group Development of the Canadian COVID-19 emergency department rapid response network population-based registry: a methodology study. CMAJ Open. 2021 Mar 17;9(1):E261–70. doi: 10.9778/cmajo.20200290. http://cmajopen.ca/cgi/pmidlookup?view=long&pmid=33731427 .9/1/E261 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Canadian COVID-19 Emergency Department Rapid Response Network. [2023-05-28]. https://www.ccedrrn.com/
- 30.Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, Lijmer JG, Moher D, Rennie D, de Vet HC, Kressel HY, Rifai N, Golub RM, Altman DG, Hooft L, Korevaar DA, Cohen JF, STARD Group STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015 Oct 28;351:h5527. doi: 10.1136/bmj.h5527. http://www.bmj.com/lookup/pmidlookup?view=long&pmid=26511519 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Ligne du temps COVID-19 au Québec. Institut national de santé publique du Québec. [2023-04-06]. https://www.inspq.qc.ca/covid-19/donnees/ligne-du-temps .
- 32.Inscription obligatoire en tant que personne vaccinée. Gouvernement du Québec. 2018. Apr 20, [2023-04-06]. https://www.quebec.ca/sante/vos-informations-de-sante/registre-de-vaccination-du-quebec/inscription-obligatoire-en-tant-que-personne-vaccinee .
- 33.Britton A, Fleming-Dutra KE, Shang N, Smith ZR, Dorji T, Derado G, Accorsi EK, Ajani UA, Miller J, Schrag SJ, Verani JR. Association of COVID-19 vaccination with symptomatic SARS-CoV-2 infection by time since vaccination and delta variant predominance. JAMA. 2022 Mar 15;327(11):1032–41. doi: 10.1001/jama.2022.2068. https://europepmc.org/abstract/MED/35157002 .2789294 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Tartof SY, Slezak JM, Fischer H, Hong V, Ackerson BK, Ranasinghe ON, Frankland TB, Ogun OA, Zamparo JM, Gray S, Valluri SR, Pan K, Angulo FJ, Jodar L, McLaughlin JM. Effectiveness of mRNA BNT162b2 COVID-19 vaccine up to 6 months in a large integrated health system in the USA: a retrospective cohort study. Lancet. 2021 Oct 16;398(10309):1407–16. doi: 10.1016/S0140-6736(21)02183-8. https://europepmc.org/abstract/MED/34619098 .S0140-6736(21)02183-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Hajian-Tilaki K. Sample size estimation in diagnostic test studies of biomedical informatics. J Biomed Inform. 2014 Apr;48:193–204. doi: 10.1016/j.jbi.2014.02.013. https://linkinghub.elsevier.com/retrieve/pii/S1532-0464(14)00050-1 .S1532-0464(14)00050-1 [DOI] [PubMed] [Google Scholar]
- 36.Sample size software. NCSS: Statistical Software. 2012. [2023-05-28]. https://www.ncss.com/software/pass/
- 37.McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22(3):276–82. https://europepmc.org/abstract/MED/23092060 . [PMC free article] [PubMed] [Google Scholar]
- 38.R Core Team R: a language and environment for statistical computing. R Project for Statistical Computing. [2023-05-28]. https://www.r-project.org/
- 39.Harrell Jr FE. Regression modeling strategies. Comprehensive R Archive Network (CRAN) 2023. May 08, [2023-05-28]. https://CRAN.R-project.org/package=rms .
- 40.Revelle W. Procedures for psychological, psychometric, and personality research. Comprehensive R Archive Network (CRAN) 2023. Mar 18, [2023-05-28]. https://CRAN.R-project.org/package=psych .
- 41.Stephenson M, Olson SM, Self WH, Ginde AA, Mohr NM, Gaglani M, Shapiro NI, Gibbs KW, Hager DN, Prekker ME, Gong MN, Steingrub JS, Peltan ID, Martin ET, Reddy R, Busse LW, Duggal A, Wilson JG, Qadir N, Mallow C, Kwon JH, Exline MC, Chappell JD, Lauring AS, Baughman A, Lindsell CJ, Hart KW, Lewis NM, Patel MM, Tenforde MW, Network Investigators IV. Ascertainment of vaccination status by self-report versus source documentation: impact on measuring COVID-19 vaccine effectiveness. Influenza Other Respir Viruses. 2022 Nov;16(6):1101–11. doi: 10.1111/irv.13023. https://europepmc.org/abstract/MED/35818721 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Lorini C, Santomauro F, Donzellini M, Capecchi L, Bechini A, Boccalini S, Bonanni P, Bonaccorsi G. Health literacy and vaccination: a systematic review. Hum Vaccin Immunother. 2018 Feb 01;14(2):478–88. doi: 10.1080/21645515.2017.1392423. https://europepmc.org/abstract/MED/29048987 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Rowlands G. Health literacy. Hum Vaccin Immunother. 2014;10(7):2130–5. doi: 10.4161/hv.29603. https://europepmc.org/abstract/MED/25424830 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Rolnick SJ, Parker ED, Nordin JD, Hedblom BD, Wei F, Kerby T, Jackson JM, Crain AL, Euler G. Self-report compared to electronic medical record across eight adult vaccines: do results vary by demographic factors? Vaccine. 2013 Aug 20;31(37):3928–35. doi: 10.1016/j.vaccine.2013.06.041. https://europepmc.org/abstract/MED/23806243 .S0264-410X(13)00819-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Vassallo A, Shajahan S, Harris K, Hallam L, Hockham C, Womersley K, Woodward M, Sheel M. Sex and gender in COVID-19 vaccine research: substantial evidence gaps remain. Front Glob Womens Health. 2021 Nov 01;2:761511. doi: 10.3389/fgwh.2021.761511. https://europepmc.org/abstract/MED/34816252 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Heidari S, Goodman T. Critical sex and gender considerations for equitable research, development and delivery of COVID-19 vaccines. World Health Organization. 2021. Apr 18, [2023-04-06]. https://cdn.who.int/media/docs/default-source/immunization/sage/covid/gender-covid-19-vaccines-sage-background-paper.pdf?sfvrsn=899e8fca_15&download=true .
- 47.Vijayasingham L, Heidari S, Munro J, Omer S, MacDonald N. Resolving sex and gender bias in COVID-19 vaccines R&D and beyond. Hum Vaccin Immunother. 2022 Dec 31;18(1):2035142. doi: 10.1080/21645515.2022.2035142. https://europepmc.org/abstract/MED/35143380 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Fischinger S, Boudreau CM, Butler AL, Streeck H, Alter G. Sex differences in vaccine-induced humoral immunity. Semin Immunopathol. 2019 Mar;41(2):239–49. doi: 10.1007/s00281-018-0726-5. https://europepmc.org/abstract/MED/30547182 .10.1007/s00281-018-0726-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Zintel S, Flock C, Arbogast AL, Forster A, von Wagner C, Sieverding M. Gender differences in the intention to get vaccinated against COVID-19: a systematic review and meta-analysis. Z Gesundh Wiss. 2022 Jan 07;:1–25. doi: 10.1007/s10389-021-01677-w. https://europepmc.org/abstract/MED/35018277 .1677 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Ulrich AK, McKearnan SB, Lammert S, Wolfson J, Pletcher J, Halloran ME, Basta NE. Validity of university students' self-reported vaccination status after a meningococcal B outbreak. J Am Coll Health. 2022 Apr;70(3):824–9. doi: 10.1080/07448481.2020.1772270. https://europepmc.org/abstract/MED/32672510 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.King JP, McLean HQ, Belongia EA. Validation of self-reported influenza vaccination in the current and prior season. Influenza Other Respir Viruses. 2018 Nov;12(6):808–13. doi: 10.1111/irv.12593. https://europepmc.org/abstract/MED/30028081 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Zheng C, Shao W, Chen X, Zhang B, Wang G, Zhang W. Real-world effectiveness of COVID-19 vaccines: a literature review and meta-analysis. Int J Infect Dis. 2022 Jan;114:252–60. doi: 10.1016/j.ijid.2021.11.009. https://linkinghub.elsevier.com/retrieve/pii/S1201-9712(21)00857-2 .S1201-9712(21)00857-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Skowronski DM, Febriani Y, Ouakki M, Setayeshgar S, El Adam S, Zou M, Talbot D, Prystajecky N, Tyson JR, Gilca R, Brousseau N, Deceuninck G, Galanis E, Fjell CD, Sbihi H, Fortin E, Barkati S, Sauvageau C, Naus M, Patrick DM, Henry B, Hoang LM, De Wals P, Garenc C, Carignan A, Drolet M, Jassem AN, Sadarangani M, Brisson M, Krajden M, De Serres G. Two-dose severe acute respiratory syndrome coronavirus 2 vaccine effectiveness with mixed schedules and extended dosing intervals: test-negative design studies from British Columbia and Quebec, Canada. Clin Infect Dis. 2022 Nov 30;75(11):1980–92. doi: 10.1093/cid/ciac290. https://europepmc.org/abstract/MED/35438175 .6570607 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Lamontagne F, Rowan KM, Guyatt G. Integrating research into clinical practice: challenges and solutions for Canada. CMAJ. 2021 Jan 25;193(4):E127–31. doi: 10.1503/cmaj.202397. http://www.cmaj.ca/cgi/pmidlookup?view=long&pmid=33298404 .cmaj.202397 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Etheredge LM. A rapid-learning health system. Health Aff (Millwood) 2007 Mar;26(2):w107–18. doi: 10.1377/hlthaff.26.2.w107.hlthaff.26.2.w107 [DOI] [PubMed] [Google Scholar]
- 56.Smith R, Hubers J, Farraye FA, Sampene E, Hayney MS, Caldera F. Accuracy of self-reported vaccination status in a cohort of patients with inflammatory bowel disease. Dig Dis Sci. 2021 Sep;66(9):2935–41. doi: 10.1007/s10620-020-06631-6. https://europepmc.org/abstract/MED/32995995 .10.1007/s10620-020-06631-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Laurence A, Lewis P, Gately C, Dixon A. Influenza and pneumococcal vaccination: do older people know if they have been vaccinated? Aust N Z J Public Health. 2016 Jun;40(3):279–80. doi: 10.1111/1753-6405.12423. https://onlinelibrary.wiley.com/doi/10.1111/1753-6405.12423 . [DOI] [PubMed] [Google Scholar]
- 58.Lu X, Zhang L, Du H, Zhang J, Li YY, Qu J, Zhang W, Wang Y, Bao S, Li Y, Wu C, Liu H, Liu D, Shao J, Peng X, Yang Y, Liu Z, Xiang Y, Zhang F, Silva RM, Pinkerton KE, Shen K, Xiao H, Xu S, Wong GW, Chinese Pediatric Novel Coronavirus Study Team SARS-CoV-2 Infection in Children. N Engl J Med. 2020 Apr 23;382(17):1663–5. doi: 10.1056/NEJMc2005073. https://europepmc.org/abstract/MED/32187458 . [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Khan MS, Nguyen TX, Lal S, Watanapongvanich S, Kadoya Y. Hesitancy towards the third dose of COVID-19 vaccine among the younger generation in Japan. Int J Environ Res Public Health. 2022 Jun 08;19(12):7041. doi: 10.3390/ijerph19127041. https://www.mdpi.com/resolver?pii=ijerph19127041 .ijerph19127041 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Diesel J, Sterrett N, Dasgupta S, Kriss JL, Barry V, Vanden Esschert K, Whiteman A, Cadwell BL, Weller D, Qualters JR, Harris L, Bhatt A, Williams C, Fox LM, Meaney Delman D, Black CL, Barbour KE. COVID-19 vaccination coverage among adults - United States, December 14, 2020-May 22, 2021. MMWR Morb Mortal Wkly Rep. 2021 Jun 25;70(25):922–7. doi: 10.15585/mmwr.mm7025e1. doi: 10.15585/mmwr.mm7025e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Baack BN, Abad N, Yankey D, Kahn KE, Razzaghi H, Brookmeyer K, Kolis J, Wilhelm E, Nguyen KH, Singleton JA. COVID-19 vaccination coverage and intent among adults aged 18-39 years - United States, March-May 2021. MMWR Morb Mortal Wkly Rep. 2021 Jun 25;70(25):928–33. doi: 10.15585/mmwr.mm7025e2. doi: 10.15585/mmwr.mm7025e2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.COVID-19 vaccine uptake and intent: Canadian community health survey (CCHS) insight. Public Health Agency of Canada. 2021. [2023-04-06]. https://www.canada.ca/en/public-health/services/publications/vaccines-immunization/covid-19-vaccine-uptake-intent-canadian-community-health-survey.html .
- 63.Sharif A, Botlero R, Hoque N, Alif SM, Nazmul Karim M, Islam SM. A pragmatic approach to COVID-19 vaccine passport. BMJ Glob Health. 2021 Oct;6(10):e006956. doi: 10.1136/bmjgh-2021-006956. https://gh.bmj.com/lookup/pmidlookup?view=long&pmid=34610907 .bmjgh-2021-006956 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 64.Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A. Accuracy and quality of immunization information systems in forty-one low income countries. Trop Med Int Health. 2009 Jan;14(1):2–10. doi: 10.1111/j.1365-3156.2008.02181.x. https://onlinelibrary.wiley.com/doi/10.1111/j.1365-3156.2008.02181.x .TMI2181 [DOI] [PubMed] [Google Scholar]
- 65.Baum U, Kulathinal S, Auranen K. Exposure misclassification bias in the estimation of vaccine effectiveness. PLoS One. 2021 May 13;16(5):e0251622. doi: 10.1371/journal.pone.0251622. https://dx.plos.org/10.1371/journal.pone.0251622 .PONE-D-20-10809 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Definitions used to determine the interrater agreement between the self-reported vaccination status and the vaccination status determined by the Québec Vaccination Registry at the time of emergency department index visit.
Characteristics of all participants by concordant or discordant vaccination status.
Contributors to the Canadian COVID-19 Emergency Department Rapid Response Network.
Data Availability Statement
Data are available on reasonable request. For investigators who wish to access Canadian COVID-19 Emergency Rapid Response Network data, proposals may be submitted to the network for review and approval by the network’s peer review publication committee, the data access and management committee, and the executive committee as per the network’s governance. Information regarding submitting proposals and accessing data may be found on the Canadian COVID-19 Emergency Department Rapid Response Network website [9].
