Dashboard development for near real-time visualization of COVID-19 vaccine safety surveillance data in the Vaccine Safety Datalink

Abstract

The Vaccine Safety Datalink (VSD) conducts active surveillance and vaccine safety research studies. Since the start of the U.S. COVID-19 vaccination program, the VSD has conducted near real-time safety surveillance of COVID-19 vaccines using Rapid Cycle Analysis. VSD investigators developed an internal dashboard to facilitate visualization and rapid reviews of large weekly automated vaccine safety surveillance data. Dashboard development and maintenance was informed by vaccine surveillance data users and vaccine safety partners. Key metrics include population demographics, vaccine uptake, pre-specified safety outcomes, sequential analyses results, and descriptive data on potential vaccine safety signals. Dashboard visualizations are used to provide situational awareness on dynamic vaccination coverage and the status of multiple safety analyses conducted among the VSD population. This report describes the development and implementation of the internal VSD COVID-19 Vaccine Dashboard, including metrics used to develop the dashboard, which may have application across various other public health settings.

1. Introduction

Vaccines are a critical public health component to ending the COVID-19 pandemic, which as of March 4, 2022, has caused more than 78.9 million cases of COVID-19 disease and 952,223 deaths in the United States (U.S.) [1]. Since December 11, 2020, the U.S. Food and Drug Administration (FDA) has issued Emergency Use Authorizations (EUA) for three COVID-19 vaccines in the United States: Pfizer-BioNTech mRNA vaccine (BNT162b2) (Pfizer), Moderna mRNA-1273 vaccine (Moderna), and Janssen Biotech, Inc. (Johnson & Johnson) Ad26.COV2.S vaccine (Janssen) [2]. Subsequently, on August 23, 2021, the FDA approved Comirnaty (COVID-19 Vaccine, mRNA), known as Pfizer under the EUA, for individuals 16 years of age and older; the Pfizer COVID-19 vaccine currently under the EUA is in effect for younger persons. On January 31, 2022, the FDA approved Spikevax, known as Moderna under the EUA, for individuals 18 years of age and older[2]. Since 1990, the Vaccine Safety Datalink (VSD) has monitored the safety of U.S. licensed vaccines by conducting surveillance and targeted research studies on rare, unusual adverse events following immunization, and provided critical, timely scientific information to the Centers for Disease Control and Prevention (CDC) Advisory Committee on Immunization Practices (ACIP)[3]. The VSD is a collaboration between the CDC Immunization Safety Office and nine integrated healthcare systems across eight states. The participating healthcare systems (hereafter referred to as “sites”) serve approximately 12 million persons annually, or 3.6% of the U.S. population, with all major demographic groups represented and no major differences in sex, race, ethnicity, and education attainment between the VSD and the 2010 US Census population[4].

With a heightened awareness of, and focus on, COVID-19 vaccines, rapidly extracting and succinctly visualizing COVID-19 vaccine administration and safety data facilitates access to information among partners. Dashboards have the power to concisely summarize relevant information from large amounts of data, allowing for rapid detection of trends and outliers[5]. Throughout the COVID-19 pandemic, dashboards have been successfully used in health care and public health settings to track COVID-19 cases, deaths, and testing, as well as to aid management of patient flow in clinical settings[6], [7], [8], [9], [10]. The VSD initiated near real-time active safety surveillance using Rapid Cycle Analysis (RCA) for COVID-19 vaccines from the start of the U.S. COVID-19 vaccination program, and developed a novel internal VSD COVID-19 Vaccine Dashboard to facilitate timely analysis and effective dissemination of vaccine safety results [11]. This report describes the development and implementation of the VSD COVID-19 Vaccine Dashboard.

2. Materials and methods

2.1. Data sources

Creation of the VSD COVID-19 Vaccine Dashboard, which consolidates and visualizes summary coverage and safety data from eight of the nine VSD sites, was possible due to VSD’s well-established distributed data model (DDM) and dynamic data files (DDFs). The DDM enables sites to gather and maintain their site’s data on a secure server, sharing limited data sets for specific studies with CDC via a secure server. Dynamic data files allow for the ongoing capture of near real-time event-based data. Each site’s DDFs continuously capture updated health care information on its members on a weekly basis, including but not limited to vaccinations, hospitalizations, clinical encounters, emergency room visits, COVID-19 testing, enrollment, and demographic characteristics using a standardized data dictionary. Vaccination data are linked with health outcome data, both of which are captured during routine patient care visits. Each site creates a standardized set of patient files with unique study identification numbers using their electronic health record (EHR) system, and CDC obtains relevant data from site files to create specific datasets for analyses[12], [13]. Data displayed in the VSD COVID-19 Vaccine Dashboard include the COVID-19 RCA vaccine safety surveillance data, and COVID-19 disease surveillance data[11]. As part of RCA, the observed number of pre-specified outcomes of interest in a defined risk window following COVID-19 vaccines are compared to the expected number; if the observed rate is significantly higher than the expected rate, this indicates a “statistical signal.” If such a signal is identified, additional analyses are conducted to determine if there is a true association, in which case a formal epidemiologic investigation may be undertaken[13].

2.2. Dashboard development

The VSD COVID-19 Vaccine Dashboard displays key COVID-19 vaccine and health outcome metrics to facilitate rapid review of weekly automated data. The dashboard was developed using Tableau software (Version 2020.4 and later). Dashboard development began in early December 2020. During the initial design phase, a small group of subject matter experts met weekly for one month and exchanged frequent e-mail communication to create draft dashboard visualizations. The group consisted of a VSD co-Principal Investigator leading the COVID-19 RCA surveillance project, a VSD Health Scientist familiar with VSD RCA surveillance data and data visualization, and a VSD Epidemiologist charged with developing the dashboard. The small group reviewed available COVID-19 RCA data, brainstormed possible metrics and data visualizations of interest, and explored ways to share visualizations weekly across the VSD to facilitate rapid review of large amounts of data. Throughout January 2021, draft dashboard visualizations were shared with VSD staff for input during weekly meetings and via e-mail to further refine the metrics of interest and preferred visualization content and layout. Taking an iterative approach, the Epidemiologist compiled feedback after each meeting and incorporated suggested changes prior to the next meeting for further input. In February 2021, when VSD leadership approved the initial version of the dashboard, it was published on the internal CDC Tableau Server for CDC VSD staff use, and a process was established for weekly dashboard updates and quality assurance data checks. Currently, surveillance data are collated from sites on a weekly basis and analyzed using SAS (Version 9.4). SAS data sets are output into a shared internal folder, and a Tableau workbook connected to the SAS datasets is refreshed and published weekly on the Tableau Server. Once data are published, an e-mail is sent to CDC VSD staff with a link to the VSD COVID-19 Vaccine Dashboard along with a summary of any visualization changes since the previous week. A weekly e-mail is also sent to VSD sites.

The dashboard initially contained only COVID-19 RCA vaccine safety surveillance data, which included vaccine uptake, demographics, pre-specified safety outcomes following vaccination, and results of over 1,000 statistical analyses conducted weekly. However, as COVID-19 RCA surveillance data matured and accumulated, new areas of interest and metrics suggested by CDC and VSD site staff during weekly meetings were added to better understand the relationship between COVID-19 vaccination and persons hospitalized for COVID-19 disease, and vaccination uptake among sub-populations of interest. To implement these changes, COVID-19 disease surveillance data were added to the dashboard and visualized. Conversely, dated visualizations created in the early stages of dashboard development were replaced with more streamlined visualizations, which decreased the number of visualizations in the dashboard.

3. Dashboard content

Numerous COVID-19 vaccine RCA and COVID-19 disease surveillance variables are used to create key metrics monitored in the dashboard. These metrics serve as the foundation for dashboard content, allowing viewers to quickly review large amounts of data and easily share visualizations. Currently, key metrics monitored include COVID-19 vaccine uptake among all persons and among pregnant persons, demographics, pre-specified safety outcomes, results of potential associations between vaccine receipt and safety outcomes (i.e., signals), and COVID-19 disease hospitalizations among all persons and among pregnant persons. Definitions for some of the key variables that feed into dashboard metrics are described below.

3.1. Population, vaccine, and pre-specified safety outcomes identification

The vaccinated population consists of enrolled VSD members who are vaccinated and age-eligible to receive COVID-19 vaccination, as authorized by the COVID-19 vaccine EUAs. The age range of the population is adjusted when new age groups are authorized to receive COVID-19 vaccine. Individuals must be enrolled in one of the VSD sites on the day of their COVID-19 vaccination to be included in the vaccinated population. COVID-19 vaccines are identified using the CDC CVX code set[14]. VSD sites also capture vaccinations that occur outside the healthcare system (e.g., retail pharmacies, mass vaccination clinics, workplace; see 2022 publication by Groom et al. for more details)[15]. The 23 pre-specified COVID-19 RCA vaccine safety outcomes (i.e., medically-attended outcomes) (Table 1 ) are identified using the International Classification of Diseases 10th Revision (ICD-10) diagnosis codes. Diagnosis codes for most pre-specified outcomes are restricted to those assigned in the emergency department and inpatient settings; outpatient codes are included for some outcomes. COVID-19 disease among VSD members is identified using ICD-10 COVID-19 diagnosis codes and site-specific internal diagnosis codes, and/or a laboratory test positive for SARS-CoV-2 on or after January 1, 2020. Patients hospitalized with a diagnosis of COVID-19 disease since March 1, 2020 are identified using ICD-10 and site-specific internal diagnosis codes[16]. Pregnancies since December 14, 2020 are identified using the VSD’s custom dynamic pregnancy algorithm (DPA) based on ICD-10 diagnosis codes, procedure codes, estimated date of delivery, and last menstrual period date[17]. The denominator for vaccine coverage is based on VSD enrollment and is dynamic, changing from month to month.

Table 1.

Vaccine Safety Datalink COVID-19 Rapid Cycle Analysis surveillance pre-specified outcomes displayed in the VSD COVID-19 Vaccine Dashboard.

#	VSD Outcomes	Abbreviation	Settingsa	Risk Window (Days)b	Exclude if COVID 19 in X prior daysc	Monitoring Onlyd	Chart Reviewe
1	Acute disseminated encephalomyelitis	ADEM	E, I	1–21, 1–42			Yes
2	Acute myocardial infarction	AMI	E, I	1–21, 1–42	30 days
3	Acute respiratory distress syndrome	ARDS	E, I	1–21, 1–42	42 days	Yes
4	Anaphylaxis	ANAPH	E, I	0–1		Yes	Yes
5	Appendicitis	APPND	E, I	1–21, 1–42
6	Bell's palsy	BP	E, I, O	1–21, 1–42	30 days
7	Cerebral venous sinus thrombosis	CVST	E, I	1–21, 1–42	30 days		Yes
8	Convulsions / seizures	SZ	E, I	1–21, 1–42 (day 0 included for children)	30 days
9	Disseminated intravascular coagulation	DIC	E, I	1–21, 1–42	42 days
10	Encephalitis / myelitis / encephalomyelitis / encephalopathy	ENCEPH	E, I	1–21, 1–42	30 days
11	Guillain-Barré syndrome	GBS	E, I	1–21, 1–42			Yes
12	Immune thrombocytopenia	ITP	E, I, O	1–21, 1–42	30 days
13	Kawasaki disease	KD	E, I	1–21, 1–42
14	Multisystem Inflammatory Syndrome in Children & Adults	MISC, MISA	E, I			Yes
15	Myocarditis / pericarditis	MYOC	E, I	1–21, 1–42	30 days		Yes, under 40 years of age
16	Narcolepsy and cataplexy	NARC	E, I, O			Yes
17	Pulmonary embolism	PE	E, I	1–21, 1–42	30 days
18	Stroke, hemorrhagic	HSTK	E, I	1–21, 1–42	30 days
19	Stroke, ischemic	ISTK	E, I	1–21, 1–42	30 days
20	Thrombosis with Thrombocytopenia Syndrome	TTS	E, I	1–21, 1–42			Yes
21	Thrombotic thrombocytopenic purpura	TTP	E, I	1–21, 1–42	30 days
22	Transverse myelitis	TM	E, I	1–21, 1–42			Yes
23	Venous thromboembolism	VTE	E, I, O	1–21, 1–42	30 days

^aE = Emergency Department, I = Inpatient, O = Outpatient.

^bOnly the first event of an outcome that occurred during the risk interval will be counted, to avoid double counting the same outcome.

^cExclusion for outcomes of interest for comparative analyses.

^dOutcomes monitored only; not included in comparative analyses.

^eChart review of outcomes shortly after case detection to confirm possible cases and collect additional data.

3.2. Dashboard structure and metrics

VSD staff identified seven key categories, or sections, to display in the VSD COVID-19 Vaccine Dashboard, including: COVID-19 vaccine administration; COVID-19 vaccine coverage; descriptive summaries of outcomes following COVID-19 vaccination; statistical analyses of COVID-19 vaccine outcomes; COVID-19 disease hospitalization counts; descriptive summaries of simultaneous vaccine administration with COVID-19 vaccine; and descriptive summaries of COVID-19 disease and COVID-19 vaccination during pregnancy. Relevant metrics were defined within each category (Table 2 ). Metrics were used to create visualizations that were then inserted into single-page dashboards; each dashboard contains at least one visualization. Currently, there are 28 unique single-page dashboards that are grouped under eight section headers (i.e., “Home”, “RCA”, “Outcome Specific”, “Coverage”, “Hospitalizations”, “Simultaneous Vaccines”, “Detailed Tables”, and “Pregnancy”; range 2–8 single-page dashboards under each section header). Tableau’s bult-it data visualization features allow for quick creation and customization of both visualizations and single-page dashboards. Different visualization methods were used to aid with rapid reviews of data (e.g., bar charts, histograms, heat maps, line charts, etc.). A navigation panel was developed on the left side of the dashboard with section headers to aid users in quickly accessing the single-page dashboards. See Fig. 1 for an example of the VSD COVID-19 Vaccine Dashboard’s “Home” page.

Table 2.

Categories and metrics included in the internal Vaccine Safety Datalink COVID-19 Vaccine Dashboard.

Category	Metricsa
COVID-19 vaccine administration	-Number of total doses administered Number of total doses administered per 100,000 persons Number of persons who received at least one dose Number of persons fully vaccinated Change in number doses administered since last week (total doses, total doses per 100,000 persons, persons who received at least one dose, persons fully vaccinated) Number of persons who received at least one dose and number persons fully vaccinated over time, by week Number of doses administered by vaccine manufacturer and dose number, where applicable Number of persons who received at least one dose and number persons fully vaccinated by demographics (i.e., sex, age group, race/ethnicity, VSD site, high-risk for COVID-19 disease status, and COVID-19 disease history status) and by week
COVID-19 vaccine coverage	-Number and proportion of VSD population who received at least one dose Number and proportion of VSD population fully vaccinated Number and proportion of VSD population who received vaccine, by vaccine manufacturer and dose number Number and proportion of VSD population who received dose 1 and number and proportion of VSD population who received dose 2, by vaccine manufacturer and demographics Number and proportion of VSD population who received at least one dose and number and proportion of VSD population fully vaccinated by demographics and by week
Descriptive summaries of outcomes following COVID-19 vaccination	-Number of pre-specified outcomes that occurred by outcome type, vaccine manufacturer and dose number Number of pre-specified outcomes that occurred by outcome type, vaccine manufacturer and dose number, average days to onset/diagnosis, sex, pregnancy status, and history of COVID-19 disease status Outcome-specific visualizations that show number of outcomes by vaccine manufacturer and dose number, demographics, pregnancy status, and days to outcome diagnosis Number of pre-specified outcomes by days since onset and rate per one million person-days, by pre-specified outcome
Statistical analyses of COVID-19 vaccine outcomes	-Cumulative incidence of pre-specified outcomes during pre-specified post-vaccination risk intervals Rate ratios to estimate the ratio of outcome incidence in the risk interval to the outcome incidence in the comparison interval with 95% confidence intervals Sequential analyses results of the one-sided null hypothesis that the vaccine does not increase risk of outcome during the risk interval
COVID-19 disease hospitalization counts	-Number of total COVID-19 hospitalizations among VSD population over time, by month Number of total COVID-19 hospitalizations among VSD population over time (by month), compared to total number of dose 1 and dose 2 COVID-19 vaccine doses administered over time (by month) Number of total COVID-19 hospitalizations among VSD population over time (by month), by demographics Number of total lab-positive COVID-19 hospitalizations among VSD population over time (by month), by demographics Number and proportion of deceased lab-positive COVID-19 hospitalized persons by sex and age group (denominator = all lab-positive COVID-19 hospitalized persons)
Descriptive summaries of simultaneous vaccine administration with COVID-19 vaccine (i.e., administration of a non-COVID-19 vaccine on same day as a COVID-19 vaccine)	-Number of persons who received simultaneous vaccines with any dose of COVID-19 vaccine compared to number persons who initiated COVID-19 vaccine Number of persons who received simultaneous vaccinations with COVID-19 vaccine, by COVID-19 vaccine manufacturer, dose number, and demographics Number of persons who received simultaneous vaccines, by simultaneous vaccine name Number of persons who received simultaneous vaccines, by simultaneous vaccine name and by week
Descriptive summaries of COVID-19 disease and COVID-19 vaccination during pregnancy	COVID-19 Vaccine Coverage Number of pregnant persons in VSD population who received at least one dose of COVID-19 vaccine prior to or during most recent pregnancy Timing of COVID-19 vaccine receipt among pregnant persons Number of COVID-19 vaccines received by pregnant persons, by vaccine manufacturer and dose number Number of pregnant persons who initiated COVID-19 vaccination and number of pregnant persons fully vaccinated by age group and race/ethnicity Cumulative proportion of COVID-19 vaccine uptake among pregnant persons by age group and race/ethnicity COVID-19 Disease Hospitalization Number and proportion of pregnant persons hospitalized (denominator = total number women hospitalized of childbearing age) Number of pregnant persons hospitalized by race/ethnicity and VSD site Minimum, median, and maximum: 1) woman’s age at COVID-19 diagnosis in years; 2) number hospitalizations during pregnancy; 3) duration of hospital stay in days; 4) time from COVID-19 diagnosis to delivery; 5) gestational age at time of delivery in weeks Number and proportion of pregnancy outcomes among women with an outcome (i.e., live born delivery; spontaneous abortion; live born or stillborn delivery; stillborn; therapeutic abortion; ectopic pregnancy) Number babies born preterm

^aAll COVID-19 vaccine doses are those administered to patients enrolled at participating VSD sites.

Fig. 1.

Vaccine Safety Datalink COVID-19 Vaccine Dashboard “Home” page.

4. Results

The internal VSD COVID-19 Vaccine Dashboard has been accessible to CDC VSD staff via the internal CDC Tableau Server since February 2021. Single-page dashboards became available at different time points throughout dashboard development and implementation. CDC staff supporting the COVID-19 response who do not have a CDC Tableau Server account are able to view and interact with the dashboard under a “guest” profile. Additionally, a PDF one-pager highlighting key COVID-19 vaccine administration and safety metrics, presentation-ready PowerPoint slides, and a Tableau Reader workbook of the VSD COVID-19 Vaccine Dashboard are shared weekly with VSD sites that contribute weekly surveillance data. Between February 2021 and February 2022, the VSD COVID-19 Vaccine Dashboard had been viewed over 1,500 times. The number of viewers per each single-page dashboard ranged from one to 59 (“guest” was counted as one user). The top five single-page dashboards viewed were: 1) “Home” landing page, which provides an overall summary of COVID-19 vaccinations to date and across demographics, overall coverage, number of outcomes by vaccine and dose number, and sequential analyses signals; 2) “Anaphylaxis” outcome-specific page, which provides detailed information on the automated and chart-confirmed number of anaphylaxis cases that can be filtered by COVID-19 vaccine type; 3) “RCA Sequential Analyses” page, which shows a table with all sequential analyses to date and can be filtered by analysis week, outcome, vaccine type and dose number, concurrent comparison type, risk interval, and signal; 4) “Simultaneous Vaccines – All Ages” page, which summarizes the number of persons who received a simultaneous vaccine by COVID-19 vaccine type and dose number, and can be filtered by demographics; and 5) “Hospitalizations Summary” page, which shows the total number of COVID-19 disease hospitalizations and the total number of COVID-19 vaccine doses administered over time in the VSD, and can be filtered by demographics.

Dashboard features allow users to quickly review and share visualizations from the dashboard. The single-page dashboards include both interactive and static visualizations. Interactive visualizations allow viewers to drill down to data views by selecting parameters of interest, such as the timeframe, demographics, outcomes, and COVID-19 vaccine types and dose numbers. Interactive dashboard visualizations are viewed monthly during VSD meetings to inform COVID-19 vaccine safety surveillance discussions. For example, weeks and outcomes of interest are selected in the “RCA Analysis – mRNA Vaccines” dashboard, which summarizes results of analyses of associations between mRNA vaccines and safety outcomes over time, to compare how rate ratios and p-values change over time for each outcome. For a detailed description of COVID-19 RCA methods and statistical analyses conducted, please refer to the 2021 publication by Klein et al.[18]. The “Home” page is an example of a single-page static dashboard. On this most frequently viewed dashboard page, viewers can quickly see that through February 26, 2022, over 16.4 million COVID-19 vaccine doses were administered in the VSD, with 77.3% of the age-eligible VSD population receiving at least one dose and 74.5% being fully vaccinated (see Fig. 1). Viewers can download both interactive and static dashboards as an image, crosstab, PDF, PowerPoint, or Tableau Workbook to include in presentations and easily share with others.

Visualizations created by the VSD COVID-19 Vaccine Dashboard have been used to support situational awareness and inform policy makers’ decision making. Select dashboard visualizations are used in weekly presentations to the ACIP COVID-19 Vaccine Safety Technical (VaST) Work Group to provide updates on the status of COVID-19 vaccination coverage, outcomes to date, and multiple safety analyses conducted among the VSD population. One example of a dashboard presented to the VaST is the COVID-19 RCA Outcomes Summary of automated Bell’s Palsy cases identified following Moderna dose 1 (Fig. 2 ). The Bell’s Palsy dashboard displays several key variables of interest on one page that allows for quick review. Additionally, visualizations of aggregate VSD COVID-19 surveillance data from the Tableau dashboard have been used in presentations made during ACIP meetings to inform the public about the completeness of COVID-19 vaccine administration in the VSD (Fig. 3 )[19].

Fig. 2.

COVID-19 Outcomes Summary dashboard of automated Bell’s Palsy cases following Moderna dose 1. HR = High Risk for COVID-19 disease; HX = History; NH = Non-Hispanic.

Fig. 3.

Example of COVID-19 Vaccine Dashboard visualization used in a June 23, 2021 Advisory Committee on Immunization Practices presentation on COVID-19 vaccine safety updates.

5. Discussion

The CDC VSD staff developed an internal VSD COVID-19 Vaccine Dashboard for rapid visualization and review of COVID-19 vaccine administration and safety data, which informs CDC and ACIP leadership about COVID-19 vaccine safety. On average, approximately 500,000 COVID-19 vaccine doses among VSD site enrollees have been added weekly since the start of the COVID-19 vaccination program (range: 36,546–1,004,163). With over 1,000 statistical analyses conducted weekly on the COVID-19 RCA data, dashboard users have found the interactive features to be very helpful in quickly filtering dashboard visualizations to view pertinent information. As reported in the results section, the following five single-page dashboards had the most views: 1) “Home” landing page, 2) “Anaphylaxis” outcome-specific page, 3) “RCA Sequential Analyses” page, 4) “Simultaneous Vaccines – All Ages” page, and 5) “Hospitalizations Summary” page. One aspect that all of the top five single-page dashboards have in common is that they were designed to feature the most pertinent information for each respective section, and to serve as the section’s landing page (i.e., section header). When a section header is selected, one or more related single-page dashboard names are listed underneath it, allowing the viewer to select which single-page dashboard they may want to view next. For example, Fig. 1 shows the “Home” page button selected, with one single-page dashboard button directly underneath it that navigates to the “Outcomes List” page when clicked. Over the past year, as data were refreshed weekly, the updated VSD COVID-19 Vaccine Dashboard was shared with VSD and CDC staff as a link embedded in e-mail communication. When clicked, this link directed viewers to the “Home” landing page on the Tableau Server. This in part, helps to explain why the “Home” page had the greatest number of views overall; nearly 400% more views than the second most commonly viewed “Anaphylaxis” page. Additionally, the “Home” page features key metrics from various sections of the dashboard, including number of doses administered, vaccination coverage, counts of automated outcomes, and signals to date in the vaccinated concurrent comparator sequential analysis. It is likely that some viewers found most of the information they needed on the “Home” page. For viewers who wanted to explore the COVID-19 vaccine data in more detail, beyond what was featured on the “Home” page, they likely navigated to other section headers of interest (e.g., “Outcome Specific”, “RCA”, “Simultaneous Vaccines”, and “Hospitalizations”).

There are some limitations to the data displayed in the dashboard. Vaccine administration data may be incomplete, and although all sites have processes to capture vaccinations that occur outside of the healthcare system, there are data lags for inputting vaccinations which vary across sites. Additionally, COVID-19 vaccination status on individuals may rarely be inaccurate due to data entry errors. As the CDC Tableau Server is internal to the agency, sites are unable to access it directly. To circumvent this limitation, sites are provided with PDF, PowerPoint, and Tableau Reader dashboard files weekly, and are also presented with relevant visualizations during weekly meetings.

The VSD COVID-19 Vaccine Dashboard development has been an iterative process, with visualizations and single-page dashboards evolving as new COVID-19 vaccines became available, new safety outcomes emerged, and vaccination recommendations expanded to younger age groups. The CDC VSD staff also regularly reviewed the CDC COVID Data Tracker to ensure alignment in terminology and to gain new ideas for visualizing vaccination data[20]. The use of Tableau software allows for flexibility in both design and content, with visualizations frequently updated to keep pace with the changing landscape of COVID-19 vaccinations, and to answer emerging vaccine safety questions of interest among CDC VSD and site staff and leadership. Continuous feedback from VSD staff about the design and layout of these visualizations ensures that dashboard content stays relevant. A recent example of this is the expansion of the VSD COVID-19 Vaccine Dashboard to include data on COVID-19 vaccine booster doses. As COVID-19 vaccine recommendations continue to evolve the dashboard may also include vaccine safety analyses among the younger (less than five years old) pediatric population.

It is vital that vaccine safety surveillance dashboard development and maintenance is continuously informed by key vaccine surveillance data users and vaccine safety partners. Vaccine surveillance data dashboards are useful tools for rapidly producing visualizations to aid in data exploration and to support situational awareness and policy makers’ decision-making. The internal VSD COVID-19 Vaccine Dashboard has been an essential tool for quickly visualizing dynamic data throughout the public health vaccination response to the COVID-19 pandemic, helping leadership quickly assess and answer critical vaccine safety questions. In the broader landscape of the U.S. government’s response to the COVID-19 pandemic, dashboards have played an important role, at both the federal and state levels. For example, in Indiana, dashboards utilizing clinical and public health data on COVID-19 disease supported state and local health departments with COVID-19 response, aiding in detection of a local community outbreak and tracking pandemic disease spread[6]. This report describes the development and implementation of the VSD COVID-19 Vaccine Dashboard, including potential metrics that may be displayed. Dashboards visualizing COVID-19 vaccination data may have application across various public health settings, including state and local health departments, to identify vaccination gaps across age, sex, race/ethnicity, and location.

Author contributions

All CDC VSD authors contributed to the conceptualization of the report and all authors reviewed and edited the report. TAK led the writing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.