An audit test evaluation of state practices for supporting access to and promoting Covid-19 vaccinations

Abstract

A field experiment, using a paired audit testing design with testers of different racial and language profiles, was conducted to document and evaluate individual encounters in inquiring about COVID-19 vaccinations in the U.S. states. Testers communicated with state health department and major vaccination site staff about obtaining the COVID-19 vaccine and assessed the extent to which evidence-informed communication tactics for encouraging take-up were employed. The audit testers included individuals representing Latinx identities, given research showing they face greater hardships in navigating vaccine infrastructure and place less trust in public immunization efforts. Data were collected in phone and electronic communications between mid-June and mid-August of 2021. Empirical analyses confirmed that states vary considerably in how clearly officials communicate vaccination requirements and procedures, and in what they ask of individuals before providing the opportunity to receive the COVID-19 vaccine. The Spanish-speaking tester was more likely to encounter negative or racialized language—primarily implicit in nature—such as calls abruptly ended (vs. attempting to secure language support) and requests for additional identification or personal information before continuing with vaccination registration. Examples of overtly negative or racist encounters included condescending comments about Latinx testers' identification (or perceived undocumented status) and the Spanish-speaking tester's communication in Spanish. Analysis of an index of good practices constructed from the audit data revealed that very few strategies for promoting vaccinations were regularly implemented. In regression analyses, an additional point on the good practices index predicted a 0.133 percentage point increase in the percent of the population receiving the first vaccine dose, suggesting the lack of implementation of these good practices may represent a missed opportunity to increase COVID-19 vaccination rates. We identified exemplars in the communications that the federal government could disseminate to rapidly improve state practices and the accessibility of accurate information on COVID-19 vaccination opportunities.

“Nothing is more expensive than a missed opportunity.” H. Jackson Brown, Jr.

1. Introduction

The United States is struggling to reach the 70 percent (fully) vaccinated rate that a population needs to control COVID-19 through herd immunity, assuming no clustering of unvaccinated groups that can contribute to disease outbreaks (Shen et al., 2021). As of the beginning of 2022, there were still states with rates of full vaccination below 50 percent. Among the factors consistently associated with lower intent to vaccinate are race, age (under 60 years), lower education, and political ideology, as well as fears driven by historical repression and harms associated with experimental medical research on communities of color, although disparities by race have been narrowing with time (Ndugga et al., 2022; Salmon et al., 2021; Shen et al., 2021).

The medical literature provides clear guidance—including strategies for promoting COVID-19 vaccination and communication tactics based on vaccine hesitancy levels—that those in policy and health care roles can employ to encourage vaccination, particularly in communities of color that may be less likely to have information on COVID-19 safety procedures and vaccine access (CDC, 2020; Wood and Shulman, 2021). Examples include: ensuring accessibility and ease of take-up; creating reminders and incentives through social media and networks; and developing targeted education and persuasive communications. In a randomized control trial (RCT) conducted in a California county, Chang et al. (2021) found positive impacts of public health video messages (e.g., affirming the safety of the vaccines) on vaccination intentions, although negatively framed messages (about the potential health consequences of not getting vaccinated) decreased 30-day vaccination rates for some subgroups.

While federal government policy explicitly conveys that everyone has a right to the coronavirus vaccine regardless of immigration status, the U.S. states vary considerably in how clearly officials communicate vaccination expectations and requirements, and in what they ask of individuals before administering vaccines (Shen and Orenstein, 2020). This disparate messaging persists even though the Department of Homeland Security has called it “a moral and public health imperative to ensure that all individuals residing in the United States have access to the vaccine” (DHS, 2021). As tracked by the Kaiser Family Foundation (2021), the registration process for COVID-19 vaccination differs across (and sometimes within) states, as vaccination sites can impose additional rules that create potential barriers to access, including policies or procedures that may exacerbate racial and ethnic gaps in vaccinations. For example, 26 states have restricted access to COVID-19 vaccinations to people who live and work in the state and require proof of residency with documents such as a driver's license or work identification (ID). Moreover, only about one-fourth of state websites convey clearly that undocumented immigrants are eligible to be vaccinated or attempt to address fears about negative effects on immigration status, and relatively few states with residency requirements allow undocumented immigrants to obtain driver's licenses or state ID cards.

In this research, we aimed to document the experiences of individuals attempting to access COVID-19 vaccinations in all U.S. states through an audit study, that is, a field experiment that allows researchers to examine actions or behavior in real-world scenarios. In developing this study, we drew on the concept of “administrative burden,” defined as overly burdensome or “onerous” individual encounters with government in policy implementation (Burden et al., 2012). Examples of administrative burden include time and effort invested to understand how to access a public program or benefit and the costs associated with complying with rules or procedures for accessing the benefit, such as producing required documentation. Although administrative burdens can serve legitimate purposes in administering public programs—e.g., requiring the completion of forms for assessing the veracity of claims on public funds and implementing procedures that facilitate efficient rationing of limited resources—they may also impose excessive costs on those attempting to access public benefits (Herd and Moynihan, 2019). Moreover, historically underserved and disadvantaged groups often struggle more in overcoming the learning, compliance, and psychological costs (e.g., stigma) they encounter (Heinrich, 2016; Nisar, 2018), which may exacerbate racialized inequities in access to public benefits (Ray et al., 2020). We therefore sought to understand how administrative burdens may contribute to government challenges in increasing COVID-19 vaccination rates, as well as how evidence-informed practices for promoting vaccinations might counter these burdens and boost COVID-19 vaccination rates. As vaccines were widely available at the time of our study, our focus is not on allocative exclusion that might prohibit vaccine access (Olsen et al., 2020), but rather on the accessibility, quality and tone with which information about vaccinations was conveyed in interactions with agency staff.

In conducting the correspondence audit study, we created identities for hypothetical individuals—an English-speaking White, an English-speaking Latinx, and a Spanish-speaking Latinx—to request information from state health departments and a major vaccination site within each state on how to register for and receive the vaccine. The decision to specify testers representing Latinx identities via names, accent, and English speaking abilities was based on evidence of added hardships they face when navigating the U.S. vaccine infrastructure, and lower trust in public immunization efforts among populations at disproportionate risk for COVID-19 (Salmon et al., 2021), as well as research suggesting that administrative burdens are embedded in inherently racialized organizations that mediate citizen access to state-provided benefits (Ray et al., 2020). For example, in a recent audit study analysis of discrimination, Olsen et al. (2020) distinguished three types of discriminatory behavior: unconscious bias in interactions arising from implicit, negative beliefs about others; explicit, “taste-based” discrimination reflecting more hardened, negative attitudes toward others; and statistical discrimination based on assumptions about individuals as members of a group. Although we are not able to objectively differentiate among these three types of discrimination in tester interactions, we collect both quantitative and qualitative data to assess the nature of tester interactions and the accessibility and quality of information provided through them.

The testers called and sent emails to state health departments and major vaccination sites (MVS) in all U.S. states and the District of Columbia (DC). We included a MVS in each state given reports that these sites “often make up their own rules (KFF, 2021), i.e., exercising bureaucratic discretion that the administrative burden literature shows can discriminate with intent (Nisar, 2018; Ray et al., 2020). We also compiled state website documentation on COVID-19 vaccination registration processes, requirements, and communications, as well as other publicly available data on state vaccination rates, vaccine registration priorities and requirements, social distancing regulations, proportions of COVID-19 at-risk adults, political ideology, and other state population characteristics.

In the analyses we perform, we first quantify the extent to which we observed the application of recommended strategies and practices for promoting COVID-19 vaccination in interactions with state health departments and MVS. We examine differences in the application or use of these strategies and practices by: (i) state agencies vs. MVS, (ii) racial profile/language of the tester, and (iii) form of contact (phone, email, response form or chat). In addition, we construct an index of “good practices” for promoting COVID-19 vaccination based on existing research and generate index scores from the audit data collected, compiled across the audit testers. We then use the good practices index to predict state COVID-19 vaccination rates—including first dose and series completion by age group—while controlling for state COVID-19 vaccination policies, social distancing regulations, and other state characteristics. We also present insights from qualitative data collected in the audit study that allow for deeper understanding of the individual encounters with state agencies and MVS and how language of communication and perceived immigration status affected access to COVID-19 vaccination information. We conclude with policy recommendations for reducing barriers to COVID-19 vaccinations and supporting our collective progress toward community immunity.

2. Study design, data and methods

The correspondence audit study employed a paired testing design (Gaddis, 2018) in which the organizations being audited—state health departments (or other state agencies managing COVID-19 vaccination distribution) and a MVS within each state—received both emails and phone calls from testers with different (racial/ethnic) attributes. The three testers (English-speaking White, English-speaking Latinx, and a native language, Spanish-speaking Latinx) followed an Institutional Review Board (IRB)-approved script (Appendix A) to request information on how to register for and receive the vaccine. We were limited by study resources to testing in a single gender; however, given that a previous U.S.-based study (Margolius et al., 2021) of COVID information hotline callers reported that 67 percent of callers were females, we went with the gender that is more likely to make these inquiries. Other identifying information that might be requested from the testers in the interactions (age, address, health insurance, employment status, and access to transportation) was established to be the same for each tester (see Appendix A). If asked what type of identification they had available, testers indicated they had a consular ID (a type of ID commonly used among some immigrants). Zip codes for the tester location were randomly selected within each state using a national zip code database and were offered when an agency representative sought to identify a vaccine location for them. The testers asked about ID requirements, if health insurance was required, and what to do when they got to the vaccination site (signaling some concern about the vaccine's potential effects). To minimize burden to the organizations, we asked for this information in a few simple questions that we expected would involve minimal effort to address.

All phone calls and electronic communications were initiated during typical business hours of 9 a.m. to 5 p.m. (or within the identified business hours), and the testers made multiple attempts as necessary to establish contact, varying the days and times of the calls and recording the dates of attempts and notes on the efforts. For each attempt, the testers recorded the time they waited on hold, if they left a voice message and requested a call back, the content of automatic responses, and when a phone number was disconnected or any other reason for a failure to ultimately make contact. For electronic communications, the audit testers waited a minimum of two weeks for responses to email and chat attempts or response form requests before recording that no response was received, and copies of interactions (which might span multiple emails) were retained in email accounts or documents (for chat exchanges or response forms). With three testers collecting data from two entities (state agency and MVS) by phone and by email in the 50 states and DC, there were potentially 612 sets of interactions to result from the communication attempts and to evaluated.

Publicly available data on the state agencies managing COVID-19 vaccinations and the MVS in the 50 U.S. states and DC were compiled to facilitate the audit testing, including: the agency name (typically a state health department) and the agency's listed phone number for public inquiries (frequently a COVID information hotline), and email address for COVID-19 vaccine information; this same information was also extracted for the state's MVS. This information was identified in early June 2021, and the testers began contacting state agencies and MVS in mid-June 2021. In the first two weeks of June, the testers conducted inter-rater reliability tests of their execution of the protocol (script) by phone and email with a non-U.S. state site (Puerto Rico) to ensure consistent approaches in use of the prompts and follow-up questions in the script. Data collection was completed by mid-August 2021.

3. Data collection

To capture information on tester interactions with state agencies and MVS, we created a REDCap data collection form that was IRB-approved and completed by the testers following each interaction. The REDCap form (shown in Appendix B) includes the following main topics of data collection: (i) form and language of communication; (ii) the length of time to connection with an individual or wait time for a response; (iii) availability of translation support and languages offered; (iv) information received on the COVID-19 vaccination and where to receive it; (v) information on ID requirements; (vi) availability of transportation support; (vii) health insurance requirements; (viii) other information provided about vaccine procedures, appointment reminders and potential side effects; (ix) evaluation of the communication in regard to use of strategies for encouraging vaccine use, and (x) evaluation of the communication in regard to support for meeting documentation requirements. The data elements concerning tactics for encouraging vaccine use and communications about documentation requirements were derived from a New England Journal of Medicine study (Wood and Schulman, 2021), which draws on consumer and survey research and behavioral economics to identify strategies to prioritize in interactions with the public based on vaccine-hesitancy levels.

The total number of tester-organization interactions captured in the database was 457 (of 612 possible), which reflects that some state agencies and vaccination sites could not be reached. In 40 of the 50 states plus DC, at least one tester failed to connect by phone and/or email with a state agency and/or MVS. Of the 155 failures to complete a connection, only 18 (11.6%) were attempts to communicate with a state health department; the rest were failed connections with MVS, and many of these (51) were breakdowns or non-response in electronic forms of communication. The large share of failed communications with MVS may have in part reflected their more limited infrastructure and some site closings as vaccine demand was waning. It was frequently noted by the testers that auto-responses or re-directions by an auto-attendant led to “dead ends” for information on COVID-19 vaccinations. For example, one tester recorded that she made three attempts at phone contact with an MVS and all went to a voicemail, which offered the auto-response: “COVID vaccine not available."

In addition to the original audit study data collection, we compiled publicly available data on the U.S. states and DC on vaccination rates (CDC, updated daily); vaccine registration priorities and requirements, social distancing regulations, shares of COVID-19 at-risk adults, and health insurance rates (Kaiser Family Foundation); political ideology and voting outcomes; immigration policy (National Conference of State Legislatures), and other state population characteristics. Table 1 presents a summary of these data and the specific measures that were included as control variables in the empirical models predicting state vaccination rates.

Table 1.

Publicly available data compiled by state.

Vaccination Rates (recorded June 14, 2021, CDC) and At-risk Populations	Vaccine Policies and Preventative Regulations	Other State Characteristics
% partially vaccinated (first dose received)	Vaccine registration priorities	State population demographics
% fully vaccinated (series complete)	Vaccine requirements - no vaccine residency requirement	Geographical region
Vaccination rates by age group - adults age 18+ and adults age 65+	Social distancing regulations - state easing restrictions, state pausing restrictions, lifted stay at home orders	Political control/ideology – 2016 national election vote margin in favor of Democrats
Share of at-risk adults age 18+ and share of at-risk adults age 65+	Travel/quarantine policies - mandatory travel quarantine, travel quarantine in place	Driver's license or state ID allowed for undocumented immigrants
	Statewide mask requirement, ban on large gatherings	Paid sick leave
	Business limits and closures - non-essential business limits, some non-essential businesses open	Health insurance rates – uninsured, share of private sector enrollees self-insured
	Emergency declaration	Poverty rate
	Medicaid coverage of vaccinations	Medicaid spending per enrollee
	Vaccination offered at no cost

Source/Notes: SOURCE KFF (Kaiser Family Foundation) Public COVID-19 Data and Policy Actions. NOTES The table includes broad categories of data compiled and specific measures included in regression analyses.

4. Methods of analysis

A first objective of the analysis was to assess the state-audit tester interactions, particularly the use of recommended strategies and practices for promoting COVID-19 vaccinations. The interactions were examined: by state agencies vs. MVS and by form of contact (phone, email, response form or chat) to evaluate the consistency of the practices and information provided, and by the racial profile/language of the tester to assess possible racial/ethnic discrimination in any differential responses. We performed cross-tabulations and computed chi-square tests to determine whether the relationships between the interactions and the agency/site, form of communication, and tester profile were statistically significant at α < 0.05.

We next used the audit study data on state/MVS responses to tester inquiries, including the use of tactics or strategies for encouraging vaccine use, to construct an index of “good practices” for supporting and promoting vaccinations among state populations. The index consists of 15 audit study data elements, shown in Table 2 along with their means. We performed one-way analysis of variance tests to examine whether good practices for vaccine promotion varied by state agencies vs. MVS, form of communication, and tester language/racial profile.

Table 2.

Index of good practices for promoting COVID-19 vaccinations.

Data Element	N	Mean
Translation available	452	0.478
Received vaccine information	457	0.794
Received vaccine location information	457	0.659
Provided educational material regarding the vaccine	455	0.134
Used persuasive language to encourage vaccine use	455	0.068
Used narratives or nudges to encourage vaccine use	455	0.055
Supported the ability to access information in a native language	455	0.174
Did not use any patronizing or judgmental language	455	0.736
Did not use any negative language pertaining to the politicizing of identity and/or identification	455	0.732
Did not use any negative language that was racialized in nature	455	0.756
No ID requirements	455	0.642
Dispelled concerns about documentation requirements	455	0.213
Connected me to trusted individuals or community vaccine advocacy group	455	0.013
No health insurance requirement	452	0.981
Transportation support offered	452	0.064

Source/Notes: SOURCE Authors' original audit study data collection. NOTES Index of Good Practices Mean = 6.0, Median = 6, Standard deviation = 2.45, Maximum = 13, Minimum = 0.

We then estimated multivariate regression models to assess whether the application of good practices for vaccine access and encouragement (as measured with the index variable) predicted COVID-19 vaccination rates in U.S. states. The outcome variables included: the % of the state population that completed one dose of the COVID-19 vaccine; the % of the state population that completed the full (two dose) vaccine series; the % of those age 65 years and over that completed one dose (and the full vaccine series), and the % of those age 18 years and over that completed one dose (and the full vaccine series). The other explanatory variables, shown in Table 1, are included to control for other state policies or attributes that might affect vaccination rates, such as vaccination requirements and priorities, regulations encouraging population health, region, and other state characteristics that might reduce (or increase) barriers to vaccination (e.g., paid sick leave, health insurance and immigration enforcement) or vaccination resistance (e.g., political ideology). Standard errors in the regressions were clustered by state and adjusted for heteroskedasticity (robust clustered standard errors). We interpret the model results as associations, not causal relationships, given there may be other factors that also affect state vaccination rates that we do not observe.

We analyzed the qualitative data captured through REDCap and identified experiences encountered primarily by the Latinx testers (English with a Spanish accent and Spanish speaking). We categorized these experiences of COVID- 19 vaccine access as follows: translation services availability, state residency requirements for vaccine access, authorized immigration status for vaccine access, referral to vaccine sites for identification requirements, and negative connotations in relation to immigration status. We counted the frequency of each type of interaction and utilized the qualitative data to assess whether communication regarding language access or immigration status plausibly reflected explicit, taste-based discrimination vs. implicit or statistical discrimination.

5. Study findings

In Table 3 , we present the findings of statistical tests for differences in state/MVS responses to the audit testers and their application of recommended strategies and practices for promoting COVID-19 vaccination by: state agencies vs. MVS, the racial profile/language used by the testers, and form of contact (phone vs. electronic communications). Looking across the columns in Table 3, it is immediately apparent that there were very few differences between state agencies and MVS in their responses and communication practices as documented by the testers. For the three items showing statistically significant differences, health department representatives were less likely to provide information on how to receive the vaccine and were more likely to respond in ways that were perceived as being racially discriminatory or judgmental.

Table 3.

Differences in observed practices by organization, racial profile/language, and form of communication.

	Statistically Significant Difference in Means
	State Health Dept.(HD) v. MVS (VS)	Language: E = English-speaking, S=Spanish-speaking	Communication form: P = phone, N = electronic (nonverbal)
Practices to Reduce Barriers to Vaccination
Translation available	HD = 49%, VS = 46% (p = 0.738)	E = 38%, S = 66% (p = 0.000)	P = 67%, N = 22% (p = 0.000)
Provided info. on how to receive the COVID-19 vaccine	HD = 77%, VS = 85% (p = 0.022)	E = 88%, S = 59% (p = 0.000)	P = 79%, N = 82% (p = 0.007)
Provided info. regarding COVID-19 vaccine locations	HD = 67%, VS = 66% (p = 0.200)	E = 75%, S = 48% (p = 0.000)	P = 67%, N = 66% (p = 0.202)
Transportation support to receive the vaccine was offered	HD = 7%, VS = 6% (p = 0.977)	E = 8%, S = 3% (p = 0.085)	P = 7%, N = 6% (p = 0.802)
Established that health insurance is not needed to receive the vaccine	HD = 2%, VS = 2% (p = 0.682)	E = 3%, S = 1% (p = 0.074)	P = 3%, N = 1% (p = 0.084)
Provided additional information about what to do upon arrival at the vaccine location	HD = 32%, VS = 33% (p = 0.366)	E = 37%, S = 23% (p = 0.000)	P = 38%, N = 27% (p = 0.002)
Provided additional information that addressed concerns regarding side effects	HD = 15%, VS = 17% (p = 0.356)	E = 18%, S = 10% (p = 0.000)	P = 13%, N = 20% (p = 0.171)
Offered to provide email or text reminders about the appointment	HD = 4%, VS = 4% (p = 0.673)	E = 5%, S = 2% (p = 0.180)	P = 6%, N = 2% (p = 0.000)
Connected to trusted individuals or community vaccine advocacy group	HD = 1%, VS = 2% (p = 0.599)	E = 0%, S = 3% (p = 0.022)	P = 0%, N = 3% (p = 0.044)
Strategies to Encourage Vaccination
Provided educational material regarding the vaccine	HD = 14%, VS = 12% (p = 0.549)	E = 18%, S = 3% (p = 0.000)	P = 7%, N = 23% (p = 0.000)
Used persuasive language to encourage vaccine use	HD = 8%, VS = 6% (p = 0.389)	E = 8%, S = 5% (p = 0.250)	P = 5%, N = 9% (p = 0.076)
Used narratives or nudges to encourage vaccine use	HD = 7%, VS = 4% (p = 0.224)	E = 6%, S = 4% (p = 0.385)	P = 8%, N = 3% (p = 0.018)
Supported the ability to access information in a native language	HD = 17%, VS = 18% (p = 0.850)	E = 2%, S = 50% (p = 0.000)	P = 24%, N = 9% (p = 0.000)
Did not use any patronizing or judgmental language	HD = 71%, VS = 78% (p = 0.104)	E = 88%, S = 43% (p = 0.000)	P = 81%, N = 64% (p = 0.000)
Did not use any negative language pertaining to the politicizing of identity/identification	HD = 70%, VS = 78% (p = 0.045)	E = 90%, S = 37% (p = 0.000)	P = 81%, N = 63% (p = 0.000)
Did not use any negative language that was racialized in nature	HD = 72%, VS = 81% (p = 0.027)	E = 91%, S = 43% (p = 0.000)	P = 85%, N = 63% (p = 0.000)
Practices Regarding Identification
Clarified that personal identification was necessary to receive the vaccine	HD = 46%, VS = 46% (p = 0.804)	E = 49%, S = 33% (p = 0.000)	P = 59%, N = 29% (p = 0.000)
Established that a state license, photo ID, and/or proof of residency were necessary to receive the vaccine	HD = 16%, VS = 21% (p = 0.168)	E = 24%, S = 6% (p = 0.000)	P = 26%, N = 8% (p = 0.000)
Established that the person needs to have a state license to receive the vaccine	HD = 5%, VS = 7% (p = 0.531)	E = 8%, S = 0% (p = 0.002)	P = 9%, N = 2% (p = 0.003)
Copy of a utility bill with a corresponding name and address is proof of identification	HD = 2%, VS = 3% (p = 0.562)	E = 1%, S = 3% (p = 0.226)	P = 3%, N = 2% (p = 0.283)
Inquired about the personal identification available to me	HD = 8%, VS = 10% (p = 0.748)	E = 10%, S = 8% (p = 0.247)	P = 17%, N = 1% (p = 0.000)
Consulate identification was not an acceptable form of identification	HD = 1%, VS = 1% (p = 0.798)	E = 1%, S = 2% (p = 0.152)	P = 2%, N = 0% (p = 0.093)
Addressed questions about ID requirements	HD = 64%, VS = 64% (p = 0.918)	E = 74%, S = 42% (p = 0.000)	P = 74%, N = 51% (p = 0.000)
Provided a list of documents that were acceptable forms of documentation	HD = 2%, VS = 2% (p = 0.682)	E = 11%, S = 1% (p = 0.000)	P = 8%, N = 7% (p = 0.837)
Probed for more information regarding documentation	HD = 7%, VS = 5% (p = 0.407)	E = 7%, S = 4% (p = 0.221)	P = 10%, N = 1% (p = 0.000)
Dispelled concerns about documentation requirements	HD = 19%, VS = 24% (p = 0.188)	E = 26%, S = 11% (p = 0.000)	P = 21%, N = 22% (p = 0.741)
Provided information on how to obtain the required documentation	HD = 1%, VS = 1% (p = 0.550)	E = 1%, S = 0% (p = 0.169)	P = 1%, N = 1% (p = 0.469)
Referred to specific vaccination sites or providers for information on ID requirements	HD = 23%, VS = 25% (p = 0.603)	E = 26%, S = 21% (p = 0.244)	P = 22%, N = 26% (p = 0.456)

Source/Notes: SOURCE Authors' original audit study data collection. NOTES One-way analysis of variance tests used to assess statistical significance (at α < 0.05) of differences in means.

In contrast to the consistency in responses and practices between state agencies and MVS, the responses and practices as experienced by the audit testers varied substantially by racial profile/language of the testers. As the interactions of the two English-speaking testers (using English and Latinx names) with state agencies and MVS were experienced more similarly than the Latinx tester who communicated in Spanish only, we present the differences by tester language in Table 3, and then we elaborate on the racialized experiences of the testers below. As expected, for the tester that communicated in Spanish, she was more likely to have translation made available by phone and to be supported in accessing the information in her native language, as communications could not proceed in the absence of translation. Yet in eight different interactions, the state agency or MVS staff abruptly ended the call rather than attempting to secure language support, and in another nine interactions of the Spanish-speaking tester, the call had to be terminated because they could not communicate. While the English-speaking testers did not require translation to receive the information requested, they also documented cases in which translation was not available, such as when automated message conveyed information only in English and included no options for translation. A number of other statistically significant differences in the experiences of the English-vs. Spanish-speaking testers shown in Table 3 reflect that the Spanish-speaking tester (without access to translation) was subsequently less like to receive information on how and where to receive the COVID-19 vaccination or educational materials about the vaccine.

Even though all three audit testers responded that they possessed a consular ID if probed in the interactions, the Spanish-speaking tester was significantly less likely to have her questions about ID requirements addressed and her concerns dispelled (and in a nonjudgmental manner) than the English-speaking testers. In addition, only the two testers with Latinx identities experienced negative judgments about having a consulate ID. For example, in one phone encounter when the English-speaking Latinx tester indicated that she had a consular ID, the respondent asked (in a condescending tone) if that was “like an alien card.” The two Latinx testers recorded five such instances of explicitly racialized interactions about identification (or perceived undocumented status), and the Spanish-speaking tester experienced three additional overtly negative retorts about her communication in Spanish.

This also suggests that the large majority of perceived negative or judgmental exchanges recorded by the testers were implicit in nature. For example, the English-speaking Latinx tester noted that a respondent would not register her for a vaccine appointment and provide any information on vaccine locations unless she gave a “full street [home] address,” and this was experienced in 15 different interactions. Evidence of state residency was also requested from the testers in more than 20 different interactions in order to receive additional information or to be registered for the vaccine. For instance, the Spanish-speaking tester was told that the representative could not answer any of her questions about identification requirements if her form of ID did not have a Georgia address, and she could therefore not be registered to receive the vaccine. In another 10 interactions, the Latinx testers were told that they would need to bring a social security number or permanent ID card to the vaccine location. It was also fairly common for all three of the testers to be referred to specific vaccine locations for the information on ID requirements. The English-speaking tester with the English name noted that one respondent expressed exasperation with the ID question and said: “Just take what you have and see what they say.” Through the lens of administrative burden, the additional documentation/ID requirements imposed by state health department or MVS staff contributed to higher learning, compliance and sometimes psychological costs (e.g., stigma associated with not speaking English or not having a typical ID card like a driver's license) of getting vaccinated, with the added burdens falling disproportionately on the Spanish-speaking tester.

Regarding the form of communication, contacting state agencies and MVS by phone (vs. email or other electronic forms) had both advantages and disadvantages. In general, the audit testers encountered more practices oriented toward promoting COVID-19 vaccination in their phone inquiries, such as the use of nudges to make vaccination appointments and offers to send appointment reminders, although educational materials about the vaccine were more often provided electronically. Translation was more likely to be facilitated by phone, although phone respondents were also more likely to ask about the tester's ID and to indicate that ID and residency were required to get vaccinated (a potentially stigmatizing practice). Communications by chat were less likely to provide information on how to get the vaccination and on where vaccinations could be received.

6. Findings on good practices

The summary statistics on good practices for promoting COVID-19 vaccinations indicate a wide range of index scores, from 0 to 13 on a scale with a maximum value of 15, and a mean and median of 6. The states with the highest average scores (above 7) on the good practices index (in rank order, beginning with the highest) included Indiana, Colorado, California, Tennessee, Wisconsin, Oregon and Kentucky. Looking at Table 2, which indicates the frequency with which particular practices were observed, it is clear that very few of the strategies recommended for promoting vaccinations⁵ were implemented in practice. For example, persuasive language and nudges or narratives to encourage vaccination, which are no- or low-cost strategies, were employed by only 5–7 percent of state agency or MVS representatives in the interactions with audit testers. In addition, even though the script (tester characteristics) suggested challenges with transportation (no driver's license), transportation support was only offered in about 6 percent of encounters. This occurred despite the fact that publicly available, web-based information indicated that free rides to vaccination locations were available by Uber or Lyft and other publicly-funded supports.

The analysis of variance test by state agency vs. MVS found no statistically significant differences in the application of good practices by organization type (state agency mean = 5.89, MVS mean = 6.16, p = 0.260). Alternatively, the analysis of variance test by racial profile/language of the testers showed statistically significant differences by the racial profile of the respondent, with the lowest average score on the good practices index experienced by the Latinx audit tester speaking Spanish (Spanish-speaking tester mean = 5.08, English-speaking testers = 6.43, p = 0.000). Regarding the form of communication, inquiries made by phone scored the highest on the good practices index (phone mean = 6.33, email mean = 5.48, chat mean = 5.55, response form mean = 5.71, p = 0.009).

Table 4 presents the results of multivariate regression models predicting state COVID-19 vaccination rates (for completing the first dose and for completing the full series), including estimates for each covariate included in the models. The findings show that in both models, the good practices index was a statistically significant and influential predictor of vaccination rates, and the estimated association was stronger for the first dose. For each additional point on the good practices index, the percent of the population that received the first dose increased by 0.133 percentage points, while the percent of the population that completed the vaccine series increased by 0.085 percentage points (holding all else constant). More restrictive (preventative) state policies for addressing COVID-19 were positively associated with vaccination rates, along with political (2016 election) margins favoring the Democrats. The share of adults ages 18 and older at higher risk for COVID-19 was also positively associated with state vaccination rates, while the share of private sector self-insurance enrollees was negatively associated with state vaccination rates. In addition, states in the West and South had significantly lower predicted vaccine rates than the Northeast. Not only were the predicted relationships (covariate coefficients) in the direction expected based on existing knowledge, but the percentage of state variation in vaccination rates explained in both models was high, about 93% for the model estimating first dose completion and nearly 90% for the model predicting full vaccine series completion.

Table 4.

Results of multivariate regressions predicting state COVID-19 vaccination rates.

(N = 455)	Completed 1st vaccine dose		Completed vaccine series
Predictor variable	Coefficient	Std. error	Coefficient	Std. error
Good practices index	0.133	(0.04)	0.085	(0.04)
Share of at-risk adults age 18+	127.79	(32.07)	133.83	(26.59)
Share of at-risk adults age 65+	−172.80	(29.20)	−175.39	(27.06)
State easing restrictions	−0.41	(1.57)	1.83	(1.45)
Paused release of restrictions	4.59	(2.99)	6.83	(3.10)
Lifted stay at home orders	−0.08	(1.44)	−1.66	(1.63)
Mandatory travel quarantine	3.22	(1.16)	2.41	(1.24)
Travel quarantine in place	−0.08	(2.51)	2.11	(2.84)
Non-essential business limits	3.39	(2.93)	1.39	(2.48)
Some non-essential businesses open	4.49	(2.88)	0.83	(2.72)
Ban on large gatherings	4.37	(1.41)	2.59	(1.20)
No restaurant limits	−1.52	(2.03)	−2.06	(1.65)
Statewide mask requirement	5.72	(2.23)	3.73	(2.40)
Emergency declaration	−4.71	(1.88)	−7.74	(2.21)
No vaccine residency requirement	−0.53	(0.87)	0.43	(0.96)
Vaccination offered at no cost	−1.35	(1.54)	−0.08	(1.51)
Paid sick leave	0.90	(1.67)	3.54	(1.76)
Share private sector enrollees self-insured	−26.59	(10.54)	−11.60	(11.41)
Vote margin in favor of Democrats	12.47	(3.51)	6.23	(3.41)
Driving privileged for unauthorized immigrants	1.07	(1.65)	0.65	(1.63)
Midwest region	−3.55	(2.17)	−1.04	(2.38)
South region	−6.37	(2.23)	−3.47	(2.41)
West region	−8.08	(1.63)	−6.51	(2.01)
Constant	57.11	(10.38)	41.87	(11.35)
R-squared	93.1%		89.6%

Source/Notes: SOURCE Authors' original audit study data analysis. NOTES Statistically significant coefficients at α < 0.05 indicated in boldface.

Table 5 presents a summary of the estimated coefficients on the good practices index for additional model specifications, including models separately predicting state COVID-19 vaccination rates for subpopulations 18 years and older and 65 years and older, and models assessing the measure of good practices for phone communications only. The results show again that the implementation of good practices for promoting COVID-19 vaccinations were more important or influential for encouraging individuals to get the first dose, with slightly larger associations for the subpopulation of people 65 years and older. The associations were also stronger (larger) when the index of good practices was estimated specifically for communications by phone, with the percent of the population that received the first dose increasing by 0.22 percentage points for each additional point on the good practices index. This suggests that states could potentially increase COVID-19 vaccination rates among their populations by improving training and boosting the use of strategies and communication tactics for promoting vaccinations among state agency and MVS staff.

Table 5.

Associations of good practices with state COVID-19 vaccination rates.

	N	Completed 1st vaccine dose		Completed vaccine series
Subpopulations	455	Coefficient	Std. error	Coefficient	Std. error
Population 18 and older	455	0.155	(0.043)	0.101	(0.044)
Population 65 and older	455	0.172	(0.043)	0.087	(0.056)
Good practices via phone - all	260	0.182	(0.059)	0.070	(0.079)
Good practices via phone - pop. 18 and older	260	0.220	(0.065)	0.152	(0.069)
Good practices via phone - pop. 65 and older	260	0.215	(0.071)	0.150	(0.093)

Source/Notes: SOURCE Authors' original audit study data analysis. NOTES Statistically significant coefficients at α < 0.05 indicated in boldface.

7. Additional qualitative findings

As shown on the REDCap form in Appendix B, the audit testers recorded comments throughout their interactions with state agency and MVS staff, which we compiled and analyzed. We present additional insights from this qualitative analysis, focusing on those that are informative for improving state policies for increasing COVID-19 vaccinations. For example, as discussed above, some state agency staff, including those from Texas, New York, Georgia, Arizona, and Mississippi, would not register the testers for an appointment without a home address, while others indicated that proof of residency was required to obtain a vaccination in the state (e.g., Arkansas, Florida, Maryland, and New Jersey). These stated requirements were not always consistent with state policy documentation posted on websites or communicated through email. In conversation with a Mississippi MVS, a tester was asked to provide her social security number (SSN), but when the tester indicated she did not have one, she was told she could walk-in and would not need to bring anything. However, the same representative said that she would be asked to provide her name, date of birth, and SSN and to verify her address at the vaccination place.

The audit testers most frequently received inconsistent or imprecise information about state policies on ID requirements. Some were uncertain about how to answer the question, “Do I need to bring an ID with me?“; a common response was to recommend bringing a driver's license, state ID or some other photo ID. A respondent from Illinois indicated: “I believe probably any photo ID is probably fine, just to prove who you are.” Others (e.g., in Maine, Nebraska, Oregon, Virginia, West Virginia, and Wisconsin) made it clear that no ID was required and dispelled concerns about ID requirements. For example, an Oregon state health department staff member specifically stated that identification should not be required and encouraged the tester to report back if any providers said it was required. However, when the audit testers indicated that they did not have a driver's license, and if asked, offered that they had a consular ID, many state agency and MVS staff did not know what a consular ID was and would either direct them to the specific vaccination site (e.g., the nearest Walgreens) or convey uncertainty. Delaware and California were among the few states that explicitly conveyed that a consular ID was an acceptable form of identification. A few other states, including California, made it clear in publicly available documentation that undocumented individuals did not have to provide identification to get vaccinated. A Rhode Island State Health Department email response conveyed that “… we are not checking IDs at vaccination sites for a number of reasons, including concerns we have heard from numerous partners about the need to ensure equitable access to vaccination opportunities for all, including people who are undocumented or do not have identifying documents.”

The state health department and MVS staff rarely acted on cues the testers provided by offering educational information on the types of vaccines and their side effects or reassuring the testers that side effects were known to be minimal. As indicated earlier, transportation options were rarely discussed (in only 6% of interactions), even with the cue that the inquirer did not have a driver's license. As noted above, the Spanish-speaking tester frequently experienced additional challenges in accessing information about how to get vaccinated for COVID-19, with some respondents ending the call when the tester began speaking in Spanish or stating “No Spanish” or “I don't speak Spanish” before terminating the call. The quality of translation, when provided, was widely varying, and key information was sometimes not translated by the third-party translator service. There was a lack of language-based resources via every form of communication (phone, email, chat, and response form). In some chat boxes (e.g., Georgia, Indiana, and Maryland), Spanish language was not recognized, which generated error messages.

8. Conclusions

The audit study we conducted to document the experiences of individuals inquiring about COVID-19 vaccinations in the U.S. drew on existing research (Wood and Schulman, 2021) to evaluate these encounters and the communication tactics used by health departments and MVS for promoting vaccinations. Our research confirmed assertions (Shen and Orenstein, 2020) that front-line staff in state health departments and MVS vary considerably (across states) in how clearly they communicate vaccination requirements and procedures, and in what they ask of individuals attempting to find out how to access the COVID-19 vaccine. We identified some exemplars, such as California, which via both verbal and electronic forms of communication, was more likely to provide educational materials about the vaccine and thorough information about where to go and what to expect at vaccination sites, as well as to dispel concerns about ID and other requirements for vaccine access. While the use of communications practices was fairly consistent between state health departments and MVS, respondents from state health departments were more likely to use (implicitly or explicitly) negative and sometimes racialized language. In addition, although translation was more likely to be facilitated by phone, respondents were also more likely to indicate that ID and state residency were required to get the COVID-19 vaccination.

Analyses of the good practices index constructed from the audit data—documenting communication tactics employed for encouraging vaccination take-up and the provision of information on how and where to receive the COVID-19 vaccine and on ID, state residency and health insurance requirements—revealed that very few of the strategies and communications tactics recommended for promoting vaccinations were regularly implemented in practice. In addition, we found that the audit tester speaking Spanish was less likely to receive information on how and where to get the vaccine, and she was more likely to encounter negative language in interactions with state health department and MVS staff. More generally, the lack of implementation of these good practices may represent a missed opportunity for states to increase their COVID-19 vaccination rates, given that we found statistically significant associations between the good practices index scores and state vaccination rates. Although we do not claim to have identified a causal relationship (in the absence of an RCT), we believe that these findings, combined with qualitative analysis, point to important ways in which states may potentially improve their vaccination promotion efforts.

For example, electronic forms of communication were highly varied by state and often very limited in the information they provided in response to basic questions about vaccination access. The federal government might consider disseminating a model for these electronic communications, such as California's chat box option, which could quickly expand the accessibility of rich and accurate information regarding the COVID-19 vaccine and how to get it. Auto-generated emails and chat boxes could also be configured to provide high-quality translation, which would eliminate disparities for non-native English speakers. Another straightforward recommendation that follows from this study is to provide better training to state agency and MVS staff who interact with the public to support vaccination take-up. This training could ensure that staff have correct and complete information on any requirements for accessing the vaccine in the state, including ID requirements, as well as transportation, education and other supports that might encourage follow-through to obtain the vaccination. In addition, state agencies might go a long way toward reducing burdens for non-native English speakers by establishing phrases in relevant languages to be played or conveyed when an inquiry is made in another language, such as “please hold while I get a translator” (as we observed in DC, Kansas and South Carolina). Lastly, federal agencies such as the Department of Health and Human Services might play a stronger role in disseminating good practices to support vaccination access and take-up and in coordinating outreach across state agencies, focusing in particular on states with lower vaccination rates and weaker existing practices. This would be consistent with the current administration's “National Strategy for the COVID-19 Response and Pandemic Preparedness” and recent calls for a nationwide “comprehensive, digital, real-time, integrated data infrastructure for public health” that would help to restore public in government and our public health institutions (Emanuel et al., 2022).

Credit author statement

Carolyn J. Heinrich: Conceptualization, Methodology, Investigation, Data Collection, Formal analysis, Project administration, Writing – original draft, Writing – review & editing. Sayil Camacho: Investigation, Data Collection, Formal analysis, Project administration, Writing – original draft, Writing – review & editing. Kaitlin Binsted; Investigation, Data Collection, Formal analysis, Writing – review & editing. Shadlan Gale: Data Collection, Formal analysis.

Appendix A. Audit Study Script

Three different communication strategies for emails and phone calls

• 1.Tester with white profile communicates in English.
• 2.Tester with Latinx profile communicates in English but with a Latin accent.
• 3.Tester with Latinx profile communicates in Spanish.

Audit tester names (typical female, common names for the ethnic profile) and language of communication

Anne Evans.

Language – English.

Ana Maria Alvarado.

Language - English with Spanish accent.

Maria Guadalupe Hernandez.

Language - Spanish.

Other profile information

Age: 35years old.

Address: No permanent address; if asked, staying with a friend.

Health insurance: No health insurance.

Employment status: Not employed.

Transportation: No car or other personal transportation.

Phone call script (English in black, Spanish in gray text):

Hi, my name is ______, and I want to find out how I can get the COVID-19 vaccine. Can you tell me what I need to do to sign up and where I go?

Hola, mi nombre es [Maria Guadalupe] y quiero saber como puedo recibir la vacuna de Covid-19. ¿Me puedes decir que tengo que hacer para apuntarme y donde debo ir?

Do I need to bring an ID with me?

¿Necesito que traer mi identificación conmigo?

[If a driver's license is suggested]:

I don't have a driver's license. Would I need to be licensed in this state?

No tengo licencia para manejar. ¿Tendrá que estar licenciada en este estado?

[If asked to bring a utility bill or other such document with an address]:

I don't have a utility bill [or rental agreement, etc.] in my name.

No tengo ninguna factura en mi nombre.

[If asked what kind of ID you have]:

I have a consular ID.

Tengo mi identificación del consulado.

Do I need to have a health insurance card?

¿Debo tener una carta de aseguranza?

What do I do when I get to the vaccination place?

¿Qué hago cuando llegue al lugar donde nos dan la vacuna?

If I am a bit worried about how it will affect me, can someone come with me?

Estoy preocupada de cómo ve va a afectar. ¿Alguien pude venir conmigo?

Email script (English in black, Spanish in gray text):

Hi, I am not sure what I need to do to get the COVID-19 vaccine. Can you tell me how I can sign up and where I should go? Also, do I need to bring an ID with me?

Thanks.

Anne (or Ana Maria or Maria Guadalupe).

Hola, No se lo que tengo que traer para recibir la vacuna de covid-19. ¿Me puedes decir donde debo ir y como hacer una cita? También quiero saber si tengo que traer mi identificación conmigo.

Gracias.

Maria Guadalupe.

Possible follow-ups:

Depending on what is suggested for an ID:

[If a driver's license is suggested]:

What if I don't have a driver's license? Do I have to have a license in this state?

¿Qué pasa si no tengo licencia para manejar? ¿Tendrá que estar licenciada en este estado?

[If asked to bring a utility bill or other such document with an address]:

I don't have a utility bill [or rental agreement, etc.] in my name.

No tengo ninguna factura en mi nombre.

[If asked what kind of ID you have]:

I have a consular ID.

Tengo mi identificación del consulado.

Other follow-up questions to include in an email:

Do I need to have a health insurance card?

¿Debo tener una carta de aseguranza?

What do I do when I get to the vaccination place?

¿Qué hago cuando llegue al lugar donde nos dan la vacuna?

If I am a bit worried about how it will affect me, can someone come with me?

Estoy preocupada de cómo ve va a afectar. ¿Alguien pude venir conmigo?

Appendix A. Supplementary data

The following is the Supplementary data to this article: