Abstract
Background
The COVID-19 pandemic has exacerbated historical inequities for people with disabilities including barriers in accessing online information and healthcare appointment websites. These barriers were brought to the foreground during the vaccine rollout and registration process.
Objective
This cross-sectional study aimed to examine accessibility of U.S. state and territory COVID-19 information and registration centralized websites.
Methods
The Johns Hopkins Disability Health Research Center created a COVID-19 Vaccine Dashboard compiling COVID-19 information and vaccine registration web pages from 56 states and territories in the United States (U.S.) reviewed between March 30 through April 5, 2021 and analyzed accessibility using WAVE Web Accessibility Evaluation Tool (WAVE). WAVE identifies website accessibility barriers, including insufficient contrast, alternative text, unlabeled buttons, total number of errors, and error density. Web pages were ranked and grouped into three groups by number of errors, creating comparisons between states on accessibility barriers for people with disabilities.
Results
All 56 U.S states and territories had COVID-19 information web pages and 29 states had centralized state vaccine registration web pages. Total errors, error density, and alert data were utilized to generate accessibility scores for each web page, the median score was 259 (range = 14 to 536 and IQR = 237) for information pages, and 146 (range = 10 to 281 and IQR = 105) for registration pages.
Conclusions
These results highlight barriers people with disabilities may encounter when accessing information and registering for the COVID-19 vaccine, which underscore inequities in the pandemic response for the disability community and elevate the need to prioritize accessibility of public health information.
Keywords: COVID-19 vaccine, Health services for persons with disabilities, Barriers to accessing information and registering for COVID-19 vaccine, Website accessibility of public health information, Disability inclusion
The COVID-19 pandemic has disproportionately impacted the disability community, compounding already existing barriers that individuals with disabilities face in accessing healthcare, as well as public health information.1, 2, 3, 4, 5 These gaps in access create inequities in accessing COVID-19 testing, treatment, and vaccination.1 , 2 People with disabilities are more likely to have underlying conditions that impact the immune system and make them exponentially more susceptible to severe and negative COVID-19 outcomes than the general population.2 , 4 This may result in some individuals with disabilities requiring personal care assistance, and the disruption of caregiving and support services due to the need for social distancing may also increase the risk for health complications unrelated to COVID-19.2 In particular, individuals with disabilities have a multitude of unique obstacles within the health care system including, but not limited to, difficulties with transportation and barriers in accessing information that have been exacerbated by COVID-19 regulations such as the social distancing mandates.6 People with disabilities are also more likely than people without disabilities to live in nursing homes or congregate care settings, which have been associated with much higher risk of COVID-19 exposure.2, 3, 4 In addition, for certain disabilities (such as sensory disabilities), there can be differential societal implications of social measures taken to combat the spread of COVID-19. For example, someone who is Blind cannot tell if the people around them are wearing a mask or maintaining social distancing. For these combinations of reasons, people with disabilities are more at risk and differentially impacted by the pandemic, and therefore prioritized for COVID-19 vaccine allocation.
The disability community has historically faced barriers to accessing information.1, 2, 3, 4, 5 The impact of these gaps in access have been elevated during the pandemic.1 , 2 Inaccessibility of COVID-19 information poses significant barriers, as individuals with disabilities may face longer delays in obtaining information about COVID-19 and receiving COVID-19 vaccine, creating inequities in health and daily life. Gaps in accessibility are compounded by the COVID-19 vaccine registration websites, potentially resulting in greater challenges to register for vaccination for people with disabilities.7, 8, 9, 10 For example, one study analyzing data collected from U.S. state health department websites reported multiple website accessibility barriers including lack of mobile access to COVID-19 vaccine websites, absence of non-English language options, poor readability, and failure to disclose critical information regarding web-based scheduling or availability.10 While this article is focused on U.S. COVID-19 website, gaps in website accessibility are widespread. One Australian study assessed a variety of WCAG violations, such as low background contrast or lack of alternative or descriptive text, on local ACT COVID-19 information websites while a Canadian article evaluated the potential impact between WCAG score errors in Ontario public health websites and the ease in which those with disabilities had access to COVID-19 vaccine appointments.11 , 12
This project focused on website accessibility, as under the Americans with Disabilities Act (ADA) and the Rehabilitation Act of 1973, states have a responsibility to provide accessible information to people with disabilities. The ADA mandates against discrimination on the basis of disability. Although Title III of the ADA focuses on “public accommodations and commercial facilities”,13 internet websites are not explicitly mentioned. The Department of Justice has taken the position to consider websites public domain, and as a result, website accessibility is protected under Title III of the ADA. Therefore, under the ADA, state COVID-19 information and registration websites need to be accessible to individuals with disabilities.13 In addition to the ADA, Section 504 of the Rehabilitation Act of 1973 requires all recipients of federal government aid to ensure their activities, programs, and services are accessible to people with disabilities, including websites.14 In 2018, Section 508 of the Rehabilitation Act adopted the internationally accepted Website Content Accessibility Guidelines (WCAG) which legally requires federal agencies (or parties doing business with a federal agency) to ensure all pages of their websites are accessible to individuals with disabilities.14
This project aimed to examine and quantify the accessibility of U.S. COVID-19 state vaccine information and registration websites and disseminate this information via the Johns Hopkins Disability Health Research Center COVID-19 Vaccine Dashboard. Although website errors and accessibility have been reliably measured by the Web Accessibility Evaluation Tool (WAVE), which has been used to assess website accessibility in many settings15, 16, 17 including a study on two local Australian government COVID-19 websites, this project aimed to track the accessibility of COVID-19 vaccine information across the U.S.11
Methods
Data collection
This cross-sectional study utilized data on website accessibility metrics determined from state/territory public health department COVID-19 vaccine web pages across all 50 U.S. states, the District of Columbia, and 5 territories collected by searching the internet for the most up-to-date web page Uniform Resource Locator (URL). As the data was publicly available and did not involve human subjects, IRB approval was not required. Web pages were monitored weekly to ensure that each URL was still active and to update the states and territories that created new web pages. The entire website was not scored. Two types of web pages were analyzed: COVID-19 information and COVID-19 vaccine registration. COVID-19 information web pages were determined based on containing state/territory-wide guidelines on business regulations, social distancing policies, vaccine eligibility, etc. Our analysis focused on the main COVID-19 state information webpage or landing page. As a result, no additional webpages were studied (only one COVID-19 information webpage URL was analyzed for each U.S. state or territory). A registration web page was determined based on containing a centralized portal for residents to register for COVID-19 vaccine appointments. Our analysis was limited to single state/territory-level vaccine pre-registration or registration government web pages and did not include data from states or territories without a centralized state-level registration vaccine portal. For example, states or territories that simply listed the contact information of each locale's health department or displayed an interactive map of which providers and commercial pharmacies had vaccines available were not captured. In addition, states or territories that listed multiple providers, each offering independent pathways for registration, were not included. These web pages were excluded, despite being centralized (i.e., hosted by state- and territory-wide departments of health), because these web pages only served as intermediary information hubs between the resident and vaccine provider. During data collection, state COVID-19 vaccine websites were split evenly between two primary coders, G.J and D.H. Each person had 4 days to collect the data, after which checks were completed, graphs created, and the data was published in the Johns Hopkins COVID-19 Vaccine Dashboard each Wednesday. Data from reviews conducted between March 30 until April 5, 2021 are included in these analyses.
Assessment of web page accessibility
The accessibility scores and rankings for COVID-19 web pages were generated from automatic testing data provided by WAVE, a web accessibility evaluation tool administered by WebAIM at Utah State University. WAVE is a commonly utilized accessibility testing tool that has been in development since 2001. WAVE analyzes patterns in web site codes to identify accessibility barriers such as insufficient contrast, alternative text, unlabeled buttons, total number of errors, and error density. A free, online version of WAVE is publicly available for download at https://wave.webaim.org/. Browser extensions for Chrome and Firefox are also available. The WAVE API was utilized for automated testing of the COVID-19 web pages.18
The number of detected accessibility errors, the density of those errors on the page (errors divided by number of page elements), and the number of likely/potential accessibility issues “(“Alerts” within WAVE) were considered when generating the rankings. Automatically detectable accessibility issues do not represent all accessibility issues, but typically indicate barriers for users with disabilities and non-compliance with accessibility guidelines and best practices. Accessibility scores were generated for each web page by rank ordering the 56 COVID-19 information pages (and 29 vaccine registration pages) on number of errors, error density, and likely/potential issues. Weightings were then applied to each of the rank orders (ties were treated equally) to calculate an overall access score—with error ranks being assigned a score of 6, error density ranks being assigned a score of 3, and likely/potential issues assigned a score of 1. This access score is not necessarily a good measure of overall accessibility but provides a normalized metric for comparisons across the web pages. The dashboard presented pages ordered or ranked by the overall access score.
For information web pages, WAVE was run automatically, while for registration web pages, accessibility scores were calculated by navigating to the first web page of the registration site where user-specific information (e.g., name, age, etc.) was entered. Rather than capturing the entire website, the first relevant information or registration page was considered reflective of all web pages of each site. WAVE was run automatically if the URL was linked to the correct web page of the vaccine portal. However, WAVE was run manually (using a browser extension) on the next page of the registration portal if account creation or CAPTCHA (completely automated public turing test to tell computers and humans apart) was required on the first page before entering user-specific data for 9 of the 20 COVID-19 vaccine registration web pages (leaving 20 of the 29 being scores using automated WAVE methods). In manual assessments, testers captured total errors, error density, and alert data—the same data collected via the automated process for pages that could be directly accessed by WAVE. However, automated WAVE data captured more in-depth analyses of the types of errors, such as contrast errors, empty links/buttons, images without alternative text, and unlabeled form inputs. If a state or territory required the creation of a new account to register for vaccination, the account creation page was used to assess accessibility: no new accounts were created by the researchers during this process.
Dashboard
These data were used to create the Johns Hopkins Disability Health Research Center COVID-19 Vaccine Dashboard.19 The dashboard was made publicly available, focused on ensuring accessibility of information, and was updated weekly to both enhance accessibility to vaccine registration URLs and highlight important considerations about mitigating disparities in COVID-19 information access for people with disabilities.19 The initial dashboard release focused on COVID-19 vaccine prioritization for people with disabilities, but was expanded to also examine state-level COVID-19 vaccine information and registration website accessibility. These analyses present a cross-sectional snapshot of the accessibility data for the week of March 30 to April 5, 2021. This week was selected as it is the “baseline” when all state website accessibility data began to be tracked.
Statistical analysis
Accessibility rankings were assigned based on an inverse relationship with accessibility score, as higher accessibility rankings corresponded to lower frequency of errors (i.e., lower accessibility score). Contrast errors, error density, total errors, accessibility scores, rankings, and URLs of COVID-19 vaccine information and registration web pages were tabulated. Additional properties of registration web pages—account requirement, pre-registration status, and CAPTCHA implementation—were also tabulated. For both information and registration web pages, the mean, standard deviation, median, range, and interquartile range (IQR) were calculated and tabulated for accessibility scores, total errors, error density, insufficient contrast errors, empty links/buttons, images without alternative text, and unlabeled form inputs. Only those registration web pages that did not preclude automatic WAVE testing due to CAPTCHA or account creation before accessing the main page were included in the table for combined statistics. Accessibility scores for information and registration web pages in the states and territories that hosted both types were plotted as bar graphs. Information and registration web pages were divided into three groups based on their accessibility scores to compare and contrast centralized web pages in various U.S. states and territories. The top third contained the fewest accessibility errors while the bottom third contained the most accessibility errors. Maps color-coded by group were created for easy visualization and comparison of information and registration web page accessibility. The correlation between state information and registration pages was calculated using Pearson's correlation coefficient (and a paired t-test to obtain the p-value) for the subset of 29 states/territories that had centralized registration portals. Statistical analyses were complete using R statistical software (version 3.6.3; The R Foundation).
Results
Accessibility scores were calculated for 56 U.S. states and territories with COVID-19 information web pages (Table 1 ), and the 29 states that had a centralized (i.e., state/territory-wide) COVID-19 registration web page as of April 5, 2021 (Table 2 ). Total accessibility scores for each state and territory information and registration web pages were examined (Fig. 1 a, b, c). The within-state correlation between information and registration page accessibility scores was 0.304 (p < 0.001). States with information and registration web pages that were both in the top third of states with the fewest accessibility errors were California, New Jersey, Maine, and Ohio. States (and one territory) with both types of web pages in the bottom third with the most accessibility errors were New York, Nebraska, Northern Mariana Islands, Florida, Idaho, New Mexico, and Illinois. The most common issues were: (1) text with insufficient contrast to the background (which is more difficult to read, particularly for users with certain visual disabilities), (2) empty links and buttons that would be announced by screen readers but lack descriptions, (3) images without descriptive alternative text for screen reader users, and (4) form inputs not properly labeled with descriptive text (Table 3 ). Other accessibility issue types, such as if a page code did not define the natural language of the document or if headings or ARIA code were used incorrectly, were also considered in the accessibility scores.
Table 1.
COVID-19 information web page accessibility ranking by United States state/territory (data as of April 5, 2021).
Data as of April 5, 2021.
Table 2.
COVID-19 vaccine registration web page accessibility ranking by United States state/territory.
| Rank | State/territory | Score | Total errors | Error density | Requires account creation/log-in | Pre-registration | Has CAPTCHAa | CAPTCHAa audio option | URL |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Nevada (NV) | 10 | 0 | 0.00% | No | No | No | No | https://vax4nv.nv.gov/patient/s/ |
| 2 | California (CA) | 14 | 0 | 0.00% | No | No | No | No | https://myturn.ca.gov/screening |
| 2 | Maine (ME) | 14 | 0 | 0.00% | No | No | No | No | https://vaccinateme.maine.gov/screening |
| 2 | Massachusetts (MA) | 14 | 0 | 0.00% | No | Yes | No | No | https://vaccinesignup.mass.gov/#/ |
| 5 | Indiana (IN) | 17 | 0 | 0.00% | No | No | No | No | https://vaccine.coronavirus.in.gov/ |
| 6 | Virginia (VA) | 20 | 0 | 0.00% | No | Yes | No | No | https://vax.preregister.virginia.gov/#/ |
| 7 | Alabama (AL) | 22 | 0 | 0.00% | No | No | No | No | https://govstatus.egov.com/vaccine-eligibility-form |
| 8 | District of Columbia (DC) | 99 | 1 | 0.23% | No | Yes | No | No | https://v51r5a21s.dc.gov/cvq_dc/ |
| 8 | New Jersey (NJ) | 99 | 1 | 0.15% | No | Yes | Yes | Yes | https://covidvaccine.nj.gov/ |
| 10 | Ohio (OH) | 110 | 1 | 1.12% | No | No | No | No | https://gettheshot.coronavirus.ohio.gov/screening |
| 11 | Maryland (MD) | 111 | 1 | 0.83% | Yes | Yes | No | No | https://onestop.md.gov/users/sign_up?registration_context=%2Fpreregistration |
| 12 | Tennessee (TN) | 130 | 2 | 0.36% | No | No | Yes | Yes | https://vaccinate.tn.gov |
| 13 | Arizona (AZ) | 131 | 2 | 1.38% | Yes | No | No | No | https://podvaccine.azdhs.gov/signup |
| 14 | Oklahoma (OK) | 139 | 2 | 1.00% | No | No | Yes | Yes | https://vaccinate.oklahoma.gov/en-US/ |
| 15 | Delaware (DE) | 146 | 2 | 1.56% | No | No | Yes | No | https://vaccinerequest.delaware.gov/s/de-vms-screening?language=en_US |
| 15 | Vermont (VT) | 146 | 2 | 2.44% | Yes | No | No | No | https://vermont.force.com/events/s/selfregistration |
| 17 | Missouri (MO) | 154 | 2 | 1.79% | No | No | No | No | https://modhss.iad1.qualtrics.com/jfe/form/SV_231d5TxZxkGedCt |
| 18 | Minnesota (MN) | 166 | 3 | 0.13% | No | No | No | No | https://vaccineconnector.mn.gov/covid-19%20vaccine/ |
| 19 | West Virginia (WV) | 175 | 4 | 0.35% | Yes | No | No | No | https://member.everbridge.net/747122446041089/new |
| 20 | Wisconsin (WI) | 182 | 3 | 0.73% | No | No | Yes | Yes | https://vaccinate.wi.gov/en-US/ |
| 21 | New York (NY) | 200 | 4 | 1.44% | No | No | Yes | No | https://am-i-eligible.covid19vaccine.health.ny.gov/Public/prescreener |
| 22 | Idaho (ID) | 204 | 5 | 0.62% | No | Yes | No | No | https://covidvaccine.idaho.gov/register |
| 23 | Illinois (IL) | 221 | 7 | 2.95% | No | No | No | No | https://covidvaccination.dph.illinois.gov/questionnaire |
| 24 | Nebraska (NE) | 223 | 3 | 3.49% | No | No | Yes | Yes | https://vaccinate.ne.gov/en-US/ |
| 25 | Georgia (GA) | 229 | 8 | 1.43% | No | Yes | No | No | https://myvaccinegeorgia.com/en/site/ALL |
| 26 | Northern Mariana Islands (MP) | 233 | 17 | 0.70% | No | Yes | No | No | https://www.vaccinatecnmi.com/covid-19/registration/ |
| 27 | New Hampshire (NH) | 250 | 12 | 3.20% | No | No | No | No | https://sonh-community.force.com/providers/s/ |
| 28 | Florida (FL) | 274 | 32 | 8.99% | No | Yes | No | No | https://myvaccine.fl.gov/#/RegistrationForm |
| 29 | New Mexico (NM) | 281 | 57 | 3.44% | No | No | Yes | No | https://cvvaccine.nmhealth.org |
Abbreviations: CAPTCHA=Completely automated public turing test to tell computers and humans apart
Data as of April 5, 2021.
Fig. 1.
United States COVID-19 information and vaccine registration web page accessibility (data as of April 5, 2021). (a) COVID-19 information and (b) COVID-19 vaccine registration pages ranked in three groups by number of accessibility errors. (c) COVID-19 information and COVID-19 vaccine registration web page accessibility scores for the 29 states and territories with both types.
Table 3.
Statistics for common COVID-19 information web page accessibility errors across the United States.
| Issue type | Number of issues detected (% of all errors) | Mean issue per web page (SD) |
|---|---|---|
| Insufficient text contrast | 549 (78.2%) | 9.80 (18.06) |
| Empty links and buttons | 55 (7.8%) | 0.98 (1.69) |
| Missing alternative text | 46 (6.6%) | 0.82 (2.91) |
| Missing input labels | 18 (2.6%) | 0.34 (0.72) |
COVID-19 information web page accessibility scores for all 56 states and territories ranged from 14 (Minnesota) to 536 (Mississippi) with a median score of 259 (Table 4 ). Errors ranged from zero (Minnesota, Kansas, Louisiana, California, Arkansas, Washington, and Maryland) to 110 (Illinois) with a median value of 5.50. Mississippi (the state with the lowest accessibility score) had 81 total errors. Error density ranged from 0% to 9.67% with a median value of 0.90%. Insufficient contrast errors ranged from 0 to 104 with a median of 3.50. Errors with empty links and buttons had a range of 0–8 with a median value of 0. Images without alternative text ranged from 0 to 21 with a median of 0, and errors with form inputs not labeled had a median value of 0 with a range of 0–3.
Table 4.
Statistics for COVID-19 information web page accessibility data across the United States.
| Accessibility | Mean (SD) | Median (range) | Interquartile range |
|---|---|---|---|
| Accessibility scores | 274.20 (151.37) | 259 (14, 536) | 172, 409.25 |
| Total errors | 12.54 (19.72) | 5.50 (0, 110) | 3, 15 |
| Error density, % | 1.41 (1.68) | 0.90 (0, 9.67) | 0.38, 2.28 |
| Insufficient contrast errors | 9.80 (18.06) | 3.50 (0, 104) | 1, 12.25 |
| Empty links/buttons | 0.98 (1.69) | 0 (0, 8) | 0, 1 |
| Images without alternative text | 0.82 (2.91) | 0 (0, 21) | 0, 1 |
| Form inputs not labeled | 0.34 (0.72) | 0 (0, 3) | 0, 0 |
Abbreviations: SD = standard deviation
Data from 50 US States, the District of Columbia, and 5 U.S. territories, as of April 5, 2021.
Of the 29 states and territories that hosted both types of websites, accessibility scores from the 29 centralized COVID-19 information web pages ranged from 14 to 530 with a median score of 324. Errors ranged from zero to 110 with a median of 7 and a mean of 14.66 (SD 22.66). Error density had a median of 0.01 (range of 0–0.10).
The accessibility scores for state COVID-19 vaccine registration web pages ranged from 10 (Nevada) to 281 (New Mexico), with a median score of 146 (Table 5 ). Error density ranged from 0 (Nevada, California, Maine, Massachusetts, Indiana, Indiana, Virginia, and Alabama) to 0.09 (Florida) with a median percentage of 0.01. Seven of the 29 states had zero total errors (Nevada, California, Maine, Massachusetts, Indiana, Virginia, and Alabama) and New Mexico (the state with the lowest accessibility ranking) had 57 total errors. Errors for the 29 COVID-19 vaccine registration web pages had a median of 2 (Table 5). Four states (Maryland, Arizona, Vermont, and West Virginia) required account creation for access and eleven states (Massachusetts, Virginia, District of Columbia, New Jersey, Maryland, Idaho, Georgia, Northern Mariana Islands, and Florida) were open for pre-registration of the vaccine (Table 2). There were eight state COVID-19 registration web pages with CAPTCHA (New Jersey, Tennessee, Oklahoma, Delaware, Wisconsin, New York, Nebraska, and New Mexico), of which five (New Jersey, Tennessee, Oklahoma, Wisconsin, and Nebraska) had an audio option for completion, which enhances accessibility for individuals with visual impairments (Table 2).
Table 5.
Statistics for registration web page accessibility scores, total errors, error density, contrast errors, empty links/buttons, images without alternative text, and unlabeled form inputs.
| All 29 registration web pages | Mean (SD) | Median (range) | Interquartile range |
|---|---|---|---|
| Accessibility scores | 138.41 (85.52) | 146 (10, 281) | 99, 204 |
| Total errors | 5.90 (11.84) | 2 (0, 57) | 1, 4) |
| Error density, % |
1.77 (2.06) |
1.10 (0, 9.67) |
0.36, 2.49 |
| 20 registration web pages |
Mean (SD) |
Median (range) |
IQR |
| Insufficient contrast errors | 0.95 (1.82) | 0 (0, 6) | 0, 1 |
| Empty links/buttons | 0.35 (0.81) | 0 (0, 3) | 0, 0 |
| Images without alternative text | 0.45 (0.83) | 0 (0, 3) | 0, 1 |
| Form inputs not labeled | 2.40 (6.44) | 0.50 (0, 29) | 0, 1.25 |
Abbreviations: SD = standard deviation
Data from 20 of the 29 registration web pages that have details on types of errors (as the remainder were extracted manually using the WAVE browser plug-in), as of April 5, 2021.
Of the 29 registration web pages, 20 were assessed using automatic WAVE testing while the other 9 (California, Maine, New Jersey, Tennessee, Oklahoma, Wisconsin, Idaho, Nebraska, and New Mexico) exhibited barriers that made manual extraction necessary (Table 5). For these 20 states, insufficient contrast errors ranged from 0 to 6 with a median of 0. The median value of errors regarding empty links and buttons was 0 with a range of 0–3. Web pages containing images without alternative text ranged from 0 to 3 with a median of 0, and errors with form inputs not labeled ranged from 0 to 29 with a median value of 0.50.
Discussion
By creating a dashboard comparing COVID-19 information and registration website accessibility, we found inequities in access to this critical information in the U.S. These results highlight important gaps in the opportunity to access COVID-19 information and vaccines for people with disabilities. The mean number of errors detected across these web pages (12.50 per page) was lower than the mean accessibility errors (51.40 per page) detected on a sample of 1 million web pages from across the internet.20 This lower mean number of accessibility errors may due to the fact this dashboard only examined state government websites, which generally have fewer error when compared to private webpages.20 Even though the accessibility of COVID-19 web pages scored better than most web pages, the importance of the information on these pages demands that any accessibility barriers be addressed, as disparities in COVID-19 outcomes and vaccination rates among the disability community may be compounded by these gaps in information access. Website errors and accessibility have been reliably measured by the Web Accessibility Evaluation Tool (WAVE) in many settings,15, 16, 17 including a study on two local Australian government COVID-19 websites, but not to the scale of encompassing all territory-wide websites.11
The majority of state rankings did not have consistent accessibility rankings across COVID-19 vaccine information and registration websites. Only a few states had within-state consistency in these rankings, such as California where the COVID-19 vaccine information website was ranked 4th, and the COVID-19 vaccine registration ranked 2nd. The reasons for the within-state inconsistency in accessibility ranks are unknown. It is possible there are within-state variations in the allocation of resources and personnel focusing on website accessibility that may contribute to these differences. It is also possible that COVID-19 vaccine information websites may have been created earlier than registration websites, allowing more time for states to improve information website accessibility based on the public's feedback. States without centralized vaccine registration websites also had discrepancies between information and registration rankings, as no ranking was possible for registration websites in these cases.
It is important to consider how the disability community is impacted by barriers in accessing COVID-19 vaccine information. For instance, CAPTCHA usually requires visual input from the user, creating difficulties for individuals who are blind or are visually impaired. CAPTCHA may therefore create additional delays in or prevent vaccine registration. Delays in vaccine registration that may have resulted from inaccessible websites may result in greater risk for people with disabilities to contract COVID-19, a disease which has been proven to have disproportionally negative outcomes for this population group.2 Acknowledging barriers for website accessibility, the federal government has recently devoted substantial funding towards reducing barriers in accessing COVID-19 vaccines for the people with disabilities by providing access to direct phone numbers for vaccine registration and providing transportation to vaccine appointments.21 Information on these efforts, however, is primarily available via the state COVID-19 websites. While this critical funding will likely go far towards closing gaps in accessing vaccines, there remains limited resources and efforts towards ensuring COVID-19 information and registration is actually accessible on state public health web pages. It is important for web designers to prioritize accessibility and include disability advocates and people with disabilities at the outset of website design. Only by partnering with the disability community can web designers create public health websites which are accessible and informative for everyone. While website accessibility is an important contributing factor in barriers to COVID-19 vaccination uptake, this is only one component of the COVID-19 inequities impacting the disability community. Further data is needed to identify gaps in accessibility of all aspects of COVID-19 vaccine distribution, including transportation to the COVID-19 vaccine appointment and accessibility at the vaccine clinic site.
Despite its potential impact, the limitations of this project must be considered. While this project and resulting Dashboard focus on website accessibility, there remains important work in collecting data documenting and identifying gaps in accessibility of the physical vaccine locations and quantifying vaccine rates among people with disabilities to determine inequities. Additionally, the accessibility score/ranking system is primarily designed around people with visual disabilities, and there are barriers to online accessibility for people with other types of disabilities that were not captured. These may include, for example, video captioning for people with hearing loss, or plain language summaries for people with intellectual, executive and cognitive disabilities. Further, some websites collect more information per page (i.e., long format) while other websites collect less information per page but include multiple pages (i.e., wide format). A website with a wide format may have fewer total errors per page but may have a higher percentage of errors with respect to how much information is on a single page. Information web pages were more often in long format, likely accounting for their notably higher accessibility scores compared to those for registration web pages (Table 3, Table 4). To mitigate this, the accessibility scores included error density—the number of errors by web page elements (i.e., how dense errors are within the page content and functionality). Our method measured accessibility scores for the first page of the registration website, which is indicative of patterns in subsequent pages; however, users may abandon the process if significant accessibility barriers are encountered on subsequent pages, and future research should focus on developing a metric for capturing accessibility data across many pages of the same website rather than just the first page. Some COVID-19 vaccine registration web pages required pre-registration, including entering driver's license credentialing, or creating an account. This limitation may have created bias in our sample as we did not gain access to websites requiring pre-registration to avoid entering false information and burdening the system. While WAVE was used in this project, it is only one type of method to evaluate website accessibility. This tool analyzes patterns in web site code and design that align with accessibility compliance failures with a very high level of reliability. However, WAVE, like all automated tools, cannot fully assess all aspects of end user accessibility, and the issues it does analyze are primarily focused on users with visual disabilities. Analyzing plain language or assessing video captioning presence/quality or testing for keyboard accessibility issues would necessitate manual testing that was not conducted as part of this study. Additional aspects of accessibility, such as options for end users to register for a vaccine or get additional supports via phone, were not considered, primarily because web sites are a primary mechanism for accessing such information and functionality. Despite these limitations, the issues detected by WAVE nearly always align with negative impact on users with disabilities and thus provide a useful measure of accessibility. In addition, the COVID-19 vaccine roll-out is rapidly evolving, and some states or territories may have adjusted their information and vaccine registration portals since initial data collection.
Conclusion
This project highlights the barriers that people with disabilities may have encountered when accessing state information and registration COVID-19 vaccine websites in the U.S during the spring of 2021. Our results underscore the addressable, yet persistent inequities in the pandemic response for the disability community.22 The accessibility of public health information must be prioritized and supported in order to ensure that the disability community is no longer left behind.
Acknowledgments
We thank the following team members for their work on the COVID-19 Dashboard Project: Caroline Cerilli, Jessica Campanile, Leah Smith, Fiona Sweeney, Maureen van Stone, Tracy Waller.
Funding
American Association of People with Disabilities (AAPD).
Disclaimer
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the AAPD.
Conflicts of interest
All authors declare no competing interests.
Availability of data and material
The COVID-19 website accessibility scores and rankings used in this study are publicly available on the Johns Hopkins Disability Research Center Website.
Author contributions
DH and GJ contributed to data collection, manuscript writing, and interpretation of results. VV contributed to conception of the work, data collection, manuscript writing, and interpretation of results. JS contributed to data analysis, and interpretation of results. SE contributed to the conception of the work and data collection. JZ and GY contributed to data analysis and visualization. KA contributed to the conception of the work. BKS contributed to conception of the work, manuscript writing, and interpretation of results. All authors had full access to all data in the study, contributed edits of the manuscript, and had final responsibility for the decision to submit for publication.
References
- 1.Epstein S., Campanile J., Cerilli C., Gajwani P., Varadaraj V., Swenor B.K. New obstacles and widening gaps: a qualitative study of the effects of the COVID-19 pandemic on U.S. adults with disabilities. Disability and Health Journal. March 2021:101103. doi: 10.1016/j.dhjo.2021.101103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Lund E.M., Forber-Pratt A.J., Wilson C., Mona L.R. The COVID-19 pandemic, stress, and trauma in the disability community: a call to action. Rehabil Psychol. 2020;65(4):313–322. doi: 10.1037/rep0000368. [DOI] [PubMed] [Google Scholar]
- 3.Sabatello M., Burke T.B., McDonald K.E., Appelbaum P.S. Disability, ethics, and health care in the COVID-19 pandemic. Am J Publ Health. 2020;110(10):1523–1527. doi: 10.2105/ajph.2020.305837. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Pulrang A. The Disability Community Fights Deadly Discrimination amid the COVID-19 Pandemic. Forbes. https://www.forbes.com/sites/andrewpulrang/2020/04/14/the-disability-community-fights-deadly-discrimination-amid-the-covid-19-pandemic/?sh=48083850309c
- 5.Okoro C.A., Hollis N.T.D., Cyrus A.C., Griffin-Blake S. Prevalence of disabilities and health care access by disability status and type Among adults — United States, 2016. MMWR Morbidity and Mortality Weekly Report. 2018;67(32):882–887. doi: 10.15585/mmwr.mm6732a3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Shalaby W.S., Odayappan A., Venkatesh R., et al. The impact of COVID-19 on individuals across the spectrum of visual impairment. Am J Ophthalmol. 2021;227:53–65. doi: 10.1016/j.ajo.2021.03.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Epstein S., Ayers K., Swenor B.K. COVID-19 vaccine prioritisation for people with disabilities. Lancet Public Health. 2021;6(6) doi: 10.1016/s2468-2667(21)00093-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kayla Hui M.P.H. People with Disabilities Are Experiencing Barriers to Covid-19 Vaccination. Verywell Health. https://www.verywellhealth.com/people-with-disabilities-barriers-covid-19-vaccine-5115883
- 9.Keegan J., Lecher C. We ran tests on every state's COVID-19 vaccine website – the markup. https://themarkup.org/blacklight/2021/03/24/we-ran-tests-on-every-states-covid-19-vaccine-website
- 10.Howe J.L., Young C.R., Parau C.A., Trafton J.G., Ratwani R.M. Accessibility and usability of state health department COVID-19 vaccine websites. JAMA Netw Open. 2021;4(5) doi: 10.1001/jamanetworkopen.2021.14861. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Yu S.Y. A review of the accessibility of ACT COVID-19 information portals. Technol Soc. 2021;64:101467. doi: 10.1016/j.techsoc.2020.101467. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Rotenberg S., Downer M.B., Brown H.K., et al. COVID-19 vaccination for people with disabilities. Science Briefs of the Ontario COVID-19 Science Advisory Table. 2021;2(35) doi: 10.47326/ocsat.2021.02.35.1.0. [DOI] [Google Scholar]
- 13.U.S. Department of Justice. Civil Rights Division. Public Access Section . U.S. Dept. of Justice, Civil Rights Division, Public Access Section; Washington, D.C.: 1991. The Americans with Disabilities Act : Title III Regulations : Part 36 Nondiscrimination on the Basis of Disability in Public Accommodations and Commercial Facilities. [Google Scholar]
- 14.(WAI), W. C. W. A. I . Web Accessibility Initiative (WAI); November 13, 2021. Web Content Accessibility Guidelines (WCAG) Overview.https://www.w3.org/WAI/standards-guidelines/wcag/ (n.d.) [Google Scholar]
- 15.Ahmi A., Mohamad R. Evaluating accessibility of Malaysian public universities websites using AChecker and WAVE. J Inf Commun Technol. 2016;15(2) doi: 10.2139/ssrn.3550314. [DOI] [Google Scholar]
- 16.Adepoju S.A., Shehu I.S. 2014 3rd International Conference on User Science and Engineering (i-USEr); September 2014. Usability Evaluation of Academic Websites Using Automated Tools. [DOI] [Google Scholar]
- 17.Lujan-Mora S., Navarrete R., Penafiel M. 2014 First International Conference on eDemocracy & eGovernment (ICEDEG); April 2014. Egovernment and Web Accessibility in South America; pp. 77–82. [DOI] [Google Scholar]
- 18.WAVE API. WAVE Web Accessibility Evaluation Tool. https://wave.webaim.org/api/
- 19.Vaccine Prioritization Dashboard The Johns Hopkins Disability Health Research Center. https://disabilityhealth.jhu.edu/vaccine-2/
- 20.The WebAIM MillionAn Annual Accessibility Analysis of the Top 1,000,000 Home Pages. WebAIM; 2021. https://webaim.org/projects/million/#errors [Google Scholar]
- 21.U.S. Department of Health and Human Services HHS to Expand Access to COVID-19 Vaccines for Older Adults and People with Disabilities. HHS.gov. https://www.hhs.gov/about/news/2021/03/29/hhs-to-expand-access-to-covid-19-vaccines-for-older-adults-and-people-with-disabilities.html
- 22.Swenor B.K. Including disability in all health equity efforts: an urgent call to action. Lancet Public Health. 2021;6(6) doi: 10.1016/s2468-2667(21)00115-8. [DOI] [PMC free article] [PubMed] [Google Scholar]