Data Representativeness in Accessibility Datasets: A Meta-Analysis

Abstract

As data-driven systems are increasingly deployed at scale, ethical concerns have arisen around unfair and discriminatory outcomes for historically marginalized groups that are underrepresented in training data. In response, work around AI fairness and inclusion has called for datasets that are representative of various demographic groups. In this paper, we contribute an analysis of the representativeness of age, gender, and race & ethnicity in accessibility datasets–datasets sourced from people with disabilities and older adults—that can potentially play an important role in mitigating bias for inclusive AI-infused applications. We examine the current state of representation within datasets sourced by people with disabilities by reviewing publicly-available information of 190 datasets, we call these accessibility datasets. We find that accessibility datasets represent diverse ages, but have gender and race representation gaps. Additionally, we investigate how the sensitive and complex nature of demographic variables makes classification difficult and inconsistent (e.g., gender, race & ethnicity), with the source of labeling often unknown. By reflecting on the current challenges and opportunities for representation of disabled data contributors, we hope our effort expands the space of possibility for greater inclusion of marginalized communities in AI-infused systems.

1. INTRODUCTION

As AI-infused systems¹ become ubiquitous, ensuring that they work for a diversity of groups is vital [29, 56, 108]. Performance disparities in these systems could lead to unfair or discriminatory outcomes for historically and culturally marginalized groups, such as on the basis of gender, race, or disability [12, 18, 44, 149, 162, 172]. One fundamental source of disparities is the lack of representation in datasets used to train machine learning models and benchmark their performance [108, 162, 179]. A notable example comes from Treviranus [166], where during a simulation, she found that machine learning models for autonomous vehicles would run over someone who propels themselves backward in a wheelchair. Merely adding training examples of people using wheelchairs did not have the intended effect in this case; the algorithm failed with a higher confidence [166]. Treviranus suspected ‘backward propelling’ was still an outlier.

In this important discussion on AI fairness and inclusion, tensions around data representativeness involving disability [60, 79, 118] have also arisen. Data sourced from accessibility datasets can help AI-infused systems work better when deployed in real-world scenarios, both for assistive and general-purpose contexts [29, 75, 169]. However, privacy and ethical concerns are especially pronounced in this community, as disclosure of disability can pose risks associated with re-identification and further discrimination e.g., for one’s healthcare and employment [169, 179]. People who have distinct data patterns, like in the case of disability, are also more susceptible to data abuse and misuse [1, 60, 167]. In addition, even if AI-infused systems are trained with diverse data, this does not inherently challenge the power structures in which these systems are embedded, which may be the actual source of harm and marginalization for disabled people [7]. For example, a more equitable AI-infused system for diagnosing autism does not necessarily correspond to greater well-being of autistic people, because it may cement the power that medical institutions have to diagnose and gatekeep [7].

We contribute to these discussions via our exploration of representation in accessibility datasets, which reveal nuanced patterns of representation and marginalization along intersectional lines. In this work, we conducted a metadata analysis of existing accessibility datasets (1984–2021, N=190) spanning multiple communities of focus and data types to understand the representation and reporting of demographic attributes including age, gender, and race & ethnicity of data contributors. We used the publicly available documentation and resources of these datasets to explore the potential opportunities and limitations for increasing data representativeness.

Our analysis shows mixed results for diverse representation of age, gender, and race & ethnicity. For age, we found that older adults are particularly well-represented, but this did not apply across all communities of focus (with Autism, Developmental, and Learning communities being notable exceptions). Gender representation skewed towards men/boys being more represented overall but varied widely by community of focus. We also found that well-documented structural marginalization in certain communities are reflected in accessibility datasets. For example, women/girls are underrepresented in Autism datasets, corresponding to existing diagnosis gaps [55, 130]. Marginalization is further embedded on a meta level, such as the case of binary categories for gender classification in the collection and reporting of gender data within datasets. Furthermore, we did not find consistent norms for reporting data, with the lack of standardized documentation, evolving practices, and variability of categories used across age, gender and race & ethnicity.

The contributions of this work are 1) a systematic examination of whether those sourcing data from the disability community are succeeding in representing diverse demographics, via an intersectional analysis along the axes of age, gender, and race & ethnicity as well as a meta-analysis of reporting methods; 2) codes of 190 existing accessibility datasets annotated with demographic metadata²; and 3) connections to larger conversations about the implications of representation, data stewardship, and epistemological challenges of data collection. We contend that data representativeness must be analyzed contextually using a critical lens, to accurately assess the potential and implications of greater inclusion of marginalized communities in AI-infused systems.

2. RELATED WORK

Sociocultural diversity has received attention in a wide range of disciplines, such as encouraging gender or ethnic diversity in teams or communities [21, 41, 74], with different concepts of diversity applied in research and applications [159]. More so, AI research has adopted diversity considerations deeply in the ongoing challenge of responsible and ethical AI [24, 42, 113]. Much conversation has been associated with the concepts around balanced representation of sub-groups (e.g., equal participation of racial sub-groups within a focal group) [47]. A growing number of studies have explored bias and performance disparities of AI systems concerning representation [38, 108], especially influenced by demographic attributes like age [36, 97, 124], gender [18, 83, 142, 162], race [18, 96], socioeconomic status [34], and disability status [56, 179]. Often such evaluations found the source of concerns as the under-representation of certain demographic groups in the training data underlying predictive and inferential algorithm [108, 162, 179], calling for action to create more balanced datasets across different demographics. In response, we have seen efforts like constructing image datasets balanced in race, gender, and age (FairFace dataset [80]) or text corpora with gender-balanced labels (GAP [175]).

In support of the current discourse around diversity in AI data, researchers have argued that datasets sourced from people with disabilities and older adults can play an important role [75, 79, 118] such as improving speech recognition with stammering data [40] and object recognition with photos taken by blind people [75]. Calls for action from this community often center around including disability in AI fairness discussions as it pertains to model performance, data excellence, and privacy [48, 77, 126, 168]. Increasing disability representation, however, is complex; there are myriads of challenges in collecting and sharing datasets from this group [1, 143]. Consent and disclosure can be problematic regarding sensitive disability status. Ethical concerns also arise given that datasets collected to mitigate AI bias for people with disabilities can be used against them by detecting their disabilities, leading to further discrimination risks [118]. There are also existing social biases and stereotypes reflected in data representing disability (e.g., [63, 70]), which may produce AI-infused systems that reinforce greater harms and marginalization of people with disabilities [7]. Efforts aiming to increase inclusion thus need to be carefully considered [163].

To recognize the opportunities and limitations of accessibility datasets in the conversation of diversity in broader AI, we first need to understand the current status of representation in accessibility datasets. Prior work investigating issues associated with diversity in AI datasets has mostly focused on examining differences in model performance across pre-defined demographic attributes to draw implications for diversity [18, 34, 162]. This often leaves inquiries about the benefits and appropriate implementation of diversity in data unanswered [47], except for a few exceptions (as shown in Table 1) that explicitly analyzed datasets or issues related to datasets in terms of demographic representation like gender and other sociocultural attributes (e.g., language) to explore the root causes of bias and misrepresentation. These studies concluded that such AI datasets (often image datasets) are skewed towards certain demographics, uncovering under-representation of older adults [109, 128], darker-skin, and females [109, 185], and lack of geographical diversity [148].

Table 1:

Prior work on analysis of broader AI and accessibility datasets with varying sample sizes.

	Data	# of Datasets	Age	Gender	Race	Skin Color	Geography	Sociocultural
Accessibility
Bragg et al. [15]	Sign Language Datasets	n=NA						•
Kaushal et al. [82]	Clinical Image Datasets	n=74					•
Broader AI
Dodge et al. [39]	C4 Webtext Corpora	n=1					•	•
Merler et al. [109]	Face Image Datasets	n=7–8	•	•		•
Park et al. [128]	Face Image Datasets	n=92	•
Scheuerman et al. [142]	Face Image Datasets	n=92		•	•
Shankar et al. [148]	Open Images, ImageNet	n=2					•
Yang et al. [185]	ImageNet	n=1	•	•		•

While representation has been discussed broadly across HCI and accessibility [1, 100] or within specific communities [114, 138], we have only seen a few studies analyzing representation and characteristics pertained to AI training datasets in related work [15, 82]. They are yet constrained to very specific tasks and applications. Additionally, discussions of biases against people with disabilities are found to be manifested in complex ways that require intersectional attention [63, 150]. This research complements prior work, by analysing existing accessibility datasets across the communities, to encourage holistic, societal implications for data representativeness including people with disabilities and older adults.

3. METHOD

Our aim is to conduct a broad investigation of what and how demographic attributes are represented in accessibility datasets—not only in terms of disability representation but also age, gender, and race. To this end, we leverage a recently compiled collection of accessibility datasets, sourced from people with disabilities and older adults. We analyze any available information on the data contributors’ demographics in associated academic publications, sharing sites, and documentation. Here, we discuss the dataset collections, explain our coding and analysis approach, and reflect on our method and limitations. Reflecting on author positionality, we note that this research was conducted by Asian, Afro-Latina, and white scholars, four of whom identified as women, one identified as non-binary, and two identified as disabled. Research in accessibility ranged from first year grad students to a professor who has been publishing accessibility research for about thirteen years.

3.1. Accessibility Datasets in Our Collection

Recently, Kacorri et al. (2020) launched a data surfacing repository, called IncluSet, as a result of putting together a collection of datasets sourced from people with disabilities and older adults that were manually located over a multi-year period [76]. An underlying promise of these datasets is their potential for training, testing, or benchmarking machine learning models. The work was later extended to investigate the risks and benefits of collecting, reporting, and sharing accessibility datasets, analyzed in terms of 10 communities of focus, 7 data formats, and 3 data access methods [79]. We leveraged the accessibility datasets (1984–2021, N=190) included in the existing collection of IncluSet and their groupings (i.e., communities of focus) as the basis for our investigation. Figure 1a illustrates the distribution of the datasets across the communities of focus. The datasets, including their annotations, are of different data types, as shown in Figure 1b. For example, there are voice recordings of people with speech impairments [25], video recordings of Deaf signers [69], text written by people with dyslexia [134], stroke gestures by people with motor impairments [171], photos of everyday objects taken by blind people [88], eye-tracking data from autistic children [43], and activity data from older adults [91].

Figure 1:

Distribution of accessibility dataset count across all communities of focus (a) and data types (b).

Identifying publicly available documentation for these datasets often depended on how they were shared. Out of 190 datasets, about 84 can be downloaded directly and 41 can be accessed upon request—e.g., through a webpage from the dataset creators or an online repository with a summary of the dataset. Summaries vary highly from a few lines to detailed descriptions of the contents of the dataset and how it was collected. Even though none of the datasets had explicitly adopted standardized documentation such as datasheets for datasets [54], some followed a systematic documentation dictated by the platforms where the datasets were stored such as Synapse.org. Associated academic publications were often referred to in the web documentation to link more detailed information about the data collected, though these sources did not always come with consistent information such as the number of data contributors, which could be easily updated on the web documentation. Dataset downloads sometimes came with relevant summary files, including a spreadsheet listing demographic information about people represented in the data. The remaining 65 datasets in the collection did not include any sharing intent with no sources available other than their academic publications. We still include these datasets in our analysis, in accordance with prior work analyzing accessibility datasets [77, 79].

3.2. Manual Coding and Analysis

We conducted an exploratory analysis where our formulation of what-to-code was based on (a) whether demographic information about the data contributors is available, (b) how is it collected and reported, and (c) how are accessibility datasets distributed among demographic groups within communities of focus.

Specifically, beyond the existing codes in Kamikubo et al. [79], we extracted information related to demographic attributes following prior surveys on datasets and studies in accessibility and AI that examined diversity and representation (summarized in Section 2.3). A total of three annotators (a PhD student in Information Studies, a Masters’ student in HCI, and an undergraduate student in Math) were involved in the process, where at least two reviewed the documentation for each dataset and discussed to correct any disagreement and error. They had different levels of familiarity with accessibility and AI. We extracted the following diversity-related information from the documentation, when available:

Age.

We note how any age-related information is obtained (e.g., self-reported, inferred, or unknown), reported (e.g., individual level, year of birth, age bins, and/or aggregate statistics), and shared (e.g., a separate file). We only calculate aggregated statistics from individual-level data when reporting findings and plotting distributions.

Gender.

We note the labels used (e.g., sex, gender), if any; the categories used; the number of data contributors that belong to the categories used; and how metadata was obtained (e.g., self-reported or inferred) and shared (e.g., spreadsheet or publication). In response to concerns raised by trans and information science scholars that the sex/gender distinction can invalidate trans and intersex identities while veiling the socially constructed nature of sex categories, for this paper we use the term “gender” to refer to discussions of characteristics of data contributors (that may be labeled by researchers as either gender or sex) [46, 142, 146].

Race and ethnicity.

Race is a multidimensional and complex concept, not a singular, biological construct with distinct limits into which people can be classified. Alone, race and ethnicity, do not reveal much about an individual’s experiences. As race and ethnicity can be viewed through multiple socially constructed lenses [17], we started with broad coding techniques to identify any information that pertains to these demographic attributes, including potential ethnic and cultural descriptors like geography and language. Manly [104] suggests that these attributes are proxies for or interrelated with unexamined variables, such as education and socioeconomic status. To better our understanding of race/ethnicity, it is central to deconstruct and examine the confounding influences of ethno-racial factors. We note any categories used to refer to data contributors’ racial groups, such as those defined in the census [19] and group ethnic and cultural metadata like nationality, geography, and language under other sociocultural information. Based on the metadata identified, we update the annotation scheme by specifically going over how this information is obtained and shared. Metadata related to education included information in terms of how it is obtained, reported, and shared; language included information on dialect and skills earned which may interact with education; geography included information on data contributors’ birthplaces and the recruitment location; and other information such as nationality or socioeconomic status when available.

3.3. Reflections on Limitations

Annotation consistency.

Annotation tasks are notably difficult, especially if they involve manual inspection of large data requiring particular skills and knowledge. Given that we inspected both dataset documentations and scholarly articles from various publication venues across many research disciplines and sub-disciplines (e.g., Linguistics, Acoustics, Physiology, Computer Vision, HCI, Accessibility), it was unavoidable to go through a messy process to correct errors and disagreement in our codes. The annotators’ varying levels of familiarity with accessibility and AI were also sources of difficulty. This is not a surprise. Even similar annotation tasks that were more limited in scope (i.e. within the field of accessibility), were characterized as “challenging and effortful” [100]. To address the challenges, as the coding process initially started with two annotators (PhD and undergraduate level), we invited a third member (Master’s level) to have a detailed pass. The PhD student took a final pass to ensure that the annotations were agreed upon at least by two annotators.

We also experienced difficulty in programmatically extracting demographic-related metadata. This often created disparities among the annotators in identifying the relevant information from the documentation. We did not find a consistent, standardized method. For example, some methods we used included manually reviewing web documentation that provided summary statistics in writing [135] or table [2] formats; downloading files containing participants’ demographic data (e.g., age, gender) together with collected data points [164] or a separate csv file on participant demographics [6]; or extracting metadata from filenames [65]. Without standardized documentation and evolving practices, whether datasets contained demographic-related metadata was often unknown prior to downloads. In addition, without proper explanation of the labels used for demographic categories, such as in one dataset [6] that provided a supplementary spreadsheet with a label ‘1’ under the Race column for each participant, we could not find the meaning of this information.

Lack of documentation.

As discussed in the Results, information on age, gender and race/ethnicity was in many cases sparse. When available, it was often unclear how the demographic-related metadata was obtained. Thus, we could not verify the source of classifications (such as for gender). Few datasets explicitly documented that the reported information was e.g., “according to self-reports” [191]. Even fewer made inferences on these demographics e.g., “using proprietary classifiers” [177] or “based on visual inspection” [151]; typically these inferences were employed on data collected over the web. Specifically, we observed that three datasets indicate estimations on data contributors’ age; all three are solicited from user interactions with a web search engine with users’ age reported being “over the age of 40 years inferred from their date of birth as reported at registration to Bing” [189] or “inferred using proprietary Bing classifiers”[177, 178].

White et al. [177, 178] employed a similar approach for gender. Whereas Shi et al. [151, 152] determine the gender of individuals by visually inspecting sign language videos from YouTube and the signers’ social media; they used the code “Other” for videos including people whose gender was deemed unknown or where there were multiple signers. While we have included the codes for these datasets in our collection as a reference for future researchers, we don’t include them in our analysis of ‘reported’ demographics; inferences can be inaccurate, perpetuate bias, and perpetuate exclusion (e.g. via binary classification of nonbinary individuals).

None of the datasets in the collection inferred or estimated demographics that pertain to race/ethnicity or other metadata related to nationality, geography, language, and education. Yet, this part of our analysis is the weakest one as it solely relies on a small number of datasets where the race/ethnicity information was specifically ‘reported’; the majority (8) came from US institutions and one from UK even though the institutions of data stewards in the collection spanned across 42 countries from Asia, Africa, North America, South America, Europe, and Australia. Thus, our analysis of this demographic is inherently limited. Only limited reporting of race/ethnicity may be due to a number of factors, such as differences in census reporting among Western and non-Western countries, a prevailing consensus that racial designations do not identify genetically distinct populations, and the likelihood of misuse (e.g., privacy risks for disabled people) [84, 122, 147]. Cooper et al. suggest that “the correlation between the use of unsupported genetic inferences and the social standing of a group is glaring evidence of bias and demonstrates how race is used both to categorize and to rank order subpopulations.” [31]. However, since federal and state legislation in the US have established evident discriminatory practices against African Americans, Hispanics, Asians, and other groups, racial categorization can be utilized to reflect intersectional gaps that are a product of racial stratification practices. Thus, considering the sociocultural and political contexts of different regions to further understand the decision to utilize racial categories is critical. We did not see within the scope of this paper a systematic way to report the somewhat sparse metadata across codes related to data contributors’ nationality, geography, language, and education and tie them to sociocultural and political contexts of different regions. Nonetheless, we include these codes in our annotations for future reference.

Non exhaustive collection.

One of the main limitations of this work remains the fact that the list of datasets in the collection is not exhaustive. While somewhat systematic, the identification of these samples is itself noisy and prone to cascading biased decisions from the researchers collecting them and those that opt/know to include their datasets in the IncluSet repository. The lack of inclusion criteria related to when these datasets were introduced or whether they are currently in use and to what extent, could lead to systematic misalignment between current efforts and past trends. This is exacerbated by the fact that many datasets that are actually employed currently in commercial AI-infused products are not accessible for this type of analysis; representation of different demographic groups could be perhaps deduced via biased performance results (e.g., [18]) but that is beyond the scope of this work. Thus, any insights from our analysis may not be generalizable beyond the research community.

4. RESULTS

Of 190 datasets whose publication and documentation we reviewed, the most commonly found types of demographic-related metadata are age (46.8%) and gender (54.2%), followed by few datasets reporting race (4.7%) and education (12.1%). We find that 71 datasets (37.4%) did not include any information related to the aforementioned types of metadata. These numbers differ from publications that also focus on health, wellness, accessibility, and aging, where few share data; when looking at 792 HCI studies, Abbott et al. (2019) found a distribution of 69.7%, 67.3% and 6.6% on age, gender, and ethnicity, respectively [1]. This difference could be due to tensions inherent in collecting “sensitive attribute data” [1, 11, 16] and concerns related to participant consent and re-identification risks [1]. A similar trend is seen among available metadata with respect to how others can access the datasets. Among those that are not publicly shared, 69.2% reported at least one of the demographics, compared to 57.1% for publicly shared and 53.7% for shared upon request.

In this section, we present our findings surrounding such “sensitive attribute data” in accessibility datasets across communities of focus (Figure 2). To better understand the current status in terms of reporting and including different demographic groups and variables, we focus on the following demographics: age, gender, and race and ethnicity. In our analysis, we compare with existing categories used to represent demographic variables in social data collection (e.g., racial categories in census [174]), and investigate representativeness within accessibility datasets.

Figure 2:

Proportion of accessibility datasets across all communities including metadata related to the age, gender, race, education, or other sociocultural factors about their data contributors. Many datasets (e.g., in the Hearing group) did not contain any metadata.

4.1. Age

A total of 6050 people within the communities of focus contributed data to the 89 datasets whose information on age was included. Their weighted average age was 43.6 (std=26.3). For the remaining of the report, statistics are reported at the dataset level (i.e. sampling distribution of the mean) even though the sample size across datasets varies highly from 1 to 990 people (mean=66.8, std=144.5). Data on age from control groups are not included in the analysis.

4.1.1. What Is Reported.

Datasets mostly reported such information in aggregate though some (36.0%) reported age at an individual level. Aggregate information includes minimum age (1.1%), range (15.7%), median (1.1%), average (20.2%), or a combination (25.8%). Typically, age was reported separately for target (i.e., disability) and control groups (e.g., [45]), contributors’ gender (e.g., [170]), and dataset purpose (e.g., training versus validation [86]). Few report on all groups together (e.g., [22]). Data anonymization is a core component of data management to minimize risk of disclosure while preserving its utility for analysis [81]. However, we find that a majority of the datasets did not incorporate these strategies. For example, bucketing by age groups (e.g., 18–30, 31–45, 46–60 years [107]) was only found in 7 datasets (7.9%).

Only 5 datasets reported median and 3 datasets reported both mean and median. More than half (58.4%) indicate standard deviation, including those reporting age at the individual level for which it can be calculated. All three, mean, standard deviation, and range, can be found for less than half (42.7%) of the datasets (e.g., “The mean age of the subjects was 54.9 ± 13.4 (SD) yr (range 36–70 yr)” [64]). Meanwhile, some documentation noted only the minimum (e.g., “participants aged 50 or older” [180]) or the age requirement for participation (e.g., “18 or older” [13]).

4.1.2. Why Is It Reported.

Most often datasets did not specify why the ages were obtained and reported. It could be an effect of perceived norms and standards for questionnaires within the research community, which often include age questions [68, 161]. Age is an established variable that helps understand the general characteristics of participants. Its distribution may reflect the quality of data collection and analysis [5]; not accounting for age can threaten the generalizability of the work especially when there is a treatment effect heterogeneity in age or other factors that may covary with age (e.g., [121]). Some datasets mention efforts to match age between target and control groups (e.g., [26, 160]) or note age matching as not feasible (e.g., [111]). Others mention age as a confounding variable e.g., for early detection of Parkinson’s disease based on touchscreen typing patterns [72]. Some datasets mentioned the goal of including data from diverse age groups to assess age-related decline of cognitive or mobility performance [91, 116]. For example, in a dataset acquiring age-related pen-based performance [116], participants were grouped based on cognition changes (‘young’ for 18–55, ‘pre-old’ for 56–75, and ‘old’ for 75+). Grouping varies across communities; in an attempt to build a diverse sign language corpus, researchers binned groups as 18–35 years, 36–50 years, 51–64 years, and 65+, rationalizing their decision based on language transmission variability within the Deaf community [141].

4.1.3. Representation Across Communities of Focus.

Figure 3 illustrates with violin plots the sampling distribution of mean age in datasets across communities, where the white dot represents the median, the thick gray bar in the center indicates the interquartile range, and the thin gray line shows the rest of the distribution, except for points that are determined to be “outliers.” Kernel density estimations on each side of the gray lines show the distribution shape. Wider sections indicate a higher probability that datasets will have a mean age of the given value; the skinnier sections indicate a lower probability. We note that datasets vary in their sample size, which is not accounted for by this visualization.

Figure 3:

Sampling distribution of ‘reported’ mean age, which differs across communities. Means are calculated on varying sample sizes.

We find that mean age in datasets differs across communities, with some communities particularly inclining towards samples with a certain target age (e.g., children, older adults). To better understand the age representation exhibited in accessibility datasets, the remainder of the section follows age groups discussed or referred to in prior literature in terms of technology (e.g., ‘older adults’ as 65+, ‘oldest-old adults’ as 85+) [128], disability-related policies (e.g., ‘children’ between 3 to 21 covered in IDEA [94]), and the communities of focus (e.g., ‘toddlers’ of 18 to 36 months in developmental assessment [30]). Of course, variations exist across studies [154] as there is no rigid definition for these groupings.

Older adults.

Many accessibility datasets represent older adults. Among the datasets that contained some form of age-related information, 48.3% included at least one older adult (65+), and 6.7% at least one oldest-old adult (85+). The highest proportion of older adults was in the Cognitive and Health groups, reporting at least one older adult in 83.8% and 73.3% of their datasets, respectively. This may not be surprising, as these groups focus on cognitive and physical decline that can relate to age—e.g., the risk of onset of dementia (e.g., Alzheimer’s disease) increases with older age [131]. Specifically, the Cognitive group had datasets with the highest mean of mean age (mean=61.7, std=12.4) which were often cross-listed with the Mobility and Speech groups including speech or motion data of patients with Parkinson’s disease (e.g., [71, 140]). The oldest participant, aged 89, was reported in the Cognitive and Health groups in the image dataset capturing daily activities of those with episodic memory impairment [89]. Communities that lack older adult representation are Autism, Developmental, and Learning, reflecting a broader gap in research pertaining to these groups [66, 73, 130, 139]. This can be due to many factors; for example, many autistic older adults experienced a severely delayed diagnosis [102]. Many adults with learning disabilities live in institutions such as nursing and residential homes, in which they arrive “before their 65th birthday” with “few opportunities to get out” [165].

Children and youth.

Children and youth are also represented in accessibility datasets; about a quarter (24.7%) of the datasets whose information on age was included contained data sourced by at least one person younger than 18 years old. It increases to 33.7% when including those 21 or younger, as the age criteria for study participation is often noted as 18 or older [13, 45]. Perhaps this reflects some of the ethical challenges in collecting data from children [32] as the process for obtaining consent, assent, or parental permission is more complex for those under the legal age [112]. While overall there are few datasets sourced from youth, they tend to concentrate in the Developmental (85.7% of datasets in this group include at least one person <18) and Learning (100.0%) groups. Datasets in the Learning group often focus on dyslexia (e.g., [53, 115]), where diagnosis is critical at early ages. Data from toddlers (18 to 36 months old) are typically seen in the Development group for the purpose of developmental assessment (e.g., [30]). They mostly involve speech data, sourced by stuttering children [58, 182] or late talkers [120]. The youngest reported age across all the accessibility datasets was 16 months, in a dataset sourced from autistic children [181], though not many (33.3%) datasets reporting age in the Autism group included those under the age of 18. The groups that lack data from children and youth are Vision, Hearing, and Mobility. We suspect that this is reflective of the most common purpose for collecting data such as image and video from this age group, which is to better assess and diagnose; disabilities related to one’s vision, hearing, and mobility have long established methods and instruments that might not require such datasets.

Younger and middle-aged adults.

When looking at younger adults (over 18), we find that surprisingly, many (9) datasets with mean age in the Autism group tend to include people between the age of 18 and 44, with an overall mean of mean age 24.0 (std=13.8). This is in striking contrast with the broader research on autism, where the majority (94%) tends to focus on infants, toddlers, children, and adolescents [73] due to a focus on early diagnosis and intervention [117, 127]. Datasets including younger adults in this group were often collected in the context of assistive technologies (e.g., evaluating text readability and comprehensibility via gaze fixations [45, 183, 184].)) Looking further at datasets skewed towards younger and middle-aged adults, the age range of Hearing and Vision groups was limited, even though visual and hearing impairments could be associated with older age [14, 95]. The datasets in the Hearing and Vision groups that reported age have an overall mean of mean age 28.3 (std=4.2) and 48.7 (std=3.6), respectively. This can be partially explained by how these datasets were collected. For example, the majority (66.7%) of datasets in the Vision group did not include any age information; they were collected from thousands of users via real-world applications (e.g., [57, 78]), where user demographics may not be available or omitted due to privacy concerns. Similarly, in the Hearing group the majority of datasets do not include age information; they tend to collect sign language from online sources (e.g., [92, 151]).

Diverse ages.

We observe that the Language group has the largest age variability. Among others, they include data sourced from children with epilepsy (e.g., [160]), adolescents with language impairment (e.g., [176]), and older adults with aphasia (e.g., [3, 35]). Often datasets in this group come from clinical settings such as the FluencyBank found in TalkBank [101], a shared database established in 2002 for studying human communication. Perhaps this collaborative effort among a wide range of disciplines could explain the variability of datasets spanning across different communities over the years. Datasets in Speech also capture different age groups. Some can be found in TalkBank, including spoken phrases of older adults with Alzeimer’s disease [105] as well as children [182] and adults [187] who stutter.

4.2. Gender

A total of 5598 people within the communities of focus contributed data to the 103 datasets whose information on gender was included. Again, we include information at a dataset level even though the sample size across datasets varies highly from 1 to 818 (mean=59.6, std=106.6). Data on gender for the control groups are not included in the analysis.

4.2.1. What Is Reported.

Gender metadata was commonly reported with the number of data contributors in the form of writing (e.g., “10 blind participants (5 female) ranging in age from 18 to 63 years old” [9]) or table (e.g., a M/F column [6]). Of datasets reporting such metadata, we observed that a binary classification was used (female/male, women/men, girls/boys), with only one dataset in our collection reporting data on the “other” category [49]. However, it is difficult to draw conclusions from this alone, as few datasets reported their method of gendering contributors. Without this, we cannot distinguish between self-identification (e.g., as part of a demographics questionnaire), or an external inference influenced by implicit assumptions (e.g., by the study designers or validators). Furthermore, if participants were asked to self-identify, they may have been limited to choosing from binary options.

4.2.2. Why Is It Reported.

Similar to age being asked in standard demographic questions [68], datasets often included gender information as part of the data distribution, without specifically describing the goal of collecting such information.

Nonetheless, we can attempt to extrapolate the reasoning for some datasets, especially when they contain particular data formats. The highest presence of gender information was in datasets that collected audio (66%) compared to video (27%) or image (32%). Perhaps, this is reflective of an assumption of the influence of gender among those working with speech data. Datasets that capture motion e.g., gait of Parkinson’s disease patients [170], also attempt (about 50% of them) to account for physical measurement differences represented in data by using gender as a proxy.

In order to keep the study design as “unbiased” as possible, some datasets reported that gender (and/or age) was “balanced” in the test group (e.g., “roughly balanced for gender of the 249 participants, 52% (n= 129) were women” [141]), but efforts to balance distribution between target and control groups were much more common (e.g., [170], [125]).

4.2.3. Representation Across Communities of Focus.

Gender demographics vary across the world, with most countries having a female³ share of the population between 49% and 51% [137]. However, overall, accessibility datasets that include gender information tend to be imbalanced with men and boys (60.1%) who are more represented on average⁴ than women and girls (39.9%). This is also evident in Figure 4a, which illustrates with violin plots the sampling distribution of gender representation in datasets across communities of focus, where the vertical dash lines indicate the quartiles and each side of the distribution shows kernel density estimations for ‘women/girls’ and ‘men/boys’. This illustration also highlights how the gap is more prominent in some communities than others.

Figure 4:

Sampling distribution of gender representation across accessibility datasets. The representation gap is more prominent in some communities than others.

Specifically, we see a clear imbalance in the representation of data contributors in the Autism and Developmental groups; on average, 33.1% (std=8.1) and 27.9% (std=9.8) are women and girls, respectively. Such highly skewed representation has been actively discussed in the evaluation and diagnosis of autistic children, given that boys constituted 81% of the sample of children [55]. One widely cited male-to-female diagnosis ratio is approximately 4:1 [51]. However, when the ASD participants are controlled for cognitive impairments, this number changes [85, 98, 103, 106, 138]. About 50–55% of autistic children are estimated to be intellectually disabled (ID) [98]. Among ID autistic children, the male-to-female ratio is significantly smaller, at 2:1 [67]. In autistic children labeled as “high functioning”, the existing literature points to a higher male-to-female ratio, about 6:1. Researchers have theorized an explanation for this relationship could be the tendency of (so-called) “high-functioning” autistic females to “mask” or “camouflage” core autistic traits [90, 133]. A growing body of evidence suggests that current diagnostic criteria for ASD may fail to account for these phenomena and the subtleties in behavior, leading to misdiagnosis and late-diagnosis for minority gender groups (e.g., women, girls, non-binary) [87].

While many communities of focus portray gender disparity in their represented samples, it is not seen in the Vision group, with the average of 50.2% (std=3.2) consisting of women per dataset. According to 2018 U.S. disability statistics [186], 45.3% of visually disabled people were male, and 54.7 % were female. The slight skew towards women has been identified by researchers in this community as possibly attributable to differences in life expectancy by gender in addition to increased risk of visual impairments with age (e.g., macular degeneration) [59], which women are noted to be at higher risk of than men [156].

4.3. Race & Ethnicity

Race is a complex and sensitive demographic variable [52, 145]. Only 9 (5%) accessibility datasets reported metadata on contributors associated with racial or ethnic groups, typically captured by demographic surveys (e.g., [19]). Modern racial classification systems construct race using both observable physical features (e.g., skin color) and nonobservable characteristics such as culture and language [27]. Thus, ‘other’ related demographic information we found could perhaps be utilized to draw some connections and inferences about race, including the place of birth [23], native language [72], or dialect [188]. However, in past studies they have led to issues of forced classification and error [11, 123]. Therefore, in this section we don’t make that connection. We report only on datasets with explicit racial and ethnic information.

4.3.1. What Is Reported.

The categories we found delineating racial composition were mostly ‘White’ and ‘Black’ [144], with variations of reporting them as ‘White-Caucasian’ or ‘Caucasian’ and ‘African-American’ [160, 182, 191]. For other racial groups, data were ambiguously grouped together (e.g., “62% Caucasian, 30% African-American and 10% other” [160]) or can be extrapolated by subtracting what was reported as the proportion of the ‘white’ category only [190]. The use of these terms also highlight the limitations of the taxonomical racial categories; ‘Caucasian’, for example, is rather discussed as outdated and disproved [119].

Similar to age and gender, race was reported separately for target and control groups (e.g., [190]). Notably, one speech dataset sourced from stuttering children aimed at a race-matched (as well as age- and gender-matched) cohort of children [132]—here, both stuttering and non-stuttering groups had 2 African American children and 1 child of mixed racial ancestry. This was also the only dataset in the collection reporting about mixed race, although we saw an attempt to collect data on race, including ‘Mixed’, from a demographic questionnaire in a study on Parkinson’s disease [13].

4.3.2. Why Is It Reported.

Looking at datasets whose data on race was collected and/or reported, they are often related to medical research associated with studies on specific disorders. Specifically, they include speech samples collected from people with aphasia [144], Parkinson’s disease[190], Alzheimer’s disease [6], and epilepsy [160] to study early detection of impairments underlying cognitive disturbance. In medical research domains, there are controversies around collecting data on race, raising both benefits and risks given disparities in health outcomes established for racial minorities [50, 62]. Concerns also lie in the taxonomy of the categories used, which have brought efforts to standardize and improve methods of obtaining and reporting data on race [8, 50]. Recent guidelines [50] suggest including an explanation of who identified participant race & ethnicity and reasons for collecting the data. We did not find disclosure of the source of the classifications among the datasets included (e.g., self-report, observation), nor a justification of why it was collected.

4.3.3. Representation Across Communities of Focus.

It was hard to distinguish the data between race and ethnicity or other sociocultural information, especially when the data spans multiple concepts and forms of classification (e.g., “129 of Caucasian, 14 of African American, 2 of Hispanic, and 2 of Asian origin” [182]). For example, in US, guidelines that inform data collection for census note that the concept of race is separate from the concept of Hispanic origin [173].

For the few datasets that reported data contributors’ race and ethnicity, the norms of how to report were highly inconsistent. Thus, with high variability and a small sample, we could not leverage standardized methods to analyze racial group composition among the communities of focus. The categories we saw (often in Cognitive and Language) were associated with ‘white’ or ‘non-white’, portraying one group as primary over another. Mixed race was rarely indicated, which is problematic given changes in racial categories (e.g., in the US census) reflecting racial mixture [20].

5. DISCUSSION

Our overarching goal lies in understanding the current state of representativeness of marginalized groups in AI datasets (along the axes of age, gender, and race & ethnicity) with a specific focus on disabled data contributors. This is relevant to the greater discourse around AI, ethics, and fairness, as marginalized communities tend to be under-represented in data [47], perpetuating cycles of exclusion as technology advances even for technologies that meant to promote inclusion such as assistive technology. We contribute to this important ongoing discussion through our analysis of 190 accessibility datasets. Specifically, we examine representation gaps and trends that can potentially lead down the road to further harm for the people who stand to be adversely affected by emerging, potentially ubiquitous technology. In this section, we recap and discuss the challenges and opportunities for representation while considering directions the accessibility field could take to carefully include marginalized communities in AI-infused systems.

5.1. Addressing Challenges and Seizing Opportunities for Representation

Our analysis revealed unique challenges in ensuring representation of intersecting demographics in accessibility datasets. Some representation gaps are attributable to societal and cultural norms and biases that operate intersectionally. For example, communities lacking older adult representation are Autism, Developmental, and Learning. This reflects not only a broader research gap on these groups [66, 73, 130, 139] but also discrimination at the intersection of disability and age; e.g., many autistic older adults live without an accurate diagnosis [102]. Similarly, looking at the intersection of disability and gender, we observe a gap for Autism, Developmental, and Learning groups, where men and boys were often over-represented. These cases can have pernicious implications characterized not only by the communities of focus but also long established research frameworks that propagate existing societal marginalization, highlighting the importance of making gender-specific changes (e.g., diagnostic criteria for autism [37, 87]).

In annotating accessibility datasets, we also surfaced how socially constructed identity categories such as race and gender are reproduced. Similar to Scheuerman’s meta-analysis of gender in face datasets [142], by analyzing information such as reasons for reporting/data collection and labels used for metadata categories, we contribute a sociological meta-examination through which the research and data collection process itself can be analyzed for bias. For example, we found that the notion of a gender or sex binary was not explicitly challenged in our collection; only one dataset reported data on the “other” gender category. This may have downstream effects in shaping machine learning model design and subsequent problems/contexts—for example, in binary gender classification, which may harm nonbinary communities through technology-enabled misgendering [61].

We also found that there is very little reporting of how identity labels were associated with data contributors, whether through selfi-dentification or external assumption (e.g., via preformed binary categories). We recommend greater transparency in disclosing these aspects of the data collection process, and for gender in particular, to include nonbinary, self-describe, and prefer not to disclose options, as recommended in the related literature [158].

At the same time, we acknowledge the implementation challenges that may need to be addressed to support transparency—e.g., how to produce a set of questions which do not elicit information leading to unintentional misuse or unwanted societal biases for data contributors. We emphasize that careful reflection on this process is needed on the part of researchers who are collecting and reporting contributor data, including implications of use (e.g. surveillance) and any potential harms enacted by power structures through the systems we build. Aligning with recent research [110], we recommend an examination and contextualization of data representativeness grounded in political, economic, and socio-cultural lenses, integrating insights from scholars in fields such as critical disability studies [28], trans/gender studies [157], and histories of social movements [136] into an analysis of power relations. As an example, one could draw from recent work by disability studies scholars examining the context the data is collected in (i.e., for AI systems vs for visibility and activism) and how representation impacts are also context-dependent [93].

5.2. Developing Participatory Approaches to Data Stewardship

This challenge of partitioning the pool of accessibility datasets into sub-communities was very real in our analysis, as the groupings that we opted for may not necessarily reflect the identities of individual data contributors. Recent work exploring challenges for collecting disability data suggests the voices of contributors to be reflected and provides best practices to ask about disability status [10]. Perhaps, to mitigate harms experienced by those from marginalized communities who are misclassified, we can extend this approach to other categories such as race and gender. Specifically, we urge researchers to come up with approaches for more meaningful engagement of data contributors in the data stewarding process. Echoing Shneiderman’s motto [153], we recommend “researchers in the loop, disabled contributors in the group”.

One way we could go about this is to employ participatory approaches to the data collection lifecycle in which users have the opportunity to enact their values in how their data is collected, maintained, shared, and interpreted in and out [33, 99]. Of course, this would require careful consideration of the many moving pieces in the Fairness, Accountability, Transparency, and Ethics (FATE) landscape both in terms of parties involved as well as exchange and access mechanisms; Bragg et al. [15] provide a wonderful starting point for this discussion in the context of the Deaf community. For example, to avoid inadvertently extractive approaches, and aligning with recent literature, we recommend meaningfully compensating participants for their work as data contributors [155]. In this vein, we also recommend developing long term relationships with data contributors and their communities (where possible) to facilitate sustainable and mutually beneficial collaboration, especially when designing and evaluating AI-infused systems that use contributor data [155, 163]. Disability community-led initiatives can help concentrate research efforts on those most likely to have a positive impact; the idea generation phase may be particularly fruitful when rooted in first person lived experience (e.g. as provided in [129]).

5.3. Addressing Epistemological Implications in Future Work

We encountered epistemological limitations at various stages in the annotation and analysis process. One such limitation is the extent to which strong claims can be made about overall representativeness, due to the lack of reporting and global statistics for disability, age, gender, and race. In addition, our findings are intrinsically linked to existing sociocultural contexts and hierarchies. Our analysis of accessibility datasets showcases these epistemological limitations. By acknowledging these limitations, we hope to spark conversations on the inclusion of marginalized communities in AI-infused systems and its myriad challenges. In future efforts, we recommend the following for broader research implications:

Exploration of disabled people’s concerns around representation.

Increasing representativeness may not always be beneficial; it may perpetuate injustice as extensions of existing systems of oppression and power. As explored in the previous section, it is vital to include first person disabled perspectives on representativeness and inclusion, as well as data collection and sharing practices. Future work remains in exploring contributor concerns such as privacy [60, 77] and surveillance [7], especially for multiple marginalized contributors.

Analyzing other sociocultural factors.

A more in-depth analysis of the sociocultural contexts in which datasets were produced, not just what was reported, could lead to interesting insights. A quick inspection of our datasets revealed that when data involves children, specifically in studies of developmental disability, we sometimes find family information, such as socioeconomic status [132] or parental education [58, 160]. Future work could explore representation along axes of level of education, language, nationality, and socioeconomic status of the data contributors, as well as intersections between them. It would also be interesting to explore the influence of dataset origin (i.e. from the HCI vs medical research community) on demographic representation as they may opt for different models of disability.

Accounting for dataset impact.

Our analysis of the implications of representation is complicated by the fact that datasets vary in research impact. Potential indicators of impact include the number of citations, the models they are used to train or benchmark, the venues in which they are published, and whether they originate from academia or industry. Future work remains in investigating and defining impact indicators and metrics, and weaving those insights into discussions of representativeness.

Beyond accessibility datasets.

While any insights from our analysis may not be generalizable beyond the research community, our findings present an opportunity for broader AI communities to strive towards more representativeness—along disability and other dimensions—by including accessibility datasets in their training data. For example, AI datasets have been critiqued for being heavily skewed towards younger adults, and under-representing older adults [128]. In contrast, accessibility datasets yield a wide variability of age groups. In future research, we strive to connect our discussions of representation gaps with larger trends for broader AI datasets and investigate whether accessibility can be used as a lens to diversify representation for the broader AI community.

6. CONCLUSION

We conducted a detailed analysis of data representativeness among 190 accessibility datasets, with an emphasis on the intersections of disability with age, gender, and race & ethnicity. While we found diverse representation of age in accessibility datasets, we identified gaps in gender and race & ethnicity representation among these datasets. Our findings illustrate the implications of historical and social contexts. Although we acknowledge there are limitations when collecting these demographic variables, going forward, we propose a participatory approach when collaborating with disabled contributors and encourage transparency regarding data collection purpose and maintenance throughout the process. We hope our effort elucidates the current challenges in representation among the accessibility community while expanding the space of possibility for greater inclusion of marginalized communities in AI-infused systems more broadly. Finally, we hope that our efforts provoke conversations on data representativeness through a critical and epistemological lens.