Table 1.

Summary of Types of Bias that Can Influence the Development and Application of AI Models

Bias Type	Definition	Example
Sociocultural Foundations
Structural inequalities	Institutions, governments, and organizations have inherent biases that cause some groups (e.g., race, gender, age) to be favored over others; can be in the form of education, housing, health care, environment, media, etc.	Compared to children in wealthier areas, children living in under-resourced neighborhoods often receive fewer educational resources, consume lower quality air and water, and have poorer municipal services (Evans, 2004). It is an important preliminary step to consider how underlying structural inequalities might impact the model development process because such foundations can unconsciously impact our perceptions and decision-making. For example, children without access to high-quality educational resources may have lower test scores. If we aim to build algorithms to predict important outcomes, then an important foundational step would be to consider how structural inequalities might relate to the outcome and thus lead to biased results (Churchwell et al., 2020)
Historical bias	Bias in data resulting from the context of a given historical time.	Algorithms built from Google News articles exhibited bias by over-associating certain occupations with men versus women (e.g., men as computer programmers and women as homemakers). Because the algorithms were built from historically sexist text data, the bias was carried forward to the resulting algorithm (Bolukbaski et al., 2016).
Homogeneous teams	Teams lacking diversity can increase the chances of implicit biases going unnoticed because the members of the team may all may share the same biases.	Scientific explanations of human fertilization have traditionally considered sperm to be “active” and “competitive” and eggs to be “passive” and “receptive” (Campo-Engelstein & Jonson, 2013). However, new evidence shows that these prior explanations of human fertilization are inaccurate, with eggs playing a more active role in determining which sperm is fertilized than previously thought (Fitzpatrick, 2020). Members of other genders could have helped to identify biased thinking in the conceptualization and theory of human fertilization.
Data Collection
Representation bias	Occurs when the collected sample is not diverse enough to represent the population for whom an application is made.	In automated speech recognition systems from five of the top tech companies, there is a 35% word-error rate on average for Black and African American speakers compared to 19% for non-Hispanic White speakers, demonstrating that the input dataset was not representative enough to account for the speech of people from different cultural and racial and ethnic backgrounds (Koenecke, 2020; Bajorek, 2019).
Measurement bias	Occurs when the measures and features researchers decide to use when building models are more accurate in some groups than others.	Algorithms might use automated speech transcription as a feature to predict diagnosis. However, the feature itself is biased because it produces more word-detection errors in women and minoritized groups compared to non-Hispanic White men (Bajorek, 2019). Additionally, measurement biases can often arise in age-heterogeneous samples because tests developed for young adults may not work for older adults (Zeidner, 1987).
Aggregation bias	Occurs when a target group is treated as a monolith and false conclusions are made because the model did not account for group diversity.	Hemoglobin AIc is regularly used to diagnose and monitor diabetes. However, research shows that levels of Hemoglobin AIc differ across ethnicities, contributing to bias in diagnostic classification (Ford et al., 2019).
Model Building
Confirmation bias	Occurs when researchers favor information that supports pre-existing beliefs, causing them to avoid looking for information that may be to the contrary.	Wakefield et al. (1998) inaccurately linked the MMR vaccine to autism. Though the study was retracted from the British Medical Journal in 2010 after evidence that Wakefield manipulated and ignored much of the data, the disproven claim still impacts community perception of vaccines today (Belluz, 2019; Deer, 2011).
Label bias	In supervised machine learning, labels must be applied to the training data and then fed to the machine learning algorithm so that the AI can properly predict what future values will be. However, these labels may not always represent all the possible labels for a given variable or can be inaccurate, resulting in biased predictions.	Individuals were tasked with identifying and labeling Twitter posts as hate speech for a machine learning algorithm. Inaccuracy in labeling the tweets of African American people directly led to the algorithm’s decision to inaccurately label future African Americans’ tweets as hate speech (Davidson et al., 2019; Sap et al., 2019).
Feature selection bias	In classification models, feature selection is a technique used to select the variables that will be the best predictors for a given target value or to reduce the number of unrelated variables that may influence predictions. Bias in this technique occurs when unrepresentative or inaccurate features are selected.	In mammogram interpretation, images are difficult to interpret and must be cleaned to draw accurate conclusions. Removal of irrelevant image information is necessary to denoise the image. Sometimes the images may contain additional content that is insufficient to be used as features. Failure to exclude such noise could result in algorithms producing poor generalizations across different images that may contain a variety of irrelevant background content (Tian et al., 2021).
Model Performance and Evaluation
Class imbalance bias	Occurs when facet a has fewer training samples compared to facet b in the dataset; this results in models preferentially fitting larger facets at the expense of smaller facets, which can result in greater error rates for facet a. Models are also higher risk of overfitting smaller datasets, which can cause larger test error for facet a.	If a machine learning model is trained primarily on data from middle-aged individuals, it might be less accurate when making predictions involving younger and older people. Similarly, when there are very few examples in the dataset of people from minoritized backgrounds compared to the many examples for the majority, predictions for minorities may be less accurate (Amazon Web Services; Ling & Sheng, 2010).
Covariate shift	When the test data distribution does not match the distribution of the training data due to shifts in the target population with time.	Covariate shift can occur for image categorization and facial recognition where models achieve high accuracy on a labeled training dataset, but model accuracy can decrease when deployed with live data. For example, subtle changes in lighting could shift data distribution points lowering model accuracy (Trotter, 2021). Light quality and intensity are covariates that impact the relationship between the features and labels. If light quality and intensity change over time (e.g., changes in the time of day the picture was taken or the type and quality of lights used in the home or office), the algorithm performance can be impacted and rendered less accurate over time.
Evaluation bias	When a model demonstrates false efficiency because the training set used is unrepresentative of a minority group and then later demonstrates efficiency on the test set due to it being equally unrepresentative. Thus, showing accuracy but false generalizability.	The use of inappropriate and disproportionate benchmarks for evaluation of application in imbalanced datasets that contain mainly lighter-complexion individuals; these benchmarks are used in the evaluation of facial recognition systems that are biased on skin color and gender (Buolamwini & Gebru, 2018; Mehrabi et al., 2021)
Human Inference and Deployment
Deployment bias	Occurs when a model is deployed in a setting or context for which it was not designed.	Models developed to predict risk of recidivism are often used to determine a defendant’s length of stay, which was not the model’s intended use (Kehl & Kessler, 2017).
User interaction bias	Occurs when users’ behavior and interaction with a given application or website can introduce bias into the algorithm.	YouTube’s restricted mode was censoring LGBTQIA+ content. They used user input to inform an automated detection algorithm to determine whether content was appropriate. LGBTQIA+ content, even though it was not explicit, was censored due to people flagging such videos. The algorithm used that information and thus classified similar videos as inappropriate. This issue represents how user interaction and the dominant society’s moral values could further marginalize historically marginalized groups (Bensinger & Albergotti, 2019).
Feedback loop bias	Occurs when a model with an existing bias further reinforces that bias via human-application interaction and the use of newly collected data to feed back into the algorithm for further prediction and training.	Predictive policing models utilize arrest data as training datasets. If police are more likely to make more arrests in a more heavily policed area, using arrest data to predict crime hotspots will disproportionally channel policing efforts to already over-policed communities (Chouldechova & Roth, 2018; Lum & Isaac, 2016).

Note. The list of bias types that can affect AI is not exhaustive.