Picture a data scientist: a call to action for increasing diversity, equity, and inclusion in the age of AI

Abstract

The lack of diversity, equity, and inclusion continues to hamper the artificial intelligence (AI) field and is especially problematic for healthcare applications. In this article, we expand on the need for diversity, equity, and inclusion, specifically focusing on the composition of AI teams. We call to action leaders at all levels to make team inclusivity and diversity the centerpieces of AI development, not the afterthought. These recommendations take into consideration mitigation at several levels, including outreach programs at the local level, diversity statements at the academic level, and regulatory steps at the federal level.

INTRODUCTION

When we ask children to draw a scientist, less than 28% draw a woman¹. Where this picture may be outdated for professions such as medicine and biology, it remains woefully accurate for data scientists working on artificial intelligence (AI). Globally, women hold less than one-third of data science jobs and this share has seen a small decline since 2018.² Data scientists from ethnic and racial minority backgrounds are especially underrepresented. Moreover, authorship for scientific publications within the AI field is unbalanced across gender^3,4 and if current trends hold, gender parity for data science authors will not be reached for another century.⁵ In addition, grant funding is negatively biased against consortia with a higher share of female^6–8 and Black principal investigators.⁹

Previous work asserts that a diverse informatics workforce broadens the research agenda and facilitates the development of equity-centered technologies.¹⁰ Here, we build on this argument by focusing on recent research pertaining to diversity, equity, and inclusion in the data science field. This perspective demonstrates how a lack of diversity, equity, and inclusion in the data science profession might hamper the ethical and responsible development of AI and is especially problematic for its application in healthcare. We outline potential solutions and urge leadership to provide consistent and coordinated guidance to increase team diversity.

FAIRNESS IN AI ALGORITHMS

The promise of AI for healthcare is undeniable, but critics of the technology have emphasized the risks posed by biased or unfair AI algorithms exacerbating societal inequalities.¹¹ Bias in AI algorithms is common and can for example be caused by measurement error, missing data, or underrepresentation.^12,13 Here, we focus on the type of algorithmic bias that can systematically and harmfully disadvantage a particular group by producing discriminatory predictions for gender, race, or other protected identities.^14,15

Healthcare algorithms are rife with algorithmic bias. Recently, an algorithm widely used by health systems for resource allocation was found to display racial bias.¹⁶ According to this algorithm, Black patients were only provided with similar levels of care when they were sicker than White patients. Many other examples exist, such as an AI underdiagnosing under-served populations such as female patients in chest X-ray pathology classification.¹⁷ Also, models predicting in-hospital mortality for intensive care unit (ICU) patients showed poor model calibration for Black, Hispanic, and Asian patients with respect to the White majority group.¹⁸

Moreover, the development of modern-day health technologies often lacks the inclusivity needed to service a wide range of people. Take Apple’s 2014 Health application: the application was marketed to provide a comprehensive health check, but its team overlooked the charting of women’s menstrual cycles.¹⁹ Forgetting one of the oldest forms of self-tracking, the application illustrates a blind spot in women’s needs. As another example, inclusive treatment may be lacking for certain medical procedures, such as pulse oximetry and mechanical ventilation. Pulse oximetry measurements were found to systematically overestimate oxygen levels for darker-skinned patients^20,21 and Black ICU patients were found to be less likely to receive ventilation treatment and more likely to receive a shorter treatment duration.²²

DIVERSITY, EQUITY, AND INCLUSION FOR AI DEVELOPMENT

Clearly algorithmic bias, non-inclusive design, and biased medical procedures can eventually lead to inequitable outcomes across subgroups and societal harm. Despite its tremendous potential for all populations, AI applications can and will exacerbate inequalities when algorithmic bias is left unchecked. Increasing the diversity of AI development teams is one mitigation step that could help to counteract algorithmic bias. Some preliminary work examines how team diversity may contribute to mitigating bias throughout the AI lifecycle. First, including a diverse perspective from designers, coders, health practitioners, and end-users may lead to products that better serve the needs of their respective communities.^23–25 For example, diverse team members may aid in better anticipating the likely impacts of certain model choices on different subgroups and modes of failure.²⁵ Second, a recent study found that AI developers from the same demographic group were more likely to develop models with the same prediction errors compared to AI developers outside of that group.²⁶ Hence, composing a diverse development team from the get-go may assist in addressing algorithmic bias by averaging out these prediction errors across developer subgroups.²⁴ Finally, team diversity can aid in broadening the actual questions being addressed by AI, for example developing a model predicting appointment no-shows versus a model predicting barriers to appointments, such as timing of appointments. This is not to say that team diversity is a panacea nor that women can only design products for women, Black people for Black people, etc. On the contrary: team diversity is an important option in the assortment of mitigation strategies available, such as improving the representativeness of the AI training data²⁵ and educating developers on fair AI practices.²⁶ A combination of these strategies is needed to effectively combat algorithmic bias.

WHAT DRIVES THE LACK OF DIVERSITY, EQUITY, AND INCLUSION?

Several causes may contribute to the lack of diversity in AI development teams, such as the lack of role models and the “leaky pipeline,” a metaphor describing how marginalized groups progressively leave science, technology, engineering, and mathematics (STEM) subjects in the period between preschool and college. Work environments that lack a diversity focus may be another contributor.²⁷ The latter may be especially problematic when good professional traits recognized in the majority group are valued differently in minority groups. For example, ambition is valued more negatively in female academics than in male academics.²⁸ Moreover, professionals with minority backgrounds may be further disadvantaged by what is referred to as a “minority tax”: overtime spent on diversity initiatives that come at the expense of other activities more directly beneficial to one’s career.²⁹ In academia, this may consist of time spent on committees to meet diversity quotas rather than on promoting their career through research and teaching.

A CALL TO ACTION

We encourage leadership at all levels to provide consistent and coordinated guidance to increase team diversity. Various mitigation steps exist in the literature. At the community and local governance level, encouraging involvement in outreach programs could inspire members from minority backgrounds to pursue STEM subjects. For example, the American Medical Informatics Association’s (AMIA) First Look program introduces women to informatics and provides mentoring and career resources.¹⁰ For the academic sector, incentive structures promoting diversity could be tied to the hiring and retention of diverse faculty, journal editors, and senior leadership, especially where AI research is conducted.^30–32 A great example is the Diversity, Equity, and Inclusion (DEI) Task Force that advises AMIA on these matters.³³ When evaluating and allocating grants, committees can require diversity of the investigative team, including diversity of senior investigators and key personnel.⁹ Academic success and impact could be valued beyond citations to also encompass mentoring, teaching, and well-being.³⁴ In industry, raising awareness about biases via seminars or programs, helping employees to better understand their biases, and targeted hiring practices to increase diversity may aid to address the problem.^26,31,35,36 At the federal level, regulatory oversight could be expanded to include evaluations on the performance of AI systems across populations, ensuring the reliability of AI tools across underrepresented populations.^37,38 In tandem with regulatory oversight, monitoring, and auditing are needed at all levels to secure the fair and inclusive use of AI. At the level of the manufacturer, surveillance systems and vigilance in terms of incident reports and safety warnings that include societal or population harm should be employed to provide post-market monitoring of algorithmic bias.^39,40 Moreover, the care organization where the AI is to be implemented is advised to draft a monitoring plan, ensuring that their target population is represented in the training data. In this plan aspects such as the monitoring of the expected and unexpected effects of the AI on clinical practice may be described.⁴⁰

CONCLUSION

The problems associated with biased or unfair AI go beyond mere technological challenges. It is a problem ingrained in the inequities of our societies and requires structural change at the organizational level to affect change at the technological level. Diverse and inclusive AI teams form an important mitigation strategy toward achieving equitable and fair AI development. Team inclusivity and diversity should therefore become the centerpieces of AI development, not the afterthought. This will benefit not only the AI technology itself, but the entire society that may one day be reliant on it.

FUNDING

Research reported in this publication was supported by the National Library of Medicine of the National Institutes of Health under Award Number R01LM013362. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

AUTHOR CONTRIBUTIONS

AAHdH, MMvB, and TH-B conceived the idea, wrote the initial draft, edited, and approved the final manuscript.

Data Availability

No new data were generated or analyzed in support of this research.