Abstract
Background
Equitable representation of racially and ethnically diverse subpopulations in clinical trials continues to be a problem, and trial participants do not always reflect the demographics of the population that the investigational product will be used to treat. The imperativeness of equitable representation of clinically relevant populations in clinical trials has implications for improving health outcomes, increasing knowledge about the safety and efficacy of new treatments across a wider population, and broadening access to innovative treatment options offered in clinical trials.
Methods
The purpose of this study was to understand organizational elements that are involved in the active implementation of racially and ethnically diverse inclusive recruitment practices for biopharmaceutical-funded trials in the United States. Semi-structured, in-depth interviews were used in this qualitative study. The interview guide was designed to explore the perceptions, practices and experiences of 15 clinical research site professionals related to recruiting diverse trial participants. Data analysis utilized an inductive coding process.
Results
Five themes were identified pertaining to the actual implementation of inclusive recruitment practices that provided explanations for organizational components: 1) provision of culturally appropriate, general disease and clinical trial education 2) organizational structure tailored for diverse recruitment 3) strong sense of mission related to improving healthcare through clinical research 4) culture of inclusion 5) inclusive recruitment practices evolving based on learning.
Conclusion
The findings from this study offer insight into improving access to clinical trials by focusing on organizational change initiatives.
1. Introduction
While the United States population is growing more diverse, people of color and other underserved populations face health disparities and issues with healthcare access driven by underlying social and economic inequities [1]. Rates of chronic disease disproportionately impact racially and ethnically diverse populations [2]. The disproportionate rates of COVID-19 illness and death affecting communities of color further highlighted longstanding health disparities and healthcare access inequities that continue in the United States [1,3]. Individuals from racially and ethnically diverse backgrounds have less access to and are provided with less opportunity to participate in a clinical trial and are consistently underrepresented as participants in clinical research [[4], [5], [6], [7], [8], [9], [10], [11]]. Improving diversity in clinical research can promote equality in health and healthcare. Additionally, it is important to identify any variability in drug treatment response across subgroups as subpopulation variation has informed discrete prescribing recommendations for several approved drug products [[12], [13], [14], [15], [16], [17]].
The National Institutes of Health (NIH) mandated the inclusion of women and members of diverse groups and their subpopulations (distinguished by racial, ethnic, and/or cultural heritage) as participants in NIH-funded clinical research since 1993 [18]. Although not mandated, the Food and Drug Administration (FDA) set forth expectations that clinical trial populations should reflect the characteristics of clinically relevant populations (with regard to age, gender, race and ethnicity) for which the investigational product would ultimately be treating [11,19]. Despite these expectations, persistence of systemic barriers contribute to the lack of progress in achieving clinically relevant trial populations. Emphasis is often placed on patient level barriers, including lack of trust in the healthcare system and in clinical research [[20], [21], [22]]. Factors impacting participation rates expand beyond an individual patient level as individuals from diverse communities are interested in learning about clinical trials and tend to enroll when given the opportunity [[23], [24], [25]].
The COVID-19 pandemic sparked a renewed focus on addressing disparities in clinical trial diversity with representation in COVID-19 vaccine studies publicly scrutinized [[26], [27], [28]]. Research manufacturing companies have demonstrated an accelerated commitment to design more inclusive trials that expand access and opportunity for clinical research to historically underrepresented and underserved populations even beyond COVID-19 [29,30]. Pfizer announced their commitment to achieving diverse participation in all trials they sponsor [31]. Although not representative of the disparate impact of COVID-19 in the United States, the intent and efforts taken to ensure representation in COVID-19 vaccine and treatment trials resulted in expanding access to these trials [3,29,[32], [33], [34]].
Findings in the literature suggest that the successful enrollment of historically underrepresented populations in NIH-funded research may have been partially due to research site characteristics [9,35]. Many research sites either lack the experience or do not have ample resources to support diversity, equity, and inclusion recruitment efforts, such as community outreach activities [20,22,36]. Joseph and Dohan (2009) explored organizational barriers to enrolling diverse participants in cancer clinical trials and found organizational climate and research infrastructure negatively impacted recruitment [36]. Regarding recruitment of diverse clinical trial participants, organizational elements are not widely cited in the literature outside of perceptions of barriers elicited from Principal Investigators (PIs) reported in NIH-funded research (that wasn't specifically exploring organizational components) [4,22]. More research is needed to understand how organizational characteristics may play a role in overcoming institutional barriers and improving access for historically underrepresented and underserved populations.
This study is focused on the significant public health problem of underrepresentation of racially and ethnically diverse populations in clinical trials by exploring organizational elements involved in the active implementation of inclusive recruitment practices for biopharmaceutical-funded trials in the United States as experienced by clinical research site stakeholders involved in recruitment.
2. Methods
2.1. Study design
A qualitative design utilizing in-depth, semi-structured telephone interviews was used to gain insight from individuals who work at a clinical research site regarding trial recruitment. Questions were constructed to be broad to allow interviewees to freely share their perceptions and experiences with additional questions designed to prompt comprehensive responses. For example, one question asked was, “What do you see playing a role in your recruitment efforts,” with a follow-up prompt, “How about with inclusive recruitment?”
While the terms ‘inclusive’ and ‘diverse’ are very broad, for purposes of this research, these terms were defined to describe racially and ethnically diverse populations. The overarching research question was:
What organizational elements are involved in the active implementation of racially and ethnically diverse recruitment practices for biopharmaceutical-funded trials in the United States?
This article includes partial findings from a broad study that applied the socio-ecological model as the conceptual framework. The socio-ecological model focuses on interrelationships between individuals and their social and physical environment under the premise that behavior is both affected by and affects the environmental system that it is part of. The ecological environment is a nested arrangement of structures that can affect and be affected by individual and organizational behavior at multiple levels that include intrapersonal, interpersonal, organizational, community, and public policy levels of influence [37,38].
2.2. Sample selection
Purposeful sampling was used to identify interviewees based on the eligibility criteria to learn about issues of central importance to the study purpose [[39], [40], [41]]. Snowball sampling where interviewees were asked to identify additional qualified clinical research site stakeholders was also utilized. The PI used publicly available data from CenterWatch.com to create a recruitment list inclusive of clinical trial research staff located in geographic areas of the United States with a large ethnically diverse population, such as Miami and New York City, as well as those involved with clinical trials in disease areas that disproportionately affect diverse populations. Upon Seton Hall University Institutional Review Board approval in October 2018, the PI emailed the Study Solicitation to individuals on the recruitment list. Individuals confirmed their interest by clicking on a hyperlink directing interested individuals to a registration form where they answered an eligibility screening question. Individuals that met the online eligibility criteria were contacted by the PI to further confirm eligibility. Respondents were selected to participate in interviews on a first come, first served basis provided they met the eligibility criteria.
In this study, interviewees were involved with actively recruiting racially or ethnically diverse participants for biopharmaceutical-sponsored clinical trials in the United States. Other inclusion criteria included being 18 years of age or above and having the ability to understand and speak English.
The recruitment period was 5 months in duration and 165 individuals were invited for study participation, of which 15 individuals participated who provided their insights on and experiences with recruiting diverse participants into clinical trials across 15 research sites. There are no sample size limitations in qualitative inquiry. Instead, respondents are selected to provide information-richness to the study findings [40]. Inclusion of 15 interviewees is in accordance with established guidance by qualitative experts to be sufficient for providing in-depth coverage of the phenomenon [[42], [43], [44]]. Rigorous methods were used to maintain trustworthiness in the findings and to ensure findings relay the interviewees’ voices.
2.3. Data collection
Interviews were conducted November 2018 through March 2019. Prior to conducting each interview, the PI assigned interviewees a number to be used to identify for transcription and data analysis to maintain confidentiality. A signed informed consent was also obtained. Each interview was approximately 1 hour in duration and audio recorded with verbal consent. At the end of the interview, questions related to characteristics of the interviewees and the organizations in which interviewees worked at the time of the interview were asked verbally.
2.4. Data analysis
Each recorded interview was transcribed verbatim using Express Scribe Transcription Software. A transcription key was used to capture voice inflection and emphasis participants placed on their words. The transcripts were checked by playing the audio recording back and ensuring that the recording matched the transcription. ATLAS.ti data analysis software version 8 was utilized for organizing interview transcripts and facilitating data analysis. The PI utilized methods described by Creswell (2013) and Saldana (2009) to guide the analysis process [42,45]. Prior to coding and categorizing, the PI read transcripts in their entirety to get a general sense of the information collected and reflect on the overall meaning. Holistic codes were applied to chunk data into broad topic areas. The PI formed first level descriptive codes inductively based on meaningful, recurrent segments of data. Attribute codes were assigned for demographic data. The PI reviewed the first cycle codes, identified commonalities and created second level pattern codes to organize by similar themes. Simultaneous coding was used, in which more than one code may be applied to the same chunk of data [45]. Deductive coding was then used whereby the codes were categorized by each level of the socio-ecological model (structure code book). After all data were coded, major categories were developed based on the PI's analytic interpretation of patterns that were identified. Overarching themes were developed by the further classification of several categories that formed a common idea. As with the codes, the categories and themes were classified by the socio-ecological model levels.
All data for three interview transcripts were coded initially in order to develop a preliminary codebook that included definitions and examples to guide data analysis and maintain consistency. An independent reviewer with expertise in qualitative data analysis audited the codes by coding data from a portion of a transcript independently, using the preliminary codebook developed. The independent reviewer and PI met to compare their analyses, discuss discrepancies, and come to a consensus. The codebook was refined based on the code review process and used as a guide for assigning codes to the data for the remaining transcripts. The codes were continuously reviewed and defined throughout the coding process.
Many techniques were taken to enhance trustworthiness and demonstrate rigorous methods were used to maintain the accuracy of findings as best described by the researcher and interviewees [40,42]. Efforts included maintaining an audit trail that demonstrated how data were collected, how categories were derived and how decisions were made in deriving study findings [44,46]; using a codebook to maintain consistency when coding [39,47]; using an independent reviewer to analyze and code transcripts, as well as to audit the creation of categories [42]; and sharing direct quotations from the interview data to make transferability judgments possible for the reader [42,44].
3. Results
3.1. Interviewee and organizational demographics
Table 1 provides a summary of the characteristics of the interviewees as well as the respective organizational characteristics in which interviewees worked during the time of the interview, as perceived and reported by the interviewees.
Table 1.
Interviewee and Organizational Demographics | n |
---|---|
Gender | |
Male | 8 |
Female | 7 |
Race | |
White | 9 |
Black or African American | 3 |
Asian | 2 |
Native Hawaiian or Other Pacific Islander | 0 |
American Indian or Alaska Native | 0 |
Mixed | 1 |
Ethnicity | |
Hispanic, Latino or Spanish origin | 2 |
Not Hispanic, Latino or Spanish origin | 13 |
Type of Organization Where Interviewee Worked | |
Dedicated research institution | 8 |
Academic medical center | 2 |
Community physician/medical practice | 3 |
Public hospital | 2 |
Current Role Within Organization | |
Ownera | 3 |
Principal Investigatora | 3 |
Research/Recruitment Lead | 10 |
Executive Director | 1 |
Funding Source for Trials Interviewees Were Involved Withb | |
Biopharmaceutical Sponsors | 15 |
NIH | 3 |
Racially/Ethnically Diverse Staff Roles on Interviewee's Research Teamb,c,d | |
Principal Investigator | 10 |
Sub-Investigator | 10 |
Recruiter | 14 |
Study Coordinator | 11 |
Study Nurse | 9 |
Diverse Staff not a part of the research team | 1 |
Demographics of Clinical Trial Participants Recruited – Reported by Interviewees | |
10% or less racially or ethnically diversed | 3 |
11–39% racially or ethnically diversed | 4 |
40–69% racially or ethnically diversed | 3 |
70–100% racially or ethnically diversed | 4 |
Extent to Which the Diversity of Clinical Trial Participants Reported by Interviewees Reflect Demographics in Surrounding Communityd | |
Less diverse participation than surrounding community demographics | 4 |
Same diverse participation as surrounding community demographics | 6 |
More diverse participation than surrounding community demographics | 4 |
Note. Due to missing responses, n values may not sum to 15.
Some interviewees indicated they are both an owner and Principal Investigator, thus n values exceed 15.
Due to multiple responses per participant, n values may exceed 15.
Recruiter, Study Coordinator and Study Nurse are the same role in some organizations.
Individuals from racial or ethnic groups including Black or African American, Asian, Native Hawaiian or Other Pacific Islander, American Indian or Alaska Native, Hispanic or Latino.
Of the interviewees, 8 were male and 7 were female. Interviewees were also asked if they considered themselves to be of Hispanic ethnicity, as well as which racial designation best describes them. Interviewees were asked to respond to both the race and ethnicity questions. Eight of the fifteen interviewees self-identified as racially or ethnically diverse.
Over half of the interviewees (eight) worked for dedicated research institutions, three worked for a community physician/medical practice, two worked for an academic medical center and two worked for a public hospital. The majority of interviewees (10 out of 15) were responsible for overseeing recruitment activities within their organizations. Three interviewees owned and opened the organization specifically to improve access to clinical trials to diverse patients, and two also served as PIs. Three interviewees reported receiving NIH funding in addition to biopharmaceutical funding for the trials they conduct. Interviewees were asked about five study staff roles and whether or not individuals in those roles in their respective study teams were racially or ethnically diverse. The majority (14) reported having diverse staff as part of their research teams. Ten interviewees reported having diverse PIs and sub-Investigators and 14 reported having diverse recruiters on their research teams. Unlike most of the interviewees, one interviewee responded that diverse staff were not part of the research team but were part of their regular practice.
Seven interviewees reported that the percentage of diverse participants in trials conducted by their organizations was 40% or above. While six interviewees reported that the diversity of their clinical trial participants reflected the demographics of the surrounding community, four interviewees reported recruiting more diverse trial participants than their surrounding community demographics.
3.2. Themes: organizational elements involved with active inclusive recruitment practices
Five overarching themes explaining organizational elements involved with active inclusive recruitment practices were constructed. Organizational elements include characteristics such as policies, culture, processes, structure, and mission. In this study, organizational elements were bound by the research sites where interviewees worked.
3.2.1. Organizational commitment to inclusive recruitment related to providing community education
A prominent theme that emerged is organizational commitment to inclusive recruitment related to providing community education. Interviewees discussed the critical importance of providing disease and general clinical trial education to diverse communities prior to presenting a specific trial opportunity (quote 1, Table 2). Education was discussed primarily in the form of community events like health fairs or partnering with community organizations such as churches and local advocacy chapters. Interviewees discussed having research staff educate communities about specific diseases and options for managing those diseases, including clinical trials as a potential option. Education was also provided on clinical trials in general to familiarize the community with clinical research. Interviewees discussed the need for providing culturally appropriate community education (quote 2). For example, educational content needs to be delivered in the local language and tailored to cultural preferences.
Table 2.
Theme | Example quotes |
---|---|
Organizational commitment to inclusive recruitment related to providing community education | 1. “So it's educating first. That then is built on, once we have that established, then we build on discussing recruitment into trials for specific … trials.” (Female, Mixed Race, Principal Investigator) |
2. “… it's not a one hit wonder what we've started with … certain communities … it's integrated now in our regular outreach to the community. We have to catch up with the minority groups so our education in regards of clinical research is mainly focused on the minority groups and this shows for example even in our media activities so we created uh um programs in Spanish in our social media … for example and all of this is in Spanish simply to … communicate and engage with the minority.” (Female, non-Hispanic White, Research/Recruitment Lead) | |
3. “You recruit patients for a trial, this … particular demographic by first educating them. By offering them something valuable that they can use without the pressure … or … the need to even have to discuss a study … when you do this in in an effective manner, it means that you have to put up a lot of resources and quite honestly, many businesses are not willing to invest over the long term and this is where they go wrong … when you're dealing with a certain demographic you cannot be there for that trial for just that study, your investment has to go beyond that, beyond the study number one and then before the study even starts … so you can't look at numbers on a spreadsheet to determine … that piece of advertisement worked – it doesn't work that way.” (Male, Hispanic, Owner) | |
Organizational commitment to inclusive recruitment related to organizational structure | 4. “… we do make an effort to have people um who … have comfort and knowledge uh of the communities we're trying to … recruit from … you want to have people in your organization who look like the people you're trying to recruit right and who sound like the people you're trying to recruit and who grew up in the neighborhoods …” (Male, non-Hispanic White, Research/Recruitment Lead) |
5. “… the full infrastructure that's needed it's not just recruitment I've gotta have … a doctor to do physical exams who um can speak Spanish … a Study Coordinator who uh can explain all of the procedures and … who speak Spanish if that's the only language … I've gotta have a full-time person who can … do that, multiple full-time people who can serve that whenever we need it. And that's … hard.” (Male, non-Hispanic White, Research/Recruitment Lead) | |
6. “… we made the investment of opening up our clinic on Saturday, which was a hard thing for us to do because you know from a business perspective it's an expensive thing to do Saturdays. You know additional staffing, physician time, you know you have limitations in terms of how … late you can see patients because of courier issues, I mean there's just a multitude of things and so it requires an up-front investment and a continued sustained approach …” (Male, Hispanic, Owner) | |
Strong sense of mission related to improving healthcare through clinical research | 7. “… today's research studies are tomorrow's standard of care procedures.” (Female, non-Hispanic White, Research/Recruitment Lead) |
8. “… excellence in clinical research accelerating access to treatments that change lives …” (Female, non-Hispanic White, Research/Recruitment Lead) | |
9. "… the actual mission of trying to get clinical research to the community … over the last 15 years we've that's what we've been doing. We've been enrolling people from the community setting, I've been training up uh individuals who are often medical assistants uh to become coordinators and they go on to become certified coordinators and learn the industry. And we've been involving our community-based physicians in research …" (Female, non-Hispanic White, Owner and Principal Investigator) | |
Instilling a culture of inclusion | 10. “… of course like all my staff they would love to um encourage … a minority um group to participate in a trial … we always like I said sit together to figure out … what's the best way to uh reach out to them.” (Male, Asian, Research/Recruitment Lead) |
11. “… this was the reason why the founder … of our site 12 years ago um even founded the clinical research part he's still an active endocrinologist and realized back then already that the healthcare system itself is not enough to help everybody because coverage is not always a given … this could be um a good benefit that clinical research has to offer …” (Female, non-Hispanic White, Research/Recruitment Lead) | |
Inclusive recruitment practices evolving based on learning | 12. “… we're continually evolving and uh trying to get understanding of what would help more …” (Male, Black, Owner and Principal Investigator) |
13. “[We] do not advertise for specific trials because we have found that when we do that uh our population or the patients that are coming to us don't um buy the name of the trial … they show up and say “I'm not in the study, I'm not doing this, I'm not doing that” … that's why we … try to keep our recruitment efforts mainly from our doctors' database and doctor to doctor referral.” (Male, Hispanic, Research/Recruitment Lead) |
Many interviewees emphasized the importance of having a continued presence in the community, which requires a long-term investment (quote 3). One recruitment lead mentioned that they have been giving talks in the Black community for several years and it sill hasn't translated to participation. While it was recognized that awareness needs to be generated in diverse communities prior to engaging with community members about a specific study opportunity, this requires a financial investment. Some interviewees shared additional community educational initiatives they wished they could implement, but funding was a barrier.
3.2.2. Organizational commitment to inclusive recruitment related to organizational structure
The second theme is organizational commitment to inclusive recruitment related to organizational structure, which needs to be conducive and tailored for recruiting diverse participants. This includes hiring diverse staff from the local community, ensuring staff members speak the same language of the patients that are being recruited and adjusting hours of operation (quote 4).
While interviewees discussed their organization's commitment to creating an organizational structure in order to be better at recruiting diverse participants, they also recognized room for improvement. Some interviewees discussed funding constraints for hiring staff members that speak additional languages in order to explain study procedures or perform cognitive testing in their patient's native language. One recruitment lead discussed the need for a full research infrastructure in order to offer a trial in another language, which includes hiring several full-time people such as a physician, recruiter and Study Coordinator who can all speak that particular language (quote 5). Interviewees discussed their organization's commitment towards overcoming barriers to making structural changes just as often as discussing barriers that prevent structural enhancements (quote 6).
3.2.3. Strong sense of mission related to improving healthcare through clinical research
Interviewees reported working for organizations with missions related to improving healthcare through clinical research. Approximately half of the interviewees spoke about their organization's mission of conducting research in relation to improving healthcare for overall society by developing medications that will be the future of care and prevention (quotes 7 and 8). Other interviewees discussed their organization's mission of conducting research in relation to improving health outcomes for patients by facilitating access to treatment options available in clinical trials to those who are underrepresented in the healthcare system. Interviewees described a strong sense of mission in that their organizations' missions influenced work across the organization and guide recruitment efforts. For example, one Owner/Principal Investigator discussed efforts to fulfill their organizational mission of bringing research to the community such as onboarding community-based staff and training community-based physicians to be Investigators (quote 9).
3.2.4. Instilling a culture of inclusion
Interviewees discussed instilling a culture of inclusion with regard to inclusivity in clinical research. Interviewees mentioned different aspects of their organizational cultures, including working among a diverse study team and understanding each other's cultural backgrounds, as well as efforts to ensure trial information reaches and resonates with diverse populations (quote 10). Some interviewees discussed having a culture of inclusion derived from leadership support or directive. This included imperatives for promoting equitable access to clinical research (quote 11).
3.2.5. Inclusive recruitment practices evolving based on learning
Interviewees discussed how their organization's approach to inclusive recruitment practices evolves, sharing changes initiated based on past experiences and lessons learned (quote 12). One recruitment lead discussed reprioritizing their efforts after having failed attempts to recruit diverse participants into a trial - later learning that community engagement was required in order for them to be successful. A Principal Investigator also shared learnings related to needing a wider net for recruitment and that meant going out into the community. Another recruitment lead mentioned that general advertising did not work for the diverse population his organization recruited so they stopped advertising for specific trials (quote 13).
The voice of the interviewees is incorporated in Table 2 and delineated by each theme.
4. Discussion
4.1. Organizational commitment to inclusive recruitment related to providing community education
One of the key elements to successfully recruiting diverse participants is providing disease and general clinical trial education prior to presenting a specific trial opportunity [[48], [49], [50]]. Despite being recognized as important, such action is not being taken nor is providing community education a common practice at the site level [50,51]. The literature suggests that research sites engaged with educating their local communities are providing education at the same time as trial recruitment, even when receiving NIH funding or resources [[51], [52], [53], [54], [55], [56]]. With exception are federally designated specialty centers that provide ongoing community education in partial fulfillment for maintaining designation status [48,49,57,58]. The success and sustainability of long-term community engagement is yet to be documented among industry-funded studies, although industry stakeholders acknowledged that community education prior to focusing on study recruitment is not occurring [50].
The NIH oftentimes provides funding for research infrastructure at sites including providing strategic guidance or funding for hiring and training diverse staff to support recruitment activities [9,52,53]. Pharmaceutical sponsors provide recruitment support to sites on a study-by-study basis, leaving it up to research sites to establish a long-term commitment to community education, should they wish. Individuals in this study discussed how their organizations made the commitment to invest in providing general education prior to discussing a study. This concept is not widely practiced and is one area in which the interviewees displayed an exceptional practice unique to many research sites.
4.2. Organizational commitment to inclusive recruitment related to organizational structure
Individuals in this study also discussed how their organization demonstrated its commitment to inclusive recruitment by investing in structural enhancements. Research center infrastructure has been identified as a barrier to diverse recruitment [36]. Even with infrastructure investments and leadership supportive of research initiatives within their organizations, interviewees discussed continuing barriers. Many expressed the need for additional research staff members and language capabilities, which are known limiting factors for inclusive recruitment [5,36]. Interviewees also shared examples of how their organizations overcame barriers to improving their research infrastructures and enhanced their capabilities to better meet the needs of diverse patients. The majority of organizations represented by interviewees in this study have diverse staff on their research teams. Ensuring research staff are the same race or ethnicity and speak the same language as patients being recruited has contributed to achieving diverse representation in past studies [21,53,55,59]. In fact, site personnel race and ethnicity was found to be correlated with the diversity of patients enrolled in clinical trials [60].
4.3. Strong sense of mission related to improving healthcare through clinical research
Findings from this study suggest that incorporating clinical research components and promoting access to clinical research within a research site's mission statement help facilitate active inclusive recruitment efforts. While not widely described in the literature, Joseph and Dohan (2009) found research activities at a safety-net hospital were insufficiently supported due in part to the organization's primary mission not encompassing clinical research [36]. Individuals in this study discussed recruitment activities driven by their mission. Believing in the mission was an important aspect leading to principles of working.
4.4. Instilling a culture of inclusion
Regnante et al. (2019) reported that cancer research centers recruiting a high percentage of diverse trial participants demonstrated an inclusive organizational culture by dedicating leadership roles to diversity and hiring diverse staff [49]. In addition to leadership support deriving a culture of inclusion, co-workers of interviewees in this study helped create an inclusive culture prevalent across their organizations.
4.5. Inclusive recruitment practices evolving based on learning
The participating organizations also had flexible processes. Recruitment approaches changed based on past experiences, as a better understanding of how to adapt to patients' cultural preferences and needs was gained. Crosson et al. (1999) discussed how organizational learning provides a competitive advantage [61]. Individuals who participated in this study discussed enhancements made based on organizational learning that improved their organizations’ competencies and allowed them to improve recruitment practices.
5. Conclusions
In this study, broad ranges of perspectives were obtained from research stakeholders who provided their first-hand experiences of organizational elements involved with inclusive recruitment practices. The peer-reviewed literature on inclusive recruitment practices in clinical trials is in the context of NIH-funded trials and these findings offer a new contribution related to industry-funded trials.
The reported enrollment of 40% or more diverse participants in trials conducted by seven organizations was unique among industry-funded research sites in that recruitment was on par with diverse representation in the US population, which was 39% at the time of data collection [62]. At these organizations, recruitment was more equitable than many organizations conducting industry-funded trials as overall, 17% of participants in industry-sponsored studies are racially or ethnically diverse [8]. Four individuals reported recruiting more diverse trial participants on average than nearby community demographics, another indicator that organizational characteristics may play a role in achieving clinical trial diversity.
Due to the qualitative nature of this study, there are inherent limitations. The qualitative method selected for this study was designed to be exploratory and therefore limiting transferability of findings. Thick description was provided through interviewee voice for readers to make transferability judgments to their own unique settings. The convenience sample created potential for self-selection bias. It is important to note that data were collected prior to the COVID-19 pandemic. The authors also want to note that while they purposefully attempted to use inclusive language in this article, at times the word ‘minority’ was used to portray interviewee voices. From a social justice perspective, culturally sensitive language related to equity, power, and identity, among others, is recommended [63]. The use of the word ‘minority’ is commonly accepted in scientific fields [64,65].
The findings from this study provide a foundation for exploring environmental elements that may be involved with inclusive recruitment. Understanding research site characteristics facilitated an understanding of what elements are involved in the active recruitment of diverse participants into industry-funded clinical trials at an organizational level. The interview findings may help to inform the application of practices that promote clinical trial diversity. Findings suggest the following organizational characteristics be exhibited:
-
1.
Committing to a long-term community engagement plan by providing culturally tailored education
-
2.
Establishing an organizational structure to eliminate barriers to success through practices such as hiring community-based research staff and tailoring language capabilities to community needs
-
3.
Defining a strong sense of mission that incorporates diversity in clinical research and establishing principles of working that support the mission
-
4.
Instilling a culture of inclusion across all hierarchal levels of the organization
-
5.
Flexibility to continually refine inclusive recruitment practices based on learning
By applying similar inclusive recruitment practices, industry stakeholders involved with clinical research can better support organizational initiatives focused on improving access to biopharmaceutical-sponsored clinical trials for racially and ethnically diverse participants. Improving clinical trial diversity requires a multi-stakeholder effort across the broader clinical trial ecosystem, of which research sites represent one component. Applying the initiatives discussed at a research site level are among many required across a multitude of stakeholders in order to achieve sustainable progress with improving access and inclusivity in clinical research.
Declaration of competing interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: The authors received no specific funding for this work. The corresponding author has one competing interest to disclose: place of employment during the time of data collection was IQVIA. The other authors have no conflicts of interest to disclose.
References
- 1.Ndugga N., Artiga S. KFF; 2021 May 11. Disparities in Health and Health Care: 5 Key Questions and Answers.https://www.kff.org/racial-equity-and-health-policy/issue-brief/disparities-in-health-and-health-care-five-key-questions-and-answers/ [Internet] [cited 2021 Jun 19] [Google Scholar]
- 2.United Health Foundation . 2021. America's Health Rankings.https://www.americashealthrankings.org/learn/reports/2021-disparities-report [Internet] disparities report; 2021 [cited 2022 Apr 1] [Google Scholar]
- 3.Artiga S., Corallo B., Pham O. KFF; 2020 Aug 17. Racial Disparities in COVID-19: Key Findings from Available Data and Analysis.https://www.kff.org/racial-equity-and-health-policy/issue-brief/racial-disparities-covid-19-key-findings-available-data-analysis/ [Internet] [cited 2021 Feb 13] [Google Scholar]
- 4.Boden-Albala B., Carman H., Southwick L., Parikh N.S., Roberts E., Waddy S., Edwards D. Examining barriers and practices to recruitment and retention in stroke clinical trials. Stroke. 2015 Aug;46(8):2232–2237. doi: 10.1161/strokeaha.114.008564. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Durant R.W., Wenzel J.A., Scarinci I.C., Paterniti D.A., Fouad M.N., Hurd T.C., Martin M.Y. Perspectives on barriers and facilitators to minority recruitment for clinical trials among cancer center leaders, investigators, research staff, and referring clinicians: enhancing minority participation in clinical trials (EMPaCT) Cancer. 2014 doi: 10.1002/cncr.28574. [Internet] Mar 18 [cited 2018 Sept 3];120:1097-105. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Eggly S., Barton E., Winckles A., Penner L.A., Albrecht T.L. A disparity of words: racial differences in oncologist-patient communication about clinical trials. Health Expect. 2013 Aug 2;18(5):1316–1326. doi: 10.1111/hex.12108. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Artiga S., Kates J., Michaud J., Hill L. KFF; 2021 Jan 26. Racial Diversity within COVID-19 Vaccine Clinical Trials: Key Questions and Answers.https://www.kff.org/racial-equity-and-health-policy/issue-brief/racial-diversity-within-covid-19-vaccine-clinical-trials-key-questions-and-answers/ [Internet] [cited 2021 Jun 19] [Google Scholar]
- 8.Center for Information & Study on Clinical Research Participation (CISCRP) Proportion of study volunteers by race and ethnicity in clinical research studies. J. Natl. Cancer Inst. 2017;109(4):7. doi: 10.1093/jnci/djx077. [DOI] [PubMed] [Google Scholar]
- 9.Duda C., Mahon I., Chen M.H., Snyder B., Barr R., Chiles C., Falk R., Fishman E.K., Gemmel D., Goldin J.G., Brown K., Munden R.F., Vydareny K., Aberle D.R. Impact and costs of targeted recruitment of minorities to the national lung screening trial. Clin. Trials: J. Soc. Clin. Trial. 2011 Jan 17;8(2):214–223. doi: 10.1177/1740774510396742. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Food and Drug Administration. U.S. Department of Health and Human Services . 2014 Aug. FDA Report: FDA Action Plan to Enhance the Collection and Availability of Demographic Subgroup Data.https://www.fda.gov/media/89307/download [Internet] [cited 2019 Mar 1] [Google Scholar]
- 11.Food and Drug Administration . U.S. Department of Health and Human Services; 2020 Nov. Enhancing the Diversity of Clinical Trial Populations – Eligibility Criteria, Enrollment Practices, and Trial Designs Guidance for Industry.https://www.fda.gov/media/127712/download [Internet] [cited 2021 Mar 1] [Google Scholar]
- 12.Carson P., Ziesche S., Johnson G., Cohn J.N. Racial differences in response to therapy for heart failure: analysis of the vasodilator-heart failure trials. J. Card. Fail. 1999 Sep;5(3):178–187. doi: 10.1016/s1071-9164(99)90001-5. [Internet] [cited 2018 Mar 1] [DOI] [PubMed] [Google Scholar]
- 13.Chen M.S., Lara P.N., Dang J.H., Paterniti D.A., Kelly K. Twenty years post-NIH Revitalization Act: enhancing minority participation in clinical trials (EMPaCT): laying the groundwork for improving minority clinical trial accrual. Cancer. 2014 Mar 18 doi: 10.1002/cncr.28575. [Internet] [cited 2018 Mar 1];120:1091-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Cohn J. Clinical experience with perindopril in African-American hypertensive patients: a large United States community trial. Am. J. Hypertens. 2004 Feb;17(2):134–138. doi: 10.1016/j.amjhyper.2003.09.017. [Internet] [cited 2018 Mar 1] [DOI] [PubMed] [Google Scholar]
- 15.Hershman D., Weinberg M., Rosner Z., Alexis K., Tiersten A., Grann V.R., Troxel A., Neugut A.I. Ethnic neutropenia and treatment delay in African American women undergoing chemotherapy for early-stage breast cancer. JNCI J. Nat. Cancer Inst. 2003 Oct 14;95(20):1545–1548. doi: 10.1093/jnci/djg073. [Internet] [cited 2018 Mar 1] [DOI] [PubMed] [Google Scholar]
- 16.Paez J.G. EGFR mutations in lung cancer: correlation with clinical response to gefitinib therapy. Science. 2004 Jun 4;304(5676):1497–1500. doi: 10.1126/science.1099314. [Internet] [cited 2018 Mar 1] [DOI] [PubMed] [Google Scholar]
- 17.Ramamoorthy A., Pacanowski M., Bull J., Zhang L. Racial/ethnic differences in drug disposition and response: review of recently approved drugs. Clin. Pharmacol. Ther. 2015 Jan 20;’(3):263–273. doi: 10.1002/cpt.61. [Internet] [cited 2018 Mar 1] [DOI] [PubMed] [Google Scholar]
- 18.National Institutes of Health . U.S. Department of Health and Human Services; 2017 Dec 6. NIH Policy and Guidelines on the Inclusion of Women and Minorities as Subjects in Clinical Research.http://grants.nih.gov/grants/funding/women_min/guidelines_amended_10_2001.htm [Internet] [cited 2021 Feb 13] [Google Scholar]
- 19.Food and Drug Administration . U.S. Department of Health and Human Services; 2016 Oct 26. Collection of Race and Ethnicity Data in Clinical Trials: Guidance for Industry and Food and Drug Administration Staff.https://www.fda.gov/media/75453/download [Internet] [cited 2019 Mar 1] [Google Scholar]
- 20.Ejiogu N., Norbeck J.H., Mason M.A., Cromwell B.C., Zonderman A.B., Evans M.K. Recruitment and retention strategies for minority or poor clinical research participants: lessons from the healthy aging in neighborhoods of diversity across the life span study. Gerontol. 2011 May 11;51(Supplement 1):S33–S45. doi: 10.1093/geront/gnr027. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Ford M.E., Siminoff L.A., Pickelsimer E., Mainous A.G., Smith D.W., Diaz V.A., Soderstrom L.H., Jefferson M.S., Tilley B.C. Unequal burden of disease, unequal participation in clinical trials: solutions from African American and Latino community members. Health Soc. Work. 2013 Feb 1;38(1):29–38. doi: 10.1093/hsw/hlt001. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Williams I.C., Corbie-Smith G. Investigator beliefs and reported success in recruiting minority participants. Contemp. Clin. Trials. 2006 Dec;27(6):580–586. doi: 10.1016/j.cct.2006.05.006. [Internet] [cited 2018 Mar 1] [DOI] [PubMed] [Google Scholar]
- 23.Diehl K.M., Green E.M., Weinberg A., Frederick W.A., Holmes D.R., Green B., Morris A., Kuerer H.M., Beltran R.A., Mendez J., Gines V., Ota D.M., Nelson H., Newman L.A. Features associated with successful recruitment of diverse patients onto cancer clinical trials: report from the American College of Surgeons Oncology Group. Ann. Surg Oncol. 2011 Jun 17;18(13):3544–3550. doi: 10.1245/s10434-011-1818-9. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Markman M., Petersen J., Montgomery R. An examination of the influence of patient race and ethnicity on expressed interest in learning about cancer clinical trials. J. Cancer Res. Clin. Oncol. 2007 Jun 28;134(1):115–118. doi: 10.1007/s00432-007-0263-4. [Internet] [cited 2018 Mar 1] [DOI] [PubMed] [Google Scholar]
- 25.Wendler D., Kington R., Madans J., Wye G.V., Christ-Schmidt H., Pratt L.A., Brawley O.W., Gross C.P., Emanuel E. Are racial and ethnic minorities less willing to participate in health research? PLoS Med. 2005 Dec 6 doi: 10.1371/journal.pmed.0030019. [Internet] [cited 2018 Mar 1];3(2):e19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Aleccia J. 2020 Jul 28. The Color of COVID: Will Vaccine Trials Reflect America's Diversity?https://nmpoliticalreport.com/2020/07/28/the-color-of-covid-will-vaccine-trials-reflect-americas-diversity/ The NM Political Report [Internet] [cited 2021 Feb 13] [Google Scholar]
- 27.Cohen E., Vigue D. CNNhealth; 2020 Aug 16. Covid-19 Vaccine Trials Have Been Slow to Recruit Black and Latino People -- and that Could Delay a Vaccine.https://www.cnn.com/2020/08/16/health/covid-19-vaccine-trial-black-minority-recruitment/index.html [Internet] [cited 2021 Feb 13] [Google Scholar]
- 28.Johnson C.Y. Washington Post; 2020 Aug 27. Large U.S. Covid-19 Vaccine Trials Are Halfway Enrolled, but Lag on Participant Diversity.https://www.washingtonpost.com/health/2020/08/27/large-us-covid-19-vaccine-trials-are-halfway-enrolled-lag-participant-diversity/ [Internet] [cited 2021 Feb 13] [Google Scholar]
- 29.Garijo B. 2020 Oct 9. We Must Use This Moment to Improve Diversity in Clinical Trials — and Not Just for Vaccines.https://time.com/5896416/vaccine-covid-diversity-clinical-trials-merck/ Time [Internet] [cited 2021 Feb 13] [Google Scholar]
- 30.Highsmith Q., Freedman J. Genentech; 2020 May 22. A Call for More Inclusive COVID-19 Research.https://www.gene.com/stories/a-call-for-more-inclusive-covid-19-research [Internet] [cited 2021 Feb 13] [Google Scholar]
- 31.Rottas M., Thadeio P., Simons R., Houck R., Gruben D., Keller D., Scholfield D., Soma K., Corrigan B., Schettino A., McCann P.J., Hellio M.P., Natarajan K., Goodwin R., Sewards J., Honig P., MacKenzie R. Demographic diversity of participants in Pfizer sponsored clinical trials in the United States. Contemp. Clin. Trials. 2021 Jul doi: 10.1016/j.cct.2021.106421. [Internet] [cited 2022 Feb 2];106:106421. [DOI] [PubMed] [Google Scholar]
- 32.Centers for Disease Control and Prevention . U.S. Department of Health and Human Services; 2021 Feb 12. COVID-19 Hospitalization and Death by Race/ethnicity.https://www.cdc.gov/coronavirus/2019-ncov/covid-data/investigations-discovery/hospitalization-death-by-race-ethnicity.html [Internet] [cited 2021 Feb 13] [Google Scholar]
- 33.The Janssen Pharmaceutical Companies of Johnson & Johnson . Fierce Pharma; 2022 Jan 3. Janssen takes multifaceted approach to ensuring diversity, equity, and inclusion in its COVID-19 vaccine trial.https://www.fiercepharma.com/sponsored/janssen-takes-multifaceted-approach-to-ensuring-diversity-equity-and-inclusion-its-covid [Internet] [cited 2022 Apr 1] [Google Scholar]
- 34.Tirrell M., Miller L. CNBC; 2020 Sep 4. Moderna Slows Coronavirus Vaccine Trial Enrollment to Ensure Minority Representation, CEO Says.https://www.cnbc.com/2020/09/04/moderna-slows-coronavirus-vaccine-trial-t-to-ensure-minority-representation-ceo-says.html [Internet] [cited 2021 Feb 13] [Google Scholar]
- 35.Wright J.T., Cushman W.C., Davis B.R., Barzilay J., Colon P., Egan D., Lucente T., Nwachuku C., Pressel S., Leenen F.H., Frolkis J., Letterer R., Walsh S., Tobin J.N., Deger G.E. The antihypertensive and lipid-lowering treatment to prevent heart attack trial (ALLHAT) Contr. Clin. Trials. 2001 Dec;22(6):659–673. doi: 10.1016/s0197-2456(01)00176-3. [Internet] [cited 2017 May 4] [DOI] [PubMed] [Google Scholar]
- 36.Joseph G., Dohan D. Recruiting minorities where they receive care: institutional barriers to cancer clinical trials recruitment in a safety-net hospital. Contemp. Clin. Trials. 2009 Nov;30(6):552–559. doi: 10.1016/j.cct.2009.06.009. [Internet] [cited 2018 Mar 1] [DOI] [PubMed] [Google Scholar]
- 37.Bronfenbrenner U. Toward an experimental ecology of human development. Am. Psychol. 1977;32(7):513–531. doi: 10.1037/0003-066x.32.7.513. [Internet] [cited 2017 Jan 15] [DOI] [Google Scholar]
- 38.McLeroy K.R., Bibeau D., Steckler A., Glanz K. An ecological perspective on health promotion programs. Health Educ. Q. 1988 Dec;15(4):351–377. doi: 10.1177/109019818801500401. [Internet] [cited 2017 Jan 15] [DOI] [PubMed] [Google Scholar]
- 39.Creswell J.W. Research Design: Qualitative, Quantitative and Mixed Methods Approaches. fourth ed. SAGE Publications, Inc; Thousand Oaks, CA: 2014. p. 273. [Google Scholar]
- 40.Patton M.Q. Qualitative Research & Evaluation Methods. third ed. Sage; Thousand Oaks, CA: 2002. [Google Scholar]
- 41.Portney L.G., Watkins M. Foundations of Clinical Research: Applications to Practice. second ed. Prentice Hall; Upper Saddle River, NJ: 2000. p. 742. [Google Scholar]
- 42.Creswell J.W. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. third ed. SAGE Publications; Thousand Oaks, CA: 2013. [Google Scholar]
- 43.Guest G., Bunce A., Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2013;18(1):59–82. [Google Scholar]
- 44.Lincoln Y.S., Guba E. Sage Publications; Beverly Hills, CA: 1985. Naturalistic Inquiry; p. 416. [Google Scholar]
- 45.Saldana J. SAGE Publications; Thousand Oaks, CA: 2009. The Coding Manual for Qualitative Researchers. [Google Scholar]
- 46.Merriam S.B. Qualitative Research and Case Study Applications in Education. second ed. Jossey-Bass Publishers; San Francisco, CA: 1998. p. 275. [Google Scholar]
- 47.Miles M.B., Huberman M. Qualitative Data Analysis: an Expanded Sourcebook. second ed. Sage Publications; Thousand Oaks: 1994. p. 338. [Google Scholar]
- 48.Etkin C.D., Farran C.J., Barnes L.L., Shah R.C. Recruitment and enrollment of caregivers for a lifestyle physical activity clinical trial. Res. Nurs. Health. 2011 Nov 14;35(1):70–81. doi: 10.1002/nur.20466. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Regnante J.M., Richie N.A., Fashoyin-Aje L., Vichnin M., Ford M., Roy U.B., et al. US cancer centers of excellence strategies for increased inclusion of racial and ethnic minorities in clinical trials. J. Oncol. Pract. 2019 Apr;15(4):e289–e299. doi: 10.1200/jop.18.00638. [Internet] [cited 2021 Feb 4] [DOI] [PubMed] [Google Scholar]
- 50.Society for Women’s Health Research . US Department of Health and Human Services (USHHS), Food and Drug Administration, Office of Women’s Health. U.S. Food and Drug Administration; 2011 Sep 22. Dialogues on Diversifying Clinical Trials: Successful Strategies for Engaging Women and Minorities in Clinical Trials: Final Report on the Dialogues on Diversifying Clinical Trials Conference.https://www.fda.gov/science-research/womens-health-research/dialogues-diversifying-clinical-trials [Internet] [cited 2018 Mar 1] [Google Scholar]
- 51.Livaudais-Toman J., Burke N., Napoles A., Kaplan C. Health literate organizations: are clinical trial sites equipped to recruit minority and limited health literacy patients. J. Health Dispar. Res. Pract. 2014;7(4):1–13. [PMC free article] [PubMed] [Google Scholar]
- 52.Cook C., Kosoko-Lasaki O., O'Brien R. Satisfaction with and perceived cultural competency of healthcare providers: the minority experience. J. Natl. Med. Assoc. 2005;97(8):1078–1087. [PMC free article] [PubMed] [Google Scholar]
- 53.Hinton L., Carter K., Reed B.R., Beckett L., Lara E., DeCarli C., Mungas D. Recruitment of a community-based cohort for research on diversity and risk of dementia. Alzheimer Dis. Assoc. Disord. 2010 Jul doi: 10.1097/wad.0b013e3181c1ee01. [Internet] [cited 2020 Feb 1]:1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Aponte-Rivera V., Dunlop B.W., Ramirez C., Kelley M.E., Schneider R., Blastos B., Larson J., Mercado F., Mayberg H., Craighead W.E. Enhancing Hispanic participation in mental health clinical research: development of a Spanish-speaking depression research site. Depress. Anxiety. 2013 Aug 19;31(3):258–267. doi: 10.1002/da.22153. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Germino B.B., Mishel M.H., Alexander G.R., Jenerette C., Blyler D., Baker C., Vines A.I., Green M., Long D.G. Engaging African American breast cancer survivors in an intervention trial: culture, responsiveness and community. J. Cancer Survivorship. 2010 Oct 1;5(1):82–91. doi: 10.1007/s11764-010-0150-x. [Internet] [cited 2018 Mar 2] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Sturgeon K.M., Hackley R., Fornash A., Dean L.T., Laudermilk M., Brown J.C., Sarwer D.B., DeMichele A.M., Troxel A.B., Schmitz K.H. Strategic recruitment of an ethnically diverse cohort of overweight survivors of breast cancer with lymphedema. Cancer. 2017 Sep 7;124(1):95–104. doi: 10.1002/cncr.30935. [Internet] [cited 2018 Mar 1] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.American Cancer Society . 2014 Nov. National Cancer Institute Cancer Center Program Information.https://www.cancer.org/treatment/finding-and-paying-for-treatment/choosing-your-treatment-team/nci-cancer-center-programs.html [Internet] [cited 2020 Feb 28] [Google Scholar]
- 58.National Institutes of Health. U.S . Department of Health and Human Services; 2018 Jan. Alzheimer's Disease Research Centers (Funding Opportunity Announcement RFA-AG-19-001)https://grants.nih.gov/grants/guide/rfa-files/RFA-AG-19-001.html [Internet] [cited 2020 Apr 15] [Google Scholar]
- 59.National Academies of Sciences . The National Academies Press; Washington, DC, USA: 2016. Engineering, and Medicine. Strategies for Ensuring Diversity, Inclusion, and Meaningful Participation in Clinical Trials: Proceedings of a Workshop. [PubMed] [Google Scholar]
- 60.Tufts Center for the Study of Drug Development . Tufts University; 2021 Nov. New Study Finds Site Personnel Race and Ethnicity Highly Correlated with Diversity of Patients enrolled:Impact Report.https://csdd.tufts.edu/in-the-news [Internet] [cited 2022 Feb 15] [Google Scholar]
- 61.Crossan M.M., Lane H.W., White R.E. An organizational learning framework: from intuition to institution. Acad. Manag. Rev. 1999 Jul;24(3):522–537. doi: 10.5465/amr.1999.2202135. [Internet] [cited 2017 Sept 21] [DOI] [Google Scholar]
- 62.United States Census Bureau . 2019. Quick Facts: United States.https://www.census.gov/quickfacts/table/AGE135215/00 [Internet] [cited 2020 Apr 15] [Google Scholar]
- 63.American Psychological Association . 2022. Inclusive Language Guidelines.https://www.apa.org/about/apa/equity-diversity-inclusion/language-guidelines [Internet] [cited 2022 Apr 19] [Google Scholar]
- 64.Forbes . 2021. Why It's Time for the Term ‘minority’ to Disappear in STEM Disciplines.https://www.forbes.com/sites/marshallshepherd/2021/11/03/why-its-time-for-the-term-minority-to-disappear-in-stem-disciplines/?sh=63ef94d153cd [Internet] Nov 3 [cited 2022 Sept 25]. Available from: [Google Scholar]
- 65.National Science Foundation, National Center for Science and Engineering Statistics . 2019. Women, Minorities, and Persons with Disabilities in Science and Engineering: Special Report NSF 19-304.https://www.nsf.gov/statistics/wmpd Mar 8 [cited 2022 Oct 01]. Available from: [Google Scholar]