Abstract
We present innovative research practices in psychiatric genetic studies to ensure representation of individuals from diverse ancestry, sex assigned at birth, gender identity, age, body shape and size, and socioeconomic backgrounds. Due to histories of inappropriate and harmful practices against marginalized groups in both psychiatry and genetics, people of certain identities may be hesitant to participate in research studies. Yet their participation is essential to ensure diverse representation, as it is incorrect to assume that the same genetic and environmental factors influence the risk for various psychiatric disorders across all demographic groups. We present approaches developed as part of the Eating Disorders Genetics Initiative (EDGI), a study that required tailored approaches to recruit diverse populations across many countries. Considerations include research priorities and design, recruitment and study branding, transparency, and community investment and ownership. Ensuring representation in participants is costly and funders need to provide adequate support to achieve diversity in recruitment in prime awards, not just as supplemental afterthoughts. The need for diverse samples in genetic studies is critical to minimize the risk of perpetuating health disparities in psychiatry and other health research. Although the EDGI strategies were designed specifically to attract and enroll individuals with eating disorders, our approach is broadly applicable across psychiatry and other fields.
Keywords: diversity, equity, health disparities, inclusion, recruitment
1. INTRODUCTION
Research hesitancy is real, especially for psychiatric and genetic studies. Many individuals have concerns about privacy, including fears of genetic data being shared with insurance companies (in some countries) or undocumented status being shared with immigration officials and other law enforcement agencies. These concerns are understandably exacerbated by histories of mistreatment of certain groups by the scientific community. Mistrust of health systems in general due to historical failings, gaps in care, and medical racism can also fuel reluctance to participate. Stigma about having a psychiatric disorder remains pervasive in many communities. All of these factors lead to underrepresentation of certain groups and overrepresentation of others, which threatens the completeness and accuracy of our science, which may in turn perpetuate inequities. Underrepresentation of non‐European ancestry populations in psychiatric genetics research is well documented and a major challenge for the field (Duncan et al., 2019; Peterson et al., 2019); although, the down‐stream implications on health disparities are underappreciated. Ensuring the representativeness of a sample is neither automatic nor easy and requires considerable planning, funding, and community engagement. Many funders set goals for minority recruitment but fail to provide adequate funding to do what is necessary to achieve those recruitment goals.
Failure to enroll adequately diverse individuals threatens both the completeness of our understanding of disease and can perpetuate health disparities if treatments that emerge from genetic research only reflect the science conducted on individuals who represent a majority group. For example, medications developed based on genetic research conducted solely on European‐ancestry populations may not target the biology of individuals from un‐studied groups, or the safety, efficacy, and dosing may differ in non‐European individuals. Likewise, many psychiatric disorders are presumed to be caused by both genetic and environmental factors. Identification of environmental risk factors that influence disorder emergence in, for example, a cisgender female European ancestry‐only sample may only capture a small slice of environmental risk factors and be blind to a host of other environmental factors that increase the risk of disease in other genders, biological sexes at birth, or ancestries, again resulting in an incomplete understanding of the disease.
In the United States, the National Institutes of Health Revitalization Act of 1993, which later became public law, marked an improvement towards ensuring that women and racial/ethnic minorities were included in funded clinical trials (Chen, Lara, Dang, Paterniti, & Kelly, 2014). Similar efforts to promote ethnic, racial, and cultural diversity in clinical trials are in place in Australia (Australian Government, 2007) and Europe (Witham et al., 2020). In New Zealand, it is a constitutional requirement that the indigenous Māori population be afforded equitable health outcomes (Durie, 1989), although there are still major disparities in health outcomes in that country.
Despite these laws and mandates, many communities remain underrepresented in behavioral, biological, and genetic research (Chen et al., 2014). For example, it is well known that Latin American populations are characterized by a rich genetic admixture of Native American, European, and African ancestries reflecting complex and diverse social, immigration, and colonial histories. Participants from Latin America have been underrepresented in DNA‐based research historically due to the complexities associated with analyzing data from highly admixed samples (Bryc et al., 2010; De Ver Dye et al., 2021). Novel analytic methods are poised to eliminate this practice. Moreover, researchers from low and middle income countries (LMICs) also have fewer resources to conduct large‐scale genetic studies, which contributes to lower global representation. Achieving federally mandated enrollment targets for “minority recruitment” may allow checking of a box, but does not ensure adequate statistical power to derive meaningful scientific conclusions about underrepresented groups.
Enrolling individuals from diverse backgrounds in research is not a given. In fact, if participation hesitancy and systemic mistrust are not addressed directly, enrollment will default to European‐ancestry overrepresentation. Although it is possible to use known epidemiology as a starting point in setting enrollment goals, the under‐detection, under‐referral, and under‐treatment of underrepresented groups can skew both epidemiological and clinical estimates of prevalence and incidence (Konkel, 2015; Kurt et al., 2016; Sharma & Palaniappan, 2021). Moreover, broadening recruitment is not simply a matter of translating materials into different languages, but rather necessitates the inclusion of personnel who identify with the underrepresented groups and the development of culturally sensitive and culturally adapted approaches in order to gain acceptance and ownership by underrepresented communities (Occa, Morgan, & Potter, 2018). Cultivating research that approaches values, culture, circumstances, and priorities of diverse populations may increase participation, and eventually, potentially impact the health disparities affecting these communities.
Many strategies and approaches are consistent with a co‐design and production framework, which has become research best practice in many countries around the world (InsideOut Institute, 2021). Co‐design and production go beyond consultation to encompass the formation of authentic partnerships for the full research life cycle from inception to dissemination with all stakeholders, subjects of, participants in, and recipients of the outcomes of research. These include consumers, carers, clinicians, and healthcare providers. Co‐design includes their knowledge and experience in the design and execution of research ensuring engagement from the outset, transparency, and equity of knowledge and responsibility. The approach also has the benefit of bringing the research participants into the process, forming key relationships, building trust, dialogue, and valuable communication channels that aid in representative recruitment and meaningful results. Funders need to incorporate co‐design into rating metrics and financially support the time and effort it takes to develop, test, and deploy truly co‐designed research studies.
We use our experience with the Eating Disorders Genetics Initiative (EDGI) research study as an exemplar. The lack of diversity in all eating disorder research (i.e., not just genetics) is pronounced and is exacerbated by long‐standing stereotypes related to who is at risk for developing eating disorders (Cheng, Perko, Fuller‐Marashi, Gau, & Stice, 2019; Lester & Petrie, 1998; Marques et al., 2011; Wilfley, Pike, Dohm, Striegel‐Moore, & Fairburn, 2001). Historically, most of the literature examining etiological, biological, and genetic risk factors contributing to the development of eating disorders has been—and to some extent continues to be—conducted on White, cisgender, female participants (e.g., Bruch, 1978; Graber, Brooks‐Gunn, Paikoff, & Warren, 1994; Miller & Pumariega, 2001; Racine et al., 2017; Striegel‐Moore & Bulik, 2007). Although substantial advances in eating disorder genetics research have occurred (e.g., anorexia nervosa), most of the genome‐wide studies have predominantly examined female samples of European ancestry (Boraska et al., 2014; Duncan et al., 2017; Wang et al., 2011; Watson et al., 2019), with some inclusion of Asian populations (e.g., Boraska et al., 2014; Bulik, 2020; Nakabayashi et al., 2009). Our intention is to present our experiences to assist other researchers in developing and tailoring their enrollment approaches to the specific needs of their patient group.
2. METHODS
In this section, we outline a series of approaches to ensure representation of individuals from underrepresented communities in psychiatric genetic studies. The approaches outlined emerge from existing literature, consultation with university core services, trial and error associated with the EDGI study, and input from research team members. The most important thread through all of these suggestions is availability and transparency. If the intention is to recruit individuals for a study via community outreach (and even if the study employs clinic‐based recruitment), being responsive and transparent to requests for information, interviews, or other content is central to success. Having more than one spokesperson or advocate for the study—and preferably individuals from diverse backgrounds—will improve visibility and engagement in research. Believing “if you build it, they will come” is destined to fail. Recruitment of a truly representative sample is difficult and requires time, adequate funding, planning, patience, and consistent attention.
2.1. Research design
2.1.1. Creating a movement; creating an inclusive community
Breaking down traditional barriers of "us" (researchers/clinicians) and "them" (patients and family members) is a fundamental step that can aid in engaging affected individuals in research. Being purposeful in building a diverse leadership team and including patients and families in genetic studies early in the process, for example, research priority setting or study planning stages, can ensure stakeholder engagement before even beginning to recruit participants (Tong et al., 2019). We use the terms “patient” and “participant” to refer to individuals with personal lived experience of the disorder under study. This does not necessarily imply that they have ever been formally diagnosed or received treatment for the illness, or that they are currently ill. Many principles reviewed here also apply to the recruitment of controls; although tailored approaches are often necessary to recruit individuals who have no vested interest in the phenotype under study.
2.1.2. Actively engaging stakeholders and communities
Engaging stakeholders (patients, caregivers, health care providers, community members, advocates, activists, and other non‐researcher stakeholders) is challenging but essential to ensure that the sample of participants you enroll truly represents how your particular phenotype or disorder exists in the world. Several approaches exist to engage the broader community and can be tailored to the needs and preferences of different groups.
Neutral stakeholder or community engagement studios (i.e., small group of opinion generators led by a neutral facilitator) can be held to provide input and feedback on the planning, design, implementation, translation, and dissemination of research. Such studios are invaluable tools to ensure that research appropriately addresses issues that are most important to relevant communities and that recruitment materials “speak” to the participants you hope to enroll. Depending on the specific disorder under study, the aims of recruitment may differ. In EDGI, for example, we developed strategies to ensure enrollment of individuals with diverse body size, socioeconomic status, race, ethnicity, ability, biological sex assigned at birth, gender identity, treatment history, and diagnostic status. Regular engagement of community partners can boost public trust in health research and ensure bi‐directional communication and stakeholder input from study conceptualization through dissemination. Humbly seeking and incorporating feedback on recruitment materials and aspects of the study itself from individuals from underrepresented groups can build trust in the research process. Field testing of materials is essential and can save money and time and avert adverse responses from the community.
Community involvement in genetic research can and should go beyond brief consultation and soliciting feedback. Recognize that the topic of DNA is often interwoven with contextual factors such as religious and cultural beliefs, histories of exploitative research, use of blood quantum to deny tribal membership and land rights, and more. Being aware of these contexts, histories, and views in study population communities, and taking steps to mitigate systemic power imbalances serves to increase engagement and transparency.
In New Zealand, for example, best practice guidelines for research on genomics and biobanking with Māori communities have been developed based on building trust between indigenous peoples and research groups, enhanced accountability for the researchers, improving equity, and knowledge translation leading to improved outcomes for the sampled group (Hudson et al., 2016; Hudson et al., 2020; Hudson, Milne, Reynolds, Russell, & Smith, 2010). Once open communication and collaboration is facilitated, communities' terms for engagement in genetic studies and providing samples should be respected. For Aborigines, Torres Strait Islanders, and Māori, for example, some of these terms include requirements for trusting relationships, knowledge about ongoing sample handling, and who has access to the samples and why (Beaton et al., 2017; Kowal & Anderson, 2012). Researchers considering procedures that include the use of marginalized and historically exploited group samples in any country should engage with these groups before proposing projects, as a lack of awareness or inappropriate engagement may perpetuate exploitative practices, cause unnecessary delays, and contribute to distrust and hesitancy.
2.1.3. Enlisting ambassadors and spokespeople
Studies can benefit from engaging individuals who speak about their experiences participating in research, their own experience with mental illness or related conditions, and their reasons for participating. Effective study ambassadors reach populations of individuals who might not typically be reachable via standard university‐based outreach. Study teams may spend considerable time brainstorming and contacting high profile celebrities or influencers with connections to the disorder under study to be ambassadors or spokespeople for research studies, often with limited success. Personal connections are typically the best (and sometimes only) way to reach influencers. Unfortunately, cold emails or calls often fail to get past professional gatekeepers. Engaging more ambassadors with moderate reach will amplify efforts without the lost time spent trying to “land a big fish.” Personal connections with individuals who have extensive reach are of course invaluable, and should be nurtured when they exist. Although many ambassadors are willing and able to donate their time, more extensive involvement should be remunerated and earmarked as an allowable expense on grants. Re‐tweeting a study message does not take much effort on the part of an ambassador; however, speaking at community meetings, reaching out to their community, and more time‐ and effort‐intensive involvement should be acknowledged both with remuneration and by acknowledgements in scientific articles and lay publications as well.
2.1.4. Ensuring that participants see themselves represented
One of the most frequently encountered reasons for individuals deciding not to participate in research is that they do not see themselves represented either in study materials or in study personnel (Butler et al., 2013). This takes on added significance for a disorder such as eating disorders, which have been wrongly portrayed for decades as disorders of white, young, affluent women. Visuals, vignettes, ambassadors, featured guest experts with lived experience, and study personnel should all represent the groups one hopes to enroll (Waheed, Hughes‐Morley, Woodham, Allen, & Bower, 2015). Representation refers to much more than skin tone, and highlighting a broad range of experiences of psychiatric disorders is much more impactful than recruitment images including people with various skin tones. It is valuable to find ways to share the work and experiences of individuals who identify with the underrepresented communities.
2.1.5. Being mindful of language
Language to discuss both mental illnesses and underrepresented groups continually evolves. Community advisors are well positioned to assist with using language that is current and does not offend or confuse. The People First Respectful Language Modernization Act of 2006 (Council of the District of Columbia, 2006) required the use of respectful language when referring to people with disabilities by putting the person before the disability and not referring to them by their disability. Examples in psychiatry include “person with an eating disorder” (rather than anorexics and bulimics) or “person with schizophrenia” (rather than schizophrenics). At the same time, there is also pushback against this in some communities. For example, some adults with autism spectrum disorder prefer to be called “autistic” rather than “person with autism,” associating the latter with out‐of‐touch allies and paternalistic attitudes (Kenny et al., 2016). Language and identity politics are continually evolving. Taking time to learn how relevant groups of people talk about themselves and their experiences is worthwhile and necessary to avoid alienating key allies and sources of participants (Williams, Housman, & McDonald, 2020). The American Psychological Association has prepared comprehensive Equity, Diversity, and Inclusion Inclusive Language Guidelines (American Psychological Association, 2021). Periodic review of terms used may be necessary over the course of a longer study.
Similar considerations hold when developing recruitment materials for various ancestry groups where it is important to us the most current language to refer to the group (e.g., Latinx, Latine), while it is important to acknowledge that not all individuals from that ancestry may refer to themselves in the same manner. Explanations about why certain terms are used can serve to clarify intentions. Appropriate language to describe individuals in the LGBTQIA+ community also evolves. Ensure language is accurate and specific when referring to gender (man, woman, nonbinary, Two Spirit, etc.) versus sex assigned at birth (male, female, and intersex). Many universities have resources to assist with ensuring that recruitment language is current, appropriate, and respectful.
For example, the same word may have different meanings for Spanish‐speaking populations with different regional dialects. The process of translating materials has to be rigorous. For example, doing translations, back translations, and reviewing in focus groups can help ensure accuracy of assessments, but may not suffice. Inspired by the methodology of DuBay and Watson (2019), to translate EDGI materials to Spanish, we partnered with six native Spanish speakers from three different Latin American countries. This group included an individual with an eating disorder, a parent of a child with an eating disorder, a certified translator, healthcare workers in a variety of fields, and an eating disorder specialist. After being trained by EDGI staff in dimensions of equivalence in the translation of research assessments, EDGI materials were translated, reviewed, discussed, agreed upon, and finally certified by a Spanish‐speaking EDGI staff member.
Moreover, assessments need to be adapted culturally to ensure their applicability to the region in which they are being deployed. A vivid example is that a treatment history questionnaire in the United States would be meaningless in a country with socialized or universal healthcare.
2.2. Recruitment
2.2.1. Study branding
Marketing research highlights the power of visible, eye‐catching, attractive, and consistent branding. Companies strive to develop legitimacy and brand‐loyalty in their customers, and the same can be done in research. The eating disorders field first established the Anorexia Nervosa Genetics Initiative (ANGI), which led to global recognition and buy‐in from patients, families, bloggers, and advocates. This movement expanded to EDGI, which allowed us to broaden our inquiry to other eating disorders. In each case, we ensured that every participating site, every publication, and every social media account globally included the EDGI name and logo. Various color palettes were developed for different countries and institutions, but the brand remained constant across sites.
Particularly on social media, a study should have a consistent and identifiable brand “personality.” In addition to using established fonts and color palettes, the tone of the brand is important because messages that convey a detached or academic flavor or include medical jargon often do not resonate with the desired audience.
2.2.2. Recruiting individuals across the lifespan
Older adults are a growing part of the world's population and might be important to include for any particular study. An important consideration for recruiting older individuals for psychiatric genetic studies is the exclusive dependence on internet‐based recruitment methods. Although millions of adults use the internet (Pew Research, 2017), a sole focus on digital recruitment could result in selection bias, particularly for older adults who do not have access to or facility with the internet. Careful consideration of where older adults spend time on the internet can inform recruitment strategies, and for those who do not have access to the internet, publications aimed at older individuals, societies or community centers that cater to older individuals, and places of worship can be effective avenues to recruit older individuals (Hunsaker & Hargittai, 2018; Miller et al., 2016). Finally, having representative ambassadors who are themselves older and who are comfortable talking about their own participation can inspire older individuals to participate.
2.2.3. Optimizing social media
Many individuals with lived experience of psychiatric disorders rely on social media for information and a sense of community. Although social media definitely has potential adverse effects on mental health (Alonzo, Hussain, Stranges, & Anderson, 2021; Bashir & Bhat, 2017; Karim, Oyewande, Abdalla, Chaudhry Ehsanullah, & Khan, 2020; O'Reilly et al., 2018), it can also be a mechanism to distribute factual information about psychiatric disorders and recruit individuals for research participation. The use of social media for recruitment must be strategic and ads and posts need to address the individuals you hope to recruit effectively. The success of a campaign can be influenced by the content, the appropriateness of the visuals, and type of social media (e.g., younger participants on TikTok to older participants on Facebook). A combination of paid, recruitment‐focused advertisements, and a variety of organic content is best. Careful testing of different paid ad campaigns with catered content for specific audiences can ensure that the approach is effective and is reaching the individuals you are intending to reach. As an example, Facebook paid advertisements were a feasible and effective tool to recruit Latin and East Asian cancer survivors who showed interest in future psychosocial research (Tsai, Zavala, & Gomez, 2019). More broadly, a systematic review suggested that Facebook is an effective and cost‐efficient recruitment method to enroll participants of all ages, to any psychosocial, health, or medical research (Thornton et al., 2016). Shifting attitudes toward specific social media platforms or shifts in user demographics could lead to recruitment bias or change receptivity to research participation notifications. Other platforms may be equally as effective in countries where Facebook is restricted. It is wise to develop and analyze metrics to determine which approaches are working and which are not effective, and adapt campaigns accordingly.
Social media can be very effective when the approach is carefully planned. Quality content can be time‐consuming to develop, and often varies across social media platforms. Social media accounts should release content that is engaging, informative, and interesting—no one wants to follow a page that mostly shares recruitment messages. Content can involve sharing recruitment updates, blogs about topical issues and research updates, as well as public lectures and activities undertaken by the researchers/clinicians. Messages can be amplified by directly asking colleagues, friends, and influencers to re‐post your content. Inherent in using social media for recruitment is acknowledging the percentage of the population that uses social media and the percentage that does not have or has restricted internet access. It is important to note that this is not a “free” method of recruitment. Social media ad campaigns can be costly and should be acceptable budget items in grant applications.
2.2.4. Joining podcasts
Podcasts have enjoyed a resurgence of interest in recent years (Chu, Chang, & Lin, 2021; Palenque, 2016; Richter, 2021). Both organizations (e.g., federal funding bodies, corporations, and specialty organizations) as well as individual advocates, activists, parents, and patients host podcasts that discuss psychiatric illnesses. When invited to be on a podcast, be open‐minded. Even if their listenership is small, you could still be heard by individuals whose reach is large and in a different space than other recruitment activities.
2.2.5. Reaching traditional media
Traditional media (television, radio, and newspapers) remains an important avenue for recruitment and dissemination of information about research. When contacted by a reporter, it is best to respond immediately. Reporters will send out multiple feelers in the hope of someone responding, and even if your response indicates that you will respond shortly, you will remain on their list. Always ask directly that they put the URL for the research in the article/show, and that the link appears on their show notes. Study personnel can benefit from a “Must Mention” guide as a reminder of points that should be made about the study. Providing reporters with fact sheets about the disorder under study, including details about the investigation streamlines information transfer. It is also worth considering a tip sheet for appropriate and inappropriate images for stories about the study or phenotype. Many advocacy organizations provide such guidelines for the media.
2.2.6. Recruiting without the internet
An important consideration is that underrepresented groups may have limited access to the internet or familiarity with various platforms. Although billions of people of all ages use the internet, an exclusive focus on digital recruitment could result in selection bias. Solely focusing on digital recruitment by nature excludes those in geographic locations with limited internet coverage. It also excludes those with limited or no access to hardware (computers and smart phones) and internet service provision. A solution involves consideration of where underrepresented groups with limited internet access interact and engage with people and environments. For example, the presence of ambassadors in such social and physical environments can promote visibility, accessibility, and trust. Again, time and effort by study ambassadors in direct engagement with their communities should be remunerated and acknowledged.
2.3. Transparency
2.3.1. Addressing hesitancy directly and genuinely
Addressing research hesitancy directly can be uncomfortable, but failing to do so can be invalidating. Stories about adverse experiences are passed down through generations and within communities, sowing seeds of distrust and concerns about exploitation and fueling intergenerational trauma. An unwillingness to engage around these difficult topics conveys a lack of empathy and understanding. Having individuals on staff who can speak directly to concerns about privacy and stigma is enormously important. Stakeholder studios can be used to develop and test messages that address these concerns directly. It is critical to solicit feedback about recruitment campaigns to ensure that they continue to achieve the desired goals as the study progresses and to remain available to respond to questions or concerns. Likewise, being forthright in explaining the steps that are taken to protect individuals and their privacy can make participants aware of the layers of oversight built into the research (e.g., ethical committees, national regulatory offices, and funding agencies). Honesty about the risks to the individual and to their privacy is both essential and ethical.
2.3.2. Making the research process transparent
Videos, infographics, or animated shorts can demystify the research process. Participants are comforted by knowing exactly what will happen to their information and their samples and who is analyzing data or handling their samples. In addition, being transparent about why you are seeking participation from underrepresented individuals can be valuable for engagement. Clear explanations are needed about why you are seeking individuals from specific groups and the benefits it may hold for those groups in the future. This counters the perception that researchers are just conducting a tick box diversity exercise. Ambassadors can refer interested potential participants to the resource rather than trying to explain the research process themselves. Ambassadors should be well equipped with tools to easily disseminate information about the study.
2.3.3. Being specific
It is rarely effective to recruit individuals from diverse backgrounds by developing a campaign that targets “individuals from diverse backgrounds.” Co‐production, or working alongside members of the groups you hope to recruit from the outset is important in developing a tailored set of strategies, led and voiced by relevant lived experience. This may lead to a set of community‐informed recruitment strategies that directly speak to the specific individuals or groups you hope to include in the research. Related to the above point of potential participants needing to see themselves reflected in recruitment materials and study personnel, approaching multiple underrepresented groups as a monolithic entity or a list is unlikely to achieve the desired outcome. As much as possible, develop campaigns that are truly co‐produced and respectful to specific groups that you hope to engage.
2.4. Investing in the community
2.4.1. Finding trusted locations (physical or virtual) to advertise the study
When developing recruitment strategies, discuss with the community consultants appropriate recruitment venues for underserved populations. In eating disorders, we have worked with churches, hair and nail salons, fitness centers, advocacy organizations, and patient organizations. In the Latinx community, primary care clinics, community health centers, and churches can be key for recruitment because of the trust placed in the primary care physician and clergy by many individuals in that community. Local international grocery stores, community bulletin boards, and independent newspapers can augment recruitment efforts. One study of Latinx and African American healthy volunteers reported varying willingness to participate in health‐related research depending on the research sponsor (Thetford, Gillespie, Kim, Hansen, & Scarinci, 2021). Local doctors and university hospitals were cited as the most trusted sources and for‐profit and tobacco companies the least trusted. For underserved populations, virtual locations offer a safe space to belong. By engaging leaders of virtual communities and advertising through their communication channels (e.g., social media groups, online ambassadors, digital magazines, or newsletters), the advertisement can spread to otherwise hard‐to‐reach populations.
2.4.2. Developing of trust with underserved populations via education
Related to research participation hesitancy is also a sense of mistrust of research, especially when the researchers are from privileged groups (e.g., cisgender individuals of European origin). It is important to deliver a message about the advantages of taking part in genetic studies for each participating group, such as highlighting that participation contributes to the development of better treatments, improved understanding of the causes of illness, while also being clear about the risks. A rich example provided by a Black participant in a feedback studio was, "How will my participation in this study help other Black women?" Being prepared to discuss the history of maltreatment in research (e.g., Tuskegee experiments with African Americans in the United States), as well as the current realities of medical racism, and clearly delineating what is being done to protect their rights as participants, is an important component of transparency. Having personnel from different backgrounds involved in recruitment and available for questions is important for building trust. It is also desirable to offer something in return. For example, if an underrepresented community organization or group has a newsletter, publication, podcast, or other communication pathway, an offer to write or speak about the research or disorder can demystify the process, underscore the advantages of participating to the individual and the community, and increase trust. As a gesture of reciprocity, researchers can offer to share any resources, posts, services, or another informative piece that aligns with the organization's values. Furthermore, it is important to ask specific organizations if they would like to be acknowledged in publications, and if so, work with them to develop appropriate acknowledgements.
2.4.3. Building capacity
Engaging individuals from diverse backgrounds in the research process is a natural segue to capacity building (i.e., helping individuals and organizations from underrepresented groups obtain, improve, and retain the connections, skills, knowledge, and other resources to secure funding and conduct research themselves). Reciprocal relationships with community organizations and underserved populations may involve educating groups about the findings, implications, and applications in ways that are culturally meaningful to the group. Such group education increases capacity and autonomy to apply the findings. Data and sample collection in populations, regions, or countries are perfect opportunities to build capacity within those areas. Ensuring that study personnel from underrepresented groups are in leadership roles not only builds research capacity, but also reflects and speaks to the communities that are being approached for participation. Embedding training and mentorship opportunities into collaborations helps build the scientific workforce and increases the likelihood of long‐term collaborative relationships. Funds and time should be built into budgets and timelines to underscore the dedication to capacity building. Increased availability of bilateral or multi‐lateral grants partnering higher income countries and LMICs could serve the dual purpose of diversifying global collections of samples and data and while building capacity for independent research.
3. DISCUSSION
Although increased efforts towards diversity and inclusion of participants in clinical trials, the field of genetics has additional hurdles, challenges, and requirements for broadening research participation. Unless researchers prioritize enrolling individuals from diverse backgrounds in genetic studies, overrepresentation of European‐ancestry individuals will persist, and science and downstream healthcare will remain biased and inequitable. Recruiting from diverse populations needs to be a priority that is supported by adequate funding and deliberate strategies. Simply requiring certain percentages of “minority participants” is not adequate support from funding bodies. Several countries have woven co‐design of research into the funding model, and included it as a metric for success in competitively funded applications (Boyd, McKernon, Mullin, & Old, 2012; O'Brien, Fossey, & Palmer, 2021; Richard et al., 2017; Wade, Hart, Mitchison, & Hay, 2021). Adequate resources must be made available to achieve true representativeness in genetic research. Budget items for the approaches outlined herein should be included and supported in the primary awards, not as afterthoughts or supplements. Costs for translation, ambassador effort and community consultation, project staff, creativity, and outreach are required to succeed in diverse recruitment goals.
Ongoing and reciprocal interaction with community groups is essential for building trust, as is soliciting and incorporating feedback from individuals from diverse communities. Most importantly, research personnel need to be genuine and transparent and, as much as possible, make the research process accessible and valuable to members of the community (Box 1).
BOX 1. Recommendations for recruitment of participants from diverse backgrounds.
(1) Research design—concepts, constructs, measurement
Recognize and center unique experiences, challenges, and concerns of individual underrepresented communities and across the lifespan.
Implement co‐design strategies throughout the research process and humbly solicit feedback from individuals from diverse communities (about measurement, procedures, concepts, etc.).
Ensure accurate translation and cultural adaptation and sensitivity in instrumentation.
Be mindful of language used to describe underrepresented groups and changes over time.
Be informed about historical, political, and cultural contexts of genes and genetic research and respect communities' terms for participation.
Budget adequate funds, time, and effort for recruitment of diverse populations.
(2) Recruitment and study branding
Create a consistent, recognizable brand for your study via social media, podcasts, and traditional media.
Ensure that recruitment materials and study personnel represent the groups you hope to recruit to participate.
Make extra effort to engage local leaders, influencers, and other prominent individuals within the respective communities; involve them in recruitment.
(3) Transparency
Address hesitancy directly; acknowledge and be open to discussions about past abuses, persisting medical racism, and intergenerational trauma.
Demystify the research process with written, audio, or video materials that explain what happens to information and samples and who has access to them.
Make the research as transparent and accessible as possible.
Be forthright and genuine about potential breaches of confidentiality and their potential impact.
(4) Community investment in and ownership of the study
Employ individuals from underrepresented groups into research leadership positions; build capacity.
Remunerate and/or acknowledge study ambassadors or groups in scientific and lay dissemination materials.
Provide regular study and recruitment updates to the community.
Reciprocate with public lectures, blogs, or other resources for the community.
Create and distribute lay summaries of findings and implications of the research.
FUNDING INFORMATION
Cynthia M. Bulik is supported by NIMH (R01MH120170, R01MH124871, R01MH119084, R01MH118278); Brain and Behavior Research Foundation Distinguished Investigator Grant; Swedish Research Council (Vetenskapsrådet, award: 538‐2013‐8864); Lundbeck Foundation (grant no. R276‐2018‐4581). Christina Sanzari is support by NSF GRFP (grant no. 1645421). Rachel Goode is supported by NIDDK (K23 DK129832). Ya‐Ke Wu reports funding from International Society of Nurses in Genetics (ISONG). Martin A. Kennedy, Jennifer Jordan, Michaela A. Pettie, Hannah L. Kennedy, Lana Cleland are supported by R01MH120170.
CONFLICT OF INTEREST
Cynthia M. Bulik reports: Shire (grant recipient, Scientific Advisory Board member); Idorsia (consultant); Pearson (author, royalty recipient); Equip Health Inc., (Clinical Advisory Board). Other authors report no conflicts.
MacDermod, C. , Pettie, M. A. , Carrino, E. A. , Garcia, S. C. , Padalecki, S. , Finch, J. E. , Sanzari, C. , Kennedy, H. L. , Pawar, P. S. , Mcgough, M. M. , Iwashita, A. , Takgbajouah, M. , Coan, D. , Szakasits, L. , Goode, R. W. , Wu, Y.‐K. , Reyes‐Rodríguez, M. L. , Vacuán, E. M. T. C. , Kennedy, M. A. , … Bulik, C. M. (2022). Recommendations to encourage participation of individuals from diverse backgrounds in psychiatric genetic studies. American Journal of Medical Genetics Part B: Neuropsychiatric Genetics, 189B:163–173. 10.1002/ajmg.b.32906
Although author order is a practical necessity, all authors have made unique and important contributions to this piece of work that may not be reflected in author order.
Funding information Brain and Behavior Research Foundation; International Society of Nurses in Genetics; Lundbeckfonden, Grant/Award Number: R276‐2018‐4581; National Institute of Mental Health, Grant/Award Numbers: R01MH118278, R01MH124871, R01MH120170, R01MH119084; National Science Foundation, Grant/Award Number: 1645421; Vetenskapsrådet, Grant/Award Number: 538‐2013‐8864
DATA AVAILABILITY STATEMENT
Data sharing is not applicable to this article as no new data were created or analyzed in this study.
REFERENCES
- Alonzo, R. , Hussain, J. , Stranges, S. , & Anderson, K. K. (2021). Interplay between social media use, sleep quality, and mental health in youth: A systematic review. Sleep Medicine Reviews, 56, 101414. 10.1016/j.smrv.2020.101414 [DOI] [PubMed] [Google Scholar]
- American Psychological Association . (2021). Inclusive language guidelines . Retrieved from https://www.apa.org/about/apa/equity-diversity-inclusion/language-guidelines.pdf
- Australian Government . (2007). National Statement on Ethical Conduct in Human Research . Retrieved from https://www.nhmrc.gov.au/sites/default/files/documents/attachments/National%20Statement/e72.pdf
- Bashir, H. , & Bhat, S. (2017). Effects of social media on mental health: A review. International Journal of Indian Psychology, 4, 125–131. [Google Scholar]
- Beaton, A. , Hudson, M. , Milne, M. , Port, R. V. , Russell, K. , Smith, B. , … Wihongi, H. (2017). Engaging Maori in biobanking and genomic research: A model for biobanks to guide culturally informed governance, operational, and community engagement activities. Genetics in Medicine, 19(3), 345–351. 10.1038/gim.2016.111 [DOI] [PubMed] [Google Scholar]
- Boraska, V. , Franklin, C. S. , Floyd, J. A. B. , Thornton, L. M. , Huckins, L. M. , Southam, L. , … Bulik, C. M. (2014). A genome‐wide association study of anorexia nervosa. Molecular Psychiatry, 19(10), 1085–1094. 10.1038/mp.2013.187 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Boyd, H. , McKernon, S. , Mullin, B. , & Old, A. (2012). Improving healthcare through the use of co‐design. The New Zealand Medical Journal, 125(1357), 76–87. [PubMed] [Google Scholar]
- Bruch, H. (1978). Obesity and anorexia nervosa. Psychosomatics, 19(4), 208–212. [DOI] [PubMed] [Google Scholar]
- Bryc, K. , Velez, C. , Karafet, T. , Moreno‐Estrada, A. , Reynolds, A. , Auton, A. , … Ostrer, H. (2010). Genome‐wide patterns of population structure and admixture among Hispanic/Latino populations. Proceedings of the National Academy of Sciences of the United States of America, 107(Suppl 2), 8954–8961. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bulik, C. M. (2020). Eating disorders genetics in Asia. The International Journal of Eating Disorders, 54, 184–186. 10.1002/eat.23445 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Butler, J., 3rd , Quinn, S. C. , Fryer, C. S. , Garza, M. A. , Kim, K. H. , & Thomas, S. B. (2013). Characterizing researchers by strategies used for retaining minority participants: Results of a national survey. Contemporary Clinical Trials, 36(1), 61–67. 10.1016/j.cct.2013.05.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chen, M. S., Jr. , Lara, P. N. , Dang, J. H. , Paterniti, D. A. , & Kelly, K. (2014). Twenty years post‐NIH Revitalization Act: Enhancing minority participation in clinical trials (EMPaCT): Laying the groundwork for improving minority clinical trial accrual: Renewing the case for enhancing minority participation in cancer clinical trials. Cancer, 120(Suppl 7), 1091–1096. 10.1002/cncr.28575 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cheng, Z. H. , Perko, V. L. , Fuller‐Marashi, L. , Gau, J. M. , & Stice, E. (2019). Ethnic differences in eating disorder prevalence, risk factors, and predictive effects of risk factors among young women. Eating Behaviors, 32, 23–30. 10.1016/j.eatbeh.2018.11.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chu, C. F. , Chang, C. C. , & Lin, Y. H. (2021). Podcasts, the pandemic, and a paradigm shift in medical careers. Journal of the Formosan Medical Association, 121, 1–2. 10.1016/j.jfma.2021.08.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Council of the District of Columbia . (2006). People First Respectful Language Modernization Act of 2006 . Retrieved from https://odr.dc.gov/page/people-first-language.
- De Ver Dye, T. , Tavarez, Z. Q. , Perez Ramos, J. G. , Fernandez, I. D. , Vega, C. V. , Vega Ocasio, D. M. , … Groth, S. W. (2021). Participation in genetic research among Latinx populations by Latin America birth‐residency concordance: A global study. Journal of Community Genetics, 12(4), 603–615. 10.1007/s12687-021-00538-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- DuBay, M. , & Watson, L. (2019). Translation and cultural adaptation of parent‐report developmental assessments: Improving rigor in methodology. Research in Autism Spectrum Disorders, 62, 55–65. [Google Scholar]
- Duncan, L. , Shen, H. , Gelaye, B. , Meijsen, J. , Ressler, K. , Feldman, M. , … Domingue, B. (2019). Analysis of polygenic risk score usage and performance in diverse human populations. Nature Communications, 10(1), 3328. 10.1038/s41467-019-11112-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Duncan, L. , Yilmaz, Z. , Gaspar, H. , Walters, R. , Goldstein, J. , Anttila, V. , … Bulik, C. M. (2017). Significant locus and metabolic genetic correlations revealed in genome‐wide association study of anorexia nervosa. The American Journal of Psychiatry, 174, 850–858. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Durie, M. H. (1989). The Treaty of Waitangi and health care. The New Zealand Medical Journal, 102(869), 283–285. [PubMed] [Google Scholar]
- Graber, J. , Brooks‐Gunn, J. , Paikoff, R. , & Warren, M. (1994). Prediction of eating problems: An 8‐year study of adolescent girls. Devlopmental Psychology, 30(6), 823–834. [Google Scholar]
- Hudson, M. , Garrison, N. , Sterling, R. , Caron, N. , Fox, K. , Yracheta, J. , … Carroll, S. (2020). Rights, interests and expectations: Indigenous perspectives on unrestricted access to genomic data. Nature Reviews. Genetics, 21(6), 377–384. 10.1038/s41576-020-0228-x [DOI] [PubMed] [Google Scholar]
- Hudson, M. , Milne, M. , Reynolds, P. , Russell, K. , & Smith, B. (2010). Te Ara Tika: Guidelines for Māori research ethics: A framework for researchers and ethics committee members . Retrieved from Auckland: Health Research Council of New Zealand.
- Hudson, M. , Southey, K. , Uerata, L. , Beaton, A. , Milne, M. , Russell, K. , … Cheung, M. (2016). Key informant views on biobanking and genomic research with Maori. The New Zealand Medical Journal, 129(1447), 29–42. [PubMed] [Google Scholar]
- Hunsaker, A. , & Hargittai, E. (2018). A review of Internet use among older adults. New Media & Society, 20(10), 3937–3954. 10.1177/1461444818787348 [DOI] [Google Scholar]
- InsideOut Institute . (2021). Australian eating disorders research & translation strategy 2021–2031. Australian Government, Department of Health.
- Karim, F. , Oyewande, A. A. , Abdalla, L. F. , Chaudhry Ehsanullah, R. , & Khan, S. (2020). Social media use and its connection to mental health: A systematic review. Cureus, 12(6), e8627. 10.7759/cureus.8627 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kenny, L. , Hattersley, C. , Molins, B. , Buckley, C. , Povey, C. , & Pellicano, E. (2016). Which terms should be used to describe autism? Perspectives from the UKautism community. Autism, 20(4), 442–462. 10.1177/1362361315588200 [DOI] [PubMed] [Google Scholar]
- Konkel, L. (2015). Racial and ethnic disparities in research studies: The challenge of creating more diverse cohorts. Environmental Health Perspectives, 123(12), A297–A302. 10.1289/ehp.123-A297 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kowal, E. , & Anderson, I. (2012). Genetic research in Aboriginal and Torres Strait Islander communities: Continuing the conversation. Melbourne, Australia: Lowitja Institute. [Google Scholar]
- Kurt, A. , Semler, L. , Jacoby, J. L. , Johnson, M. B. , Careyva, B. A. , Stello, B. , … Smulian, J. C. (2016). Racial differences among factors associated with participation in clinical research trials. Journal of Racial and Ethnic Health Disparities. 10.1007/s40615-016-0285-1 [DOI] [PubMed] [Google Scholar]
- Lester, R. , & Petrie, T. (1998). Physical, psychological, and societal correlates of bulimic symptomatology among African American college women. Journal of Counseling Psychology, 45, 315–321. [Google Scholar]
- Marques, L. , Alegria, M. , Becker, A. E. , Chen, C. N. , Fang, A. , Chosak, A. , & Diniz, J. B. (2011). Comparative prevalence, correlates of impairment, and service utilization for eating disorders across US ethnic groups: Implications for reducing ethnic disparities in health care access for eating disorders. The International Journal of Eating Disorders, 44(5), 412–420. 10.1002/eat.20787 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Miller, E. , Nowson, C. , Dunstan, D. , Kerr, D. , Solah, V. , Menzies, D. , & Daly, R. (2016). Recruitment of older adults with type 2 diabetes into a community‐based exercise and nutrition randomised controlled trial. Trials, 17(1), 467. 10.1186/s13063-016-1589-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Miller, M. , & Pumariega, A. (2001). Culture and eating disorders: A historical and cross‐cultural review. Psychiatry, 64(2), 93–110. 10.1521/psyc.64.2.93.18621 [DOI] [PubMed] [Google Scholar]
- Nakabayashi, K. , Komaki, G. , Tajima, A. , Ando, T. , Ishikawa, M. , Nomoto, J. , … Shirasawa, S. (2009). Identification of novel candidate loci for anorexia nervosa at 1q41 and 11q22 in Japanese by a genome‐wide association analysis with microsatellite markers. Journal of Human Genetics, 54(9), 531–537. [DOI] [PubMed] [Google Scholar]
- O'Brien, J. , Fossey, E. , & Palmer, V. J. (2021). A scoping review of the use of co‐design methods with culturally and linguistically diverse communities to improve or adapt mental health services. Health & Social Care in the Community, 29(1), 1–17. 10.1111/hsc.13105 [DOI] [PubMed] [Google Scholar]
- O'Reilly, M. , Dogra, N. , Whiteman, N. , Hughes, J. , Eruyar, S. , & Reilly, P. (2018). Is social media bad for mental health and wellbeing? Exploring the perspectives of adolescents. Clinical Child Psychology and Psychiatry, 23, 601–613. [DOI] [PubMed] [Google Scholar]
- Occa, A. , Morgan, S. E. , & Potter, J. E. (2018). Underrepresentation of Hispanics and other minorities in clinical trials: Recruiters' perspectives. Journal of Racial and Ethnic Health Disparities, 5(2), 322–332. 10.1007/s40615-017-0373-x [DOI] [PubMed] [Google Scholar]
- Palenque, S. (2016). The power of podcasting: Perspectives on pedagogy. Journal of Instructional Research, 5, 4–7. [Google Scholar]
- Peterson, R. E. , Kuchenbaecker, K. , Walters, R. K. , Chen, C. Y. , Popejoy, A. B. , Periyasamy, S. , … Duncan, L. E. (2019). Genome‐wide association studies in ancestrally diverse populations: Opportunities, methods, pitfalls, and recommendations. Cell, 179(3), 589–603. 10.1016/j.cell.2019.08.051 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Pew Research . (2017). Technology use among seniors . Retrieved from https://www.pewresearch.org/internet/2017/05/17/technology-use-among-seniors/
- Racine, S. E. , VanHuysse, J. L. , Keel, P. K. , Burt, S. A. , Neale, M. C. , Boker, S. , & Klump, K. L. (2017). Eating disorder‐specific risk factors moderate the relationship between negative urgency and binge eating: A behavioral genetic investigation. Journal of Abnormal Psychology, 126(5), 481–494. 10.1037/abn0000204 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Richard, L. , Piper, D. , Weavell, W. , Callander, R. , Iedema, R. , Furler, J. , … Palmer, V. J. (2017). Advancing engagement methods for trials: The CORE study relational model of engagement for a stepped wedge cluster randomised controlled trial of experience‐based co‐design for people living with severe mental illnesses. Trials, 18(1), 169. 10.1186/s13063-017-1878-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Richter, F. (2021). The steady rise of podcasts. Statista . Retrieved from https://www.statista.com/chart/10713/podcast-listeners-in-the-united-states/
- Sharma, A. , & Palaniappan, L. (2021). Improving diversity in medical research. Nature Reviews Disease Primers, 7, 1–2. [DOI] [PubMed] [Google Scholar]
- Striegel‐Moore, R. H. , & Bulik, C. M. (2007). Risk factors for eating disorders. The American Psychologist, 62(3), 181–198. [DOI] [PubMed] [Google Scholar]
- Thetford, K. , Gillespie, T. W. , Kim, Y. I. , Hansen, B. , & Scarinci, I. C. (2021). Willingness of Latinx and African Americans to participate in nontherapeutic trials: It depends on who runs the research. Ethnicity & Disease, 31(2), 263–272. 10.18865/ed.31.2.263 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Thornton, L. , Batterham, P. J. , Fassnacht, D. B. , Kay‐Lambkin, F. , Calear, A. L. , & Hunt, S. (2016). Recruiting for health, medical or psychosocial research using Facebook: Systematic review. Internet Interventions, 4, 72–81. 10.1016/j.invent.2016.02.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tong, A. , Synnot, A. , Crowe, S. , Hill, S. , Matus, A. , Scholes‐Robertson, N. , … Craig, J. C. (2019). Reporting guideline for priority setting of health research (REPRISE). BMC Medical Research Methodology, 19(1), 243. 10.1186/s12874-019-0889-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tsai, W. , Zavala, D. , & Gomez, S. (2019). Using the Facebook advertisement platform to recruit Chinese, Korean, and Latinx cancer survivors for psychosocial research: Web‐based survey study. Journal of Medical Internet Research, 21(1), e11571. 10.2196/11571 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wade, T. D. , Hart, L. M. , Mitchison, D. , & Hay, P. (2021). Driving better intervention outcomes in eating disorders: A systematic synthesis of research priority setting and the involvement of consumer input. European Eating Disorders Review, 29(3), 346–354. 10.1002/erv.2759 [DOI] [PubMed] [Google Scholar]
- Waheed, W. , Hughes‐Morley, A. , Woodham, A. , Allen, G. , & Bower, P. (2015). Overcoming barriers to recruiting ethnic minorities to mental health research: A typology of recruitment strategies. BMC Psychiatry, 15, 101. 10.1186/s12888-015-0484-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wang, K. , Zhang, H. , Bloss, C. S. , Duvvuri, V. , Kaye, W. , Schork, N. J. , … Price Foundation Collaborative Group . (2011). A genome‐wide association study on common SNPs and rare CNVs in anorexia nervosa. Molecular Psychiatry, 16(9), 949–959. 10.1038/mp.2010.107 [DOI] [PubMed] [Google Scholar]
- Watson, H. , Yilmaz, Z. , Thornton, L. , Hübel, C. , Coleman, J. , Gaspar, H. , … Bulik, C. (2019). Genome‐wide association study identifies eight risk loci and implicates metabo‐psychiatric origins for anorexia nervosa. Nature Genetics, 51, 1207–1214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wilfley, D. E. , Pike, K. M. , Dohm, F. A. , Striegel‐Moore, R. H. , & Fairburn, C. G. (2001). Bias in binge eating disorder: How representative are recruited clinic samples? Journal of Consulting and Clinical Psychology, 69(3), 383–388. [DOI] [PubMed] [Google Scholar]
- Williams, R. D., Jr. , Housman, J. M. , & McDonald, J. D. (2020). If we build it, will they come? Challenges of adapting and implementing a smoking cessation program for the LGBTQ community in southcentral Texas. Health Promotion Practice, 21(1_suppl), 118S–123S. 10.1177/1524839919882385 [DOI] [PubMed] [Google Scholar]
- Witham, M. D. , Anderson, E. , Carroll, C. , Dark, P. M. , Down, K. , Hall, A. S. , … INCLUDE Writing Group . (2020). Developing a roadmap to improve trial delivery for under‐served groups: Results from a UK multi‐stakeholder process. Trials, 21(1), 694. 10.1186/s13063-020-04613-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Data sharing is not applicable to this article as no new data were created or analyzed in this study.