Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2024 May 1.
Published in final edited form as: AJOB Empir Bioeth. 2023 May 1;14(4):185–196. doi: 10.1080/23294515.2023.2201478

Community Engagement in Precision Medicine Research: Organizational Practices and Their Impacts for Equity

Janet K Shim 1, Nicole Foti 2, Emily Vasquez 3, Stephanie M Fullerton 4, Michael Bentz 5, Melanie Jeske 6, Sandra Soo-Jin Lee 7
PMCID: PMC10615663  NIHMSID: NIHMS1897209  PMID: 37126431

Introduction

In recent decades, the field of precision medicine research (PMR), which studies the relationships among genes, behaviors, and environmental exposures, has been challenged by a lack of diversity in research participation (Need and Goldstein 2009; Popejoy and Fullerton 2016). In response, multiple initiatives now seek to include greater proportions of historically underrepresented populations in research. Community engagement (CE) has emerged as a key intervention to help achieve this goal, as well to acknowledge past harms in biomedical research and address persistent questions about trust and the conventional relationships that PMR researchers, and participants have with one another.

Our entry and contribution to debates about how to situate the invitation to underrepresented groups to participate in PMR within the broader social and political context and lived experiences of communities is an empirical one. Since 2018, we have followed how various PMR stakeholders responded to the call for greater diversity and inclusion: how they funded and designed studies, recruited populations, conducted data collection and analysis, and engaged diverse participants and communities. By empirically examining the range of perspectives, practices, and experiences of these activities, we aim to surface the normative assumptions, political dimensions, and social and ethical implications of implicit and overt choices made by PMR studies around CE practices.

In this paper, we examine how stakeholders approach three sets of questions: First, who to engage? That is, who is thought to represent the community? Second, what is CE, and how is it organized and conducted? And third, why engage communities; what is engagement for and what is it intended to produce? We find that ad hoc, opportunistic, and instrumental decisionmaking on who to engage, how community input is solicited and fits into study workflow, and what engagement was for had significant consequences for the time and resources devoted to CE and the ultimate impacts it has on research. These choices were also often contingent on previous, upstream decisions about research priorities and resources in ways that study personnel, in the moment, were frequently unable to discern. Such prior decisions cascaded into significant downstream consequences for what CE practices could achieve within the context of the PMR studies. In particular, we argue that the ways in which CE practices were implemented at times undermined the goals of inclusion and equity, potentially jeopardizing ongoing research relationships with communities and constraining the ability of PMR studies to provide equitable benefit. We end by offering reflections on the significance of these constraints for CE and their ramifications for relationships among researchers, research institutions, participants and communities.

Background

CE has been defined in many ways and covers a wide range of activities, as Tindana and colleagues (2015, 1) suggest through their broad definition of CE as “a process of working collaboratively with a group or groups of people on a shared goal or common interest.” CE and its related terms (e.g., engagement of participants, stakeholders, publics, citizens, etc.) are frequently used interchangeably, and the purpose, strategies, processes, and outcomes of CE are often left ill-defined (Richardson et al. 2021). Many case studies of CE used in genomics and precision medicine research offer reflections on the ethical motivations, structures and practices, and lessons learned (Lemke et al. 2010; Lyles et al. 2018; O’Daniel et al. 2022; Tuttle et al. 2021). Some studies highlighted the impact of power imbalances for CE while conducting research in underserved communities. In their review of genomic research conducted in Africa, Tindana and colleagues (2015; see also Agongo et al. 2021; Folayan, Oyedeji, and Fatusi 2015; Marsh et al. 2010) note that CE practices tended to support early phases of the research life course, such as participant recruitment. Brunger and Wall (2016) underscore the critical consideration of “research fatigue,” defined as the experience of being asked the same questions without results being shared with the community, and the burden of being asked to engage in, approve, consult, and collaborate on research not directed at the needs of the community.

Multiple scholars (Cheung 2018; Dheensa et al. 2018; Moodley 2017; Sabatello and Appelbaum 2017; Tindana et al. 2017) critique biobanking efforts as extractive, especially against the backdrop of structural inequities in health and health research (see Lee 2021). This literature identifies a range of ethical issues including: benefits that are diffuse, overly promissory, and not equity-centered; informed consent issues that arise from data use that may occur at an unknown time in the future; and an increasing push to make data openly available, that tends to benefit researchers in the global North and wealthier institutions. Some biobanks have implemented CE efforts to navigate issues arising from increasing commercialization, including intellectual property and ownership of samples, data, and downstream products (Cohn et al. 2015; Dry et al. 2017; Burgess et al. 2008; O’Daniel et al. 2012; Beaton et al. 2017).

An important thread underscored in CE scholarship is that it requires a clear warrant for research to be accountable. Pratt and de Vries (2018) argue that the absence of an ethical obligation and philosophical rationale for CE leads funders to view CE as “good practice” but stops short of mandating or providing sufficient resources for it. Molyneux and Bull (2013) identify “instrumental” goals such as increasing buy-in and enrollment, but note these can conflict with “intrinsic” goals of enacting inclusion and demonstrating respect. Finally, CE goals may invoke “transformative” aims to address power differentials and marginalization in current practices of scientific knowledge production (see also Reynolds and Sariola 2018; Versfeld et al. 2018; Gichuru et al. 2018). Key and colleagues’ (2019) model of CE integrates health equity indicators (e.g., power and control, mutual benefit) and contextual factors (e.g., history, trust). Pratt and de Vries (2018) assert that “shared health governance” should be a goal for global health research in low-and middle-income countries that aims to address health inequities, while Pratt, Cheah, and Marsh (2020) argue for “solidarity.”

A handful of scholars have developed typologies of CE practices, paying particular attention to varying levels of engagement and their effects on both research and community relationships. Seeley et al. (1992), for example, proposed that CE occurs along a contract/consultative/collaborative spectrum, while Tindana et al. (2015) further elaborated this into a typology based on whether approaches inform, consult, involve, collaborate, or empower.

Other models for conducting CE are often designed to materially express specific values like transparency and trust (Grayson, Doerr, and Yu 2020) and change prevailing practices (Sanders Thompson et al. 2020). These frameworks address common critiques that participation in and methods of CE are not truly democratic, and offer alternative practices that emphasize co-ownership, co-creation, and engagement at every research stage (Moodley and Beyer 2019; Murtagh et al. 2017; Wilson et al. 2019). Reynolds and Sariola (2018) argue that inclusive and meaningful CE must occur throughout all stages of knowledge production process, build skills and capacity of all parties, and aim to redress historical inequalities in North–South collaborations. Finally, Wilkins and Alberti (2019, 6) point to needed changes in the politics of knowledge: “A substantial barrier to community engagement is the traditional way that knowledge is accepted as legitimate in the academy—that is, it must be disciplinary, expert led, hierarchical, and university based.” In response to ongoing concerns, several institutional models have been developed to facilitate community-engaged research in genomics and PMR (e.g., Blacksher et al. 2021; Fohner et al. 2019).

Despite much attention to CE in genomics research, we identified only one study that examined engagement practices across multiple studies. Haldeman et al. (2014) examined CE among six US-based biobank sites and identified variability in definitions of “community,” CE practices, and reasons why biobanks undertake CE. No studies to our knowledge have combined an examination of different CE approaches with a comparative analysis of their effects, across the life course of the research studies, particularly those mandated to increase diversity and inclusion of underrepresented populations. However, the existing literature underscores the high stakes of how CE is implemented and practiced for whether it produces meaningful collaborations, community-research partnerships, and shared benefits, or whether it is seen as instrumental, transactional, and hierarchical ‘science as usual.’ Our data help to fill this gap, by uncovering the upstream-downstream dependencies that shape who, when, how, and with what effects CE is conducted, and how prior decisions about research goals and resources facilitate or constrain different kinds of CE practices. These practices in turn exact consequences for the relationships connecting communities, participants, researchers, and research institutions, and their potential to promote or hinder equity and inclusion.

Methods

This paper is part of a larger project investigating how diversity and inclusion are interpreted and operationalized in PMR studies, and impact practices such as enrollment, retention, engagement, data collection and analysis, and return of results. We followed five PMR studies across three consortia funded by the US National Institutes of Health (NIH). These five studies were selected because they were located in geographically diverse areas and used heterogeneous strategies to engage, recruit, and retain underrepresented individuals as research participants. Institutional review board approval was obtained from the University of California, San Francisco and Columbia University.

Our sources of data include: a content analysis of 76 documents, including study materials and funding announcements issued by the NIH; approximately 450 hours of observations of study activities, including site and consortium working group calls, and in-person or virtual meetings; and 125 in-depth interviews. Interviews included 102 initial and 23 follow-up interviews, each lasting 60–90 minutes, with a purposive sample that included funders of PMR research (n=4), PMR investigators (n=30), research team members (n=64), and research participants/participant advisory board members (n=4) (see Table 1 for aggregate demographic characteristics). Interviewees were purposively recruited based on their involvement in our five study sites or the PMR consortia to which they belonged. Semi-structured interviews explored the engagement, recruitment, and retention of underrepresented participants, as well as how study teams were operationalizing diversity mandates in their research. Interview guides were tailored, to enable us to ask about roles and activities relevant for each participant. Interviewees were offered a $50 gift card for each interview. All interviews were recorded and professionally transcribed verbatim.

Table 1.

Interview Sample Characteristics

Number Percent*

Gender
 Women 65 64%
 Men 37 36%
 Nonbinary or Other 0

Race
 Black or African American 16 16%
 American Indian and Alaska Native 0
 Asian American 6 6%
 Native Hawaiian and Other Pacific Islander 1 1%
 White 72 71%
 Two or more races 3 3%
 Not reported 4 4%

Ethnicity
 Hispanic/Latino 9 9%
 Not Hispanic/Latino 92 90%
 Not reported 1 1%

Age
 18–65 93 91%
 65+ 9 9%

Total 102
*

Rounded to nearest one percent.

All data, including documents, observation fieldnotes, and interview transcripts, were analyzed using the principles of constructivist grounded theory (Charmaz 2014). Consistent with this methodology, we treated documents, observation fieldnotes, and interviews all as raw data and analyzed them as one comprehensive dataset: we did not triangulate among different data sources, although we did use information gathered from one source to inform other data collection (e.g., crafting interview questions based on observations). The research team generated an initial codebook based on study aims, the existing literature, and issues that emerged repeatedly during data collection. Using the qualitative analysis software, Dedoose, the research team coded a small sample of documents, interview transcripts, and observation fieldnotes with this initial codebook; additional codes that arose through our inductive examination of the data were then included in the codebook. Through consecutive waves of independent and joint coding, followed by discussion and reconciliation, we finalized a codebook of 75 codes (some of which also have 2–14 sub-codes). For this analysis, we extracted a subset of data tagged with specific codes related to participant and CE; we then engaged in multiple (at least two) rounds of review and extensive memoing on these data, by different team members to allow for variable and iterative analysis and interpretation. Emerging interpretations and patterns with supporting data were iteratively presented to the whole research team and discussed and refined, leading to the generation of categories of codes and themes that led to this paper’s findings.

Results

Our data illustrate wide-ranging perspectives on who is thought to represent the community, how CE is implemented, and what CE is for and what the goals of CE ought to be. We did not discern any systematic differences by stakeholder type or group, but instead observed a range of views within each group. Below we discuss these views and provide illustrative data and quotes in Tables 2–5.

Who Is the ‘Community’? Targeted Versus Ad Hoc Outreach

Interviewees had varied perspectives on who best to include as community representatives (Table 2). They often described the importance of working with community insiders to open doors for them and broker relationships, and sought to engage communities through “trusted messengers.” However, in practice, many researchers reported using fairly ad hoc, opportunistic ways of tapping individuals for community input, based on convenience and networks of community members already known to them (as illustrated in 2A).

Table 2.

Perspectives on Who Represents the ‘Community’

Summary Illustrative Quotes & Examples
2A Ad hoc, opportunistic identification of community advisors based on preexisting connections “A lot of that [CE] we do with just people who…we all have either personal or professional relationships with … They are well plugged into various groups. We also just reach out to people who engage with us, or who we interact with, or we happen to go to a seminar and they’re presenting. And so it’s kind of a mish mash of people … then we were kind of able to reach out to groups that we hadn’t heard … from and kind of broaden our catchment.” (Research staff)
“Basically we said to the recruiters, ‘Is there somebody that you thought would be good?’ … And they’re like, ‘I got this guy,’ and ‘I got that guy.’ And we invited them. And some people accepted and some people didn’t.” (Research staff)
“We had an in-person meeting where we invited some of the stakeholders from the local groups … The listserv for people that can participate in the [community] feedback loop [for the study] came from the members that attended that in-person meeting. We are trying to increase the number of people that are on the listserv to include other people that weren’t at that particular in-person meeting. I know that I’ve had challenges at my site trying to identify people that might be appropriate for that sort of group.” (Research staff)
2B Selection based on personal characteristics & experience with research Interviewer: “What makes [the community advisor] a great collaborator, partner voice to have on board?”Investigator: “It’s their enthusiasm, their level of engagement and they’re just great sort of spokespeople … really compelling … the way they represent themselves … very motivated and interested in helping.”
“[The community advisors] have been quite involved with [the research institution] and are real advocates.” (Investigator)
2C Upstream decisions about who represents the community can shape, downstream, the nature and content of the community feedback given Excerpted from observation fieldnotes on a panel discussion and Q&A that included research staff from one of our study sites: “An audience member asked the panel how they identified individuals to serve on a community advisory board: ‘How are you making sure they are representative?’ The panelists looked at each other, some punted on answering, and some in the audience chimed in to give an array of responses: ‘some CAB [community advisory board] members volunteered and some have been recruited and are part of collaborating organizations,’ ‘there was a call for community partners to join,’ and ‘participants who expressed an interest. These were … known people in the community whose voice we value.’ The panelists’ hesitance and responses suggest an inability to know whether who they consulted with were sufficiently representative of the community’s interests, and that perhaps this question might not even have been considered at all.”

When researchers described other strategies for selecting community members to engage with, they often weighed such criteria as personal characteristics, preexisting relationships with the research institution, connections to communities, bandwidth to take on collaborations, and previous experience with biomedical research. Valuing such criteria for choosing community advisors often meant that they were enthusiasts of research and had experience navigating and interacting with research and healthcare institutions (2B).

Such practices—where community representatives were selected in ad hoc fashion, and criteria for engagement included those who had existing, positive relationships with the researchers or the research institution, and willing and available to be involved—underscore that decisions about who to engage impact the outcomes of engagement. Relying on such community advisors, rather than a broader group of others who may not be as aligned or involved with the research community, likely yields different feedback (2C). When the intent is to engage communities who are underrepresented in research, the practices of soliciting community input from preexisting connections and willing brokers, and those with the availability and ability to partner with research institutions, problematize who, and in turn what concerns, are included and heard, or excluded and rendered invisible.

How Should Community Engagement Be Done? The Importance of the Organizational Chart and Workflow

Researchers and funders understood it was crucial to embed engagement expectations and activities into the study process, and then consistently and actively support them. However, in general, their awareness of the need to embed CE into the conduct of studies did not consistently result in institutionalized practices. We found a great deal of diffuseness around what exactly constitutes engagement. Across the studies we followed, CE included distributing materials about their studies at community meetings, senior breakfasts, and neighborhood fairs. CE also included giving presentations at churches, barbershops, libraries, and businesses, and sharing information in neighborhood publications and media outlets. As one participant put it, “pretty much we try to go anywhere, anywhere that we can get the word out.”

Organizational nodes and activity streams in which CE occurs, and how work and decisionmaking authority flows through or around them, were influential in determining its impacts on research (Table 3). This included whether community members could determine which issues they considered, based on their own criteria, and revise or even veto proposals made upstream. The proximity or distance between where CE occurred and where research decisions were made was also highly consequential: the organizational structure of a study both reflected and reproduced explicit and implicit choices about whether and how community input should impact its conduct. One study was organized such that workflow must pass through community advisors, whose approval was required for research to proceed (3A). In contrast, other studies had hub-and-spoke or parallel track models for community input, where research staff reached out for feedback, but the workflow did not necessarily require that such input be collected and addressed before moving forward (3B).

Table 3.

Perspectives on How to Engage Communities

Summary Illustrative Quotes & Examples
The Importance of Organizational Structure
3A Community input integrated into study workflow “Every protocol change goes through their [community] board. Every single protocol change has to be approved by their [community] board.” (Funder)
3B Community input occurs in a parallel process or hub-and-spoke part of study workflow “We had a … meeting for stakeholder representatives … It was a fantastic meeting, just to hear all the different perspectives from these individuals … But it was also very clear from the post meeting discussion that [future meetings] were planning scientific talks that were intended for a scientific audience … And we’ve not had [community and patient stakeholders] back to another … meeting, but we’ve had separate interactions with them throughout.” (Research staff)
“There are just multiple infrastructures set up to get input … Multiple folks on … from different perspectives … we help it go through. It goes through us, but then we send it out to key advisors at all levels.” (Research staff)
Recognition, Accountability, and Return
3C Essential role of community expertise “I’m looking at … hearing from [community partners] instead of just hearing from researchers … understanding that this may be an opportunity for [researchers] to learn … We [researchers] don’t have all the answers … We need to sometimes shut up and listen … [and] need to become uncomfortable sometimes a little bit more … maybe to get out their comfort level a little bit and get more down into the community and ask them questions and get kind of more involved … I can let [community stakeholders] know that .. they’re valued and that we don’t want to start a meeting without you being here and those kinds of things, and that you’re the expert and not me because you live in the community I’m trying to work with.” (Investigator)

“Our research team has seen the value [of community feedback] even going to the IRB, saying, ‘Well, we were advised to do this.’ And it seems to hold a different kind of weight.” (Research staff)
3D Limited use of community input Interviewer: “Is there anything from the community consultation that’s really changed the way that you’re rolling out [the study]?”Investigator: “Yeah, I think the consent process has been modified … we figured out after a short while that the families have the closest relationship with the nurses at the bedside and with the doctor who’s involved in the care … And we figured out that we’ll do a lot better if they sort of back us up and help to introduce us … So that that was an example of something we modified early on as we learned, what was working and what was not.”
Interviewer: “Did any of those discussions prompt changes in the way that the study was designed or approaches to any aspect of [the study]?”Research staff: “I don’t remember a case when it did make a difference, which either means we’re not thinking deeply enough or that we planned pretty well in the first place.”
3E Importance of accountability for addressing community input “In a different study … it was clear that the advisory board individuals felt that their feedback was not being heard or applied … They were like, essentially, ‘Why are we giving feedback that’s not being implemented?’ … We need to be very transparent and make sure that the entire team [understands] this is valuable stuff … We’re not asking questions just to hear ourselves think … that we work hard to implement [their ideas]. And if we can’t implement [community feedback], explain why. And just keeping that communication open … It’s definitely a relationship that’s built on trust, public humility, going back and forth, and just being very honest and transparent.” (Research staff)
“There’s vehicles for doing the [engagement] work. And … these vehicles are only as good as the folks that drive them. I mean, you can have these community advisory boards and if you don’t use them and they are just to check a box, you don’t have the right folks at the table, or you’re not committing the right folks, then hell, it’s just something that you put in grant, you put in your documents … I have had folks reach out… They wanted access so I can get them to the community, but there’s no real commitment in doing things; just to kind of check the box and throw your name down there to add to credibility …you know, you got to be committed to … doing the work.” (Investigator)

Other key aspects of how CE was implemented revolved around whether community members were recognized as experts, and how this was communicated (3C). Relatedly, community advisors often did not know what happened to their feedback, whether it was explicitly considered in decisionmaking, and what impact their involvement had for how the study was conducted. For their part, researchers also often struggled to recall how their studies used the community input that had been collected, and the examples of study changes they gave were relatively minor and proscribed (3D). Thus a final point of emphasis in the conduct of CE was the importance of accountability to the community, and whether their input was built into and impacted how studies were conducted (3E). Everyday decisions and the minutiae of study implementation both reflected and shaped where power was located in the organization, workflow, and decisionmaking processes, and whether those study infrastructures allowed for collaboration and partnership, or continued hierarchy.

What Is Community Engagement For?

Researchers and funders gave variable and sometimes contradictory responses to questions about the purpose and ultimate impact of CE (Table 4). In many instances, we noted that CE was invoked in the context of facilitating research, fostering study “awareness,” doing “outreach,” and recruitment and retention of participants (4A) (see also Jeske et al. 2022). Such understandings of the nature and goals of CE gave it a transactional cast (4B), that it was a set of instrumental activities that could be quantified and measured by the number of individuals “touched” and recruited. These assessments of what constituted ‘successful’ CE were encapsulated in what one research team member referred to as the “bean counting” nature of engagement.

Table 4.

Perspectives on What Community Engagement Is For

Summary Illustrative Quotes & Examples
CE for Study Facilitation
4A CE in the service of study procedures “The engagement is seen as really closely tied to recruitment and enrollment … generating interest and enthusiasm for the [study].” (Research staff)
4B CE as instrumental, transactional “The metrics are basically, did 30 people come to your barbecue?” and “the number of people who we engaged … the people who got materials… Did people sign up?” (Research Staff)
CE for Community Benefit
4C CE to benefit community health “You have to understand how does being in a community, and the interactions, and the connections that people have, and the conversations and the thought processes that individuals have in the community … understanding that so that we can disseminate and develop interventions…You can’t just go and say, ‘Here’s cardiovascular disease information. Take it and run with it.’ … Understanding how you take uptake, retention and really how people take action on the messages. So the goal is to really develop sustainable intervention … And changing behavior.” (Research staff)
“I’m hoping that [the investigators] are trying to always … ask … ‘What does this mean to the community that I’m serving that I’m trying to impact and how do we get there? How do we get that brief or that publication transferrable to where it’s palatable that it can be given to the community in some form or fashion? How can it be disseminated and translated?’” (Investigator)
4D Community-centered research questions “Working with those communities to design a project that is going to include them and things that are of interest to them. So, what are they going to get out of this? Is there something that their community is particularly interested in studying?… If people were saying … this is a question we really want to know the answer to … then … the community [could be more] willing to contribute the data to answer that question… The resource would become richer. I think that the science that was being done would be more diverse, and this process would become more transparent, and therefore engender greater trust which would bring more people into [the study] over time … There’s just … a lot of knock-on effects there.” (Research staff)
CE for Building Relational Capital
4E CE to cultivate & maintain long-term relationships “I think the biggest thing to me is funding community-based research that persists over time…what we hear over and over and over again for our more diverse advisers is you can’t just pop into a community, give them results, and then leave … Increasing funding to maintain relationships over time so that you are able to stay connected to communities and it’s part of your work, I think, is the most important thing … That’s difficult when you have competing people for grants and you might have one grant and then you don’t get the grant the next time … Funding more cohort studies [so] that there’s expected funding over time to keep people engaged, and funding outreach and community engagement is probably something that I would just think would be most helpful.” (Investigator)
“I don’t think that [the researchers] were really equipped for how long and how deep the effort is needed to create through community engagement over time.” (Research staff)

In contrast, other participants, often (but not only) those involved in community-based studies, conceptualized CE in broader terms, focusing instead on how it might fundamentally embed community priorities and concerns in the research. One way in which some of our interviewees interpreted the imperative to translate research findings with and for communities was a focus on how study findings in the form of interventions and programs would fuel health related changes (4C). Others emphasized an even more transformational move—one that is well known to those who do community-based participatory or community-engaged research—that involves shifting what scientific questions guide research and who gets to decide (4D).

Finally, a third way some researchers and funders understood the purpose of CE was as long-term investments in relationships with and changes in communities (4E). In these conceptions, our interviewees emphasized that the return, rather than being measured in near-term (and easily quantifiable) increases in recruitment or even translations of study findings into better community health, may only be apparent years after an individual study has concluded. Given this, multiple interviewees expressed frustration with the limitations of the grant-funded model for CE in PMR, in which researchers cultivated community connections that served their purpose for one grant, but could not be sustained across studies.

“Pantomime Community Engagement”

How research teams made choices about who to engage, what engagement activities to support, how to solicit and integrate community input and/or decisionmaking into the workflow of the study, and what CE was for had significant consequences for the time and resources devoted to CE. Indeed, these choices were often conditioned upon upstream perceptions and decisions about study goals, competing priorities, and resource availability. We tracked, across many of the studies we followed, how constrained time and resources cascaded into tradeoffs that culminated in what one participant called “pantomime community engagement,” that is, performative practices that imitated CE (Table 5).

Table 5.

Experiences of “Pantomime Community Engagement”

Summary Illustrative Quotes & Examples
5A Insufficient resources for CE “It’s like you have a period of time to recruit what you can recruit … There is no patience from funding groups to the groups that are reaching out to the [community] partners, to these populations. It requires patience. It requires trust-building. It requires relationships … [But] that has not been my experience. You don’t have the luxury, you don’t have the time and resources to continue and maintain relationships with these communities.” (Funder)
5B Perceived tensions between science & CE “I totally don’t think that we all recognized … the level that we would have to do [CE] to keep the numbers up… But those are kind of two distinct things that I think at times are in tension with one another.” (Research staff)
“Somebody [in study leadership] was like, ‘This money, we don’t spend that on engagement’ … [CE] wasn’t a valued activity … She’s like, ‘Why would we spend money on engagement? We should spend it only on science’ … We have no science unless you have engagement … The people that are actually responsible for outreach and engagement are having to fight every day for every dollar.” (Research staff)
5C “Checkbox” CE “They’re mostly paying lip service to meaningful inclusion … We’re not doing the work.” (Research Staff)
“[A researcher] mentioned like ‘We’re doing so much to engage the community,’ because that’s what they want to hear. But in reality, we just don’t have the manpower to do that.” (Research staff)

Many participants expressed frustration with the lack of time and funding to properly prepare and conduct engagement activities before the press of recruitment, enrollment, and data collection all begin. Almost all research staff we spoke to said that they were not sufficiently funded to undertake CE activities in meaningful or fulsome ways (5A). Investigators talked about tough decisions when teams needed to cut budgets, often resulting in CE and diversity-related objectives being curtailed because, simply put, such activities were expensive and did not count as budgetary priorities because they were not seen as part of the science (5B).

The overall effect of this lack of time and funding to do CE is that many were not conducting what even their own staff would regard as purposeful CE. Instead, our interviewees spoke of a kind of ‘check box’ attitude where studies considered CE to be complete once certain tasks (e.g., meeting with community members, getting feedback on some study aspect or materials) had been checked off rather than when the underlying goal of more deeply engaging communities was achieved (5C). At times, capitalizing on the ambiguity and variability of what constituted engagement described above, activities were retroactively labeled as being CE when they were not originally conceived as such. Participants understood such actions to stem from a belated recognition of the importance of CE and the perceived need to demonstrate that CE was being done, when it had not been intentionally planned or resourced. Some of our interviewees were deeply concerned about the potentially serious consequences when studies do not or feel unable to invest in cultivating long-term relationships with communities and conducting research that benefits them. As one research staff member put it: “That’s the covenant between participants and researchers, and we have to fulfill our promise or else, what are we even doing? … Why bother?” Otherwise, she warned, “it is one giant data grab. It is any possible data that could be gathered from a human being, all of those data, every single piece of data that can be gotten. All of it.”

Discussion: Time, Equity, and Trust

Cumulatively, the sense of urgency, insufficient funds, the priority accorded to preserving “the science,” and the subsequent ways that engagement was done in improvised, sporadic ways, all exact downstream costs. Those involved in PMR frequently experienced CE and the research itself as two separate tracks that competed for resources and often could not be coordinated in a sequence that made sense: the research could not, or would not, wait for the engagement. Not only did “the science” dictate the pace that engagement efforts must follow, but structures and practices were not in place to enable engagement to impact the science. Under these conditions, CE may in fact lead to harms, as Brunger and Wall (2016, 1870) note: “for communities feeling the effects of research fatigue, the ‘good’ of community engagement may lead to harms to overstretched and under-resourced communities; community engagement takes work.” We therefore note three takeaways from our study:

Commitment to CE requires new temporal expectations:

Changing the temporal constraints on funding was seen as especially key. Research staff talked about building in engagement before the study and continuing it after its conclusion. Others spoke of the necessity of investing in sustainability beyond single studies, making “longitudinal goals” and “historical investments” with respect to CE. These experiences echo the guidance offered by many CE scholars: Wilkins and Alberti (2019, 6), for example, argue that research funding should “specifically include opportunities for community engagement, allow expenditures that are aligned with community partners’ needs and expertise.” However, a fair number of our participants sense that science is proceeding as usual, while, as one put it, community researchers “go their own pathway, trying to use duct tape and string to get access to resources and … create science that’s meaningful to their community.”

CE to achieve inclusion and equity requires rethinking altruism and benefit:

Our findings interrogate the current ethical paradigm for thinking about research participation as acts of altruism in a voluntary gesture without expectation of direct tangible benefits (Lee 2021). That is, while research may have societal and downstream benefits to future individuals, participants themselves will likely not benefit, and this understanding has structured and shaped ideas about fairness and reciprocity between researchers and participants. However, goals of diversity and inclusion of underrepresented communities calls into question whether this framework is sufficient. Instead, the research relationship—and research itself—require reconsideration.

Research infrastructures must be rebuilt, not to promote efficiency, but to support meaningful research relationships:

One alternate approach is to give up efficiency in return for deep engagement, reflecting an altogether different politics of knowledge—not science as usual, but research practices premised on equity and shared power and decisionmaking, among community members, other stakeholders, and scientists. This is the spirit in which Wilson et al. (2019, 94) call for “authentic relationship with the community” that demands “listening, observing, and participating in the community in such a way that we can understand and articulate its needs, allowing those needs to shape the research agenda.”

In a similar vein, Adams, Burke, and Whitmarsh (2014, 180; see also Alleva 2006, Müller 2014) open a conversation about “slow research” and what kind of science might serve global health goals:

We are all being asked to be productive in ways that create a sense of having to do more and to do it faster … These demands are not simply a result of how we conceptualize efficiency or capacity. Rather, they are directly related to problems of funding (capital), globalization (scale), and quality and focus (method). A call for slow research interrogates these demands. Slow research is not about doing less over time, although there is a temporal concern … It entails working with an ethic or set of values and strategies that valorize different things from the emergent norms. Slow research calls for a deliberate shift in the way we do our work and the ways in which that work and its products are valorized.

In these conceptualizations, research starts with the local—being there together with communities, the importance of the particular and specific details and histories of places and peoples. This attention to the local is also in relationship to knowledge, such that we understand knowledge to be situated in, produced by, and make sense within the local context. Slow research also asks that we resist the celebration of speed and efficiency and utility, so that we might see the benefits of extended time, long relationships, and deep understanding. Citing the time and resource intensive nature of CE, Pratt and de Vries (2018) acknowledge this as potentially having “the perverse effect of slowing research or preventing otherwise worthwhile studies from being conducted”; however, they emphasize that such attention to the process of conducting health research contributes equally to the quality of the knowledge it generates as well as to its being just.

These calls for an alternate approach to research introduce necessary questions: How much is science willing to relinquish? And what is at stake if we continue with the science status quo? The heterogeneity we found in our participants’ conceptions of what CE is for and what it ought to accomplish indicates that researchers and funders of PMR hold wide-ranging ideas about their roles and obligations in relation to CE. The often ad hoc ways in which decisions about CE practices were made suggest that research teams were either less than reflective about how CE was enacted and the consequences of those choices, and/or unable to implement CE in ways that they believed were appropriate, meaningful, or ethical. And the occasional retroactive labeling of activities as CE when they were not initially defined as such both illustrates an understanding of the importance of CE but also constitutes a problematic practice that can undermine trust and short-circuit more intentional strategies for conducting it. Our findings thus reveal that while there is already some willingness to fundamentally change how PMR is done, we face a significant gap in what it will take to make this practically possible. Closing this gap requires fundamentally remaking our infrastructures that support PMR and indeed, biomedical research more generally.

PMR persistently, if not uniquely, valorizes big data that is efficiently collected and well harmonized. This understanding drives the formation of large research consortia, because they enable the amalgamation of big data and create organizational structures within which specific and different data might be reconciled, erasing the local. Recent mandates for PMR studies to reflect the diversity of the US are rooted in the argument that doing so will yield data that can be truly useful for all populations and diverse scientific needs. Yet if PMR were to continue practices of “pantomime” and check-box CE, this could deeply undermine the legitimacy of this argument. As Juengst (2003, 196) points out, “to the extent that [CE] is used as a moral placebo to give the impression that ‘every effort’ is being made to protect and respect genetic populations … however well intended, it becomes a misleading, self-defeating attempt to achieve an inappropriate goal.”

A choice to pursue “normal” PMR without deep CE thus risks our continued ability to include diverse populations, to forge new relationships with communities, to doing things differently. Alternatively, to realize the potential for meaningful diversity and inclusion, and for PMR to be conducted in equity-promoting ways, there must be a willingness to entertain tradeoffs between what conventionally counts as ‘scientific progress’ and CE goals that might well challenge those assumptions. That is, we argue that PMR must reimagine the goals of scientific discovery in ways that center community interests. This requires profound shifts in community engagement practices, towards approaches that share power and decisionmaking in the conduct of precision medicine research.

Acknowledgments:

We are grateful to our study participants for sharing their time, perspectives, and experiences with us during observations and interviews. We also appreciate the input and collaboration of Aliya Saperstein, Caitlin McMahon, and Larissa Saco. Research reported in this publication was supported by the National Human Genome Research Institute of the National Institutes of Health under award number R01HG010330. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Footnotes

Conflict of Interest:

The authors declare that they have no conflicts of interest.

Contributor Information

Janet K. Shim, Department of Social and Behavioral Sciences, University of California, San Francisco

Nicole Foti, Department of Social and Behavioral Sciences, University of California, San Francisco.

Emily Vasquez, Department of Sociology, University of Illinois-Chicago.

Stephanie M. Fullerton, Department of Bioethics & Humanities, School of Medicine, University of Washington

Michael Bentz, Division of Ethics, Department of Medical Humanities and Ethics, Columbia University.

Melanie Jeske, Institute for the Formation of Knowledge, University of Chicago.

Sandra Soo-Jin Lee, Division of Ethics, Department of Medical Humanities and Ethics, Columbia University.

References

  1. Adams Vincanne, Burke Nancy J., and Whitmarsh Ian. 2014. Slow research: Thoughts for a movement in global health. Medical Anthropology 33: 179–97. 10.1080/01459740.2013.858335. [DOI] [PubMed] [Google Scholar]
  2. Agongo Godfred, Debpuur Cornelius, Amenga-Etego Lucas, Nonterah Engelbert A, Kaburise Michael B., Oduro Abraham, Ramsay Michele, and Tindana Paulina. 2021. Community engagement and feedback of results in the H3Africa AWI-Gen Project: Experiences from the Navrongo Demographic and Health Surveillance site in Northern Ghana. AAS Open Research 4: 15. 10.12688/aasopenres.13081.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Alleva Lisa. 2006. Taking time to savour the rewards of slow science. Nature 443, no. 7109: 271. 10.1038/443271e. [DOI] [PubMed] [Google Scholar]
  4. Beaton Angela, Hudson Maui, Milne Moe, Ramari Viola Port Khyla Russell, Smith Barry, Toki Valmaine, et al. 2017. Engaging Māori in biobanking and genomic research: A model for biobanks to guide culturally informed governance, operational, and community engagement activities. Genetics in Medicine 19: 345–51. 10.1038/gim.2016.111. [DOI] [PubMed] [Google Scholar]
  5. Blacksher Erika, Hiratsuka Vanessa Y., Blanchard Jessica W., Lund Justin R., Reedy Justin, Beans Julie A., Saunkeah Bobby, et al. 2021. Deliberations with American Indian and Alaska Native people about the ethics of genomics: An adapted model of deliberation used with three tribal communities in the United States. AJOB Empirical Bioethics 12: 164–78. 10.1080/23294515.2021.1925775. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Brunger F, and Wall D. 2016. “What do they really mean by partnerships?” Questioning the unquestionable good in ethics guidelines promoting community engagement in Indigenous health research. Qualitative Health Research 26: 1862–77. 10.1177/1049732316649158. [DOI] [PubMed] [Google Scholar]
  7. Burgess Michael, O’Doherty Kieran, and Secko David. 2008. Biobanking in British Columbia: Discussions of the future of personalized medicine through deliberative public engagement. Personalized Medicine 5: 285–96. 10.2217/17410541.5.3.285. [DOI] [PubMed] [Google Scholar]
  8. Charmaz Kathy. 2014. Constructing Grounded Theory. 2nd ed. Thousand Oaks, CA: Sage. [Google Scholar]
  9. Cheung Alexander T. M. 2018. Including everyone but engaging no one? Partnership as a prerequisite for trustworthiness. The American Journal of Bioethics 18: 55–7. 10.1080/15265161.2018.1431711. [DOI] [PubMed] [Google Scholar]
  10. Cohn Elizabeth Gross, Husamudeen Maryam, Larson Elaine L., and Williams Janet K.. 2015. Increasing participation in genomic research and biobanking through community-based capacity building. Journal of Genetic Counseling 24: 491–502. 10.1007/s10897-014-9768-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Dheensa Sandi, Samuel Gabrielle, Lucassen Anneke M., and Farsides Bobbie. 2018. Towards a National Genomics Medicine Service: The challenges facing clinical-research hybrid practices and the case of the 100 000 Genomes Project. Journal of Medical Ethics 44: 397–403. 10.1136/medethics-2017-104588. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Dry Sarah M., Garrett Sarah B., Koenig Barbara A., Brown Arleen F., Burgess Michael M., Hult Jen R., Longstaff Holly, et al. 2017. Community recommendations on biobank governance: Results from a deliberative community engagement in California.” PLOS ONE 12: e0172582. 10.1371/journal.pone.0172582. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Fohner Alison E., Volk Kyle G., and Woodahl Erica L.. 2019. Democratizing precision medicine through community engagement. Clinical Pharmacology and Therapeutics 106: 488–90. 10.1002/cpt.1508. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Folayan Morenike O., Oyedeji Kolawole S., and Fatusi Olawunmi A.. 2015. Community members’ engagement with and involvement in genomic research: Lessons to learn from the field.” Developing World Bioethics 15: 1–7. 10.1111/dewb.12020. [DOI] [PubMed] [Google Scholar]
  15. Gichuru Evans, Kombo Bernadette, Mumba Noni, Sariola Salla, Sanders Eduard J., and van der Elst Elise M.. 2018. Engaging religious leaders to support HIV prevention and care for gays, bisexual men, and other men who have sex with men in coastal Kenya. Critical Public Health 28: 294–305. 10.1080/09581596.2018.1447647. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Grayson Shira, Doerr Megan, and Yu Joon-Ho. 2020. Developing pathways for community-led research with big data: A content analysis of stakeholder interviews. Health Research Policy and Systems 18: 76. 10.1186/s12961-020-00589-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Haldeman KM, Cadigan RJ, Davis A, Goldenberg A, Henderson GE, Lassiter D, and Reavely E. 2014. Community engagement in us biobanking: Multiplicity of meaning and method.” Public Health Genomics 17: 84–94. 10.1159/000357958. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Jeske Melanie, Vasquez Emily, Fullerton Stephanie M., Saperstein Aliya, Bentz Michael, Foti Nicole, Shim Janet K., Soo-Jin Lee Sandra. 2021. Beyond inclusion: enacting team equity in precision medicine research. PLoS One 17: e0263750. 10.1371/journal.pone.0263750. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Juengst Eric T. 2003. Community engagement in genetic research: The ‘slow code’ of research ethics?” In Populations and Genetics: Legal and Socio-Ethical Perspectives, edited by Knoppers Bartha M., 181–97. The Netherlands: Martinus Nijhoff. http://brill.com/view/book/edcoll/9789047402930/B9789047402930_s010.xml. [Google Scholar]
  20. Key Kent D., Furr-Holden Debra, Lewis E. Yvonne, Cunningham Rebecca, Zimmerman Marc A., Johnson-Lawrence Vicki, and Selig Suzanne. 2019. The continuum of community engagement in research: A roadmap for understanding and assessing progress.” Progress in Community Health Partnerships: Research, Education, and Action 13: 427–34. 10.1353/cpr.2019.0064. [DOI] [PubMed] [Google Scholar]
  21. Lee Sandra Soo-Jin. 2021. Response to Open Peer Commentaries: Distinguishing the ‘gift’ from ‘donation’ as a path toward reciprocity and relational ethics. The American Journal of Bioethics 21: W1–3. 10.1080/15265161.2021.1905102. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Lemke Amy A., Wu Joel T., Waudby Carol, Pulley Jill, Somkin Carol P., and Trinidad Susan Brown. 2010. Community engagement in biobanking: Experiences from the eMERGE network.” Genomics, Society and Policy 6: 50. 10.1186/1746-5354-6-3-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Lyles Courtney R., Lunn Mitchell R., Obedin-Maliver Juno, and Bibbins-Domingo Kirsten. 2018. The new era of precision population health: Insights for the All of Us Research Program and beyond. Journal of Translational Medicine 16: 1–4. 10.1186/s12967-018-1585-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Marsh Vicki M., Kamuya Dorcas M., Mlamba Albert M., Williams Thomas N., and Molyneux Sassy S.. 2010. Experiences with community engagement and informed consent in a genetic cohort study of severe childhood diseases in Kenya. BMC Medical Ethics 11: 13. 10.1186/1472-6939-11-13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Molyneux Sassy, and Bull Susan. 2013. Consent and community engagement in diverse research contexts: reviewing and developing research and practice: Participants in the Community Engagement and Consent Workshop, Kilifi, Kenya, March 2011. Journal of Empirical Research on Human Research Ethics 8: 1–18. 10.1525/jer.2013.8.4.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Moodley Keymanthri. 2017. Legitimacy, trust and stakeholder engagement: Biobanking in South Africa.” Asian Bioethics Review 9: 325–34. 10.1007/s41649-017-0035-7. [DOI] [Google Scholar]
  27. Moodley Keymanthri, and Beyer Chad. 2019. Tygerberg Research Ubuntu-Inspired Community Engagement Model: Integrating community engagement into genomic biobanking. Biopreservation and Biobanking 17: 613–24. 10.1089/bio.2018.0136. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Müller Ruth. 2014. Racing for what? Anticipation and Acceleration in the work and career practices of academic life science postdocs. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research 15. 10.17169/fqs-15.3.2245. [DOI] [Google Scholar]
  29. Murtagh Madeleine J., Minion Joel T., Turner Andrew, Wilson Rebecca C., Blell Mwenza, Ochieng Cynthia, Murtagh Barnaby, Roberts Stephanie, Butters Oliver W., and Burton Paul R.. 2017. The ECOUTER Methodology for stakeholder engagement in translational research. BMC Medical Ethics 18: 24. 10.1186/s12910-017-0167-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Need Anna C., and Goldstein David B.. 2009. Next generation disparities in human genomics: concerns and remedies. Trends in Genetics: TIG 25: 489–94. 10.1016/j.tig.2009.09.012. [DOI] [PubMed] [Google Scholar]
  31. O’Daniel Julianne M., Rosanbalm Katie D., Boles Larry, Tindall Genevieve M., Livingston Troy M., and Haga Susanne B.. 2012. Enhancing geneticists’ perspectives of the public through community engagement. Genetics in Medicine 14: 243–9. 10.1038/gim.2011.29. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. O’Daniel Julianne M., Ackerman Sara, Desrosiers Lauren R., Rego Shannon, Knight Sara J., Mollison Lonna, Byfield Grace, et al. 2022. Integration of stakeholder engagement from development to dissemination in genomic medicine research: Approaches and outcomes from the CSER consortium. Genetics in Medicine 24 (5): 1108–19. 10.1016/j.gim.2022.01.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Popejoy Alice B., and Fullerton Stephanie M.. 2016. Genomics Is failing on diversity. Nature 538, no. 7624: 161–4. 10.1038/538161a. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Pratt Bridget, Cheah Phaik Yeong, and Marsh Vicki. 2020. Solidarity and community engagement in global health research. The American Journal of Bioethics 20: 43–56. 10.1080/15265161.2020.1745930. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Pratt Bridget, and de Vries Jantina. 2018. Community engagement in global health research that advances health equity. Bioethics 32: 454–63. 10.1111/bioe.12465. [DOI] [PubMed] [Google Scholar]
  36. Reynolds Lindsey, and Sariola Salla. 2018. The ethics and politics of community engagement in global health research. Critical Public Health 28: 257–68. 10.1080/09581596.2018.1449598. [DOI] [Google Scholar]
  37. Richardson Emma Z.L., Bandewar Sunita V.S., Boulanger Renaud F., Mehta Rukshan, Lin Tinya, Vincent Robin, Molyneux Sassy, Goldstone Arisa, and Lavery James V.. 2021. Addressing diversity and complexity in the community engagement literature: The rationale for a realist review. Wellcome Open Research 5 (March): 1. 10.12688/wellcomeopenres.15525.2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Sabatello Maya, and Appelbaum Paul S.. 2017. The precision medicine nation. Hastings Center Report 47: 19–29. 10.1002/hast.736. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Thompson Sanders, Vetta L, Ackermann Nicole, Bauer Kyla L, Bowen Deborah J, and Goodman Melody S. 2020. Strategies of community engagement in research: Definitions and classifications. Translational Behavioral Medicine 11: 441–51. 10.1093/tbm/ibaa042. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Seeley JA, Kengeya-Kayondo JF, and Mulder DW. 1992. Community-based HIV/AIDS research--Whither community participation? Unsolved problems in a research programme in rural Uganda. Social Science & Medicine 34: 1089–95. 10.1016/0277-9536(92)90282-u. [DOI] [PubMed] [Google Scholar]
  41. Tindana P, Campbell M, Marshall P, Littler K, Vincent R, Seeley J, de Vries J, Kamuya D, and the H3Africa Community Engagement Working Group. 2017. Developing the science and methods of community engagement for genomic research and biobanking in Africa. Global Health, Epidemiology and Genomics 2. 10.1017/gheg.2017.9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Tindana Paulina, de Vries Jantina, Campbell Megan, Littler Katherine, Seeley Janet, Marshall Patricia, Troyer Jennifer, et al. 2015. Community engagement strategies for genomic studies in Africa: A review of the literature. BMC Medical Ethics 16: 24. 10.1186/s12910-015-0014-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Tuttle Katherine R., Knight Richard, Appelbaum Paul S., Arora Tanima, Bansal Shweta, Bebiak Jack, Brown Keith, et al. 2021. Integrating patient priorities with science by community engagement in the Kidney Precision Medicine Project. Clinical Journal of the American Society of Nephrology 16: 660–8. 10.2215/CJN.10270620. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Versfeld Anna, Scheibe Andrew, Shelly Shaun, and Wildschut Janine. 2018. Empathic response and no need for perfection: Reflections on harm reduction engagement in South Africa. Critical Public Health 28: 329–39. 10.1080/09581596.2018.1443204. [DOI] [Google Scholar]
  45. Wilkins Consuelo H., and Alberti Philip M.. 2019. Shifting academic health centers from a culture of community service to community engagement and integration. Academic Medicine : Journal of the Association of American Medical Colleges 94: 763–7. 10.1097/ACM.0000000000002711. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Wilson Wylin D., Jackson Faye Hall, and Harrell Joan R.. 2019. Framework for Ethical Community Engagement (ECE) with underserved populations in the rural South: A help for bioethics and healthcare promotion. Journal of Health Care for the Poor and Underserved 30: 91–104. 10.1353/hpu.2019.0120. [DOI] [PubMed] [Google Scholar]

RESOURCES