Lessons Learned While Starting Multi-Institutional Genetics Research in Diverse Populations: A Report from the Clinical Sequencing Evidence-Generating Research (CSER) Consortium

Abstract

Background:

Increasing diversity in clinical trial participation is necessary to improve health outcomes and requires addressing existing social, structural, and geographic barriers. The Clinical Sequencing Evidence-Generating Research Consortium (CSER) included six research projects to enroll historically underrepresented/underserved (UR/US) populations in clinical genomics research. Delays and project re-designs emerged shortly after work began. Understanding common experiences of these projects may inform future trial implementation.

Methods:

Semi-structured interviews with six CSER principal investigators and seven project managers were performed. An interview guide included questions of research/clinical infrastructure, logistics across sites, language, communication, and allocation of grant-related resources. Interviews were recorded, transcribed verbatim; transcripts were analyzed using inductive coding, thematic analysis and consensus building.

Results:

All projects collaborating with new clinical sub-sites to recruit UR/US populations. Refining trial logistics continued long after enrollment for all projects. Themes of challenges included: sub-site customization for workflow and genetics support, conflicting input from participant advisory groups and approval bodies, developing research personnel, complex data management structures, and external changes (e.g. subcontractors ending contracts) that required redesign. Themes of beneficial lessons included: domains with prior experience were easier, develop project champions at each sub-site, structure communication within the research team, and simplify research design when possible.

Conclusions:

The operational aspects of expanding clinical research into novel sub-sites are significant and require investment of time and resources. The themes arising from these interviews suggest priority areas for more quantitative analyses in the future including multi-institutional approval policies and processes, data management structures, and incremental research complexity.

Background

Lack of diversity in clinical research limits improvements in health outcomes for all populations. Decades after the National Institutes of Health (NIH) formally prioritized inclusion of historically underrepresented or medically underserved (UR/US) populations in clinical research [1], barriers to participation still exist. Numerous attempts to reach new populations demonstrate that researchers must proactively develop strategies to overcome existing social, structural, and geographic barriers [2–4], While lack of diversity among individuals who participate in research is not unique to the genomic medicine context, the implications for key outcomes such as clinical utility are especially staggering. In a study of 1800 variants in genomic databases, over half came from individuals of European ancestry, other ancestral groups represented under 15% [5]. This unbalanced representation makes identification and interpretation of clinically relevant genetic variants in non-European ancestry populations far less accurate [6]. Consequently, increasing diversity in genomic research is a priority in the genetics community [7].

The Clinical Sequencing Evidence-Generating Research Consortium (CSER), multi-institutional collaborative research program funded by the National Institutes of Health, focuses on implementing genomic technology in clinical settings[8]. The second phase of CSER prioritized enrollment of UR/US populations in its request for applications by expecting projects to “recruit a minimum of 60% of patients who come from racial or ethnic minority populations, underserved populations, or populations who experience poorer medical outcomes”[9]. In 2017, six projects were funded for this second phase of the consortium, each project with its own clinical indications for genomic testing [10, 11] but with similar operational aspects at the local, multi-institutional, and consortium levels (Figure 1). Each project included a multi-disciplinary core of investigators and research staff responsible for the design, implementation, and oversight of individual projects and cross-consortium research questions. Projects were at the interface of research and clinical care at multiple sub-sites; five projects closely integrated into clinical care and one project relied on clinicians to identify and refer potential participants to a web-based research interface. All projects formed new collaborations with clinical sub-sites to reach UR/US populations. Finally, research participants interacted with research through four basic steps: 1) participants were identified, screened and enrolled if eligible and willing; 2) specimens were collected for genomic studies and delivered to the project-specified clinical laboratory(ies) for processing and analysis; 3) results of genomic tests were returned to the participant and clinical provider; 4) participants completed project-specific and consortium-level surveys.

Figure 1.

Organizational and operational structure of Clinical Sequencing Evidence-Generating Research Consortium projects

Delays and project re-designs emerged shortly after funded work began on these projects. Cross-consortium survey development and standardization posed challenges in the first year [12], but these did not fully account for the scope of concerns investigators raised in meetings or reported to consortium leadership. We report findings from interviews with CSER principal investigators (PIs) and Project Managers (PMs) to make our experiences useful for future research efforts aimed to enroll historically UR/US populations.

Methods:

We conducted semi-structured interviews with key informants from each CSER project using an interview guide (see Appendix). The interview guide was developed by a study team of CSER consortium investigators and program managers (PM) with experience in clinical trial design, management and health care systems. Concepts relevant to project start-up or implementation were identified from review of the publicly-available portions of submitted quarterly progress reports, observations from CSER meetings, and lived experiences of the study team. The interview guide included open-ended questions and detailed probes regarding the following topics: definitions of project start-up; setting up the research/clinical infrastructure; logistics within the main site and across sub-sites; language and communication with research participants; and allocation of grant-related resources. The guide also included questions about running large clinical trials during the COVID-19 pandemic, qualitative findings from which are presented elsewhere (under review). The interview guide was pilot tested among members of the study team and refined prior to conducting interviews.

In-depth interviews with project principal investigators (PI) and PMs were conducted via video conferencing and lasted 60–90 minutes each. The interviewer was a trained study team member from a different CSER project to allow collection of unbiased information and capture the detailed information that would be more visible to an outside observer than a member of the same project team. Interviews were recorded, transcribed verbatim (otter.ai, Los Altos, California) and checked for accuracy prior to analysis. Interview transcripts were analyzed using inductive coding and thematic analysis. After members of the coding team (HR, HSS, JB, NS, PM) each reviewed 2–3 transcripts independently, they held regular meetings to discuss suggested codes and develop a preliminary code book through a consensus approach. They finalized the codebook through an iterative review process. Each transcript was then coded by two members of the study team, with any discrepancies in coding resolved through deliberation and consensus-building. We developed themes based on codes and organized the themes into high-order domains. This project was considered exempt from review by the Baylor College of Medicine Institutional Review Board (IRB). Interviewee quotes are coded to maintain anonymity.

Results

Six PIs and seven PMs participated in interviews during summer and autumn of 2020. Because our study was design to understand both the common and unique experiences of the six studies to guide future collaborative research, our findings are organized into two domains, challenges and lessons learned, with multiple themes within each. Illustrative key words and quotes for each theme are presented in Table 1 and throughout the text.

Table 1.

Domains, Themes, and Illustrative Quotes:

Themes	Sample Key Words	Illustrative Quotes
Domain: Challenges
Multiple institutions	Research/Clinical Interface Clinicians Staff Physical Space Culture Payment	“the issues were different in two places. [Sub-site 1] was about payment. And at [Sub-site 2] it was because it’s a general hospital…they have a pediatric ward, but they don’t have someone who can draw blood at the outpatient … department.”(4–2) “And even engaging providers to do referrals…it took a little while to get into their clinical workflow, and to get them engaged, and trusting of the study staff and all that kind of stuff.” (5–1) “I think part of the training that went into the research coordinators, at least on our end, was like really training them to cultivate relationships with the clinic staff, and who could, you know, either facilitate our job or make it harder, in many ways…there’s always like a gatekeeper and you have to, make friends with the gatekeeper to make it easier…to do a lot of things that you have to do” (5–3) So what works at one clinical site doesn’t always work at another clinical site, as I’m sure you know… we’ve done a lot of trying to find what works at one site, and implement that as best we can at other sites, so that, you know, the testing can get to the right people. But that has proven to be a challenge, just based on the way that the hospitals work and things that are kind of downstream in (project). (6–2)
Genetics	Specimens Genetic councilors	“One of the little wrinkles in our project is that we have a duplicate sample that goes to our clinical genetics laboratory. So the pathology department has a molecular lab and the duplicate sample goes there so that we can do any clinical confirmation and reporting of findings so that we can have a CLIA sample and so that, you know, samples coming from these other institutions had to then get picked up from … and brought over, essentially ferried over to the molecular genetics lab.” (3–1) “So you know, one of those three [research coordinators] would collect the samples, and take it back to the lab. And the reason for that is, it was just harder to set up the billing and … it wasn’t straightforward to do … maybe one day… And so we literally used to transport them by hand.” (4–1) “We do duals or trios [genomic samples on biologic parents]…if they showed up it was just easy to get the samples…But we had to do a lot of calling and emailing, all communications, for the other parent to send their samples.” (4–2) “So it,… it took us a while to smooth out all the wrinkles there by virtue of the complexity of the sample flow.” (5–1)
Research Input and Approval	Harmonization Cross-consortium Advisory groups Institutional Review Boards Translation	“…adding that site took an inordinate amount of time. I mean, we just had many challenges. They wanted a separate IRB. We didn’t agree with that. But they really insisted. And then the IRB took a long time, there were millions of calls…we were able to access another population group and to add another provider. So I think one of the positives were worth it…I think I should have thought about it more, because I didn’t anticipate the degree and the work and time that it would take to be involved.” (4–1) “It goes beyond language, right? Like the culture and understanding where they’re coming from, and particular needs they have.” (5–3)
Personnel	Staffing Contracts Roles	“We needed to hire bilingual recruiters…both are the ones that we got were new to research. And so we spent a lot of time training them up on…basic research procedures, how to talk with people… how to follow all the rules and consent and those kinds of things…There aren’t a lot of bilingual folks in [site], especially not ones with research experience. And, you know, I don’t know if we’re unusual, but our recruitment positions are sort of entry level in terms of pay. And so even when you find people with the research experience, they wanted more than we were allowed to pay them.” (1–2) “We were a little delayed in our study database development because, until we actually received the funding, it’s hard to pay people for work…it would have been ideal to already have all that done at enrollment.” (2–2) “One of the challenges was the people we brought on board. And we’re, you know, taking the genetic counselors who were starting their first job. So part of the, you know, part of it was trying to create an environment where they had sufficient mentorship and their roles while being part of a research unit. “(5–1) “We would lose really good study coordinators… because they’re making career moves of their own. So trying to identify new study coordinators and get them motivated and interested, has been a little bit of a struggle, but somehow I think we’ve been able to overcome it for the most part.” (6–2)
Data Management Structures	Data Data Transfer Monitoring and Tracking	“We load [information into study-related data management tool]…once patients are deemed eligible by their risk assessment. And then they’re automatically loaded into tracking…if they are [sub-site A] patients, their electronic medical record data just automatically comes in as well. [Sub-site B] had to hand enter it behind the scenes, because we don’t have access to that… their system doesn’t allow us to have access to PHI prior to someone being consented…which I understand. But that was a big barrier, and certainly extra work for the [sub-site B] team. Now we did have patients hand-enter most of the data themselves at the time they started…like their phone number, and their date of birth and their address, and their medical record number. So we put quite a bit of the burden on the patient.” (1–2) “Maybe we’re at over 3000 variables in this database at this point? Yeah, it’s huge… we really rely on it in every aspect of our study” (5–2) “So everything that is entered … any phenotype the person has, or anything like that, is manually entered by the nurse coordinators. We kind of streamline the information that we get for analysis that way, but it does add an additional step for them. But since none of them use the same EMR this was a really good workaround, I think…getting the result into the medical record was a completely underestimated task. We, when we found that that had happened, we were backlogged a couple months, where they were like, Oh, yeah, I have these and they haven’t been scanned into the EMR. And we’re like…Can you just scan them? And we realized that that’s not how it works. That was a challenge. (6–2)
External Changes	Expectation Change	“Lots of turnover. And so almost felt like we kept having to start over to build the team, there was just no stability” (3–2). “And we were working with a company who had a tool that we were supposed to partner with to do this, but the company decided to divest in the development of that tool. So then that tool no longer became available to us…So we actually had to develop the tool ourselves. That was an unexpected software build” (5–1)
Lessons
Experience		“I actually will say one big thing, which we’re sort of trying to work on at [site], is having a patient stakeholder group ready to go, that any project could use because we did spend so much time trying to find those people that then once we finally got them, and we’re getting feedback, we had so little time to actually use their feedback. We did use their feedback, but not as well as we would have liked.” (1–2) “We certainly benefited greatly from the fact that her group already had the infrastructure in place to do that. And again, I think if we had had to come up with that ab initio, it would have cost even more. Yeah, that kind of infrastructure is really valuable to be able to build on top of.” (3–1) “I don’t think that, you know, there was any challenge doing due to being multi-disciplinary. I think most of the challenges was, we were all new and all new to our roles” (4–1) “We were very lucky because we weren’t starting from scratch… So there’s a stakeholder board that specifically was advising and engaging on genomic research in genomic medicine existing at [project], There was also another stakeholder group of primary care physicians…with particular emphasis on underserved populations was a second stakeholder group… So to some extent, we lucked out in the sense that we were engaging with groups who were very familiar with genomic research, very knowing of and experienced in sort of dealing with some of the issues that come up in genomic research in genomic medicine. And were already very educated and expert in some of these questions. And so we could really get like, you know, hit the ground running in terms of engaging. (5–1)
Project Champions	Appointed versus emerging Relationships	“We would have a key person at each site who was their site representative, and those people were checking in together. And that really helped smooth out some of the problems that we were having.” (1–1) And she’s … very committed to her patients, and she’s very well liked. Like everybody who meets her just loves her. So she can pull the strings in [sub-site] to get those people to support her to do this. So we’re, we’re taking advantage, … of her very specific situation that she can actually get this done in [sub-site]. (4–2) Our approach to the sites was to have a physician champion for each disease area at [each] site, so that they really could help communicate the goals of the project and … champion that and remind their own colleagues of the study. (5–2) “…definitely identify those champions at each university. I think that took a time… But I think now we kind of know at least the traits of what we want those people to have. And so kind of getting our collaborators to help us identify those people and making sure that they have enough time and effort to be able to spend on the project so that we can basically get a true coordinator at each site.” (6–2)
Communication	Interactions Teams	“I think it probably took us at least two years to…figure out really good working relationships across all of those things….Figuring out a way to communicate amongst all the workgroups in an efficient way was a real challenge. And also, in addition to 10 workgroups, we have 8 different sites, institutions involved…so just building a communication structure for that complicated of a project where all the pieces interconnect with each other was a huge challenge.” (1–1) “We had to create a forum for everyone to be contributing, listening to each other, understanding each other and then making decisions together. And plus add to that that these people come from different disciplines, social scientists and clinicians and lab people. So in the beginning even like among the smaller PI groups we had to do some team building. You know. ‘We can’t work like this like if you just compete with each other or if you um, don’t understand or know where its coming from.’… it was a lot of building the team and understanding.” (4–2)
Simplify	Design Uncertainty	“We all had to have these incredibly broad teams and lots of components and patient interaction stuff and clinical utility and right the whole ball of wax. But that being said, probably our tactical error was trying to have it all in one study… So that’d be the main thing is keep it simple.” (3–1) “We have 3 aims and we have sub-aims …And those are all independent groups…They enroll from their sites. And then we have the lab. We have the wet lab and then the dry lab because we also process everything.” (4–2) “And just, you know, we have so many consent forms in one language, then you double that with English and Spanish and just the sheer volume… of handling the IRB component was something else that I think … I wasn’t prepared for.” (5–2)

DCC: Data Coordinating Center; EMR: Electronic medical record; PHI: Protected health information

Project teams’ conceptualization of their project being “up and running” played out in multiple forms. One interviewee described their project as up-and-running when the main protocol was approved by the first IRB, another described this milestone as when work-groups were established within their project and they were “setting out what the work is going to be…and starting to do that work” (5–1). The remaining interviewees conceptualized up-and-running as when they enrolled the first research participant. Regardless of this anchoring definition, “up and running” seemed to be a fluid concept, especially with integrations of sub-sites, and all projects continued refining logistical aspects of project administration long after reaching this milestone.

Challenges

Multiple Institutions:

All projects formed new collaborations with the purpose of reaching UR/US populations, many of the collaborations were strategically selected because of their catchment of diverse populations. Examples of sub-sites included an urban community-based health facility, federally qualified health systems, and a community-based medical home, and rural hospitals. Some of the challenges encountered may be expected during the early phases of any multi-institutional clinical research project and each sub-site required customized approval, infrastructure development, and research and clinical workflow. However, sub-sites with high UR/US were often located in resource-limited healthcare systems and communities with less marginal capacity to support research efforts. Several interviewees reported initial plans to start all multiple sub-sites simultaneously but changed to sequential starts because of the effort required to adapt to each sub-site. Two projects halted collaboration with sub-sites that served predominantly UR/US populations before enrollment could begin because of extensive hurdles.

Genetics:

Performing genetic research testing requires specimens (blood and/or cheek swab) be collected from patients and transferred to appropriate laboratories. Some projects also requested specimen collection from biological parents and/or required specimens be routed to multiple laboratories. After testing, the genetic laboratory interpreted the results within clinical and research contexts. Each project included genetic counselors (GC) although the direct interaction with participants varied according to each project’s research question(s). Genetic-related patient-facing materials were developed to support participants and providers for each project as well. Inclusion of genetic support, specimen collection and shipment to testing laboratories, and returning genetic results required customization of workflows for each sub-site.

Input from Multiple Stakeholders:

Creating standardized tools to distill complex genetics and/or research topics into language accessible by participants with input from participant advisors while simultaneously addressing numerous IRB requirements was a source of delays and rework as cited by all projects (Figure 2). Each project included advisors representing their unique research population. Although the structure and scope of function of these groups varied, all projects sought input from their advisors for project-specific and consortium-wide patient-facing materials (e.g. consent forms, educational materials, surveys). Furthermore, every sub-site required customized IRB approval. Three projects used centralized IRB structures to streamline the approval processes at sub-sites, however, these projects encountered sub-sites not familiar with, comfortable with, or prepared for central IRB processes. For at least one sub-site, lack of experience of the IRB reviewers with genetics terminology and ethical concerns prolonged time to approval. “So, I think what I described as some of the slowness, and the slowness and the responsiveness of the IRB’s, and the back and forth with the IRB’s and things like that, that was a common thing across both sites.” (5–1) Time and patience were required to finalize the details.

Figure 2:

Simplified flow diagram of approval and translation of patient materials at multiple institutions

All projects translated their consent forms, surveys, recruitment fliers and other patient-facing materials into Spanish. Only one project submitted their English and Spanish materials to IRBs simultaneously, all other projects waited until English versions were approved at all sub-sites prior to translation. As a consequence, Spanish materials were not available for most projects at the same time as English materials; one project was completing their Spanish material translations at the time of these interviews, three years into project funding. Translation into additional languages was described as desirable but cost prohibitive. Projects relied on bilingual research staff to help with translations [13]. One site initially planned on translating the research materials in languages other than Spanish (e.g. Cantonese), but could not due to funding. “Working in different languages should not be an afterthought. It’s expensive… if you want to reach diverse populations then we need to be able to have more money or more money for translations.” (4–2)

Personnel:

The infrastructure for clinical research includes personnel to address regulatory bodies and policies, track participant enrollment and participation, and perform research-specific efforts (e.g. administering surveys, analyzing data). Projects were required to customize research infrastructure to each sub-site’s existing system. If a sub-site had clinical research personnel, they might have time to support this project but not be experienced in genetics. Conversely, a sub-site may have an existing clinical genetics team, but members of that team would not have adequate time to support research. All projects cited the need to hire research staff or allocate research effort for this project. Personnel turnover unrelated to the project was a common challenge resulting in delays while replacement personnel were identified. Funding personnel in settings where little or no established clinical research infrastructure was cited by multiple PI’s as a challenge because paying for a part of a person’s time was not a usual practice in many settings. “We’re budgeting for 50% of a project coordinator. Well, where’s the other half of that person’s salary coming from?” (3–1) An additional challenge noted by three projects hoping to enroll participants more comfortable in languages other than English was attracting, identifying, and hiring qualified bi-lingual or multi-lingual research personnel. Two projects described difficulties recruiting GCs to the UR/US facilities. “We tried [to hire Spanish speaking GC’s]. I don’t believe that that field is super diverse yet…” (5–2). Interviewees reported needing to provide significant training in research methods and scenarios where the core project members took on additional roles that could not be delegated to the UR/US sub-site because of staffing. Furthermore, these were ambitious projects unlikely to be performed unless externally funded and institutional policies may prevent opening and advertising new positions until funding was in hand potentially delaying hiring by months. “We had to hire a lot of people…I think I hired something like 18 people in 18 months…and that’s just at my research center…all the other sites are also having to hire as well.” (1–1)

Data Management Structures:

All projects developed data management structures customized to their research, a process considered complex and labor-intensive by all interviewees. Data collection fell into four main categories: 1) patient data including clinical features, phenotype, and socio-demographics, 2) genomics data including the raw genomics testing results and variant interpretation, 3) extensive survey data including multiple project-specific and consortium-wide surveys, and 4) event tracking including consents, enrollments, sample tracking, and longitudinal follow-up dates (Figure 3). Data management structures were developed at the primary project site with data shared in category-specific formats between the project, sub-sites, consortium, and funding organizations. Processes for collecting data and reporting genomic results were customized to workflows for each sub-site; interacting with sub-site electronic health records ranged from seamless integration to manual extraction and input. Data management structures were closely integrated with research infrastructures and communications with participants and providers. As projects became more complex, so did their data management structures. When changes occurred at the consortium level (e.g. a change in a consortium-level survey question) or in the requirements of a sub-site’s IRB, data management structures required revisions.

Figure 3.

Data management structures and transfer of data sharing requirements for each CSER project. EHR: electronic health record

External Changes:

Interviewees described changes unrelated to their project that significantly impacted their progress. The rapid pace of genomics research and clinical uptake also required new adaptions in the few years between grant writing, funding and formally launching these projects. “I think over four years, it’s a long time in genetics.” (4–1) For example, one project required redesign because the standard clinic workflow changed from delivering genomic results in-person to via telephone. In another project, a different genomics research project started at a sub-site during the grant review process creating competition for patients and resources and ultimately requiring study redesign. Other external changes were more unique. A technology subcontractor cancelled their contract to develop a patient education platform after funding started requiring the project to develop alternative solutions. Another sub-site institution underwent administrative acquisition and restructuring resulting in contract renegotiations and additional IRB review.

Lessons Learned Through Multi-site Project Implementation

Experience:

Whenever possible, projects leveraged existing infrastructure, technology and relationships. Four projects participated in the previous round of funding with CSER and described building off their existing research team and data management structures. Interviewees described domains where structures already existed (e.g. participant advisory boards) to be less challenging and more impactful than starting from scratch. Knowledge of the IRB process at a sub-site, either through previous experience or through identifying champions with experience, was reported as useful in facilitating the process.

Project champions:

A theme of developing “project champions,” either intentionally or organically, at sub-sites was commonly identified as a tactic perceived to be associated with success or in response to the prompt “what would you do differently in the future?” Project champions were described as people who passionately promoted the research and rallied others around it. Some project champions were formal research investigators or staff. Others were physicians, nurses, support staff, or in other roles, who were not formally funded by the project but were enthusiastic about the potential value of genomic testing and/or project success in their sub-site.

Communication:

These were large and complex projects with multi-disciplinary and geographic diversity of projects that led to slow collaboration in some projects. Structured frequent interactions was noted by several projects as beneficial.

Simplify:

In addition to the inherent complexity associated with research in genetics, UR/US populations, novel collaborations, and consortium-level harmonized data collection [12], individual projects designs were complicated. All had multiple project study questions related to genomics, clinical outcomes, and participant education/understanding. Some of these also involved randomization or stratification of participants. Interviewees noted that these multiple demands were difficult to address simultaneously and that this added to delays early on. Several noted they would make their projects far less complicated in hindsight.

Discussion

The CSER consortium took a deliberate step towards including historically UR/US populations in clinical genomics research [14]. Despite the significant previous clinical research experience by CSER investigators, these projects were still challenging in their early phases. This qualitative exploration of the challenges and successes of launching the CSER consortium projects demonstrate the investment required to perform multi-institutional human subjects research intersecting with clinical care. Genetics is a highly technical and nuanced field of medicine that carries additional ethical and data management challenges. While these specifics of the genetics field may limit the generalizability of some of our findings, the majority of our experiences are applicable to other human subjects research and are particularly relevant to expanding research into UR/US populations. Identifying the specific challenges CSER consortium investigators faced and how they addressed these challenges may guide the design, implementation, and funding of future clinical research hoping to recruit a more diverse participant population.

To reach new populations, CSER investigators collaborated with clinical settings delivering care to UR/US populations. Previous clinical research efforts have demonstrated that meeting populations in their communities helps overcome barriers of trust and geography, but requires significant research team time and flexibility [2]. The operational aspects of expansion into novel sub-sites is easily underestimated. We present here a qualitative analysis of our experiences which was not designed to compare the efforts required to start research in sites that serve UR/US populations versus more established research institutions. Our analysis suggests that every clinical sub-site required development of personal relationships and customized infrastructure and workflow design. However, our findings demonstrated these collaborations took significantly more resources and time than was expected by seasoned investigators or allocated in project budgets or the granting mechanism. Additional attention to and support of the logistics of appropriately performed human subjects research at multiple sites is necessary to encourage researchers to successfully include previously under-represented populations.

These interviews identified the cumulative effects of institutional barriers unrelated to genetics as a challenge of multi-institutional clinical research. Delays and redundancies are well known problems associated with numerous IRBs reviewing the same research protocol [15–17]. Starting in 2018, the NIH required the use of single (a.k.a. central) IRBs for funded research studies [18]. Three CSER projects attempted to use these in 2017, however, they found themselves educating sub-sites on the central IRB processes suggesting that implementing this policy will take time. Communication in languages other than English and incorporating feedback from stakeholders are elements of the US/UR framework developed by CSER [14]. While the focus of all sites was to enroll diverse populations, it is likely that the delays caused by the translation and approval processes [19] were counter-productive to fulfilling this goal.

Another institutional barrier was the inability to begin work until grant funding was in-hand. At some institutions this included posting for new positions or developing data management structures. Early delays in developing central project infrastructure will likely amplify downstream delays at sub-sites, especially at those sub-sites with less marginal support for research personnel. Failure to prepare for institutional delays may result in unrealistic expectations around time required to be “up and running.”

Clinical trials have become increasingly complex over time. As the investment required to start a trial increases, so have the number of procedures and secondary aims [20]. We identified multiple complexities that impacted our research progress worthy of consideration for future study design: combining advanced technologies, multiple novel patient populations, and clinical trials that incorporated both local and consortium-level components. Our analysis focused on the perspective of the projects and research sites, however, the consortium also faced challenges in developing centralized data management structures [21] and cross-consortium research questions [12]. Each additional increase in trial complexity creates additional challenges in an exponential rather than linear fashion [22]. It is possible that by trying to do too much, we cannot accomplish our primary goals.

As a research community we build on the shoulders of the giants before us. Experience was an important theme in these multi-disciplinary projects. Previous investment in infrastructure such as data management cores or previous consortium experience was reported by our interviewees as positive. Conversely, development of novel processes or relationships required significant effort. We do not mean to imply that only experienced groups should receive funding for future research, rather we believe this observation acknowledges that investment of time and resources to infrastructure development provides value beyond a single granting cycle. Alternative grant mechanisms that incorporate a feasibility or planning phase and allow for development of initial milestones before further implementation, such as the two-phase UG3/UH3 approach [23], are a step in the right direction.

Meanwhile, the efforts required to operationalize human subjects research is understudied. If we desire to impact the whole population, not merely advance our scientific technology, more attention to what it takes to perform such research well is necessary. The themes developed from these interviews suggest some priority areas for more quantitative analyses in the future including multi-institutional approval policies and processes, data management structures, and incremental complexity of research.

Abbreviations:

CSER

Clinical Sequencing Evidence Generating Research consortium

DCC

Data Coordinating Center

GC

Genetic Counselor

UR/US

Under-represented / Underserved