Evaluating Systemic Changes to Support Clinical and Translational Health Research

Abstract

In evaluation research, “programs” are often conceptualized as clearly bounded, narrow in scope, focused on specific outcomes, using a well-defined linear causal model, and hence, suitable for standard evaluation methods. The evaluation work reported here was carried out in a more challenging context, where large, complex, interwoven systems were targets for change as a means to influence a diffuse array of outcomes. Our evaluation of an NIH-funded program to improve statewide infrastructure for clinical and translational health research (“Advance-CTR”) used qualitative data provided by investigators who used the program’s services, were funded awardees, or were members of an internal advisory committee (leadership representatives from partnering institutions). We examined perceived barriers to systemic changes to enhance research, as well as how systems have changed due to the Rhode Island Advance-CTR program’s efforts, to what degree, and with what effects. Using the causal logic of our program to connect these more distal systemic outcomes to the services and components of Advance-CTR, we discuss the effects this program has had on researchers and their environments, contributing to the development of sustainable programs of research that ultimately improve the health and well-being of our state’s residents.

Introduction

In his landmark “NIH roadmap” paper, Zerhouni (2003) advocated for increased emphasis on accelerating the impact of scientific investigations on improving health. Moving basic science from “the bench to the bedside” is a central aim of clinical and translational research but the process is not straightforward (Fudge et al., 2016). The National Institutes of Health created specialized Clinical and Translational Science Awards (CTSA) and Clinical and Translational Research (CTR) programs in 2007 (National Institutes of Health, 2007) to promote the development of infrastructure and human resources required to conduct clinical and translational research. Intended outcomes include an enhanced ability of institutions and investigators to develop competitive clinical and translational research programs, with increased and sustained collaboration and coordination of clinical and translational activities, and ultimately, an impact on health (National Center for Advancing Translational Sciences, 2021).

Rhode Island’s Advance-CTR program was established in 2016 by a National Institute of General Medical Sciences (NIGMS) Institutional Development Award (IDeA). Advance-CTR is a collaborative enterprise between two universities, three independent health care systems, and the Rhode Island Quality Institute, which manages the state’s health information exchange. This program is comprised of a set of required Cores. Each Core has well-defined activities directed at clear and logical outcomes. These include pilot awards for early stage clinical researchers; professional development and mentoring connections; consultations for research design, statistical analysis, and proposal development; access to health-related databases; community engagement and outreach; an administrative core to coordinate all of this; and an evaluation core to assess project processes and outcomes.

Aims for our evaluation draw directly on the expectations for evaluation of IDeA-CTR projects as presented in the funding opportunity announcement from NIGMS, moving from process evaluation of the activities of the cores; to proximal effects directly due to funded core activities as they are integrated into the existing research support structures of the collaborating institutions; and on to more distal effects -- sustainable changes in the collaborating institutions and the links between them that can potentially have longer-term impacts on health outcomes in the state. We believe this program logic requires a multifaceted examination of program impacts that recognizes the temporal aspect of the causal process, calling for evolving evaluation priorities and strategies as the earlier, more directly assessible changes play a role in the longer-term effects on infrastructure for translational research, i.e. increased organizational capacities, inter-institutional policies, and new mechanisms to enhance collaboration and community engagement.

Individual project elements with specified program activities and clear near-term effects fit into a relatively straightforward program evaluation context, while longer-term goals represented in our program’s logic pose a greater challenge for evaluation. These longer-term goals include developing infrastructure to augment the translation of science and enhancing the ability of institutions to sustain a clinical and translational research enterprise. With the components of our evaluation reported here, we were exploring the qualitatively perceived effects of both the shorter-term proximal systemic changes having to do with incorporation of added resources tied directly to Advance-CTR Core activities, and the more distal systemic impacts leading to reduced institutional barriers for the conduct of research. Although the latter must be seen as still developing due to the relative infancy of Advance-CTR, a means for understanding and measuring their development and effects were an important goal of the work reported here. At this point in the evolution of Advance-CTR, our research aimed to address the current level of perceived impact at both levels, along with the remaining barriers to overcome.

The rationale for these foci was informed by our own needs assessment (Willey et al., 2018), conducted in the first year of the project, and subsequent use of Group Concept Mapping (Kogut, Fede, Hayward, & Stevenson, 2019; Fede, et al, 2021), conducted in the second year of the project, to examine the concerns and ideas of investigators and research administrators in Rhode Island. We found that, in addition to the lack of pilot funding and protected time for research, the most frequently cited barriers to translational research included a need for data analysis support services, pre- and post-award grant administration support, and inter-institutional IRB collaboration. When asked what they considered most important for improving the quality and quantity of CTR in Rhode Island, respondents’ top-rated concerns were improving inter-institutional collaboration and building connections between researchers. The ratings differed between the various organizations partnering with Advance-CTR, suggesting priorities and needs varied among investigators from different settings.

The work reported here was intended to be responsive to these concerns by delineating qualitative perspectives on project effects addressing research support needs at both the proximal and distal levels described above. More specifically, we wanted to investigate 1) focused, practical aspects of systems changes targeted by the core activities; and 2) broad, sustainable changes in policies and infrastructure as more distal, still developing, outcomes. We envisioned that investigators would provide essential feedback to enable an understanding of how the services and trainings provided by Advance-CTR have improved the capacity for research at the investigator level. We also believed that those in leadership roles at Advance-CTR partner institutions could provide further insights about the effects of Advance-CTR on organizational research infrastructure. Our own evaluation feedback was also intended to play a role in affecting broader systemic change, and we discuss that along with major challenges that we have encountered in seeking a better understanding of how these changes come about. Like other evaluators confronted by complex developmental changes intended to affect interacting systems, we also sought ideas for new domains for metrics for that context and intended this qualitative exploration to help with that goal.

Methods

Using qualitative data provided by three distinct stakeholder groups: service users, funded awardees, and members of the project’s Internal Advisory Committee (leadership representatives from the participating institutions), we examined to what degree Advance-CTR’s efforts have changed the systems supporting translational research in Rhode Island, including but not limited to the resources, processes, and culture for conducting translational research at the partner institutions. These three stakeholder groups comprise the majority of Advance-CTR’s “customer base”, and as such, their insights are very useful for ascertaining whether the goals of the CTR are being attained.

We conducted this evaluation in two phases: Phase one was conducted by interviewing service users and awardees. Their experiences with Advance-CTR were similar in that they were consumers of CTR services and recipients of its funding and therefore, could respond with first-hand knowledge of the research infrastructure at their institutions and the support/resources provided by CTR. The second phase was conducted with the program’s Internal Advisory Committee (IAC). These respondents were not direct consumers of CTR services or funding, but were advocates of our program’s efforts at their institutions and could speak to systemic change occurring at both the proximal and distal levels of impact. The IAC had also previously been a target for our own attempts to use evaluation feedback as a lever to promote those more distal and complex systems changes, which made it a particularly relevant source of information on the extent of those changes and remaining barriers.

Phase One

Service User Interviews.

In the fourth and fifth years of our initial five-year award, participants for these interviews were self-selected from the eligible pool of all Advance-CTR service users (n = 300) by responding to an email request soliciting their participation. Those who utilized at least one of the three Advance-CTR service cores (Biomedical Informatics and Cyberinfrastructure Enhancement (BMI); Clinical Research Design (CRC); Biostatistics, Epidemiology and Research Design (BERD)) within the past four years were eligible. Twenty-six people responded to this email request, and all 26 were then contacted by Advance-CTR staff about potential dates and times for an interview. Nine individuals declined to participate in an interview; thus, individual interviews were conducted with 17 service users by two trained facilitators.

The interviews were held by video conferencing or telephone, with interviewees acknowledging their consent to participate by reviewing a description of the study and related consent information. Participants then clicked a hyperlink to join the videoconference appointment. Before starting the interview, the facilitator verbally confirmed the interviewee had read and understood the consent document provided in the link for the session.

These interviewees (service users) were diverse in their levels of research experience, institutional affiliation, service use frequency, and other demographic data (see Table 1 for details).

Table 1.

Demographics of the Users of Advance-CTR Services who Participated in Individual Interviews

Gender	n	Rank	n	Service Core Used	n
Female	12	Assistant Professor	5	Biostatistics	13
Male	5	Associate Professor	5	BMI	6
Race/Ethnicity		Professor	3	CRC	2
White	11	Research Fellow/Associate	3	Institution
Not reported	4	Instructor	1	Lifespan	8
Asian	1	Degree		Care New England	4
Latinx	1	PhD	13	Brown	3
Black African American	0	MD	4	URI	2

Interviews lasted between 20 and 60 minutes (average length = 30 minutes) and were recorded with notes taken by the facilitators using a debriefing template that was developed prior to the conduct of the interviews. Among the questions asked were (see Appendix for all interview questions and debriefing template):

• How did the services you used make a difference in your research?

• How can we do better? What resources are still needed?

Awardee Interviews.

We also interviewed awardees who had received internal Advance-CTR support through one of the program’s award mechanisms (Pilot Projects Program, Mentored Research Awards, Big Data Pilot Projects). The pool of eligible awardees focused on those with award start dates at least one year prior to the interviews (n = 40) to ensure they had sufficient time to have conducted their research in order to have informed responses regarding their experiences with Advance-CTR. Of these 40 individuals, we selected a pool that represented awardees from the various institutions, levels of experience, award mechanisms, and would be of manageable size to conduct, resulting in 17 persons receiving a solicitation email requesting their participation in a group interview. Of these 17, nine indicated they were interested in participating in the group interview sessions. Due to institutional restrictions enacted in response to COVID-19, the group feedback session that was originally designed to be conducted in person was revised to a video conferencing format. Three group feedback sessions were conducted via videoconferencing with a total of seven persons who had received funding awards from Advance-CTR (two sessions had two awardees per session and one session had three awardees participate); all of which were conducted by the same primary facilitator.

The group interview participants ranged in levels of experience, were racially and gender diverse, and also varied by institution and award mechanism (see Table 2 for details).

Table 2.

Demographics of Awardees Participating in the Group Interview Sessions

Gender	n	Rank	n
Male	4	Assistant Professor	7
Female	3
Race/Ethnicity		Degree
White	4	PhD	4
Asian	2	MD	2
Unknown	1	MD/PhD	1

Sessions lasted approximately one hour (range: 55–90 minutes) and were also recorded with notes documented by the facilitator using a pre-designed debriefing template. Among the questions asked during these interviews were:

• What were you able to do as a result of the funding that you would not have otherwise been able to do?

• How did the award change your product and/or help you professionally?

• How can we do better? What resources are still needed?

• How has your research had (or will have) an impact on the health, health equity, and/or health disparities of Rhode Islanders?

Phase Two

Internal Advisory Committee (IAC) Input.

The project’s IAC is composed of 18 individuals in key administrative roles for supporting translational research at each of the six institutional partners that form Advance-CTR. In the spring of 2018 (the project’s second year), we had presented data to the IAC about researchers’ concerns regarding systemic barriers which emerged from a group concept mapping exercise (GCM) with investigators (Fede et al., 2021). Some of the strongest recommendations derived from the GCM dealt with “institutional” and “administrative” barriers – inter-institutional IRB processes, subcontracting, a need for “blanket agreements” to facilitate collaborations across institutions, and improved ways to identify and connect with potential collaborators from other institutions. Additional within-institution improvements included the need for additional or new organizational structures and processes such as more technical support for grant proposal development, and incentives for mentoring new researchers. These findings were used to stimulate a discussion among IAC members of ways in which the involved institutions could improve their research support structures beyond the directly funded activities generated by the grant.

Two years later (during the fourth year of the grant award), IAC members were asked to complete a pre-meeting worksheet (see Appendix) to give their views of progress at their institution over the past five years and the impact of Advance-CTR at their institution. Example items were informed by the group concept mapping results of year two and included, Reducing barriers to conducting research (e.g., administration, pilot funds) and Improving structures for supporting internal multi-disciplinary collaborations. This worksheet served as the basis for a 60-minute discussion, led by the Tracking and Evaluation Core Director, during which nine members of the IAC who were in attendance shared their insights and perspectives. IAC members were also encouraged to take time to think about the items presented and add to the worksheet as needed. This worksheet was emailed to IAC members in a follow-up message and encouraged those members who were not in attendance to provide their input. The IAC members’ feedback, both oral and written, provided the data for this facet of the study.

Analysis

Although explicit coding of Phase One and Phase Two participants’ responses was not conducted due to the a priori framework provided by the interview questions, commonality and coherence of emergent concepts and themes were analyzed initially by two of the authors. For the service user and awardee interviews, data were triangulated using the debriefing templates as the primary source for data analysis with the audio recordings utilized for additional clarification or information using an open categorization framework for each interview question. Transcripts of the interviews were printed and common themes were given a one- or two-word descriptor with supportive sentences noted. These broad categories, ideas, or topics were then compared and when divergent themes were noted between these two authors, two additional authors examined the data and consensus was accepted as the standard. The questions asked of these participants were intentionally quite direct and limited in scope (e.g., How did the services used make a difference in your research? What resources do you still need?); therefore, responses were likely to align with the content areas identified by the authors prior to these interviews (e.g., “service satisfaction”, “resources”). Moreover, participants had had a pre-existing relationship with Advance-CTR and were thus familiar with its resources and aims. This limited the themes that were likely to emerge. From the contributed responses, quotes were identified to exemplify themes within these areas, with unique experiences also noted.

The worksheets completed by the IAC members and meeting notes from their session were also analyzed in a similar fashion to delineate themes in response to each of the interview questions posed. Within the a priori frame provided by the eight open-ended questions, all of the authors of this paper provided input to the inductive development of themes/concepts that linked concepts across the individual questions, and also served as “checks” to the analysis of these data. When we undertook an integration of the data across the three separate sources, we chose an overarching pair of intentional categories to organize all of the content: proximal effects directly linked to grant-funded activities, and sustainable distal effects (achieved or not) at the organizational and inter-organizational levels. Thus, our analysis was driven by the specific questions and letting themes emerge within these categories from the responses provided by all three sources.

Results

Of primary interest in this study were responses by service core users, funded awardees, and IAC members’ addressing the expansion of research support infrastructure through increased organizational capacities, new mechanisms to promote collaboration or community engagement, and inter-institutional policies that are intended to enhance clinical and translational research that improves health, health equity, and/or health disparities in Rhode Island.

Proximal Impacts: Direct impact of grant-generated activities on research support infrastructure:

All of the awardees, as well as several of the service core users and members of the IAC, stated Advance-CTR had a high impact via core-provided interventions on the translational research infrastructure in Rhode Island. Examples given included small pilot grants, career development assistance, and Advance-CTR-sponsored, skill-building trainings and consultative services. These were seen as particularly valuable as demonstrated by the following insights:

• “Seed funding really helped protect my time from having more clinical and administrative duties until I had more significant funding afterward.”

• “I don’t think I could have gotten IRB approval without their [Advance-CTR] help.”

• “My team got better access to computing resources, data, and consultations on grant writing.” IAC members suggested that, although not solely attributable to Advance-CTR, external funding at their institution had increased and Advance-CTR had played a role. For instance, the Pilot Projects Program was specifically mentioned as a vital pathway for the collection of pilot data which led to more significant support. Assistance provided by Advance-CTR for resubmission efforts was also noted as an example of an institutional gap being filled by Advance-CTR. Overwhelmingly, IAC member responses were very positive about the current usefulness of support structures and processes facilitated by Advance-CTR, such as mentor training, pilot funding, and statistical consultation. IAC members also noted the value of the many professional development opportunities, such as Good Clinical Practice, Public Responsibility in Medicine and Research (PRIM&R), and Society of Clinical Research Associates certifications.

Distal Impacts: Indicators of improvements in long-term organizational and inter-organizational structures and policies

New mechanisms to promote collaboration and community engagement.

One of the most salient impacts Advance-CTR has had on the awardees, service users, and their partner institutions has been the development of better ways to facilitate collaborations among researchers. As one person reported, “CTR has enabled inter- and intra-institutional collaborations that were not possible before CTR was funded. We’ve established rigorous methodological cores that weren’t here before. There were no standard places to go. It’s been invaluable.” Another awardee stated, “I interacted with a lot of cores and people, including [name], who is now a mentor on my K application. I got to know people through CTR.” Similarly, an awardee stated that Advance-CTR support helped her better understand the infrastructure of her institution, and gave her the confidence to develop external collaborations, commenting, “I really appreciate CTR’s encouragement and facilitation of seriously interdisciplinary collaborations. Being with [hospital] and [university]--they are extremely challenging systems to navigate--and I now have the confidence to do so.” Finally, one IAC member noted a change with longer-term implications was an expanded organizational commitment to seek research-intensive faculty for clinically-oriented programs.

Remaining barriers and potential improvements.

According to our respondents, there is still room for improvement, particularly for the more distal outcomes. IAC members discussed the significant institutional barriers to cross-collaborations and felt some uncertainty about what may be helpful in the felling of those barriers, and in particular what role, if any, Advance-CTR might be able to provide in facilitating better cross-institution collaboration by “flattening” barriers. When asked to elaborate, the top issues involved how funds are allocated across institutions and IRB approval. Advance-CTR has been working on the latter issue from the outset, but with limited success. As one service user noted, “Coordination between IRBs is the hardest nut to crack, the biggest waste of time, and a difficult issue for junior investigators who don’t have a big staff to handle this dysfunctional craziness. Anything that CTR can do to take the edge off of that would foster greater collaboration within and between institutions. Not any one IRB – they are totally siloed –overarching coordination/standardization is needed.”

Many of the participants also mentioned ways in which they thought Advance-CTR could work to improve collaborations and community engagement. As mentioned by one service user, “It is hard to establish relationships with hospitals, especially as a new faculty member. Advance-CTR was able to help connect me with clinicians and as a new faculty member that was critical. It was also suggested that junior researchers who had been successful (e.g., R01 level funding) need help building on that success moving their lab to center funding. This could be a resource for Advance-CTR as these newly established investigators are potential mentors and collaborators for beginning researchers. The development of more stable research collaboratives is one promising aspect of system change, begetting a need for new metrics in this area.

Evolving structures for community engagement represent another promising area for the development of additional outcome metrics. Because many of the Advance-CTR partner institutions have already developed community advisory boards, sometimes competing for members, Advance-CTR could aim to develop more effective and efficient ways to coordinate links between particular areas of academic research expertise and related community health concerns.

Several comments by these interviewees pointed to the long-term problem of sustainability. How will services, training, and internal pilot research funding mechanisms be financially supported in the absence of IDeA-CTR funding? As shared by one awardee, “The question really is - how do you create vibrant cores? You need money and time.” While another acknowledged the sustainability concern in this way, “Advance-CTR is an important safety net for my research. It’s a really helpful service that I hope continues. It will harm the career trajectories of young, talented investigators if this resource goes away.” The need for evolution of internal and inter-institutional policies and funding mechanisms was acknowledged, with one interviewee stating, “We’ve created the expectation that there is this collaborative space. In the next five years we should focus on, how do we turn this space into transformational work? That will be critical to maintaining the credibility of the cores and moving research forward.” Others saw potential for improvements (e.g. building support for statistical consultation into grant proposals), yet also expressed frustration with attempts to reduce these institutional barriers. Furthermore, there was also an appreciation of the differences across institutions in the levels of need in particular areas of support to provide sustainable improvements in research infrastructure and productivity. Specifically, one user expressed concern about women and under-represented minorities (URM) in this way, “CTR needs to invest in those pipelines early when women and URM fall out of the system. CTR has been helpful to me but there are definitely some unmet needs in that group. You haven’t targeted those groups as much as you could.”

COVID-19 effects:

Several of the interviewees were expanding their research endeavors to include the impact of COVID-19 on the state’s subpopulations, pointing to another place where Advance-CTR could develop new kinds of assistance. As one person stated, “Even though I focus on digital health, almost all studies have some in-person component. It would be great if CTR could put together some information for conducting studies remotely or how to salvage studies that have been impacted [by the pandemic].”

Discussion

In evaluation research, “programs” are often conceptualized as clearly bounded, narrow in scope, focused on specific outcomes, using a well-defined linear causal model, and hence relatively straightforward for evaluation. The evaluation work reported here illustrates our approach to more challenging circumstances, in which large, complex, interwoven systems were targets for change as a means to influence a diffuse array of outcomes. Early in our project we compiled research stakeholder perspectives on barriers and needs to feed into the change process, and followed up with qualitative explorations of the extent of improvement at two levels of anticipated effects – proximal system adaptations directly associated with grant-funded activities, and distal longer-term and more fully integrated systems changes offering the possibility of sustained structural support for clinical and translational research in our state. By examining current progress and barriers for the more distal systemic outcomes related to the services and components of the Rhode Island Advance-CTR, we aimed to contribute to the development of resilient programs of research that ultimately improve the health and well-being of our state’s residents, and in particular, those deemed most vulnerable.

We asked Advance-CTR awardees, service users, and Internal Advisory Committee members whether the range of Advance-CTR activities had benefited institutional systems. Although our sample sizes are small, the qualitative data provided rich sources of insight and the experiences reported were remarkably consistent across these participants. Participants felt Advance-CTR had increased the likelihood of health-related research being funded, promoted health-related research mentoring, provided more opportunities for multi-disciplinary collaborations, and aided in reducing some institutional barriers to conducting research. Advance-CTR’s funded services (biostatistics, biomedical informatics, clinical research support) have, over time, become well-established, effective, and integrated into the participating organizations. Advance-CTR’s award programs have filled critical gaps by providing pilot funding and technical support services that were previously unavailable on a statewide level. Increased support for multi-disciplinary and multi-institutional collaborations was reported by both researchers themselves and the organizational leaders from our IAC sample.

However, we also learned that much remains to be done. Many participants noted that despite the progress made during the first years of Advance-CTR, the program’s promise to impact community engagement and outreach, establish policies to facilitate research across institutions, and reduce significant barriers to conducting health-related research remains largely unfulfilled. Community engagement was a consideration of Advance-CTR’s pilot funding decisions during its first four years, but there was no dedicated Core. Project evaluation has attempted to use these various reported perspectives to inform Core leaders of potential interventions into the “systems” implicated for change. Several strategies have been employed by the evaluators to increase the salience of these efforts and to facilitate opportunities for discussion and planning for action by those in positions to devise feasible solutions, implement them, and make these changes. Evaluation feedback conveying concerns raised by investigators has stimulated greater attention to these concerns within the Advance-CTR structure, and via the institutional representatives on our IAC. As noted there have already been some documented effects at the level of sustainable inter-institutional policies.

Recognizing the challenges of changing complex systems drew us to the qualitative methods on which we have reported here. Each organization partnering with Advance-CTR has its own internal set of stable pathways and processes for developing research proposals, identifying promising opportunities for external funding, constructing multi-investigator collaborations and subcontracts, funding new work, assuring compliance via IRB review, etc. Within organizations, those pathways and processes have been developed over time to meet the varying needs of constituencies and individuals within research-related organizational components. Well-established feedback loops maintain existing institutional processes and structures. Although the new statewide infrastructure made possible by the CTR award has facilitated research collaborations and increased research output across the state, it has primarily led to the development of temporary new accommodations in order to absorb the funding, both within and between the represented organizations (i.e., subcontracting awards, allocation of resources, and skills-sharing).

Those kinds of changes are apparent in our data; however, changing longer-term trajectories will require continuing attention to reduce the likelihood of return to prior ways of operating. As reported by interviewees, one promising avenue for sustainable change appears to be the development of research “labs” or centers that may themselves develop cross-institutional connections and methods for mentoring new researchers, facilitating IRB review, negotiating subcontracts, establishing inter-institutional infrastructure for data sharing, and establishing standard operating procedures and IRB Authorization Agreements (IAAs), as examples. New positive feedback from success in achieving external funding can expand these positive outcomes of Advance-CTR support.

A future direction for project evaluation addresses institutional structures for coordinating clinical and translational research endeavors with community health concerns and engagement with community perspectives throughout the research process, to ensure that programs of research emanating from our IDeA-CTR are achieving genuine impact within the communities they aim to serve. Additionally, adding institutional programs and services for coordinating mid-level researchers to ensure they remain successful in their research enterprise will provide opportunities for collaborations, mentorship, and overall sustainability to the institutional infrastructure already established by Advance-CTR. Finally, the continuation of the institutional services and award mechanisms currently in place is essential to the success of the individuals, both junior and veteran researchers alike, associated with Advance-CTR.

These qualitative results are also useful for identifying promising new metrics for focusing on the infrastructure change level as we go forward. In particular, means for tracking the development of research labs and centers will take us beyond the individual level of one-off inter-researcher collaborations. Additionally, measures of increased efficiency in use of community representatives to facilitate long-term community engagement can add to our understanding of how best to make our evaluation relevant for improving systemic effectiveness in support of the conduct and use of research that will ultimately improve health outcomes in our state.

Lessons Learned

Large, complex organizational structures can be positively influenced by an infusion of resources targeting infrastructure development, but sustainability is more challenging. A logic model connecting more proximal and directly addressable changes with longer-term intended structural outcomes can usefully inform both evaluators and project leaders. Early examination of stakeholder-perceived needs and barriers can be fed back into the change process and later used to organize follow-up evaluation of changes and remaining issues. Stakeholders at multiple levels within the targeted organizations have differing perspectives and roles for the change process, all of which are worthwhile for engagement by evaluators.

Critical voices missing from our interviews were those of patients, community organization leaders and staff, and members of the local communities we serve. In future iterations of our efforts we hope to include these perspectives as the views of these stakeholders are vital. We also did not include the staff of Advance-CTR as data sources. Often, these individuals are the ones doing the critical work of connecting researchers to services and potential collaborators, and aiding them in navigating the intra and inter-institutional systems. The inclusion of their opinions of the work and processes would also make for a more robust and complete picture of the impact and efficacy of the Center; thus, we heartily recommend others in similar programs to encompass these perspectives in their scope of work. Moving forward, we intend to incorporate those voices, add them to our evaluation feedback along with continuing feedback from our primary researcher clientele, and also add new metrics more sensitive to both the community engagement process and the research infrastructure advances that our program.logic envisions.

Highlights.

• Examination of challenges for strengthening clinical and translational research infrastructure across state

• Qualitative interviews conducted with technical support service users, pilot research award recipients, Internal Advisory Committee members

• Identification of sustainable improvements at institutional and inter-institutional levels

• Impact on community engagement and outreach still needed and difficult to evaluate

APPENDIX A: INTERVIEW SCRIPTS

Service User Interviews (Individual)

1. What service(s) did you use?

1. How did it help?

2. What did you expect before the consultation?

3. Did the services meet your expectations? If so, how, if not, why?

4. Did the service change what you planned to do?

2. How did the service(s) make a difference?

a. Prompts:

1. Performed data analysis

2. Provided guidance other mentors couldn’t

3. Helped me gain new skills (research methods, data management, statistical analysis, etc)

4. Reviewed the proposal

5. Connected me to new collaborators

6. Helped me navigate across institutions

7. Increase in score or improved score

8. Access to conferences, trainings, other resources

3. How can we do better?

1. What resources are still needed?

2. Would you use us again? Why/why not

3. Would you recommend our services to our colleagues? Why/why not

4. What other Advance-CTR offerings did you participate in as a result of the consultation?

1. Trainings, workshops, seminars

2. Symposia

3. Conferences (Association for Clinical & Translational Science, etc.)

5. How has your research had (or will have) an impact on health, health equity, and/or health disparities in Rhode Island?

• If so, how; if not, what’s needed?

• What’s next?

  • Grant applications, patents, publications, new collabs, new tech

  • Do you feel your chances of success are higher after working with Advance-CTR? Why/why not

Awardee Interviews (Group Sessions)

1. What did you expect before you received the award?

1. Did the funds, services, and resources provided by Advance-CTR through the award meet your expectations? Why/why not

2. What were you able to do as a result of the funding that you would not have otherwise been able to do?

1. Did you develop new skills as a result of the award?

   1. Improved my ability to conduct research (in what ways?)

   2. Qualitative: Interviewing, NVivo, etc.

   3. Biostatistics: REDCap, study design, analysis, software (Sas, R, etc)

   4. Informatics: Study design, management, analysis of big data sets, navigating URSA Stronghold environment/cross institutional, sensitive data, coding

   5. Clinical Research: Regulatory protocols (IRB, IACUC), using a biobank, grant writing (specific aims, hypothesis)

2. Multi-PI Pilots: How did you learn about team science as a result of the award?

3. PIs: How did your role as the contact PI help you grow as a leader?

   1. How did the award allow you to work across institutions?

      1. Infrastructure access

      2. Workflows/processes

      3. Collaborators

3. How did the award change your product and/or help you professionally?

1. Improved my ability to conduct research (in what ways?)

2. Connected me to new collaborators

3. Helped me navigate across institutions

4. Access to conferences, trainings, other resources

5. Strengthened relationships with mentors

6. Mentors/collaborators provided guidance others couldn’t

7. Received extramural funding

8. For GRA awardees: Improved score or increase in score

4. How can we do better?

1. What resources are still needed?

2. How was the pre-award process and closeout?

3. Communication with Advance-CTR and overall support you received?

2. How has your research had (or will have) an impact on health, health equity, and/or health disparities in Rhode Island?

1. If so, how; if not, what’s needed?

2. What’s next?

   1. Grant applications, patents, publications, new collabs, new tech

   2. Do you feel your chances of success are higher after working with Advance-CTR? Why/why not

How has Advance-CTR assisted you in addressing COVID-19 disruptions to your research? (maybe give some examples here if necessary: letters to program officials, navigating communications from funders, protocol changes)

How well did this assistance/help address your needs?

What additional ways can Advance-CTR assist you in addressing COVID-19 disruptions to your research?

APPENDIX B: Debriefing Template for Interviews

Interview ID:

Date:

Time:

Duration:

Location:

Facilitator:

Observer/Note taker (if there was one):

Tone of the interview:

How did it go? Was the conversation easy to have? What was the demeanor of the participant?

Ability to Complete Agenda/Agenda Adherence:

Did you get through all of the agenda? Were there interruptions? Was there anything you didn’t cover?

Interview Description:

What Worked Well:

Problems/Challenges:

Observations that will not be evident from reading transcript:

Lessons Learned:

Changes needed to the agenda/questioning strategies:

Issues to Follow Up With:

Note Taker’s Suggestions to Interviewer: (if appropriate)

Things learned that you didn’t know or think about before this interview?

Saturation?

SUMMARY of DISCUSSION

1. What service(s) did you use?

1. How did it help?

2. What did you expect before the consultation?

3. Did the services meet your expectations? If so, how, if not, why?

4. Did the service change what you planned to do?

2. How did the service(s) make a difference?

a. Prompts:

1. Performed data analysis

2. Provided guidance other mentors couldn’t

3. Helped me gain new skills (research methods, data management, statistical analysis, etc)

4. Reviewed the proposal

5. Connected me to new collaborators

6. Helped me navigate across institutions

7. Increase in score or improved score

8. Access to conferences, trainings, other resources

3. How can we do better?

1. What resources are still needed?

2. Would you use us again? Why/why not

3. Would you recommend our services to our colleagues? Why/why not

4. What other Advance-CTR offerings did you participate in as a result of the consultation?

1. Trainings, workshops, seminars

2. Symposia

3. Conferences (Association for Clinical & Translational Science, etc.)

5. How has your research had (or will have) an impact on health, health equity, and/or health disparities in Rhode Island?

• If so, how; if not, what’s needed?

• What’s next?

  • Grant applications, patents, publications, new collabs, new tech

  • Do you feel your chances of success are higher after working with Advance-CTR? Why/why not

How has Advance-CTR assisted you in addressing COVID-19 disruptions to your research? (maybe give some examples here if necessary: letters to program officials, navigating communications from funders, protocol changes)

How well did this assistance/help address your needs?

What additional ways can Advance-CTR assist you in addressing COVID-19 disruptions to your research?

APPENDIX C: IAC WORKSHEET

Background:

At the IAC meeting held on March 29, 2018, the Tracking and Evaluation core of Advance-CTR presented the top 10 themes generated from our concept mapping study involving researchers in RI, who were asked:

“How can we increase the quality and quantity of clinical and translational research in RI?”

Responses rated as most important are ranked to the right.

1. Provide seed grants to encourage new collaborations across basic science and clinical faculty

2. Have a common IRB and IACUC between Brown, Lifespan hospitals, and CNE hospitals

3. Create blanket collaboration agreements between institutions within the RI CTR network

4. Reduce administrative barriers to collaboration

5. Improve support for grant writing

6. Improve pre-award support for grant applications

7. Create recurring networking opportunities that connect researchers from different domains

8. Create a unified statewide directory of researcher expertise and projects

9. Connect researchers with common interests

10. Ensure that junior faculty have experienced mentors with a track record of funding

Agenda for Our Upcoming IAC Meeting

In follow-up to our prior meeting, we will discuss progress towards the enhancement of research capacity at our partner institutions occurring since the inception of Advance-CTR in 2016. Please consider the items below and be prepared to discuss your responses during the meeting. Indicate your view of A) the magnitude of progress at your institution (or unit) during the past 4 years, and B) how Advance-CTR has contributed to these items. Space is provided for personal notes.

Name: ____________________________________ Affiliation: __________________________________

Item	A. Your view of progress at your institution during the past 5 years:	B. Your view of the impact of Advance CTR at your institution:
Increasing the likelihood of health-related research grants being funded
Reducing barriers to conducting research (e.g. administrative, pilot funds)
Promoting health-related research mentoring
Showcasing health research results
Engaging the community
Improving structures for supporting internal multi-disciplinary collaborations
Developing cross- institutional policies to facilitate research
Other areas (write in):