Making Contributions and Defining Success: An eDelphi Study of the Inaugural Cohort of CIHR Health System Impact Fellows, Host Supervisors and Academic Supervisors

Abstract

Context:

The Health System Impact (HSI) Fellowship, an innovative training program developed by the Canadian Institutes of Health Research's Institute of Health Services and Policy Research, provides PhD-trained health researchers with an embedded, experiential learning opportunity within a health system organization.

Methods/Design:

An electronic Delphi (eDelphi) study was conducted to: (1) identify the criteria used to define success in the program and (2) elucidate the main contributions fellows made to their organizations. Through an iterative, two-round eDelphi process, perspectives were elicited from three stakeholder groups in the inaugural cohort of the HSI Fellowship: HSI fellows, host supervisors and academic supervisors.

Discussion:

A consensus was reached on many criteria of success for an embedded research fellowship and on several perceived contributions of the fellows to their host organization and academic institutions. This work begins to identify specific criteria for success in the fellowship that can be used to improve future iterations of the program.

Introduction

Health systems of developed countries are facing complex and multifaceted challenges in meeting the needs of patients and providers. Sophisticated research and analytic expertise are critically needed to aid in the design, implementation and evaluation of evidence-based solutions (CIHR 2018c). A capacity-building initiative gaining international recognition is embedding PhD-trained researchers into health system organizations outside of traditional academic settings (Cheetham et al. 2018; Ghaffar et al. 2017; Marshall et al. 2014, 2016; Vindrola-Padros et al. 2017, 2019). According to previously published case studies, embedding academic researchers into health systems is a promising approach to facilitate the integration of evidence- and practice-informed research in policy implementation, build research capacity and meet stakeholder and patient needs (Cheetham et al. 2018; Ghaffar et al. 2017; Hamelin and Paradis 2018; Paradis et al. 2017; Vindrola-Padros et al. 2017). A more rigorous evaluation of the embedded research model is ongoing in the UK (National Institute for Health Research 2017).

In October 2017, the Canadian Institutes of Health Research's Institute of Health Services and Policy Research (CIHR-IHSPR) launched the Health System Impact (HSI) Fellowship. The HSI Fellowship is a $5.5 million national initiative to train the next generation of embedded researchers to promote health system transformation in Canada (Bornstein et al. 2018; Tamblyn et al. 2016). The HSI Fellowship aims to: (1) prepare PhD graduates for a wide variety of careers (both inside and outside of academia) by enriching post-doctoral training with opportunities for “real-world” experience within health system organizations and (2) demonstrate the value of adding PhD-trained researchers to organizations outside traditional academic settings. More information about this dynamic program is available in the CIHR funding opportunity (CIHR 2018c). A central facet to this program is the co-mentorship of fellows by high-level, senior leaders in both the academic and the health system setting.

Although embedded research training opportunities are gaining popularity, there is limited Canadian research evaluating the preliminary impacts of these fellows within the health system organization. To the best of our knowledge, only one narrative review has explored the role embedded researchers could play in quality improvement efforts in healthcare settings (Vindrola-Padros et al. 2017). Although the review included a wide array of organizations across the world, there was no Canadian representation, an exclusive focus on quality improvement and little emphasis on the training of early-career researchers. Therefore, more evidence is needed from the Canadian context to inform program development, assess the impact of embedded fellows on health systems and facilitate the success of future fellows.

This present study fills a gap in the literature by: (1) identifying the criteria of a successful fellowship as defined by HSI fellows, host supervisors and academic supervisors and (2) elucidating each group's perspectives on key contributions made by fellows to their respective academic and health system organizations after the first year of operation of the HSI Fellowship.

Methods

An eDelphi (Diamond et al. 2014; Hasson et al. 2000) study was conducted to elicit the perspectives of fellows, host supervisors and academic supervisors regarding: (1) the criteria that each group uses to define a successful fellowship and (2) key contributions fellows have made in their academic and health system organizations. The eDelphi method was chosen due to the distribution of fellows and their host organizations and academic institutions across Canada.

The entire 2017 cohort of fellows (n = 46), their host organization supervisors and academic supervisors were invited to complete the survey (CIHR 2018b). As the first cohort of this innovative program, participants were considered the experts of their experience. Data were collected through two rounds of online questions administered by CIHR-IHSPR. These were sent four weeks apart, with two weekly follow-up reminders sent to non-responders.

In Round 1, two open-ended questions were administered to each of the three stakeholder groups asking their views on the main contributions of the fellow to their host organization and the criteria they would use to define a successful fellowship. For each stakeholder group, a team of three researchers analyzed the qualitative data generated in Round 1 using content analysis to identify themes (Hsieh and Shannon 2005). Items to be included in Round 2 were developed by each team from the list of themes for their respective stakeholder group. Next, one researcher who was not part of any of the three qualitative analysis groups identified themes common to the three groups as well as unique themes in each group. Prior to distribution of the Round 2 survey, all the authors met to discuss the themes. From this discussion, it was decided that common themes would be presented to all three stakeholder groups in the Round 2 survey to facilitate comparison across the groups and themes unique to one group would be presented to that group only.

In Round 2, participants were asked to rank their agreement with the themes generated in Round 1 on a 5-point Likert-type scale (range: 1 to 5). Descriptive statistics (mean and standard deviation [SD]) were used to assess the level of agreement (item endorsement). Although there is no universally agreed upon proportion to define consensus, a conservative threshold of 70% (mean 3.5/5) was set a priori (Hasson et al. 2000). All quantitative analyses were performed using R software (R Core Team 2019). Ethical clearance for the study was approved by the University of Waterloo Ethics Review Board (#32105).

Results

Of 134 potential participants, 113 completed the first survey, resulting in an overall 85% response rate for Round 1 (fellows, n = 42/46 [91%]; host supervisors, n = 34/43 [79%]; academic supervisors, n = 37/45 [82%]). The Round 2 survey was only sent to individuals who participated in Round 1. In Round 2, 78 of 113 participants completed the survey, resulting in an overall 69% response rate (fellows, n = 36/42 [86%]; host supervisors, n = 21/34 [62%]; academic supervisors, n = 21/37 [57%]).

Criteria of a Successful Fellowship Experience

Round 1

Seven common criteria of a successful fellowship were identified across two or more stakeholder groups in Round 1 (Figure 1). In addition to common criteria across groups, some criteria of success were identified by only one group of stakeholders. Fellows identified better understanding of their desired career trajectory (employment preparedness) as a success criterion. Host supervisors identified four additional criteria, including capacity of adaptation to different working environments, openness to new learning experiences, team cohesion (collegial interaction) and positive experience by the fellow. Finally, academic supervisors considered measurable impacts within the health system and building a foundation for fellow research success as criteria for defining success.

Figure 1.

Shared themes of criteria of a successful fellowship experience identified by two or more stakeholder groups

HSIF = Health System Impact Fellowship.

Round 2

All common criteria of success reached consensus (Figure 2). Among the common success criteria, academic supervisors had the highest level of agreement on all items, followed by fellows and host supervisors, respectively, with the exception of better knowledge of health systems and the role of health policy. The sole criterion with higher agreement among host supervisors versus fellows was advancing the host organization's mandate or priorities.

Figure 2.

Level of agreement with identified criteria of success by stakeholder group

Pink = reached consensus, level of agreement of 3.5 or more.

Grey = did not reach consensus, level of agreement less than 3.5.

HSIF = Health System Impact Fellowship.

Among the unique criteria for success, only difficult to define success (mean 2.09; SD 0.87) did not reach the level of consensus. Fellows rated better understanding of their desired career trajectory to an average score of 4.22 (SD 0.87). The average scores of the host supervisors' specific criteria were 4.05 (SD 0.83) for capacity of adaptation to different working environments; 4.15 (SD 0.75) for openness to new learning experiences; 4.15 (SD 0.88) for team cohesion; and 4.40 (SD 0.80) for positive experience by the fellow. The mean scores for the academic supervisors' specific criteria were 3.52 (SD 1.03) for measurable impacts within the health system and 4.24 (SD 0.89) for building the foundation for fellow research success.

Key Contributions of the Fellows – Round 1

Seven common themes emerged among all three groups highlighting the key contributions of fellows to their respective organizations (Figure 3). In addition to the common themes, the HSI Fellowship impact project and grantsmanship (applying for or obtaining funding) were contributions identified only by fellows. The strong work ethic of the fellow was a contribution only identified by the host supervisors. Academic supervisors identified two additional contributions: revealing the strengths and weaknesses of current academic training and identifying challenges and barriers to learning health systems.

Figure 3.

Level of agreement with key contributions of fellows to their respective organizations by group

Pink = reached consensus, level of agreement of 3.5 or more.

White = did not reach consensus, level of agreement less than 3.5.

Round 2

The results of Round 2 are presented in Figure 3. All three stakeholder groups had the highest level of agreement on the same three common key contributions (although the order of rankings differed). These included: research/analytic skills and expertise, networking, building relationships and collaboration and new perspectives, interventions, processes, methods or strategies. In comparison with host supervisors, the fellows' level of agreement was higher across all items with respect to key contributions to their respective organization. Whereas both fellows and academic supervisors reached consensus on leadership and providing mentorship or education as key contributions made by the fellow, host supervisors did not. All three groups strongly disagreed that fellows provide limited to no contribution to their respective organizations. As this item was negatively worded, lack of consensus in this case actually represents agreement across the three groups that fellows made some contribution to their host organization.

Among the unique themes identified by respondents, only grantsmanship (mean 2.94; SD 1.59), identified by fellows, did not reach the level of consensus. All other unique themes reached consensus in their respective groups. Fellows rated the HSI Fellowship impact project to an average score of 4.14 (SD 1.09). The average score of host supervisors for strong work ethic was 4.29 (SD 0.56). The mean score for key contributions identified by academic supervisors were 3.95 (SD 0.80) for revealing the strength and weakness of the current academic training and 3.65 (SD 0.88) for identifying challenges and barriers to learning health systems.

Discussion

The objectives of this study focused on identifying the criteria of success for an embedded research fellowship and the perceived contributions of the fellows to their host organization and academic institutions. To our knowledge, this study is the first to use an eDelphi approach to examine and compare the experiences of three stakeholder groups participating in an embedded research training opportunity. Insights gained through this research will be explored further and implications for each group, and the program as a whole, will be presented.

A high level of agreement was found across the participant groups on the common criteria of success as well as fellows' contributions to respective organizations. Although fellows and academic supervisors strongly agreed, the primary differences in the perspectives were found among the host organizations. This may be attributed to the heterogeneity of the host organizations as they have different objectives and more variability in their characteristics, structures and organizational cultures than academic institutions. Host organizations included in the inaugural cohort of the HSI Fellowship represented over 30 different public, private for-profit, not-for-profit and Indigenous health organizations (CIHR 2018b). The high level of agreement, particularly within fellows and academic supervisors, may be a result of their familiarity with a post-doctoral program design and program core competencies.

Criteria of a Successful Fellowship Experience

Consensus was shared among fellows and host and academic supervisors pertaining to their criteria for HSI Fellowship program success; however, there was variability of agreement within groups for some criteria. Although reaching the threshold for consensus across all three groups, academic or research productivity as a criterion of success demonstrated less within-group agreement for fellows and host supervisors. This is not surprising given the HSI Fellowship's stated program objectives, which focus primarily on experiential learning and enriched competency development outside the traditional academic setting. Despite calls for new or additional metrics to assess real-world research impact, peer-reviewed publications, grants and teaching continue to be valued above demonstrable improvements in health and health systems (Cruz Rivera et al. 2017; Robert 2016). This misalignment of program goals and contemporary academic career path rewards may create tension between academic and host supervisors, leading to unmanageable workloads as fellows attempt to succeed in both worlds. In the UK, embedded researchers have highlighted similar challenges of defining their project scope and maintaining academic identity (Vindrola-Padros et al. 2017, 2019).

Contributions of the Fellows

Academic supervisors and fellows had a high level of agreement on contributions of the fellow, reinforcing the fact that the design of the HSI Fellowship program (Tamblyn et al. 2016) aligns with the core competencies identified for the program (Bornstein et al. 2018). All three groups reached consensus that a key contribution of fellows included research and analytic skills/expertise, which is consistent with previous literature on the characteristic features of embedded research (Vindrola-Padros et al. 2017).

Fellows and academic supervisors reached consensus on all the common key contributions of fellows identified in Round 1; however, host supervisors did not reach consensus on fellows' contributions for leadership and mentorship. This may reflect the differences in culture between academic institutions and host organizations. For example, some host organizations may not perceive fellows as leaders or mentors because they have established hierarchies that are based on tenure within their or similar organizations rather than academic qualifications. In academic settings, however, post-doctoral fellows are in leadership and mentorship roles, including teaching, supervising theses and research projects and/or more general mentorship for undergraduate and graduate students (Åkerlind 2005, 2009; Chen et al. 2015). It may also reflect the program's design, which featured leadership and mentorship as a core competency for fellows to develop with the mentorship and support of their health system and academic supervisors.

Strengths and Limitations

There are several strengths to the present study. First, this study included perspectives from the three key stakeholder groups collaborating in an innovative training initiative to better understand key features of success and fellows' contributions. By examining the program from these three different perspectives, we have been able to elucidate areas where there may be tensions or misalignment between stakeholders. This study also begins to establish specific criteria for success in the fellowship that can improve future iterations of the program.

There are also several limitations of the current study. First, it is unclear how findings may have been affected if we had presented a list of all themes, both common and unique, to all participant groups in Round 2 of the eDelphi. Second, although there are structural differences in the three programs (e.g., one-year, two-year, start-up grant) (CIHR 2018a) included in the inaugural cohort, the study design did not enable us to stratify results based on fellowship type. Furthermore, the application processes were different. Either the organization formed a relationship and pursued work with a fellow (CIHR-IHSPR 2016) or the fellow forged a relationship with the preferred organization (CIHR 2018c). It remains to be determined if both approaches are of similar benefit. Variations within and across the three participant groups and the degree of consensus must be interpreted with caution.

Third, there were limited options presented to respondents that may not have captured their full range of perspectives. Although this is a general limitation of Delphi studies (Keeney et al. 2001), in this study, it was not possible to disentangle contributions based on fellows' previous experience and skills versus those based on program elements. Fourth, this study was conducted after the first year of the program, while many of the fellowships were still in progress or recently completed. Therefore, it may have been too early to gain a complete understanding of the criteria for success or the value of the fellows' roles. Finally, there may also have been differences between respondents and non-respondents for which there is insufficient consideration.

Implications

Although findings confirm high agreement among the three groups on criteria for success and fellows' contributions, discrepancies between the three groups suggest opportunities for program improvement. First, there is an opportunity to strengthen communication regarding the intention of the fellowship and facilitate discussion of the criteria for success among members of each fellowship triad. This includes structured communication between the funder and all parties at the program outset and communication between fellows and their hosts and academic supervisors throughout the fellowship. Currently, the program requires that the three parties complete a professional development plan at baseline and at three and 12 months. This plan is an important guide for goal-setting (for the fellowship and long-term career), self-reflection and supervisor assessment. An expectation management activity might be an interesting complement to reflect on the tripartite relationship in addition to the fellow's career.

Our findings highlight the heterogeneity within groups and the potential for competing priorities of the host organizations and academic institutions to impact fellowship success. Existing program orientation and training provided for all three groups may be enhanced based on this new knowledge concerning the criteria for success and fellows' key contributions.

Conclusion

This study is a first step toward gaining an understanding of the experiences of a complex, pan-Canadian embedded researcher training program from the perspectives of three key stakeholder groups. A consensus was reached on many criteria of success for an embedded research fellowship and on several perceived contributions of the fellows to their host organization and academic institutions. Future research should include a qualitative, in-depth exploration of the experiences of HSI fellows, host supervisors and academic supervisors to expand on emerging definitions of success and the program as a contributor to learning health systems.