The Rockefeller Team Science Leadership training program: Curriculum, standardized assessment of competencies, and impact of returning assessments

Abstract

The ability to effectively lead an interdisciplinary translational team is a crucial component of team science success. Most KL2 Clinical Scholars have been members of scientific teams, but few have been team science leaders. There is a dearth of literature and outcome measures of effective Team Science Leadership in clinical and translational research. We focused our curriculum to emphasize Team Science Leadership, developed a list of Team Science Leadership competencies for translational investigators using a modified Delphi method, and incorporated the competencies into a quantitative evaluation survey. The survey is completed on entry and annually thereafter by the Scholar; the Scholar’s primary mentor and senior staff who educate and interact with the Scholar rate the Scholar at the end of each year. The program leaders and mentor review the results with each Scholar. The survey scales had high internal consistency and good factor structure. Overall ratings by mentors and senior staff were generally high, but ratings by Scholars tended to be lower, offering opportunities for discussion and career planning. Scholars rated the process favorably. A Team Science Leadership curriculum and periodic survey of attained competencies can inform individual career development and guide team science curriculum development.

Introduction

The overall vision of the Rockefeller University Center for Clinical and Translational Science (CCTS), supported by the National Institutes of Health Clinical and Translational Science Award (CTSA) program, is to develop, demonstrate, evaluate, and disseminate innovative programs to achieve translational success and to integrate these into a seamless “Learning Clinical Research Enterprise” that uses outcome data to drive quality improvement for the benefit of human health. The Rockefeller University 3-year Masters’ degree KL2 Clinical Scholars Program, which began in 2006 and is supported by both CTSA and University resources, is a key component of the CCTS, and its goal is to prepare translational team science leaders who are able to use their skills to improve human health [1]. It is designed to provide an optimal environment for junior translational investigators to develop team science and leadership skills by designing, conducting, analyzing, and reporting a human participant protocol under the supervision of an expert senior scientific mentor and a multidisciplinary group of Team Science Educators/Evaluators (TSE/Es) who comprise the Senior Staff of the CCTS. We have approximately 15–20 Scholars in the program at any time.

Our motivation to focus on Team Science Leadership is rooted in our belief that the ability to effectively lead an interdisciplinary translational team is a crucial component of team science success, supported by feedback from prior graduates who expressed insecurity about their Team Science Leadership skills immediately after graduation, and by our observation that despite our Scholars having extensive experience participating as team science members, they have little experience as team science leaders. While there is universal recognition of the importance of team science and team leadership in the conduct of clinical and translational science (CTS) [1–3], and an enormous literature from industry on team leadership and organization [2,4], there is a remarkable dearth of outcome measures of effective team leadership in CTS. Moreover, while the business literature on corporate leadership is robust, in our literature review, we could not identify any team leadership competencies that are specific for serving as a team leader of clinical and translational research. Specifically, PubMed searches in July 2021 for papers with the terms “team science outcome” or “team science competencies” in their titles yielded no papers. The 2011 CTSA Core Competencies in Clinical and Translational Research touched on Translational Teamwork and Leadership [5] but were very general and did not provide a mechanism to assess outcomes. Even a search of the NCI Team Science Toolkit [6] did not identify tools for outcome assessment of Team Science Leadership. To address these gaps, we enhanced our curriculum to emphasize Team Science Leadership, developed a list of team science leadership competencies specific for translational investigator team leaders, and implemented a Team Science Leadership quantitative evaluation survey to be completed annually by the Scholar, the Scholar’s primary mentor, and the TSE/Es. We also implemented a process to review the results with the Scholars to ensure that they receive feedback on their progress at a time when they can plan a course of action to ensure that they master the competencies before graduating. Analyzing the results of the survey has also provided crucial additional information that has enhanced our ability to: 1) teach team leadership skills, 2) provide opportunities for trainees to develop their skills via experiential learning, 3) evaluate trainees’ competencies, 4) provide feedback to trainees about their performance, 5) assess whether providing feedback results in improved performance, and 6) identify opportunities to strengthen individual components of team science training. This paper describes our Team Science Leadership curriculum, presents the process we followed to create the Rockefeller Team Science Leadership competencies and survey, and provides early results on psychometrics, means and variability of items, and lessons learned on the utility of the survey.

Methods

Team Science Leadership Curriculum

The Team Science Leadership curriculum is outlined in Table 1 and discussed in more detail in Supplementary Materials.

Table 1.

Elements of the Team Science Leadership curriculum

1.	Distribution of Team Science Leadership Self-Assessment Survey and Team Science Leadership Evaluation Survey to incoming Scholars. Self-assessments conducted using both instruments.
2.	A series of tutorials focused on Team Science Leadership, including topics related to :why leadership is important; what makes for effective leadership; different styles of leadership; case-based analysis of leadership responses to a series of typical academic leadership challenges; adages reflecting principle of leadership; leadership literature; and the importance of team diversity.
3.	Trainings in preparation for leading a human participant protocol as the responsible Principal Investigator (PI): Good Clinical Practice; Protection of Human Subjects; Responsible Conduct of Research; 15-hour structured course and one-on-one tutorials on research methods and biostatistics that stress the importance of leading and managing research and analytic teams; 10-session seminar series on Rigor, Reproducibility and Reporting that stresses the importance and value of multidisciplinary teams; and regulatory requirements for clinical investigation, including registration in ClinicalTrials.gov.
4.	Constitution of a multidisciplinary team by Scholar PI, including, depending on protocol, biostatistician, bioinformatician, research nurse, research coordinator, information technology expert, participant recruitment expert, laboratory technician, and research pharmacist.
5.	Preparation of a Pilot Grant application requesting funding for the protocol.
6.	Preparation of human participant protocol with one-on-one training under the Rockefeller University Center for Clinical and Translational Science (CCTS) Navigation [13] and the Community-Engaged Navigation [14] programs, the Centralized Research Participant Recruitment [15] program, Research Hospitalist [16] initiative, and Informed Consent for Next-Generation Sequencing [17] initiative.
7.	Oral presentation of the human participant protocol to the Rockefeller University Institutional Review Board.
8.	Training by senior Nurse Practitioner Research Coordinator in conduct of team science meetings, along with constructive feedback after senior Coordinator observes meetings.
9.	One-on-one review by senior Coordinator of Scholar’s response to challenging hypothetical scenarios, e.g., nurse administration of incorrect dose of study drug to two participants (Is this your responsibility? How will you determine exactly how the error occurred? What is your action plan to prevent this from happening again? To whom will you report this error? How will you communicate what you have found, and plan to do, to each of your team members?).
10.	Management training skills.
11.	Early audits of Scholar protocols after the first few participants are enrolled to ensure compliance with all protocol, regulatory, and IRB requirements.

We introduced the Team Science Leadership initiative to the Clinical Scholars by providing them with a Team Science Leadership Self-Assessment document with a series of questions about

Team Science Leadership for self-assessment (Table 2), and if they chose, discussion with their primary mentor and/or the leaders of the Clinical Scholars program. The document covers the Scholar’s experiences and/or views of current and past experience in participating in teams, developing and articulating a vision for a team, delegation of tasks, oversight of team members, decision-making, conflict resolution, models of leadership and role modeling, and self-reflection on the Scholars’ current strengths and weaknesses as a team leader. In addition, Scholars rate themselves on their Team Science Leadership Competencies (see below) on entry into the program and yearly using the same survey used by the TSE/Es. To broaden the Scholars’ scope in contemplating their roles as leaders, we searched for other leadership models that might be instructive for comparison. We selected a program on leadership, The Art of Teams: Achieving Excellence as Equals, that uses the relationship among the members of a string quartet as a paradigm of shared and shifting team leadership, reflecting the relative roles of the instruments as dictated by the music and the overall team structure of the quartet [7].

Table 2.

Rockefeller University Clinical Scholars Team Science Leadership self-assessment survey

Current and past team experience How many different teams do you participate in or lead? What are your positive and negative reflections on your experience as a member of a medical care team during your medical training? Have you ever received training in team leadership? How do you tailor your leadership style when leading a basic science team versus a clinical research team versus a team that includes industry members? How do you see yourself as a leader in your current role? Current and past team experience How many different teams do you participate in or lead? What are your positive and negative reflections on your experience as a member of a medical care team during your medical training? Have you ever received training in team leadership? How do you tailor your leadership style when leading a basic science team versus a clinical research team versus a team that includes industry members? How do you see yourself as a leader in your current role? Developing and articulating a vision Do you have a clear vision for your team, and if so, how do you articulate it and engage team members in sharing your vision? Did you engage team members in developing or refining the vision? How do you make certain everyone on your team understands the goals and desired outcomes? Delegation of tasks and oversight of team members Under what circumstances do you delegate authority to team members? How do you maintain quality control over the tasks that you delegate? How do you determine the degree of independence granted to individual team members? How do you make certain that the person you delegate a task too feels able to accomplish the task? Under what circumstances do you share team leadership and how do you signal that to other team members? Decision-making Which methods of communication do you use (e.g., phone, email, text, in person, group meetings) and how do you decide which method to use? How do you solicit advice from more senior people who are either on your team or who you know and respect? How do you resolve intra-team conflict? Leader role modeling How do you demonstrate that you take ultimate responsibility for all actions of the team? How do you insure that you are recognizing the contributions of team members equitably? How do you mentor team members in developing their leadership skills? What do you do to establish trust among team members and demonstrate your own accountability? How do you consult with other team members before making decisions and do you share your thought process with other team members? How do you support your team members? How do you lead collaborations between your team and other teams? Self-reflection What do you enjoy the most about leading your team? What do you do well as leader? How do you think you could improve as a leader?

Team Science Leadership Competencies and Teams Science Leadership Survey

As TSE/Es, we collectively recognized the need to define the competencies for effective Team Science Leadership for translational research studies. By combining our domain-specific expertise and experience, we created a list of competencies that we assessed to be essential for an outstanding translational scientist to lead a multidisciplinary team in conducting a human participant research protocol (Supplemental Figure 1). All 15 Senior Staff members participated in a modified Delphi method [1,8,9] for generating and reviewing both the major domains and the items within domains. Over the course of 6 months of weekly meetings, the domains were first agreed upon, and then the items within domain were created, deleted, and revised until neither new items nor major revisions were suggested. These draft items were then shared with all of the current Scholars for feedback and revision. They were also shared with colleagues from Yale University and the University of Pennsylvania in a series of face-to-face meetings at each site. The list of competencies was finalized to begin the project, but with agreement that the competencies would need to be reviewed and modified periodically.

The final list converged on five domains: 1. Foundational Leadership Competencies containing nine items centered on creating and leading teams, creating and communicating a vision, building trust among team members, mentoring team members, and establishing lines of open communication; 2. Professionalism Competencies containing five items that describe aspects of responsibility and modeling ethical and supportive behavior; 3. Team Building and Team Sustainability containing 12 items that describe more granular level aspects of team communications, recognition of team member achievements, conflict resolution, seeking advice, and taking responsibility; 4. Appropriate Use of Resources and Execution of Study containing six items that describe elements of appropriate budgeting, staffing, overcoming obstacles, organizing continual quality improvement, and delegating responsibility appropriately; 5. Regulatory Accountability containing five items describing issues relating to compliance with Good Clinical Practice (GCP), federal and state laws and regulations, and issues related to data management, security, and sharing.

Prior to the Team Science Leadership initiative, we periodically reviewed our perceptions of the progress of individual Scholars during our weekly TSE/Es meetings. During these discussions, we appreciated the value of sharing our perceptions since we each observed the Scholars in different roles, interacting with different people, and performing different tasks. We thus recognized that by summing our individual observations we had a much more comprehensive assessment of the Scholars’ progress. We therefore set a goal of systemizing the information from all TSE/Es and combining it with the Scholars’ mentors’ evaluations and integrating it into a comprehensive plan for competency evaluation, feedback, and improvement. To achieve this goal, we developed a research protocol describing the evaluation of Scholars’ Team Science Leadership competencies and received approval from the Rockefeller Institutional Review Board prior to implementing the study. To evaluate the Team Science Leadership competencies of the Scholars, we converted the competencies into a survey in which each competency is rated based on a six-point Likert scale score (0–5), with additional space for free text comments (Table 3). We then implemented the survey in a REDCap [10] format to facilitate data collection and preserve the confidentiality of the reviews. Each Scholar evaluates herself or himself, each mentor evaluates her or his Scholar, and each TSE/E evaluates each Scholar across as many items and dimensions as are appropriate, based on content area and exposure to the Scholar’s activities in that area (Supplemental Figure 2) The evaluations by the TSE/Es, mentor, and Scholars are combined into a written report that includes 1. the aggregated evaluations by the mentor and TSE/Es, presented as the mean ± SD score for each competency, 2. an adjacent column with the Scholars’ self-evaluation scores, 3. an adjacent column listing the difference in the aggregated score and the Scholar’s score for each competency, and 4. free text, anonymized comments by the TSE/Es and mentor. After field testing the survey with one Scholar, we made minor revisions to the survey and to the REDCap administration platform.

Table 3.

Team science competencies

Item	TSE/E + Mentor Mean (SD)	Scholar Mean (SD)	Average Difference (SD) (TSE/E − Scholar)
A. Foundational leadership competencies for Team Science Educators/Evaluators (TSE/E) + Mentors, and Scholars, and Mean Differences
Establishes a compelling vision and sets appropriate goals	4.77 (.48)	3.33 (0.89)	1.42 (.98)
Creates a culture that values and supports diversity	4.83 (.44)	3.86 (1.17)	1.09 (1.20)
Fosters an environment of mutual trust	4.79 (.49)	3.80 (1.08)	0.96 (1.24)
Develops and nurtures collaborations and external relationships	4.90 (.30)	3.86 (0.83)	0.99 (1.06)
Explores opportunities for growth and development on a continuous basis	4.71 (.68)	2.93 (1.48)	1.55 (1.26)
Anticipates obstacles and devises strategies to overcome them	4.54 (.79)	3.0 (1.18)	1.35 (1.26)
Supports and mentors all members of the team	4.78 (.55)	3.27 (1.38)	1.47 (1.46)
Anticipates the need for resources to carry out initiatives and obtains them in a timely manner	4.57 (.84)	3.31 (1.25)	0.67 (1.15)
Establishes and oversees a communication structure and processes that insure that both oral and written communication with and among all team members and other stakeholders is timely and effective	4.70 (.72)	3.07 (1.25	1.99 (1.58)
Foundational leadership competencies scale (α = 0.81)	4.80 (.31)	3.35 (0.75)
B. Professionalism competencies for Team Science Educators/Evaluators (TSE/E) + Mentors, and Scholars, and Mean Differences
Accepts responsibility as PI for the conduct of all aspects of the study	4.77 (.73)	4.07 (0.79)	0.61 (1.0)
Serves as a model of the highest professional and ethical standards	4.90 (.29)	4.08 (0.79)	0.88 (0.81)
Commits to continuing education	4.71 (.59)	3.64 (1.28)	1.17 (1.34)
Commits to transparency, invites feedback from all team members, and implements ideas that garner broad support	4.88 (.33)	4.0 (0.85)	0.94 (0.94)
Recognizes and rewards contributions of all team members	4.80 (.72)	3.93 (0.88)	0.97 (0.98)
Professionalism Scale Mean (α = 0.71)	4.81 (.20)	3.94 (0.76)
C. Team building and team sustainability for Team Science Educators/Evaluators (TSE/E) + Mentors, and Scholars, and Mean Differences
Invites participation in building vision	4.65 (0.65)	3.36 (1.15)	1.45 (1.06)
Articulates vision and goals clearly and unambiguously	4.81 (0.39)	3.0 (1.31)	1.72 (1.31)
Effectively identifies, selects, and recruits talented team members	4.56 (0.74)	3.13 (1.06)	1.58 (1.06)
Insures timely and effective bidirectional communication with all team members and among team members	4.92 (0.26)	3.80 (1.26)	0.94 (1.07)
Demonstrates respect for team members via active listening and rapid follow-up, and sensitivity to both verbal and nonverbal communication	4.91 (0.41)	3.0 (1.56)	1.37 (1.18)
Leads team meetings effectively, with defined agenda, adequate time for discussion, and adherence to starting and stopping times. Summarizes meeting and creates action plan with clear assignment of responsibility and expected completion dates	4.68 (0.54)	3.0 (1.57)	1.47 (1.4)
Celebrates milestones and accomplishments	4.88 (0.32)	3.57 (0.85)	1.39 (1.02)
Recognizes and acknowledges strengths of team members	4.77 (0.51)	3.38 (0.96)	1.42 (0.67)
Takes responsibility for all team errors and immediately develops corrective action plan based on detailed analysis of system failures leading to error	4.58 (0.90)	3.41 (0.66)	1.19 (1.11)
Intervenes as rapidly as possible to resolve conflicts, listening carefully and in confidence to all parties, and mediating resolution via building out fairly and equitably from shared values and goals	4.69 (0.92)	3.67 (0.89)	1.06 (1.46)
Seeks assistance from more senior investigator(s) and administrators when unable to successfully address problems	4.70 (0.74)	4.15 (0.68)	0.48 (0.77)
Communicates with, and is available to, clinical staff in labs and Hospital participating in the study	4.81 (0.51)	2.86 (1.29)	1.71 (1.03)
Team building and team sustainability scale (α = 0.93)	4.79 (0.31)	3.31 (0.84)
D. Appropriate use of resources and execution of study for Team Science Educators/Evaluators (TSE/E) + Mentors, and Scholars, and Mean Differences
Budgets effectively and insures that there are adequate financial resources to support projects	4.67 (0.61)	4.0 (0.73)	0.65 (078)
Takes initiative in planning for appropriate staffing and identifying appropriate space, equipment, and other resources needed to conduct study	4.70 (0.51)	4.10 (0.67)	0.71 (0.75)
Formulates with the help of others effective and innovative strategies to achieve goals and reformulates strategy as appropriate to address unexpected obstacles and/or new opportunities	4.82 (0.45)	3.71 (0.61)	1.10 (0.65)
Monitors results and team function continuously and makes adjustments when necessary as rapidly as possible	4.79 (0.41)	3.59 (0.79)	1.25 (0.92)
Delegates responsibilities appropriately while still maintaining oversight and performing systematic review of actions taken by others	4.65 (0.56)	2.71 (1.43)	2.01 (1.3)
Provides constructive critical feedback to members of the team discretely and at appropriate intervals	4.74 (0.68)	3.15 (1.21)	1.40 (0.82)
Appropriate use of resources and execution of study scale (α = 0.83)	4.76 (0.42)	3.49 (0.66)
E. Regulatory accountability for Team Science Educators/Evaluators (TSE/E) + Mentors, and Scholars, and Mean Differences
Knows, understands, and transmits to team members’ information about applicable regulations related to hospital accreditation, protection of human subjects, Good Clinical Practice (GCP), research sponsor requirements, FDA, New York State Department of Health, and Rockefeller University policies, as well as local, state, and federal laws	4.70 (.61)	2.81 (0.98)	2.02 (1.10)
Knows acceptable methods of data analysis and proper methods of transmission of data to regulatory agencies and to appropriate databases to comply with data-sharing responsibilities	4.87 (0.43)	3.29 (1.07)	1.62 (1.10)
Monitors results and team function continuously and makes adjustments when necessary as rapidly as possible	4.83 (0.38)	3.41 (1.08)	1.14 (0.90)
Creates financial plan in accord with University and sponsor requirements and oversees budget and payments	4.66 (0.54)	3.41 (0.79)	1.26 (0.98)
Understands responsibilities in protecting intellectual property and complies with University and sponsor requirements for invention disclosures	4.93 (0.27)	3.75 (0.96)	0.83 (0.98)
Regulatory accountability (α = 0.97)	4.89 (0.32)	3.28 (0.93)

Note: Scholar ratings less than or equal to 3.0 and TSE/E versus Scholar ratings greater than 1.0 are in bold.

The Return of Results process consists of the CTSA PI and the Director of the Clinical Scholars Program prereviewing the report to make sure that any negative comments are framed constructively. They then meet with the Scholar and the Scholars’ mentor to review the findings and focus on identifying areas where the Scholar thinks she or he needs additional training or opportunities to develop the competencies. The review also focuses on comparing the Scholars’ self-evaluation on each competency to the aggregate evaluation by the Scholar’s primary mentor and all of the TSE/Es. Where appropriate, the group identifies additional resources or programs that may help the Scholar develop the competency. After the meeting, Scholars are asked to complete a short anonymous survey (Table 4) providing feedback on the value of participating in the survey and the Return of Results process.

Table 4.

Survey for clinical scholars to report the value of receiving results from the team science competency survey

1) Did you find the information of value in thinking about your Team Science Leadership skills?
1 – Not at all	2	3	4	5	6 – Very valuable
2) Taking the evaluations as a whole, were you surprised by the evaluations?
1 – Not at all	2	3	4	5	6 – Very surprised
3) Do you think that you will change any of the things you do as a team science leader based on the evaluation?
1 – Definitely will not	2	3	4	5	6 – Definitely will
4) What do you think is the best way to return the evaluations to Clinical Scholars?
A. In person by:
1 – Program Director(s)
2 – Mentor
3 – Program Director(s) + Mentor
B. Electronically by:
1 – Program Director(s)
2 – Mentor
3 – Program Director(s) + Mentor

The first administration of the competency survey was in the Fall of 2019, followed by a second administration in the Fall of 2020, which included the incoming scholars of the 2020 cohort.

Statistical Analysis

Means and standard deviations were calculated for each item using the 0–5 scale, which spanned from “Not At All Well” to “Extremely Well,” respectively, so that a higher score indicated a greater sense of competence of the item. If the TSE/E or mentor evaluators did not have a basis for making a judgment, they selected “Not Applicable” and the result was set to missing. Equal weighted scales for each domain were calculated if greater than 50% of the items in the scale were non-missing. Data from Scholars and TSE/Es were then merged to calculate item differences as the mean of the combined scores of the TSE/Es and mentor minus the Scholar score so that a positive difference indicated that the TSE/E and mentor scores were higher than the Scholar’s score.

A Cronbach’s alpha was computed for each subscale to assess the internal consistency and inform reliability. To help better understand the scale structure, a factor analysis with principal component dimension extraction followed by orthogonal rotation was employed within each domain. Subsequent scree plots of the eigenvalues were examined to assess the dimensionality of the subscales. All analyses were conducted in SAS Studio V94.

Results

A total of 31 Team Science Raters (which included TSE/Es and mentors) made a total of 115 ratings across 15 scholars (11 unique, 4 rated twice). The number of TE/Es and mentors rating of the Scholars ranged from 2 to 11. All of the mentors evaluated their Scholars. The mentors’ ratings were incorporated into the aggregated TSE/Es ratings to preserve anonymity. The average Cronbach’s alpha coefficient was 0.85 (range 0.71–0.97), where cells with fewer items had predictably lower coefficients. Visual assessment of the scree plots from the exploratory factor analysis supports a single-factor solution within each domain, where the first eigenvalue of each scale accounted for greater than 50% of the variance and the second typically less than 15%.

Table 3A presents the nine items that comprise the Foundational Leadership Competency scale, along with the average ratings by TSE/Es–mentors and Scholars, and the average difference between the ratings. In general, the TSE/Es mentors scored the Scholars higher that the Scholars scored themselves; TSE/Es mentors thought Scholars excelled at “Developing and nurturing collaborations and external collaborations” and had the most room to improve in “Anticipating obstacles and devising strategies to overcome them”, while Scholars in general thought they excelled at “Creating a culture that values and supports diversity”, and in “Developing and nurturing collaborations and external relationships”, while they felt least competent in “Exploring opportunities for growth and development”. The largest disparity in ratings between TSE/Es–mentors and Scholars in this scale was in communication. The internal consistency of the scale as measured by Cronbach’s alpha was 0.81.

Similarly, Table 3B presents the five items describing Professional Competencies. TSE/Es–mentor scores were uniformly high, and they uniformly rated the Scholars higher on these constructs than did the Scholars themselves, where the largest disparity was for the items describing the Scholars’ commitment to staff education. The internal consistency of the scale as measured by Cronbach’s alpha was 0.71.

Table 3C describes the characteristics for the 12 items comprising the Team Building and Team Sustainability scales. Here too, TSE/Es–mentor scores were higher, with the highest positive discrepancy for two items describing communication skills, one for the articulation of vision and goals and the second for communicating with clinical teammates. The internal consistency of the scale as measured by Cronbach’s alpha was 0.93.

Table 3D presents item characteristics for the six items representing the Appropriate Use of Resources and Execution of Study scale. The largest discrepancy between scores of TSE/Es mentors and Scholars was the item about delegation of responsibility. The internal consistency of the scale as measured by Cronbach’s alpha was 0.81.

Table 3E describes the five items comprising the Regulatory Accountability Scale. TSE/Es–mentors believed scholars were much stronger on hospital regulations and policy than did the scholars. The internal consistency of the scale as measured by Cronbach’s alpha was 0.97.

Table 5 presents the comments provided by scholars who completed the survey about their experiences in having the meeting with their mentor team and receiving the results of the evaluation. Overall, the comments reflected an appreciation for being exposed to these constructs in addition to the traditional clinical and research skills, and a recognition that these skills are key to a successful translational science career.

Table 5.

Clinical scholar feedback after return of results

Item	Response
Did you find the information of value in thinking about your Team Science Leadership skills?	I felt this was a good exercise in self-evaluation. It made me reflect on my successes and failings with regard to Team Science Leadership.
	I had never thought about science leadership as a concept before. It helped me to understand the importance if this concept as scientific project becomes more complex and teams are growing larger. Additionally, it was valuable to identify my own strengths and weaknesses.
Taking the evaluations as a whole, were you surprised by the evaluations?	The comments were much more positive than expected. Parts of it likely come from differences in culture.
	I tend to underestimate my abilities and was surprised by the evaluation. The results also helped me to identify areas that I can work on with the goal to improve my science leadership abilities.
	I feel very appreciated in my daily work, but the results boosted my confidence and determination about being able to achieve my goals.
Do you think that you will change any of the things you do as a team science leader based on the evaluation?	I already changed some of my behavior.
	I will try to place more emphasize on Team Science Leadership in my projects. I appreciate that a scientific project requires a vision and leadership.
	The results highlighted that I under evaluated my skills, and I hope that they will result in me being able to present myself, my science, and my abilities with more confidence when looking for my next position.

Discussion

Effective Team Science Leadership is vital to success of translational research, but few programs focus specifically on developing leadership skills in this area [3,11]. This paper describes our Team Science Leadership curriculum; the development of Team Science Leadership competencies for translational investigators leading multidisciplinary teams that develop and conduct human participant protocols; the value of having the Scholar, the Scholar’s mentor, and a diverse group of TSE/Es evaluate the Scholar’s progress in mastering the competencies; and the value of sharing the information with the Scholar for future career development planning. By having the Scholars rating themselves on entry and yearly thereafter and having the mentors and TSE/Es rating the Scholars at the end of each year in the program, we provide feedback in a timely way that allows the Scholars, mentors, and program leaders to individualize paths to attain the competencies by the time the Scholar graduates.

We converted the competencies into a semi-quantitative survey format and evaluated the psychometric characteristics of the survey responses. In general, the five resulting scales had high internal consistency and factor structure. Of note, evaluators tended to rate Scholars as being more competent across the range of leadership characteristics than the Scholars rated themselves. In reviewing the survey results with the Scholars, the leaders used these discrepancies to initiate conversations with the Scholars to better understand their perceptions and why they felt less confident about their competence than did the mentor and TSE/Es. This feedback was especially important in helping the Scholars develop self-confidence in their abilities. Overall, Scholars rated their participation favorably, commenting that it helped them appreciate better that Team Science Leadership skills are important in becoming successful translational scientists.

Our study has several limitations. First, while the survey was developed by people experienced in translational research, with input from and collaboration with experts from two other institutions, important competencies may be missing from the list or not optimally described. Moreover, our competencies focused on the leadership skills required to lead a clinical research protocol team, the central component of our training program, and thus may not address competencies required for research in other areas of the translational spectrum or at different points in a translational career. In addition, the survey was implemented at only our relatively small, basic science- focused institution. Therefore, both the results and Scholar experiences may be different at other institutions with a larger number of Scholars, with different Scholar recruitment and selection priorities, and different educational programs. In particular, some other KL-2 Scholar programs focus on junior faculty who are further along in their careers than our Scholars and so are already serving as team science leaders. To gain a broader perspective, we are currently collaborating with other academic institutions to gain additional experience on the generalizability of our findings. In addition, the factor structure, which supported single dimensions within scales, and the Cronbach’s alphas, which describe fairly high-scale reliability, will need to be replicated with a larger and more heterogeneous sample before drawing generalizable conclusions. For statistical and psychometric purposes, it would be optimal to have all evaluators score all Scholars on all dimensions. However, as expected, not all TSE/Es had sufficient exposure or knowledge of the Scholars to rate them on all dimensions (e.g., a TSE/E who interacted with a Scholar only on regulatory issues or budgets may not be able to evaluate the Scholar’s ability to express a vision), and so “missing” responses were common. Fourth, some TSE/Es did not complete all items on the survey, so it was not possible to discern whether this was due to an inability to rate or a failure to complete the survey. Finally, we consider the competencies a living document and so anticipate modifying the lists as we learn from our own experience and from collaborators at other institutions.

With these limitations in mind, the development, deployment, and results of the survey provide interesting information. First, the survey provides a starting point for discussing Team Science Leadership competencies for translational investigators who aspire to lead their own multidisciplinary groups. Second, the results of the survey provide information to Scholars based on the perceptions of their mentor and an experienced group of TSE/Es, nearly all of whom have interacted with numerous Scholars during their careers, and who observe the Scholars in a wide range of roles required for translational success. Third, the results of the survey inform the elements we include in our training program on a yearly basis. One ancillary benefit is that by participating in the process, and reviewing the competencies, the Scholars’ mentors undergo training in mentoring in Team Science Leadership.

Despite Scholars’ abundant experience in being members of clinical and scientific teams, few have formally thought of themselves as Team Science Leaders. That is why we have integrated the survey into a much fuller curriculum in Team Science Leadership to ensure that Scholars develop these competencies.

We try to emphasize that we do not expect Scholars to master all of the team science competencies rapidly, but rather want to help them identify the key skills and attributes needed and intentionally plot a multi-year path to achieve the goal that can be incorporated into the Scholar’s Individual Development Plan. We previously developed a Graduate Tracking Survey System (GTSS) [1,12] to assess the progress of our Scholars after graduation and it provides outcome data on our Scholars’ success as translational science team leaders. Over time, by merging the process-based assessments generated by the survey with the outcome data from the GTSS, we hope to gain a better understanding of the factors that are associated with Team Science Leadership success. That information will help us craft curricula and assessments to optimize our program. We are eager to share our survey with other institutions as we continue to refine and modify the competencies, survey, and curriculum.

Conclusion

Our Team Science Leadership initiative highlights the importance of defining Team Science Leadership competencies. Training Scholars in those competencies by embedding them in an experiential and didactic curriculum focused on clinical and translational science, measuring Scholars’ attainment of the competencies over time using a quantitative survey, and providing periodic feedback to Scholars with the focus on ensuring that the Scholar masters the competencies by the end of the program. We are eager to share our materials with educators at other institutions to help refine the competencies and to enhance external validity and continuous improvement, as well as to help systematize Team Science Leadership skills measurement and training in translational science across the CTSA network.

Disclosures

The authors have no conflicts of interest to declare.