Challenges in Measuring Benefit of Clinical Research Training Programs – the ASH Clinical Research Training Institute Example

Abstract

The American Society of Hematology developed the Clinical Research Training Institute (CRTI) to address the lack of training in patient oriented research among hematologists. As the program continues, we need to consider metrics for measuring the benefits of such a training program. This article addresses the benefits of clinical research training programs. The fundamental and key components are education and mentorship. However, there are several other benefits including promotion of collaboration, job and advancement opportunities and promotion of work-life balance. The benefits of clinical research training programs need to be measured so that funders and society can judge if they are worth the investment in time and resources. Identification of elements that are important to program benefit is essential to measuring the benefit of the program as well as program planning. Future work should focus on the constructs which contribute to benefits of clinical research training programs such as CRTI.

Introduction

The need to develop clinician scientists with a focus on patient oriented research has been recognized in many specialties including hematology. Patient oriented research is important to ensure that basic science and clinical research ultimately impact on patient outcomes in a positive and meaningful manner [1, 2]. Several National Institutes of Health (NIH) working groups have noted the lack of effective clinical research training programs and the lack of mentorship, resulting in the inability of emerging clinician scientists to obtain peer reviewed funding [3]. Recently, Dr. Harold Varmus stated, “We must prepare this coming generation of clinical researchers to be competitive in seeking research grant support and be fully aware of the complexity of conducting sound clinical research”[4].

In response to this need, the NIH developed several initiatives, namely the K30 support mechanism [5], a loan repayment program, and the Clinical and Translational Science Award (CTSA) to name just a few [6]. The American Society of Hematology (ASH) similarly identified this need over a decade ago [7] and consequently created the ASH Clinical Research Training Institute (CRTI). The inaugural class was held in 2003 and last year, CRTI celebrated its tenth year anniversary. The CRTI program is currently a one-year program which begins with a week long summer workshop typically held in August. The program focuses on senior fellows and junior faculty with an intended career in patient oriented hematology research. Trainees may focus on either malignant or benign hematology in pediatric or adult medicine. In the initial few years of the course, attendees were exclusively from North America. More recently, a purposeful expansion to a very limited number of trainees outside of North America has occurred with previous trainees coming from Australia, Germany, the United Kingdom, and Singapore. The year-long process is led by two CRTI co-directors; these positions are elected by a CRTI Oversight Committee and rotate annually.

The trainee submission process includes a summary of the research proposal to be developed during CRTI along with a letter of support from the home mentor, a curriculum vitae and a career development plan. The chosen 20 trainees first meet each other at the summer workshop. The summer workshop is also attended by approximately 20 faculty members with a successful research program in patient oriented research. Faculty also include 5 to 6 statisticians and representatives from key funding agencies including the National Heart, Lung and Blood Institute and the National Cancer Institute. Faculty may also include the current ASH president and often a current or previous editor of Blood.

During the summer workshop, the program begins with each trainee presenting his/her research proposal. The current program includes formal evaluation of presentation skills with individual feedback by one of the two course co-directors. The rest of the week then consists of didactic sessions in the morning and small group sessions in the afternoon where trainees work in groups of 2 to 4 faculty members, 2 to 4 trainees and one statistician on protocol refinement. The evenings consist of dinner tables with focused discussions such as work-life balance and how to develop a research team, career retrospectives from both junior and senior faculty members and social events. The week long program ends with each trainee presenting the updated proposal.

The CRTI program also includes two subsequent re-convening of the small groups, one in December during the annual ASH meeting and one the following May. During the reunion of the small groups, each trainee presents a progress update followed by suggestions from faculty and other trainees on strategies to overcome road blocks. During these sessions, important career mentorship also occurs and in particular, very personal and difficult career experiences may be shared. In 2011, the CRTI instituted a formal mentorship program in which each trainee is matched with a CRTI faculty member who provides unbiased advice and mentorship throughout the year. Contact is prescribed to occur a minimum of four times through the year but may occur much more frequently.

ASH CRTI recently submitted and was successful in obtaining a NIH R25 training award, which afforded the opportunity to reflect critically on the purpose and evaluation of the program. The purposes and benefits of CRTI around education and mentorship are readily understood. However, these are not the only benefits of such training programs. If the evaluation process only included these elements, it would fail to capture several important aspects of the program. As funding becomes more restricted over time, it is important to highlight these benefits such that programs can continue to flourish as long as they continue to add value.

Consequently, the purpose of this manuscript is to articulate the benefits of the ASH CRTI program and to begin the process of considering a suitable outcome or set of outcomes to measure the benefit of clinical research training programs. These metrics will need to be developed and, as with any outcome measure, be psychometrically sound and useful. There are other clinical research training programs such as “Methods in Clinical Cancer Research” organized by the American Association for Cancer Research and American Society of Clinical Oncology that may overlap with CRTI in areas related to hematological malignancies. To our knowledge, outcome measures for this training program have not been reported.

One of the early steps in developing an outcome measure is to identify important constructs such that content validity can be examined. The following sections will outline some potential constructs to be included in measuring benefits of a clinical research training program followed by challenges in creating an outcome measure to capture these constructs. Ethical approval does not apply to this study.

Methods and Results

A: Potential Constructs of CRTI Benefit

Education

Education encompasses both core methodological knowledge and practical knowledge about research implementation. At CRTI, education is accomplished through traditional mechanisms such as didactic and small group sessions. Principles learned during the didactic portion are applied to each trainee’s project during the small group sessions. An innovative recent addition has been “sessions on demand” where each year’s trainees can identify their specific needs and learn in concurrently delivered workshops. For example, sessions on demand topics have included meta-analysis, decision analysis, how to interview, and the STATA statistical package. These additional sessions are chosen at the beginning of each week. Time allotted to these sessions has been increased over time reflecting their popularity.

A solid foundation in clinical research knowledge is a prerequisite to being a successful clinical researcher. Areas include, but are not limited to study design, biostatistics, regulatory considerations, patient reported outcomes and biomarkers [8]. Sessions also focus on grant mechanisms, how to write a successful grant, how to give an effective presentation, and how to write and publish research findings. Other sessions describe how to navigate the Clinical Trials Evaluation Program and how to become involved in co-operative and collaborative groups.

One of the more popular sessions at CRTI has been an interactive session on the ethical conduct of research. In this session, a series of controversial ethical situations is described; two potential courses of action are given for each scenario. Each co-director takes one of these courses and describes his/her rationale for the decision. The trainees are then invited to give their opinion on which decision is better and the rationale behind their choice. Faculty then provide their input and describe their own experiences with similar situations. This session has been extremely successful as it allows decades of experience to be shared among this diverse audience.

While there is no debate about the importance of education, the best approach to evaluate this education is less clear. Some have advocated for pure knowledge-based evaluation. While feasible, this method of evaluation may not be directly applicable in terms of the probability of trainee success. A second approach is to encompass evaluation of education in terms of the overall success of CRTI graduates with respect to grants and publications. Alternatively, others have used and preliminarily evaluated the psychometrics of an instrument called the Clinical Research Appraisal Inventory (CRAI) which was designed to measure participants’ clinical research self-efficacy. There is a 92 and 76 item version which both include 10 domains such as Conceptualizing a Study, Designing a Study, Collaborating with Others, and Funding a Study as examples [9]. The optimal strategy to evaluating education has not been identified in this context.

Mentorship

Many have noted the importance of mentorship in navigating a successful research career and a lack of mentorship at local institutions [10]. Throughout the history of CRTI, there has been great emphasis on the provision of mentorship by CRTI faculty although initially, mentorship occurred informally. Mentorship has included providing project and career advice including review of K and R award applications, opportunities for collaboration and publication, and opportunities for leadership position within research collaborations including cancer cooperative groups. In August 2011, a convenience sample of 14 CRTI faculty identified that on average, these faculty published 13 papers on which a CRTI mentee or graduate was a co-author and 4 grant applications on which a CRTI mentee or graduate was a co-author (personal communication, ASH Sept 2013).

While these data provide support for the previous informal mentorship program, it is possible that less assertive trainees may not have identified a CRTI mentor. Consequently, CRTI implemented a formal mentorship program in 2011 in which each trainee is matched to a CRTI faculty who is responsible for providing career and project advice over the year-long program. Qualitative comments for this program have been favorable.

In spite of the positive qualitative feedback, it remains a challenge to measure the adequacy and benefits of the CRTI mentorship program. Further, CRTI has not formally evaluated the mentors themselves. Measuring both of these aspects from a quantitative perspective is likely to be important and two tools designed to measure these aspects are the Mentorship Profile Questionnaire and Mentorship Effectiveness Scale [11].

Promotion of Collaboration

Innovative research may be the product of collaboration among researchers with different perspectives [12]. One of the major benefits of CRTI is the promotion of collaboration among different groups of clinicians and methodologists. First, CRTI promotes collaboration between pediatric and adult hematologists. CRTI trainee and faculty selection specifically considers ensuring representation from both groups. Opportunities for collaboration between pediatric and adult hematologists are infrequent but may be particularly fruitful since experiences and approaches are likely to differ. A better understanding of different perspectives on a problem is likely to yield better research.

Second, CRTI promotes collaboration between malignant and benign hematologists. This type of collaboration may be particularly beneficial when focused on supportive care research. For example, studies of blood product support, anti-coagulation and hematopoietic growth factors in cancer patients are examples where such collaborations are needed. Study of leukemia arising from benign hematologic conditions such as severe congenital neutropenia is another area in which collaboration would be beneficial.

Third, CRTI promotes collaboration between investigators from different nations. Traditionally, CRTI faculty and trainees have come from Canada and the United States. However, more recently, CRTI has expanded outside of North America. As clinical and laboratory research becomes more global in nature, this facilitation of collaboration between nations is important.

Finally, CRTI promotes collaboration between different types of clinical researchers such as qualitative scientists, clinical trials specialists, health services researchers, and translational scientists. Such intra-disciplinary collaboration should foster the best science which capitalizes on the best methods to address a specific research question.

Promotion of these collaborations is a positive attribute of CRTI. However, how to measure the extent of collaboration that develops as a result of CRTI and how this collaboration influences trainee outcomes has never been quantified. How to capture this construct is unknown.

Job Opportunities and Networking

Since CRTI is focused on trainees with a planned career in hematological patient oriented research, it is anticipated that many trainees will seek academic positions at medium and large sized institutions. One of the important advantages of CRTI is provision of networking opportunities which could facilitate identification of available positions and securing an interview for those positions. Further, CRTI faculty have played an important role in terms of providing letters of reference for candidates.

Career promotion may also be facilitated by CRTI. In particular, the presence of key leaders in ASH such as the ASH President provides a mechanism by which trainees and faculty members may be promoted through the ASH organization. Similarly, the presence of leaders affiliated with pediatric and adult cancer co-operative groups may also facilitate promotion of CRTI trainees with membership within these groups.

Work-life Balance

Traditionally, clinical research training curricula inevitably address the issue of work-life balance and emphasize the importance of emotional, social and physical health and healthy lifestyle choices. However, balancing work hours and a healthy lifestyle for academic clinicians may be challenging. While chosen CRTI faculty have been successful academically, there is no guarantee that these faculty members can role model work-lifestyle balance.

The CRTI curriculum addresses this issue during didactic sessions and over round table discussions at dinner. A unique aspect of the CRTI program has included daily exercise. For the last several years, a group of trainees and faculty members walk for 1 to 2 hours each morning prior to the first CRTI formal session. These walks provide an avenue to participate in daily activity and illustrate how it can be incorporated into an academic lifestyle. The advantage of walking over other activities is facilitation of conversation between CRTI attendees.

It is unclear whether this issue should be included as an outcome when measuring CRTI benefits. Nonetheless, since it is a beneficial aspect of the program, further consideration is warranted.

B. Challenges to Developing an Outcome Measure for Clinical Research Training Programs

Many of the challenges that we encountered in conceptualizing outcome measures for CRTI are not unique to hematology and thus, are generalizable to other clinical research training programs. Two general ways to measure the benefit of CRTI is to measure the success of previous trainees and to more specifically measure the constructs being targeted. These approaches are not exhaustive or mutually exclusive.

One way to measure benefit is by determining the success of previous trainees as identified by publications, grants and academic positions. However, this approach is not ideal related to confounding and selection bias. The key challenge to measuring the contribution of CRTI to the success of trainees is control group selection. Trainees at a comparable level who did not apply to CRTI are unlikely to be a suitable control group since applicants are likely systematically different than non-applicants. In our R25 application, we measured success by comparing the top 20 applicants to CRTI (i.e. CRTI attendees) to the subsequent 20 applicants and were able to show benefit in terms of K awards, R awards and publications (Sung personal communication August 26, 2013). However, such a comparison still suffers from selection bias since the highest ranked 20 candidates are likely to be systematically different than the next 20, thus explaining their success with the application process. Further, this analysis suffers from inadequate power given the available sample size.

A second approach to measuring the benefit of CRTI is to develop or use specific or global scales. Specific scales would measure constructs that are thought to be important in program success. If a new measure is developed, then it must undergo the same rigor in development as any outcome measure with evaluation of its psychometric properties. This approach may be a more sensitive measure of program benefits. The first step in developing such an outcome measure is to identify the constructs that contribute to program benefit. Identification would likely start with a literature review and consensus methodologies. The identification of a suitable control group is also a challenge with this approach. However, depending on the nature of the measure, pre-post designs may be feasible.

To conclude, we believe that clinical research training programs such as CRTI are fundamental to the development and success of clinical researchers and the clinical research community. However, as with any program, evaluation will be an increasingly important consideration in order to understand where resources should be allocated. Identifying constructs which contribute to benefit is important to facilitate evaluation and program development. Future work should focus on the constructs which contribute to benefits of clinical research training programs such as CRTI.