Addiction Research Training Programs: Four Case Studies and Recommendations for Evaluation

Abstract

The presence of structured addiction research training programs helps to ensure that the scientific workforce includes well-trained, diverse scientists necessary to reduce the negative impact of alcohol, drug, and tobacco use disorders. While the field has made significant progress in the development of standards for clinical training in addiction medicine, there remains significant room for improvement in the training of addiction researchers, as well as opportunities to synergize across addiction research training programs. The purpose of this paper is to describe four NIH-sponsored addiction research training programs, highlight critical components, and provide recommendations for more comprehensive and effective program evaluation. Moving forward, evaluation of addiction research training programs would be enhanced by the use of conceptual models to inform process and outcome evaluations, the application of innovative methods to ensure long-term data collection, the improvement of mentorship evaluation measures, and the integration of training methods from other fields of study. We encourage NIH and others in the field to be proactive in establishing core metrics for evaluation across programs. Furthermore, centralized tracking of NIH-funded addiction research trainees, analysis of aggregate data across programs, and innovative methods to effectively disseminate program materials and processes is recommended.

INTRODUCTION

Alcohol, drug, and tobacco use disorders are some of the most significant public health problems today in terms of overall economic burden and as long-standing causes of health disparities (Bouchery et al., 2006; CDC, 2014; National Drug Intelligence Center, 2011). Research training is critical to the development of a diverse, well-trained scientific workforce prepared to reduce the burden of addiction in our society (IOM, 2007; NIDA, 2015). Well-qualified researchers, who bring diverse backgrounds, priorities, and perspectives (i.e., underrepresented racial and ethnic groups, individuals with disabilities, and individuals from disadvantaged backgrounds), are imperative to developing a more effective scientific workforce (Moskowtiz & Thompson, 2001; NIDA, 2015; Yager et al., 2004).

The available evidence indicates that research training programs are effective at increasing trainee involvement in research. In a survey of over 3,000 doctoral-level medical school faculty, Pincus and colleagues (Pincus et al., 2013) found that post-doctoral research training was associated with greater research engagement. Ley and Rosenberg (2005) reviewed trends in the physician-scientist pipeline using data from NIH and other professional sources. Results indicated an increasing interest in research, possibly fueled by the NIH loan repayment programs created in 2002. An innovative, culturally-informed mentorship program at the University of New Mexico demonstrated increased trainee productivity (i.e., grants and publications), as well as increasing the value placed on cultural diversity, community service, and community-based participatory research designs as a method to support racial and ethnic minority faculty (Viets et al., 2009). Others have found that strategies to enhance the engagement of students in research training experiences (e.g., mentorship before, during and after training) help increase the supply of addiction researchers (Bland & Ruffin, 1992; Lambert & Garver, 1998; Reynolds et al., 1998).

In addition to the development of effective training programs in addiction research, there is a need to develop effective, standardized tools for evaluating such programs. A report by the National Research Council noted that it was virtually impossible to evaluate the “success” of existing training programs due to lack of standardized measures and limited aggregate data management (National Research Council, 2005). The report further points to the need for greater coordination among programs across NIH Institutes and a broader operationalization of success beyond the milestone of obtaining independent NIH funding. At present, there is limited published information describing existing addiction research training programs, trainees, and evaluation methods.

The objectives of this paper are to: 1) describe the program components of four currently funded NIH-sponsored addiction research training programs, and 2) provide recommendations for enhancing curriculum and evaluation methods. The paper aims to contribute to the growing field of addiction research training by providing specific details on program implementation challenges that may help inform the design, implementation, and evaluation of future research training programs.

ADDICTION TRAINING PROGRAM MODELS

Program Description

A primary impetus for this manuscript arose through discussions at national meetings and at a medical education conference with training program leadership. Of particular concern was the lack of structured training programs in addiction, and the dearth of literature on best practices for evaluating research training programs. Thus, the programs included in this manuscript were self-selected, but provide a general sense of the types of currently funded addiction research training programs, as well as a foundation of evaluation strategies from which other programs might benefit. Below is a brief description of each program. Table 1 presents the characteristics and major components of each program.

Table 1.

Training Program Characteristics and Evaluation Components

Program Characteristics	DART Medial University of South Carolina	SARET New York University	TRACC City University of New York and Columbia University Medical Center	TREND Memorial Sloan Kettering Cancer Center and City College of New York
Funding Institute/ Mechanism	NIDA R25	NIDA R25	NIDA R25	NCI P20
Funding Years	2006-current	2007-current	2013–2019	2014–2018
Area of Focus	Addiction and associated research methods	Addiction and associated research methods	Translational addiction science	Tobacco and tobacco-related cancer disparities
Trainees/year	14–16	12 (summer) Variable (year-long)	4–6	Variable
Total Enrolled (thru 2015)	132	80	16	14
% Female	54.5%	60.0%	55.0%	87.5%
% Underrepresented Minority	13.6%	26.0%1	100%	75.0%
Trainee Discipline/Level
Undergraduate Students	X		X	X
Medical Students	X	X	X	X
Dental Students		X
Nursing Students		X
Master’s Students	X	X	X	X
Psychiatry Residents	X
Medical Residents
Graduate Students	X	X	X	X
Early Stage Investigators				X
Program Components
Mentorship	X	X	X	X
Mentorship Team	X	X
Pilot Project/Funding	X	X	X	X
Didactic Seminars	X	X	X
Conference Attendance	X	X	X
Coursework/Workshops	X	X		X
Webinars/Educational Modules		X
Training Institute			X
Primary Evaluation Indicators
Scholarly productivity (manuscripts, presentations)	X	X	X	X
Honors/awards	X	X
Academic milestones	X		X	X
Research-related funding	X	X	X	X
Research self-efficacy			X2	X2
Knowledge/competencies				X
Engagement in addiction research	X	X		X
Mentorship quality	X	X	X	X3

¹This figure only includes students from 2013–2015 cohorts;

²Forester, Kahn, & Hesson-McInnis, 2005;

³Fleming et al., 2013

The Drug Abuse Research Training Program (DART, NIDA R25, PIs Drs. Sudie Back and Kathleen Brady) is located in the Department of Psychiatry and Behavioral Sciences at the Medical University of South Carolina (MUSC) in Charleston, SC (Back et al., 2011; Cluver et al., 2014; Book et al., 2015). The primary goal of DART is to increase the number of physician-scientists with the skills needed to conduct patient-oriented addiction research. The DART program was initiated in 2006 and is comprised of a 2-year research track with weekly seminars for psychiatry residents, and a 10-week summer research program designed to develop the “pipeline” of trainees by extending recruitment to undergraduate, graduate, and medical students. The pillars of the DART program are research mentorship, an intensive core research training curriculum, training activities tailored to meet each trainee’s needs, and funds for pilot project research. The summer research program also includes a formal presentation given by trainees at the annual “DART Summer Research Day”.

The Substance Abuse Research Education and Training program (SARET, NIDA R25, PI Dr. Gourevitch) resides at New York University, bridging its School of Medicine, College of Dentistry, College of Nursing and School of Social Work. SARET is designed to educate and spur interest among students in these four disciplines (at the undergraduate and graduate levels) about substance use disorders and the fundamentals of clinical research. Interactive web-based learning modules focused on diverse aspects of substance use content and research methods are integrated into courses at all four schools. Each summer, a smaller number of students from each school join together with research faculty in an intensive summer-long mentored research experience. Efforts are made to pair students with a mentor from a discipline other than their own to increase interprofessional exposure. During the summer, trainees participate in formal didactic sessions that supplement the web-based curriculum; students also attend site visits to substance use disorders treatment programs, a harm reduction center, and a human subjects research lab.

The Translational Research Training in Addiction at CCNY and CUMC program (TRACC, NIDA R25, PIs Drs. Hien and Ruglass) is a combined partnership of the City College of New York (CCNY) and Columbia University Medical Center (CUMC). TRACC trains medical and psychology graduate students from underrepresented racial and ethnic groups in translational addiction research with a specific emphasis on neuroscience. TRACC scholars are mentored over 2–3 years by expert scientists while working in research labs and developing and implementing their own pilot research studies. Weekly seminars during the academic year and a week-long summer training institute provide supplemental didactic material and workshop experience in a wide range of research methods and addiction content.

The Translational Research Education and Training to Eliminate Tobacco Disparities program (TREND, NCI P20, PIs Drs. Sheffer and Ostroff) is a joint effort between City College of New York (CCNY) and Memorial Sloan Kettering Cancer Center (MSK). The goal of TREND is to reduce tobacco-related health disparities through the development and provision of tobacco-related research, training, and education to students and other early stage investigators underrepresented in the field. The program provides mentoring, support and pilot research project funding for early stage investigators; research training for undergraduate and graduate students; and enhanced CCNY clinical curriculum on knowledge and competencies about the treatment of tobacco dependence and tobacco-related disparities. Supported by a P20 Partnership Planning Grant, TREND aims to establish a sustainable, collaborative research and research training partnership, and develop procedures, curriculum, and evaluation processes to inform the establishment of a sustainable R25 training program in tobacco-related health disparities.

Program Similarities and Differences

In accordance with the NIH and IOM missions, all four training programs have an overarching goal to increase the number and diversity of addiction research scientists. There is significant synergy across the programs with regard to design, methods, and evaluation processes. For example, all programs offer mentored research experience and didactic training (e.g., seminar series, online educational modules, curriculum augmentation), three of the four programs provide pilot project funds, and two of the four provide funding to attend and/or present at national conferences. Evaluation endpoints are also similar and include objective outcomes (e.g., scholarly activity, academic promotion or career advancement, engagement in research-related activities) and processes deemed related to outcomes (e.g., research self-efficacy, mentorship quality, knowledge acquisition, career intentions). The programs also target diverse training levels: all four include undergraduate, graduate, and medical students, with one program also targeting medical residents and another program targeting early stage (i.e., post-doctoral, junior faculty) investigators. Program eligibility reflects the documented need to target students earlier in the academic pipeline, especially women and trainees from underrepresented groups, whose representation among researchers declines following the undergraduate education (Lautenberger et al., 2014; National Academy, 2011). Targeted disciplines are diverse: three of the programs train healthcare providers (i.e., nursing, dental, and medical students), two programs train psychology graduate students, and one program each focuses on psychiatry, social work, and substance use disorders counseling.

RECOMMENDATIONS FOR EVALUATION OF ADDICTION RESEARCH TRAINING PROGRAMS

1. Utilize standardized mixed-method evaluation measures

It is recommended that NIH utilize a set of core measures in all addiction training programs to enable cross-program collaboration and comparison, as well as improve ongoing trainee tracking and program evaluation. Standardized measurement would assist in assessing the impact of research training programs and improvements in the representation of a diverse scientific workforce. It would also allow for ongoing psychometric testing to improve the reliability and validity of process and outcome measures over time and with diverse populations of trainees (e.g., by demographics, by discipline, by training level).

Process evaluation is particularly important because traditional objective outcomes (e.g., publications, grants, academic promotion, career choice) are relatively long-term and low-frequency which may lack the sensitivity to capture variability or inform program adaptation. This is particularly true for students and other more junior trainees where traditional objective outcomes may be particularly far off. Measures of process evaluation might include: attitudes, beliefs, professional identity and research identity development, sensitivity of clinicians to addiction issues and to working with people with substance use disorders, and intention to and confidence in pursuing addiction-related research careers. The quality of mentorship is regarded as an important process-related factor. Measures to inform mentorship quality might include: trainee and mentor match, mentorship engagement process, and cultural understanding and support. In programs focused on younger trainees, greater practical support, structured supervision, and more explicit guidance of program expectations is needed.

2. Select measures based on conceptual models of scientific training or academic career development

Models to guide training and career development exist, but typically have not been used to guide addiction research training proposals. Admittedly, none of the programs described in this manuscript currently use a conceptual model of adult learning to guide the evaluation plan. It is recommended that addiction research training programs utilize evidence-based academic career development models (e.g., behavior change, non-traditional adult learning) and other theoretically-informed models with consistent and systematic outcomes that include process variables. The “Career-Success Model” developed at the University of Pittsburgh and later advocated by members of NIH’s Clinical and Translational Science Awards (CTSA) (Lee et al., 2012; Rubio et al., 2011) is one such possible model. Although developed specifically for training physician-scientists, the model has potential to be broadly applied. Importantly, career success (comprised of objective and subjective domains) is influenced by both personal (e.g., demographics, education, personality) and organizational factors (resources, mentoring, conflicting demands) (Rubio et al., 2011).

Other models and theories of academic training should also be explored and evaluated within addiction research training programs. These could include a) social cognitive theory (Lent, Brown, & Hackett, 1994) which has been applied to academic training and could be especially salient to understanding the mentoring process, b) the Theory of Proximal Development (Vygotsky, 1978) which incorporates the idea of providing “scaffolding” until a skill is learned, and c) the Progressive Mentoring Model (Santora, Mason, & Sheahan, 2013) which focuses on collaborative, bi-directional relationships and multiple levels of mentorship (i.e., expert, upper, mid, and lower level mentors). Logic models can be used to facilitate evaluation of education programs by targeting short, medium, and long-term outcomes (Van Melle, 2016).

3. Enhance mentorship support and mentorship quality

Mentorship “teams” that consist of mentors with diverse expertise at various career levels who co-mentor a trainee can be a particularly beneficial model for delivering high quality mentorship. As an example, SARET co-investigators, who are senior faculty from four disciplines, serve in the role of senior co-mentor for trainees who are also paired with a project-specific mentor. Across the training programs, mentors have vocalized the need for guidance and support to provide higher quality mentorship, especially when mentoring trainees from disciplines different from their own or trainees earlier in the academic pipeline. Exploring ways in which mentorship training could be integrated into the development and ongoing implementation of research training programs should be considered (Anderson, Silet, & Fleming, 2012). For example, the NIH-supported National Research Mentoring Network (NRMN) is a consortium of biomedical professionals, institutions and trainees with the goal of increasing the nation’s capacity for delivering evidence-based mentorship through training, virtual mentorship, and offering NIH-funded pilot funding increasing scientific workforce diversity (see www.nrmnet.net).

4. Integrate training and evaluation scholarship from other fields

For example, strong attention is given to evaluation within the inter-professional development field (e.g., Madden et al., 2006) and could be utilized for NIH addiction research training programs. Another important development within the NIH CTSAs is the “Key Function Committee on Evaluation” which produced a special issue of Evaluation and Health Professions devoted to describing evaluation across CTSA programs (Pincus et al., 2013). Although the overall goals, and thus evaluation needs, of the CTSAs are more complex and wide-ranging than addiction research training programs, there are likely cross-cutting themes, conceptualization, and identified resource needs that would be useful.

5. Promote long-term, technology supported, data collection

to describe the impact of addiction training programs and career development. The research career development trajectory is long, extending past the formal end of training programs. Developing a practical and feasible plan for longitudinal tracking is essential and best practices in tracking of scholars should be explored. For example, the SARET program has maintained up-to-date email address information for 65 of 68 past participants to date via Google, Facebook, etc., and recently received a 79% response rate to a long-term evaluation survey. This strong response rate is helped by a financial incentive provided to former trainees for completion of evaluation tools. Another possibility is to explore the use of a centralized tracking mechanism (e.g., NIH Commons) for NIH-funded trainees. The SARET program also tracks trainee progress over time using automated literature searches constructed in collaboration with university librarians. The TRACC program, which includes undergraduate trainees, is exploring ways to utilize long-term mentorship networks to provide post-training program support. This is especially important for trainees who do not necessarily move to the next level of research training to become independent investigators but are still interested and able to integrate collaborative research into their clinical careers.

Importantly, funding within the NIH training program mechanism must be allocated toward supporting the expenses associated with short- and long-term evaluation. Although the majority of funding within NIH training grants is appropriately allocated towards trainee resources, there is need to increase resources committed to formal evaluation such as for key personnel dedicated to program evaluation and analysis, as well as dissemination of outcomes.

6. Create training programs with diverse and interdisciplinary directors and investigators

Especially in programs that cross academic institutions and disciplines, it is important to partner with key stakeholders to support logistical needs of the program, such as class scheduling, regulations on absences (to attend conferences or participate in other research-related activities), curriculum requirements and integration of addiction content into existing courses (Ho et al., 2008). This can be more easily done in those instances where addiction training or research is a required part of pre-existing curriculum or focus, but this is often not the case in many health professional schools. The need for addiction research training across disciplines is increasing and it may be the case that schools will be looking for ways to enhance currently limited opportunities.

CONCLUSION

Research training programs are imperative to increasing the number, diversity and quality of addiction scientists available to effectively address the nation’s public health problem of addiction. Significant resources have been devoted to develop addiction research training programs that build the capacity and diversity of the academic pipeline, but much is yet to be done. Based on the four, NIH-sponsored research training programs described in this paper, we recommend core metric requirements for model-driven evaluation in future programs, as well as enhanced methods to track NIH-funded trainees, and analysis of aggregate process and outcomes data across programs. Standardizing measurements and synergizing across addiction research training programs will help advance the field and the quality of the next generation of addiction researchers.