Predicting Dissemination of a Disaster Mental Health “Train-the-trainer” Program

Abstract

Objective

Disaster mental health (DMH) is vital to comprehensive disaster preparedness for communities. A train-the-trainer (TTT) model is frequently used in public health to disseminate knowledge and skills to communities, although few studies have examined its success. We report on the development and implementation of a DMH TTT program and examine variables that predict dissemination.

Methods

This secondary analysis examines 140 community-based mental health providers’ participation in a TTT DMH program in 2005–06. Instructors’ dissemination of the training was followed for 12-months. Bivariate and multivariate analyses were conducted to predict dissemination of the training program.

Results

Sixty percent of the trainees in the DMH TTT program conducted trainings in the 12-month period following training. The likelihood of conducting trainings was predicted by a self-report measure of Perceptions of Transfer of Training. The number of individuals subsequently trained (559) was predicted by prior DMH training and gender. No other variables predicted dissemination of DMH training.

Conclusion

The TTT model was moderately successful in disseminating DMH training. Intervention at the organizational and individual level, as well as training modifications, may increase cost-effective dissemination of DMH training.

Introduction

Terrorist attacks on September 11, 2001, the catastrophic aftermath of disasters such as Hurricane Katrina in 2005, and the growing threat of a pandemic influenza together raise considerable concern about the nation’s public health preparedness.¹ Many state and community preparedness efforts focus largely on detecting the potential for deadly terrorist strikes, developing disease surveillance systems, and planning for mass evacuations from disaster-pending areas.^1,2 Disaster mental health sequelae, however, must also be addressed as part of comprehensive disaster preparedness.^3–6

Few states have developed comprehensive disaster mental health training programs that prepare a professional mental health workforce to respond to disaster victims’ psychosocial needs. Moreover, traditional mental health curricula, whether in medicine, psychology, social work or nursing professions, fail to include disaster-related training as a core element. Project Liberty, a community-based outreach program following the 9/11 terrorist attacks, was an extensive effort made in New York City and the surrounding area to train mental health professionals and paraprofessionals to provide supportive services.⁷ Services were effective in assisting community members to regain previous levels of functioning and to access mental health resources as needed.^7–9 The authors concluded, “… an effective public mental health response to terrorism requires a population-based approach that anticipates a wide range of individual and collective reactions, ranging from emotional distress, to changes in behavior, to emerging psychiatric illness.” ⁷

An effective population-based approach requires broad-based training of a critical mass of individuals who are prepared to respond to disasters and other public health emergencies when they strike.^{5, 10–12} A train-the-trainer (TTT) model is frequently used in business,¹³ education,¹⁴ and interventions,^15,16 to disseminate knowledge and skills. In this model, master trainers teach curriculum content, as well as the process of delivering the course, to instructor-trainees who are then charged with conducting subsequent trainings to target groups. Despite the wide-spread use of TTT programs for training large groups of people in a relatively brief period of time, few studies have examined TTT models, dissemination to the target audience or cost effectiveness. Evidence is particularly scant for public mental health programs including disaster mental health.¹⁷

To begin to address this gap in the literature, we report on naturally occurring dissemination of a DMH New York State training program developed through a partnership between the University of Rochester Medical Center (URMC), the New York State Department of Health, and the New York State Office of Mental Health. A TTT model was used to disseminate DMH curriculum to new instructors across the state. Given the preliminary nature of our evaluation of a secondary data set, primary research questions are: 1) How successful is a TTT model in disseminating a DMH program as part of a multi-community public health initiative? That is, do newly trained instructors proceed to train others as expected? 2) What factors predict which instructors will successfully disseminate, both in terms of predicting if the trainings were conducted and in terms of the number of second generation trainees ultimately trained?

Methods

Setting and DHM Training

Beginning in 2004, the New York State Department of Health funded the Department of Psychiatry at the URMC to develop, implement, and monitor a state-wide disaster mental health training initiative. The program included the development of evidence-informed curricula to train mental health professionals to respond to the psychological consequences of disaster.

Two master trainers (JH, VC) delivered the 3-day Disaster Mental Health TTT workshop across the state. A total of 8, 3-day train-the-trainer workshops were delivered to representatives of 51 counties. One master trainer (JH) provided ongoing technical assistance to newly trained instructors over the next 12 months. (Information on the DMH TTT program is available from the authors.)

Participants

County mental health directors selected participants from 57 counties. A total of 51 counties were represented; some counties sent more than two participants. Candidates were informed of the requirement to disseminate the program as part of the TTT model within 12 months of the completion of trainings held during 2005 and 2006.

Data collection

DMH instructor participants completed surveys before and after the training. Program facilitators collected data on the number of trainings new DMH instructors conducted 12 months following the trainings using a web-based portal. To receive credit for attending the training, second generation trainees in the community sites – the target group – were required to complete a survey that identified DMH instructor(s) by name and the date of training. For analyses, we credited instructors with completing trainings and the number of new trainees who received their training over 12 months. Instructor pairs were each given credit for co-facilitated training. This methodology provided an objective measure of program dissemination without relying on instructor self-reports.

Measures

The pre-training questionnaire consisted of several demographic and pre-training items including: gender, race, education level, previous disaster mental health training, and work. A 14-item, multiple choice Knowledge about Disaster Mental Health assessment designed for this study was also administered. Items included questions about reactions to disaster, purpose of mental health intervention, and administrative structures in emergencies. The post-training questionnaire included: 1) a second 14-item Knowledge about DMH questionnaire matched for difficulty; and, 2) a 14-item measure of Perception of Transfer of Training (PTT), based on three subscales of The Learning Transfer System Inventory (LTSI).¹⁸ The PTT assesses trainees’ perceptions about 3 factors that impede facilitator transfer of training: Motivation to Transfer, Capacity to Transfer, and Learning Readiness to do so. PTT internal reliability for this sample is high (Cronbach’s alpha = .85).

Impact of the DMH TTT was measured by the number of trainees who received DMH training following TTT workshop training, and is reported as frequency count data.

Data Analyses

Data was stored in Excel and transported into SPSS 16 for descriptive analysis. SAS (version 9.1) and Stata (version 9) were used to perform descriptive statistical analysis, logistic analysis and Zero-truncated Poisson analysis.¹⁹ Logistic regressions were used to model whether subsequent trainings were conducted by newly trained DMH instructors and variables with a p value of <.20 were entered into the full multivariate logistic regression model. For instructors who had conducted at least one subsequent DMH training, Zero-truncated Poisson regressions were used to model the number of the subsequent trainings. Again, simple logistic regressions were conducted and variables with a p value of <.20 were entered into the full multivariate Zero-truncated Poisson analysis.

The URMC Research Subjects Review Board approved this study.

Results

Demographics

Six (6) participants were excluded from analyses because they were judged to be limited in the ability to disseminate the training (i.e., retired, unemployed, or currently a student). Demographic information is presented in Table 1 for the sample (n = 134).

Table 1.

Participant Characteristics

Demographic and Pre-training Variables (N=134)
Variable		Mean or Freq	SD or proportion
Age		48.7	8.20
Gender	Male	51	38
	Female	83	62
Community size	<15,000	52	38.81
	15,000–75,000	27	20.15
	>75,000	55	41.04
Work Setting	community mental health setting/private practice	29	21.64
	healthcare setting (e.g., physician office)	16	11.94
	agency/clinic operated by government (county, state)	89	66.42
Role	primarily clinical	34	25.37
	primarily administrative	43	32.09
	combination clinical and administrative	45	33.58
	research/teaching/education	12	8.96
Degree/Education	MD/DO/PhD/PsyD	26	19.40
	less than doctorate	108	80.60
Prior DMH training	none	22	16.42
	1–4 hrs	17	12.69
	5–16 hrs	48	35.82
	17+ hrs	47	35.07
Prior DMH Experience	none	80	60.61
	1	26	19.70
	2+	26	19.70

Knowledge of Disaster Mental Health (KDMH)

Paired Wilcoxon Signed Rank Test assessed pre-post training Knowledge of Disaster Mental Health differences because the scores were not normally distributed. Results showed a significant increase in correct responses from pre-to post training. Participants’ post training scores (X =12.42, SD =1.33) were significantly higher than baseline knowledge scores (X = 9.43, SD =1.70), p < .0001.

Perception of Transfer of Training (PTT)

Participants completed the PPT immediately following the TTT workshop. Group mean was 44.9 (SD = 6.0).

Primary analyses

We are primarily interested in learning how likely it is that newly trained instructors will conduct trainings and what factors predict successful dissemination. Eighty of the 134 participants (59.7%) conducted one or more trainings (training at least 1 second generation participant). A bivariate logistic analysis was conducted to investigate the relationship between the likelihood that a newly trained instructor conducted any DMH trainings during the 12-month period following TTT workshop. All demographic data pre-training variables and total PTT were entered separately to predict the dependent variable, training, as a dichotomous variable. Variables included: community size, working setting, prior DMH experience and PTT. Results (Table 2) showed only PTT scores significantly predicted whether a newly trained DMH instructor conducted any training in the subsequent 12-month period (p. <.05). Newly trained instructors with higher PTT scores were more likely to conduct subsequent trainings. The odds ratio of conducting any subsequent training was 1.075 (CI: 1.006, 1.148) when the PTT score increases one unit.

Table 2.

Multivariate analysis predicting subsequent training (dichotomous) by newly trained instructors (n = 134)^*

Dependent Variable: Training or no training conducted
Variables		# of participants	% conducted subsequent training	Estimate	Standard Error	Wald chi-square	Pr > chisq	Odds Ratio	95% Wald Confidence Limits
Community Size	15,000–75,000	27	70.4%	0.1121	0.5607	0.0400	0.2833	1.119	0.373	3.357
	>75,000	55	50.9%	−0.5904	0.4527	1.7007		0.554	0.228	1.346
	<15,000	52	63.5%	-	-	-		-	-	-
Work setting	healthcare setting (e.g., physician office)	16	37.5%	−0.7858	0.6888	1.3017	0.0694	0.456	0.118	1.758
	agency/clinic operated by the government	89	65.17%	0.5292	0.4969	1.1342		1.698	0.641	4.496
	community mental health setting/private practice	29	55.2%	-	-	-		-	-	-
Prior DMH Experience	1	26	57.7%	−0.0187	0.4971	0.0041	0.5875	0.981	0.370	2.600
	2+	26	76.9%	0.5470	0.5494	0.9911		1.728	0.589	5.073
	None	80	56.3%	-	-	-		-	-	-
Perception of Transfer of Training (PTT]				0.0722	0.0337	4.5849	0.0323*	1.075	1.006	1.148

1Variables with p value from the bivariate model < .20 were included in the multivariable model

^*p < .05

Our second analysis (Table 3) used the same approach to predict the number of individuals trained by those DMH instructors (n=80). A total of 559 trainees were trained in the “second generation.” The significant variables were gender and prior DMH training hours, controlling for age, community, and PPT. On average, females trained 2.6 more new trainees than did males. Instructors with more prior DMH trainings (17+ hours) trained 4.9 more trainees than those without any prior DMH training.

Table 3.

Multivariate analysis predicting number of individuals trained by newly trained instructors 12 months after TTT (n = 80) *

Dependent Variable: Number of trainees trained by DMH instructors who conducted trainings (n = 559)
Variables		#of participants	Mean # trained in 12 months (SD)	Estimate	Standard Error	Z score	Pr > ChiSq
Gender	Female	46	14.3 (6.76)	.2090534	.0718324	−2.91	0.004**
	Male	34	11.2 (7.22)	-	-	-
Community size	15,000–75,000	19	13.1 (8.13)	−.0878182	.0791883	−1.11	0.0647
	>75,000	28	11.4 (5.92)	−.2511687	.0770749	−3.26
	<15,000	33	14.2 (7.29)	-	-	-
Prior DMH Training	1–4 hrs	8	10.3 (4.13)	−.036213	.1492882	−0.24	0.0008***
	5–16 hrs	20	12.0 (7.10)	.1829854	.1212724	1.51
	17+ hrs	32	15.2 (7.83)	.368119	.1186003	3.10
	None	10	11.0 (4.52)	-	-	-
Age				.0043011	.0042236	1.02	0.309
Perception of Transfer of Training (PTT)				.0081048	.0061685	1.31	0.189

1variables with p value from the bivariate model < .20 were included in the multivariable

^**p<.01

^***p<.001

Discussion

We found widespread interest in New York communities to include Disaster Mental Health training in a comprehensive preparedness plan. Almost every invited county sent two trainees to a TTT program with the understanding that the 3-day training would result in subsequent dissemination of the DMH program by the newly trained instructors to other mental health professionals in their communities. In addition, training resulted in a significant increase in knowledge about DMH for newly trained instructors. Sixty percent (60%) of TTT participants conducted DMH trainings during the following 12-months, which resulted in 559 “second generation” DMH trainees. Therefore, despite being chosen as “good candidates” by mental health leadership in their counties and a clear expectation that dissemination was a criterion for attending the training, 40% of trainees did not provide training to others.

To date, TTT programs have not routinely measured success in achieving the goal of widespread dissemination. The assumption is that the training will “spread” as intended and the TTT model proliferates. Our finding that 40% of the participants in the TTT for DMH training did not use their training as intended must be measured against the zero return. Participants were absent from their workplaces for 3 days, incurring loss of productivity. The resources involved in the training itself, as well as the food and lodging of participants, add further to the calculable expense. Given this scenario and fiscal constraints across the public and private sectors, it is critical to understand the obstacles and facilitating factors for successful TTT programs in DMH. In particular, “low tech” (i.e., low cost, easily implemented, little reliance on technology) strategies to enhance outcomes could improve the cost effectiveness of the popular model.

We did not directly assess transfer factors at the 51 participant sites --the cost to do so was beyond the scope of the training. Instead, we administered a brief self-report of participants’ perceptions. Although it may seem self-evident that workplace factors are likely predictors of subsequent trainings, the current study demonstrates that trainees from a variety of settings accurately assess the likelihood that they will disseminate the DMH training. The practical implication for TTT programs is that a simple, inexpensive survey of trainee perceptions predicts successful dissemination providing training administrators with the opportunity to intervene and increase the impact of the training. Intervention may occur at the organizational level (e.g., contact supervisors about resources required for subsequent trainings or the importance of clear supervisory support in the workplace for dissemination of DMH training), or individual level (e.g., provide additional opportunities for practice and technical support for those who report low levels of readiness). An extension of this approach would be to assess trainee perceptions of transfer of training before conducting DMH TTT programs. It may be that potential trainees can provide important information about workplace or site readiness to disseminate training.

Our data indicate that participants with the most prior training trained an average of almost 5 more individuals than those participants who had no previous or less experience. From an adult learning perspective, new learning is most meaningful and retained when the learner’s previous experiences are valued and integrated with new learning.²⁰ A training program that has the dual goal of learning new content, as well as learning the process of training others, may not be reasonable for trainees without previous experience to support their transformation to “expert trainer” role. Thus, for those individuals who went on to train others, previous relevant training experience provided a foundation for the DMH TTT program. The curriculum and training format presented here appears to have been most successful for participants with 2 days (or more) of previous DMH training. One “low tech” strategy to enhance TTT success may be to select participants with significant previous training.

An alternate strategy would be to modify the current training. Two levels of training could be offered – a beginner program for those with minimal previous DMH training which would focus heavily on content and process skills, and an advanced program for those with prior training, which would review content and focus more on process skills. Training methods could also be a factor in the ultimate dissemination of the DMH curriculum. One training approach is to incorporate active learning approaches. Behavioral rehearsal and role play practice, small group problem-solving, and expert feedback have been shown to enhance active engagement and learning among adult learners in a variety of settings¹⁴ and to enhance self-efficacy and readiness to transfer training to the workplace.^21–23 Moreover, these activities provide the opportunity for instructors to assess, either formally or informally, trainee knowledge and skills about the content as well as the process of training others.

On average, females trained 2.5 more individuals than males. We are not certain why women ultimately trained more individuals than men. Further study of how gender may be a factor in DMH training is recommended as is replication of these findings.

There are clear limitations to this study. In terms of design, we report on a secondary data analysis of a limited number of variables in a naturally occurring implementation and dissemination of a DMH curriculum. The study lacks the scientific rigor of random assignment and systematic procedures for follow-up contact with new instructors. Primary data collection on the newly trained instructors was limited over the follow-up period to whether they were named as instructors by new trainees. Although this is an objective measure superior to self-report, errors --including underreporting if instructors trained after the one year window--may have occurred. Observational assessment of actual skills in delivering the training would be a useful measure of trainer ability and may be associated with the number of individuals subsequently trained.²⁴ Observational methods are, however, prohibitively costly for most program evaluations. Finally, our findings may not be generalizable to other DMH trainings or other TTT programs.

Despite the limitations, this is the first report on a TTT model for DMH training dissemination in the field that we are aware of which provides methods for other public health education efforts. Our findings suggest the success of TTT procedures can be measured and cost effectiveness should be considered. The most important outcome measure is whether trainees (new instructors or their trainees) actually go on DMH missions as a consequence of participation in the training. Future studies might include longitudinal follow-up of a cohort of “first” and “second” generation trainees to examine how prepared these professionals were to meet the challenge of disaster victims’ mental health needs.