Large-Scale Evaluation of Satisfaction, Intent to Use, and Confidence with an Online Learning Course for a Well-Established, Evidence-Based Trauma Treatment: TFCBTWeb2.0

Abstract

Web-courses for learning evidence-based treatments (EBTs) are increasingly used to improve the dissemination and implementation of evidence-based practice in psychology. Most research on web-courses has focused on engagement and knowledge acquisition, and limited research has evaluated learner satisfaction with training and intent to use EBTs. Further, even when aspects of satisfaction and learner intentions are examined, factors that may contribute to these variables are often overlooked. The current study sought to address these limitations by examining individual, training, and system level factors related to web-course satisfaction, intent to use EBTs, and confidence working with trauma-exposed families following completion of TFCBTWeb2.0, a publicly available web-course for trauma-focused cognitive-behavioral therapy (TF-CBT). Data from 80,749 learners who completed TFCBTWeb2.0 and who represented a wide range of professionals were examined. Most learners (> 90%) were satisfied with the course, intended to use the material, and had high confidence about working with trauma-exposed families. Moreover, those learners who completed the course more slowly (vs. faster completion time) and had more knowledge gain, as well as those who enrolled during the COVID-19 pandemic (vs. pre-pandemic), reported higher satisfaction with the course. Collectively, these results highlight the need to examine both individual and system level factors when considering web-course satisfaction, intent to use EBTs, and confidence working with client populations. Suggestions for TF-CBT trainers and supervisors are provided for how to improve training and learners’ willingness to use TF-CBT.

Web-based training courses have been developed for a wide variety of evidence-based treatments (EBTs) for psychological concerns among children (Jackson et al., 2018). Typically, web-courses require learners to complete a set of self-paced modules that explain and sometimes demonstrate key concepts and strategies. Web-courses can be used independently or as supplements to other training efforts or approaches. For example, learners of Parent-Child Interaction Therapy (PCIT) can choose between in-person, web-based synchronous, or web-based asynchronous training options (e.g., PCIT Incorporated, 2021). Alternatively, learners who wish to attend any approved workshop training in Trauma-Focused Cognitive-Behavior Therapy (TF-CBT; Cohen et al., 2017) must complete a web-course prior to the workshop (e.g., TF-CBT National Therapist Certification Program, 2024).

Evidence across both small-scale (e.g., learners from a single institution or state, < 100 learners) and large-scale (e.g., learners from varied professions, > 1000 learners) web-course evaluations has found that such courses are acceptable to professional audiences and effective in improving knowledge of EBTs. For example, Heck and colleagues (2015) and Kasparik and colleagues (2022) both conducted evaluations of English and German web-courses for TF-CBT (Cohen et al., 2009; 2017) among large samples of behavioral health and medical providers. Both studies found increased knowledge of the treatment components among learners, and that earlier career individuals were more likely to complete the course compared to learners with more experience. In another example, Maguire and colleagues (2020) examined The Essentials, a web-course designed to help providers learn to assess and treat eating disorders in children and adults. In their sample of over 1000 providers, pre- to post-course self-reported improvements were observed in confidence, knowledge, skills, and attitudes toward working with individuals with eating disorders. Although there is considerable variability across web-courses in terms of the training strategies and technological elements, extant studies support the idea that web-courses have become an effective and integral part of training for many child-focused EBTs and, more broadly, a critical part of EBT dissemination and implementation efforts (Jackson et al., 2018). The utility of web-based training was highlighted during the COVID-19 pandemic, when in-person training efforts were halted due to community efforts to stop the spread of the virus, and virtual training strategies were widely adopted (e.g., Frye et al., 2022; Kern & Tague, 2022).

Despite their increased use, most published work has focused on web-course engagement and knowledge gain, overlooking factors such as learner satisfaction, intent to use the EBT, and confidence working with the populations the EBTs intends to support. Although limited evidence exists for web-courses, empirical evidence from other training approaches suggests that such aspects of training experiences relate to important provider and patient outcomes. For example, beliefs about intent to use and confidence/self-efficacy with trauma focused EBTs post-training have been found to be associated with greater implementation of the EBT with patients in the months following training (e.g., Ackland et al., 2023; Ruzek et al., 2017). Understanding satisfaction with these types of training approaches may be also critical to improving training practices within larger EBT training models. For example, if a web-course serves as an individual’s first introduction to an EBT, or is a key step in obtaining more advanced training. A poor experience (e.g., low satisfaction) with that web-course could deter an individual from seeking additional training and using the EBT in practice. This may especially be the case for treatments that involve elements of exposure therapy for children and adolescents, which many therapists approach with concern (Neelakantan et al., 2019).

Indeed, when satisfaction is assessed, not all learners who engage in web-course trainings report positive experiences. For example, satisfaction responses from small-scale evaluations of web-courses for CBT and other treatments with both adults and children tend to find that anywhere from 5% to 20% of participants report low to very low satisfaction (e.g., Bennett-Levy et al., 2012; Gryglewicz et al., 2017). Initial evidence from large-scale studies from more widely available web-courses suggests similar patterns. For example, Maguire and colleagues (2020) found that approximately 6% of web-course completers reported being unsatisfied or very unsatisfied with The Essentials. In their evaluation of the German TF-CBT web-course, Kasparik and colleagues (2022) also found some indicators of low satisfaction and hesitancy to deliver the treatment. Collectively, the findings suggest that some learners may have negative experiences with web-courses or may not feel able to use the target treatment in practice, which in turn has the potential to negatively influence EBT implementation. However, satisfaction, intent to use, and confidence factors related to web-course trainings are rarely examined. Even when these variables are examined, they are typically considered only at the level of the entire sample and in a descriptive format. Therefore, it is unclear which learners may be least satisfied or most reluctant to implement a treatment, and why. According to a systems-contextual (SC) approach to improving evidence-based practice (Beidas & Kendall, 2010), researchers should identify factors beyond the training itself (e.g., knowledge gained, length of training), such as characteristics of learners and the contexts under which they are participating. This knowledge may help inform noted gaps in use of a treatment with actual patients following training (e.g., Frank et al., 2021). Thus, an important step for EBT dissemination and implementation efforts is to increase understanding of individual and system level factors that may explain course satisfaction, hesitancy to use EBTs, and confidence following web-based trainings.

In related research on other types of youth treatment training approaches, factors beyond the training experience at both the individual and system/environmental level play an important role in satisfaction, intent to use EBTs, and confidence working with the intended population. For example, agency/organizational support have been linked to learner satisfaction in workshop, supervision, and other types of intervention training, often suggesting that greater perceived support increases participation and satisfaction with training (Herschell et al., 2010; Marriot et al., 2023). Further, evaluations of web-courses for EBTs have also shown that individual and system level factors appear to contribute to completion rates, knowledge gain, and other efficacy related web-course components. For example, early career learners tended to complete web-courses for TF-CBT at higher rates than later career learners (e.g., Heck et al., 2015). Despite these potential differences identified in other EBT training domains, no research to date has explored whether learner characteristics or context of the learning environment relate to satisfaction, intent to use the target EBTs, and confidence working with the target populations of web-course trainings.

Current Study

To this end, the current study examined multiple aspects of satisfaction, intent to use, and confidence from an evaluation survey from learners who completed TFCBTWeb2.0 (Table 1), a publicly available web-course focused on understanding and implementing TF-CBT (Cohen et al., 2017), a well-established treatment for children exposed to trauma (Dorsey et al., 2017). Building from an SC perspective, the current study explored both individual level (e.g., professional identity, degree, years of experience) and system/context level factors (e.g., participation during the COVID-19 pandemic, sponsorship or funding for web-course registration payment), in addition to training related factors (i.e., completion time, knowledge gained), that may contribute to multiple aspects of learner satisfaction, intent to use EBTs, and confidence working with trauma exposed children and families (Figure 1). This was achieved through evaluation of registration, completion, and course evaluation for TFCBTWeb2.0. Given the lack of previous empirical evidence, most analyses were exploratory. However, given findings related to learner characteristics and overall web-course engagement (e.g., completion rates; Heck et al., 2015), it was hypothesized that earlier career learners (e.g., those with less than five years of experience) would be (1) more satisfied with the course formatting and course material and (2) more likely to apply the material to practice, compared to later career learners (e.g., learners with more than 10 years of experience).

Table 1.

TFCBTWeb2.0 Evaluation Survey Questions

Survey Question	Survey Component
1. The Web site was easy to navigate and find my way around.	User Experience Satisfaction
2. The material was presented in a manner that made it easy to learn.	User Experience Satisfaction
3. The Web site has a pleasant “look and feel” (i.e., pictures, fonts, graphics, colors, designs).	User Experience Satisfaction
4. The Web site has directions that are clear and easy to follow.	User Experience Satisfaction
5. The video demonstrations helped me better understand the written material.	User Experience Satisfaction
6. The Developmental Considerations helped me understand how to apply techniques to different age groups.	Course Elements Satisfaction
7. The Clinical Challenges sections helped me understand how to adapt the treatment techniques for my clients.	Course Elements Satisfaction
8. Information from the Cultural Considerations section helped me understand how to implement the treatment with culturally diverse groups.	Course Elements Satisfaction
9. The step-by-step technique instructions were useful in learning the treatment techniques.	Course Elements Satisfaction
10. The Foundations of TF-CBT module helped me understand for what clients TF-CBT is appropriate and not appropriate.	TF-CBT Component Satisfaction
11. I understand the principles and components of TF-CBT.	TF-CBT Component Satisfaction
12. I understand how to structure and conduct a typical TF-CBT session.	TF-CBT Component Satisfaction
13. The course helped me understand the key components of Psychoeducation about Trauma.	TF-CBT Component Satisfaction
14. The course helped me understand the key components of Parenting and Behavior Management Strategies.	TF-CBT Component Satisfaction
15. The course helped me understand the key components of Relaxation Techniques.	TF-CBT Component Satisfaction
16. The course helped me understand the key components of Affect Identification and Regulation.	TF-CBT Component Satisfaction
17. The course helped me understand the key components of Cognitive Coping.	TF-CBT Component Satisfaction
18. The course helped me understand the key components of Creating the Trauma Narrative.	TF-CBT Component Satisfaction
19. The course helped me understand the key components of Cognitive Processing of the Trauma.	TF-CBT Component Satisfaction
20. The course helped me understand the key components of Sharing the Trauma Narrative with caregivers.	TF-CBT Component Satisfaction
21. The course helped me understand the key components of facilitating In Vivo Mastery to overcome avoidance symptoms.	TF-CBT Component Satisfaction
22. The course helped me understand the key components of Enhancing Safety and Future Development.	TF-CBT Component Satisfaction
23. I am likely to use TF-CBT with many of my clients.	Intent to Use
24. I am likely to use evidence-based treatments with many of my clients.	Intent to Use
25. I feel more confident about my ability to work with traumatized children and their families.	Confidence

TF-CBT = Trauma-focused Cognitive-Behavioral Therapy.

Figure 1.

Hypothesized Model

Q = Corresponds to the item number on the evaluation survey. The indicator variables and the factor loadings for the three satisfaction latent variables were modeled; although, they are not shown in the model above for reader clarity.

Methods

Online Web-Course Description and Procedure

The current study utilized data from learners who completed TFCBTWeb2.0 (https://www.musc.edu/tfcbt2), an 11-module web-course covering the theoretical foundations and session-by-session instructions for delivering TF-CBT (Cohen et al., 2017). Each module provides learners with instructions on implementing TF-CBT techniques; video demonstrations of treatment skills; both cultural and developmental considerations in treatment delivery; and information on addressing frequently experienced clinical challenges. Specific modules include: foundations of TF-CBT, psychoeducation, parenting skills, relaxation, affect identification and regulation, cognitive coping, trauma narration and processing (I & II), in vivo mastery, conjoint parent-child sessions, and enhancing safety and future development (Cohen et al., 2017). Each module was created to require approximately one hour to complete. This time to read the provided text, watch the module’s embedded demonstrations, and complete pre- and post-tests. To enroll in the web-course, individuals completed an online registration and provided payment ($35.00 enrollment fee). Some learners had their enrollment fee waived and were provided complementary access to TFCBTWeb2.0. Typically, this involved an employer or educational institution paying for learner registrations. Progress through the course is self-paced, and learners completed a pre- and post-module knowledge test for each module. Post-test completion is required to access the subsequent module. After completing all modules, if learners wanted to obtain an official completion certificate, learners were prompted to complete the course evaluation. Completion certificates are typically required for course credit, verification of completion to employers, and for national certification. Also, licensed psychologists were required to complete the evaluation survey to obtain access to the post-course test if they wished to receive American Psychological Association (APA) approved continuing education units. Data were de-identified and considered program evaluation, so informed consent was not obtained.

Participants (i.e., Learners)

For the purposes of the current study, all learners who had registered, paid (or been paid for), and completed all 11 modules of TFCBTWeb2.0 were eligible. Learners were excluded if they did not complete the course evaluation, or if they completed the course in less than 5 hours, which is approximately less than half the 11-hour intended completion time. This exclusion eliminated course completers who could not have meaningfully engaged in the course material (e.g., not watching all the video demonstrations, not reading all the text). Data were extracted at the end of January 2023 for those who had enrolled in the web-course prior to July 1^st, 2022. This time frame was chosen as it permitted all learners at least six months to complete the course. Since TFCBTWeb2.0 was launched in January 2018, this provided data from learners who registered over the course of approximately 4.5 years.

Measures

Learner Characteristics.

Learners’ demographic characteristics were extracted from the course registration database. This included information on professional identity (such as psychology, clinical counseling, social work, etc.), years of experience (0–5, 6–10, 10–20, and 20+ years of work experience), and degree type (e.g., MA, MSW, Ph.D., MD). Learners also provided information on their gender, which was a non-forced response question that allowed learners to select either (a) male, (b) female, (c) other or no response. Further, learners provided information on their current location (i.e., country or state for learners in the United States). Learners were coded as to whether they enrolled and paid independently (self-enrollment) or whether they received complimentary access to the web-course (sponsored enrollment). Years of experience, measured as a 4-point ordinal variable from least to most years, and enrollment type (0 = sponsored enrollment, 1 = self-enrollment) were included in the structural model as a predictor variables for satisfaction factors, intent to use, and confidence.

Web-Course Timing.

Information on learner engagement in the web-course was extracted from course meta-data. For the purposes of the current study, data were extracted on date of registration and date of course completion (i.e., completing the post-test of the last module). These dates were then used to calculate the total completion time, which was included in the structural model as a continuous predictor variable. An indicator of enrollment relative to the COVID-19 pandemic onset was created using the registration date, where learners were categorized into pre-pandemic and peri-pandemic group based on whether the registration occurred after 03/31/2020. The end of March 2020 was selected as the COVID-19 date because this was an approximate time when most of the U.S. had enacted lockdown orders (Centers for Disease Control and Prevention, 2023). This binary variable (0 = peri-COVID-19 enrollment, 1 = pre-COVID-19 enrollment) was also included in the structural model predicting the aspects of satisfaction, intent to use, and confidence.

Knowledge Acquisition.

To examine learners’ knowledge acquisition, learners were required to complete a pre-test before each web-course module and then they were asked the same set of questions at the end of the module. Each set of questions pertained to information presented in that module. Questions for each module were developed by the treatment and web-course developers and were based on items adapted from the first iteration of TFCBTWeb (Smith & Saunders, 2005). Learners were not given the correct answers for each test until after completion of the post-test. For the current study, the pre-test and post-test sum scores for each module for each web-course were extracted. A knowledge change score was calculated for each module by subtracting learners’ pre-test score from the post-test score. The difference scores for all of the 11 modules were then summed together to create a total knowledge change score, which was included as a predictor variable in the structural model.

Course Evaluation.

Learners completed a 25-item evaluation survey adapted from the original TFCBTWeb (Smith & Saunders, 2005). The full course evaluation is provided in Table 1. The evaluation survey included three satisfaction domains. Five items pertained to User Experience Satisfaction, which focused on the web-course’s presentation. Four items assessed Course Elements Satisfaction, which focused on specific types of content that are included in each course module (e.g., Developmental Considerations). Third, there was one item per module assessing the degree to which the course facilitated the learner’s understanding of each module’s content, with the exception that there were three items related to the content of Module 1- Foundations of TF-CBT. These 13 items were combined to form the TF-CBT Components Satisfactionsubscale. In addition to these measures of satisfaction, there were two items related to Intent to Use, the first assessing intention to use TF-CBT after completing the course, and the second assessing intention to use EBTs, generally, after the course. Finally, one item assessed Confidence in working with traumatized children and families. For all items, responses were made on a four-point Likert-scale from 1 (“Strongly Disagree”) to 4 (“Strongly Agree”). The User Experience Satisfaction, Course Elements Satisfaction, and TF-CBT Components Satisfaction subscales were examined as latent variables in the full structural equation model; the two Intent to Use items and single Confidence item were examined separately in the model as endogenous variables.

Data Analysis

All data analyses were performed in SPSS Software V.27 (IBM Corporation, 2020) and R software V.4.2.3 (R Core Team, 2023). In the first part of the analysis, differences in completion status for the satisfaction survey among those who completed the web-course were evaluated across the variables of interest (i.e., years of experience, COVID-19 Status, and Invitation Status) using chi-square (χ²) tests of independence and by Cramér’s V statistic. Significant differences via the χ² tests were further examined using post-hoc, pairwise z-tests.

In the next part of the analyses, the contributions of training and contextual factors to dimensions of learner satisfaction, intent to use, and confidence were analyzed using structural equation modeling (SEM) through the lavaan package in R (Rosseel, 2012). The hypothesized model is shown in Figure 1. From the course evaluation, three satisfaction latent variables were created for the User Experience Satisfaction (5 items), Course Elements Satisfaction (4 items), and TF-CBT Components Satisfaction (13 items) components of the survey, which each served as dependent variables. Additionally, the two Intent to Use items and Confidence item were included as dependent variables, each as individual outcomes. The covariances between the User Experience Satisfaction, Course Elements Satisfaction, and TF-CBT Component Satisfaction latent variables, as well as each of two Intent to Use and single Confidence items, were modeled. These dependent variables were regressed onto all the following predictor variables: (a) years of experience, (b) total knowledge gain, (c) completion time, (d) COVID-19 group, and (e) enrollment type. This also included the three latent variables for satisfaction when predicting the Intent to Use and Confidence items. The professional identity or degree type variables were not included as predictors in the SEM analyses due to the imprecision of some grouping characteristics (e.g., master’s degree learners might hold terminal master’s or be progressing toward a doctoral degree) and potential overlap with other predictors (e.g., student status and years of experience).

To build the model, a robust weighted least squares with mean and variance adjustment (WLSMV) estimator was utilized, which has been shown to estimate more accurate and unbiased model parameters when using items on an ordinal scale (e.g., Li, 2016; Tarka, 2017). Among the 79,430 learners who completed the web-course and satisfaction survey, there were 8572 learners (10.79%) who had at least one item on the course evaluation missing. The percentage missing for each item ranged from 2.2% to 2.9%. This was because for learners who registered prior to October 2019 (i.e., 1.75 years after the course launched in January 2018), the course evaluation was not programmed to require responses for all items prior to completion. After October 2019, learners had to answer each question, or the evaluation would not be submitted. An additional 39 (<.01%) learners did not have start and completion times captured in the database due to a programming error. Thus, missing data were considered missing at random and managed using multiple imputation by fully conditional specification (imputations = 10). This approach has been shown to outperform other methods of missing data management with large datasets that include continuous and non-continuous variables (Liu & De, 2015). To evaluate model fit, the following fit indices were evaluated using the following cutoff values (Kline, 2015; Hu & Bentler, 1999): the chi-squared test statistic, the null root-mean-square residual error of approximation (RMSEA; ≤ .08), the standardized root mean square residual (SRMR; ≤ .08), the comparative fit index (CFI; ≥ .90), and the Tucker-Lewis Index (TLI; ≥ .90).

Results

The characteristics and group proportions for the learners included in the current study are provided in Table 2. During the study’s timeframe, 174,193 learners registered for TFCBTWeb2.0, and among those, 88,503 (50.81%) paid for and completed the course. Among the 88,503 learners who completed, 7754 learners (8.76%) were removed from the data analysis for completing the course in five hours or less (i.e., Invalid Completers- Table 2). This resulted in a final sample size of 80,749 learned who completed the web-course. On average, web-course completers finished the course in 42.62 days (SD = 112.23; Median = 9.22; range: 5 hours to 1761 days). Most learners identified as female (83.05%), followed by male (11.91%) and “Other” (.40%); 4.64% declined to answer the gender question. Most learners identified their profession as social work (45.14%), followed by counseling (31.00%), psychology (12.88%), marriage/family therapy (7.99%), nursing (1.17%), psychiatry (.95%), and then no response (.87%). Moreover, most learners reported having a master’s degree (70.38%), followed by pre-master’s degree student (24.90%), doctoral degree (4.05%), and medical degree (.66%). Learners who completed the course reported representation from over 100 countries, with the majority of learners (94.37%) being from the U.S. (including states, territories, or freely associated states).

Table 2.

Learner Demographics for Completers of TFCBTWeb2.0

	Valid Completers				Invalid Completers
Learner Characteristics	Total Completers N (% of Group)	Completers with Satisfaction Survey n (% of Group)	Completers without Satisfaction Survey n (% of Group)	Percentage of Completers who finished Satisfaction Survey- χ2 (df) / Cramér’s V	N (% of Group)
Years of Experience				19.591 (3) / .016 ***
<5 Years	55510 (68.74%)	54505 (68.70%)	1005 (71.33%)	98.19%a	5662 (73.02%)
5–10 Years	13564 (16.80%)	13316 (16.78%)	248 (17.60%)	98.17% a	1234 (15.91%)
10–20 Years	7901 (9.78%)	7812 (9.85%)	89 (6.32%)	98.87%b	642 (8.28%)
20+ Years	3774 (4.67%)	3707 (4.67%)	67 (4.76%)	98.22% a,b	216 (2.79%)
COVID-19 Status				43.077 (1) / .023 ***
Peri-COVID-19 Pandemic	40065 (49.62%)	39488 (49.77%)	577 (40.95%)	98.6%a	3710 (47.85%)
Pre-COVID-19 Pandemic	40684 (50.38%)	39852 (50.23%)	832 (59.05%)	98.0%b	4044 (52.15%)
Invitation Status				23.015 (1) / .017 ***
Self-Enrollment	70461 (87.26%)	69172 (87.18%)	1289 (91.48%)	98.2%a	6959 (89.75%)
Sponsored Enrollment	10288 (12.74%)	10168 (12.82%)	120 (8.52%)	98.8%b	795 (10.25%)

Bold values and

^***= p < .0005

^**= .0005 ≤ p < .001

^*= .001 ≤ p < .005. Total Valid Completers N = 80749. Total Invalid Completers N = 7754. Percentages represent the proportion of learners per each characteristic variable per column based on completion status. Proportion values with similar subscripts indicate non-significant differences between those groups, as determined by a post-hoc, pairwise z-tests (p < .0005 for all significant differences observed).

Group proportions for differences in completion rates of the course evaluation were examined across learner characteristics (Table 1). Across all learner characteristics, over 98% of learners finished the course evaluation. Several significant differences emerged among all groups of characteristics suggested that certain proportion of learners were more likely to complete the evaluation compared to others (all χ²’s > 19.591, all p’s < .0005). Learners with 10–20 years of experience were more likely than learners with 5–10 or less than five years to complete the evaluation. Further, peri-COVID-19 pandemic enrollees and learners whose enrollment was sponsored were more likely to complete the evaluation. All effect sizes for these differences were small, as indicated by all Cramér’s V’s < .03.

Course Evaluation Item Responses

The mean values for each evaluation item are presented in Supplementary Table S1, and the correlations between each of the items and primary study variables are presented in Supplementary Table S2. For all evaluation items, responses ranged from 1 (i.e., Strongly Disagree) to 4 (i.e., Strongly Agree), and all mean values were greater than 3.30. For all items, 2.1% to 5.4% of all learners rated the item in Strongly Disagree or Disagree. The highest satisfaction ratings for the web-course structure and content items were found for item 1 (“The Web site was easy to navigate and find my way around”), item 4 (“The Web site has directions that are clear and easy to follow”), and item 9 (“The step-by-step technique instructions were useful in learning the treatment techniques”). Among the 75,165 learners who completed all 13 TF-CBT Component Satisfaction items, the overall module mean satisfaction score was 3.47 (SD = .55; range 1.00–4.00), indicating a quite high level of satisfaction. Just 1.85% had a mean module score below 2.00 (in the Disagree to Strongly Disagree range). The highest rated items pertained to Module 1 (Foundations of TF-CBT), Module 7 (Trauma Narration and Processing I) and Module 10 (Conjoint Parent-Child Sessions). The lowest rated items pertained to Module 1 (Foundations of TF-CBT; item 12) and Module 3 (Parenting; item 14). Additionally, high mean values were observed for Intent to Use and Confidence items; item 24 (“I am likely to use evidence-based treatments with many of my clients”) having the highest mean among the items.

Model Examination

The model results are presented in Table 3. The model demonstrated satisfactory fit, χ² (368) = 27336.245, p < .0005, RMSEA_(.030-.031)= .030, SRMR= .020, CFI = .997, TLI= .998. All estimated covariances and all factor loadings for the course elements, user experience, and TF-CBT components satisfaction latent variables were significant (all p’s < .0005). There was minimal variance explained by the predictors when examining the three satisfaction latent variables (i.e., all R²’s = .004). There was notable variance explained for the two Intent to Use outcomes related to both TF-CBT and general EBTs, as well as for the Confidence outcome (i.e., all R²’s > .460).

Table 3.

Path Estimates for the Structural Equation Model

	Full Path Model Results
	User Experience Satisfaction		Course Elements Satisfaction		TF-CBT Components Satisfaction		Future Use of TF-CBT		Future Use of EBTs		Working with Traumatized Children/Families
Path Model Parameters	Unstandardized Coef. Estimates (95% CI)	Standardized Coef. Estimates	Unstandardized Coef. Estimates (95% CI)	Standardized Coef. Estimates	Unstandardized Coef. Estimates (95% CI)	Standardized Coef. Estimates	Unstandardized Coef. Estimates (95% CI)	Standardized Coef. Estimates	Unstandardized Coef. Estimates (95% CI)	Standardized Coef. Estimates	Unstandardized Coef. Estimates (95% CI)	Standardized Coef. Estimates
Years of Experience	−.024 (−.033,−.016)	−.021 ***	−.034 (−.043,−.206)	−.029 ***	−.029 (−.037,−.021)	−.024 ***	.004 (.000,.008)	.005	−.010 (−.013,−.006)	−.014 ***	.007 (.003,.010)	.009 ***
Total Knowledge Gain	.002 (.001,.003)	.018 ***	.004 (.002,.005)	.026 ***	.003 (.002,.004)	.022 ***	−.001 (−.002,−.001)	−.014 ***	−.001 (−.002,−.001)	−.017 ***	−.001 (−.001,−.000)	−.007*
Completion Time	.000 (.000,.000)	.026 ***	.000 (.000,.000)	.011*	.000 (.000,.000)	.017 ***	.000 (.000,.000)	.009 ***	.000 (.000,.000)	.011 ***	.000 (.000,.000)	.008 ***
Enrollment Type (Ref: Self)	−.084 (−.107,−.062)	−.028 ***	−.081 (−.103,−.058)	−.027 ***	−.093 (−.114,−.071)	−.031 ***	−.016 (−.026,−.006)	−.008**	−.008 (−.017,.000)	−.005	−.019 (−.027,−.011)	−.010 ***
COVID-19 Enroll (Ref: Peri-)	−.090 (−.105,−.706)	−.045 ***	−.081 (−.096,−.067)	−.040 ***	−.073 (−.087,−.058)	−.036 ***	.017 (.010,.024)	.013 ***	.002 (−.004,.007)	.002	.013 (.007,.019)	.011 ***
User Experience Satisfaction		NA		NA		NA	.086 (.073,.100)	.133 ***	.146 (.134,.159)	.247 ***	.066 (.054,.078)	.107 ***
Course Elements Satisfaction		NA		NA		NA	.105 (.088,.121)	.161 ***	.091 (.076,.105)	.151 ***	.082 (.068,.096)	.135 ***
TF-CBT Components Satisfaction		NA		NA		NA	.268 (.257,.279)	.413 ***	.230 (.220,.230)	.385 ***	.334 (.324,.343)	.546 ***
Model R2	.004		.004		.004		.465		.567		.585
	User Experience Satisfaction		Course Elements Satisfaction		TF-CBT Components Satisfaction
Measurement Models’ Factor Loadings	Standardized Coef. Estimates	R 2	Standardized Coef. Estimates	R 2	Standardized Coef. Estimates	R 2
Item 1	.810 ***	.656	NA		NA
Item 2	.855 ***	.731	NA		NA
Item 3	.825 ***	.680	NA		NA
Item 4	.866 ***	.751	NA		NA
Item 5	.770 ***	.593	NA		NA
Item 6	NA		.867 ***	.751	NA
Item 7	NA		.879 ***	.772	NA
Item 8	NA		.790 ***	.624	NA
Item 9	NA		.889 ***	.791	NA
Item 10	NA		NA		.854 ***	.730
Item 11	NA		NA		.865 ***	.749
Item 12	NA		NA		.831 ***	.691
Item 13	NA		NA		.892 ***	.796
Item 14	NA		NA		.891 ***	.794
Item 15	NA		NA		.894 ***	.800
Item 16	NA		NA		.906 ***	.821
Item 17	NA		NA		.915 ***	.837
Item 18	NA		NA		.909 ***	.826
Item 19	NA		NA		.918 ***	.842
Item 20	NA		NA		.912 ***	.833
Item 21	NA		NA		.903 ***	.815
Item 22	NA		NA		.906 ***	.821

N = 79430. Bold values and

^***= p < .0005,

^**= .0005 < p < .001

^*= .001 ≤ p < .005. Model Fit: χ² (368) = 27336.245, p < .0005, RMSEA_{(.030−.031)}= .030, SRMR= .020, CFI = .997, TLI= .998. Standardized covariances between the dependent variables: User Experience Satisfaction ~~ Course Elements Satisfaction = .915, User Experience Satisfaction ~~ TF-CBT Components Satisfaction = .854, Course Elements Satisfaction ~~ TF-CBT Components Satisfaction = .887, Future Use of TF-CBT ~~ Future Use of EBTs = .34, Future Use of TF-CBT ~~ Working with Trauma Children/Families = .303, and Future Use of EBTs ~~ Working with Trauma Children/Families = .276.

Enrollment Type: 0 = Self-enrollment, 1 = Sponsored enrollment. COVID-19 Enroll: 0 = During/peri-COVID-19 pandemic, 1 = Before/pre-COVID-19 pandemic.

Coef. = Coefficient. 95% CI = 95% Confidence Interval.NA = Not applicable. TF-CBT = Trauma-focused Cognitive-Behavioral Therapy. EBT = Evidence-based treatments.

There were several significant predictors associated with the outcome variables of interest in the model at the p < .0005 level. Consistent with hypotheses, completers’ years of experience were negatively associated with all satisfaction latent variables and the Intent to Use EBTs outcome. Learners with more years of experience were less satisfied with multiple aspects of the course and had lower intentions of using EBTs with clients, compared to learners with fewer years of experience. However, the opposite was observed for confidence in working with children and families exposed to trauma. Learners with more years of experience reported higher confidence about working with traumatized children and their families compared to learners with fewer years of experience.

Total knowledge gain was positively associated with all three satisfaction latent variables. This suggests that learners who learned more during the course were more satisfied with all course elements, compared to learners who learned less. In contrast, total knowledge gain was negatively associated with Intent to Use TF-CBT and EBTs in general. Longer completion times were positively associated with User Experience Satisfaction, TF-CBT Components Satisfaction, and Intent to Use TF-CBT and EBTs in general, and Confidence in working with trauma-exposed children and their families. Enrollment type was negatively associated with all satisfaction latent variables, suggesting that learners who were sponsored (i.e., did not have to personally pay the enrollment fee) were somewhat less satisfied with the course. Further, enrollment type was also negatively associated with learner Confidence. COVID-19 completion timeframe was significantly negatively associated with all three satisfaction latent variables. Learners enrolling in the course prior to the COVID-19 pandemic tended to report lower satisfaction with the course. Moreover, COVID-19 completion timeframe was positively associated with judgments about the likelihood of using TF-CBT and confidence in working with children and their families exposed to trauma, with enrollees prior to the pandemic reporting higher intent to use TF-CBT and confidence in working with traumatized families. All three Satisfaction latent variables were positively associated with the Intent to Use and Confidence outcomes. Higher satisfaction was associated with higher self-rated likelihood of using TF-CBT, using EBTs more generally, and greater confidence working with trauma exposed children and families. Of note when comparing the size of the standardized coefficients, TF-CBT components satisfactions appeared to have the largest positive association with intent to use of TF-CBT, EBTs, and working with traumatized children and families.

Discussion

Web-courses have become an important part of the training on, as well as the dissemination and implementation of, EBTs. Initial evidence from both large-scale and small-scale programs has demonstrated the efficacy of such courses in helping learners understand concepts and skills (Jackson et al., 2018; Heck et al., 2015). However, much less research has focused on course satisfaction, learner’s reports of intent to use EBTs, and confidence on a large-scale. The current study sought to address these gaps in knowledge by examining learner satisfaction, intentions, and confidence associated with completing TFCBTWeb2.0, a web-course for learning TF-CBT (Cohen et al., 2017) within the context of a systems-contextual approach.

Overall, learners tended to report very favorable views of TFCBTWeb2.0, including its style, structure, and content. The proportion of learners who reported a favorable experience with the course (i.e., > 90%) appears consistent with other large-scale evaluations of online EBT trainings (e.g., Kasparik et al., 2022; Maguire et al., 2020). The current study’s findings build on the literature base by demonstrating that learner acceptability and satisfaction with web-course training can be fairly robust across a diverse set of learners. For TFCBTWeb2.0, all types of learners perceived benefit from the web-course, suggesting that learners from multiple mental health discipline and of varying experience levels can have high overall satisfaction with a web-course focused on introducing learners to the foundational principles and methods of a trauma-focused EBT. It is worth considering whether more specialized training geared to an individuals’ professional background or experience might be beneficial at other points in the training process.

While learner evaluations were generally positive, there were areas of relatively lower satisfaction. Notably, one of the items with the lowest satisfaction was the item assessing understanding how to structure and conduct a typical TF-CBT session. Although the overall mean value indicated that most learners clearly were satisfied with the coverage of this topic, additional clarification and specificity about how to administer TF-CBT sessions might be helpful for trainers and supervisors to consider when working with individuals using TF-CBT. Future modifications to the course could address this issue more thoroughly, but at present, trainers and TF-CBT supervisors can be mindful of this issue in their training activities. For example, this might include ensuring trainees feel comfortable or confident with knowing how to start, administer a technique, and then end a session, as well as incorporate other treatment needs within the session, such as administration of measures or long-term treatment planning.

Another relatively lower-rated item (but still in the agree-strongly agree range) pertained to the parenting and behavioral management strategies module of TFCBTWeb2.0. This finding is consistent with studies that have examined providers’ difficulties in implementing TF-CBT. For example, Ascienzo and colleagues (2020) examined reports from mental health providers about which components of TF-CBT were most difficult to implement. The authors found that the parenting skills component ranked second among TF-CBT components in terms of perceived difficulty, behind only trauma narrative creation. These findings have two important implications. First, the web-course could be modified to provide more thorough information about this component, or link to material that provides a more basic, foundational coverage of some of the parenting techniques suggested within TF-CBT. Second, TF-CBT trainers and clinical supervisors should be aware that they may need to provide some TFCBTWeb2.0 learners who complete the web-course with more in-depth support related to these concepts.

In contrast, the ramifications of the current study on uptake of information related to trauma narration and processing, consistently reported as the most difficult TF-CBT skill for practitioners to implement (Ascienzo et al, 2020, Hanson et al., 2014; Neelakantan et al., 2019), may be more complex. In the present study, satisfaction associated with the trauma narration components was among the highest rated aspects of TFCBTWeb2.0. This disconnect suggests some intriguing possibilities. First, the “difficulty” in using among TF-CBT learners reported in previous studies is possibly associated with the perceived difficulty of talking to trauma-exposed youth about the emotionally arousing and distressing events that they have experienced. It is possible that the step-by-step instructions explicating the process of narrative creation and processing, and the video demonstrations of expert therapists engaging in those techniques, helps learners see how the concepts are supposed to work. That potential demystification of the process might be quite helpful without specifically addressing a learner’s anxieties actually enacting the behavior. This is consistent with basic processes of improving self-efficacy and the relative value of observing versus enacting target behavior (Bandura, 1986). This, too, may inform both future iterations of web-courses, to the extent that technological advancements will permit the introduction of virtual or remote “practice activities” into online courses, and also current training and supervision in TF-CBT. Supervisors and training consultants can ensure that their trainees receive support around implementing what they have learned with respect to trauma narration and processing.

When considering factors at the individual level of the SC approach, data also revealed some differences in outcomes across various learner characteristics. Years of experience had a small association with lower satisfaction in the structural model. Many web-courses (including TFCBTWeb2.0) are specifically designed for early career individuals because their primary purpose is to introduce key concepts of a treatment to those who are likely unfamiliar with the treatment. Thus, one explanation for these findings may be that more experienced learners did not find the information as novel or helpful compared to learners with fewer years of experience, which lowered their satisfaction. This idea is bolstered by the findings related to knowledge gain, as those learners who demonstrated greater knowledge gain tended to report a higher satisfaction. Although views toward evidence-based practice were not assessed in the current study, another possible explanation for lower satisfaction scores among more experienced learners could be that these learners had less favorable views toward evidence-based practice compared to learners with fewer years of experience, as has been shown in previous literature (e.g., Aarons, 2004). TFCBTWeb2.0 emphasizes throughout the course that TF-CBT is a well-established trauma treatment and contains a section listing research that provides the empirical basis for those assertions. To the extent that learners find a focus on evidence off-putting or inconsistent with their philosophical approach to treatment, this could explain lower satisfaction with the course. In addition, some more experienced learners might have been hoping the course would move beyond introductory concepts and into more clinically complex scenarios. While there are sections focused on clinical challenges, the primary goal of the course is to introduce TF-CBT.

Several other notable patterns emerged within the structural model related to how engagement with the web-course might have influenced satisfaction and intention to use TF-CBT. Across multiple aspects of satisfaction, learners who took more time completing the course reported higher satisfaction, compared to learners who went more quickly through the course. These findings suggest that courses on EBTs could benefit from strategies that ensure learners move through the course at an appropriate pace. This could be accomplished in several ways, such as setting limits on learners’ progress through the course or publishing total time spent on the course on a completion certificate to dissuade such behavior. This might also include ensuring slow learners complete the web-course in an adequate time frame, such as sending email reminders. It is hypothesized that many of the faster learners may have taken the course rapidly in order to meet an external completion deadline, such as attending an in-person workshop or an academic coursework, continuing education, or employer-imposed deadline. It may also be the case that trainers and supervisors of individuals engaging in TFCBTWeb2.0 monitor or track progress to ensure these individuals are not completing the web-course in a manner that would not allow for proper retention or learning of the web-course material.

While completion time was positively associated with intent to use TF-CBT and EBTs more generally, knowledge gain was negatively associated with the same outcome variables. This finding appears counter-intuitive; it is logical to assume that greater knowledge of a treatment would make someone more likely to use it. However, this pattern may be consistent with some training approach studies that suggest that knowledge gain after in-person training is not consistently associated with skills use and treatment use (Herschell et al., 2010). Learners who experienced the most knowledge gain in TFCBTWeb2.0 may be those with the least amount of experience. For example, a graduate student completing TFCBTWeb2.0 as part of their intervention coursework may learn quite a lot about TF-CBT theories and techniques and still – appropriately – be quite anxious about the prospect of providing the treatment to real patients, or may not have opportunities to do so in their training. Indeed, a post-hoc analysis did demonstrate a small but significant negative correlation (r = −.11) between years of experience and knowledge gain. These findings speak to the importance of training models that include both didactic/observational and experiential components (e.g., Frank et al., 2020). Employers and supervisors who work with individuals learning TF-CBT must not assume that completion of TFCBTWeb2.0 is sufficient for implementation of TF-CBT. This also relates to not assuming knowing TF-CBT treatment skills will be sufficient in knowing how to work with children and their families who have experienced trauma or other stressful life experiences, as neither course related variables (e.g., knowledge gain) were related to confidence in working with children and families exposed to trauma. Trauma-informed care involves more than knowing the components of TF-CBT (e.g., Guevara et al., 2021), and learners who want to use TF-CBT should seek additional opportunities to practice their skills. However, the current knowledge base supports the current practice of using the web-course as an introductory training element that can be supplemented by more intensive workshop and/or experiential training and supervision/consultation (TF-CBT National Therapist Certification Program, 2022).

Related to contextual factors in the SC approach, self-enrollment was also associated with higher satisfaction outcomes and intent to work with traumatized children and families in the structural model. Self-enrollers may perceive the course as more valuable since they invested more personal resources than those who had their fee waived or paid by someone else. It may also be that some sponsored learners were less likely to have an inherent interest in the training, creating a slightly less positive view of the course compared to self-enrolled learners. It is important to note that even though sponsored learners reported lower satisfaction levels than self-enrollers, satisfaction rates were still quite high overall in both groups. Because organizational and supervisory support has been shown to be important for ensuring implementation and use of TF-CBT (e.g., Aarons et al., 2009; Lang et al., 2017), it does not seem warranted to discourage employers and institutions from sponsoring their staff, students, or other types of trainees for these types of trainings, especially if the treatment involved is likely to be applicable in their clinical setting.

Some small but significant differences also emerged based on the temporal relationship of registration to the onset of the COVID-19 pandemic. For 24 of the 25 items on the course evaluation, learners who enrolled during the COVID-19 pandemic reported more positive responses than learners who registered prior to the pandemic. This was also observed in the structural model analysis, such that enrollment in the course pre-pandemic was associated with slightly lower scores on all satisfaction-related variables. One possible explanation for this finding related to higher satisfaction with participation during the pandemic may be associated with the general increased reliance and acceptance of online training formats during the pandemic due to decreased access to in-person training options (e.g., Frye et al., 2022; Kern & Tague, 2022). TFCBTWeb2.0 may have benefited slightly from learners’ increasing familiarity and comfort with virtual learning methods as the pandemic progressed. In contrast to satisfaction, participating in the web-course during the COVID-19 pandemic was negatively associated with the intention to use TF-CBT and confidence working with children and families exposed to trauma. This observation could be grounded in the lack of information in TFCBTWeb2.0 on telehealth implementation, which became a prominent (and often necessary) treatment delivery method due to the pandemic. When TFCBTWeb2.0 was published in 2018, there was much less literature on delivering TF-CBT via telehealth or within other contextual factors related to the COVID-19 pandemic (e.g., Stewart et al., 2017). Thus, when the peri-pandemic learners were evaluating their intentions to use TF-CBT, the absence of telehealth-related instruction within the web-course may have influenced their responses. Since 2018, there has been much more literature published on the use of TF-CBT vie telehealth approaches and with respect to specific COVID-19 phenomenon (e.g., Gusler et al., 2023; Racine et al., 2020). Based on these findings, one recommendation for trainers and supervisors is to incorporate this literature base to ensure that learners have the ability to flexibility adapt TF-CBT in an empirically based manner to clients via telehealth or within the context of other COVID-19 related factors (e.g., death of a loved one due to the COVID-19 virus).

Limitations

The findings from the current study are tempered by several limitations. First, the course evaluation could be accessed only by learners who completed the course. Thus, satisfaction ratings do not represent learners who did not complete. This reporting bias may have contributed to the overall high scores. An additional limitation was the collection of only a small range of learner characteristics (e.g., minimal demographics related to professional training, limited selection of years of experience variable). While this strategy was an intentional design element of course registration that was intended to reduce registrant burden, it prevented a more fine-grained analysis of learner demographics, professional status, learner theoretical orientation, acceptance of EBTs, and other variables of potential interest. Another limitation is that the outcomes related to intention and confidence in using EBTs, TF-CBT, and working with traumatized families in the structural equation model were all assessed using a single item. Use of a more comprehensive measure that assesses these domains (e.g., Evidence-Based Practice Attitude Scale; Aarons, 2004) may have provided a more informative indicator of how participation in the web-course influenced these factors. Future work should explore this area in a planful manner. Finally, it is important to note that the findings presented here are specific to TFCBTWeb2.0 and therefore may not pertain to all treatment-focused web-courses. There is considerable variability in the way web-courses are organized, constructed, and presented to learners, both in terms of user experience (e.g., the “look and feel” of a web-course) and the pedagogical elements (e.g., the organization of information, depth of material, availability of video demonstrations, etc.). It is probably best not to consider all “web-based training courses” as a homogenous group, but to focus more specifically on the course elements and components included in a specific offering.

Conclusions

Despite these limitations, the current study’s findings help to further inform the field on dissemination and implementation of TF-CBT via a publicly available and widely-used web-course. Consistent with a systems-contextual model, evidence from this study demonstrates that individual, training, and contextual factors appear to contribute to satisfaction and intentions to use TF-CBT after completing TFCBTWeb2.0 across a diverse sample of professionals. The findings of this study suggest future refinements for research on TFCBTWeb2.0 (and any future iterations of the course), and promising directions for developers of other web-based training courses. Future research on web-courses designed to help educate learners on EBTs for children should ensure there is examination of these various factors in relation to satisfaction, intent to use the treatment, and confidence, rather than only examining entire sample trends or descriptives. More strategic data collection and analyses will help inform the development of future EBT web-courses, as well as determine what additional training strategies may be necessary through continued education and supervision to ensure youth and their families properly receive EBTs.