Key Points
Question
Can lay counselor–provided trauma-focused cognitive behavioral therapy improve mental health outcomes for children in Kenya and Tanzania who experienced parental death?
Findings
In this randomized clinical trial of 640 children, 12 weeks of trauma-focused cognitive behavioral therapy was more effective than usual care in improving posttraumatic stress in children in rural Kenya, urban Kenya, and urban Tanzania (but not rural Tanzania) after treatment. At the 12-month follow-up, differences were maintained only in rural and urban Kenya, where children experienced greater stress and adversity.
Meaning
Lay counselor–provided trauma-focused cognitive behavioral therapy improved outcomes for children who experienced parental death, but outcomes differed by context, with some children improving even without trauma-focused cognitive behavioral therapy by the 12-month follow-up.
Abstract
Importance
Approximately 140 million children worldwide have experienced the death of one or both parents. These children, mostly in low- and middle-income countries, have higher rates of mental health problems than those who have not experienced parental death. Cognitive behavioral therapy (CBT) may improve the well-being of these children, but to our knowledge there have been no randomized clinical trials specifically focused on this population.
Objectives
To test the effectiveness of trauma-focused CBT (TF-CBT) for improving posttraumatic stress (PTS) in children in Kenya and Tanzania who have experienced parental death, to test the effects of TF-CBT on other mental health symptoms, and to examine the feasibility of task-shifting with greater reliance on experienced, local lay counselors as trainers and supervisors.
Design, Setting, and Participants
A randomized clinical trial conducted in urban and rural areas of Tanzania and Kenya compared TF-CBT and usual care (UC) for 640 children aged 7 to 13 years who were recruited from February 13, 2013, to July 24, 2015. All children had experienced the death of one or both parents and had elevated PTS and/or prolonged grief. Interviewers were masked to study condition. Participants were followed up for 12 months after the randomized clinical trial. Statistical analysis was performed from February 3, 2017, to August 26, 2019. All analyses were on an intent-to-treat basis.
Interventions
In the intervention condition, 320 children received 12 weeks of group TF-CBT delivered by lay counselors who were supervised weekly. In the UC condition, 320 children received community services typically offered to this population.
Main Outcomes and Measures
The primary outcome was PTS, evaluated using a continuous, standardized measure. Other mental health symptoms and child-guardian relationship were also measured.
Results
A total of 640 children (320 girls and 320 boys; mean [SD] age, 10.6 [1.6] years) were included in the study. Trauma-focused CBT was more effective than UC for PTS in 3 of 4 sites after treatment (end of 3-month randomized clinical trial): rural Kenya (Cohen d = 1.04 [95% CI, 0.72-1.36]), urban Kenya (Cohen d = 0.56 [95% CI, 0.29-0.83]), and urban Tanzania (Cohen d = 0.45 [95% CI, 0.10-0.80]). At 12-month follow-up, TF-CBT remained more effective than UC in both rural (Cohen d = 0.86 [95% CI, 0.64-1.07]) and urban (Cohen d = 0.99 [95% CI, 0.75-1.23]) Kenya. At 12-month follow-up in Tanzania, children who received TF-CBT and UC had comparable rates of improvement (rural Tanzania, Cohen d = 0.09 [95% CI, −0.08 to 0.26]; urban Tanzania, Cohen d = 0.11 [95% CI, −0.09 to 0.31]). A similar pattern was seen for secondary outcomes, with stronger effects observed in Kenya, where children experienced greater stress and adversity (eg, more food scarcity, poorer guardian health, and greater exposure to traumatic events).
Conclusions and Relevance
This study found that TF-CBT was more effective than UC in reducing PTS among children who experienced parental death in 3 of 4 sites in Kenya and Tanzania. At 12-month follow-up, TF-CBT was more effective in reducing PTS only among children in rural and urban Kenya.
Trial Registration
ClinicalTrials.gov identifier: NCT01822366
This randomized clinical trial examines the effectiveness of trauma-focused cognitive behavioral therapy for improving posttraumatic stress in children in Kenya and Tanzania who have experienced parental death.
Introduction
Approximately 140 million children worldwide have experienced the death of one or both parents, most of whom live in low- and middle-income countries (LMICs).1 These children, considered single orphans (one parent died) or double orphans (both parents died) by the United Nations Children’s Fund’s definition,2 have higher rates of posttraumatic stress (PTS), prolonged grief, depression, anxiety, and behavior problems than nonorphaned children.3 Orphaned children are exposed to stressors associated with parental death, including loss of social support, disrupted education, unstable living situations,4,5 and orphan-related stigma.6,7,8
Treatment recommendations from the World Health Organization for addressing PTS and prolonged grief include cognitive behavioral therapy (CBT) with a focus on trauma.9 The World Health Organization recommends task shifting, in which lay counselors deliver therapy under supervision, given the substantial shortage of mental health professionals in LMICs.10 Some evidence for the effectiveness of trauma-focused CBT (TF-CBT) in African nations exists,11,12,13 but to our knowledge, randomized clinical trials (RCTs) have not examined its effectiveness when lay counselors both train and supervise other lay counselors, nor when TF-CBT is focused exclusively on parental death. In work preceding this trial, members of this research group conducted studies in Tanzania and Kenya to understand mental health problems,14 the cross-cultural feasibility and appropriateness of TF-CBT,15 and the lay counselors’ experiences with delivery of TF-CBT.16
This article describes posttreatment and 12-month follow-up outcomes from a large RCT of TF-CBT for orphaned children in Kenya and Tanzania. Tanzanian lay counselors from a previous study,14,16 rather than external experts, predominantly led the training and provided the supervision. To include a potential solution for areas in which expertise is not locally available, supervision of Kenyan counselors was conducted remotely by Tanzanian trainers. This trial was a collaboration with Ace Africa Kenya and the Tanzania Women Research Foundation.
Methods
Participants
The participants were 640 children who experienced parental death (320 children per country) and their guardians. Inclusion criteria were the death of one or both parents, the child being 7 to 13 years of age and residing in a family home (vs orphanage or group home), the child’s and guardian’s willingness to participate, ability to speak Kiswahili, and scoring above locally defined cutoffs for PTS (child-reported or guardian-reported score of ≥18 on the 17-item Child PTSD Symptom Scale [score range, 0-51, where 0 indicated no PTS symptoms and 51 indicated the highest possible reporting of PTS symptoms]17) and/or prolonged grief (child-reported score of ≥35 on the 28-item Inventory of Complicated Grief [score range, 0-112, where 0 indicated no prolonged grief symptoms and 112 indicated the highest possible reporting of prolonged grief symptoms]18). Cutoffs were set 1 SD above the mean from a local reliability and validity study. Children were excluded if their parent(s) died when they were 3 years or younger, if their parent(s) died within the previous 6 months, or if the children had substantial cognitive impairment. The trial protocol is in Supplement 1. Local and national institutional review boards, including the National Institute for Medical Research in Tanzania, the Kenya Medical Research Institute, and Duke University, approved the study. Guardians provided written informed consent and children provided written assent to participate. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline.
Setting
The study was conducted in the bordering East African nations of Tanzania and Kenya, which have distinct political and cultural histories but share a common trade language, allowing for cross-site counselor training and supervision.
Procedure
Recruitment and Consent
Recruitment occurred in 5 cohorts in each country from February 13, 2013, to July 24, 2015. Cohorts focused either on an urban or a rural area and on younger (7-10 years) or older (11-13 years) children. Potential participants were identified by local community members (eg, community health volunteers), leaders (eg, chiefs or pastors), orphan-serving organizations, and schools. The identifying community member made first contact with guardians.
Interviewers met separately with each child and guardian to obtain informed consent or assent and to administer the screening questions. Eligible children completed additional baseline questionnaires. Interviewers had paper slips with printed study identification numbers (masked to condition). Guardians selected 1 paper slip for themselves and their child. Interviewers provided the selected identification number and the guardian and child’s names to the study coordinator, who had lists of identification numbers randomized to condition. The coordinator relayed the assigned condition to guardians, keeping interviewers masked. Interviews continued until 64 children (32 in the intervention group and 32 in the UC group) were identified per cohort. Interviewers collected data at baseline (before treatment), after treatment (end of the 3-month RCT), and at 6- and 12-month follow-ups after study completion. The windows for data collection were 2 weeks for baseline and after treatment and 2 to 3 weeks for follow-ups. We focus on findings after treatment and at the 12-month follow-up in this article.
Interviewers
Interviewers had experience with children (eg, served as interviewers on other child-focused studies) and at least a bachelor’s degree. They received 1 week of training focused on question-by-question analysis of cultural appropriateness and translation quality (modifying for Kiswahili dialects), practicing interviews, finalizing crisis response protocols, and identifying and responding to vicarious traumatization. After training, interviewers practiced with feedback until they were certified by coordinators, who had extensive experience training and supervising interviewers.15,19 Interviewers periodically shadowed each other and were observed by the in-country coordinator, who maintained weekly communication with the US team. Interviewers were retained throughout the duration of the study.
Safety Protocol
Coordinators oversaw development and implementation of a safety protocol,20 including steps for assessing risk and referral options for formal (eg, health care professionals) and informal resources (eg, church elders).
Intervention
Trauma-focused CBT is an evidence-based treatment for sequelae from trauma exposure, with grief-specific elements.21 Empirical support comes from RCTs of TF-CBT conducted in high-income countries,22 with growing evidence from RCTs of TF-CBT conducted in LMICs.11 Trauma-focused CBT was pilot tested in Tanzania with promising feasibility and initial clinical outcomes.15 Called Pamoja Tunaweza (Together We Can; S.D., K.O., L.K.M., J.A.C., unpublished data, 2012) locally, all TF-CBT elements were included. Treatment focused specifically on parental death. Cultural modifications were informed by focus groups before the pilot and lay counselor input. These modifications included group (vs individual) modality; the use of local metaphors to explain concepts; simplified, step-by-step guides for sessions; and language (delivering TF-CBT in Kiswahili, referring to the sessions as “class” vs therapy, and no psychological labels).
Trauma-focused CBT included 12 group sessions and 3 to 4 individual sessions for 12 consecutive weeks.15,23 Child groups and guardian groups met concurrently, with joint child-guardian activities in the final 5 sessions. Child group sessions were led by 2 counselors, and guardian group sessions were led by 1 counselor, with counselors randomly assigned to groups to allow examination of counselor-level effects. Group sessions 1 to 4 focused on psychoeducation and coping strategies. Guardians learned the same information to support children and apply strategies in their own lives. They also learned behavior management skills. In individual sessions delivered during weeks 4 to 7, counselors met with each child to conduct imaginal exposure, called the “trauma narrative” in TF-CBT, and to begin processing trauma-related cognitions. One counselor met individually with each child to talk about the death of their parent(s) and surrounding events. The counsler then met with the child’s guardian to review the trauma narrative and build emotional support for the child. Group sessions 5 to 8 involved developing plans for situational exposure (ie, trauma reminders), children reviewing their own trauma narratives individually, continued processing of trauma-related cognitions, and sharing the trauma narrative individually with their guardians. Group sessions 9 to 12 focused on grief-specific elements (eg, resolving ambivalent feelings and redefining the relationship).
Usual care included services to which children had access that could affect mental health (predominantly educational support, as well as some mental health care).
TF-CBT Training and Supervision
The Tanzania Women Research Foundation and Ace Africa Kenya each hired 6 counselors. All counselors received 2 weeks of in-person training following procedures outlined in the apprenticeship model24 (eg, demonstration and participant practice with trainer feedback). Training was led by 3 Tanzanian lay counselors with experience from the pilot study,15 who became supervisors and had received 4.5 days of in-person, trainer-focused training. One of us (S.D.; an expert in TF-CBT) took a secondary role (eg, observing and consulting at the beginning and end of each training day).
Booster training sessions were held in 2014 (Tanzania only) and 2015. During the course of the study, 4 Tanzanian counselors left the Tanzania Women Research Foundation; replacement counselors were trained through these booster sessions. The replacement counselors first participated as the second counselor in child group sessions, enabling them to gain experience before serving as lead counselors for child or guardian groups. Tanzanian supervisors provided weekly supervision in person for Tanzanian counselors and by Skype audio or text message (when internet connectivity was poor) for Kenyan counselors. Counselors role-played and practiced with supervisor feedback. Annually, Kenyan counselors received 1 day of in-person supervision. Supervisors received weekly hour-long supervision from one of us (S.D.), who reviewed counselor reports in advance (30-45 minutes; intentional low-dose expert involvement).
Measures
We selected standardized measures based on a qualitative study in the region14 as well as work in both countries.15,19 Measures were translated and back-translated for accuracy. We conducted a reliability and validity study in Kenya (100 children and 1 guardian per child) examining internal, interrater, and test-retest reliability and discriminant validity for our primary outcome (ie, PTS), prolonged grief, and a functioning measure developed locally, following established procedures.25 Indicators of reliability were acceptable for PTS and grief but not for the functioning measure. Discriminant validity was not established. As in other studies using the same validation approach,25 the lack of discriminant validity likely was owing to testing methods using agreement between 2 raters26,27 (child-guardian) on the existence of mental health problems after parental death as the standard when a criterion standard (eg, diagnostic interview) is unavailable.
Child and guardian characteristics were assessed at baseline using demographic characteristics and health survey questions widely used in LMICs. Additional questions came from a 5-country longitudinal orphan study (including Tanzania and Kenya).19 Subjective health was measured using 1 question from the 36-question Medical Outcomes Short Form.28 The Life Events Checklist8,29,30 assessed traumatic event exposure.
Posttraumatic stress (avoidance, reexperiencing, and hyperarousal) in relation to parental death was assessed via the 17-item Child PTSD Symptom Scale17 using both child (primary outcome) and guardian report. Response options for each item were 0 to 3 (score range, 0-51, where 0 indicated no PTS symptoms and 51 indicated the highest possible reporting of PTS symptoms). The 17-item Child PTSD Symptom Scale had high internal consistency (α = .80-.85) and reasonable combined test-retest and interrater reliability (r = 0.43-0.62).
Children reported on prolonged grief using the 28-item Inventory of Complicated Grief.18 Response options for each item were 0 to 4 (score range, 0-112, where 0 indicated no prolonged grief symptoms and 112 indicated the highest possible reporting of prolonged grief symptoms). Sample item content includes feeling lonely since the death, missing the deceased, and having a hard time believing the death is real. Melhem and colleagues18 found that the 28-item Inventory of Complicated Grief significantly differentiated prolonged grief from PTS and depression in children who experienced parental death. In our reliability study, the 28-item Inventory of Complicated Grief demonstrated high internal consistency (α = .93) and acceptable test-retest and interrater reliability (r = 0.69). Behavioral difficulties were assessed using the broadband externalizing scale, and depressive and internalizing symptoms were assessed using the broadband internalizing scale from the Child Behavior Checklist31 (guardian report) and Youth Self-Report32 (completed by children ≥11 years), supplemented by items assessing unique local symptoms from a qualitative study.14 The Child Behavior Checklist and Youth Self-Report scales demonstrated high internal consistency at baseline (α = .73-.89). The child-guardian relationship was assessed using the closeness and conflict scales from the 15-item Child-Parent Relationship Scale (guardian report).33 Both scales demonstrated acceptable internal consistency at baseline (α = .62-.65).
Counselors’ TF-CBT knowledge was assessed using a 14-item knowledge test after training (total potential points for the 14 items were 29, scored 0%-100% correct, where 0% indicated the lowest TF-CBT knowledge level and 100% indicated the highest TF-CBT knowledge level). Fidelity was assessed through review of counselor reports, session audio recordings, and discussion. Supervisors rated both adherence (following the treatment manual) and competence (skillful delivery) on a scale of 1 to 6 (where 1 indicated the lowest skill level and 6 indicated the highest skill level).
Statistical Analysis
Statistical analysis was performed from February 3, 2017, to August 26, 2019. We estimated power (1 − β) to detect differences as small as Cohen d = 0.35 between intervention and UC at 0.80 (α = .05) with intraclass correlation coefficients as large as 0.15. All analyses were on an intent-to-treat basis, with all randomized participants included in the analysis and analyzed according to original group assignment regardless of treatment dose received.34 There were limited missing data after treatment (6 of 640 [0.9%]) and at 12 months (16 of 640 [2.5%]) and no differences between participants with or without missing data. Owing to limited missingness, we did not impute missing values. We used partially nested multilevel models to predict symptoms at both time points from condition (TF-CBT vs UC) while controlling for baseline symptom levels and covariates (child sex, age, area [urban or rural], and country). We tested all covariate-by-condition interactions35 with an a priori threshold for retention set at P < .01 because not including interactions can bias coefficient estimates,36 but we refrained from interpreting exploratory moderators. Other potential effect modifiers not included as covariates were tested following our retention rule. A few exhibited main effects; however, inclusion did not alter treatment effects or country-by-area interactions. To measure clinical effect magnitude, we calculated effect size estimates using Cohen d.37 All P values were from 2-sided tests, and the results for primary outcomes were deemed statistically significant at P < .05 except where noted (ie, covariate-by-condition interactions were set at P < .01 and secondary outcomes were Bonferroni adjusted).
Children in the group-modality TF-CBT intervention condition were nested within groups; children in the UC condition were not. Following best practices to model partial nesting,38 we estimated a fixed intercept and a random slope for intervention. Because the intercept estimates the symptom level in the UC condition (condition = 0), a random intercept is not necessary because UC children were not nested.38 Estimating a random slope for effects of condition allows for the accounting of clustering effects within groups by estimating group-level differences in treatment effects. We also tested whether allowing individual residuals to vary across conditions improved model fit using deviance testing. We report effects after treatment and at the 12-month follow-up. To reduce the risk of false discovery, we report Bonferroni-adjusted P values for secondary outcomes.39
Results
Baseline Characteristics
Of 861 children screened, 643 were eligible, and 640 were enrolled and randomized to receive TF-CBT or UC (Figure 1). Most children were single orphans; approximately half lived with a biological parent. Table 1 illustrates differences between countries and areas (urban or rural). Kenyan children had poorer health (χ24 = 107.96 [n = 640]; P < .001), greater food scarcity (χ22 = 290.93 [n = 637]; P < .001), poorer guardian health (χ24 = 100.37 [n = 640]; P < .001), more missed school (χ21 = 120.74 [n = 634]; P < .001), more children in the home (t635 = 7.55; P < .001), and nearly double the exposure to traumatic events (t637 = 10.33; P < .001). Rural Kenyan children reported the highest level of food scarcity (χ22 = 23.11 [n = 320]; P < .001), and urban Kenyan children reported the highest level of exposure to traumatic events (t318 = 6.08; P < .001).
Table 1. Demographic and Baseline Characteristics of Children and Guardians by Country and Area (Rural and Urban).
Characteristic | Kenya, No. (%) | Tanzania, No. (%) | ||
---|---|---|---|---|
Urban (n = 192) | Rural (n = 128) | Urban (n = 192) | Rural (n = 128) | |
Child | ||||
Female sex | 96 (50.0) | 64 (50.0) | 96 (50.0) | 64 (50.0) |
Age, mean (SD), y | 10.7 (1.6) | 10.2 (1.7) | 10.9 (1.6) | 10.5 (1.8) |
One parent died | 155 (80.7) | 103 (80.5) | 145 (75.5) | 109 (85.2) |
Both parents died | 23 (12.0) | 15 (11.7) | 28 (14.6) | 10 (7.8) |
Biological mother died | 56 (29.2) | 41 (32.0) | 74 (38.5) | 42 (32.8) |
Biological father died | 159 (82.8) | 102 (79.7) | 145 (75.5) | 94 (73.4) |
Traumatic events experienced, mean (SD) [range], No.a | 5.5 (2.7) [0-13] | 4.5 (2.5) [1-12] | 3.0 (2.4) [0-11] | 3.3 (2.5) [0-12] |
School attendance | ||||
Currently in school | 188 (97.9) | 128 (100) | 186 (96.9) | 128 (100) |
Missed any school in past 2 wk | 111 (57.8) | 72 (56.3) | 30 (15.6) | 19 (14.8) |
Gone to bed hungry last 14 d | ||||
None of the time | 63 (32.8) | 15 (11.7) | 171 (89.1) | 118 (92.2) |
Some or all of the time | 129 (67.2) | 113 (88.3) | 21 (10.9) | 7 (5.5) |
Child’s health | ||||
Very good or good | 115 (59.9) | 69 (53.9) | 177 (92.2) | 120 (93.8) |
Fair or poor | 76 (39.6) | 59 (46.1) | 15 (7.8) | 8 (6.3) |
Guardian | ||||
Age, mean (SD) [range], y | 44.5 (12.7) [19-83] | 42.3 (10.1) [23-72] | 43.2 (12.7) [19-93] | 49.7 (14.8) [20-90] |
Marital status | ||||
Married | 62 (32.3) | 55 (43.0) | 51 (26.6) | 34 (26.6) |
Widowed | 122 (63.5) | 73 (57.0) | 112 (58.3) | 79 (61.7) |
Female sex | 172 (89.6) | 111 (86.7) | 176 (91.7) | 115 (89.8) |
Relationship to child | ||||
Biological parent | 104 (54.2) | 63 (49.2) | 88 (45.8) | 48 (37.5) |
Grandparent | 44 (22.9) | 21 (16.4) | 49 (25.5) | 43 (33.6) |
Aunt or uncle | 28 (14.6) | 33 (25.8) | 26 (13.5) | 13 (10.2) |
Otherb | 16 (8.3) | 11 (8.6) | 27 (14.1) | 24 (18.8) |
Health | ||||
Very good or good | 90 (46.9) | 69 (53.9) | 166 (86.5) | 109 (85.2) |
Fair or poor | 100 (52.1) | 56 (43.7) | 26 (13.5) | 16 (12.5) |
Children in the household, mean (SD), No. | 4.23 (2.14) | 4.39 (2.26) | 2.83 (1.39) | 2.81 (1.60) |
Traumatic event exposure by child report.
Other guardian relationship to child includes sibling, stepparent, adoptive parent, other relative, and nonrelative.
Treatment Outcomes
Children receiving the TF-CBT intervention improved more than those receiving UC. The random effect of treatment accounts for differences in treatment effects across counselors, which were very close to 0 (intraclass correlation coefficients 0.000 and 0.011-0.07), suggesting few to no differences in treatment outcomes by TF-CBT group. Children receiving UC (140 of 320) mostly received education-related services (tuition or school fees and uniforms; 114 of 140 were Kenyan). Only 29 children in the UC group received mental health services (28 of 29 were Kenyan). At baseline, child-reported PTS and grief were moderately correlated (r = 0.62). Child-reported and guardian-reported PTS were weakly correlated (r = 0.26).
After treatment, there were consistent differences in treatment effects across area and country for PTS and nearly all secondary outcomes. A 3-way interaction among condition, area, and country significantly improved model fit (b = –7.72 [95% CI, –12.37 to –3.08]; P = .002) (Table 2) and produced normal model residuals. At 12-month follow-up, we observed differences by condition and country (b = –5.92 [95% CI, –7.96 to –3.89]; P < .001) but not area; the 3-way interaction was no longer significant. Because of these interactions, we describe change at both time points within each condition, area, and country (Table 3), although at 12 months only differences by country were significant. Age significantly interacted with PTS after treatment; no interactions beyond condition-by-country were significant at 12 months. We report adjusted P values for highest-order interactions for secondary outcomes because those P values represent multiple tests of the hypothesis (all simple main effects represented within model).40
Table 2. Treatment Effects on Child-Reported Posttraumatic Stress After Treatment and at 12-Month Follow-up.
Variable | b (95% CI)a | |
---|---|---|
After Treatment | At 12 mo | |
Condition | 3.81 (1.99 to 5.63)b | 6.70 (5.06 to 8.34)b |
Baseline score | 0.27 (0.15 to 0.39)b | 0.15 (0.08 to 0.22)b |
Country | –1.68 (–3.64 to 0.29) | –1.93 (–3.30 to –0.56)c |
Sex | –0.45 (–1.55 to 0.66) | 0.71 (–0.26 to 1.67) |
Age | 0.14 (–0.35 to 0.63) | –0.25 (–0.55 to 0.06) |
Area | 1.20 (–1.13 to 3.54) | –0.21 (–1.23 to 0.82) |
Condition × country | –1.13 (–3.98 to 1.71) | –5.92 (–7.96 to –3.89)b |
Condition × age | –0.92 (–1.61 to –0.22)d | NA |
Condition × baseline score | –0.08 (–0.24 to 0.09) | NA |
Condition × area | 4.34 (0.97 to 7.72)d | NA |
Country × area | –2.33 (–5.55 to 0.89) | NA |
Condition × country × area | –7.72 (–12.37 to –3.08)c | NA |
Constant | 12.74 (11.37 to 14.11)b | 9.39 (8.11 to 10.68)b |
Observations, No. | 631e | 624e |
Abbreviation: NA, not applicable.
The posttreatment assessment was conducted at treatment completion, which was 3 months from baseline for both conditions. The 12-month follow-up assessment was conducted 1 year after treatment (completion of the 3-month RCT), which was 15 months from baseline for both conditions.
P < .001.
P < .01.
P < .05.
Four participants were missing data on age at baseline and were excluded from the final models (because age was included as a covariate).
Table 3. Posttreatment and 12-Month Outcomes for Child-Reported Posttraumatic Stress by Study Condition, Country, and Areaa.
Site | Children, No. | Child PTSD Symptom Scale Score, Mean (SD)b | Effect Sizes (95% CI) | ||||||
---|---|---|---|---|---|---|---|---|---|
TF-CBT Group | Usual Care Group | ||||||||
Baseline | After Treatment | At 12 mo | Baseline | After Treatment | At 12 mo | After Treatment | At 12 mo | ||
Kenya | |||||||||
Urban | 192 | 24.70 (6.32) | 12.82 (6.49) | 10.33 (6.55) | 22.03 (6.99) | 15.92 (6.73) | 16.27 (9.08) | 0.56 (0.29 to 0.83)c | 0.99 (0.75 to 1.23)c |
Rural | 128 | 24.86 (7.97) | 14.00 (5.77) | 9.30 (5.78) | 24.61 (7.63) | 22.44 (9.69) | 16.38 (8.79) | 1.04 (0.72 to 1.36)c | 0.86 (0.64 to 1.07)c |
Tanzania | |||||||||
Urban | 192 | 23.40 (5.71) | 10.82 (7.78) | 7.39 (5.04) | 21.23 (6.06) | 12.78 (7.88) | 8.24 (5.23) | 0.45 (0.10 to 0.80)d | 0.11 (−0.09 to 0.31) |
Rural | 128 | 25.66 (6.79) | 10.17 (7.78) | 8.53 (6.23) | 23.41 (7.35) | 8.98 (6.41) | 8.15 (5.10) | –0.12 (−0.47 to 0.24) | 0.09 (−0.08 to 0.26) |
Abbreviation: TF-CBT, trauma-focused cognitive behavioral therapy.
There were 3 urban cohorts and 2 rural cohorts (thus differences in sample sizes). Outcome means are reported based on raw data, while effect sizes are model-estimated effect sizes for each outcome by area and country. The posttreatment effect size represents the difference in mean change between the TF-CBT and usual care conditions from baseline to after treatment. The 12-month effect size represents the difference in mean change between the TF-CBT and usual care conditions from baseline to the 12-month follow-up. Effect sizes are reported by country, area, and condition, even though at 12 months only the condition × country interaction is significant.
Range from 0 to 51, where 0 indicated no posttraumatic stress symptoms and 51 indicated the highest possible reporting of posttraumatic stress symptoms.
P < .001.
P < .05.
Table 3 illustrates estimated effect sizes for PTS by country and area after treatment and at the 12-month follow-up. Results indicated that, compared with UC, TF-CBT was more effective both after treatment and at 12 months in improving PTS in children in urban Kenya (Cohen d = 0.56 [95% CI, 0.29-0.83] after treatment; Cohen d = 0.99 [95% CI, 0.75-1.23] at 12 months) and in rural Kenya (Cohen d = 1.04 [95% CI, 0.72-1.36] after treatment; Cohen d = 0.86 [95% CI, 0.64-1.07] at 12 months) (Table 3; Figure 2). Treatment effects were observed only after treatment in children in urban Tanzania (Cohen d = 0.45 [95% CI, 0.10-0.80] after treatment; Cohen d = 0.11 [95% CI, −0.09 to 0.31] at 12 months), with no observed effects in rural Tanzania (Cohen d = –0.12 [95% CI, −0.47 to 0.24] after treatment; Cohen d = 0.09 [95% CI, −0.08 to 0.26] at 12 months).
Secondary Treatment Outcomes
Guardian-reported PTS and prolonged grief followed a similar pattern (eTables 1, 2, and 3 in Supplement 2): TF-CBT was significantly more effective after treatment and at 12 months in improving guardian-reported PTS in urban Kenya (Cohen d = 0.71 after treatment; Cohen d = 1.15 at 12 months) and rural Kenya (Cohen d = 1.47 after treatment; Cohen d = 0.98 at 12 months). There were small effects after treatment and at 12 months in urban Tanzania (Cohen d = 0.32 after treatment; Cohen d = 0.15 at 12 months) and rural Tanzania (Cohen d = 0.24 after treatment; Cohen d = 0.16 at 12 months). Trauma-focused CBT was significantly more effective in improving prolonged grief at both time points in urban Kenya (Cohen d = 0.36 after treatment; Cohen d = 0.75 at 12 months) and rural Kenya (Cohen d = 0.81 after treatment; Cohen d = 0.60 at 12 months) and only after treatment in urban Tanzania (Cohen d = 0.34 after treatment; Cohen d = 0.15 at 12 months). There were no observed effects in rural Tanzania at either time point (Cohen d = –0.07 after treatment; Cohen d = 0.12 at 12 months).
Levels of behavior problems were low at baseline; thus, we are not reporting effects (eTables 4, 5, and 6 in Supplement 2). Outcomes were similar for Child Behavior Checklist and Youth Self-Report standardized scales analyzed with and without added local items; analyses without local items are reported. Treatment effects for depressive and internalizing symptoms generally followed the same pattern of PTS and grief (eTables 7, 8, and 9 in Supplement 2). Treatment effects on the child-guardian relationship—closeness and conflict—varied, with greater effects on closeness (vs conflict) in Kenya and at 12 months (urban Kenya, Cohen d = 0.27 after treatment; Cohen d = 0.72 at 12 months; rural Kenya, Cohen d = 0.46 after treatment; Cohen d = 0.54 at 12 months) (eTables 10, 11, and 12 in Supplement 2).
Implementation Outcomes
Counselors demonstrated a high level of knowledge of TF-CBT after training (mean percentage correct: Kenya, 86.6%; Tanzania, 85.2%). The mean (SD) supervisor ratings of counselors’ TF-CBT adherence (Kenya, 5.48 [0.56]; Tanzania, 5.36 [0.53]) and competence (Kenya, 5.39 [0.57]; Tanzania, 5.32 [0.60]) were high across groups and throughout the duration of the study.
Discussion
This study demonstrates that TF-CBT can be effective in improving PTS in children who experienced parental death but that outcomes may differ by context. After treatment, TF-CBT was more effective than UC in urban Kenya, rural Kenya, and urban Tanzania but not in rural Tanzania. At 12-month follow-up, differences by condition were observable only in urban and rural Kenya. Secondary outcomes generally followed a similar pattern, with TF-CBT having the strongest effects after treatment for rural Kenyan children, and effects at 12 months mostly limited to urban and rural Kenyan children.
In contrast to children receiving UC in Kenya, children receiving UC in Tanzania experienced symptom reduction at 12-month follow-up similar to that seen in children who received TF-CBT. On average in Tanzania, children in both the TF-CBT and UC conditions experienced a substantial reduction in PTS scores at 12 months (60%-70% reductions from baseline), with PTS scores well below the locally defined cutoff. We have a separate qualitative analysis underway to explore potential explanations for improvement within the UC condition in Tanzania. Comparability in symptom reduction across the UC and TF-CBT conditions in Tanzania does not seem to be explained by greater receipt of service among Tanzanian children in the UC condition. It also does not seem to be explained by differences in counselors’ TF-CBT fidelity. Even with counselor attrition and replacement in Tanzania, supervisors’ ratings of TF-CBT fidelity remained high, and there did not seem to be treatment effects by counselor or specific TF-CBT group.
One potential explanation is that participants in Kenya and Tanzania had different experiences of stress and adversity. Among other differences, Kenyan children and guardians had poorer health, more food scarcity, and higher exposure to traumatic events. These differences reflect national statistics41 and may have contributed to symptom maintenance among Kenyan children in the UC condition. Improvement among Tanzanian children in the UC condition, with relatively lower stress and adversity, may reflect findings from a longitudinal study of grief in US children who experienced parental death, some of whom naturally recovered over time.42 Looking to the TF-CBT evidence base for potential explanations, RCTs including 12-month follow-up demonstrated maintained effects of TF-CBT,43,44,45 although all those RCTs were conducted in high-income countries and most focused on children who had been sexually abused.
This study advances the science of task-shifted mental health interventions in a few ways. We included a 12-month follow-up. With few exceptions,46 most global mental health studies have included only posttreatment or 6-month follow-ups. We saw notable differences at 12 months compared with after treatment, including symptom reductions in children in the UC condition in both urban and rural Tanzania and seemingly larger effects in urban Kenya. Finally, we intentionally included less expert involvement in training and supervision (supervision, 1.5-2 hours per week vs 6-8 hours per week) and greater reliance on local lay counselors as trainers and supervisors. They developed expertise through their own TF-CBT delivery, and with training and support, they served as lay counselor trainers and supervisors in this study.
The positive outcomes in Kenya were obtained by counselors who received remote audio or text (vs in-person) supervision. Research in high-resource settings47,48,49 and growing evidence in low-resource settings50 highlights that treatment manuals and training alone are insufficient. Posttraining supervision is needed. However, the scale-up of such supervision is limited by the cost and inconvenience of face-to-face supervision.51 Our findings of treatment effectiveness in Kenya with remote supervision provide some additional empirical support for potentially lower-cost supervision approaches in low-resource settings.
Limitations
The study has important limitations. It was a single-blind study, with interviewers masked to condition. However, participants were aware of assignment and could have disclosed information or had biased expectations from assignment. Local supervisors were experienced former counselors with TF-CBT expertise, which would not be available everywhere. We cannot confidently determine why children in the UC condition in Tanzania experienced improvement. As can be common in mental health service delivery in high-income countries,52 we experienced counselor turnover in Tanzania. Many children who experienced parental death live in group homes and orphanages; our findings likely could be replicated in these settings, as long as settings were “family-like” environments, but we cannot be certain. Finally, a small transportation reimbursement was provided that may have accounted for high rates of attendance and may present a barrier to sustainment.
Conclusions
To our knowledge, this is the largest TF-CBT RCT conducted to date and the first study of TF-CBT in LMICs examining 12-month follow-up effects. After treatment, TF-CBT was effective in 3 of 4 sites in Kenya and Tanzania. At the 12-month follow-up, TF-CBT was more effective only in the 2 Kenyan sites. Children experiencing greater stress and adversity seemed to benefit more from TF-CBT. Positive outcomes in Kenya are notable, given greater reliance on lay counselor training and remote supervision. Our findings suggest a need for treatment studies with longer follow-up periods and inclusion of children in varying contexts to understand which children most need treatment and are unlikely to improve without intervention.
References
- 1.UNICEF The state of the world’s children 2016: a fair chance for every child. https://www.unicef.org/publications/files/UNICEF_SOWC_2016.pdf. Published June 2016. Accessed January 7, 2019.
- 2.UNICEF Orphans. https://www.unicef.org/media/media_45279.html. Updated June 16, 2017. Accessed June 12, 2019.
- 3.Atwine B, Cantor-Graae E, Bajunirwe F. Psychological distress among AIDS orphans in rural Uganda. Soc Sci Med. 2005;61(3):555-564. doi: 10.1016/j.socscimed.2004.12.018 [DOI] [PubMed] [Google Scholar]
- 4.Foster G, Makufa C, Drew R, Mashumba S, Kambeu S. Perceptions of children and community members concerning the circumstances of orphans in rural Zimbabwe. AIDS Care. 1997;9(4):391-405. doi: 10.1080/713613166 [DOI] [PubMed] [Google Scholar]
- 5.Urassa M, Boerma JT, Ng’weshemi JZL, Isingo R, Schapink D, Kumogola Y. . Orphanhood, child fostering and the AIDS epidemic in rural Tanzania: project to support AIDS control in Mwanza Region. Health Transit Rev. 1997;7(suppl 2):141-153. [Google Scholar]
- 6.Messer LC, Pence BW, Whetten K, et al. Prevalence and predictors of HIV-related stigma among institutional- and community-based caregivers of orphans and vulnerable children living in five less-wealthy countries. BMC Public Health. 2010;10:504. doi: 10.1186/1471-2458-10-504 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Murray LK, Haworth A, Semrau K, et al. Violence and abuse among HIV-infected women and their children in Zambia: a qualitative study. J Nerv Ment Dis. 2006;194(8):610-615. doi: 10.1097/01.nmd.0000230662.01953.bc [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Whetten K, Ostermann J, Whetten R, O’Donnell K, Thielman N; Positive Outcomes for Orphans Research Team . More than the loss of a parent: potentially traumatic events among orphaned and abandoned children. J Trauma Stress. 2011;24(2):174-182. doi: 10.1002/jts.20625 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Tol WA, Barbui C, van Ommeren M. Management of acute stress, PTSD, and bereavement: WHO recommendations. JAMA. 2013;310(5):477-478. doi: 10.1001/jama.2013.166723 [DOI] [PubMed] [Google Scholar]
- 10.World Health Organization Task Shifting: Global Recommendations and Guidelines. Geneva, Switzerland: World Health Organization; 2008. [Google Scholar]
- 11.Murray LK, Skavenski S, Kane JC, et al. Effectiveness of trauma-focused cognitive behavioral therapy among trauma-affected children in Lusaka, Zambia: a randomized clinical trial. JAMA Pediatr. 2015;169(8):761-769. doi: 10.1001/jamapediatrics.2015.0580 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.O’Callaghan P, McMullen J, Shannon C, Rafferty H, Black A. A randomized controlled trial of trauma-focused cognitive behavioral therapy for sexually exploited, war-affected Congolese girls. J Am Acad Child Adolesc Psychiatry. 2013;52(4):359-369. doi: 10.1016/j.jaac.2013.01.013 [DOI] [PubMed] [Google Scholar]
- 13.McMullen J, O’Callaghan P, Shannon C, Black A, Eakin J. Group trauma-focused cognitive-behavioural therapy with former child soldiers and other war-affected boys in the DR Congo: a randomised controlled trial. J Child Psychol Psychiatry. 2013;54(11):1231-1241. doi: 10.1111/jcpp.12094 [DOI] [PubMed] [Google Scholar]
- 14.Dorsey S, Lucid L, Murray L, et al. A qualitative study of mental health problems among orphaned children and adolescents in Tanzania. J Nerv Ment Dis. 2015;203(11):864-870. doi: 10.1097/NMD.0000000000000388 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.O’Donnell K, Dorsey S, Gong W, et al. Treating maladaptive grief and posttraumatic stress symptoms in orphaned children in Tanzania: group-based trauma-focused cognitive-behavioral therapy. J Trauma Stress. 2014;27(6):664-671. doi: 10.1002/jts.21970 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Woods-Jaeger BA, Kava CM, Akiba CF, Lucid L, Dorsey S. The art and skill of delivering culturally responsive trauma-focused cognitive behavioral therapy in Tanzania and Kenya. Psychol Trauma. 2017;9(2):230-238. doi: 10.1037/tra0000170 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Foa EB, Johnson KM, Feeny NC, Treadwell KR. The Child PTSD Symptom Scale: a preliminary examination of its psychometric properties. J Clin Child Psychol. 2001;30(3):376-384. doi: 10.1207/S15374424JCCP3003_9 [DOI] [PubMed] [Google Scholar]
- 18.Melhem NM, Porta G, Walker Payne M, Brent DA. Identifying prolonged grief reactions in children: dimensional and diagnostic approaches. J Am Acad Child Adolesc Psychiatry. 2013;52(6):599-607.e7. doi: 10.1016/j.jaac.2013.02.015 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Whetten K, Ostermann J, Whetten RA, et al. ; Positive Outcomes for Orphans (POFO) Research Team . A comparison of the wellbeing of orphans and abandoned children ages 6-12 in institutional and community-based care settings in 5 less wealthy nations. PLoS One. 2009;4(12):e8169. doi: 10.1371/journal.pone.0008169 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Murray LK, Skavenski S, Bass J, et al. Implementing evidence-based mental health care in low-resource settings: a focus on safety planning procedures. J Cogn Psychother. 2014;28(3):168-185. doi: 10.1891/0889-8391.28.3.168 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Cohen JA, Mannarino AP, Deblinger E. Treating Trauma and Traumatic Grief in Children and Adolescents. New York, NY: Guilford Press; 2006. [Google Scholar]
- 22.Dorsey S, McLaughlin KA, Kerns SEU, et al. Evidence base update for psychosocial treatments for children and adolescents exposed to traumatic events. J Clin Child Adolesc Psychol. 2017;46(3):303-330. doi: 10.1080/15374416.2016.1220309 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Deblinger E, Pollio E, Dorsey S. Applying trauma-focused cognitive-behavioral therapy in group format. Child Maltreat. 2016;21(1):59-73. doi: 10.1177/1077559515620668 [DOI] [PubMed] [Google Scholar]
- 24.Murray LK, Dorsey S, Bolton P, et al. Building capacity in mental health interventions in low resource countries: an apprenticeship model for training local providers. Int J Ment Health Syst. 2011;5(1):30. doi: 10.1186/1752-4458-5-30 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Bolton P. Cross-cultural validity and reliability testing of a standard psychiatric assessment instrument without a gold standard. J Nerv Ment Dis. 2001;189(4):238-242. doi: 10.1097/00005053-200104000-00005 [DOI] [PubMed] [Google Scholar]
- 26.Haroz EE, Bass JK, Lee C, Murray LK, Robinson C, Bolton P. Adaptation and testing of psychosocial assessment instruments for cross-cultural use: an example from the Thailand Burma border. BMC Psychol. 2014;2(1):31. doi: 10.1186/s40359-014-0031-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Bass J, Murray SM, Mohammed TA, et al. A randomized controlled trial of a trauma-informed support, skills, and psychoeducation intervention for survivors of torture and related trauma in Kurdistan, Northern Iraq. Glob Health Sci Pract. 2016;4(3):452-466. doi: 10.9745/GHSP-D-16-00017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Michalopoulos LM, Unick GJ, Haroz EE, Bass J, Murray LK, Bolton PA. Exploring the fit of western PTSD models across three non-western low- and middle-income countries. Traumatology. 2015;21(2):55-63. doi: 10.1037/trm0000020 [DOI] [Google Scholar]
- 29.Gray MJ, Litz BT, Hsu JL, Lombardo TW. Psychometric properties of the life events checklist. Assessment. 2004;11(4):330-341. doi: 10.1177/1073191104269954 [DOI] [PubMed] [Google Scholar]
- 30.Elhai JD, Gray MJ, Kashdan TB, Franklin CL. Which instruments are most commonly used to assess traumatic event exposure and posttraumatic effects? a survey of traumatic stress professionals. J Trauma Stress. 2005;18(5):541-545. doi: 10.1002/jts.20062 [DOI] [PubMed] [Google Scholar]
- 31.Achenbach TM, Rescorla LA. Manual for the ASEBA School-Age Forms & Profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families; 2001. [Google Scholar]
- 32.Achenbach TM. Manual for the Youth Self-Report and 1991 Profile. Burlington, VT: University of Vermont, Department of Psychiatry; 1991. [Google Scholar]
- 33.Pianta RC. Child-Parent Relationship Scale. Charlottesville: University of Virginia; 1992. [Google Scholar]
- 34.McCoy CE. Understanding the intention-to-treat principle in randomized controlled trials. West J Emerg Med. 2017;18(6):1075-1078. doi: 10.5811/westjem.2017.8.35985 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Allison P. Testing for interaction in multiple regression. Am J Sociol. 1977;83(1):144-153. doi: 10.1086/226510 [DOI] [Google Scholar]
- 36.Vatcheva KP, Lee M, McCormick JB, Rahbar MH. The effect of ignoring statistical interactions in regression analyses conducted in epidemiologic studies: an example with survival analysis using Cox proportional hazards regression model. Epidemiology (Sunnyvale). 2015;6(1):216. doi: 10.1038/nbt.3121.ChIP-nexus [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Keefe RS, Kraemer HC, Epstein RS, et al. . Defining a clinically meaningful effect for the design and interpretation of randomized controlled trials. Innov Clin Neurosci. 2013;10(5-6 suppl A):4S-19S. [PMC free article] [PubMed] [Google Scholar]
- 38.Bauer DJ, Sterba SK, Hallfors DD. Evaluating group-based interventions when control participants are ungrouped. Multivariate Behav Res. 2008;43(2):210-236. doi: 10.1080/00273170802034810 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Bland JM, Altman DG. Multiple significance tests: the Bonferroni method. BMJ. 1995;310(6973):170. doi: 10.1136/bmj.310.6973.170 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Cohen P, West SG, Aiken LS. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. 3rd ed Mahwah, NJ: Lawrence Erlbaum Associates Inc; 2003. [Google Scholar]
- 41.ReliefWeb Kenya: drought 2014-2018. https://reliefweb.int/disaster/dr-2014-000131-ken. Published 2018. Accessed October 20, 2018.
- 42.Melhem NM, Porta G, Shamseddeen W, Walker Payne M, Brent DA. Grief in children and adolescents bereaved by sudden parental death. Arch Gen Psychiatry. 2011;68(9):911-919. doi: 10.1001/archgenpsychiatry.2011.101 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Mannarino AP, Cohen JA, Deblinger E, Runyon MK, Steer RA. Trauma-focused cognitive-behavioral therapy for children: sustained impact of treatment 6 and 12 months later. Child Maltreat. 2012;17(3):231-241. doi: 10.1177/1077559512451787 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Deblinger E, Mannarino AP, Cohen JA, Steer RA. A follow-up study of a multisite, randomized, controlled trial for children with sexual abuse–related PTSD symptoms. J Am Acad Child Adolesc Psychiatry. 2006;45(12):1474-1484. doi: 10.1097/01.chi.0000240839.56114.bb [DOI] [PubMed] [Google Scholar]
- 45.Jensen TK, Holt T, Ormhaug SM. A follow-up study from a multisite, randomized controlled trial for traumatized children receiving TF-CBT. J Abnorm Child Psychol. 2017;45(8):1587-1597. doi: 10.1007/s10802-017-0270-0 [DOI] [PubMed] [Google Scholar]
- 46.Ertl V, Pfeiffer A, Schauer E, Elbert T, Neuner F. Community-implemented trauma therapy for former child soldiers in Northern Uganda: a randomized controlled trial. JAMA. 2011;306(5):503-512. doi: 10.1001/jama.2011.1060 [DOI] [PubMed] [Google Scholar]
- 47.Beidas RS, Kendall PC. Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol (New York). 2010;17(1):1-30. doi: 10.1111/j.1468-2850.2009.01187.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Fairburn CG, Cooper Z. Therapist competence, therapy quality, and therapist training. Behav Res Ther. 2011;49(6-7):373-378. doi: 10.1016/j.brat.2011.03.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Herschell AD, Kolko DJ, Baumann BL, Davis AC. The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clin Psychol Rev. 2010;30(4):448-466. doi: 10.1016/j.cpr.2010.02.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Rawson RA, Rataemane S, Rataemane L, et al. Dissemination and implementation of cognitive behavioral therapy for stimulant dependence: a randomized trial comparison of 3 approaches. Subst Abus. 2013;34(2):108-117. doi: 10.1080/08897077.2012.691445 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Gilkey MB, Moss JL, Roberts AJ, Dayton AM, Grimshaw AH, Brewer NT. Comparing in-person and webinar delivery of an immunization quality improvement program: a process evaluation of the adolescent AFIX trial. Implement Sci. 2014;9(1):21. doi: 10.1186/1748-5908-9-21 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Beidas RS, Marcus S, Wolk CB, et al. A prospective examination of clinician and supervisor turnover within the context of implementation of evidence-based practices in a publicly-funded mental health system. Adm Policy Ment Health. 2016;43(5):640-649. doi: 10.1007/s10488-015-0673-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.