Evaluation of a Web-Based Training Model for Family Peer Advocates in Children’s Mental Health

Abstract

Objective:

To compare knowledge gains from a new online to an existing in-person training program for Family Peer Advocates (FPAs).

Methods:

Data came from a pre/post study of individuals who enrolled in the web-based Parent Empowerment Program (PEP) training, 144 of whom completed the training, pre/post tests and 140 were admitted to the analyses. Knowledge was assessed with 34 questions, 29 of which were common to both the online and in-person trainings. Pre/post knowledge changes were available from the in-person training.

Results:

There were statistically significant gains in knowledge on both the 34 questions and the 29 questions common to both trainings. Comparisons of knowledge gains across the two training models did not differ (U=21082.00, z=−0.16, p=0.870).

Conclusions:

Data on knowledge gains from this accessible, affordable online training show promise for training this growing and important workforce.

Introduction.

To address mental health workforce shortages and to improve parents’ engagement in their children’s mental health services, many states have funded family peer delivered services.¹ Studies demonstrate that embedding family peer advocates (FPAs) as deliverers of services can improve show rates and retention of families.² Nationally, FPAs provide a workforce to address the well-documented and severe workforce shortages in children’s mental health,³ with a growing number of states developing certification standards and a national certification process under development. A recent survey indicates that 28 states and the District of Columbia pay for Family Peer Support Services through Medicaid (usually through a State Plan Amendment or a Medicaid waiver) with the training and credentialing processes varying by state.⁴

The New York State Office of Mental Health (NYSOMH) has a 20-year history of supporting the FPA model to increase caregiver engagement in treatment through a network of FPAs embedded in children’s services provider agencies and in family-run organizations.^5,6 In NYS, FPAs receive training and credentialing through a structured FPA credentialing process, using the Parent Empowerment Program (PEP) training that follows a health activation framework.^1,7 Further, quality indicator standards are now available for use in the state to support clear specification of FPA roles.⁸ The overall goal is to enhance parents’ capacity to parent, navigate care systems and advocate for their children through, in part, assisting them to become fully engaged as partners with provider teams who provide care to their children.

Educating and credentialing FPAs in NYS was a time intensive process involving 40 hours of didactics followed by bi-weekly consultation calls for six months.⁹ In 2010, in order to make training more accessible, PEP training moved to a train-the-trainer model, again showing statistically significant increases in knowledge acquisition and self-efficacy.¹

With the recent changes to children’s behavioral health services in NYS, FPA services are Medicaid-reimbursable. Thus, there is a need for increasing the numbers of FPAs and for broadening their training so that they can support parents and caregivers across all child serving systems (e.g., education, medical, child welfare, substance use, juvenile justice). Knowing that 40 hours of in-person training was burdensome, costly and that FPAs needed immediate access to training to qualify to be reimbursed by Medicaid upon employment, in 2014 the NYSOMH awarded New York University a contract to create a hybrid training program consisting of online learning modules, two-days of in-person training, and three months of twelve consultation calls.

Web-based training programs have become increasingly popular as a cost-effective educational mechanism, particularly when training a workforce that is spread across a large geographical area. Research suggests that web-based training is effective for training health and mental health service providers.^10-12 Therefore, the purposes of this study were to: (1) examine the gains in knowledge from completing the web-based PEP Level 1 training program; and (2) compare knowledge gains for the online training to the in-person train-the-trainer training¹ using the 29 knowledge questions that were used to assess knowledge in both training modalities.

Methods.

Data for this pre/post study came from individuals who enrolled in the PEP Level 1 online training from 2/2018 to 4/2019. Of the 258 who accessed the online modules, 144 completed the training (56%). Two participants who completed the training and three who did not were removed from the study due to missing or improperly coded data. Two were removed because their change score on the general knowledge measures presented as extreme outliers (i.e., >3 interquartile ranges from the center of the data), leaving 140 participants in the completers group to examine knowledge gains, sociodemographic and employment factors related to knowledge gains, and to compare against the results of the in-person training. The individuals in the inperson training were FPAs like those completing the online training.¹ We also examined the sociodemographic characteristics common to both training groups and found no statistically significant age, sex, ethnicity or educational differences (data not shown). The online training is a quality improvement effort and did not require Institutional Review Board review.

The traditional five-day in-person PEP training covered the following topics: The FPA Role; Engagement and Empowerment Strategies; Listening and Communication Skills; Understanding the Children’s Behavioral Health and Education Systems; Partnership and Negotiation Skills; Managing Groups; Boundaries; and Self-Care. The training approach included a mix of didactic instruction, small group discussion, and role-playing.

Because the traditional in-person training was considered effective, the new training addressed a similar set of competencies. The process of developing the online training began by reviewing the topics in the existing training, changes in the FPA scope of practice, and new competencies required by system changes. This information was gathered through discussion with current trainers, FPA supervisors, FPAs, clinical partners, and state and national leaders. The content of the existing training was then re-reviewed for relevancy with any disagreements about content retention resolved through discussion. New topics for the online training were chosen through a similar process. Content experts, experienced FPAs, and a team of educational technologists then reviewed each online module. Of the content in the earlier in-person training, 90% was retained in the new online Level 1 modules.

The new online PEP modules are accessed through a Learning Management System and are completed independently by trainees. The modules are intended to introduce the role and fundamental principles of family peer support, teach basic skills, provide an overview of key child serving systems, and introduce other relevant topics. New content was added in the areas of crisis response, planning, documentation, and a structured approach to exploring family strengths and needs. A variety of activities was built into the updated training in lieu of in-person role-play and discussion (Online Supplement, Table B contains areas addressed).

Demographic questions completed prior to training consisted of age, gender, ethnicity, language spoken, education level, and several work characteristics.

The general knowledge pre/post tests consisted of 34 multiple choice and true/false questions assessing knowledge related to the PEP curriculum content. Of note, some of the questions are designed to gauge how FPAs would respond when given certain scenarios. Twenty-nine are identical to those used in the earlier evaluation.¹ As in the earlier evaluation, scores for both the 34 and 29 items were calculated as number of correct responses.¹

Statistical analyses were performed using SPSS Statistics Software (Version 25). Chi-square tests were conducted to examine potential differences between training completers and noncompleters.

Because data from the general knowledge test violated the assumption of normality, Wilcoxon signed-rank tests were used to evaluate change in knowledge. This test was first performed using the 34-items. Data from the 29 items that were identical across trainings were then analyzed in the current sample and the previous sample. Because the previous study, allowed participants with missing items on the pre- and post-tests to be included in the comparisons, we retained participants missing no more than four item responses per test in this sample. Missing items were calculated as incorrect. A Mann-Whitney test was then used to determine if knowledge gains varied between the in-person and the web-based trainings. Differences in knowledge gains across sociodemographics were evaluated using the Kruskal-Wallis and Mann-Whitney tests.

Results.

Completers did not differ on demographic factors from noncompleters (all p>0.050). However, significant differences in employment characteristics were found between groups, including: work status as an FPA (X² [1]=28.85, p<0.001) and whether their employer requires credentialing as an FPA (X² [2]=20.53, p<0.001). Fewer noncompleters were employed as FPAs and worked for settings requiring FPA credentialing (Online Supplement, Table A).

In the sample of 140 completers, scores on the 34-item post-test (M=27.82; SD=3.89; Med=29) were statistically significantly higher than the pre-test (M=26.39; SD=3.66; Med=27), z=−5.24, p<0.001, r=−0.31. Scores on the common 29 items were also statistically significantly higher on the post-test (M=23.65; SD=3.45; Med=24.5) than on the pre-test (M=22.57; SD=3.34; Med=23), z=−4.29, p<0.001, r=−0.26. The 304 participants who received the in-person model¹ also scored statistically significantly higher on the 29 common items on the post-test (M=23.73; SD=3.29; Med=24) than on the pre-test (M=22.66; SD=3.17; Med=23), z=−6. 70, p<0.001, r=−0.27. Importantly, knowledge gains across training models did not differ, U=21082.00, z=−0.16, p=0.870 (Table 1). No sociodemographic characteristic was related to differences in knowledge gains (data not shown).

Table 1.

Change in General Knowledge Scores After Web-based and In-person Trainings

Training Type	Pre-test Score	Post-test Score	P-Value	Effect Size (r)
Web-based Training Model (N=140)
Number correct out of 34 items (Updated)
M	26.39	27.82
SD	3.66	3.89
Med.	27	29	0.000	−0.31
Number correct out of 29 items (Matched)
M	22.57	23.65
SD	3.34	3.45
Med.	23	24.5	0.000	−0.26 A
Train-the-Trainer, In- person Training (N=304)
Number correct out of 29 items (Matched)
M	22.66	23.73
SD	3.17	3.29
Med.	23	24	0.000	−0.27 A

M = Mean

SD = Standard Deviation

Med. = Median

^{A =}Mann-Whitney test: U=21082.00, Z=−0.16, p=0.870

Discussion.

Given the growing use of Family Peer Support Services, having an efficient, effective, and accessible approach to training this emerging workforce is a critical need, particularly for states with large geographical areas. The cost and accessibility challenges of in-person didactic training prompted the NYSOMH to explore online training to teach the didactic portion of the effective in-person training models.^1,9 Data from the pre-post survey of the 140 completers of the online training showed a statistically significant gain in knowledge on both the battery of 34 items and the 29 items that were common to the pre/post tests administered in prior in-person trainings. Gains in knowledge were not statistically significantly different between participants who completed the training in-person or via the web-based modules. This is a notable finding since online training is considerably more accessible, convenient, and cost effective than in-person training. Importantly, most of the training content is not specific to NYS and, therefore, could be adopted by other states—thus potentially standardizing the educational training of FPAs nationwide.

The second important finding is that no sociodemographic characterstic was related to increases in knowledge. This is notable because data suggest that the use of technology in training is difficult for older individuals and those less comfortable with technology.¹³ The latter has been identified as an important reason for dropping out of training.¹⁴

These data have certain limitations. This is a small sample of individuals from only New York State. The online program was accessed by a total of 258 individuals, 56% of whom completed the training. Although there were no sociodemographic characteristics that differentiated completers and noncompleteres, the completers may have unmeasured characteristics that are partially responsible for the knowledge gain. However, a more likely explanation is that current FPAs for whom credentialing was not required or individuals interested in becoming credentialed as FPAs may have accessed the modules to understand the new responsibilities of an FPA. Importantly, these data are only from the online training. We have no information on whether improvements in knownledge translated into changes in the FPAs’ practice behaviors.

These initial data on knowledge gains from an online training program for FPAs show promise. Having an accessible and affordable means of training this growing, important workforce is an important step forward in improving the care for vulnerable children.