Outcomes of a Metaintervention to Improve Treatment Engagement Among Young Adults With Serious Mental Illnesses: Application of a Pilot Randomized Explanatory Design

Abstract

Purpose:

The objective of this study was to conduct a preliminary evaluation of a new young adult–centered metaintervention to improve treatment engagement among those with serious mental illness.

Methods:

Young adults, clinic staff, and policy makers provided feedback on the intervention, which is a two-module engagement program provided by a clinician and person with lived experience (peer) during intake. A two-group pilot randomized explanatory trial design was conducted, comparing treatment as usual with treatment as usual plus the engagement program, Just Do You. The primary outcomes were treatment engagement and presumed mediators of program effects measured at 3 months after baseline.

Results:

The randomized explanatory trial indicated that young adults in Just Do You were more engaged in treatment than treatment as usual and that changes in several mediators of engagement occurred. Mechanisms that demonstrated between-group differences were stigma, perceived expertise of providers, trust in providers, and beliefs about the benefits of treatment. Results also provide diagnostic information on mediators that the program failed to change, such as hope, self-efficacy, and emotional reactions to treatment. These results inform next steps in the development of this promising intervention.

Conclusions:

Just Do You illustrated feasibility, acceptability and preliminary impact. It represents an innovative metaintervention that has promise for improving treatment engagement in mental health services among young adults who have a history of poor engagement.

Few evidence-based engagement programs exist for emerging and young adults with serious mental illnesses, such as schizophrenia spectrum, mood, and anxiety disorders [1,2]. Transition-age youths often report uncertainty and trepidation about mental health treatment when transitioning to the adult system [3,4]. In late adolescence and early young adulthood, rates of mental illness are high [5] and treatment engagement is low [1,6,7]. Research has shown that treatment dropout and disengagement is associated with socioeconomic position (being poor) and ethnicity (being non-white) [8,9]. The present research developed and tested a metaintervention aimed at orienting young adults to treatment interventions. It is conceptualized as an add-on component for therapeutic interventions that treat mental illness among emerging and young adults to increase treatment engagement. This report summarizes promising data for the meta-intervention collected as part of a study examining engagement in Personalized Recovery-Oriented Services (PROS). PROS are community-based recovery and rehabilitation programs that integrate evidence-based treatments and supports for adults with serious mental illnesses [10].

Just Do You

Just Do You (JDY) evolved from extensive qualitative research [11] from decision science and from traditional mental health service uses frameworks (see Figure 1). The program targets selected mediators from Figure 1 [12]. It integrates expressive art activities (music and visual arts), technology-based narratives (e.g., celebrity testimonials), psychoeducation, and motivational interviewing principles [12]. Delivery of JDY can take either a group or individual format depending on the number of individuals experiencing intake at a given point in time at a clinic. Provider teams include the clinic intake staff, clinicians, and what we call recovery role models (RRMs). RRMs possess lived experiences with the mental health system [12,13] having undergone treatment within it. Clinicians and RRMs cofacilitate two 90-minute modules offered during treatment initiation.

Figure 1.

Conceptual model.

Module I addresses mental health literacy, stigma, and hope. It describes how treatment works and includes exercises on acceptance and recovery. It uses technology-based celebrity storytelling/narratives related to receiving treatment, while teaching skills to improve communication with providers. Module II validates past experiences and instills hope about the future. The module addresses unpleasant treatment experiences, including discrimination and other forms of harm, while also discussing positive experiences. JDY draws upon the culturally meaningful and empowering dimensions of music/art to facilitate conversations about mental health. Cofacilitators use the activities as processing tools to offer more accessible ways for participants to express emotions, build relationships, develop goals, and instill hope. In addition, the RRMs are involved in mentoring, check-ins, answering questions, and relationship building with the participants between modules [12].

The use of RRMs is innovative. RRMs have directly experienced the difficult processes that can be associated with receiving a diagnosis and receiving treatment. An integral role of the RRM is discussing their lived experience with these processes that include the need to accept one’s condition and trust one’s providers along with the ability to communicate with providers about treatment decisions. JDY is based on the premise that communicating messages about treatment efficacy, stigma, and acceptance, as well as how to transition to independent housing (e.g., single-room occupancy) or search for employment, by someone with lived experience can influence key mediators associated with treatment engagement [12].

JDY also draws heavily from research in decision science, the parallel to which became evident in our formative research that resulted in Figure 1 [11]. Decision theorists have identified five classes of social-psychological variables shown in thousands of studies [14] to represent the key types of cognitions that people consider when making a decision [12,14]. As applied to treatment engagement, these are (1) behavioral beliefs about the advantages and disadvantages of engaging in treatment (e.g., the perception that it works); (2) social norms (e.g., approval or disapproval from important others in one’s life); (3) image perceptions (e.g., stigma surrounding use); (4) emotional reactions to engage in mental health services (e.g., fear); and (5) self-efficacy or one’ perceived ability to engage the system. JDY addresses these key determinants/mediators, coupled with others suggested by the engagement and recovery literatures (e.g., hope [15,16]), to form messaging about the importance of treatment engagement and how to overcome obstacles to engagement, as per Figure 1.

A unique facet of JDY is its reliance on communication science [17]. Knowing which cognitive/affective variables in the decision process to target is only part of the puzzle. One also must know how to change those variables through appropriate messaging. Communication science differentiates message reception (attention/comprehension), message acceptance (agreeing with the message), and message retention (recalling the message at later times). In general, all three processes are necessary for message effectiveness. There is a wealth of research in communication science on how to leverage these processes as a function of structuring (1) the source who delivers the message and (2) how the message is delivered, including content and communication style [17]. JDY draws on this research. For example, research shows that messages are more effective if the source of the message is perceived as trustworthy (looking out for the best interests of the recipient) and expert (knowledgeable about the topic area). Many clinicians assume patients see them as trustworthy and possessing expertise, but this is not necessarily the case [17]. The RRM (and JDY more broadly) seeks to impact such perceptions on the assumption that doing so increases engagement. In addition, JDY draws from the recent literature on narrative health communication, which uses stories as a means of communication [18–20] and on music and expressive arts, analyzing and discussing song lyrics with mental health content/messages and creating visual art and poetry as a means to improve active participation, knowledge, reduce stigma, and promote overall social-emotional health and well-being [21,22].

Finally, JDY addresses practical barriers by providing Metro-Cards for transportation and making reminder phone calls. The overall focus is on preparing, orienting, and motivating young adults so they are empowered to make decisions about continued engagement in services.

Randomized Explanatory Trials

Recently, researchers have highlighted the importance of conducting randomized explanatory trials (RETs) instead of randomized controlled trials (RCTs) when evaluating programs [12,23]. RETs are RCTs, but they include mediators to identify the mechanisms through which a program affects an outcome. This allows one to empirically test (1) if the program has, in fact, changed each target mechanism it assumes is relevant to an outcome and (2) to evaluate if the presumed mediator is, in fact, relevant. Before conducting a full-fledged RET, it is useful to advantage funding mechanisms (e.g., NIH R21s and R34s) to conduct pilot research to lay groundwork for a larger RET. The present study illustrates such pilot work by (1) demonstrating initial effects of JDY on outcomes, (2) testing if the presumed mediators targeted by JDY are related to outcomes, and (3) testing if the program affects those mediators, all on a preliminary basis.

Methods

The methods and protocols were reviewed and approved by the university human subjects committee at NYU.

Recruitment and randomization

The project team partnered with four PROS programs in poverty-impacted communities in a large urban area in the United States. Intake providers introduced the project to potential participants; if the young adults were interested, they contacted project staff directly. One hundred forty young adults in the clinics were approached to participate in the study of which 10 declined and nine did not meet inclusion criteria. Once young adults signed consent, they were randomized to condition based on a pregenerated random number via simple random assignment. Neither intake staff nor research staff knew of condition assignment until after consent had been taken and the assignment had been executed. Recruitment took place over a two-year period, 2018 to 2020 (the project began with two PROS programs in 2018 and then added two more later in the first year).

Based on power analyses, we sought a total sample size of 195. However, owing to the COVID-19 pandemic, data collection ended prematurely, with enrollment at N = 121. Participants were assigned to treatment as usual (TAU) (n = 58) or TAU + JDY (n = 63). The TAU consisted of PROS, which incorporates evidence-based treatments, rehabilitation, and social supports [10]. Eligibility included (1) the ages of 18–34 years; (2) living with a serious mental illness (mood, anxiety, schizophrenia spectrum); (3) are in the intake process or enrolled in PROS; and (4) were formerly involved with or received safety net programs/ services (e.g., Medicaid, juvenile justice, foster care, Supplemental Nutrition Assistance Program). We used a wide age range because research suggests that the developmental contexts of youths and young adults in the United States (e.g., transitioning to adult roles, marriage, moving out of parent’s homes, beginning a career) are occurring at later ages [24,25]. The Census Bureau formally defines young adults using the age range of 18–34 years [26].

Providers

The PROS program providers consisted of about 20 clinicians per clinic. Within a clinic, JDY staff consisted of a single licensed clinician (drawn from the broader PROS clinic team) and a RRM. The TAU condition received intake as usual, whereas the JDY condition received the JDY modules in addition to the standard intake protocol. The JDY program modules were provided during intake. PROS clinicians were blind to treatment condition.

Measures

All participants completed assessments at baseline and 3 months after enrollment. The study used both prior validated scales and measures unique to the study.

Engagement.

The primary engagement outcome was a modified version of the widely used engagement measure as reported by Yatchmenoff, [27] with items from the therapeutic “working relationship” and “buy-in” subscales; it summed eight items each rated on five-point disagree-agree scales (total score from 5 to 40). Sample items are “I am not just going through the motions. I’m really involved in working with the PROS staff.” We also assessed the number of sessions attended at PROS over the past two weeks, “In the past two weeks, about how often have you gone to PROS?” with a response metric ranging from “1” (one day a week) to “5” (every day of the week).

Mediators/mechanisms of change.

We measured six categories of mediators as discussed previously from decision science. Five categories (perceived advantages/disadvantages of treatment, emotional reactions to treatment, social norms, image management, and self-efficacy) relied on strong psychometric approaches tied to decision theory constructs (refer to the study by Fishbein M and Ajzen [28]) for details, with most using five-point disagree-agree scales. Examples include “Continuing my treatment will provide me with non-judgmental support” (perceived advantage/disadvantage); “When I think about the idea of continuing my medication it makes me anxious” (emotional reaction); “If others who are important to me found out that I follow up with my treatment, I would be seen by them as crazy” (image management); “How would your mother feel about you continuing your treatment for your difficulties (issues) at this time in your life?” (social norms–these were measured with a 6-point approval/disapproval scale and then averaged across 8-referents); and “It is easy for me to attend my treatment sessions” (self-efficacy). We also assessed trust in PROS providers (I trust that the mental health care staff here is sincerely working to improve my mental health and perceptions of their expertise (My providers are knowledgeable about treatment) The trust and expertise measures had 8 items. Finally, we used two measures of hope, a standardized measure of “generalized hope” [29] that had 12 items each rated an 8-point Likert scale (e.g., “I energetically pursue my goals,” and a measure of hope related to mental health (4 items using the same 8-point Likert [e.g., “I am more hopeful about my future because I have found ways to manage my mental health condition”]).

Analytic approach

RETs are best analyzed using structural equation modeling. However, in pilot research, sample sizes typically cannot sustain such modeling. Instead, we used a combination of small sample size–appropriate regression and limitation information analysis that applied traditional single degree of freedom contrasts for between-group comparisons of means at the 3-month assessments for both the mediators and the outcomes. We used Mplus to apply full information maximum likelihood methods to accommodate missing data. For relationships between mediators and outcomes, we calculated traditional correlations at the 3-month data point, again because the sample size would not sustain complex multivariate mediational modeling.

Results

Study sample

Study participants were 66% men; 38% identified as black/African American, 34% identified as Latino/Latina, 16% identified as biracial or multiracial, 4% identified as white, 2% identified as Asian, and 7% reported “other.” The participants’ average age was 26 years, with a range of 18–34 (5% were younger than 20 years; 68% were in their 20s, and 27% were in their early 30s). Sixty-one percent (n = 74) reported a current diagnosis of a schizophrenia spectrum disorder, 40% (n = 48) reported bipolar disorder, 31% (n = 37) reported depressive disorders, 19% (n = 23) reported anxiety disorders, 10% (n = 12) reported trauma and stressor-related disorders, 5% reported personality disorders (n = 6), and there were 14 additional disorders which were reported by less than five young adults (e.g., dissociative disorders, substance abuse disorders, gender dysphoria). These do not total 121 as many young adults reported more than one current diagnosis. All participants had a serious mental illness as a primary diagnosis. Formal diagnosis was determined based on responses to standardized self-reports. We analyzed the medical records of a subset of the sample (n = 30) and found that the diagnoses based on the self-report measures matched clinician diagnoses reported on the medical records for 89% of the cases.

The sample consists of individuals who receive Medicaid, Medicare, and/or Social Security/Social Security Disability Insurance. Participants also reported receipt of federally funded safety net services (e.g., Supplemental Nutrition Assistance Program, WIC, foster care, juvenile justice, and/or special education), and 41% of the sample reported having experienced homelessness. Ninety percent of the sample reported previous inpatient hospitalization(s) and 44% of the sample reported at least one suicide attempt.

Twenty-five percent of the sample reported not completing high school, 44% completed high school, with an additional 4% reporting that they completed a General Educational Development. Finally, 26% reported having completed some post-secondary education ranging from a few classes to one young adult who completed a master’s degree. At the time of the study, 7% of the sample reported that they were currently in school and 8% reported they were currently employed. We evaluated if there were any notable between-group differences as a function of treatment condition on the aforementioned variables and did not find any noteworthy disparities.

JDY completion rates and attrition

Eighty-seven percent of individuals in the JDY condition completed both JDY sessions; 5% attended one of two sessions; 8% did not attend either session. All individuals who were randomized were included in the analyses. Seventeen of the JDY sample did not complete the key study measures at 3 months and 21 of the TAU group did not do so. Of these 38 individuals, 10 were scheduled to complete the 3-month assessment but could not because of study termination owing to the pandemic. There was no evidence for meaningful differential attrition as a function of treatment condition.

Effects of JDY on engagement

Table 1 presents results for the primary outcome. A significant difference in engagement was observed between TAU + JDY and TAU at 3 months (p < .05, Cohen’s d = .59). A secondary engagement outcome measuring the number of sessions attended in the past two weeks yielded a near-significant difference (p < .056, Cohen’s d = .41) favoring TAU + JDY.

Table 1.

Effects of Just Do You

Variable	Corr	TAU + JDY mean	TAU mean	Diff	p value	Cohen d
Engagement
Engagement scale	-	35.71	32.67	3.04*	.011	.59
Sessions attended past 2 weeks	-	3.43	2.89	.54	.056	.41
Mediator
Behavioral beliefs (BB)
BB: benefits of treatment	.65*	43.46	38.47	4.99*	.01	.57
BB: Perceived benefits of medication	.43*	35.43	34.84	.59	.75	.07
BB: credibility of providers	.77*	37.38	33.70	3.68*	.002	.72
BB: trust in providers	.71*	29.07	26.29	2.78*	.02	.51
Emotional reactions (ERs)
ER: emotional reactions to treatment	.43*	20.33	18.91	1.42	.38	.19
ER: emotional reactions to medication	.37*	18.58	18.31	.27	.88	.03
Social norms (SNs)
SN: social norms (approval) for treatment	.38*	4.85	4.99	−.14	.53	.10
SN: social norms (approval) for medication	.27*	4.81	4.99	−.22	.35	.21
Image management (IM)
IM: indifference to stigma re: treatment	.47*	30.00	26.61	3.39*	.03	.47
IM: indifference to stigma re: medication	.57*	32.26	29.11	3.15	.12	.35
Self-efficacy (SE)
SE: difficulty accessing treatment	−.46*	4.50	5.05	−.55	.28	.24
SE: difficulty taking medication	−.41*	4.82	4.58	.20	.74	.85
SE: perceived control over treatment	.34*	13.30	13.47	−.02	.98	.68
SE: perceived control over medication	.35*	13.16	13.74	−.58	.34	.20
Hope (HO)
HO: generalized hope	.49*	67.34	63.03	4.31	.13	.34
HO: mental health hope	.59*	28.48	27.64	.84	.37	.20

Corr = correlation of variable with engagement scale.

^*p < .05.

Mediator associations with outcome

JDY assumed that each of the targeted mediators derived from Figure 1 would be associated with or relevant to engagement. In pilot RETs with small Ns, these assumptions can be tested by evaluating the correlation of each mediator with engagement across treatment conditions. (As noted, formal mediation analysis requires a large N that is unrealistic for pilot RETs; hence, we used less sample size demanding correlation coefficients.) If a given correlation is statistically nonsignificant, this questions assumption viability of mediator relevance. All mediators reported in Table 1 yielded statistically significant correlations and this also was true if treatment condition was statistically controlled.

These analyses ignore common/shared variance among the mediators and, of course, should be viewed as preliminary. We examined zero-order correlations among the mediators and generally found that the mediators within a category were moderately correlated (with some fluctuations) but that correlations across categories were more modest. For example, within the self-efficacy category, the correlations averaged .49 and within the hope category, they were correlated .61. The correlations across categories ranged from .02 to .63. We comment on these correlations below.

JDY effects on mediators

Pilot RETs can also preliminarily examine which of the predictive mediators that the program impacts. For mediators which are shown to be relevant and for which no program effect is evident, the program needs to be revised to strengthen influence on the unaffected mediator before a larger RET is conducted. Table 1 presents results for each mediator; significant effects were observed for four of the 16 mediators, representing behavioral beliefs about treatment, perceived expertise of providers, trust in providers, and image management surrounding treatment (stigma).

Discussion

JDY yielded promising results and is worthy of further investigation. JDY improved engagement and also impacted four mediational targets hypothesized to be important mechanisms of engagement (i.e., behavioral beliefs about advantages of treatment, credibility of PROS providers, trust in providers, and stigma surrounding treatment). Interestingly, the program was not found to impact emotional reactions to treatment, self-efficacy, hope, and social norms, even though it was intended to do so. These results provide important diagnostic information for future refinements to strengthen JDY before embarking on a formal RET. For example, the results suggest that module content and processes that focus on self-efficacy need to be revisited, perhaps by including efficacy-enhancing exercises, such as role plays of young adults and providers discussing medication planning. Furthermore, the lack of impact on hope and emotional reactions to treatment might be better addressed by increasing time with the RRMs and reinforcing RRM protocols for addressing these constructs. Regarding hope, the team determined, in hindsight and in light of the study results, that the Snyder scale for hope, which is widely used, did not map well onto intervention activities directed at increasing hope; that is, many of the items focused on dispositional hope whereas the program sought to change situational/clinical hope as focused on mental health. A better and more responsive measure should have been used.

Although there was shared variance among constructs within several mediational categories, we do not recommend necessarily aggregating or factor analyzing items within a category. Each variable/item contains unique content that concretely informs program designers what they need to focus on even if the variable shares variance with other items. In our research, items were dominated by unique variance (e.g., correlations of .60 share only 36% common variance). Researchers can sometimes be too quick to appeal to abstract constructs underlying common variance rather than focusing on the variance of each construct in its own right.

Overall, the aforementioned findings show the value of RETs over RCTs. An outcome only RCT would conclude that JDY was effective and stop there. The RET pinpoints which hypothesized mediators seem to be relevant, which of these mediators the program successfully changed, and which are the ones the program failed to change meaningfully. In this sense, RETs provide specific feedback for program improvement. RETs analyze the individual links within mediational chains and point to where these links are “broken” or, alternatively, where they are successfully joined. To be sure, such analyses require larger sample sizes and more rigorous designs than pilot RETs, but the latter can be helpful in screening relevant mediators for a larger RET and for strengthening an intervention before making the large resource investments required by a full-fledged RET. One, of course, must be careful in a pilot RET of embracing null results owing to low statistical power with small sample sizes. Nevertheless, the significance patterns and effects are of interest and informative. RETs are an exemplary method for those applying experimental therapeutics to their intervention research programs, as experimental therapeutics highlights the importance of identifying and testing the main “targets” of interventions and whether, or not, they are changed by the intervention [30].

Overall, our results suggest that JDY improves engagement among young adults with serious mental illnesses, in part, through the changes it brings about in stigma, beliefs about the advantages of treatment and beliefs that PROS providers are credible and trustworthy. These mechanisms may be useful for other engagement programs to consider among this population.

JDY is unique because its design derives from classic theories of mental health engagement [11,12] coupled with decision science [14] and communication science [17]. It makes use of RRMs and methods with expressive arts to motivate participants [12]. We ultimately intend JDY to be a metaintervention that can be used for any type of therapeutic program for young adults, but research is needed to establish its reach and generalizability. We are in the early stages of this research program.

As is true with all research, the present study has limitations. It had a relatively small sample size, in part because of early study termination owing to the pandemic and because such is the nature of pilot research. Results also cannot confidently be generalized to other populations. Although our sample was predominately in their 20s, we defined young adulthood encompassing a broader age range (18–34 years) than some research from our literature review. Our approach is consistent with developmentalists who emphasize the concept of extended adolescence and, in turn, emerging and young adulthood. For example, Steinberg [31] defines adolescence as “the stage of development that begins with puberty and ends with economic and social independence,” beginning as early as 10 years and continuing well into a person’s 20s. Our approach is also consistent with definitions by the U.S. Census Bureau [26]. The transition to young adulthood and, by implication, young adulthood itself is age-extended in the modern U.S. society. One always must interpret findings from pilot studies with caution and, of course, the quality of conclusions is tied to the quality of measures used. Limitations notwithstanding, the present study adds useful information to the field on a promising intervention which addresses a long-standing public health concern, namely lack of engagement in mental health treatment among young adults with serious mental illnesses. As well, it provides perspectives on the design of pilot RETs. Our own opinion is that complementing the RET design with the collection of qualitative data that interviews people before, during, and after treatment would be illuminating.

IMPLICATIONS AND CONTRIBUTION.

Transition-age youths and young adults experiencing mental illness often disengage in mental health treatment. Just Do You is an orientation program that aims to improve treatment engagement and was designed in partnership with stakeholders. This study reports on outcomes of a pilot randomized explanatory trial that indicates promise.