A Multidimensional Examination of the Measurement of Treatment Engagement: Implications for Children’s Mental Health Services and Research

Abstract

Objective:

The gap between rates of children’s mental health problems and their participation in services highlights the need to address concerns related to engagement in mental health services more effectively. To identify, understand, and resolve engagement concerns appropriately requires effective measurement. In this study, we employed a multidimensional conceptual framework of engagement to examine the measurement of engagement in intervention studies focused on improving children’s and/or families’ engagement in services.

Method:

We coded 52 randomized controlled trials (RCTs) of interventions designed to enhance treatment engagement published between 1974 and 2019 to examine what engagement constructs have been measured, how these constructs have been measured, who has provided information about engagement, and when and why engagement measures have been administered.

Results:

Attendance was measured in 94.2% of studies, and 59.6% of studies measured only attendance. Furthermore, most studies (61.5%) measured only one engagement dimension. One hundred twelve unique indicators of treatment engagement were used (61.6% measuring attendance). Infrequent measurement of youth (19.2% of studies) or caregiver (26.9%) perspectives was apparent. About half (54.7%) of measures were completed on one occasion, with 53.7% of measures completed after treatment was concluded.

Conclusions:

Results highlight how the field’s measurement of engagement has focused narrowly on attendance and on interventions that improve attendance. We consider promising new directions for capturing the multidimensional, dynamic, and subjective aspects of engagement, and for leveraging measurement in research and practice settings to feasibly and effectively identify, monitor, and address engagement challenges.

Roughly one in five youth in the U.S. has emotional or behavioral challenges significant enough to warrant treatment (Perou et al., 2013), yet less than half enroll in mental health services (Merikangas et al., 2010), and over 50% who enroll then terminate prematurely (Nock & Ferriter, 2005; Pellerin et al., 2010). The cost of this unmet need is considerable, as treatment dropout predicts poorer youth outcomes (Danko et al., 2016), and poor youth mental health can contribute to detriments in functioning and wellbeing into adulthood (Chen et al., 2006). Given the gap between need and participation, addressing problems related to treatment engagement – i.e., to individuals’ and families’ commitment and capacity to effectively participate in treatment – could have significant public health impact on the effectiveness of mental health services.

Fortunately, there is a substantial and growing evidence base of effective interventions designed to increase involvement in youth mental health services. These interventions have been summarized in several previous reviews (see Becker et al., 2018), and range considerably in terms of their characteristics, focus, and use within interventions. Briefly, they have been embedded within mental health clinics, but also in emergency departments and pediatric settings. They have been used to promote engagement in a variety of interventions, including parent training programs, multisystemic therapy, and child psychotherapy in general. They represent many approaches, from strategies to improve screening and referral processes, to those that orient youth and/or caregivers to treatment, to motivational enhancement strategies. Interventions also incorporate discrete clinical procedures in varying combinations, including appointment reminders, psychoeducation, and goal setting (Becker et al., 2018; Lindsey et al., 2014).

In psychology broadly, assessment is integral to intervention from start to finish. According to Meehl (1954/1996), standardized and/or statistical approaches can offer added value beyond pure clinical decision-making; using standardized measures in assessment both increases the accuracy of prediction, and provides context and comparison for the ideographic presentation of the client. As formulated by Youngstrom (2013), assessment can strengthen intervention through prediction (providing information regarding a diagnosis or other criterion of interest), prescription (informing the choice of treatment), or process (progress over the course of treatment, outcomes of treatment). The utility of an assessment approach can be evaluated based on its contributions to prognosis and treatment across these “Three P’s.”

The evidence base now affords us the opportunity to examine the field’s instrumentation related to treatment engagement. As interventions continue to mature and the collective body of research reveals what works for what purpose (cf. Paul, 1967; Becker et al., 2018), we should consider whether measurement strategies have similarly kept pace to allow us to evaluate how well our interventions are working and identify opportunities to advance measurement. The purpose of this paper was to characterize the measurement of engagement within the context of a sample of studies testing engagement interventions. Specifically, we examined what engagement constructs are measured, how these constructs are measured, who reports about engagement, and when and why engagement measures are administered. We discuss each of these parameters in turn.

Consensus has emerged among researchers that the construct of engagement is best characterized as multidimensional (e.g., Becker et al., 2015; Becker et al., 2018; Chacko et al., 2016; Haine-Schlagel et al., 2019;, Pullman et al., 2013), including domains that are social (e.g., therapeutic alliance), cognitive (e.g., expectancies related to treatment effectiveness, understanding of treatment approach), affective (e.g., emotions related to treatment such as hopefulness or frustration), and behavioral (e.g., attendance, active participation in session, homework completion). Reflecting these multiple domains, the nature of the engagement problems that occur in treatment vary (e.g., De Haan et al., 2013; Garibaldi et al., 2020; Karver et al., 2018; Wright et al., 2019). Thus, what is measured is a critical question. Ideally, the field’s instrumentation should have the capacity to accurately and reliably measure multiple domains of engagement. Prior research suggests, however, markedly disproportionate measurement of different domains; for example, in a review of engagement interventions using a five-dimensional conceptual framework for engagement (also used in the current study), attendance was measured in 94% of studies; other domains (relationship, expectancy, clarity, and homework) were each examined in less than 30% of studies (Becker et al., 2018). Given attendance alone does not predict treatment outcomes (Nock & Ferriter, 2005), and attendance and other engagement domains may be interrelated, a focus on measuring attendance might occur at the expense of advancing measurement of other social, cognitive, and/or affective treatment engagement factors and our ability to assess whether interventions improve them. Even within specific domains of engagement, definitions and methods of operationalization may diverge, leading to divergent outcomes, as Warnick et al. (2012) demonstrated in a study in which rates and predictors of attrition from youth psychotherapy varied depending on the method by which attrition was defined.

How dimensions of engagement are measured determines which inferences can be drawn about the construct and its components. The field of psychology has a tradition of ensuring that measures have strong psychometric properties and clearly defined parameters for use in various populations and contexts (Meehl, 1996). Psychometric properties of measures such as reliability and validity provide important information about the strengths and limits of measures, which in turn have implications for their use in research and practice (Cook & Beckman, 2006; Wood et al., 2007). Advancing research on the predictors of treatment engagement and outcomes of engagement interventions requires reliable, valid, and theoretically coherent measurement (cf Hock et al., 2015). An additional consideration reflects whether a measure is available to users beyond the research in which it was initially utilized. For a measure to be used in practice, it must be accessible; proprietary research measures, and measures offered at prohibitively high cost, cannot be adopted widely.

Sound assessment is also predicated on informants; who the informant is shapes what we know about the construct of interest, and we will likely obtain diverging information from different informants. Kraemer et al. (2003) proposed that various informants can be identified to represent unique contexts and perspectives, such that informant discrepancies provide complementary information that improve our capacity to comprehend a phenomenon more fully. Similarly, De Los Reyes et al. (2013) offered the Operations Triad Model (OTM) for identifying circumstances under which informant discrepancies yield meaningful information about behavior; in particular, the model suggests informant discrepancies are useful when a behavior expresses itself differently across a factor of interest (e.g., the severity of a behavior varies across settings), and the discrepant reports meaningfully reflect this variation. Integrating and building on the conceptual foundations of Kraemer et al. (2003) and the OTM (De Los Reyes et al., 2013), Makol et al. (2020) examined the incremental validity and predictive capacity of using principal components analysis to derive “trait scores” (components that reflect concerns across contexts and perspectives) from multiple informants to assess social anxiety, and found that trait scores explained more variance in observed adolescent social anxiety and predicted adolescent referral status than individual informants’ reports or a mean-derived composite of informants’ reports of social anxiety.

Several studies offer direct evidence of discrepant perspectives on treatment engagement. In a study examining concordance between therapist and client understanding of why clients terminated treatment, Hunsley and colleagues (1999) found that therapists underestimated how frequently clients’ reasons for termination involved dissatisfaction with therapy or therapist. In an examination of perceptions of barriers to youth mental health treatment (Champine et al., 2019), providers identified culturally competent care as a way to improve access and quality of treatment, yet cultural competence was not a concern raised by caregivers; however, caregivers identified barriers not noted by providers, including discomfort with in-home services, conflicting information from schools and providers regarding treatment, and inflexibility of services. As both De Los Reyes et al. (2013) and Kraemer and colleagues (2003) emphasized, discrepancies such as these highlight opportunities for refined lines of inquiry and enhanced decision-making by capturing complex constructs holistically. If we conceptualize treatment engagement as multidimensional, transactional, and inclusive of both concrete and subjective components, youth, caregiver, and provider perspectives all likely offer unique and valuable information to understand and address the array of engagement challenges that may arise over treatment.

Given that engagement is understood to be dynamic (i.e., changes over time), when we measure engagement may, similarly, offer varying information, and lend itself to varying purposes (i.e., why). For example, Nock and Ferriter (2005) distinguished between preparatory enhancement and continuous enhancement strategies for engagement; assessment at the beginning of treatment would indicate the effectiveness of any preparatory enhancement strategies implemented, but may not predict engagement at subsequent timepoints. Whenever it occurs, assessing engagement on one occasion provides a snapshot, and assessing engagement shortly after implementing an enhancement strategy provides information relative to the short-term effectiveness of the strategy; however, it may not be appropriate to extrapolate from this single measurement.

Illustratively, Langer et al. (2017) found that early in treatment, observers rated youth-therapist alliance to be higher for youth receiving manualized treatment than youth receiving non-manualized care; however, observer ratings for the two groups converged over time such that no differences between groups were observed in middle or late treatment. The authors speculated that early differences in alliance could suggest that properties of manualized treatment approaches (e.g., clear treatment structure and goals) might contribute to early alliance-building, whereas diminished differences subsequently could reflect increasing familiarity and comfort with session structure and therapist qualities for youth across conditions, or diminished enthusiasm over time for a manualized approach. These hypotheses highlight how measurement at different timepoints can be leveraged to understand different engagement phenomena. Relatedly, repeated measurement can facilitate the detection of emerging problems, and can be integrated as a dynamic piece of information to consider and respond to over the course of treatment. An engagement concern could be identified through measurement, then measured again over time to evaluate change, and whether changes are linked temporally with intervention components including those designed specifically to address the identified engagement problem and those with unintended consequences for engagement.

Current Study

Our goal was to gain insight about opportunities for advancing instrumentation around treatment engagement. We aimed to characterize the measurement of engagement within the context of intervention studies focused on improving children’s and/or families’ engagement in mental health services. We chose to focus on randomized controlled trials (RCTs) that explicitly targeted treatment engagement to understand how the most advanced children’s mental health research on the topic of engagement approached conceptualization and instrumentation; further, because participant flow is included in the Consolidated Standards of Reporting Trials, we suspected that a broader examination of engagement within children’s mental health interventions could result in a spurious inflation of studies that report treatment attrition and/or completion (i.e., attendance) as their only measures of engagement.

Specifically, this study examined the following questions: (1) What are the measurement patterns of engagement domains? (2) How have those domains been measured? (3) Whose perspectives and experiences regarding treatment engagement have been assessed? (4) When and for what purposes has treatment engagement been measured? As previous work has demonstrated that attendance is the most commonly measured engagement outcome (Lindsey et al., 2014), we hypothesized that attendance would be assessed with greater frequency than other engagement domains. Because attendance is often assessed via clinic records, we hypothesized that clinic records would serve as the most frequent information source. We hypothesized that most studies would measure engagement on a single occasion, as an intervention outcome.

Method

Selection Criteria and Sample

Forty-eight of the 52 studies in this review were initially compiled for a systematic review of engagement research undertaken to identify practice elements associated with improvements across specific domains of engagement (Becker et al., 2018). The literature search and selection process are detailed in Becker et al., 2018. In brief, the systematic review included RCTs that tested psychosocial interventions intended to improve youth or family engagement in children’s mental health services and reported outcomes for at least one measure of engagement, with a final sample of 48 articles published between 1974 and 2016.

We then conducted an independent literature search to identify additional articles published between January 2016 and December 2020. We searched PsycINFO, SocINDEX, and PubMed using the following terms: engagement OR retention OR attrition, plus an exploded “mental health services” term; we further specified “Randomized Controlled Trial” under “Article Type” in PubMed. We reviewed 563 records through PsycINFO; 181 records from SocINDEX, and 380 from PubMed. We also conducted a PubMed search with the same search terms, but specifying “Review” and “Systematic Review” for “Article Type.” Of the 209 articles returned, we reviewed abstracts for 33 potentially relevant articles, then closely examined references for five (Forman-Hoffman et al., 2017; Godoy et al., 2019; Moore, 2018; Werlen et al., 2019; Yasui et al., 2017). We examined references in four additional systematic reviews identified through PsycINFO and SocINDEX (Georgeson et al., 2020; Greef et al., 2017; Pereira & Barros, 2019; Petts & Shahidullah, 2020). Through this process we identified four additional articles appropriate for inclusion. Our final sample included 52 RCTs published between 1974 and 2019.

Across the 52 RCTs, data were included for a total of 6,340 participants between the ages of 0 and 21 years (M = 10.51, SD = 1.92; reported in 53.8% of studies). Studies tested engagement interventions in a variety of settings (or multiple settings), including clinics (67.3%), homes (42.3%), hospitals (5.7%), and the community (3.8%).¹

Coding Procedures and Reliability

Across studies, we identified 112 distinct measures of engagement, and coded them using eight codes: availability, psychometric indicator, information source, target, administration count, timing, purpose, and informed treatment (see Table 1 for code definitions and levels). Each study was analyzed independently by two coders. Coders met regularly to review code applications, focusing on clarifying coding nuances, preventing coder drift, ensuring overall consistency of code application, and resolving discrepancies through consensus (Palinkas, 2014), with the first author serving as auditor (Hill et al., 2005). Interrater reliability (see Table 1) was calculated prior to discrepancy resolution; kappas were all above published standards (i.e., at least 0.40; Fleiss, 1981).

Table 1.

Study Codes with Definitions, Levels, and Inter-Rater Reliability

Research Question	Codes	Code Definitions	Code Levels	Kappa
2	Availability	Availability of measure for use beyond the original research study setting	Available in the public domain with no associated cost; Available with associated cost; Unavailable	.84
2	Psychometric Indicator	Was at least one psychometric indicator reported for measure?	Yes; No	.98
3	Information Source	Source providing information / completing the measure	Service Records/Setting; Youth; Caregiver; Provider; Research Staff/Observer	.89
3	Target	Individual about whom the measure was completed	Youth; Caregiver; Family; Provider	.76
4	Administration Count	Number of times measure was completed over course of study	Count	1.00
4	Timing	The timing of measurement within the course of treatment	Before treatment; during/after 1st treatment session; repeatedly (measured multiple times over course of treatment); pre/post (measured before treatment and after conclusion of treatment); after conclusion of treatment; other	.78
4	Purpose	The purpose for which the measure was collected in the study	Initial evaluation (completed prior to engagement intervention); Progress Monitoring (to evaluate change in an engagement domain over the course of an intervention); Outcome Measure (dependent variable to determine intervention efficacy); Other	.57
4	Informed Treatment	Were measure data used in treatment planning, intervention selection, intervention adaptation, or another component of treatment/intervention delivery?	Yes; No	1.00

Note. “Engagement Domain” was coded for 48 studies in (Becker et al. (2018)). For the additional four studies coded in the current study, the kappa value for engagement domain was 1.00.

A ninth code, engagement domain, was reliably coded for the previously completed systematic review (Becker et al., 2018) and referred to a multidimensional measurement framework developed by Becker and Chorpita (2016) with five domains of treatment engagement organized around the acronym REACH: relationship (e.g., therapeutic alliance); expectancy (e.g., beliefs and/or expectations about how helpful treatment will be); attendance (e.g., barriers to attending or being on time for treatment sessions); clarity (e.g., shared understanding of treatment goals and approach); and homework (e.g., active participation and completion of work assigned as part of therapy by youth and family, both in and between sessions). Engagement domain was coded along with the eight codes described above for the four articles published after 2016.

Analytic Plan

Analyses were conducted using SPSS statistical software (v. 26). We addressed our four research questions as follows: (1) to assess the measurement patterns of engagement domains, we examined the frequency with which each of the five REACH domains was measured in the sample of studies, the frequency and modal number of domains measured across studies, and conditional probabilities for pairs of domains assessed across studies; (2) to assess how engagement domains have been measured, we examined the number and breadth of different types of measures within each domain; and features of the measures including measure availability, cost, and reported psychometric properties; (3) to assess whose perspectives and experiences regarding treatment engagement have been assessed, we examined the frequencies of information sources and measure targets used in studies, mean number of information sources per study, the proportion of information sources and targets associated with each domain, and the co-occurrence of information sources, and of targets, with one another; and (4) to assess the timing of measure administration and the purposes for which treatment engagement has been measured, we examined the frequency of measure administration, the modal number of times measures were completed, the timing of measurement within the course of treatment, frequencies examining the purpose of data collection overall and within REACH domains (i.e., was measure collected for purpose of initial evaluation or as an outcome of interest), and whether the data collected was used to inform the delivery of intervention.

Results

What are the measurement patterns of engagement domains?

The most frequently assessed engagement domain across studies was attendance (94.2% of studies), followed by homework (21.2%), expectancy (13.5%), clarity (11.5%), and relationship (11.5%). The modal number of REACH domains measured within a study was one: 61.5% of studies measured one domain, 26.9% of studies measured two domains, 9.6% of studies measured three domains, and 1.9% of studies measured four.

Table 2 presents the conditional probabilities of each other REACH domain being measured in a study when a given domain was measured. Fifty-nine percent of studies measured only attendance to assess engagement. When attendance was measured (i.e., the given domain in Table 2), the probability of another domain being measured ranged from 8.2% for relationship or clarity to 22.4% for homework. Attendance had a high probability of being measured (i.e., the conditional domain) when each other engagement domain was measured (i.e., when another domain was the given domain): relationship (66.7%), expectancy (71.4%), clarity (66.7%), and homework (100.0%). Homework was not measured concurrently with clarity or relationship in any studies.

Table 2.

Conditional Probabilities for Co-occurrence of REACH Domains within Studies

Given Domain	Conditional Domain
	Relationship	Expectancy	Attendance	Clarity	Homework
Relationship	—	66.7	66.7	50.0	0.0
Expectancy	57.1	—	71.4	28.6	28.6
Attendance	8.2	10.2	—	8.2	22.4
Clarity	50.0	33.3	66.7	—	0.0
Homework	0.0	18.2	100.0	0.0	—

Note. This table presents the conditional probability of other REACH domains being measured in studies when a given domain is measured. Numerals are percentages.

How have engagement domains been measured?

Indicators used to assess engagement across studies are presented in Table 3; aggregated across studies and domains, there were 112 unique indicators of engagement in this sample. Attendance was assessed using the greatest number of indicators, with 69 unique indicators across studies (61.6% of indicators). Total appointments attended was used in 22 studies; attendance at first session was used in 14 studies; and the proportion of youth/families that completed treatment (which varied according to how completion was defined) was used in 10 studies. Fifty-four indicators of attendance were used in one study each. Eighteen unique indicators were used to assess homework (16.1% of indicators), 13 indicators for expectancy (11.6%), nine for relationship (8.0%), and six for clarity (5.4%).²

Table 3.

Indicators and Measures Used to Assess Engagement by REACH Domain, with Frequency of Use Across Studies

Domain	Measures
Relationship	1 study: Adaptation to Therapy (AT), total score; AT, Relationship with Therapist subscale; AT, Child Communication subscale; Attraction-Receptivity Questionnaire*; Client Evaluation of Motivational Interviewing Scale (CEMI)*, Client Perceived Quality of Motivational Interview subscalea; Treatment Beliefs Questionnaire (TBQ) a*; Cross-Cultural Counseling Inventory-Revised*; Tolan Adapted Process Measure; Working Alliance Inventory-Short Form*.
Expectancy	2 studies: Parent Motivation Inventory (PMI)* total score. 1 study: Brief Development Survey*; CEMI*, Client Perceived Quality of Motivational Interview subscalea; Expectations of Therapy Outcome Scale*; Mental Health Services Grid Interview, Perceived Need for Treatment; Parenting Self-Agency Measure*; PMI, Readiness to Change subscale*; PMI Desire for Child Change subscale*; PMI, Perceived Ability to Change subscale*; Readiness to Change (% rating)*; Readiness Ruler*; TBQa*; Understanding Mood Disorders Questionnaire.
Attendance	22 studies: Total appointments attended. 14 studies: Attended first appointment. 10 studies: Completed treatment. 8 studies: Dropped out of treatment. 7 studies: Attended at least one session. 3 studies: Attended all scheduled appointments; Attended second appointment; No-showed to first appointment. 2 studies: Appointments attended on time; Attended intake and one therapy session; Attended first appointment or called to reschedule; Cancelled appointments (% of sample); Cancelled appointments (raw number); Families attended at least 4 sessions. 1 study: # clinical telephone support sessions; # days between appointments; # days between ER discharge and treatment initiation; # days between referral and initial screening appointment; # late arrivals to appointments; # online modules completed; # sessions within six months of ER discharge; # sessions within two months of ER discharge; Attended second visit to psychiatry or addiction program within 30 days of the first; Attended one session within six months of ER discharge; Attended one session within two months of ER discharge; Attended subsequent appointments; Attended third appointment; Barriers to Services Questionnaire (BSQ)*, # families who experienced at least one family barrier; BSQ*, # families who experienced at least one service barrier; BSQ*, # family barriers; BSQ*, # service barriers; Canceled first appointment; Changed or cancelled first appointment; Cases who used any medical services; Cases who used any mental health service; Completed all eight online modules; Completed first online module; Completed seven sessions; Duration of clinical telephone support sessions; Endorsed “other” as barrier to treatment; Enrolled in treatment; Family attended 4+ sessions; Family available for scheduled call times; Families who never attended one outpatient appointment; General Continuing Care Adherence scale*; Hours of intervention received; Mean # of minutes late to appointments; Mean # of medical visits; Mean # of mental health visits; Mental Health Services Grid Interview, # services used by family; No-showed for appointment (% of sample); No-showed for appointment (raw number); Outpatient appointments attended; Refused services; Service Assessment for Children and Adolescents (SACA), attended one session within six months of ER discharge; SACA, attended one session within two months of ER discharge; Services for Children and Adolescents - Parent Interview (SCAPI)*, cases who visited emergency room; SCAPI*, cases who had family therapy; SCAPI*, cases who had other therapy; SCAPI*, cases with overnight stay; SCAPI*, cases who visited psychiatrist; SCAPI*, cases who visited teen clinic; SCAPI*, mean # emergency room visits; SCAPI*, mean # family therapy visits; SCAPI*, mean # other therapy visits; SCAPI*, mean # visits to psychiatrist; SCAPI*, mean # visits to teen clinic; Overall attrition (% individuals who did not complete all eight online modules); Pretreatment attrition (% individuals who did not complete any of the eight online modules); Time spent completing online modules
Clarity	1 study: Therapy Expectations Questionnaire; Therapy Knowledge Survey; Therapy Survey*, Client Expectations of Therapy Structure subscale; TBQa*; Understanding of Therapy Questionnaire; Understanding of Therapy Test.
Homework	5 studies: Homework assignments completed. 1 study: Adherence Questionnaire (AQ), Quality of Parent Adherence in Past week subscale; AQ, Quantity of Parent Adherence subscale; Caregivers who provided name and address to staff; Caregivers who returned program registration; Dyadic Parent-Child Interaction Skills Coding System (DPICS)*, Clean Up Do Skills; DPICS*, Clean Up Don’t Skills; DPICS*, Negative Parenting; DPICS*, Positive Parenting; DPICS*, Play Do Skills; DPICS*, Play Don’t Skills; Days caregiver cooperates with treatment assignments; Parent Participation Engagement in Child Psychotherapy Observational Coding System; Parenting Practices Scale*; Therapist rating of Cooperation; Therapist rating of extent to which client participated; Therapist rating of participation quality*; Therapist-report measure of parent engagement*.

^aThese measures assessed multiple engagement domains.

^*[measure] indicates that a measure is free and available in the public domain.

It appeared that most measures (72.2%) were available in the public domain with no associated cost. Remaining measures were not available for use (i.e., unable to access in public domain). Most scale or survey measures (77.8%) were reported to include at least one psychometric indicator (e.g., intraclass correlation coefficient, Cronbach’s alpha, Cohen’s kappa). Availability of measures in the public domain varied across REACH domains. Attendance had the greatest proportion of measures available (93.8%), followed by expectancy (84.6%), homework (69.2%), relationship (55.6%), and clarity (33.3%). Among measures for which psychometrics are appropriate and expected (e.g., scales, coding systems, etc.), all measures used to assess attendance had at least one reported psychometric indicator (100%), followed by expectancy (83.3%), homework (83.3%), clarity (50.0%), and relationship (44.4%).

Whose perspectives and experiences regarding treatment engagement have been assessed?

Across studies, five sources provided information regarding engagement. Aggregated across domains, the most frequent information source was clinic records (86.5% of studies). Caregivers (26.9%), providers (23.1%), youth (19.2%), and research staff/observers (7.7%) also served as information sources. The mean number of information sources per study was 1.63 (SD = 0.82), and the modal number of information sources was one. Most studies relied on one information source (55.8%), followed by two sources (26.9%), three sources (15.4%), and four sources (1.9%). Assessments were completed regarding four targets (i.e., the person about whom assessments were completed). The most frequent target was the family (51.9% of studies), followed by caregiver (40.4%), youth (36.5%), and provider (5.8%). The mean number of targets per study was 1.35 (SD = 0.59), and the modal number of targets was one. Most studies (71.2%) examined one target, followed by two targets (23.1%) and three targets (5.8%). Table 4 presents the proportion of information sources and measure targets associated with each domain. Caregivers provided information related to all five domains, while youth provided information related to all domains except homework. Caregivers represented the primary information source for measures of relationship (66.7%) and expectancy (76.9%), and caregivers and youth were primary information sources for measures of clarity (each represented in 66.7% of measures). Most measures (59.4%) used to assess attendance relied on information from clinic records, and measures used to assess homework were most frequently completed by provider (44.4%). The family unit was the target of measurement for 39.1% of measures of attendance but was not a target for other domains. Youth were measure targets for all domains except homework. Caregivers were targets for measures assessing all five engagement domains; they were the only identified targets of measures assessing homework, and the predominant target for measures assessing expectancy (61.5%) and clarity (66.7%). Providers were the measure target for 44.4% of measures assessing relationship and 7.7% of measures assessing expectancy.

Table 4.

Proportion of Measures Assessing REACH Domains by Information Source and Target

Domain	Information Source
	Clinic	Youth	Caregiver	Provider	Research Staff/Observer
Relationship	0.0	44.4	66.7	22.2	0.0
Expectancy	0.0	30.8	76.9	7.7	0.0
Attendance	59.4	29.0	17.4	7.2	8.7
Clarity	0.0	66.7	66.7	0.0	0.0
Homework	16.7	0.0	16.7	44.4	38.9
Domain	Target
	Family	Youth	Caregiver	Provider
Relationship	0.0	44.4	22.2	44.4
Expectancy	0.0	30.8	61.5	7.7
Attendance	39.1	52.2	31.9	0.0
Clarity	0.0	66.7	66.7	0.0
Homework	0.0	0.0	100.00	0.0

Note. This table presents the proportion of measures assessing REACH domains by respondent and target (e.g., 30.8% of measures assessing expectancy used information from the youth). Numerals are percentages. Row percentages may exceed 100%, as some measures assessed multiple domains.

Table 5 presents the percentage and number of studies in which information sources and targets co-occurred. The highest degree of co-occurrence for information sources was between clinic and caregiver, with 21.2% of studies collecting information from both sources. Five studies (9.6%) collected information from the youth and caregiver concurrently. The greatest co-occurrence for measure targets was caregiver and family (17.3%). Caregiver and youth were both targets in 11.5% of studies.

Table 5.

Co-occurrence of Information Sources and Targets in Studies

Information Sources	Clinic	Youth	Caregiver	Provider	Research Staff/Observer
Clinic	__	11.5	21.2	15.4	5.8
Youth	__	—	9.6	5.8	0.0
Caregiver	__	__	__	9.6	1.9
Provider	__	__	__	__	3.8
Research Staff	__	__	__	__	__
Targets	Family	Youth	Caregiver	Provider
Family	__	1.9	17.3	1.9
Youth	__	__	11.5	3.8
Caregiver	__	__	—	3.8
Provider	__	__	__	__

Note. This table presents the percentage and number of studies in which information sources and targets co-occurred (e.g., 11.5% of studies used clinic and youth as information sources). Numerals are percentages.

When and for what purposes has treatment engagement been measured?

Within a study, the number of times an engagement measure was administered ranged from one to twelve; the modal number of times was one (54.7%). Measures were completed twice in 25.0% of studies and three or more times in 20.3% of studies. Most measures were completed after termination of treatment (53.7%), followed by completion in the first session (15.3%), repeated completion across the treatment episode (10.8%), completion before and after treatment (i.e., pre-post; 8.4%), completion prior to treatment initiation (8.4%), and completion at other defined time points (e.g., once after second or third session; 3.4%).

Most measures (54.5%) assessing relationship were completed multiple times over the course of treatment. Expectancy was assessed prior to treatment initiation in 35.7% of studies and before and after treatment in 35.7% of studies. Most measures (72.6%) used to assess attendance were compiled following treatment termination; most measures (66.7%) of clarity were completed before treatment initiation; and homework was assessed repeatedly across the treatment episode most frequently (46.2%). Most indicators (90.6%, n = 184) were used as outcome measures to assess the efficacy of interventions. Fewer measures were used for progress monitoring (7.9%, n = 16) and initial evaluation of engagement (1.5%, n = 3). No measures were used to inform treatment.

Discussion

In this study, we examined how treatment engagement was measured in 52 RCTs of interventions targeting engagement in children’s mental health services. Using the REACH framework, a multidimensional conceptual framework for treatment engagement used in previous studies (Becker et al., 2018; Becker et al., 2019), we scrutinized (1) what the measurement patterns were across engagement domains; (2) how those domains have been measured; (3) whose perspectives and experiences of treatment engagement have been assessed; and (4) when and for what purpose treatment engagement has been measured. Our intention was to examine how the field’s instrumentation related to engagement shapes what we know about effective interventions, and to identify opportunities for advancing science and for assessing and addressing engagement problems in practice.

In examining what the measurement patterns are across engagement domains, we found that attendance was measured in 94.2% of studies – far more frequently than other domains, but consistent with previous reviews (Lindsey et al., 2014, Becker et al., 2015). Most studies (61.5%) examined a single engagement domain, and 26.9% examined two domains, mostly attendance plus one other domain. Examining attendance may be a widely adopted strategy because it is cost effective and efficient. Absence at first appointments and treatment attrition might also be conceptualized as adverse events; much like monitoring hospitalization or suicide in intervention studies, measuring these attendance outcomes is critical because their negative downstream impact on treatment is close to absolute. However, assessing attendance without assessing other domains may result in missed opportunities to understand the nature of the engagement problem, as inconsistent or flagging attendance may be a superficial manifestation of a problem rooted in cognitive or social barriers, or appear after the optimal window for intervention (Kazdin & Wassell, 1999; Spirito et al., 2002). In other words, attendance is not synonymous with engagement; no single indicator or domain is. Instead, engagement is likely a multidimensional construct. Examining just a single domain limits our understanding of engagement as well as the relation between different domains.

In our examination of how engagement has been measured, we found considerable heterogeneity in the operationalization and assessment of engagement in each domain, from six methods for measuring clarity to 69 methods of measuring attendance. We found most scales, questions, or behavioral observation coding systems are free and publicly available (72.2%) and have at least one reported psychometric indicator (77.8%), indicating there are accessible measures that can be used to assess engagement, with information that consumers can use to assess whether the measure provides satisfactory reliability and/or validity.

Examining who supplied information regarding engagement, we found that 45 studies used information generated by the clinic, largely in keeping with the focus on attendance. Youth provided information in 10 studies. Although it seems redundant to state that a client’s own experiences and perspectives are integral to their engagement, this truism is not reflected in the instrumentation. For services targeting younger children it may be that caregivers are the primary participants in treatment. Further, regardless of client age, caregiver involvement in children’s treatment contributes substantially to youth outcomes (Dowell & Ogles, 2010), and caregiver participation is an important component of engagement given that caregivers often decide whether to initiate and/or remain in treatment. Yet caregivers provided information about engagement in only 13 studies. Caregiver and youth perspectives about treatment likely often diverge, as well (Garland et al., 2000), and it may therefore be useful to gather information from both, yet caregiver and youth were both reporters in only 9.6% of studies. “Family” was identified as the target of assessment in 51.9% of studies, but exclusively regarding attendance. Several studies examined “family” attendance without articulating whether and how youth and/or caregivers were meant to participate in treatment, obscuring the potential contributions of specific actors’ participation in the treatment process, and the distinct treatment challenges and implications for treatment outcomes that may be related to engagement problems with youth versus caregivers. Further, only 23.1% of studies included therapist as an information source, despite evidence suggesting that therapists’ perspectives – for example, their view of therapeutic alliance (Bachelor, 2013) and expectancies regarding client improvement (Meyer et al., 2002) – are associated with outcomes. Lastly, the modal reliance on a single information source suggests missed opportunities to generate nuanced and holistic understandings of the engagement processes and concerns addressed in these studies, given the value of strategically integrating multiple data sources to assess a construct of interest (De Los Reyes et al., 2013; Kraemer et al., 2003; Makol et al., 2020).

Examining when engagement was measured, we found that roughly half of the studies in our sample (54.7%) measured engagement on one occasion; another 25.0% of studies measured engagement twice. Most studies assessed engagement following the conclusion of treatment, associated with the reliance on assessing attendance outcomes in this sample of studies. Other domains were associated with different timing patterns; for example, expectancy and clarity were measured most frequently before treatment initiation, in keeping with the emphasis of the interventions in those studies on improving expectancy and clarity at the beginning of treatment through preparatory enhancement strategies (Nock & Ferriter, 2005). Relationship and homework were most often assessed repeatedly, perhaps reflecting in the first case a more established conceptualization that the therapeutic relationship changes over time, and in the second case the relative ease of tracking concrete behavioral indicators such as homework completion (Lindsey et al., 2014).

Our examination of why engagement was measured demonstrated that in 90.6% of studies measures were used to assess the outcome of an intervention, with fewer studies assessing initial levels of engagement or monitoring changes over time. Outcomes-focused studies have clarified a number of specific strategies that can improve engagement – particularly attendance at the first treatment session (e.g., Breland Noble, 2012; McKay et al., 1998) and across the course of treatment (e.g., Kutash et al., 2011) – for groups at heightened risk for engagement problems. The field has less evidence to specify how to understand which individuals demonstrate varying degrees and types of engagement problems across the course of treatment. Research to date has also not examined how engagement can be measured to inform subsequent intervention with a specific individual. This is notable given that the ultimate outcome of interest in treatment is symptom reduction or improved functioning, and engagement is a proximal, enabling factor.

Implications

These findings highlight opportunities for advancing the science and practice related to treatment engagement by more explicitly targeting questions through our measurement related to what, how, who, when, and why. An approach to the measurement of treatment engagement that captures more of its multidimensional, interactional, and dynamic nature could more accurately reflect the current consensus and extend our understanding around what comprises treatment engagement. A multidimensional approach to measurement may be particularly important because of evidence suggesting that while some empirically supported engagement interventions (e.g., psychoeducation) are associated with improvements across all engagement domains, others are associated with improved engagement in just one domain (e.g., appointment reminders, which improve attendance), and most are associated with improvements in two to three domains (Becker et al., 2018). That certain practices may be better suited to improve engagement in certain domains highlights the importance not just of assessing for engagement problems broadly but to support identification of domain-specific problems.

Characteristics of future potential measurement tools are also important (i.e., how engagement is measured). Consistent with the literature on pragmatic measurement (Glasgow & Riley, 2013), the measurement tool must be perceived by users as both useful and easy to use. Glasgow and Riley articulate a number of additional required and recommended criteria for pragmatic measures: a measure must be 1) important to stakeholders; 2) low burden for both respondents and staff; 3) actionable; and 4) sensitive to change (p. 238). Research to develop new engagement measurement tools that meet these criteria can facilitate translation into practice. Further, the relative dearth to date of studies utilizing consumer and provider perspectives, and drawing on multiple perspectives, suggests new directions for future research related to who provides data. This could include multi-informant measurement drawing on perspectives of those “closest to the action,” with assessment strategies designed to leverage discrepancies as a means to more fully understand the relevant constructs (De Los Reyes et al., 2013; Kraemer et al., 2003). Designing studies that prompt consumers (youth and caregivers) and providers to represent their experiences with service utilization and barriers would advance research by identifying who specifically might demonstrate an engagement concern; it could also illuminate transactional and interpersonal elements of engagement undetectable by examining one perspective.

Because treatment engagement is a dynamic, interactional, process-oriented construct, our field may benefit from research that approaches when to measure as “early and often.” If a concern related to treatment engagement is identified and an intervention implemented to address it, repeated assessment would facilitate evaluation of whether the problem improved following intervention, or whether a different approach might be needed. Repeated measurement can also contribute to the evidence around when different engagement domains may be activated or challenged in treatment. Given that changes in engagement over time are to be expected across domains, the interpretation of repeated assessment results may vary.

There may also be added value if studies expand why engagement is being measured. For example, accurate detection of engagement problems via multidimensional, multi-informant approaches can be used as an information-gathering step within an action cycle (Deming 1993), a goal-directed process in which actors make deliberate and informed plans, implement those plans, evaluate implementation outcomes, and make adaptations based on results that are themselves then implemented, evaluated, etc. Broadly, instrumentation may be conceptualized and applied most usefully in the context of a coordinated decision-making framework, in which a set of organizing principles define and guide how various components of service provision – treatments, clients, providers, etc. – function together in a system (Chorpita & Daleiden, 2014). Such a framework links the questions we ask in this paper. Future research around engagement instrumentation may be most useful if it is not considered separately from the system in which it is utilized; instrumentation questions of what, how, who, when, and why will be most valuable to consider in relation to one another, as part of a larger whole.

The current study examined resource-intensive intervention trials specifically focused on enhancing engagement; thus, we would expect that our results represent the “upper bound” in conceptualization and instrumentation related to measuring engagement. Although much thoughtful and rigorous research has been done to develop interventions to enhance engagement in treatment, our results highlight promising new directions for instrumentation and assessment approaches. We can design multidimensional measures to better understand when in treatment problems arise and their course over time. We can also design studies to examine the effects of engagement interventions on targeted dimensions and whether there is generalization across dimensions – for example if an intervention targets relationship, do client expectancies improve, as well? We can also seek to reduce the assessment burden by conducting item-response analyses and identifying specific items/indicators with predictive validity for downstream engagement outcomes (e.g., attendance or attrition).

The import of this work also extends to the broader field of interventions research. Despite the significant threat to high-fidelity treatment delivery posed by low engagement, interventions research in general tends to omit meaningful measures of engagement. Illustratively, an examination of Behavioral Parent Training studies (Chacko et al., 2016) found that very few studies assessed within or between-session engagement; most studies measured engagement only via attrition. This is a clinical context characterized by high attrition – 51% percent of individuals did not complete treatment across Behavioral Parent Training studies – but measuring engagement in relation to treatment outcome occurred infrequently. Measuring engagement in interventions research multidimensionally, repeatedly, across multiple informants, and within a dynamic decision-making framework can illuminate opportunities to improve engagement and thereby enhance treatment delivery and improve outcomes. Including multiple measures of engagement in interventions research can also leverage and contribute to the science of engagement, helping us strengthen conceptual models of how different engagement domains relate to one another.

In practice, we see even more striking patterns of attrition than in treatment outcomes research. As just one example, in a large study using a nationally representative sample, approximately 40% of youth enrolled in services attended only a single mental health visit, and only one third participated in a “minimally adequate” number of sessions (Saloner et al., 2014). Yet the small body of services research focused on engagement suggests that providers rarely measure engagement at all.

In a study in which providers described their experiences detecting clients’ engagement problems, they overwhelmingly reported relying on their own observation, and none used a formal measure of engagement (Becker et al., 2021); this is not ideal, given the limitations of relying solely on clinical judgment (Meehl, 1954/1996) and the documented challenges therapists have with accurately detecting engagement concerns (Hunsley et al., 1999). However, these results are also not surprising, given that the field has not yet provided a feasible (i.e., low burden) measurement approach for providers to assess multiple dimensions of engagement. There are, however, still opportunities to enhance practice. First, recognizing that engagement is multidimensional, providers can consider how to measure the subtler signs of social and/or cognitive engagement rather than focusing on attendance as the primary indicator. It may not be necessary, and certainly may not be feasible, to utilize a burdensome array of measures reflecting different dimensions, but providers could, for example, administer an established alliance measure at multiple timepoints, and supplement with questions to assess other domains. In the absence of any formal measures, providers could ask questions that elicit client perspectives related to different engagement dimensions – for example, a provider could assess expectancy by asking a client to give a subjective scale rating in response to the question, “How optimistic are you that in the end, therapy will help make things better?” Importantly, providers could connect the results of these assessments to possible solutions for intervention when problems are detected. In short, providers may benefit from a measurement approach that is low burden, detects multiple types of engagement problems, can be administered early and often, and that informs decision-making.

Limitations

This study provides a descriptive characterization of the measurement of treatment engagement in intervention research targeting engagement in children’s mental health services. Because we reviewed interventions aggregated over decades, this study does not characterize the current state of the field, and may not reflect important advances in measurement. Yet certain findings hold true across the sample, and therefore highlight new directions for future research; for example, no studies used measurement to inform intervention. Using the REACH framework could also be interpreted as privileging one perspective on the varied dimensions of treatment engagement. However, the framework is grounded in prior research (see Becker et al., 2018). Consensus in the field also holds that engagement is multidimensional, and all measures were coded as belonging to one domain or another; thus, this study still highlights that intervention research on engagement in children’s mental health has not utilized a multidimensional operationalization. Lastly, this study is limited to providing observed values; we cannot suggest expected values or benchmarks until other research addresses certain key questions: whether a hierarchy or causal structure links engagement domains; whether all engagement domains should receive equal attention; optimal schedules for assessing various domains; and optimal respondents or information sources for various domains. As the field progresses to examine these questions, we can synthesize what may be most important to measure, how and when to measure it, who is best suited to provide the information, and the best uses for that measurement to address engagement-related challenges.

Conclusion

This study examined how the measurement of treatment engagement has shaped our understanding of the construct and of effective interventions to improve it. Results highlight that the field’s instrumentation has supported research examining universally applied interventions that improve engagement in one or two domains, with a preponderance of studies examining attendance. Recent scholarship conceptualizing engagement as multidimensional, interpersonal, and dynamic clarifies new opportunities to build on current intervention research to expand our understanding of this crucial and complex phenomenon. There may be particular promise in research that measures engagement to detect and address potential engagement concerns, as that may create opportunities to leverage research evidence in practice settings. Developing a research base that utilizes multidimensional, multi-informant, repeated assessment of treatment engagement can help us identify 1) valid, reliable, and sensitive indicators of different types of engagement problems; 2) optimal timing for detecting different engagement problems and variations in different dimensions of engagement over time; 3) the associations among various facets of engagement; 4) what procedures increase which dimensions of engagement; and 5) how different engagement dimensions relate to treatment process and outcome. Research that focuses on the development of pragmatic measurement, and that considers how the assessment of treatment engagement can be folded into practice in order to help providers and agencies more effectively identify and address problems, will contribute even more to our shared pursuit of effectively supporting youth and families to benefit from mental health services. Gaining a fuller understanding of how engagement functions, and integrating actionable measurement, should be a shared priority for all researchers and providers focused on addressing children’s mental health concerns.