The Validation of a Provider-Reported Fidelity Measure for the Transdiagnostic Sleep and Circadian Intervention in a Community Mental Health Setting

Abstract

Monitoring treatment fidelity is essential to check if patients receive adequate doses of treatment and to enhance our theoretical understanding of how psychosocial treatments work. Developing valid and efficient measures to assess fidelity is a priority for dissemination and implementation efforts. The present study reports on the psychometric properties of the Provider-Rated TranS-C Checklist – a provider-reported fidelity measure for the Transdiagnostic Sleep and Circadian Intervention (TranS-C). Adults with severe mental illness (SMI; N = 101) seeking treatment in a community mental health setting received eight sessions of TranS-C. Therapists completed the Provider-Rated TranS-C Checklist at the end of each treatment session (N = 808) to indicate which modules they delivered during that session. To assess convergent validity, independent raters scored modules delivered from audio recordings of a subset of sessions (n = 257) for the modules delivered using the Independent-Rater TranS-C Checklist. Using exploratory factor analysis, a unidimensional scale composed of TranS-C’s modules was identified. Provider-Rated TranS-C Checklist scores were positively associated with the Independent-Rater TranS-C Checklist scores demonstrating convergent validity. Results indicate that the Provider-Rated TranS-C Checklist yields reliable and valid scores of providers’ delivery of TranS-C.

Fidelity has been defined as “the degree to which an intervention or program is delivered as intended” (Carroll et al., 2007) and is comprised of therapist adherence, therapist competence, and treatment differentiation (Schoenwald, 2011; Walton, Spector, Tombor, & Michie, 2017). Monitoring treatment fidelity allows us to check if patients receive the appropriate dose of the treatment, enhances our theoretical understanding of how a treatment works, and helps us differentiate whether positive or negative findings in clinical trials are due to the treatment itself or to its administration (Borrelli, 2011; Schoenwald et al., 2011). The present study examines the therapist adherence facet of fidelity, or the extent to which a provider utilizes the required procedures (Schoenwald et al., 2011). The rationale for this focus is threefold. First, adherence is the most essential element of fidelity (Carroll et al., 2007). Second, accumulating evidence indicates that adherence is associated with enhanced treatment outcome (Durlak & DuPre, 2008; Eames et al., 2008). Third, therapist adherence is under-researched (Breitenstein, Gross, et al., 2010). By establishing a novel measure of therapist adherence within a community mental health center context, this study will contribute to knowledge on fidelity measures that can be employed in routine care (Schoenwald et al., 2011).

The gold standard method of measuring fidelity involves observation via audio or video recordings of treatment sessions (Borrelli, 2011; Walton et al., 2017). Independent raters score providers on a list of predetermined criteria that providers are expected to deliver in each session according to the treatment manual (Borrelli, 2011). Observational methods are burdensome as they require additional raters, time, and resources (Schoenwald, 2011; Schoenwald et al., 2011). This burden is not feasible in routine care settings (Schoenwald et al., 2011). Hence, there is a need for brief, efficient, and accurate measures of fidelity (Schoenwald & Garland, 2013).

Provider self-report methods show promise as less burdensome and more efficient measures of adherence. For example, Hogue et al. (2014) found that providers working in an academic research setting with youth at risk for substance use were able to accurately report adherence when compared to the results of an observational fidelity measure. Similarly, a provider-rated checklist was highly correlated with an observer-rated fidelity measure for a parent training intervention delivered to low-income parents in a community setting (Breitenstein, Fogg, et al., 2010). Indeed, the psychometric properties, including convergent validity with independent-rater scales and content validity, have been established for a range of provider-reported fidelity measures with various populations and treatments including primary care providers, family therapy for adolescents with behavioral problems, and motivational interviewing and cognitive behavior therapy for adolescent substance use (Beehler, Funderburk, Possemato, & Dollar, 2013; Beehler, Funderburk, Possemato, & Vair, 2013; Hogue, Dauber, & Henderson, 2017; Hogue, Dauber, Lichvar, Bobek, & Henderson, 2015). These studies were conducted within various settings such as the Veteran’s Health Administration, academic settings, and community mental health settings. This is encouraging progress given that provider-reported measures of fidelity have distinct advantages. Not only do they measure fidelity briefly and efficiently, but they also continuously remind providers of the treatment and they have been rated as helpful by community clinicians (Bearsley-Smith, Sellick, Chesters, & Francis, 2008; Bloomquist et al., 2016). Promisingly, studies have also found that provider self-reported fidelity can predict treatment outcome in both academic and community mental health settings (Ellis, Naar-King, Templin, Frey, & Cunningham, 2007; Nakamura et al., 2014).

Despite advances in developing and validating provider self-report fidelity measures, significant gaps remain. First, few of these studies have examined fidelity when treating individuals with severe mental illnesses (SMI). Indeed, a recent review found that under five percent of the existing literature on adherence measures focused on SMI (Schoenwald & Garland, 2013). Second, despite the positive findings of provider-reported fidelity measures reviewed above, the evidence for provider-reported fidelity measures is mixed – at times, providers in both academic research and routine practice settings rated their adherence as higher than the observers’ ratings (Carroll, Nich, & Rounsaville, 1998; Hurlburt, Garland, Nguyen, & Brookman-Frazee, 2010; Martino, Ball, Nich, Frankforter, & Carroll, 2009; Mullin, Saver, Savageau, Forsberg, & Forsberg, 2016). Third, while psychometric information has been reported for several provider-reported measures, a recent review found that under five percent of these studies reported validity data (Schoenwald & Garland, 2013). Hence, by seeking to address the above gaps, this study will contribute to the growing literature of provider-reported fidelity measurement.

The focus of the present study is the Transdiagnostic Sleep and Circadian Intervention (TranS-C; Harvey & Buysse, 2017) implemented with adults with SMI (Harvey et al., 2016) in a community mental health setting. TranS-C is a modular intervention that targets psychosocial, behavioral, and cognitive contributors to sleep and circadian dysfunction. TranS-C is transdiagnostic in two ways. First, TranS-C treats a range of the most common sleep and circadian problems that can overlap and simultaneously co-occur among individuals with SMI (Giglio et al., 2010; Ohayon, 2012). Second, TranS-C is designed to be helpful across SMI. This dual focus is important because various sleep and circadian dysfunctions are common across numerous mental illnesses including depression, anxiety, bipolar disorder, schizophrenia, and substance use (Baglioni et al., 2016) and have been associated with suicidal ideation and attempts (Goldstein, Bridge, & Brent, 2008). One study has reported promising efficacy data for TranS-C (Harvey et al., 2018).

Research conducted in community mental health settings is important as these settings are major, publicly-funded providers of treatment for SMI. They provide for the poorest and most underserved members of the community who experience high rates of comorbidity and complexity. Yet, prior research has demonstrated that maintaining fidelity is challenging when interventions are implemented in community mental health settings (Lau et al., 2017; Stirman et al., 2013). One explanation for this difficulty is that portions of a complex or modular protocol may be impractical for certain populations or settings (Allen, Linnan, & Emmons, 2012). For this reason, it is essential to identify the core elements of an intervention in advance of its implementation (Walton et al., 2017). Hence, identifying the elements of TranS-C that providers frequently use with patients with a SMI in a community mental health setting will allow for the identification of the elements of TranS-C with the best “fit” prior to larger-scale implementation efforts. In addition, a comparison of provider-rated vs. independent-rater module use will also allow for another dimension of evaluating a novel fidelity measure.

The goal of this study is to develop and validate a provider self-report measure of fidelity for TranS-C – the Provider-Rated TranS-C Checklist. The first aim is to establish the internal consistency and structural validity. The second aim is to establish convergent validity by comparing ratings on the Provider-Rated TranS-C Checklist with ratings on the Independent-Rater TranS-C Checklist. The third aim is to characterize TranS-C delivery in terms of the modules most commonly delivered on the Provider-Rated TranS-C Checklist vs. the Independent-Rater TranS-C Checklist.

Methods

Participants

The 101 participants included in this study were drawn from a National Institute of Mental Health-funded randomized controlled trial that included adults who met criteria for SMI and sleep and circadian disturbance and were recruited from multiple sites within Alameda County Behavioral Health Care Services (ACBHCS; Alameda County, CA, USA) (Harvey et al., 2016). The primary trial included 121 participants. However, 20 participants were excluded from the present study as they were missing Provider-Rated Checklists. SMI was operationalized according to prior research (Kessler et al., 2003) and is defined as the presence, for 12-months, of at least one DSM mental disorder that leads to substantial interference with one or more major life activities. Participant demographics are displayed in Table 1.

Table 1.

Participant Demographics

	Provider-Rated TranS-C Checklist Sample (N=101)		Independent-Rater TranS-C Checklist Subsample (n=33)
Characteristic	Mean or N	SD or %	Mean or N	SD or %
Age (years)	47.83	12.28	48.54	11.31
Female	53	52.50	17	51.52
Race
African-American or Black	43	42.60	17	51.52
American Indian/Alaskan Native	2	2.00	2	6.06
Asian	6	5.90	1	3.03
Caucasian	38	37.60	9	27.27
Native Hawaiian or Other Pacific Islander	2	2.00	1	3.03
Mixed Race	6	5.90	1	3.03
Not specified	2	2.00	2	6.06
Ethnicity
Hispanic or Latino	16	15.80	3	9.09
Not Hispanic or Latino	84	83.20	29	87.88
Not specified	1	0.99	1	3.03
Employment
Full-time	2	2.00	2	6.06
Part-time	13	12.90	4	12.12
Unemployed	85	84.20	27	81.82
Missing	1	0.99	0	0.00
Education (years)	13.91	3.46	14.36	3.17
Highest level of education completed
High school or below	30	29.70	9	36.36
Vocational school	11	10.90	3	9.09
Some college or completed college	55	54.50	21	63.64
Graduate school	5	4.95	0	0.00
Annual personal income ($)	12,530	13,395	13,777	9,780
Annual household income ($)	24,791	24,613	23,956	22,258
Receiving government assistance	96	95.05	31	93.94
DSM diagnoses at pre treatment1
Schizophrenia spectrum disorder	48	47.52	13	39.39
Bipolar disorder2	26	25.74	6	18.18
Major depressive disorder3	24	23.76	12	36.36
Any anxiety disorder4	45	44.55	9	27.27
Obsessive compulsive disorder4	19	18.81	6	18.18
Post-traumatic stress disorder	14	13.86	8	24.24
Substance use disorder	29	28.71	11	33.33
Psychotic symptoms/features5	70	69.30	18	54.55
Sleep and circadian diagnoses at pre treatment1
Insomnia	82	81.18	28	84.85
Hypersomnolence (provisional)6	25	21.78	5	15.15
Delayed sleep phase	6	5.94	2	6.06
Advanced sleep phase	2	1.98	1	3.03
Irregular sleep-wake disorder	1	0.99	0	0.00
Restless leg syndrome	5	4.95	1	3.03
Periodic limb movements (provisional)7	5	4.95	2	6.06

Note. TranS-C = Transdiagnostic Sleep and Circadian Intervention.

¹Participants could meet diagnostic criteria for multiple problems.

²Bipolar disorder with psychotic features is listed in this category, not in the schizophrenia spectrum or psychotic disorders category.

³Depression with psychotic features is listed in this category, not in the schizophrenia spectrum or psychotic disorder category.

⁴No participants were solely diagnosed with an anxiety disorder or obsessive compulsive disorder – all also received a comorbid schizophrenia spectrum, bipolar disorder, major depressive disorder, post-traumatic stress disorder, and/or substance use disorder diagnosis.

⁵Psychotic symptoms/features includes depression with psychotic features, bipolar disorder with psychotic features, a schizophrenia spectrum or psychotic disorder diagnosis.

⁶A hypersomnolence diagnosis requires a multiple sleep latency test (American Academy of Sleep Medicine, 2014).

⁷A periodic limb movement diagnosis requires a polysomnography assessment (American Academy of Sleep Medicine, 2014).

Individuals were eligible if they met the following inclusion criteria: (a) 18 years of age or older; (b) English language fluency; (c) presence of at least one DSM-5 mental disorder for 12 months; (d) having a guaranteed bed to sleep in for the next three months; (e) receiving care for SMI at ACBHCS and consenting to regular communication between the research team and their ACBHCS psychiatrist and/or case manager; and (f) presence of one or more of the following problems, on three or more nights per week, for three months assessed via the Sleep and Circadian Problems Interview: taking 30 minutes or longer to fall asleep, waking in the middle of the night for 30 minutes or longer, obtaining less than six hours of sleep per night, obtaining nine or more hours of sleep per 24 hour period (i.e., nighttime sleep plus daytime napping), maintaining a bedtime later than 2:00am, or having more than 2.78 hours of variability in sleep-wake schedule across one week.

Individuals were excluded if they met any of the following criteria: (a) presence of an active and progressive physical illness or neurological degenerative disease and/or substance use that would make participation in the study unfeasible; (b) current serious suicide risk or homicide risk (both assessed by study staff and a case manager or psychiatrist); (c) night shift work two or more nights per week in the past three months; (d) pregnancy or breastfeeding; or (e) unable or unwilling to participate in and/or complete the pretreatment assessments.

Treatment

Treatment was delivered by nine therapists hired by the University of California, Berkeley system. Therapist demographics and training backgrounds are displayed in Table 2. The therapists traveled between the ACBHCS clinic sites to deliver treatment.

Table 2.

Therapist Demographics

Characteristic	Mean or N	SD or %
Age (years)	31.00	6.39
Female	7	77.78
Race
Asian	1	11.11
Caucasian	8	88.89
Ethnicity
Hispanic or Latino	0	0.00
Not Hispanic or Latino	9	100.00
Education (years)	18.94	2.03
Highest level of education completed
Some graduate school	4	44.44
Completed master’s degree	2	22.22
Graduate training beyond master’s degree	2	22.22
Completed doctorate	1	11.11
Area of Education/Field
Clinical Psychology	5	55.56
Social Work	3	33.33
Marriage and Family Therapy	1	1.11
Prior experience delivering CBT (years)	2.67	2.40
Prior experience delivering sleep/circadian interventions (years)	0.89	0.99
Number of participants per therapist in this studya
Provider-Rated TranS-C Checklist	11.67	7.53
Independent-Rater TranS-C Checklist	4.13	3.87

Note. N = 9.

^aSome participants (n=4) had to switch therapists due to scheduling concerns, which results in participants being counted for multiple therapists.

TranS-C (Harvey & Buysse, 2017), which was administered in eight weekly 50-minute sessions, is grounded in basic sleep science and is derived from several sources. It draws heavily from the literature indicating the efficacy of cognitive behavioral therapy for insomnia (CBT-I) for SMI (Edinger et al., 2009; Manber et al., 2008; Myers, Startup, & Freeman, 2011). It also incorporates Interpersonal and Social Rhythms Therapy (Ehlers, Frank, & Kupfer, 1988), chronotherapy (Wirz-Justice, Benedetti, & Terman, 2009), and motivational enhancement (Miller & Rollnick, 2002). TranS-C includes four cross-cutting modules featured in every session (functional analysis, education, behavior change and motivation, and goal-setting), four core modules that apply to the vast majority of participants (establishing regular sleep-wake times including learning a wind-down and wake-up routine, improving daytime functioning, correcting unhelpful sleep-related beliefs, and maintaining behavior change), and seven optional modules used less commonly, depending on the presentation (improving sleep efficiency, reducing time in bed, dealing with delayed or advanced phase, reducing sleep-related worry/vigilance, promoting compliance with CPAP/exposure therapy for claustrophobic reactions to CPAP, negotiating sleep in a complicated environment, and reducing nightmares). Core and optional modules can be delivered in any sequence and are customized to the participant based on their presentation and goals for treatment.

Measures

Provider-Rated TranS-C Checklist.

The Provider-Rated TranS-C Checklist consists of the treatment modules that comprise TranS-C. On this checklist, the core module of establishing regular sleep-wake times including learning a wind-down and wake-up routine was split into three items for clarity: establishing regular sleep-wake times, learning a wind-down routine, and learning a wake-up routine. Due to an administrative error, this checklist did not include a rating for the reducing nightmares module. Therefore, therapists could select from a list of 16 items. For a list of the items, see the note to Table 3 and Supplement 1. At the end of each treatment session, the provider is asked to select which items on the list they delivered during that specific session. There is no limit to the number of items a provider can select for each treatment session. A total of 808 checklists from 101 participants (average of 7.71 sessions per participant) were included in the analyses for this paper.

Table 3.

Interitem Correlations for the Provider-Rated TranS-C Checklist Items

Items	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
1	-
2	0.25*	-
3	0.20*	0.45*	-
4	0.10*	0.43*	0.39*	-
5	0.15*	0.30*	0.24*	0.32*	-
6	0.04	0.33*	0.23*	0.26*	0.32*	-
7	0.01	0.28*	0.25*	0.21*	0.31*	0.23*	-
8	0.02	0.20*	0.28*	0.26*	0.26*	0.19*	0.35*	-
9	−0.02	0.17*	0.19*	0.15*	0.10*	0.16*	0.18*	0.27*	-
10	−0.04	0.10*	0.06	0.16*	0.13*	0.14*	0.10*	0.12*	0.10*	-
11	0.06	0.20*	0.18*	0.26*	0.26*	0.12*	0.27*	0.31*	0.15*	0.34*	-
12	0.12*	0.11*	0.12*	0.12*	0.09*	0.13*	0.13*	0.09*	0.03	−0.01	0.09*	-
13	−0.01	0.10*	0.07*	0.15*	0.05	0.15*	0.05	0.16*	0.34*	0.07*	0.08*	0.01	-
14	0.02	0.12*	0.09*	0.08*	0.16*	0.14*	0.14*	0.23*	0.16*	0.09*	0.11*	−0.01	0.26*	-
15	0.07*	0.18*	0.16*	0.14*	0.20*	0.07*	0.04	0.23*	0.14*	0.13*	0.13*	−0.01	0.15*	0.14*	-
16	−0.04	−0.03	0.07*	0.07*	0.02	−0.04	0.01	0.05	0.13*	−0.02	0.03	0.14*	0.20*	0.06	0.01	-

Note.

^*p<0.05.

1. Functional analysis. 2. Education. 3. Motivational enhancement. 4. Goal setting. 5. Irregular sleep-wake times. 6. Difficulty winding down. 7. Difficulty waking up. 8. Daytime impairment. 9. Unhelpful beliefs about sleep. 10. Poor sleep-efficiency. 11. Too much time in bed. 12. Delayed or Advanced phase. 13. Sleep-related worry. 14. Promoting compliance with CPAP/other treatments for comorbid sleep problems. 15. Negotiating sleep in a complicated environment (e.g., group home). 16. Maintenance of behavior change.

Independent-Rater TranS-C Checklist.

The Independent-Rater TranS-C Checklist contains 19 items. Three of these items are not listed on the Provider-Rated TranS-C Checklist or are not TranS-C modules, thus were not included in the analyses. The first two were progress monitoring and encouraging the patient to follow up on referrals for a sleep apnea assessment, which are not TranS-C modules. The third was reducing nightmares, which was left off of the Provider-Rated TranS-C Checklist due to an administrative error. In parallel to the TranS-C Provider-Rated Checklist, the core module of establishing regular sleep-wake times including learning a wind-down and wake-up routine was split into three items for clarity: establishing regular sleep-wake times, learning a wind-down routine, and learning a wake-up routine. Therefore, the Independent-Rater TranS-C Checklist contains 19 items, including all 15 of the TranS-C modules. Trained independent-raters listened to audio recordings of treatment sessions and made four ratings for each of the 19 items. First, they selected the treatment element they observed to be present. Second, they recorded the duration or the time (rounded to the nearest minute) that the therapist spent on each element. Third, they rated the quality of delivery of each element on a three-point scale (1=excellent, 2=adequate, 3=inadequate). Fourth, they rated the participant’s response on a four-point scale (1=participant fully understood and embraced the module, 2=participant somewhat understood and embraced the module, 3=participant did not understand nor embrace the module, 4=unknown). These domains follow the conceptualization of fidelity offered by Allen et al. (2012) and Schoenwald et al. (2011). A scoring rule was developed which allowed raters to select multiple optional or cross-cutting modules, but required that raters select only one core module. This arose because therapists rarely selected multiple core modules at one time and because it facilitated achieving interrater reliability. For the present study, we only used the first of the four ratings that observers made – scores indicating which treatment elements observers rated as present or absent, allowing us to measure concordance with the Provider-Rated TranS-C Checklist. We randomly selected participant sessions to observe using a two-step approach. First, participants had to have completed treatment. Second, participants had to have providers who also completed the Provider-Rated TranS-C Checklists. A total of 257 session checklists from 33 participants who received treatment from eight therapists were included in this sample (average of 7.79 sessions per participant). Demographic characteristics of this subsample are displayed in Table 1.

Three independent raters were trained to code treatment session tapes. Training was conducted using a multistep approach. First, the raters were introduced to the rating scale and each independently coded a tape to compare their ratings with those of an expert rater who was also an experienced TranS-C therapist (NBG). Second, the three raters coded one tape together as a group. Third, each rater independently coded three tapes out of five randomly selected tapes (one tape overlapped with all three raters and the other four tapes overlapped with two raters). Fourth, all raters listened to these same five tapes together and discussed their ratings until consensus was reached. Last, each rater coded five more tapes out of seven randomly selected tapes (one tape overlapped with all three raters and the other six tapes overlapped with two raters). Throughout the training process, codes were compared and disagreements were discussed until consensus was achieved before progressing to the next step. Inter-rater reliability for these final seven tapes was 87.91%.

Procedure

All procedures were approved by the University of California, Berkeley, Committee for the Protection of Human Subjects. All participants provided informed consent. Participants completed a pre-treatment assessment, which included the Mini-International Neuropsychiatric Interview (Sheehan et al., 1997) to establish SMI diagnoses as well as the Duke Structured Interview for Sleep Disorders (Edinger, Bonnet, et al., 2004; Edinger, Kirby, et al., 2004) plus 1- week of sleep diary collection (Carney et al., 2012) to establish provisional sleep and circadian diagnoses. They were then randomly assigned to receive TranS-C immediately, or to delayed-treatment – wherein they were on a waitlist for eight months and then received TranS-C. Immediately following treatment and again at 6-month follow-up, participants completed outcome measures. See Supplement 2 for a flow chart illustrating the study design. Therapists completed the Provider-Rated TranS-C Checklist after each treatment session during the TranS-C treatment phase.

Data Analysis

Internal consistency of the Provider-Rated TranS-C Checklist was assessed by examining interitem Spearman’s correlations for individual Provider-Rated TranS-C Checklist ratings and Cronbach’s alpha. The factor structure of the Provider-Rated TranS-C Checklist was established using exploratory factor analysis (EFA) across the sample of completed checklists. As the data is nested within therapists, we first examined the intraclass correlation coefficients (ICCs) to determine if a standard EFA or a multilevel EFA was more appropriate. As the 16 items had low to high ICCs (range = 0.04–0.27) and there are no definitive guidelines on when to use a multilevel EFA (Reise, Ventura, Nuechterlein, & Kim, 2005), we elected to run the simpler model – a regular EFA (Pornprasertmanit, Lee, & Preacher, 2014). The suitability of the data for EFA was assessed using the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (Kaiser, 1974) and Bartlett’s test of sphericity (Bartlett, 1950). Principal axis extraction was used to explore latent factors in the set of 16 items on the Provider-Rated TranS-C Checklist. Common factor analysis was selected over principal components analysis as it has the advantage of accounting for measurement error in producing a factor solution (Widaman, 1993).

Multiple criteria were used to determine the number of factors to retain based on the recommendations of Henson and Roberts (2006). First, the Provider-Rated TranS-C Checklist ratings were assessed using Kaiser’s (1960) criterion, which involves retaining factors with observed eigenvalues greater than one. Second, the Provider-Rated TranS-C Checklist ratings were examined using Cattell’s (1966) scree test, which involves evaluating a visual representation of eigenvalues ordered from largest to smallest and retaining only those factors that fall above a steep drop off point. Third, the Provider-Rated TranS-C Checklist ratings were evaluated using Horn’s (1965) parallel analysis, which involves retaining factors with observed eigenvalues exceeding specified parameters of expected eigenvalues (based on a simulated distribution of eigenvalues from randomly generated samples). Horn’s parallel analysis was conducted in SPSS Version 22 (O’Connor, 2000). Based on the recommendations of Glorfeld (1995), factors with observed eigenvalues that exceeded the 95^th percentile of randomly generated eigenvalues were retained.

Convergent validity was assessed via Spearman’s correlations between Provider-Rated TranS-C Checklist scores and Independent-Rater TranS-C Checklist scores. We evaluated the concordance between the presence or absence of each item on the measures as well as the total number of items delivered on both measures. Means and standard deviations were calculated to identify the modules most commonly delivered on both the Provider-Rated TranS-C Checklist and the Independent-Rater TranS-C Checklist.

Results

Internal Consistency and Factor Structure

Internal consistency.

Interitem correlations of Provider-Rated TranS-C Checklist scores are presented in Table 3. The mean interitem correlation was ρ = 0.16, within Clark and Watson’s (1995) ideal range of 0.15–0.50 for mean interitem correlations. The internal consistency estimate for the Provider-Rated TranS-C Checklist was acceptable (Cronbach’s α = 0.74), according to Nunnally’s (1978) guidelines.

Exploratory factor analysis.

The 808 completed Provider-Rated TranS-C Checklists were used for EFA. The KMO index was 0.78, indicating middling sampling adequacy according to Kaiser’s (1974) guidelines. Bartlett’s test of sphericity was significant, χ²(120) = 1836.33, p < .001, indicating that the correlation matrix is factorable based on Bartlett’s (1950) guidelines. As displayed in Table 4, initial communalities for the items were in the low range: 0.07–0.37.

Table 4.

Initial Communalities and Factor Loadings for the Factor Analysis, Correlations between the Providerr-Rated Trans-C Checklist and Independent-Rater TranS-C Checklist, and Means and Standard Deviations for Items Delivered on the Checklists

	Initial communalities and one- factor solution factor loadings for the exploratory factor analysis of the Provider-Rated TranS-C Checklist items		Correlations between Items on the Provider-Rated TranS-C Checklist and Items on the Independent-Rater TranS-C Checklist		Means and Standard Deviations for TranS-C Items Delivered Based on the Provider-Rated TranS-C Checklist and the Independent-Rater TranS-C Checklista
Item	Initial Communalities	Factor loadings for the one factor solution	ρ	p	Provider-Rated TranS-C Checklist (N=101)	Independent-Rater TranS-C Checklist (n=33)
1. Functional analysis	0.11	0.18	0.10	0.12	2.41(1.71)	6.88(1.67)
2. Education	0.37	0.62	0.44	0.00***	4.19(2.41)	3.84(1.58)
3. Motivation enhancement	0.31	0.55	0.13	0.04*	2.86(2.47)	6.27(1.48)
4. Goal setting	0.30	0.57	0.28	0.00***	3.84(2.52)	1.48(0.94)
5. Irregular sleep-wake times	0.26	0.51	0.18	0.00**	2.07(2.05)	2.45(1.75)
6. Difficulty winding down	0.22	0.44	0.39	0.00***	1.93(1.63)	1.84(1.30)
7. Difficulty waking up	0.25	0.53	0.54	0.00***	1.55(1.49)	1.39(1.06)
8. Daytime impairment	0.27	0.53	0.37	0.00***	1.50(1.95)	1.42(1.37)
9. Unhelpful beliefs about sleep	0.19	0.37	0.46	0.00***	1.68(1.64)	1.06(1.17)
10. Poor sleep efficiency	0.14	0.29	0.20	0.00**	0.57(1.16)	1.58(1.25)
11. Too much time in bed	0.24	0.48	0.08	0.20	1.15(1.68)	0.18(0.73)
12. Delayed or advanced phase	0.07	0.17	0.15	0.02*	0.30(0.73)	0.36(0.65)
13. Sleep-related worry	0.20	0.30	0.35	0.00***	1.52(1.84)	2.54(1.77)
14. Promoting compliance with CPAP/other treatments for comorbid sleep problems	0.13	0.29	0.71	0.00***	0.34(1.31)	0.06(0.24)
15. Negotiating sleep in a complicated environment	0.12	0.30	0.11	0.07	0.90(1.90)	0.21(0.54)
16. Maintenance of behavior change	0.09	0.10	0.52	0.00***	1.59(1.55)	2.12(0.93)
Total number of items delivered	--	--	0.41	0.00***	--	--

Note.

^*p<0.05,

^**p<0.01,

^***p<0.001.

The Provider-Rated TranS-C Checklist does not include the following items that were included on the Independent-Rater TranS-C Checklist: progress monitoring, encouraging the patient to follow up on referrals for an assessment for sleep apnea, and reducing nightmares.

^aMean(standard deviation) presented.

Using Kaiser’s (1960) criterion, five factors had an eigenvalue greater than one, indicating a five-factor solution. Based on Cattell’s (1966) scree test, one factor had an eigenvalue above a visually determined drop-off point, indicating a one-factor solution. According to Horn’s (1965) parallel analysis, one factor had an observed eigenvalue exceeding its corresponding expected eigenvalue, again indicating a single-factor solution. Based on the evidence that Kaiser’s criterion has a tendency to overestimate the number of factors (Zwick & Velicer, 1986), and as parallel analysis has been demonstrated to be more accurate than Kaiser’s criterion and other commonly used criteria (Ledesma & Valero-Mora, 2007), a one-factor solution was extracted.

Tabachnick and Fidell (2001) recommend that coefficients of items on factors should be 0.32 or higher. As displayed in Table 4, factor loadings were above this threshold for nine out of the sixteen items and slightly below this threshold for an additional four items. The one-factor solution explained 21.45% of the variance.

Convergent Validity

Correlations between the Provider-Rated TranS-C Checklist items and the total number of items delivered and the Independent-Rater TranS-C Checklist are presented in Table 4. As evident, 13 of the 16 items were significantly correlated with each other on both of these measures. The total number of items delivered at each session on the Provider-Reported TranS-C Checklist was significantly and meaningfully associated with the total number of items rated on the Independent-Rater TranS-C Checklist (ρ = 0.41, p < .01).

Modules Delivered

Table 4 displays the means and standard deviations for the number of times each item was delivered according to the Provider-Rated TranS-C Checklist and the Independent-Rater TranS-C Checklist. We first describe the results from the Provider-Rated TranS-C Checklist. The education module and goal setting module were delivered the most frequently during treatment, occurring during on average 4.19 sessions and 3.84 sessions out of eight sessions, respectively, for each participant. Addressing delayed or advanced phase and promoting compliance with CPAP/other sleep treatments were delivered the least frequently of all the treatment modules, as they were delivered during on average 0.30 and 0.34 sessions per participant, respectively.

We found similar results for the frequency of modules delivered for the Independent-Rater TranS-C Checklist. Functional analysis and motivational enhancement were rated as delivered the most frequently, occurring on average during 6.88 sessions and 6.27 sessions out of eight sessions, respectively, for each participant. Promoting compliance with CPAP/other sleep treatments and addressing too much time in bed were delivered the least frequently of all the treatment modules, as they were delivered during on average 0.06 and 0.18 sessions per participant, respectively.

Discussion

The overarching goal of this study was to assess the psychometric properties of the Provider-Reported TranS-C Checklist. The first aim was to assess the internal consistency and structural validity. The results of the EFA indicated a single factor solution, which is consistent with the theory that the TranS-C modules address the common core of sleep health (Buysse, 2014; Harvey & Buysse, 2017). The single factor solution explained 21.45% of the total variance. The 16 Provider-Rated TranS-C Checklist items were intercorrelated with one another and demonstrated acceptable internal consistency. There was no a priori hypothesis regarding latent factors for the Provider-Rated TranS-C Checklist, which influenced the decision to run an EFA rather than a confirmatory factor analysis.

The second aim was to establish the convergent validity of the Provider-Rated TranS-C Checklist. As expected, most of the items on the Provider-Rated TranS-C Checklist, as well as the total number of items, were significantly and positively correlated with the Independent-Rater TranS-C Checklist. The three items that were not correlated were: functional analysis, too much time in bed, and negotiating sleep in a complicated environment. Upon inspecting the data, it appears that functional analysis was scored as present in almost every single treatment session by the Independent-Rater TranS-C Checklist, whereas it was infrequently scored on the Provider-Rated TranS-C Checklist. It is encouraging that therapists are engaging in more functional analysis than they recognize. Alternatively, therapists were encouraged to adapt treatment and module selection based on data from progress monitoring via the daily sleep diaries. The treatment manual also includes a more formal, structured functional analysis process with an accompanying worksheet that is completed primarily during the first couple of treatment sessions, which connects behaviors, thoughts, and feelings around sleep. It is possible that therapists considered only this structured functional analysis activity when selecting the functional analysis item on the checklist and neglected to identify other types of functional analysis commonly used in TranS-C. Future provider trainings should highlight the different types of functional analysis that can occur throughout treatment. Both too much time in bed and negotiating sleep in a complicated environment were rated less frequently on the Independent-Rater TranS-C Checklist (rated as present in only six and seven total sessions, respectively) compared to the Provider-Rated TranS-C Checklist (rated as present in 39 and 44 total sessions, respectively). We offer three possible explanations. First, these two items may be more difficult to recognize by an independent observer. Second, therapists were not given restrictions on how many items they could select on the Provider-Rated TranS-C Checklist for any given treatment session. However, the independent raters developed a scoring rule that only one core module, but more than one optional or cross-cutting module, could be selected for any given portion of a session recording (explained in the “Independent-Rater TranS-C Checklist” section). Third, perhaps therapists over-rated their delivery of these two modules (Carroll et al., 2000, 1998; Martino et al., 2009).

The correlations, ranging from weak to strong, between the Provider-Rated TranS-C Checklist and the Independent-Rater TranS-C Checklist are promising and consistent with prior studies that demonstrated that therapist-reported fidelity was accurate when compared with independent rater-reported fidelity (e.g., Breitenstein, Fogg, et al., 2010; Hogue et al., 2017, 2015). Therapists over-rated the presence of only two of the items compared to the independent-raters, which is consistent with prior research (Hurlburt et al., 2010; Martino et al., 2009). Overall, the correlations demonstrate convergent validity for the Provider-Rated TranS-C Checklist and also provide evidence for the utility and accuracy of self-report fidelity measures for clinicians treating adults with SMI.

The third aim was to describe the TranS-C modules most commonly delivered according to both the Provider-Rated TranS-C Checklist and the Independent-Rater TranS-C Checklist. Taking the Provider-Rated TranS-C first, education, goal setting, and motivational enhancement were the most frequently delivered modules. This is not surprising as these are cross-cutting modules and thus are designed to be provided in each session (Harvey & Buysse, 2017). The modules that promote compliance with CPAP/other sleep treatment and address delayed or advanced phase modules were delivered the least frequently. In terms of the former, our research team recommended that 51 participants seek further assessments for sleep apnea. However, many (approximately 47) did not do so while in treatment. Anecdotally, it often took longer than eight weeks to schedule an overnight clinical assessment for sleep apnea and to have the results evaluated. Therefore, while sleep apnea is common in adults with SMI (Stubbs et al., 2016), promoting compliance with CPAP was not the most immediate treatment target for many participants. In terms of the low frequency of the module for addressing delayed or advanced phase, while many participants exhibited a tendency toward a delayed or advanced phase, only six (5.94%) met criteria for delayed phase sleep type and two (1.98%) met criteria for advanced sleep phase type upon entering the study. On the one hand, addressing the delayed or advanced phase module may not have been indicated for the majority of participants. On the other hand, the more straightforward irregular sleep-wake times module often assisted these participants. While two of the modules that are drawn from CBT-I —unhelpful beliefs about sleep and reducing sleep-related worry—were delivered more frequently, another module also derived from CBT-I —stimulus control and sleep restriction to improve poor sleep efficiency —was delivered less frequently. We hypothesize that this may be due to the challenging circumstances faced by participants in this study. Many were experiencing instability in their SMI symptoms such that the sleep deprivation that is often experienced as part of stimulus control and sleep restriction was contraindicated. Many were living in supportive housing environments, which involve sharing a room with others whom they did not wish to disturb during stimulus control. Anecdotally, several participants reported finding the stimulus control and sleep restriction instructions somewhat difficult to understand and follow.

Moving to the Independent-Rater TranS-C Checklist, functional analysis, motivational enhancement, and reducing sleep-related worry were rated as delivered the most frequently. Functional analysis and motivational enhancement are cross-cutting modules (Harvey & Buysse, 2017), so their frequent delivery is not surprising. Also, as already discussed, providers may be under-rating their use of functional analysis. In addition, the sleep-related worry module is drawn from CBT-I and appears to be effective in reducing insomnia symptoms (Harvey et al., 2014; Harvey, Sharpley, Ree, Stinson, & Clark, 2007).

This study had several limitations. First, the psychometric properties of the Provider-Rated TranS-C Checklist were assessed in a sample of adults with SMI. Future research should examine these properties with different patient populations in order to increase generalizability. Second, the treatment providers were hired, trained, and supervised within a university setting. Future studies are needed with community clinicians in order to determine if these findings replicate (Hogue et al., 2014; Schoenwald, 2011). Third, the reducing nightmares module was not included on the TranS-C Provider-Rated Checklist due to an administrative error. Fourth, the Provider-Rated TranS-C Checklist only assessed the presence or absence of a TranS-C module in a given session as our goal was to develop a brief and efficient measure for the future implementation of TranS-C in routine practice settings. Future studies may consider using a Likert scale to assess provider-rated quality. Fifth, the TranS-C Provider-Rated Checklist was developed based on the treatment manual (Harvey & Buysse, 2017). Providers did not contribute to the content or clarity of the measure. Future research should utilize techniques such as cognitive interviewing to help improve item content (Chinman et al., 2016). Sixth, this study was not designed to assess predictive validity. Future studies should collect outcome data at each treatment session in order to examine if the delivery of each TranS-C module is associated with an improvement in sleep and circadian functioning or SMI symptoms. Future implementation trials could also consider creating a score of the percentage of total TranS-C modules delivered based on the Provider-Rated TranS-C ratings in order to predict treatment outcome. Seventh, this study did not evaluate discriminant validity. Future studies should include a measure to facilitate this analysis. Eighth, this study used a regular EFA rather than a multilevel EFA, which does not account for nesting of Provider-Reported TranS-C Checklists among therapists. However, several of the ICCs for the items on the checklist were low and there are no definitive guidelines for when to use a multilevel EFA (Reise et al., 2005). Ninth, the sleep and circadian diagnoses were made using the Duke Structured Interview for Sleep Disorders (Edinger, Kirby, et al., 2004) and supplemented with an evaluation of a seven-day sleep diary (Carney et al., 2012). However, these are tentative as additional assessments, such as polysomnography and multiple sleep latency tests, are necessary to confirm the presence of periodic limb movements and hypersomnolence. Finally, the present study only examined one component of fidelity: treatment adherence. Future research should examine therapist competence and treatment differentiation for TranS-C (Schoenwald et al., 2011).

In conclusion, these results provide preliminary data suggesting that the Provider-Rated TranS-C Checklist is a valid measure of provider fidelity for TranS-C. This provider-report measure is an efficient tool to monitor provider treatment adherence to TranS-C and is an important step in preparing for the implementation of TranS-C in routine practice settings. Establishing this benchmark of adherence will allow for the evaluation of treatment adherence once TranS-C is deployed into routine care (Hogue et al., 2017; McLeod et al., 2019). The current study is one of few to focus on treatment of adults with SMI and to include validity data. Overall, this study contributes to the burgeoning literature on self-report fidelity measures.

Highlights.

• Creating efficient, valid measures of treatment fidelity is essential

• A provider-report fidelity measure for a sleep/circadian intervention is proposed

• A unidimensional scale is extracted using exploratory factor analysis

• The measure demonstrates convergent validity