PSYCHOMETRIC EVALUATION OF A MULTI-DIMENSIONAL MEASURE OF SATISFACTION WITH BEHAVIORAL INTERVENTIONS

Abstract

Treatment satisfaction is recognized as an essential aspect in the evaluation of an intervention’s effectiveness, but there is no measure that provides for its comprehensive assessment with regard to behavioral interventions. Informed by a conceptualization generated from a literature review, we developed a measure that covers several domains of satisfaction with behavioral interventions. In this paper, we briefly review its conceptualization and describe the Multi-Dimensional Treatment Satisfaction Measure (MDTSM) subscales. Satisfaction refers to the appraisal of the treatment’s process and outcome attributes. The MDTSM has 11 subscales assessing treatment process and outcome attributes: treatment components’ suitability and utility, attitude toward treatment, desire for continued treatment use, therapist competence and interpersonal style, format and dose, perceived benefits of the health problem and everyday functioning, discomfort, and attribution of outcomes to treatment. The MDTSM was completed by persons (N = 213) in the intervention group in a large trial of a multi-component behavioral intervention for insomnia within 1 week following treatment completion. The MDTSM’s subscales demonstrated internal consistency reliability (α: .65 – .93) and validity (correlated with self-reported adherence and perceived insomnia severity at post-test). The MDTSM subscales can be used to assess satisfaction with behavioral interventions and point to aspects of treatments that are viewed favorably or unfavorably.

With the growing advocacy for engaging patients in the design of treatments and for patient-centered care, satisfaction with treatment is now recognized as an important indicator of care quality (Kendra et al., 2015; Smith et al., 2013) and an essential aspect in the evaluation of treatments’ or interventions’ effectiveness (Hundt et al., 2013; Rose, Wykes, Farrier, Doran, Sporle & Bogner, 2008). Treatment satisfaction reflects persons’ valuation of different aspects and/or attributes of the treatment, resulting from their experience with the treatment (Gold et al., 2011; Kucukarslan et al., 2013; Revicki, 2004). Such feedback is informative; it reinforces the acceptability and usefulness of aspects of the treatment and points to those that could be modified to enhance their attractiveness to potential users (Alessi & Rash, 2017; Brod et al., 2007; Schulte et al., 2011). Treatment satisfaction is also important because of its well-established association with treatment adherence and outcomes. Evidence supports this association in studies of a range of health problems, such as pain (e.g., George & Robinson, 2010), anxiety (e.g., Smith et al., 2013), smoking cessation (e.g., Alessi & Rash, 2017), and substance misuse (e.g., Schulte et al., 2011). In this paper, we describe and report on the psychometric properties of a self-report instrument measuring satisfaction with behavioral interventions.

Despite its acknowledged importance and the increasing number of studies of satisfaction with a variety of pharmacological and non-pharmacological treatments (e.g., Barbosa et al., 2012; Schulte et al., 2011), there is no clear and unifying definition of treatment satisfaction (Barbosa et al., 2012; Speight, 2005). Variability in theoretical perspectives may have contributed to the development of a range of self-report instruments measuring treatment satisfaction. The instruments consist of either single items measuring global treatment satisfaction (e.g., Fontana et al., 2006; Waltz et al., 2014) or multiple items assessing the perceived helpfulness of each treatment component (e.g., Alessi & Rash, 2017; Chang et al., 2004; Jennings, Vandelanotte, Caperchione & Mummery, 2014; McGregor, Coghlan & Dennis, 2014; Smith et al., 2013) and treatment processes such as group dynamics or climate (e.g., Levenson, Macgowan, Morin & Cotter, 2009; Reay, Fisher, Robertson, Adams & Owen, 2006), the therapists’ competence (e.g., Openshaw et al., 2012), and the relationship between therapists and patients (e.g., Oei & Green, 2008; Tierney & Kane, 2011). Other measures incorporate items assessing patients’ values or perceptions of different treatment attributes, which often include their benefits, discomfort or side effects, and convenience of use (e.g., Frick et al., 2012; Gamble et al., 2013; Gold et al., 2011; Kucukarslan et al., 2013; Rejas et al., 2013; Umar et al., 2013; Wong et al., 2015; Wyman et al., 2010).

The Treatment Satisfaction Questionnaire for Medication and the Client Satisfaction Questionnaire are two measures of treatment satisfaction that have demonstrated good psychometric properties. The Treatment Satisfaction Questionnaire for Medication was specifically designed to examine patients’ valuation of different attributes of medications that are known to influence treatment decision making (Kucukarstand et al., 2013), thereby limiting its relevance for measuring satisfaction with behavioral interventions. The Client Satisfaction Questionnaire (Attkisson & Greenfield, 2004) has been used to assess satisfaction with behavioral interventions, but its eight items do not provide for a comprehensive evaluation of all components and attributes of these interventions.

Informed by the comprehensive conceptualization and operationalization of satisfaction presented by Sidani and Epstein (2016), we developed a multi-dimensional, self-report instrument to measure satisfaction with behavioral interventions. The Multi-Dimensional Treatment Satisfaction Measure (MDSTM) consists of eleven subscales for appraising an intervention’s process and outcome attributes. The content of the measure is consistent with recent perspectives that satisfaction is multi-dimensional (Barbosa et al., 2012; Kendra et al., 2015), covering attributes of a treatment associated with its delivery and outcomes (Gold et al., 2011; Smith et al., 2013). Data obtained from a large trial of a multi-component intervention (MCI) that demonstrated efficacy for managing insomnia were analyzed to determine the psychometric properties of the MDSTM subscales. In this paper, we briefly review the conceptualization of satisfaction. We then describe the MDSTM subscales and report the results of their psychometric testing. The MDSTM subscales are relevant for measuring satisfaction with different behavioral interventions, and they were used in our study to examine satisfaction with a behavioral intervention for insomnia.

Conceptualization of Satisfaction with Treatment

Based on a critical review and synthesis of the conceptual and empirical literature, Sidani and Epstein (2016) defined satisfaction as the subjective appraisal of the process and outcome attributes of an intervention that individuals make after having received or experienced it. The process attributes relate to the intervention’s components, overall quality, and implementation.

Each intervention component is rated for its suitability and utility. Suitability refers to the appropriateness of the component in addressing the health problem and the convenience or ease of carrying out its treatment recommendations. Utility is the helpfulness of the component in managing the health problem. Overall quality refers to the attitude toward the intervention (inclusive of all the intervention components) and the desire to continue using it.

Implementation of the intervention denotes the therapists’ competence and interpersonal style, and usefulness of its format and dose. Therapists’ competence has to do with their knowledge and skills in delivering the intervention, whereas their interpersonal style reflects their ability to communicate clearly, to show warmth, and to be supportive. Usefulness of the intervention’s format and dose of implementation refers to their contribution to promoting understanding and application of the treatment recommendations.

Four treatment outcome attributes are proposed. The first two are the perceived benefits of the overall intervention in resolving the health problem and improving everyday functioning. The third attribute is the discomfort or side effects associated with its use. The fourth refers to the extent to which the person accredits the improved outcomes they experience to the intervention.

Multi-Dimensional Treatment Satisfaction Measure

The conceptualization of satisfaction with treatment informed the generation of the MDSTM subscales and items. A systematic process was followed in generating the items comprising each of the 11 subscales, thereby maintaining each subscale’s content validity.

The process began by developing a matrix that identified the specific attribute and provided its conceptual definition, as proposed in Sidani and Epstein’s (2016) conceptualization. The matrix also listed items derived from available instruments reported to measure the respective attribute. The content covered by all items identified as measuring the respective attribute was examined and analyzed to determine and describe the indicators that were consistently used to operationalize the attribute. The description of each indicator was synthesized to generate the content of the respective item.

Items reflecting indicators of the same attribute formed a subscale, and each subscale was subsequently labeled to identify the attribute that the items are intended to measure. Relevant empirical literature (e.g., systematic reviews) was reviewed to identify the outcomes expected of the behavioral intervention for insomnia evaluated in our study.

Sections and Subscales of the MDTSM

The MDTSM has two sections, and its content can be tailored to the features characterizing the behavioral intervention under evaluation. In the following, we explain what the sections contain and provide examples of their application in our study that evaluated the multi-component intervention (MCI) for insomnia. The first section identifies all components comprising the intervention, and each component is described. For example, in our study, the MCI for insomnia included three components, described in a later section: sleep education and hygiene, stimulus control therapy, and sleep restriction therapy. Each component is briefly described, using simple, easy to understand terms. For example, in our study, the stimulus control therapy component was described as “instructions on activities you must do or avoid around bedtime and on specific changes you must make in your sleep environment.” This information serves as a reference in clarifying the intervention and its components when they are being appraised.

The second section of the MDTSM contains the 11 subscales, with items assessing satisfaction with the treatment process and outcome attributes. The subscales and respective items, as applied in our study with persons with insomnia who received the MCI, are presented in Table 1. The response options include descriptors that are consistent with the items’ content, in a five-point rating scale ranging from not at all (0) to very much (4) for all items. For instance, the descriptor “logical” is used in the first item, with the rating scale: not at all logical (0), somewhat logical (1), logical (2), very logical (3), and very much logical (4).

Table 1.

Attributes, subscales and items of the Multi-Dimensional Treatment Satisfaction Measure

Domain	Attribute	Subscale	Items
Process	Treatment Components	Suitability	1. Logical / reasonable 2. Easy to use 3. Appropriate for sleep problem
		Utility	1. Useful in increasing understanding of sleep and sleep problem (i.e., insomnia) 2. Useful in increasing knowledge about ways to manage insomnia 3. Useful in increasing confidence in using recommendations to manage insomnia 4. Helpful in managing insomnia
	Overall Treatment	Attitude toward treatment	1. Personally like the treatment 2. Quality of the treatment
		Desire for continued treatment use	1. Willingness to undergo the treatment again in the future if have a serious problem with insomnia 2. Likelihood to recommend the treatment to other persons with insomnia
	Implementation	Therapist Competence	1. Understand sleep problem 2. Adequate knowledge of the nature and treatment of sleep problem 3. Competence to treat sleep problem
		Therapist Interpersonal style	1. Warmth 2. Support to person and the group members 3. Clear explanation of the treatment
		Format and dose	Booklet 1. Length is reasonable 2. Easy to understand information 3. Helpful in providing knowledge about sleep and techniques to improve sleep 4. Effective in helping learn how to practice techniques to improve sleep
			Group 1. Length of each session is reasonable 2. Total number of session is reasonable 3. Easy to understand treatment 4. Helpful in providing knowledge about sleep and techniques to improve sleep 5. Effective in helping learn how to practice techniques to improve sleep
			Telephone 1. Length of each session is reasonable 2. Total number of session is reasonable 3. Easy to understand treatment 4. Helpful in providing knowledge about sleep and techniques to improve sleep 5. Effective in helping learn how to practice techniques to improve sleep
Outcome	Perceived benefits	Perceived benefits-health problem (i.e., insomnia)	1. Time to fall asleep 2. Number of awakenings during the night 3. Amount of time spent awake during the night 4. Waking up too early 5. Overall quality of sleep 6. Total sleep time 7. Daytime well-being 8. Concentration during the day
		Perceived benefits-everyday functioning	1. Leisure activities 2. Family activities 3. Job-related activities 4. Social activities
	Discomfort	Discomfort	1. Experience of discomfort 2. If yes, please describe discomfort
	Attribution of outcomes to treatment	Attribution of outcomes to treatment	1. Successful in terms of improving insomnia 2. Effectiveness of specific techniques taught in improving sleep 3. Benefit of practicing techniques at home in improving sleep

Subscales assessing process attributes invite participants to rate each intervention component for its suitability and utility, prior to (and hence preparing participants for) evaluating the quality and implementation of the overall intervention. The Suitability subscale has three items inquiring about the extent to which the component is logical and appropriate to address the health problem. The Utility subscale has four items assessing the extent to which the component is useful in helping persons understand and manage the health problem.

Appraisal of the overall intervention is then made using two subscales: attitude and desire for continued use. The Attitude subscale has two items: one assesses the extent to which the participant liked the treatment and one assesses the overall quality of the intervention. The Desire for Continued Treatment Use subscale contains two items related to willingness to undergo the same treatment in the future if needed and likelihood of recommending it to other persons with the health problem.

The implementation of the intervention involves rating the therapist and format and dose with which it is given. The Therapist Competence subscale contains three items asking persons to rate the therapist’s knowledge about the health problem and the intervention, and skills in delivering it. The Therapist Interpersonal Style subscale contains three items asking persons to judge the therapist’s warmth, support and clarity of communication. The Format and Dose subscale asks persons to appraise the extent to which the format and dose used for providing the intervention facilitate their understanding of its content and their application of the treatment recommendations.

In our study, the MCI components were given in three formats: a booklet summarizing the sleep education and hygiene recommendations, four face-to-face group sessions for discussing the treatment recommendations and strategies to overcome barriers to their application, and two individual telephone sessions initiated by the therapist to promote continued application of the treatment recommendations. As shown in Table 1, items were developed to appraise the format and dose of the booklet (4 items), face-to-face group (5 items), and individual telephone (5 items) sessions for providing the MCI.

The treatment outcome attributes are assessed with the following subscales. First, the Perceived Benefits - Health Problem subscale includes items measuring perception of the symptoms or indicators of the health problem. In our study and as shown in Table 1, eight items measuring symptoms of insomnia comprised this MDTSM subscale. Second, the Perceived Benefits - Functioning subscale consists of items measuring performance of everyday activities. In our study, this subscale included four items related to engagement in leisure, family, job and social activities (Table 1). Third, the Discomfort subscale has two items related to the experience of discomforts or side effects with the intervention. One closed-ended item inquires about whether or not (response options: yes and no) any discomfort was experienced. The other item is an open-ended question, inviting participants to describe, in their own words, the type of discomfort experienced. Fourth, the Attribution of Outcomes to Treatment subscale has items for rating the extent to which participants attribute the outcomes (i.e., change in health problem and functioning) to the intervention. In our study, this subscale included three items (Table 1).

Our study aimed to evaluate the psychometric properties of the subscales comprising the MDTSM by examining their internal consistency reliability and construct validity. Construct validity was determined by examining if the items comprising a subscale loaded on one factor reflecting the respective treatment attribute and investigating the association of the MDTSM subscales with two conceptually related variables - treatment adherence and treatment outcome (e.g. Peyrot & Rubin, 2009; Rejas et al., 2013; Schulte et al., 2011; Tierney & Kane, 2011). Specifically, it was hypothesized that satisfaction with treatment process attributes would be more highly correlated with adherence, whereas satisfaction with treatment outcome attributes would be more highly associated with the outcome (i.e. perceived insomnia severity) measured at post-test, as implied by recent findings (Barbosa et al., 2012; Fontana et al., 2006; George & Robinson, 2010; Kendra et al., 2015; Smith et al., 2013; Wong et al., 2015).

Methods

Design

Our study was part of a large trial to evaluate the influence of treatment preferences on outcomes and the effectiveness of behavioral interventions for chronic insomnia. The trial protocol and findings pertaining to the primary aims have been reported elsewhere (Sidani et al., 2015a; 2015b). The study protocol was approved by the Research Ethics Board at participating institutions, and all participants provided written consent.

In the last week of treatment, participants were mailed a package that contained the MDTSM and the measures of treatment adherence (i.e., overall compliance with the MCI treatment recommendations) and primary outcome (i.e., perceived insomnia severity). Participants were instructed to complete the measures and return them in a postage-paid envelope, within 1 week of receiving the package. The research assistant made a telephone call 1 week following the mailing as a reminder to return the completed measures.

Sample

Persons with insomnia were eligible if they were non-institutionalized adults; 21 years of age or older; able to read and write English; complained of difficulty falling and/or staying asleep of 30 minutes or more per night on at least 3 nights per week, for at least 3 months, ascertained with relevant items on the Insomnia Interview Schedule (Morin, 1993); had been assigned to the MCI; and had completed the measures at post-test, regardless of whether or not they had attended all planned intervention sessions. Those with cognitive impairment indicated by a score < 27 on the Mini-Mental State Exam (Folstein, Folstein & McHugh, 1975), and those reporting a current diagnosis of sleep apnea, were excluded.

Of the 258 participants allocated to the MCI, 213 (82.5%) completed all measures at post-test and were included in the analysis. Participants were excluded if they dropped out of the study (n = 35) or had missing data on the adherence and outcome measures (n = 10). Applying the recommendation of having 10 cases per item for testing the internal consistency of a measure and examining the subscale items’ loadings on the respective factor (Streiner & Norman, 2014), this sample size was considered adequate for conducting these analyses separately for each subscale comprising the MDTSM. The number of items in the subscales ranged from 2 to 8, requiring a sample of 20 to 80. The accrued number of participants (N = 213) also was appropriate to detect small to moderate bivariate correlations between the MDTSM subscales and treatment adherence and outcomes (Cohen, 1992).

Multi-Component Intervention for Insomnia

The MCI consisted of three components: sleep education and hygiene (SEH), stimulus control therapy (SCT) and sleep restriction therapy (SRT). The MCI is effective in managing insomnia in different patient populations (e.g., Johnson et al., 2016; Trauer, Qian, Doyle, Rajaratnam & Cunnington, 2015). The SEH is foundational for understanding sleep, insomnia, and the rationale for behavioral therapies (Bootzin & Epstein, 2011). The SEH covered information on: stages of sleep, factors that precipitate and perpetuate insomnia, general behavioral strategies to carry out during the day (e.g., engage in physical activity), the evening (e.g., avoid caffeine and nicotine), and around bedtime (e.g., put the day to rest), and changes in the sleep environment (e.g., have a comfortable room temperature). The recommended strategies promote a good night’s sleep (Irish et al., 2015). In our study, the SEH was implemented in two formats: the therapist presented and discussed the information in the first treatment session (thereafter called Education), and handed out a booklet (thereafter called Booklet) that summarized the same information for future reference.

The SCT entailed specific instructions aimed at helping persons with insomnia re-associate the bedroom with sleepiness and the bed with sleeping. The instructions were: go to bed only when sleepy, avoid activities other than sleeping (e.g., reading) while in bed, get out of bed if unable to fall asleep or back to sleep within 15–20 minutes, wake up at the same time every day, and avoid naps (Bootzin & Epstein, 2011). The therapist explained the instructions in the first intervention session, and in subsequent sessions, reinforced them, discussed barriers to their application and engaged participants in group problem-solving to identify strategies to overcome the barriers.

The SRT involved limiting the amount of time in bed to the participants’ individual sleep time as reported in their daily sleep diaries, and developing a sleep-wake schedule that specified the bed and wake-up times. In the first intervention session, the therapist worked with the participants to schedule the bed and wake-up times, taking into account their personal lifestyle. In the remaining sessions, the therapist assisted in exploring and managing factors that interfered with participants’ adherence to the sleep-wake schedule, and revised the schedule based on improvement in sleep documented in the daily sleep diary and on an available algorithm (Germain et al., 2006).

Variables and Measures

Data on the participants’ socio-demographic characteristics, related to age, sex, marital status, education, and employment status were obtained with standard questions. The concepts of interest were satisfaction with the MCI, adherence to the MCI recommendations, and the outcome of perceived insomnia severity (measured at post-test).

Satisfaction with the MCI was measured with the MDTSM, described above and in Table 1. Subscale scores were computed as the mean of the respective items’ scores; higher scores indicated a positive appraisal of the process or outcome attributes (i.e., high levels of satisfaction).

The usability of the MDSTM subscales was examined as part of the internal pilot test of the large trial. Eight participants who received the MCI and completed the MDSTM were asked to comment on the clarity of the instructions, the items’ content and response options, and the ease of completion. All participants reported no difficulty in completing the MDSTM.

Adherence to the MCI was assessed with a self-report item inquiring about overall compliance to the MCI recommendations, using a five-point scale ranging from not at all (0) to very much (4). Single items, often with a five-point Likert-type scale, have been used as patient-reported measures of adherence to pharmacological (e.g., Clifford, Perez-Nieves, Skalickey, Reaney, & Coyne, 2014) and behavioral interventions (e.g., Matthews, Arnedt, McCarthy, Cuddihy & Aloia, 2013).

Perceived insomnia severity at post-test was measured with the Insomnia Severity Index (ISI) developed by Morin (1993). The ISI contains seven items assessing severity of insomnia and distress with the sleep problem on a five point rating scale (0 = not at all and 4 = very much). The total ISI score is computed as the sum of the items’ scores, with a possible range of 0 to 28; higher scores indicate more severe insomnia. The ISI demonstrated high internal consistency reliability (Cronbach’s alpha coefficient ≥ .90) and validity (evidenced by significant correlation with other subjective and objective measures of insomnia severity, and with measures of conceptually related variables) across studies of persons with both good and poor sleep patterns (Morin, Belleville, Bélanger & Ivers, 2011).

Data Analysis

Descriptive statistics (frequency distribution, mean and standard deviation) were applied to characterize the profile of participants. The percentage of participants with missing data on each item was computed.

The psychometric properties were examined for each subscale of the MDTSM except the one on discomfort, consisting of one item with a dichotomous response scale and one item with an open-ended response option (Table 1). The subscales’ internal consistency reliability was determined with the Cronbach’s alpha coefficient and item-total correlation coefficient. An alpha coefficient ≥ .70 and item-total correlation coefficients ≥ .30 were used as indicators of reliability of the subscales.

Factor analysis was carried out to determine whether the items in each subscale loaded on a single factor, as identified in conceptualization and operationalization of satisfaction with treatment delineated above. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was estimated prior to conducting the factor analysis. The number of factors to be extracted, using principal axis factoring, was imposed to one. A KMO value equal or greater than .50 was considered suitable for factor analysis (Williams, Onsman, & Brown, 2010). The extracted factor was accepted if the eigenvalue was ≥ 1.0, the percentage of the variance in the respective items’ responses accounted for by the factor was ≥ 10%, and the loadings of the items on the respective factor were ≥ .30 (Streiner & Norman, 2014). In addition, evidence of construct validity included statistically significant (p ≤ .01, adjusting for multiple tests of association) correlations of a small-to-moderate (.2 to .6) size, between the subscales of the MDSTM and the self-reported level of adherence and perceived insomnia severity at post-test. Pearson’s correlation coefficients (r) were computed.

Results

Participants’ Profile

On average, participants were middle-aged (56 ± 16, range: 21–90 years), well-educated (15.7 ± 3.5, range: 3–30 years of formal education), and women (59.5%). Most participants were married (58.3%) and employed on a full- (41%) or part-time (13%) basis. The majority (90%) were white.

Responses to Items

The percentage of participants with missing data was low (range: 0% to 30%), not exceeding 15% for all items except two. The latter items were related to the frequency of using medication (30%) and engagement in job-related activities (17%), which may not have been applicable to some participants.

Internal Consistency Reliability

The Cronbach’s alpha coefficient and the range of item-total correlation coefficients computed for the multi-item subscales of the MDTSM are presented in Table 2. The alpha and item-total correlation coefficients exceeded the pre-set criteria for all subscales except for the perceived suitability of two treatment components. Their low (< .70) alpha coefficients may be due to the rather limited variability in the participants’ responses to the respective items; most participants rated the components as suitable for managing insomnia. However, the item-total correlations of the respective items were > .30, supporting their internal consistency.

Table 2.

Psychometric properties of the Satisfaction with Treatment measure

DOMAIN/Attribute	Subscale	Number of items	Cronbach’s alpha	Item-total correlation	KMO	Eigen-value	% of variance	Loadings
PROCESS
Treatment components	Suitability
	SEH	3	.74	.52 – .68	.68	1.6	53.7	.62 – .90
	SCT	3	.65	.36 – .56	.61	1.3	44.6	.43 – .81
	SRT	3	.66	.32 – .58	.60	1.3	45.1	.37 – .82
Treatment components	Utility
	SEH	4	.92	.80 – .84	.83	2.9	74.5	.84 – .89
	SCT	4	.92	.80 – .86	.85	3.0	76.9	.84 – .90
	SRT	4	.93	.82 – .85	.85	3.1	78.1	.86 – .89
Overall treatment	Attitude toward treatment	2	.74	.36 – .74	.70	2.4	48.3	.50 – .83
	Desire for continued treatment use	2	.81	.70	.50	1.4	70.1	.83
Implementation	Therapist Competence	3	.88	.75 – .82	.73	2.2	73.4	.81 – .91
	Therapist Interpersonal style	3	.81	.55 – .78	.50	1.1	57.0	.34 – .60
	Format and dose
	Booklet	4	.80	.54 – .73	.62	2.3	58.6	.70 – .83
	Group	5	.81	.49 – .76	.73	2.4	48.3	.53 – .91
	Telephone	5	.87	.63 – .77	.75	3.0	60.0	.71 – .83
OUTCOME
Perceived benefits	Health problem (insomnia)	8	.86	.40 – .76	.86	3.8	47.6	.42 – .82
	Everyday functioning	4	.95	.84 – .90	.83	3.3	83.6	.86 – .94
Attribution of outcomes to treatment	Attribution of outcomes to treatment	6	.86	.51 – .78	.79	3.4	57.4	.58 – .85

Note. SEH: Sleep education and hygiene; SCT: Stimulus control therapy; SRT: Sleep restriction therapy; KMO = Kaiser-Meyer-Olkin measure of sampling adequacy

Construct Validity

Construct validity was evaluated by examining the loadings of the items comprising each subscale on one factor reflecting the respective attribute and the correlation coefficients between the subscales’ scores and treatment adherence and outcome measured at post-test. As reported in Table 2, the KMO values exceeded the cut-off point for all subscales. All items comprising each subscale loaded on one factor. The factor loadings were >.35, exceeding the preset criterion. In addition, the corresponding eigenvalues were > 1.0, and the factors accounted for > 40% of the variance in the items’ responses.

The Pearson’s correlation coefficients quantifying the relationships of the subscales of the MDTSM to adherence to the MCI and to perceived insomnia severity at post-test are reported in Table 3. Overall, the relationships were in the expected directions. The positive correlation between the MDTSM subscales and self-reported adherence to therapy indicated that high levels of satisfaction were associated with high levels of adherence, whereas the negative correlation between the MDTSM subscales and perceived insomnia severity at post-test implied that high levels of satisfaction were related to low levels of insomnia severity at post-test.

Table 3.

Correlations between treatment satisfaction, adherence, and outcome

Domain	Attribute	Satisfaction subscale	Adherence	Outcome
Process	Treatment components	Suitability
		SEH	.25 *	− .19 *
		SCT	.23 *	− .23 *
		SRT	.22 *	− .24 *
		Utility
		SEH	− .03	− .19 *
		SCT	.10	− .23 *
		SRT	.15	− .24 *
	Overall treatment	Attitude toward treatment	.25 *	− .16
		Desire for continued treatment use	.19 *	− .20 *
	Implementation	Therapist
		Competence	.15	− .23 *
		Interpersonal style	.08	− .12
		Format and dose
		Booklet	.07	− .00
		Group	.13	− .04
		Telephone	.14	− .08
Outcome	Perceived Benefits	Health problem (insomnia)	.10	− .42 *
		Everyday functioning	.00	− .23 *
	Attribution of outcomes to treatment	Attribution of outcomes to treatment	.12	− .32 *

Note. SEH: Sleep education and hygiene; SCT: Stimulus control therapy; SRT: Sleep restriction therapy

^*p < .01

Perceived suitability of the three treatment components (SEH, SCT and SRT) demonstrated statistically significant associations, of a small magnitude, with adherence and perceived insomnia severity at post-test. Perceived utility of the three components had small associations with perceived insomnia severity at post-test. The subscales reflecting satisfaction with the overall therapy (i.e., attitude toward and desire for continued use of treatment) showed statistically significant but weak relationships with perceived insomnia severity at post-test. The therapist’s competence, but not interpersonal style, was weakly correlated with perceived insomnia severity at post-test. The subscales assessing satisfaction with intervention implementation (i.e., format and dose) had no significant relationships with treatment adherence and outcome. The subscales measuring satisfaction with outcomes and attribution of outcome to treatment were not associated with adherence but were weakly to moderately associated with perceived insomnia severity at post-test.

Discussion

Treatment satisfaction represents an important element in evaluating the effectiveness of interventions. The eleven MDTSM subscales are promising for comprehensively assessing participants’ satisfaction with behavioral interventions.

The content of the MDTSM was informed by a conceptualization of treatment satisfaction that was derived from a critical review and synthesis of conceptual and empirical literature. Treatment satisfaction refers to persons’ appraisal of the intervention, which is reflected in a set of attributes pertaining to both treatment process and outcome. The MDTSM operationalizes the process attributes as the suitability and utility of the intervention components, attitude toward the overall intervention, desire for continued use, therapist’s competence and interactional style, and treatment implementation (i.e., format and dose in which it is delivered). It also covers the outcome attributes of the treatment’s benefits and discomforts and the attribution of the benefits to the intervention. Each of these attributes is represented in a subscale.

The content of the items constituting the subscales is consistent with the description of the respective attributes as presented in the conceptualization of treatment satisfaction. This consistency maintained the content validity of the measure. Further, the items’ statements were written in simple and easy to understand terms, as recommended by DeVellis (2011). In our study, participants were able to complete the MDTSM (evidenced by the findings of the pilot test and the low percentage of missing data at the item level) despite variability in some of their socio-demographic (specifically age and education) characteristics, which provides additional support of the relevance of its content to a range of participants.

The results of psychometric testing provided initial evidence supporting the internal consistency and construct validity of the subscales. The results of the reliability and factor analyses converged in supporting the interdependence of the items within the respective subscales in capturing a common concept. The Cronbach’s alpha coefficients were consistently > .70 for the subscales, except the one measuring perceived suitability of the MCI components. The item-total correlation coefficients were of a moderate-to-large magnitude (≥ .32) across the subscales, including the one measuring perceived suitability of treatment components. Similarly, the items within the respective subscales loaded on one factor, which accounted for > 40% of the variance in the items’ responses; the item loadings were of a moderate-to-high magnitude (> .35).

Overall, the observed correlations between the subscale scores and the self-reported adherence to treatment and perceived insomnia severity at post-test provide additional preliminary evidence of the construct validity of the MDTSM. As hypothesized, the associations between satisfaction with treatment process attributes and adherence were statistically significant and slightly larger than the associations between satisfaction with treatment process attributes and perceived insomnia severity at post-test; compared to satisfaction with process attributes, satisfaction with outcome attributes was more highly associated with perceived insomnia severity at post-test. These findings are comparable to those reported by others (Barbosa et al., 2012; Fontana et al., 2006; George & Robinson, 2010; Smith et al., 2013; Wong et al., 2015) and support the construct validity of the MDTSM subscales.

Adherence demonstrated small, positive relationships with subscales assessing satisfaction with the MCI process attributes: suitability of its components, attitude towards it, and desire to continue using it. These relationships are comparable to previously reported associations, suggesting that persons who consider the treatment components suitable are satisfied with them and experience minimal burden in applying them (Barbosa et al., 2012; Brod et al., 2007).

Persons reporting low levels of perceived insomnia severity at post-test rated the three MCI components as suitable and useful for managing their sleep problem; had favorable attitudes toward treatment; expressed a desire to continue using the treatment; and rated the therapists as competent. They judged the MCI beneficial in managing their insomnia and performing daytime activities, and they attributed these benefits to the MCI. The associations between these attributes and perceived insomnia severity at post-test are consistent with previous findings showing significant but small to moderate correlations between overall satisfaction and post-test outcomes of different treatments, including psychotherapy (Oei & Green, 2008), diabetes medication (Rubin & Peyrot, 2009), retinopathy treatment (Brose & Bradley, 2009), pain medication (Rejas et al., 2013), mental health services (Tierney & Kane, 2011), migraine treatment (Martin et al., 2008), and substance use treatment and 12-step groups (Kendra et al., 2015).

The observed pattern of associations suggests that: satisfaction with treatment process attributes was related to adherence; satisfaction with treatment process and outcome attributes was related to the outcome as reported by participants at post-test; and satisfaction with treatment implementation was not related to adherence or outcome at post-test. If replicated in future research, this pattern of association supports the sensitivity of the domains of the MDTSM to different aspects of participants’ experiences with treatment.

Unlike other measures of treatment satisfaction, the MDTSM contains a comprehensive set of treatment attributes for persons to take into account when evaluating treatments. Useful information is generated by persons’ evaluation of the treatment process and outcome attributes. If ethically acceptable and logistically possible, all persons assigned to receive the behavioral intervention under evaluation are requested to complete the MDSTM subscales in the context of pilot, efficacy and effectiveness studies. For consistency with the conceptual definition of satisfaction with treatment, the MDSTM subscales are administered to participants after exposing them to the intervention, to enable them in appraising the intervention’s process and outcome attributes. Participants’ responses may identify intervention components that are not well-received and issues with treatment implementation, pointing to treatment areas that are viewed favorably and unfavorably. Areas viewed unfavorably then can be refined, with the goal of enhancing satisfaction, and hence treatment uptake, adherence, and outcome. For instance, low ratings of a particular component’s utility suggest the need to explore the underlying reasons, which could guide revision of the component’s content and activities. Similarly, responses indicating inadequacy of the number of treatment sessions or dislike of the group interaction would highlight the need to revise the treatment delivery dose and format, respectively.

An advantage of the MDTSM is its appraisal of specific process and outcome attributes to yield a comprehensive assessment of satisfaction with various treatments. Each subscale operationalizes one attribute. Although we recommend using all subscales to comprehensively assess participants’ satisfaction with behavioral interventions, it is possible to administer relevant subscales separately to evaluate specific intervention attributes of concern.

Limitations

Caution should be used in interpreting the relationships between treatment adherence and perceived insomnia severity at post-test and the respective MDSTM subscales because of three study limitations. First, the single item for assessing overall adherence, while similar to the one used in previous research, may not have been sensitive to variations in levels of adherence to the range of the MCI treatment recommendations. Second, the participants’ self-reported high levels of adherence with the overall MCI recommendations could be related to social desirability bias, as suggested by Clifford et al. (2014). Third, it is possible that participants who withdrew from treatment may have been dissatisfied with the MCI, and those who completed treatment may have been satisfied with the MCI, which may have contributed to low variance in responses to some items.

Implications for Future Research

The small to moderate correlations between the MDSTM subscales measuring the process and outcome attributes and treatment adherence and outcome at post-test, as reported in our study and other trials, require further investigation. Although the interventions have demonstrated efficacy, it is unclear whether conceptual and/or methodological factors contributed to the rather weak correlations. Conceptual factors include possible differences in valuing of process and outcome attributes (e.g., outcome attributes may be considered more important than process attributes). Methodological factors include the use of non-established measures of adherence, the potential for social desirability bias in responding to the measures of adherence and satisfaction with treatment, and high attrition rates among participants dissatisfied with the behavioral intervention under evaluation.

The reliability and validity that the MDTSM subscales demonstrated in our study enhance the accuracy of measurement and inferences. However, the MDSTM measure contains a large number of items; completion of all items may introduce response burden, which may affect the quality of responses (Streiner & Norman, 2014). This potential is counterbalanced by the simplicity of the items’ content, which is likely to facilitate item comprehension. The time it took participants to complete the MDSTM subscales was not documented; completion of all items could have been time-consuming and could have contributed to missing data. Time to complete the subscales should be examined in future studies.

The results of our study supported the psychometric properties of the MDSTM subscales. Additional research is required to: evaluate the sensitivity of the subscales in capturing levels of satisfaction with different behavioral interventions for insomnia; validate the utility of the subscales in predicting continued adherence to the interventions’ recommendations and improvement in short- and long-term outcomes (other than perceived insomnia severity at post-test); and explore the utility of the MDSTM subscales in assessing satisfaction with a range of behavioral interventions targeting different health problems. The MDSTM subscales are available upon request; they must be tailored to the attributes (e.g., outcomes and mode of delivery) of other behavioral interventions as needed.

Conclusions

Treatment satisfaction is recognized as an important element in evaluating interventions. A comprehensive conceptualization guided the development of the MDTSM. The measure contains 11 subscales assessing specific treatment process and outcome attributes. Initial testing supported the reliability and validity of the subscales, which can be used to determine aspects of different treatments that are viewed favorably or unfavorably, and to examine the extent to which satisfaction with treatment influences adherence and outcomes.