Skip to main content
NCBI home page
Research in Psychotherapy : Psychopathology, Process, and Outcome logoLink to Research in Psychotherapy : Psychopathology, Process, and Outcome
. 2023 Jul 27;26(2):687. doi: 10.4081/ripppo.2023.687

Validation of the client satisfaction questionnaire: a pilot psychometric analysis in Portuguese routine practice

Sara Silva 1, Isabel Basto 1, João Salgado 1, Carla Cunha 1,✉
PMCID: PMC10483481  PMID: 37503687

Abstract

Treatment satisfaction has been found to have good therapeutic results in psychotherapy, and the 18-item version of the client satisfaction questionnaire (CSQ-18) is one of the most widely used measures to evaluate it. This study sought to carry out a pilot analysis of the psychometric’s properties and factorial structure, as well as validate the CSQ-18’s applicability to the Portuguese population within the context of routine clinical practice. It also sought to explore the association that treatment satisfaction has with general symptoms, therapeutic alliance, and expectations at an early stage of psychotherapy. The sample comprised 98 clients who were undergoing psychotherapy in a routine practice context. All these clients completed self-reported measures for symptom assessment and therapeutic relationship, namely the clinical outcome routine evaluation-outcome measure, working alliance inventory, and credibility/expectancy questionnaire, in addition to the CSQ-18. The semi-confirmatory factorial analysis demonstrated that the CSQ-18 has good psychometric properties and revealed an association between treatment satisfaction and therapeutic alliance. The results corroborate the findings of other versions of the measure and present a good adjustment model for the semi-confirmatory factorial analysis.

Key words: treatment satisfaction, validation, routine practice, psychotherapy

Introduction

According to the World Health Organization (2007), assessing treatment satisfaction is the best way to account for clients’ concerns about services and opinions about new treatments. From the therapist’s point of view, assessing satisfaction makes it possible to solicit feedback about a treatment’s implementation and make improvements to the services and treatments provided; moreover, it helps anticipate negative reactions and behaviors, such as dropout, treatment disengagement, and poor treatment adherence (European Foundation for Quality Management, 2013; Draper et al., 2001; Howard et al., 2007; Vázquez et al., 2019).

Treatment satisfaction encompasses multiple aspects, such as assessment of the services, care quality, and available treatments (Larsen et al.,1979). More specifically, according to Larsen et al. (1979), treatment satisfaction considers 9 dimensions: physical surroundings, support staff, type/kind of service, treatment staff, quality of treatment, quantity of treatment, outcome of services, overall satisfaction, and service procedures. Assessing a client’s satisfaction with treatment provides a more complete picture of the client without the bias of intervention programs (Attkisson & Zwick, 1982) and with an assessment of the quality of services (Säilä et al., 2008). Moreover, it can offer important information about the treatment a client is receiving and possibly help identify a client’s unsatisfied needs and expectations (Carlson & Gabriel, 2001; Hanson et al., 2004).

Research into satisfaction with psychological treatments

Studies of treatment satisfaction have been conducted to assess the quality of health services and establish a treatment’s acceptance and effectiveness in various contexts, such as hospitals, primary healthcare systems, group interventions, and clinical trials (Buffini & Gordon, 2015; Donovan et al., 2002; Hund et al., 2013; Kooistra et al., 2016; Palacios et al., 2018; Smith et al., 2014). However, when it comes to mental health, research has generally focused on healthcare settings related to substance abuse and mood and anxiety disorders (Buffini & Gordon, 2015; Donovan et al., 2002; Hutchison et al., 2022; Schulte et al., 2011; Smith et al., 2014).

The results generally show that satisfaction with psychological treatments tends to be high, regardless of the context (Hutchison et al., 2022; Schulte et al., 2011). Recent studies have found treatment satisfaction to be associated with good therapeutic results, improvements in quality of life, and even a decrease in hospitalizations, regardless of the treatment received (Blenkiron & Hammil, 2003; Donovan et al., 2002; Priebe et al., 2012; Smith et al., 2014). Clients who are not very satisfied with their treatments tend to disengage and drop out (Blenkiron & Hammil, 2003; Hundt et al., 2013; Lindhiem et al., 2014; Priebe et al., 2012).

Only a few studies have explored the association between treatment satisfaction and treatment factors like adherence to therapy, therapeutic alliance, and expectations (Constantino et al., 2011; Hundt et al., 2013; Smeets et al., 2008). Hundt et al. (2013) analyzed the predictors of treatment satisfaction following the conclusion of psychotherapy treatment, and their results showed that treatment satisfaction was positively associated with well-being, expectations, social support, improvement in symptoms, and treatment adherence. Moreover, further research has suggested that positive expectations about treatment can predict client satisfaction (Constantino et al., 2011; Smeets et al., 2008). Other research that explored the association between treatment satisfaction and therapeutic alliance found that the most satisfied clients had a stronger therapeutic alliance, both in the initial phase of therapy (Egede et al., 2016; Miller & Weissman, 2002; Rumpold et al., 2005; Warnecke et al., 2020) and at the end of the intervention (Donker et al., 2013; Palacios et al., 2018; Rosenvinge & Klusmeier, 2000). However, these studies tend to assess satisfaction with treatment retrospectively (Egede et al., 2016; Hundt et al., 2013; Miller & Weissman, 2002; Rumpold et al., 2005; Warnecke et al., 2020) or at the end of treatment (Donker et al., 2013; Palacios et al., 2018; Rosenvinge & Klusmeier, 2000).

Measures of treatment satisfaction

There are only a few empirically validated measures of treatment satisfaction, and most studies use qualitative measures, such as interviews and other self-developed measures (Jenkins- Guarnieri et al., 2015; Miller & Weissman, 2002; Roe et al., 2006; Schulte et al., 2011; Vázquez et al., 2019). The decision to create specific measures for different single studies may be justified, but it also raises difficulties when it comes to comparing such studies. The client satisfaction questionnaire (CSQ) is one of the most promising solutions to this problem, and it is the most widely used measure to assess treatment satisfaction, especially the 8-item version [(CSQ-8) Attkisson & Zwick, 1982; Larsen et al., 1979). This instrument, which was initially developed by Larsen et al. (1979) as a version with 18 items (CSQ-18), is a self-reported instrument based on a Likert scale used to assess treatment satisfaction. The CSQ-8 is a reduction of the CSQ-18, but both versions have good psychometrics with an a³.91 (Attkisson & Zwick, 1982; Larsen et al., 1979). In terms of the factorial structure of the original CSQ-18 version, it covers only one factor despite evaluating 9 dimensions of treatment satisfaction, namely physical surroundings, support staff, type/kind of service, treatment staff, quality of treatment, quantity of treatment, the outcome of services, overall satisfaction, and service procedure (Attkisson & Zwick, 1982; Larsen et al., 1979). The convergent validity of the measure was explored by investigating the link between treatment satisfaction and clinical symptoms, and a negative association between treatment satisfaction and clinical symptoms was found, meaning that clients with greater treatment satisfaction saw a reduction in clinical symptoms (Attkisson & Zwick, 1982; Larsen et al., 1979). Furthermore, other studies on the validity of CSQ-18 have yielded similar results, suggesting that improvements in clients’ outcomes are somehow related to the way they perceive their treatment (Sabourin et al., 1989; Vázquez et al., 2019). Indeed, this measure has been validated among multiple populations, namely French Canadians, Castilian Spanish, and Filipinos. Also, it has been shown to have good psychometric properties (𝛼>.80), implying that the measure is robust in different contexts (Matsubara et al., 2013; Roberts et al., 1984; Sabourin et al., 1989; Vázquez et al., 2019).

Overall, the previous validation studies on the CSQ-18 have reported consistent results for the one-dimensionality of this measure (Attkisson & Zwick, 1982; Larsen et al., 1979). However, the previous validation studies on the CSQ only present, as a result of exploratory factor analysis, the internal consistency of the instrument. Studies that explored adjusting the model of the factorial structure were not found.

The present study

The CSQ has not yet been validated for the Portuguese population, so this study focuses on a pilot analysis of the psychometric properties of the Portuguese European version of this instrument. Thus, in addition to analyzing the psychometric properties, we also present a pilot adjusting the model of factorial structure of the CSQ-18, applying bootstrapping, within the context of routine psychotherapeutic practice. Moreover, as a secondary but complementary goal using the same sample, we also explore the relationship that treatment satisfaction has with general symptoms, therapeutic alliance, and expectations at an early stage of psychotherapy.

Methods

Participants

This study involved 98 adult participants aged between 18 and 63 (mean age=29 years; standard deviation=9.99) of whom 68 (69.4 %) were women and 30 (30.6%) were men. Of these, 58.2% had university degrees, 39.8% had completed high school, and 2% had only finished primary school. In terms of employment status, 54.1% were employed, 34.7% were students, 7.1% were unemployed, 3.1% were students/workers, and 1% were homemakers. Most participants (76%) were single, 14.3% were married or cohabitating, and 9.2% were divorced or widows/widowers.

All participants were recruited through a university counseling service (University Institute of Maia (ISMAI), Portugal). Clients access university counseling services to receive psychological support, psychotherapy, and/or personal development. Cases may be self-referred or referred by other health services, and they involve different clinical symptoms. The participants were divided into clinical and non-clinical, according to symptom severity measured with clinical outcome routine evaluationoutcome measure 10 [(CORE-10) Barkham et al., 2013] in the assessment session. In this study sample, 83.7% of participants presented clinical symptoms, while 16.3% had nonclinical issues. Participants were informed about the study and its confidentiality conditions (i.e., to guarantee the anonymity of each participant), and all of them consented in writing to participate. For this study, we considered all clients of the service, even those who were already involved in the therapeutic process.

The clients of the university counseling service could access psychotherapy face-to-face or online, as well as a mixture of the two. The online sessions came about in response to the situation caused by the COVID-19 pandemic. More specifically, 84 clients (85.7%) received face-to-face psychotherapy, 6 (6.1%) received online psychotherapy, and 8 (8.1%) received a mix of both modalities.

Measures

The CSQ-18 (Attkisson & Zwick, 1982; Larsen et al., 1979) is a self-reported measure with 18 items that assess the client’s global treatment satisfaction within a clinical context (e.g., quality of service, kind of service, the needs met, the likelihood of recommending it to a friend, the degree of help, how problems are dealt with, overall satisfaction, and chances of returning). The client answers 18 questions based on a 4-point Likert scale (Attkisson & Zwick, 1982; Larsen et al., 1979). Thus, each item in the CSQ-18 is scored from 1 to 4, with the total score ranging from 18 to 72 (Attkisson & Zwick, 1982; Larsen et al., 1979), where higher scores indicate a greater level of satisfaction with treatment. The original version of the CSQ- 18 has been shown to have good internal consistency [α>.91; (Attkisson & Zwick, 1982)].

The CORE-10 (Barkham et al., 2013) is a self-reported measure for assessing mental health in different contexts. It is clustered into 4 subscales, namely subjective well-being, complaints and symptoms, social and personal functioning, and risk behavior (Barkham et al., 2013). The 10 items are scored on a Likert scale ranging from 0 (never) to 4 (always). The CORE-10 is actually a reduced version of the clinical outcome routine evaluation-outcome measure (Sales et al., 2012), which is a 34-item questionnaire that has been translated and adapted for the Portuguese population with good internal consistency [α>.80; (Sales et al., 2012)].

The working alliance inventory-6 items (WAI-6) includes 6 items that evaluate the quality of a therapeutic relationship using a Likert scale that ranges from 1 (never) to 5 (always) (Falkenström et al., 2015). This is a reduced version of the 12-item working alliance inventory-short revised (Horvath & Greenberg, 1989), which has been validated for the Portuguese population with an adequate degree of reliability [𝛼=.95; (Machado & Horvath, 1999)].

The credibility/expectancy questionnaire [(CEQ) Devilly & Borkovec, 2000] is a self-reported measure with 6 items for assessing a client’s credibility and expectations. More specifically, the credibility items explore whether the treatment is i) logical, ii) useful, and iii) reliable. Meanwhile, the expectations items explore i) the improvement that the patient expects; ii) whether the client feels that therapy will help him/her; iii) whether the client feels that an improvement will occur (Devilly & Borkovec, 2000). This measure is scored on a Likert scale ranging from 1 to 9, where, for example, 1 means not at all, 5 means somewhat, and 9 means very much, as well as a percentage scale ranging from 0 to 100 in 10% increments. More specifically, items 4 and 6 are coded from 0 to 100%, so they are linearly translated to the Likert scale (Devilly & Borkovec, 2000; Smeets et al., 2008). The Portuguese version of the CEQ has 5 items with 2 subscales, namely credibility with 3 items and expectations with 2 items, and it has been shown to have good internal consistency (𝛼=.78;Silva, et al., 2021).

Procedures

Authorization to translate, apply, and validate the CSQ-18 measure for the Portuguese population was requested from and granted by the measure’s creators. Two independent researchers who were fluent in both relevant languages translated the measure into Portuguese. The main researcher conducted the first translation, while the back-translation and revision were performed by a specialized translator. This study sought, and was granted, approval by the Ethics Committee (5/CEDSMAI/2019; ISMAI, Portugal).

The goals of this research were presented to all clients, both new ones and those already engaged in psychotherapy, of the university campus psychotherapy center, which is open to the general public in addition to students and staff. Inclusion in the study required clients’ informed consent to participate and allow data to be collected. Following the clinic’s protocol, all clients had an initial assessment session where they completed the CORE-10 before starting the session. The clients that were included in this study were divided into clinical and non-clinical based on the symptom severity of CORE-10 during the initial assessment session, according to the instrument cut-off point. The clients that scored 10 or below were considered non-clinical, and those with a score of 11 or above were considered clinical, following the scoring procedure of CORE-10 (Connell & Barkham, 2007). Clients who had already attended less than 4 sessions completed the CSQ-18 just before starting sessions 4 and 5, while all remaining participants completed this measure in 2 consecutive sessions. The clients included in this study completed the CSQ-18 twice, with a minimum period between sessions of 1 week and a maximum period of 3 weeks. All clients also filled in the CORE-10 before starting a session and the WAI-6 and CEQ at the end of all sessions. We used all the CSQ- 18 ratings from every participant in our exploratory factor analysis. The repeated application of the CSQ-18 allowed us to analyze the test-retest reliability. To achieve this study’s other goals, we considered only those clients who had filled in the CSQ-18 at the initial stage of the process (i.e., session 4).

Statistical analysis

Initially, descriptive statistics and tests for normality were conducted among the items through the analysis of Skweness and Kurtosis (Table 1). The Kaiser-Meyer-Olkin (KMO) measure verified the sample’s adequacy for the analysis. The item inter-correlation matrix, through the Polychoric correlation, was performed following the recommendation of Baglin (2014) for exploratory factor analysis on ordinal variables (Table 2). After the preliminary analysis, we performed an exploratory factor analysis using FACTOR (Ferrando & Lorenzo-Seva, 2017), which is a free program that facilitates such analyses (Lorenzo-Seva & Ferrando, 2006). The SPSS 25 (SPSS Inc., Chicago, IL, USA) software was also used to check the internal consistency in terms of Cronbach’s a coefficient, test-retest reliability, and Spearman correlation to achieve the remaining goals of the present study.

For factor extraction (Timmerman & Lorenzo-Seva, 2011), the following criteria were used: i) parallel analysis, as it is considered an adequate procedure to determine the number of factors to retain; ii) the Kaiser criterion, which involves retaining factors with an eigenvalue greater than 1 (Field, 2017; Kaiser,1960). Regarding the quality of the items and the factors for retaining an item, it was decided that commonality should be greater than .40 and adequate saturations should be greater than .35 (Costello & Osborne, 2005). Internal consistency was evaluated using Cronbach’s a coefficient [.60 to .70 is acceptable and greater than .70 is recommended (Field, 2017)]. The method used was maximum likelihood with orthogonal varimax rotation (Baglin, 2014; Field, 2017), considering that the variable is ordinal and independent. Bootstrapping was used to analyze the adjustment quality of the factorial model based on the indexes with empirical statistical support (Brown, 2015) given the size of the sample, specifically the robust mean-scaled chi-square, root mean square error of approximation, goodness-of-fit index, adjusted goodness-of-fit index, non-normed fit index, and comparative fit index.

Table 1.

Description of normality and homogeneity assumptions (Skewness and Kurtosis of each item) of the 18-item version of the client satisfaction questionnaire.

Item Mean Confidence interval 95% Variance Skewness Kurtosis
1 3.75 3.59-3.90 .371 -3.330 14.106
2 3.38 3.15-3.62 .822 -1.742 2.634
3 3.84 3.70-3.97 .277 -4.782 8.071
4 3.63 3.44-3.82 .537 2.702 8.345
5 3.59 3.39-3.78 .546 -2.351 6.551
6 3.64 3.48-3.79 .373 -2.554 10.810
7 3.85 3.70-3.99 .310 -4.998 27.653
8 3.62 3.46-3.78 .378 -2.430 10.135
9 3.37 3.20-3.55 .476 -1.398 4.141
10 3.70 3.54-3.85 .353 -2.998 13.526
11 3.77 3.63-3.91 .300 -3.802 20.690
12 3.71 3.55-3.87 .389 -2.973 11.815
13 3.72 3.56-3.87 .364 -3.114 13.447
14 3.76 3.61-3.90 .305 -3.676 19.640
15 3.72 3.57-3.87 .344 -3.175 14.730
16 3.56 3.38-3.73 .449 -2.037 6.554
17 3.87 3.74-3.99 .235 -5.687 39.240
18 3.68 3.49-3.87 .542 -2.931 9.126
Open in a new tab

Table 2.

Correlation matrix of items of the 18-item version of the client satisfaction questionnaire.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Item 1 - .29 .40 .17 .27 .13 .19 .28 .33 .36 .30 .36 .30 .31 .18 .33 .17 .38
Item 2 - .57 .50 .37 .29 .38 .25 .11 .29 .53 .26 .30 .16 .31 .29 .22 .15
Item 3 - .38 .33 .21 .43 .14 .35 .31 .54 .37 .23 .12 .45 .45 .53 .46
Item 4 - .62 .61 .44 .69 .47 .48 .57 .62 .66 .60 .66 .76 .24 .56
Item 5 - .56 .36 .57 .37 .44 .54 .55 .56 .54 .55 .61 .16 .46
Item 6 - .24 .47 .38 .33 .45 .59 .51 .61 .42 .66 .18 .43
Item 7 - .42 .03 .22 .48 .40 .38 .50 .49 .34 .54 .55
Item 8 - .53 .50 .43 .58 .63 .60 .67 .66 .40 .52
Item 9 - .64 .52 .50 .46 .48 .61 .60 .25 .27
Item 10 - .67 .56 .56 .50 .46 .54 .23 .36
Item 11 - .63 .59 .43 .50 .67 .19 .35
Item 12 - .63 .57 .53 .70 .17 .55
Item 13 - .66 .56 .75 .25 .61
Item 14 - .54 .63 .41 .69
Item 15 - .78 .43 .52
Item 16 - .31 .56
Item 17 - .49
Item 18 -
Open in a new tab

Results

Factor structure

The polychoric correlation was preferred over Pearson’s correlation, a decision based on the recommendation of Baglin (2014) for exploratory factor analysis on ordinal variables (Table 2). The KMO measure supported the sample’s adequacy for analysis (KMO=.90), as did the significant Bartlett’s test [𝜒2(153)=1049.6, p<.001]. To determine the number of factors being extracted, we performed a parallel analysis with an orthogonal varimax rotation. However, 4 items could not be retained due to their low quality (Table 3), with their commonality values being lower than .40 (Costello & Osborne, 2005), namely item 1 (λ=.15), item 2 (λ=.20), item 3 (λ=.26), and item 17 (λ=.19).

After removing items 1, 2, 3, and 17, the factorial analysis was repeated, and the quality of the factorial adjustment model was analyzed using bootstrapping with a 95% confidence interval (CI). The KMO measure verified the sample’s adequacy for analysis (KMO=.93, 95% CI [.92 - .93]), as did the significant Bartlett’s test [𝜒2(91)=1064.9, p<.001]. To determine the number of factors being extracted, a parallel analysis was again performed with orthogonal varimax rotation, and this revealed a factor that explained 60.96% of the variance (eigenvalue =8.53 and a percentage cumulative of 60.96). An analysis of the commonality values showed a good relationship between the items and the model (higher than .40) and high factor saturations [between .51 and .94 (Table 4)]. The adjustment indices for the factorial solution also reflected a good adjustment (Table 5).

Reliability

The internal consistency was analyzed without items 1, 2, 3, and 17 following the recommendation of Costello and Osborne (2005), with this yielding a Cronbach’s 𝛼=.88. The test-retest reliability was assessed by correlating the factor scores from the first application of the CSQ with those from the second application, and it was found to be reliable and significant, r(61)=.76, p=.00.

The associations of treatment satisfaction with general symptoms, therapeutic alliance, and expectations at an early stage of psychotherapy

To analyze the associations that treatment satisfaction has with general symptoms, therapeutic outcomes, and expectations, we used the Spearman correlation because normality requirements were not met. Only those participants who had completed the CSQ in session 4 were considered in this analysis. Treatment satisfaction and general symptoms (N=56) were found to have a moderately negative but statistically significant association, rs(56)=-.31, p=.02, meaning that at the early stage of psychotherapy, higher levels of treatment satisfaction tend to coincide with lower levels of general symptoms. Furthermore, as expected, treatment satisfaction was also significantly associated with higher levels of therapeutic alliance at an early stage of psychotherapy, rs(28)=.39, p=.04. Finally, the association between treatment satisfaction and expectations was weak and not statistically significant, rs(28)=.09, p=.66.

Table 3.

Exploratory factor solution for the 18-item version of the client satisfaction questionnaire.

Item Factor 1 Comm Mean Confidence interval 95% a
1 .38 .15 3.75 3.59-3.90 .37
2 .44 .20 3.38 3.15-3.62 .82
3 .51 .26 3.84 3.70-3.97 .28
4 .83 .68 3.63 3.44-3.82 .54
5 .70 .49 3.59 3.39-3.78 .55
6 .65 .43 3.64 3.48-3.79 .37
7 .66 .30 3.85 3.70-3.99 .31
8 .75 .56 3.62 3.46-3.78 .38
9 .61 .37 3.37 3.20-3.55 .48
10 .66 .43 3.69 3.54-3.85 .35
11 .74 .55 3.77 3.63-3.91 .30
12 .77 .60 3.71 3.55-3.87 .39
13 .80 .64 3.72 3.56-3.87 .36
14 .74 .55 3.76 3.61-3.90 .31
15 .78 .60 3.72 3.57-3.87 .34
16 .88 .78 3.56 3.38-3.73 .45
17 .43 .19 3.87 3.74-3.99 .24
18 .70 .48 3.68 3.49-3.87 .54
Open in a new tab

Table 4.

Bootstrapping exploratory factor solution for the client satisfaction questionnaire with 14 items.

Item Factor 1 Comm Mean Confidence interval 95% a
4 .83 .69 3.63 3.44-3.82 .54
5 .85 .73 3.59 3.39-3.78 .55
6 .66 .43 3.64 3.48-3.79 .37
7 .53 .28 3.85 3.70-3.99 .31
8 .92 .85 3.62 3.46-3.78 .38
9 .51 .26 3.37 3.20-3.55 .48
10 .64 .41 3.69 3.54-3.85 .35
11 .70 .48 3.77 3.63-3.91 .30
12 .94 .89 3.71 3.55-3.87 .39
13 .82 .68 3.72 3.56-3.87 .36
14 .76 .58 3.76 3.61-3.90 .31
15 .76 .58 3.72 3.57-3.87 .34
16 .89 .80 3.56 3.38-3.73 .45
18 .66 .44 3.68 3.49-3.87 .54
Open in a new tab

Table 5.

Bootstrapping adjustment of the factorial solution for the client satisfaction questionnaire with 14 items.

Index of adjustment
RMSCS ^(77)=12.882, p=.250 .
RMSEA .000 [.000, .010]
GFI .970 [.920, .990]
AGFI .964 [.906, .989]
NNFI 1.151 [.447, 1.343]
CFI 1.128 [.532,1.290]
Open in a new tab

95% CI BCa, bias-corrected and accelerated confidence intervals for 95% level of confidence. RMSCS, robust mean-scaled chi-square; RMSEA, root mean square error of approximation; GFI, goodness-of-fit index; AGFI, adjusted goodness-of-fit index; NNFI, non-normed fit index; CFI, comparative fit index.

Discussion and Conclusions

The present study set out to pilot the exploration of the psychometric properties and factorial structure of the CSQ within the context of routine practice, as well as analyze the associations that treatment satisfaction has with general symptoms, therapeutic alliance, and expectations at an early stage of psychotherapy.

Similar to what was found in other studies that have analyzed the CSQ’s factorial structure, the results suggest the existence of a single factor (Attkisson & Zwick, 1982; Larsen et al., 1979; Matsubara et al., 2013; Roberts et al., 1984; Sabourin et al., 1989; Vázquez et al., 2019), indicating that treatment satisfaction is a homogeneous variable across different countries and cultures. In addition, the results of the present study suggest removing items 1, 2, 3, and 17 because they have lower levels of saturation and retention (Costello & Osborne, 2005). This finding is not consistent with those for the original CSQ-18 version, but in the reduced CSQ-8 version, items 1, 2, 3, and 17 were also eliminated for having low levels of saturation and retention (Attkisson & Zwick, 1982; Larsen et al., 1979). Thus, the pilot validation of the European Portuguese version should comprise 14 items, and we suggest that future studies refer to it as CSQ-14-PT.

The previous validation studies did not explore the CSQ’s adjustment model, so the present study proposed a pilot analysis with bootstrap, taking into account the removal of the previously mentioned items. The results gave a good adjustment index for the presented factorial structure, but this finding should be treated with caution given the relatively small sample size.

The previous validation studies did not explore the CSQ’s adjustment model, so the present study proposed a pilot analysis with bootstrap, taking into account the removal of the previously mentioned items. The results gave a good adjustment index for the presented factorial structure, but this finding should be treated with caution given the relatively small sample size.

By analyzing the means of each CSQ item, we verified that satisfaction with the psychotherapy treatment in the routine practice context was high, with it being similar to what would be found in a hospital, community, online, or group intervention context (Buffini & Gordon, 2015; Donker et al., 2013; Donovan et al., 2002; Palacios et al., 2018; Schulte et al., 2011; Smith et al., 2014; Warnecke et al., 2020). Previous research into the association between satisfaction with psychotherapy treatment and general symptoms has found that satisfaction relates to lower levels of symptoms at the end of an intervention (Hundt et al., 2013). The results of the present study are in line with this trend, although they indicate that higher levels of treatment satisfaction at an early stage of therapy are associated with lower levels of general symptoms.

Regarding the association between a client’s satisfaction with treatment and the therapeutic alliance, the results indicate a significant relationship, such that greater satisfaction relates to a stronger therapeutic alliance at an early phase of therapy. This resembles the findings of studies of brief psychotherapy and group psychotherapy (Rumpold et al., 2005; Warnecke et al., 2020), as well as those of some studies that retrospectively analyzed the relationship between therapeutic alliance and satisfaction with treatment (Keleher et al., 2019).

The association between treatment satisfaction and expectations was not found to be statistically significant. This is contrary to previous studies (Constantino et al., 2011; Smeets et al., 2008), but it should be noted that those previous studies analyzed the predictive power of initial expectations to determine the satisfaction level at the end of the treatment (Constantino et al., 2011; Smeets et al., 2008), while our study used one point of assessment at an early phase of treatment. Combining these findings suggests that satisfaction may improve over time as the results start to manifest.

This study has some limitations that should be considered when interpreting the results, such as the limited sample size. Although it meets the recommendations for validating psychometric properties (Gorsuch, 1983), a larger sample would make the results more robust. The data collection also occurred at different stages, and this was also a limiting factor because some clients were already engaged in the therapeutic process, while others were just at the beginning. More specifically, being at different stages may have influenced the clients’ perspectives on their treatment. However, we did not identify any studies that have analyzed differences in clients’ treatment satisfaction at different stages of the therapeutic process, so this would be a research gap for a future study to fill.

The present study provides some significant indicators for the importance of assessing clients’ satisfaction with their treatment, not just at the end but also during the therapeutic process. Given that there is limited research that has analyzed the role of treatment satisfaction in psychotherapy, some aspects should be explored in future studies, particularly the predictive effect of the therapeutic alliance on satisfaction with treatment at different points in the therapeutic process. It would importantly explore the ruptures at our lower levels in the therapeutic alliance that interfere with treatment satisfaction and impact adherence to the therapeutic process. Regarding the client’s expectations, it would be interesting to explore whether the association between the client’s expectations and treatment satisfaction is mediated by symptom severity and treatment credibility. Moreover, some studies in psychiatric contexts have suggested that clients with a lower level of satisfaction tend to be more likely to drop out (Smith et al., 2014), so it would be valuable to explore whether dropping out of treatment can be predicted by low levels of treatment satisfaction across different contexts.

Funding Statement

Funding: this article was supported by the Portuguese Foundation for Science and Technology (FCT) under the Pluriannual Funding Programme for Research Units 2020-2023 UIDP/00050/2020 and through the Ph.D. Grant SFRH/BD/139752/2018, Icare4Depression Project PTDC/MHC-PCL/1991/2014, and MCTES, FSE, and UE.

References

  1. Attkisson C. C., Zwick R. (1982). The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome. Evaluation and Program Planning, 5(3), 233-237. doi: 10.1016/0149-7189(82)90074-X. [DOI] [PubMed] [Google Scholar]
  2. Baglin J. (2014). Improving your exploratory factor analysis for ordinal data: a demonstration using FACTOR. Practical Assessment, Research and Evaluation, 19(5). doi: 10.7275/dsep- 4220. [Google Scholar]
  3. Barkham M., Bewick B., Mullin T., Gilbody S., Connell J., Cahill J., Mellor-Clark J., Richards D., Unsworth G., Evans C. (2013). The CORE-10: a short measure of psychological distress for routine use in the psychological therapies. Counselling and Psychotherapy Research, 13(1), 3-13. doi: 10.1080/14733145.2012.729069. [Google Scholar]
  4. Blenkiron P., Hammill C.A. (2003). What determines patients' satisfaction with their mental health care and quality of life? Postgraduate Medical Journal, 79(932):337-40. doi: 10.1136/pmj. 79.932.337. PMID:12840123; PMCID: PMC1742729. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Buffini K. B., Gordon M. (2015). One-to-one support for crisis intervention using online synchronous instant messaging: evaluating working alliance and client satisfaction. British Journal of Guidance and Counselling, 43(1), 105-116. doi: 10.1080/03069885.2014.987723. [Google Scholar]
  6. Brown T. (2015). Confirmatory factor analysis for applied research. New York, Guilford Publications. [Google Scholar]
  7. Carlson M. J., Gabriel R. M. (2001). Patient satisfaction, use of services, and one-year outcomes in publicly funded substance abuse treatment. Psychiatric Services, 52(9), 1230-1236. doi: 10.1176/appi.ps.52.9.1230. [DOI] [PubMed] [Google Scholar]
  8. Connell J., Barkham M. (2007). CORE-10 user manual, version 1.1. Rugby, CORE System Trust & CORE Information Management Systems Ltd. [Google Scholar]
  9. Constantino M. J., Arnkoff D. B., Glass C. R., Ametrano R. M., Smith J. Z. (2011). Expectations. Journal of Clinical Psychology, 67(2), 184-192. doi: 10.1002/jclp.20754. [DOI] [PubMed] [Google Scholar]
  10. Costello A. B., Osborne J. W. (2005). Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment, Research and Evaluation, 10. doi: 10.7275/jyj1-4868. [Google Scholar]
  11. Devilly G. J., Borkovec T. D. (2000). Psychometric properties of the credibility/expectancy questionnaire. Journal of Behavior Therapy and Experimental Psychiatry, 31(2), 73-86. doi: 10.1016/S0005-7916(00)00012-4. [DOI] [PubMed] [Google Scholar]
  12. Donker T., Bennett K., Bennett A., Mackinnon A., Van Straten A., Cuijpers P., Christensen H., Griffiths K. M. (2013). Internet-delivered interpersonal psychotherapy versus internet- delivered cognitive behavioral therapy for adults with depressive symptoms: Randomized controlled noninferiority trial. Journal of Medical Internet Research, 15(5), e82. doi: 10.2196/jmir.2307. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Donovan D. M., Kadden R. M., DiClemente C. C., Carroll K. M. (2002). Client satisfaction with three therapies in the treatment of alcohol dependence: results from Project MATCH. The American Journal of Addiction, 11, 291-307. doi: 10.1080/1055049029008809. [DOI] [PubMed] [Google Scholar]
  14. Draper M., Cohen P., Buchan H. (2001). Seeking consumer views: what use are results of hospital patient satisfaction surveys? International journal for quality in health care, 13, 463-468. [DOI] [PubMed] [Google Scholar]
  15. Egede L.E., Acierno R., Knapp R.G., Walker R.J., Payne E.H., Frueh B.C. (2016) Psychotherapy for Depression in Older Veterans Via Telemedicine: Effect on Quality of Life, Satisfaction, Treatment Credibility, and Service Delivery Perception. Journal of Clinical Psychiatry, 77(12):1704-1711. doi: 10.4088/JCP.16m10951. PMID: 27835713. [DOI] [PubMed] [Google Scholar]
  16. European Foundation for Quality Management. (2013). EFQM Excelence Model 2013. Brussels: EFQM. [Google Scholar]
  17. Falkenström F., Hatcher R. L., Skjulsvik T., Larsson M. H., Holmqvist R. (2015). Development and validation of a 6- item working alliance questionnaire for repeated administrations during psychotherapy. Psychological Assessment, 27(1), 169-183. doi: 10.1037/pas0000038. [DOI] [PubMed] [Google Scholar]
  18. Field A. (2017). Discovering Statistics Using IBM SPSS Statistics (5ºed). London: SAGE Publications. [Google Scholar]
  19. Ferrando P. J., Lorenzo-Seva U. (2017). Program FACTOR at 10: origins, development and future directions. Psicothema, 29(2), 236-240. doi: 10.7334/psicothema2016.304. [DOI] [PubMed] [Google Scholar]
  20. Hanson K., Yip W. C., Hsiao W. (2004) The impact of quality on the demand for outpatient services in Cyprus. Health Economics, 13(12), 1167-1180. doi: 10.1002/hec.898. [DOI] [PubMed] [Google Scholar]
  21. Horvath A. O., Greenberg L. S. 1989. Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology, 36: 223–233. [Google Scholar]
  22. Howard P.B., Rayens M.K., El-Mallakh P., Clark J.J. (2007). Predictors of satisfaction among adult recipients of medicaid mental health services. Archives of psychiatric nursing 21(5):257-69. doi: 10.1016/j.apnu.2007.06.004. PMID: 17904483. [DOI] [PubMed] [Google Scholar]
  23. Hundt Na., Armento M. E. A., Porter B., Cully J., Kunik M. E., Stanley M. (2013). Predictors of treatment satisfaction among older adults with anxiety in a primary care psychology program. Evaluation and Program Planning, 37, 58-63. doi: 10.1016/j.evalprogplan.2013.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Hutchison M., Russell B. S., Gans K. M., Starkweather A. R. (2022). Online administration of a pilot mindfulness-based intervention for adolescents: feasibility, treatment perception and satisfaction. Current Psychology, 1-13. doi: 10.1007/s12144-022-03025-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Jenkins-Guarnieri M. A., Pruitt L. D., Luxton D. D., Johnson K. (2015). Patient perceptions of telemental health: systematic review of direct comparisons to in-person psychotherapeutic treatments. Telemedicine and E-Health, 21(8), 652-660. doi: 10.1089/tmj.2014.0165. [DOI] [PubMed] [Google Scholar]
  26. Kaiser H. F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151. 10.1177/001316446002000116 [DOI] [Google Scholar]
  27. Keleher B., Oakman J. M., Capobianco K., Mittelstaedt W. H. (2019). Basic psychological needs satisfaction, working alliance, and early termination in psychotherapy. Counselling Psychology Quarterly, 32(1), 64-80. doi: 10.1080/09515070.2017.1367271. [Google Scholar]
  28. Kooistra L. C., Ruwaard J., Wiersma J. E., van Oppen P., van der Vaart R., van Gemert-Pijnen J. E. W. C., Riper H. (2016). Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interventions, 4, 61-71. doi: 10.1016/j.invent.2016.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Larsen D. L., Attkisson C. C., Hargreaves W. A., Nguyen T. D. (1979). Assessment of client/patient satisfaction: development of a general scale. Evaluation and Program Planning, 2(3), 197-207. doi: 10.1016/0149-7189(79)90094-6. [DOI] [PubMed] [Google Scholar]
  30. Lorenzo-Seva U., Ferrando P. J. (2006). FACTOR: a computer program to fit the exploratory factor analysis model. Behavior Research Methods, 38(1), 88-91. doi: 10.3758/bf03192753. [DOI] [PubMed] [Google Scholar]
  31. Machado P., Horvath A. (1999). Inventário de aliança terapêutica – WAI. In Simões M. R., Gonçalves M. M., Almeida L. S. (Eds.), Testes e Provas Psicológicas em Portugal (Vol. 2, pp. 87-94). Braga: APPORT/SHO. [Google Scholar]
  32. Matsubara C., Green J., Astorga L. T., Daya E. L., Jervoso H. C., Gonzaga E. M., Jimba M. (2013). Reliability tests and validation tests of the client satisfaction questionnaire (CSQ- 8) as an index of satisfaction with childbirth-related care among Filipino women. BMC Pregnancy and Childbirth, 13, 235. doi: 10.1186/1471-2393-13-235. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Miller L., Weissman M. (2002). Interpersonal psychotherapy delivered over the telephone to recurrent depressives: a pilot study. Depression and Anxiety, 16(3), 114-117. doi: 10.1002/ da.10047. [DOI] [PubMed] [Google Scholar]
  34. Palacios J. E., Richards D., Palmer R., Coudray C., Hofmann S. G., Palmieri P. A., Frazier P. (2018). Supported internet- delivered cognitive behavioral therapy programs for depression, anxiety, and stress in university students: open, non-randomised trial of acceptability, effectiveness, and satisfaction. JMIR Mental Health, 5(4), e11467. doi: 10.2196/11467. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Priebe S, Golden E, McCabe R, Reininghaus U. (2012). Patientreported outcome data generated in a clinical intervention in community mental health care – Psychometric properties. Biomed Central Psychiatr, 12, 113. doi: 10.1186/1471-244X-12-113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Roberts R. E., Atrkisson C. C., Mendias R. M. (1984). Assessing the client satisfaction questionnaire in English and Spanish. Hispanic Journal of Behavioral Sciences, 6(4), 385-396. doi: 10.1177/07399863840064004. [Google Scholar]
  37. Roe D., Dekel R., Harel G., Fennig S., Fennig S. (2006). Clients’ feelings during termination of psychodynamically oriented psychotherapy. Bulletin of the Menninger Clinic, 70(1), 68-81. doi: 10.1521/bumc.2006.70.1.68. [DOI] [PubMed] [Google Scholar]
  38. Rosenvinge J. H., Klusmeier A. K. (2000). Treatment for eating disorders from a patient satisfaction perspective: a Norwegian replication of a British study. European Eating Disorders Review, 8(4), 293-300. doi: 10.1002/1099-0968(200008)8:4<293::AID-ERV346>3.0.CO;2-4. [Google Scholar]
  39. Rumpold G., Doering S., Smrekar U., Schubert C., Koza R., Schatz D. S., Bertl-Schuessler A., Janecke N., Lampe A., Schuessler G. (2005). Changes in motivation and the therapeutic alliance during a pretherapy diagnostic and motivation- enhancing phase among psychotherapy outpatients. Psychotherapy Research, 15(1–2), 117-127. doi: 10.1080/ 10503300512331327092. [Google Scholar]
  40. Sabourin S., Pérusse D., Gendreau P. (1989). Psychometric properties of the French-Canadian version of the client of Mental Health Services Satisfaction Questionnaire (QSC- 8 et QSC-18B). Canadian Journal of Behavioural Science, 21(2), 147-159. doi: 10.1037/h0079853. [Article in French]. [Google Scholar]
  41. Säilä T., Mattila E., Kaila M., Aalto P., Kaunonen M. (2008). Measuring patient assessments of the quality of outpatient care: a systematic review. Journal of Evaluation in Clinical Practice, 14(1), 148-154. doi: 10.1111/j.1365-2753.2007. 00824.x. [DOI] [PubMed] [Google Scholar]
  42. Sales C. M. D., Moleiro C. M. de M., Evans C., Alves P. C. G. (2012). The Portuguese version of the CORE-OM: translation, adaptation, and preliminary study of psychometric properties. Revista de Psiquiatria Clínica, 39(2), 54-59. doi: 10.1590/S0101-60832012000200003. [Article in Portuguese]. [Google Scholar]
  43. Schulte S. J., Meier P. S., Stirling J. (2011). Dual diagnosis clients’ treatment satisfaction - a systematic review. BMC Psychiatry, 11. doi: 10.1186/1471-244X-11-64. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Smeets R. J. E. M., Beelen S., Goossens M. E. J. B., Schouten E. G. W., Knottnerus J. A., Vlaeyen J. W. S. (2008). Treatment expectancy and credibility are associated with the outcome of both physical and cognitive-behavioral treatment in chronic low back pain. Clinical Journal of Pain, 24(4), 305-315. doi: 10.1097/AJP.0b013e318164aa75. [DOI] [PubMed] [Google Scholar]
  45. Smith D., Roche E., O’Loughlin K., Brennan D., Madigan K., Lyne J., Feeney L., O’Donoghue B. (2014). Satisfaction with services following voluntary and involuntary admission. Journal of Mental Health, 23(1), 38-45. doi: 10.3109/ 09638237.2013.841864. [DOI] [PubMed] [Google Scholar]
  46. Silva S., Barbosa E., Salgado J., Cunha C. (2021). Portuguese validation of the credibility/expectancy questionnaire in routine practice. Research in Psychotherapy: Psychopathology, Process and Outcome, 24(1). 10.4081/ripppo.2021.495 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Timmerman M. E., Lorenzo-Seva U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16(2), 209-220. [DOI] [PubMed] [Google Scholar]
  48. Vázquez F. L., Torres Á., Otero P., Blanco V., Clifford Attkisson C. (2019). Psychometric properties of the Castilian Spanish version of the client satisfaction questionnaire (CSQ- 8). Current Psychology, 38(3), 829-835. doi: 10.1007/s12144-017-9659-8. [Google Scholar]
  49. Warnecke I., Reiss N., Werner A., Luka-Krausgrill U., Rohrmann S. (2020). Treatment satisfaction and therapeutic alliance in three different treatment groups for test anxiety. Journal of Cognitive Psychotherapy, 34(2), 131-147. doi: 10.1891/JCPSY-D-18-00027. [Google Scholar]
  50. World Health Organization (2007). People-Centred Health Care: A policy framework. Geneva, Switzerland: WHO Document Production Services. [Google Scholar]

Articles from Research in Psychotherapy : Psychopathology, Process, and Outcome are provided here courtesy of PAGEPress

RESOURCES