Abstract
Background
Cognitive behavioral assessment for outcome evaluation was developed to evaluate psychological treatment interventions, especially for counseling and psychotherapy. It is made up of 80 items and five scales: anxiety, well-being, perception of positive change, depression, and psychological distress. The aim of the study was to present the metric qualities and to show validity and reliability of the five constructs of the questionnaire both in nonclinical and clinical subjects.
Methods
Four steps were completed to assess reliability and factor structure: criterion-related and concurrent validity, responsiveness, and convergent–divergent validity. A nonclinical group of 269 subjects was enrolled, as was a clinical group comprising 168 adults undergoing psychotherapy and psychological counseling provided by the Italian public health service.
Results
Cronbach’s alphas were between 0.80 and 0.91 for the clinical sample and between 0.74 and 0.91 in the nonclinical one. We observed an excellent structural validity for the five interrelated dimensions. The clinical group showed higher scores in the anxiety, depression, and psychological distress scales, as well as lower scores in well-being and perception of positive change scales than those observed in the nonclinical group. Responsiveness was large for the anxiety, well-being, and depression scales; the psychological distress and perception of positive change scales showed a moderate effect.
Conclusion
The questionnaire showed excellent psychometric properties, thus demonstrating that the questionnaire is a good evaluative instrument, with which to assess pre- and post-treatment outcomes.
Keywords: outcome evaluation, well-being, treatment effects, patient perception, anxiety, depression, questionnaire
Introduction
The acronym CBA stands for cognitive behavioral assessment. It is not limited to a battery of tests; more specifically, it is an overall approach to clinical assessment. The CBA project was originally conceived by a group of psychologists who practiced cognitive behavioral psychotherapy in general hospitals, psychiatric services, private practice, and university departments. This project produced the first assessment device named the CBA 2.0 primary scales;1–3 the CBA 2.0 was often repeatedly administered at time intervals in order to evaluate the progress of a patient after some psychotherapy sessions or in a follow-up control. Its administration in specific contexts appeared inappropriate, considering the specificity of the goal (ie, to evaluate progress). Hence, the CBA team started with a new project named CBA-OE (outcome evaluation) with the aim of streamlining the outcome of a cure/treatment action in a more focused manner. Unlike the CBA 2.0, the CBA-OE does not simply examine symptoms and psychological discomfort: It also addresses positive constructs such as psychological well-being, adjustment, coping abilities, self-esteem, and self-efficacy.
The aim of the present study is to briefly present the qualities of the psychometric properties of this questionnaire and to show the validity and reliability of its five scales, both in nonclinical and clinical subjects. Criterion-related validity, responsiveness, and convergent validity were also assessed.
Methods
Participants
Before starting the study, the protocol was reviewed and approved by a technical and scientific committee for ethical protection of persons (Azienda ULSS9, Treviso, Italy, Mental Health Department) in conformity with the Declaration of Helsinki. Each subject provided informed written consent before entering the study. Two samples entered the study. A nonclinical group including 269 participants filled in the questionnaire; the characteristics of the nonclinical sample are shown in Table 1.
Table 1.
Characteristics of the nonclinical sample voluntarily involved in the development of the CBA-OE
| Characteristic | Percentage |
|---|---|
| Sex | |
| Male | 43.6 |
| Female | 56.4 |
| Age (years) | |
| 18–30 | 25 |
| 31–50 | 25 |
| 51–65 | 25 |
| >65 | 25 |
| Nationality | |
| Italian | 98.4 |
| European community | 0.8 |
| Non-EC | 0.8 |
| Civil status | |
| Married | 56.6 |
| Single | 34.8 |
| Widow | 5.2 |
| Separated | 2.4 |
| Divorced | 1.2 |
| Living conditions | |
| With other | 92.8 |
| Alone | 6.4 |
| In community | 0.8 |
| Professional status | |
| Employed | 43.6 |
| Retired | 22.8 |
| Student | 13.6 |
| Housekeeper | 8 |
| No work | 6.8 |
| In search for job | 0.4 |
| Other | 4.4 |
| Education | |
| Second grade secondary school | 44.8 |
| First grade secondary school | 27.6 |
| University degrees | 18.8 |
| Primary school | 7.5 |
| None | 1.2 |
Abbreviations: CBA-OE, cognitive behavioral assessment for outcome evaluation; EC, European community.
Furthermore, a clinical group comprising 168 consecutively recruited outpatients (enrolled at Consultorio Familiare Services, Mental Health Departments, and Centers for Addiction of the Italian National Health Service) entered the study. A total of 130 of them were enrolled to test responsiveness and repeatability; they underwent a medium length (from 12 to 20 sessions) manualized (mostly cognitive-behavioral) psychotherapy, and eleven out of 130 also received a combined psychopharmacological intervention. This sample was made up of 50 males and 80 females (mean age =43.2) and the diagnoses, according to the Diagnostic and Statistical Manual of Mental Disorders – Fifth edition (DSM-5),4 were the following: 29.2% anxiety disorders, 23.8% depressive disorders, 9.2% personality disorders, 8.5% bipolar disorders, 8.5% trauma-related disorders, 6.2% obsessive–compulsive disorders, 3.8% schizophrenia spectrum, 3.8% sexual dysfunctions, 3.8% substance-related disorders, and 3.1% somatic symptoms disorders. Among them, 50 patients also completed other self-report measures to assess the CBA-OE convergent validity.
The remaining 38 patients were included in a waiting list to test temporal stability (mean interadministration interval =33.8 days). These were individuals who, when referring to the aforementioned services, had to wait for a psychotherapy because of organizational issues (eg, no immediate availability of operators). They were 15 males and 23 females (mean age =41.6 years), and their DSM-5 diagnoses were the following: 26.3% anxiety disorders, 21.1% depressive disorders, 10.5% personality disorders, 5.3% bipolar disorders, 13.1% trauma-related disorders, 7.2% obsessive–compulsive disorders, 2.6% schizophrenia spectrum disorders, 5.3% sexual dysfunctions, 5.3% substance-related disorders, and 2.6% somatic symptoms disorders.
Instruments
Cognitive behavioral assessment–outcome evaluation
The questionnaire originates from the family of CBA tests1,2,5–7 and comprises contents and items of the CBA-Young form.8 Focus groups were also conducted in order to identify further appropriate items to be included. From the whole set of items, we proceeded to select those evaluated consistent with the measurement of the areas of inherent anxiety, comfort, depression, awareness of positive change, and support. The row list also included items investigating other symptoms of psychological distress and uneasiness. Third, we selected some other items with higher saturation in nonclinical and clinical samples that were collected in a previous study using CBA 2.0 and CBA-Young. With this methodology, the CBA-OE comprised 98 items in its preliminary version. Subsequently to this initial phase, a group of five expert psychotherapists selected and modified the set, eliminating redundant items, formulating all of them in positive wording, and rewriting the text of some to syntactically make them briefer and simpler.
The final version of the CBA-OE is made up of 80 items and is articulated over the following five scales: anxiety (14 items; eg, “I have been upset about trivial things”); well-being (15 items; eg, “I have done things that interested and involved me”); perception of positive change, of getting others’ support, and being able to cope with difficulties (11 items; eg, “I have tried to deal with difficulties rather than avoid them”); depression (19 items; eg, “I have been tormented by feelings of guilt”); psychological distress, or serious symptoms of disorder and poor control over impulses (21 items; eg, “I have felt debased or mocked”). Instructions for the CBA-OE require participants to answer each item by referring to the previous 2 weeks.
The CBA-OE was developed in Italian and then translated in English. Figure S1 presents the layout and a sample of 23 items of the CBA-OE questionnaire. The process of translation was based on Beaton et al’s procedure.9,10 We performed a final stage of adaptation (cognitive debriefing), enrolling 15 people for each language.
Besides the CBA-OE, the following questionnaires were also administered to a subsample of the clinical group in order to test convergent validity.
State–trait anxiety inventory: Y1 and Y2 forms:11,12 The state–trait anxiety inventory: Y1 and Y2 forms (STAI-Y1 and STAI-Y2) are two 20-item self-report instruments assessing state and trait anxiety, respectively; higher scores reflect higher levels of anxiety.
Depression questionnaire:3,13 The depression questionnaire (DQ) comprises 24 dichotomous items (yes/no) evaluating clinical depression. Individuals are asked to identify their current experience of each symptom. Good psychometric properties were reported.
Psychological well-being scale:14,15 The psychological well-being scale (PWB) is a self-report measure made up of 84 items, assessing psychological well-being by evaluating six distinct dimensions: autonomy, environmental mastery, personal growth, positive relations with others, purpose in life, and self-acceptance. Both the original and the Italian version showed good psychometric properties.
Results
Reliability and model testing
Before conducting a multigroup confirmatory analysis, Cronbach’s α for each questionnaire’s dimension was calculated separately in the clinical and in the nonclinical groups in order to further test the reliability of the scales. All scales demonstrated good/excellent reliability values in both groups (anxiety: clinical group α=0.90, nonclinical group α=0.89; well-being: clinical group α=0.91, nonclinical group α=0.90; perception of positive change: clinical group α=0.80, nonclinical group α=0.74; depression: clinical group α=0.91, nonclinical group α=0.91; psychological distress: clinical group α=0.88, nonclinical group α=0.89). Furthermore, correlations between scales were computed separately in the clinical and in the nonclinical group (Table 2): Pearson’s r values ranged between −0.40 and 0.81 in the clinical group and between −0.21 and 0.83 in the nonclinical one.
Table 2.
Intercorrelations between the CBA-OE scales in the clinical and nonclinical groups
| Anxiety | Well-being | Perception of positive change | Depression | Psychological distress | |
|---|---|---|---|---|---|
| Anxiety | −0.55 | −0.40 | 0.79 | 0.69 | |
| Well-being | −0.51 | 0.71 | −0.64 | −0.51 | |
| Perception of positive change | −0.38 | 0.71 | −0.49 | −0.40 | |
| Depression | 0.83 | −0.60 | −0.47 | 0.81 | |
| Psychological distress | 0.74 | −0.26 | −0.21 | 0.78 |
Notes: Correlations for clinical subjects are reported above the diagonal. All the correlation indices are significant (P<0.05).
Abbreviation: CBA-OE, cognitive behavioral assessment for outcome evaluation.
Then, a multigroup confirmatory analysis was performed using the software Lisrel 8.8 (Scientific Software International, Inc., Skokie, IL, USA) to simultaneously test the correlated five-factor theoretical structure on both clinical and nonclinical samples; the maximum likelihood estimator was applied. The confirmatory analysis was conducted on item parcels (made up of 3–7 items) built through random assignment (Figure 1).
Figure 1.
The CBA-OE model of confirmatory factor analysis.
Note: The five factors and 24 parcels.
Abbreviations: CBA-OE, cognitive behavioral assessment for outcome evaluation; A, anxiety; WB, well-being; PC, perception of positive change; D, depression; PD, psychological distress.
First, a baseline model was estimated in order to verify the configural invariance, thus assessing whether both the clinical and nonclinical group showed the same patterns of relationships among observed variables and hypothesized latent constructs. The resulting fit indices demonstrated that the model fits the data well: despite the chi-squared being significant (χ2 SB [Satorra-Bentler scaled chi-square] [484; clinical group: N=168; nonclinical group: N=269] =1,014.29; P<0.01), the other indices (NNFI [non-normed fit index] =0.98; CFI [comparative fit index] =0.98; RMSEA [root mean square error of approximation] =0.07) are more than satisfactory.16
To assess metric invariance of the model, all of the factor loadings were constrained to equality across the two groups.16 Also, this model showed satisfactory fit indices (χ2 SB [508; clinical group: N=168; nonclinical group: N=269] =1,339.91; P<0.01; NNFI =0.98; CFI =0.98; RMSEA =0.07). Furthermore, the difference between CFI of the metric invariance model and the CFI of the configural invariance model was lower than 0.01; thus, the hypothesis of the global metric invariance can be accepted.16 Therefore, the validity of the correlated five-factor model has been proven in both groups. Estimates of factor loadings, error of variances, and correlations between factors of the baseline model are reported in Table 3. All the factor loadings were significant, overall high (the lower one was 0.62 in the clinical group and 0.59 in the nonclinical one), and substantially similar between the two groups. Such results consistently support the theoretical model and its invariance between the two groups.
Table 3.
Factorial saturation and correlations between factors from confirmatory factor analysis
| Anxiety | Well-being | Perception of positive change | Depression | Psychological distress | Error of variance | |
|---|---|---|---|---|---|---|
| A parcel 1 | 0.83 (0.84) | 0.32 (0.29) | ||||
| A parcel 2 | 0.87 (0.87) | 0.24 (0.24) | ||||
| A parcel 3 | 0.79 (0.82) | 0.38 (0.33) | ||||
| A parcel 4 | 0.82 (0.89) | 0.32 (0.21) | ||||
| WB parcel 1 | 0.82 (0.88) | 0.33 (0.23) | ||||
| WB parcel 2 | 0.86 (0.84) | 0.27 (0.30) | ||||
| WB parcel 3 | 0.82 (0.81) | 0.33 (0.34) | ||||
| WB parcel 4 | 0.81 (0.70) | 0.34 (0.51) | ||||
| WB parcel 5 | 0.85 (0.81) | 0.28 (0.34) | ||||
| PC parcel 1 | 0.62 (0.64) | 0.61 (0.59) | ||||
| PC parcel 2 | 0.84 (0.91) | 0.29 (0.18) | ||||
| PC parcel 3 | 0.78 (0.65) | 0.39 (0.57) | ||||
| D parcel 1 | 0.82 (0.82) | 0.33 (0.33) | ||||
| D parcel 2 | 0.88 (0.87) | 0.22 (0.24) | ||||
| D parcel 3 | 0.76 (0.74) | 0.43 (0.46) | ||||
| D parcel 4 | 0.87 (0.87) | 0.24 (0.25) | ||||
| D parcel 5 | 0.78 (0.85) | 0.39 (0.28) | ||||
| PD parcel 1 | 0.75 (0.59) | 0.44 (0.65) | ||||
| PD parcel 2 | 0.73 (0.73) | 0.47 (0.46) | ||||
| PD parcel 3 | 0.79 (0.85) | 0.37 (0.28) | ||||
| PD parcel 4 | 0.77 (0.81) | 0.40 (0.34) | ||||
| PD parcel 5 | 0.66 (0.66) | 0.56 (0.56) | ||||
| PD parcel 6 | 0.69 (0.73) | 0.52 (0.47) | ||||
| PD parcel 7 | 0.75 (0.84) | 0.43 (0.29) |
Notes: In brackets there are estimated variance errors of the nonclinical sample and outside brackets the clinical one. All the estimated structural variances are significant at P<0.05. Cross loadings were fixed at zero ex ante.
Abbreviations: A, anxiety; WB, well-being; PC, perception of positive change; D, depression; PD, psychological distress.
Concurrent and criterion-related validity
We assessed concurrent validity using the outcome from CBA-OE against the targets jointly identified by therapists and patients at the beginning of the intervention. The achievement of individual targets was estimated as the percentage of both clinical (eg, symptom reduction) and functional (eg, return to work) individual goals reached by each patient at the end of the intervention. Concurrent validity data for subjects who underwent psychological treatment (clinical group N=168) were compared to scores obtained by nonclinical participants. The scores were used to assess discriminant validity by testing for differences within the five scales of the CBA-OE: a multivariate analysis of variance, where the five factors were entered as dependent variables and the group (clinical vs nonclinical) was included as the independent variable, was conducted. Results highlighted a significant multivariate difference between the two groups (Wilk’s λ =0.74; F (5,450)=32.15; P<0.001; partial η2=0.26). Univariate statistics showed significant differences between groups in all five factors (Table 4). As expected, the clinical group reported higher scores than the nonclinical one in the anxiety, depression, and psychological distress dimensions, whereas it scored lower than the nonclinical group in the well-being and perception of positive change scales.
Table 4.
Results from univariate ANOVAs on mean scores for each of the CBA-OE factors
| Clinical (N= 168)
|
Nonclinical (N =269)
|
F (1,454) | P-value | η2 | |||
|---|---|---|---|---|---|---|---|
| M | SD | M | SD | ||||
| Anxiety | 1.84 | (0.82) | 1.11 | (0.66) | 113.91 | <0.001 | 0.20 |
| Well-being | 1.40 | (0.68) | 1.90 | (0.59) | 69.66 | <0.001 | 0.13 |
| Perception of positive change | 1.64 | (0.65) | 1.83 | (0.52) | 11.80 | 0.001 | 0.03 |
| Depression | 1.61 | (0.81) | 0.79 | (0.60) | 152.00 | <0.001 | 0.25 |
| Psychological distress | 1.23 | (0.67) | 0.72 | (0.54) | 80.71 | <0.001 | 0.15 |
Abbreviations: ANOVA, analysis of variance; CBA-OE, cognitive behavioral assessment for outcome evaluation; M, mean; SD, standard deviation.
Responsiveness
In order to assess the responsiveness of the CBA-OE, a blind study was conducted: therapists were required to indicate an estimate of the goal reached with the treatment by each of the 130 participants (subgroup of 168) included in the clinical group (% of target hits). All clinical subjects filled in the CBA-OE at the beginning and at the end of the psychological treatment. Results did not reveal differences between males and females in scores across time.
The complete correlational matrix is reported in Table 5. Correlations with the percentage of target hits of judges were moderate for the anxiety (0.58), depression (0.55), and psychological distress (0.52) scales; the correlations with the well-being (−0.45) and the perception of positive change (−0.36) scales were in the small–medium range. Cohen’s d was calculated to estimate the effect size, and conventionally, effect sizes were large ≥0.80; moderate ≥0.50; small ≥0.20. We used the same formula used in the clinical outcomes in routine evaluation outcome measure (CORE-OM)17 validation process: Cohen’s d = (mean after − mean before)/standard deviation before. Observed effect sizes were between the moderate (Cohen’s d =0.63) and large (Cohen’s d =0.90) range.
Table 5.
Scores obtained by the clinical sample (N=130) on the CBA-OE before and after psychotherapy
| Scale | Before treatment
|
After treatment
|
Difference before – after
|
Cohen’s d | Correlations with target | |||
|---|---|---|---|---|---|---|---|---|
| M | SD | M | SD | M | SD | |||
| Anxiety | 25.81 | 10.18 | 16.04 | 9.27 | 9.77 | 9.43 | 0.96 | 0.58 |
| Well-being | 19.08 | 9.51 | 27.69 | 8.33 | −8.60 | 8.80 | −0.90 | −0.45 |
| Perception of positive change | 18.00 | 6.86 | 22.32 | 5.92 | −4.32 | 7.09 | −0.63 | −0.36 |
| Depression | 32.72 | 13.98 | 20.12 | 11.86 | 12.60 | 13.47 | 0.90 | 0.55 |
| Psychological distress | 26.82 | 14.42 | 15.77 | 11.18 | 11.05 | 14.00 | 0.76 | 0.52 |
Abbreviations: M, mean; SD, standard deviation.
Temporal stability
A small group of 38 individuals was included in a “waiting list” and filled in the CBA-OE twice, at least 30 days apart (mean =33 days). At the first evaluation, obtained scores were similar to those reported by the under-treatment group. Scores obtained at the retest were characterized by small Cohen’s ds (range: −0.08 to 0.27). Test–retest correlations were comprised between r=0.75 and r=0.79 (Table 6).
Table 6.
Scores obtained by the clinical “waiting list” (N=38) group before and after treatment
| Scale | Test
|
Retest
|
Delta Test–retest
|
Cohen’s d | Pearson’s r | |||
|---|---|---|---|---|---|---|---|---|
| M | SD | M | SD | M | SD | |||
| Anxiety | 25.58 | 8.17 | 23.66 | 6.60 | 1.92 | 5.28 | 0.24 | 0.76 |
| Well-being | 19.18 | 8.17 | 19.84 | 7.21 | −0.65 | 5.47 | −0.08 | 0.75 |
| Perception of positive change | 18.97 | 5.48 | 20.16 | 5.42 | −1.18 | 3.76 | −0.22 | 0.76 |
| Depression | 33.45 | 10.81 | 30.53 | 10.44 | 2.92 | 6.92 | 0.27 | 0.79 |
| Psychological distress | 27.16 | 11.08 | 26.24 | 8.96 | 0.92 | 6.86 | 0.08 | 0.79 |
Abbreviations: M, mean; SD, standard deviation.
Convergent validity
A subsample of 50 clinical individuals were administered the CBA-OE, the STAI-Y1 and STAI-Y2, and DQ at the beginning of treatment. Then, convergent validity was tested, particularly focusing on the correlations between these measures and the anxiety and depression dimensions of the CBA-OE. As shown in Table 7, high correlations between the STAI-Y1/Y2 and the anxiety scale, as well as between the DQ and the depression scale of the CBA-OE, were observed.
Table 7.
Correlations between the CBA-OE and the STAI-Y1, STAI-Y2, and DQ (N=50)
| Anxiety | Well-being | Perception of positive change | Depression | Psychological distress | |
|---|---|---|---|---|---|
| STAI-Y1 | 0.59 | −0.20 | −0.23 | 0.31 | 0.26 |
| STAI-Y2 | 0.66 | −0.27 | −0.33 | 0.33 | 0.23 |
| DQ | 0.29 | −0.29 | −0.27 | 0.71 | 0.37 |
Abbreviations: CBA-OE, cognitive behavioral assessment for outcome evaluation; STAI-Y1, state-trait anxiety inventory-Y1 form; STAI-Y2, state-trait anxiety inventory-Y2 form; DQ, depression questionnaire.
Finally, to complete the assessment of convergent validity, 40 individuals filled in the CBA-OE and the PWB at the end of the psychological treatment. In particular, high associations between scores on the PWB and scores on the CBA-OE well-being and perception of positive-change dimensions were expected. As reported in Table 8, the stronger correlation emerged between the CBA-OE well-being scale and the PWB self-acceptance subscale (r=0.61), followed by the correlation between the CBA-OE well-being scale and the positive relation with others scale of the PWB (r=0.55). Also, the correlations between the CBA-OE perception of positive change scale and the PWB personal growth (r=0.59), environmental mastery (r=0.55), and purpose in life (r=0.52) scales were in the medium range.
Table 8.
Correlations between the CBA-OE well-being and perception of positive change and the PWB (N=40)
| PWB autonomy | PWB environmental mastery | PWB personal growth | PWB purpose in life | PWB positive relations with others | PWB self-acceptance | |
|---|---|---|---|---|---|---|
| CBA-OE well-being | 0.45 | 0.31 | 0.44 | 0.25 | 0.55 | 0.61 |
| CBA-OE perception of positive change | 0.40 | 0.55 | 0.59 | 0.52 | 0.22 | 0.44 |
Abbreviations: CBA-OE, cognitive behavioral assessment for outcome evaluation; PWB, psychological well-being scale.
Discussion
We have shown that the CBA-OE is characterized by psychometric properties appreciable for clinical use: it has a strong factor structure, good reliability, satisfactory criterion-related and convergent validity, and has the power to discriminate among nonclinical and “suffering-distressed” subjects. The discriminant validity is statistically well documented. In line with expectations, clinical individuals obtained significantly higher values in comparison to those of the nonclinical group in regards to the anxiety, depression, psychological distress dimensions, lower values in well-being, and perception of positive change. Associations between dimensions, particularly between anxiety, depression, and psychological distress, were quite significant in both the clinical and the nonclinical groups. This is not particularly surprising, since depression and anxiety are highly comorbid conditions characterized by both shared and distinctive features; their frequent co-occurrence, as well as the inability of traditional self-report measures to discriminate them, is well known.18,19 Furthermore, according to Clark and Watson’s model,18,20 negative affectivity (a concept similar to that of general distress) could represent a common feature shared by anxiety and depression syndromes, which are also characterized by their own peculiar elements. This could explain the high comorbidity between anxiety and depression and, considering our results, the strong correlations that emerged between these constructs.
Responsiveness reached moderate to large effects; as regard test–retest reliability, the scores obtained by those in a “waiting-list” condition were similar to those reported by individuals included in the treated group at the first evaluation, indicating an equal symptom severity of the two groups. In a similar study, Young21 found wider variations in the scores reported on two subsequent administrations of the CORE-OM on waiting list patients (the Cohen’s d varied from 0.29 to 0.58), indicating that the CORE-OM scores were lower at the start of therapy than they were at the time of the assessment. The period of a waiting list could activate a positive reaction in subjects because of the expectation of a forthcoming psychotherapy.22,23 On the contrary, in light of present results, the current study suggests that the CBA-OE is a sensitive instrument since pre–post differences among scores occurred only as a consequence of a psychotherapeutic treatment, whereas they were not observed in waiting list subjects.
The CBA-OE is characterized by at least two peculiar and robust strengths. On the one hand, it provides brief indicators of classical measures of psychological suffering (the anxiety, depression, and psychological distress scales), thus allowing assessment of outcomes after treatment. On the other hand, it permits having a measure of subjective components, such as well-being and perception of positive change, which do not stand for “healthiness” but can indicate powerful positive awareness and wishing to talk and listen. Both the well-being and perception of positive change dimensions also comprise features related to resilience and strength of the patient, possible starting points that can be used at the beginning of treatment based on the need opened up by the client.
In Italian mental health services, outcome measures to be employed in routine evaluation are still lacking. Indeed, other instruments designed for evaluating outcomes in general psychopathology are available in foreign clinical contexts, such as the outcome questionnaire-424 and the CORE-OM.17,25 Other instruments can also be used, but they assess outcomes in regard to specific disorders rather than generic psychopathology (eg, the patient health questionnaire-926 for depressive symptoms).
Apart from the global assessment of functioning27 and the symptom checklist-90-R,28 which lacks published psychometric data on the Italian version, the CORE-OM has been validated and is currently widely used in the Italian population29 with aims similar to those pertaining to our project on CBA-OE. The CORE-OM is a 34-item self-report measuring four factors: well-being, problems/symptoms, functioning, and risk. By comparing the structural features of the CBA-OE and the CORE-OM, we believe that the CBA-OE is characterized by the following advantages: it includes more items measuring well-being; it allows a specific assessment for anxiety and depression, providing a useful diagnostic measure for clinical psychologists; and it permits collection of more data, because qualitative analysis is possible. Furthermore, the measure of “perception of positive change” is a CBA-OE peculiarity.
Limitations
The main study limitation was the nondifferentiation between the clinical features/diagnoses of the employed clinical sample. Future research should verify the psychometric properties of the CBA-OE in different and homogeneous clinical samples.
Conclusion
The main strengths of the CBA-OE are its simple layout, quick administration, and proper scoring system (Supplementary material) that effectively measures psychotherapeutic outcomes, both in single individuals (Supplementary material) and wider samples of patients. Outcome measures represent a necessary means to measure evidence of quality in psychological care in both private practice and public organizations; hence, a self-administered questionnaire should be useful in indicating a good practice based on self-evaluation.
Supplementary material
CBA-OE SOFTWARE – brief description
The program is accessed through a password that can be personalized by the user.
Personal data and test data are kept in databases (resident in the computer) where information is encrypted and not accessible from outside subjects.
The software is made up of three parts:
Personal data form: the same as the paper-and-pencil version of the CBA-OE, where the patient’s personal data are recorded, including first name, family name, date of birth, disorders suffered from, etc.
Test form: the items and possible answers of the CBA-OE. Figure S1 shows a sample of 23 items of the CBA-OE questionnaire. The patient reads the items and can select an answer alongside (single-choice answer). When input is complete, confirmation is given and data are saved. The software computes the scoring of the CBA-OE and shows the results in a special part of the patient’s private data form.
Report form: a visualization of patients’ profiles. Each report shows the values of scores obtained by the patient in each scale across administrations, and a graphic section shows the patients’ scores on graded scales to make it easier to read the profile and verify treatment efficacy.
The programming language used to develop the CBA-OE software was Java, which allows it to be used on most popular operating systems, including Windows, Mac OS X, and Linux.
In order to run, the Java Virtual Machine (JVM) software must be installed on your computer. If the JVM is not installed on your computer, the installer of the CBA-OE contains the JVM version 6.0, which will be installed on your computer.
Regardless of operating system, the CBA-OE software has a windows user interface that allows the software to be used by those who do not have great computer skills.
Output
The output of the CBA-OE software (the patient’s profile) can be generated in an html or PDF file. The output sheet is divided into three areas: the first area contains data that identifies the patient, such as the identifier, psychological disorder, age, etc; the second area contains the table of scores on the CBA-OE scales; and the third area contains a graphic of the score.
Privacy
For reasons of security of personal data and privacy, access to the software is protected by user name and password. The data and test responses are stored in a database within the computer, in encrypted form.
Implementation in other languages
The software was designed and built in the English language, but the text file can be edited, replacing the labels from English with other languages, to use a version of the software in another language.
The CBA team wishes to start collaborations with colleagues speaking different languages and coming from different countries to create an international milieu of research and professional development.
CBA-OE was originally developed in Italian. English (UK), English (Canadian), French, French Canadian, German, Russian, Ukrainian, and Albanian versions are available for psychometric studies.
Contact address: giorgio.bertolotti@fsm.it
Case report
Peter M; male, 36 years old, married. Father of a 5-year-old boy.
School: degree in Business and administration. Job: Director of a municipal office.
Diagnoses: Major depression form 2 months, without previous depressive episodes present in anamnesis.
Stress events: mother dead; a preferment in his job role.
Motivation to the treatment program: high and fully compliant.
Social resources: good as well personal abilities.
Psychotherapeutic treatment plan: 15 weekly single cognitive behavioral sessions. The cognitive behavioral therapy was based on a self-help manual.1
Pharmacological treatment: started a month before the psychological sessions. It was suspended at the end of psychological treatment.
Outcome: no more depressive symptoms and full recovery of the social life habits and job satisfaction.
The table presents the five CBA-OE scales scores used for monitoring at six steps during psychotherapeutic treatment and at the conclusive session. The Z score refers to normative data obtained by the clinical sample before treatment (N=130). Figure S2 shows that, for each scale, the scores variations in Cohen’s d were assessed from T0, appointment request, to the final session T6. Table S1 shows the raw and Z scores obtained in the five scales from the appointment request to the last session.
Layout and a sample of 23 items of the CBA-OE questionnaire.
Score variations in Cohen’s d from T0, appointment request, to the final session T6.
Table S1.
Raw and Z scores obtained in the five scales from the appointment request to the last session
| T0 (appointment request) | T1 (session 1) | T2 (session 4) | T3 (session 7) | T4 (session 10) | T5 (session 13) | T6 (session 15) | |
|---|---|---|---|---|---|---|---|
| Anxiety | 41 | 39 | 38 | 30 | 24 | 17 | 14 |
| Z score | 1.36 | 1.15 | 1.04 | 0.19 | −0.45 | −1.20 | −1.52 |
| Well-being | 7 | 8 | 18 | 23 | 32 | 37 | 41 |
| Z score | −1.87 | −1.73 | −0.35 | 0.35 | 1.60 | 2.29 | 2.84 |
| Perception of positive change | 7 | 10 | 19 | 28 | 31 | 39 | 41 |
| Z score | −1.71 | −1.24 | 0.19 | 1.61 | 2.08 | 3.35 | 3.67 |
| Depression | 61 | 58 | 46 | 33 | 25 | 13 | 9 |
| Z score | 1.99 | 1.77 | 0.87 | −0.11 | −0.71 | −1.61 | −1.92 |
| Psychological distress | 41 | 38 | 33 | 26 | 19 | 12 | 7 |
| Z score | 0.90 | 0.68 | 0.32 | −0.19 | −0.69 | −1.20 | −1.56 |
Reference
- 1.Leveni D, Michielin P, Piacentini D. Un programma di terapia cognitivo comportamentale. Florence, Italy: Eclipse; 2014. SUPERARE LA DEPRESSIONE. [Google Scholar]
Acknowledgments
The authors express their very great appreciation to Outcome Evaluation – Translators and Adaptation Team
English UK version: George Cremona MD, San Raffaele Scientific Institute, Milano, Italy; Mariaelisa Santonastaso PhD;
English Canadian version: Maggi Stefania PhD, Institute of Interdisciplinary Studies and Department of Psychology, Carleton University, Ottawa, ON K1S 5B6; Elena Gallitto PhD, University of Ottawa; Rosemary Allpress (professional translator);
French version: Pierluigi Graziani Professeur des Universités en Psychologie Clinique et Psychopathologie, Nîmes University, Marie Vittorini (professional translator);
French Canadian version: Maggi Stefania PhD, Institute of Interdisciplinary Studies and Department of Psychology, Carleton University, Ottawa, ON K1S 5B6;
German version: Sylvia von Mackensen, psychologist, University Medical Centre Hamburg-Eppendorf, Hamburg, Germany; Patrizia Donolato, clinical psychologist, Cardiology Division, Bozen, Sud Tirol, Italy;
Russian and Ukraine versions: Oleksandra Yakymets, psychologist, Università Pontificia Salesiana; Oksana Sokolova (professional translator), State Saint Petersburg University, Department of Philology, Italian literature;
Ukraine version: Brent Bezo, PhD candidate, Department of Psychology, Carleton University; Albanian version: Ervis Mitrovica (professional translator – bilingual).
Footnotes
Author contributions
All authors contributed toward data analysis, drafting, and revising the paper and agree to be accountable for all aspects of the work.
Disclosure
The authors report no conflicts of interest in this work.
References
- 1.Bertolotti G, Michielin P, Sanavio E, Vidotto G, Zotti AM. A computerized approach to cognitive behavioral assessment: an introduction to CBA-2.0 primary scales. J Behav Ther Exp Psychiatr. 1990;21:21–27. doi: 10.1016/0005-7916(90)90045-m. [DOI] [PubMed] [Google Scholar]
- 2.Sanavio E, Bertolotti G, Michielin P, Vidotto G, Zotti AM. CBA-2.0 Scale Primarie: Manuale. Firenze, Italy: Organizzazioni Speciali; 1986. [Google Scholar]
- 3.Sanavio E, Bertolotti G, Vidotto G, Zotti AM. Neuroticism and type A behaviour pattern in healthy Italian blue-collar workers. Psychother Psychosom. 1996;65:145–149. doi: 10.1159/000289066. [DOI] [PubMed] [Google Scholar]
- 4.American Psychiatric Association . Diagnostic and Statistical Manual of Mental Disorders. 5th ed. Arlington, VA: American Psychiatric Publishing; 2013. [Google Scholar]
- 5.Carinci F, Nicolucci A, Ciampi A, et al. Role of interactions between psychological and clinical factors in determining 6-month mortality among patients with acute myocardial infarction. Application of recursive partitioning techniques to the GISSI-2 database. Eur Heart. 1997;18:835–845. doi: 10.1093/oxfordjournals.eurheartj.a015349. [DOI] [PubMed] [Google Scholar]
- 6.Gruppo Italiano per lo Studio della Streptochinasi nell’Infarto miocardico (GISSI) Effectiveness of intravenous thrombolytic treatment in acute myocardial infarction. Lancet. 1986;1:397–402. [PubMed] [Google Scholar]
- 7.Zotti AM, Bertolotti G, Michielin P, Sanavio E, Vidotto G. CBA-H Cognitive Behavioural Assessment – Forma Hospital. Manuale. Firenze, Italy: Organizzazioni Speciali; 2010. [Google Scholar]
- 8.Bertolotti G, Bettinardi O, Boz F, et al. Cognitive Behavioural Assessment –Young (CBA-Y). Test per la valutazione del benessere psicologico in adolescenti e giovani adulti. Trento, Italy: Erickson; 2013. Italian. [Google Scholar]
- 9.Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol. 1993;46(12):1417–1432. doi: 10.1016/0895-4356(93)90142-n. [DOI] [PubMed] [Google Scholar]
- 10.Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25:3186–3191. doi: 10.1097/00007632-200012150-00014. [DOI] [PubMed] [Google Scholar]
- 11.Spielberger CD. Manual for the State-Trait Anxiety Inventory (Form Y) Menlo Park, CA: Mind Garden; 1983. [Google Scholar]
- 12.Pedrabissi L, Santinello M. Inventario per l’Ansia di Stato e di Tratto: Nuova Versione Italiana dello STAI – Forma Y: Manuale. Firenze, Italy: Organizzazioni Speciali; 1989. [Google Scholar]
- 13.Bertolotti G, Michielin P, Vidotto G, Zotti AM, Sanavio E. Depression questionnaire (DQ) In: Nezu AM, Ronan GF, Meadows EA, McKlure KS, editors. Practitioner’s Guide to Empirical Based Measures of Depression. Norwell, MA: Kluwer Academic/Plenum Publishers; 2000. pp. 45–47. [Google Scholar]
- 14.Ryff C, Keyes C. The structure of psychological well-being revisited. J Pers Soc Psychol. 1995;69:719–727. doi: 10.1037//0022-3514.69.4.719. [DOI] [PubMed] [Google Scholar]
- 15.Ruini C, Ottolini F, Rafanelli C, Ryff C, Fava GA. La validazione italiana della Psychological Well-being Scale (PWB) Riv Psichiatr. 2003;38:117–130. Italian. [Google Scholar]
- 16.Cheung GW, Rensvold RB. Evaluating goodness-of-fit indexes for testing measurement invariance. Struct Equat Model. 2002;9:233–255. [Google Scholar]
- 17.Barkham M, Mullin T, Leach C, Stiles WB, Lucock M. Stability of the CORE-OM and the BDI-I prior to therapy: evidence from routine practice. J Psychol Psychother. 2007;80:269–278. doi: 10.1348/147608306X148048. [DOI] [PubMed] [Google Scholar]
- 18.Clark LA, Watson D. Theoretical and empirical issues in differentiating depression from anxiety. In: Becker J, Kleinman A, editors. Advances in Mood Disorders, Vol. I. Psychosocial Aspects of Depression. Hillsdale, NJ: Erlbaum; 1990. [Google Scholar]
- 19.Sanderson WC, DiNardo PA, Rapee RM, Barlow DH. Syndrome comorbidity in patients diagnosed with a DSM-III-R anxiety disorder. J Abnorm Psychol. 1990;99:308–312. doi: 10.1037//0021-843x.99.3.308. [DOI] [PubMed] [Google Scholar]
- 20.Clark LA, Watson D. Tripartite model of anxiety and depression: psychometric evidence and taxonomic implications. J Abnorm Psychol. 1991;100:316–336. doi: 10.1037//0021-843x.100.3.316. [DOI] [PubMed] [Google Scholar]
- 21.Young C. What happens when people wait for therapy? Assessing the clinical significance of the changes observed over the waiting period for clients referred to a primary care psychology service. Prim Ment Health Care. 2006;4:113–119. [Google Scholar]
- 22.Bolton D, Williams T, Perrin S, et al. Randomized controlled trial of full and brief cognitive-behaviour therapy and wait-list for paediatric obsessive-compulsive disorder. J Child Psychol Psychiatry. 2011;52:1269–1278. doi: 10.1111/j.1469-7610.2011.02419.x. [DOI] [PubMed] [Google Scholar]
- 23.Steketee G, Frost RO, Tolin DF, Rasmussen J, Brown TA. Waitlist-controlled trial of cognitive behavior therapy for hoarding disorder. Depress Anxiety. 2010;27:476–484. doi: 10.1002/da.20673. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Lambert MJ, Burlingame GM, Umphress V, et al. The reliability and validity of the outcome questionnaire. Clin Psychol Psychother. 1996;3(4):249–258. [Google Scholar]
- 25.Evans C, Connell J, Barkham M, et al. Towards a standardised brief outcome measure: psychometric proprieties and utility of the CORE-OM. Br J Psychiatry. 2002;180:51–60. doi: 10.1192/bjp.180.1.51. [DOI] [PubMed] [Google Scholar]
- 26.Kroenke K, Spitzer RL, Williams JB. The PHQ–9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–613. doi: 10.1046/j.1525-1497.2001.016009606.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Spitzer RL, Gibbon M, Williams JBW, Endicott J. Global assessment of functioning (GAF) scale. In: Sederer LI, Dickey B, editors. Outcomes Assessment in Clinical Practice. Baltimore, MD: Williams and Wilkins; 1994. [Google Scholar]
- 28.Derogatis LR. The SCL-R-90 Manual I: Scoring, Administration and Procedures for The SCL-90. Baltimore, MD: Clinical Psychometric Research; 1977. [Google Scholar]
- 29.Palmieri G, Evans C, Hansen V, et al. Validation of the Italian version of the clinical outcomes in routine evaluation outcome measure (CORE-OM) Clin Psychol Psychother. 2009;16:444–449. doi: 10.1002/cpp.646. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Layout and a sample of 23 items of the CBA-OE questionnaire.
Score variations in Cohen’s d from T0, appointment request, to the final session T6.
Table S1.
Raw and Z scores obtained in the five scales from the appointment request to the last session
| T0 (appointment request) | T1 (session 1) | T2 (session 4) | T3 (session 7) | T4 (session 10) | T5 (session 13) | T6 (session 15) | |
|---|---|---|---|---|---|---|---|
| Anxiety | 41 | 39 | 38 | 30 | 24 | 17 | 14 |
| Z score | 1.36 | 1.15 | 1.04 | 0.19 | −0.45 | −1.20 | −1.52 |
| Well-being | 7 | 8 | 18 | 23 | 32 | 37 | 41 |
| Z score | −1.87 | −1.73 | −0.35 | 0.35 | 1.60 | 2.29 | 2.84 |
| Perception of positive change | 7 | 10 | 19 | 28 | 31 | 39 | 41 |
| Z score | −1.71 | −1.24 | 0.19 | 1.61 | 2.08 | 3.35 | 3.67 |
| Depression | 61 | 58 | 46 | 33 | 25 | 13 | 9 |
| Z score | 1.99 | 1.77 | 0.87 | −0.11 | −0.71 | −1.61 | −1.92 |
| Psychological distress | 41 | 38 | 33 | 26 | 19 | 12 | 7 |
| Z score | 0.90 | 0.68 | 0.32 | −0.19 | −0.69 | −1.20 | −1.56 |