Abstract
Despite clear indications of need to improve depression treatment, practical tools that efficiently measure psychotherapy are not available. We developed a patient-report measure of psychotherapy for depression that assesses Cognitive Behavioral (CBT), Interpersonal (IPT), and Psychodynamic therapies. 420 patients with depression from a large managed behavioral health care organization completed the measure. The three subscales measuring CBT, IPT, and Psychodynamic Therapy showed good internal consistency, appropriate item-total correlations, and were supported by a 3-factor structure. Our results suggest that a patient questionnaire is a promising approach for assessing psychotherapy in quality improvement interventions.
Keywords: Measure development, Assessment, Psychotherapy
Mental health care provided in most practice settings uses procedures that are different from the treatments found to be effective in research settings (Institute of Medicine 2001). Quality improvement interventions are effective at increasing rates of evidence-based depression care (Katon et al. 1995, 1996; Wells et al. 2000). Unfortunately, managed behavioral health care settings are unlikely to be able to engage in large scale quality improvement interventions for psychotherapy because no efficient, easily administered measures of the extent to which psychotherapy procedures provided are evidence-based are available. In this study, we present initial findings on the development of a patient-report measure of the extent to which therapists use procedures of Cognitive Behavioral Therapy (CBT), Interpersonal Therapy (IPT), or Psychodynamic Therapy.
Psychotherapy for depression is an appropriate area to begin development of a patient-report measure of psychotherapy procedures. Depression has a high prevalence and early age of onset, making depressive disorders a leading cause of disability worldwide (Murray and Lopez 1996; Lopez et al. 2006). Two forms of well-defined psychotherapy for depression, CBT and IPT, have been well-established as effective interventions for depression (USDHHS 1999). Yet many depressed patients do not receive evidence-based psychotherapy (Wang et al. 2000; Young et al. 2001). Quality improvements efforts greatly improve the outcomes for depression (Gilbody et al. 2003), yet are not feasible for psychotherapy without ways to measure quality care. An efficient measure of the extent to which psychotherapy procedures consistent with CBT or IPT are used could enable managed behavioral health organization to document the impact of quality improvement interventions for psychotherapy for depression.
Current methods to determine if psychotherapy is evidence based are not feasible for quality improvement efforts. Trained observers code audio or videotapes of sessions to determine adherence to a particular psychotherapy. Although not a measure of psychotherapy, caregiver reported adherence to principles of MST have proven to be reliable and valid predictors of child outcome in randomized trials, as well as usual care settings (Schoenwald 2008). In fact, caregiver reports of adherence to principles of MST have proved better predictors than therapist reports of adherence (Schoenwald 2008). In this study, we reviewed existing rating scales and psychotherapy literature to develop a patient self-report measure of Cognitive Behavioral, Interpersonal, and Psychodynamic Therapies that could easily be administered as part of a quality improvement effort. We include Cognitive Behavioral and Interpersonal Therapies because they are well validated forms of care. We include Psychodynamic Therapy because 25% of therapists in practice use this approach (Norcross et al. 2002). The goals of our study were to present preliminary evidence of the psychometric characteristics of our patient-report measure of psychotherapy for depression.
Methods
We sought to develop a patient-report instrument to describe three psychotherapy approaches used in the treatment of depression. The goals of the analyses presented here were to evaluate the (a) psychometric properties of the measure, (b) preliminary data on validity of the scale, and (c) propose a revision of the measure based on these initial evaluations.
Measures
Psychotherapy Techniques
The instrument, the Psychotherapy Practice Scale—Patient Depression Care Version (PPS Patient), was designed to provide a tool to describe the psychotherapeutic techniques used in the treatment of depression from the patient perspective. The instrument focuses on three therapeutic approaches: CBT, IPT, and Psychodynamic Therapy. CBT and IPT techniques were included because these approaches have strong empirical support for the treatment of major depression (Butler et al. 2006; Chambless et al. 1998; Chambless and Ollendick 2001; Frank and Spanier 2006). Psychodynamic Therapy has limited empirical support but was included given the prominence of this orientation in clinical training and practice.
Content and item development was guided by review of clinical literature on efficacy and essential therapy components and existing observation coding tools of adherence and competence (including rating scales used in the NIMH Treatment of Depression Collaborative Research Program). Generated items were reviewed in consultation with at least two clinical experts in each of the three therapeutic approaches to ensure inclusion of key therapeutic techniques essential to each type of therapy. Items were further refined by conducting cognitive interviews with 12 current psychotherapy patients who completed the measure.
The questionnaire includes 30 items to assess techniques that are key components of CBT (11 items), IPT (9 items), and Psychodynamic Therapy (11 items). Item content is provided in Table 1. The instrument asked the patient to rate the frequency that their therapist used a particular technique in the course of treatment on a 7-point scale ranging from ‘never’ (1) to ‘always’ (7). Items were presented in random order.
Table 1.
Item—subscale correlations
| Items | CBT | IPT | Psycho-dynamic | % Missing | Variability | |
|---|---|---|---|---|---|---|
| Cognitive behavioral therapy items | ||||||
| C_Q11 | My therapist helped me to understand the beliefs behind my thinking. (For example, “I am a worthless person” or “If people get to know me, they won’t like me.”) | 0.74a | 0.70 | 0.64 | 3 | 1.42 |
| C_Q13 | My therapist helped me to create statements that I could use to respond to my negative thoughts | 0.75a | 0.68 | 0.62 | 2 | 1.65 |
| C_Q21 | My therapist asked me to do things that I enjoy doing between sessions | 0.63a | 0.58 | 0.49 | 4 | 1.56 |
| C_Q22 | My therapist helped me to develop a schedule of activities | 0.63a | 0.55 | 0.48 | 7 | 1.56 |
| C_Q26 | My therapist worked with me to come up with a plan for each session | 0.71a | 0.60 | 0.50 | 4 | 1.63 |
| C_Q28 | My therapist asked me to try specific things to see if my thoughts were true | 0.70a | 0.60 | 0.58 | 5 | 1.63 |
| C_Q31 | My therapist helped me to use evidence or facts to question my thoughts | 0.74a | 0.71 | 0.66 | 2 | 1.59 |
| C_Q33 | My therapist helped me to see mistakes in my thinking. (For example, all-or-nothing thinking or always fearing the worst will happen.) | 0.73a | 0.72 | 0.67 | 0 | 1.54 |
| C_Q34 | My therapist helped me to understand how my thoughts and feelings are related | 0.75a | 0.76 | 0.72 | 6 | 1.48 |
| C_Q37 | My therapist asked me to do “homework” between sessions. (“Homework” can include activities like writing something down or practicing something you learned during your session.) | 0.61a | 0.49 | 0.48 | 5 | 1.60 |
| C_Q39 | My therapist asked me about the kinds of thoughts I have when I am upset. (For example, “My spouse divorced me because I am a terrible person.”) | 0.69a | 0.68 | 0.65 | 5 | 1.58 |
| Interpersonal psychotherapy items | ||||||
| I_Q10 | My therapist and I discussed how good my relationships with other people are | 0.57 | 0.63a | 0.57 | 2 | 1.30 |
| I_Q14 | My therapist helped me to explore ways I could improve relationships with people in my life | 0.67 | 0.73a | 0.62 | 2 | 1.51 |
| I_Q16 | My therapist and I discussed how I might get along with other people in the future | 0.72 | 0.81a | 0.71 | 0 | 1.51 |
| I_Q18 | My therapist focused more on how I get along with people today than how I got along with people in the past | 0.63 | 0.70a | 0.56 | 2 | 1.53 |
| I_Q19 | My therapist and I discussed conversations that I had with other people | 0.57 | 0.70a | 0.63 | 1 | 1.54 |
| Psychodynamic psychotherapy items | ||||||
| D_Q12 | My therapist had me talk about issues as they came to my mind | 0.59 | 0.57 | .61a | 2 | 1.52 |
| D_Q15 | My therapist helped me to talk about things in my childhood that make it difficult for me to talk about issues today | 0.55 | 0.63 | 0.71a | 1 | 1.64 |
| D_Q17 | My therapist and I explored the deeper meaning of my thoughts and expectations | 0.73 | 0.74 | 0.72a | 5 | 1.54 |
| D_Q23 | My therapist and I discussed how I felt about him/her | 0.50 | 0.49 | 0.42a | 10 | 1.48 |
| D_Q27 | My therapist allowed me enough time to explore topics or ideas that I was interested in | 0.47 | 0.48 | 0.56a | 0 | 1.33 |
| D_Q29 | My therapist helped me to talk about things in my childhood that were very painful | 0.56 | 0.63 | 0.70a | 0 | 1.63 |
| D_Q35 | My therapist and I explored my fears of being rejected by other people | 0.66 | 0.71 | 0.63a | 8 | 1.63 |
| D_Q36 | I chose the issues we discussed in my therapy | 0.31 | 0.35 | 0.45a | 8 | 1.22 |
| I_Q20 | My therapist helped me to explore different ways to make a decision | 0.78 | 0.76a | 0.71 | 0 | 1.59 |
| I_Q24 | My therapist helped me to understand how my problems might be related to my relationships with others | 0.67 | 0.77a | 0.70 | 4 | 1.52 |
| I_Q25 | My therapist and I explored how I could change my social activities | 0.68 | 0.67a | 0.55 | 1 | 1.45 |
| I_Q30 | My therapist and I discussed my ideas about how others should act | 0.53 | 0.64a | 0.61 | 2 | 1.54 |
| I_Q32 | My therapist asked me about the feelings I had when I was with other people | 0.70 | 0.77a | 0.65 | 1 | 1.48 |
| I_Q38 | My therapist and I discussed the good and bad parts of how I got along with others in the past | 0.69 | 0.76a | 0.71 | 6 | 1.49 |
aItems are associated with the original theoretical scale
Patient Demographic Variables
We used administrative data to determine age and gender for patients. Survey data measured patient education.
Clinician Demographic Variables
We used administrative data to determine age, gender, education level and years of clinical experience. Survey data measured ethnic/cultural identity.
Clinician Therapeutic Primary Orientation
Clinicians were asked to select their “main theoretical orientation” when treating depression. Options included Cognitive Behavioral, Interpersonal, psychoanalytic, psychodynamic, supportive, and other. Because many of our clinicians were eclectic, we developed a measure of the extent to which they practiced each of our three major orientations. Using clinician responses on the frequency with which they delivered CBT, IPT and Psychodynamic Therapy, we classified clinicians into low (<25% quartile), medium (25–75% quartile) and high (>75% quartile) categories on each of the three therapies.
Recruitment and Participants
We selected patients from OptumHealth Behavioral Solutions (formerly United Behavioral Health) who were recently diagnosed with major depression, who had at least three outpatient psychotherapy visits within the prior 6 weeks, and who had been treated by a high-volume network provider. A high-volume provider was defined as a clinician (MD, PhD, or MSW) who had treated 10 or more adult patients in the past year. We surveyed 2,417 eligible patients. Patients were sent a cover letter and consent form, patient survey instrument (described below), and an additional consent to allow us to contact their psychotherapist by mail. All patient participants received a $10 gift card. With patient permission, we then mailed survey packets (cover letter, consent form, signed copy of patient consent, and clinician survey instrument) to their treating clinician. All study procedures were approved by the RAND and UCLA Human Subjects Protection Committees.
Analyses
Examination of Psychometric Properties
We examined the internal consistency reliability (Cronbach’s alpha; Cronbach and Warrington,1951) with the goal of achieving at least an alpha of 0.7 for each of the three hypothesized scales. We also examined the correlation between hypothesized composites. We expected that the IPT and Psychodynamic Therapy composites would be most highly correlated given that IPT is derived from Psychodynamic Therapy. We expected the composites to have low to moderate intercorrelations. We examined item total scale correlations which allowed us to determine how highly an item correlated with its hypothesized scale and how that correlation related to the item’s correlation with other scales. To further determine the psychometric properties of each item, we examined the percent missing for the item and the variability of each item.
Examination of Factor Structure
Although we designed our patient report measure to have a three-factor structure, we were uncertain whether the measure would demonstrate this structure in a sample of patients seen by clinicians in practice who were predominantly eclectic in orientation. Thus, the use of exploratory factor analysis seemed the most appropriate analytic technique for this stage of measurement development research (Floyd and Widaman 1995; Reise et al. 2000; Weersing et al. 2002). Exploratory factor analyses using oblique (promax) rotation with varimax prerotation and squared multiple correlations as communality estimates priors were used to examine the initial composites and determine if the three therapeutic approaches would form three correlated subscales. The number of factors was determined using eigenvalues, adequate loadings (<0.40) on one factor with other loadings 0.20 lower on other factors, and the interpretability of the rotated factor pattern matrix.
Preliminary Validity Analyses
We examined whether clinician ratings of their own techniques predicted patient self-rated subscale scores for similar therapeutic orientations. For patient reported scales, we use subscores designed to measure CBT, IPT, and Psychodynamic Therapy. Individual items were standardized (z-scores with a mean of 0 and standard deviation of 1), to ensure that an item’s variability did not impact its weight in the subscore. Scores were transformed into T-scores, with a mean of 50 and standard deviation of 10 for ease of interpretation. We compared scores on each of the three patient-reported subscales with clinician-reported theoretical orientation (low, medium, high) using t-tests. To determine the effect of the three types of clinician reported therapy orientation on patient reported behavior, we used a multivariate analysis of variance (MANOVA) with clinician rated behavior (CBT, IPT, Psychodynamic Therapy) as independent variables and patient reported clinician behavior (CBT, IPT, Psychodynamic Therapy) as dependent variable.
Development of a Short-Form Measure
To determine whether it is possible to shorten the measure, we examined each for poor discrimination, negative impact on internal consistency, percentage of missing, and loading on the wrong factor.
Results
Participants
A total of 420 patients (17.4% response rate) and 159 high volume clinicians responded to the survey. Patients who responded were mostly adults between the age 35 and 54 (53.8%), female (74.2%), and received only psychotherapy (versus psychotherapy and medication management, 71.1% vs. 28.9%). Their clinicians were mostly master’s level therapists (62.4%), with largest percentage falling below age 30 (66.9%).
Evaluation of 30-Item Instrument
Cronbach’s alphas for each subscale were high, CBT = 0.93, IPT = 0.93 and Psychodynamic Therapy = 0.86 (Nunnally and Bernstein 1994). Scales were highly correlated with one another (r = 0.78–0.85). As predicted, the measures of the two dynamic therapies (IPT and Psychodynamic Therapy) were more highly correlated (r = 0.85) as compared with the correlation of CBT to IPT and Psychodynamic Therapy (r = 0.82 and 0.78, respectively).
Psychometric properties evaluated included item-subscale correlations, percentage of missing for each item, and variability of the item (Table 1). Results showed appropriate primary correlations (e.g., CBT technique items correlated most highly with the CBT subscale, etc.), as well as moderate secondary correlations between the subscales. Although, correlations were fairly high among all subscales, the lowest correlation was observed between CBT and Psychodynamic Therapy items. However, the highest correlation was observed for CBT and IPT items, rather than IPT and Psychodynamic Therapy items, as originally hypothesized. The Psychodynamic Therapy subscale had larger percentages of missing items than did the CBT and IPT subscales. Variability of items was relatively similar across scales.
Factor Structure
Table 2 presents the factor loadings, eigenvalues, and percentage of variance for each factor. There were two eigenvalues over one (14.74, 1.22, 0.91, 0.65, 0.54 and so on) with the third eigenvalue approaching one (0.91). When items were assigned to a factor that had standardized regression coefficients of at least 0.40, the three factor solution was interpretable. The three rotated factors accounted for 61% of the total variance. The first factor accounted for 81% of the proportion of total variance and the seven items that loaded on that factor appeared to tap a “structured, behavioral orientation,” most consistent with CBT techniques. However, it did include one item about encouraging social activities, which was originally developed as an IPT item. The second factor accounted for 7% of the proportion of total variance and seven items loaded on this factor. These items were originally Psychodynamic Therapy and IPT items and appeared to reflect a focus on past childhood experiences as well as an examination of relationships and problems, which we called “insight about relationships”. The third factor accounted for 5% of the proportion of total variance and 5 items that reflect a “general talking” factor, where patients had control over their session. Most of these items were from the Psychodynamic Therapy scale, but an item from each of the other two scales associated with “talking” loaded on this factor. There were also items that loaded on more than one factor. Items that loaded on to more than one factor were considered nonspecific with low discrimination, as listed in Table 2. Thus, three factor solution was not necessarily consistent with the original hypothesized orientations, instead they reflected (1) structured specific behavioral factor (most similar to CBT), (2) insight oriented factor (consistent with both Psyhodynamic Therapy and IPT), and (3) unstructured talking factor.
Table 2.
Rotated Factor Loadings from a Principal Axis Factor Analysis of Items from the PPS (N = 420)
| Factors | ||||
|---|---|---|---|---|
| 1 | 2 | 3 | ||
| Items | ||||
| C_Q26 | Come up with a plan for each session | 0.75 | 0.25 | 0.12 |
| C_Q22 | Develop a schedule of activities | 0.70 | 0.25 | 0.08 |
| C_Q28 | Try specific things to see if my thoughts were true | 0.62 | 0.32 | 0.23 |
| C_Q13 | Respond to my negative thoughts | 0.61 | 0.18 | 0. 50 |
| I_Q25 | Explored how I could change my social activities | 0.61 | 0.35 | 0.23 |
| C_Q21 | Do things that I enjoy doing between sessions | 0.60 | 0.11 | 0.36 |
| C_Q37 | Do “homework” between sessions | 0.57 | 0.17 | 0.24 |
| D_Q29 | Talk about things in my childhood that were very painful | 0.17 | 0.70 | 0.31 |
| I_Q30 | Discussed my ideas about how others should act | 0.23 | 0.67 | 0.22 |
| I_Q38 | Discussed the good and bad parts of how I got along with others | 0.39 | 0.63 | 0.33 |
| D_Q15 | Talk about things in my childhood that make it difficult for me to talk about issues today | 0.16 | 0.62 | 0.39 |
| I_Q32 | Feelings I had when I was with other people | 0.44 | 0.60 | 0.30 |
| I_Q24 | Understand how my problems might be related to my relationships with others | 0.40 | 0.57 | 0.37 |
| D_Q35 | Explored my fears of being rejected by other people | 0.34 | 0.55 | 0.38 |
| I_Q16 | Get along with other people in the future | 0.45 | 0.53 | 0.44 |
| D_P12 | Talk about issues as they came to my mind | 0.28 | 0.21 | 0.62 |
| D_Q27 | Explore topics or ideas that I was interested in | 0.12 | 0.23 | 0.60 |
| C_Q34 | Understand how my thoughts and feelings are related | 0.43 | 0.38 | 0.60 |
| I_Q10 | Discussed relationships with other people | 0.32 | 0.30 | 0.51 |
| D_Q36 | Chose the issues we discussed in my therapy | 0.01 | 0.21 | 0.50 |
| Nonspecific items | ||||
| C_Pll | Understand the beliefs behind my thinking | 0.49 | 0.21 | 0.62 |
| I_Q18 | Get along with other people today | 0.44 | 0.40 | 0.35 |
| C_Q31 | Use evidence or facts to question my thoughts | 0.51 | 0.52 | 0.30 |
| D_Q23 | Discussed how I felt about the therapist | 0.40 | 0.41 | 0.07 |
| D_Q17 | Explored the deeper meaning of my thoughts | 0.38 | 0.43 | 0.58 |
| I_Q14 | Explore ways I could improve relationships | 0.44 | 0.34 | 0.51 |
| I_Q19 | Conversations that I had with other people | 0.24 | 0.48 | 0.49 |
| I_Q20 | Explore different ways to make a decision | 0.53 | 0.36 | 0.54 |
| C_Q33 | See mistakes in my thinking | 0.45 | 0.42 | 0.48 |
| C_Q39 | Kinds of thoughts I have when I am upset | 0.43 | 0.40 | 0.44 |
| Eigen value | 14.74 | 1.22 | 0.91 | |
| % of Variance | 81 | 7 | 5 | |
Bold values indicate eigen values above .50
Preliminary Evaluation of Validity
A multivariate analysis of variance (MANOVA) was conducted to determine the effect of the three types of clinician reported therapy orientation (CBT, IPT, and Psychodynamic Therapy) on three dependent variables (patient reported CBT, IPT or Psychodynamic Therapy). Clinician reported therapy orientation was categorized into low, medium, and high for each orientation. Significant differences were found among the clinician reported therapy subscale on the patient reported dynamic subscale (Wilks’ λ = 0.91, F (24,159) = 2.12, P < 0.05), suggesting the clinicians who self rated their therapy as higher on the Psychodynamic subscale were significantly more likely to have patients rate them high on the Psychodynamic subscale as well. Whereas clinician rated subscales of IPT and CBT were not as predictive of patient ratings on the therapy subscales.
Univariate analyses of variance (ANOVAs) for each dependent variable were conducted as follow-up tests to the MANOVA. The ANOVAs of the patient reported Psychodynamic Therapy (F(2,159) = 4.87, P < 0.01) and CBT (F(2,159) = 3.86, P = 0.023) were significant, whereas the Patient Reported IPT was nonsignificant (F(2,159) = 0.87, P = 0.41). Although main effect was not found for IPT, the level of IPT techniques endorsed by clinicians corresponds to the mean patient rated IPT scores, suggesting a similar trend. Table 3 presents the means and standard deviations of the patient reported therapy orientation for the three levels of each clinician reported therapy orientation (CBT, IPT, Psychodynamic Therapy) groups.
Table 3.
Patient mean scores (30 Items) by clinician (Low–Medium–High) behavioral orientation
| Clinician—CBT behavior | Clinician—IPT behavior | Clinician—psychodynamic behavior | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Low N = 46 | Medium N = 73 | High N = 40 | P-value | Low N = 47 | Medium N = 67 | High N = 45 | P-value | Low N = 47 | Medium N = 74 | High N = 38 | P-value | |
| PT Mean CBT | 3.51(1.11) | 3.77(1.04) | 4.13(0.93) | 0.02* | 3.61(1.12) | 3.87(1.08) | 3.83(0.95) | 0.38 | 3.63(1.03) | 3.82(1.10) | 3.90(1.00) | 0.48 |
| PT Mean IPT | 3.82(1.16) | 3.85(1.01) | 4.22(1.18) | 0.17 | 3.74(1.13) | 3.94(1.11) | 4.12(1.04) | 0.27 | 3.76(1.04) | 3.98(1.05) | 4.06(1.26) | 0.42 |
| PT Mean DYN | 3.91(1.11) | 3.91(0.97) | 4.17(1.00) | 0.37 | 3.74(1.14) | 4.07(1.00) | 4.08(0.87) | 0.17 | 3.67(1.03) | 3.98(0.96) | 4.35(1.03) | 0.01** |
* P < .05; ** P < .01
Shortened Version
Based on the psychometric properties of the full scale (see Table 4), we considered a shortened version of this patient self-report measure. We retained items that had best psychometric properties for its hypothesized scale for the shortened version, which resulted in 15 items assessing CBT (6 items), IPT (6 items), and Psychodynamic Therapy (3 items). These items are in bold in Table 4.
Table 4.
Evaluation of 30-item version
| Item text | Poorer discriminationa | Negative impact on internal consistencyb | Higher % missing | Loaded on wrong scale | |
|---|---|---|---|---|---|
| In thinking of your sessions with your therapist, HOW OFTEN: | |||||
| CBT | |||||
| C_Q11 | Understand the beliefs behind my thinking | ✓ | ✓ | ||
| C_Q13 | Respond to my negative thoughts | ||||
| C_Q21 | Do things that I enjoy doing between sessions | ||||
| C_Q22 | Develop a schedule of activities | ||||
| C_Q26 | Come up with a plan for each session | ||||
| CC_Q28 | Try specific things to see if my thoughts were true | ||||
| C_Q31 | Use evidence or facts to question my thoughts | ✓ | |||
| C_Q33 | See mistakes in my thinking | ✓ | |||
| C_Q34 | Understand how my thoughts and feelings are related | ✓ | ✓ | ||
| C_Q37 | Do “homework” between sessions | ||||
| C_Q39 | Kinds of thoughts I have when I am upset | ✓ | ✓ | ||
| IPT | |||||
| I_Q10 | Discussed relationships with other people | ✓ | |||
| I_Q14 | Explore ways I could improve relationships | ✓ | ✓ | ||
| I_Q16 | Get along with other people in the future | ||||
| I_Q18 | Get along with other people today | ✓ | |||
| I_Q19 | Conversations that I had with other people | ✓ | |||
| I_Q20 | Explore different ways to make a decision | ✓ | ✓ | ||
| I_Q24 | Understand how my problems might be related to my relationships with others | ||||
| I_Q25 | Explored how I could change my social activities | ||||
| I_Q30 | Discussed my ideas about how others should act | ||||
| I_Q32 | Feelings I had when I was with other people | ||||
| I_Q38 | Discussed the good and bad parts of how I got along with others | ||||
| DYN | |||||
| D_Q12 | Talk about issues as they came to my mind | ||||
| D_Q15 | Talk about things in my childhood that make it difficult for me to talk about issues today | ✓ | |||
| D_Q17 | Explored the deeper meaning of my thoughts | ✓ | ✓ | ||
| D_Q23 | Discussed how I felt about the therapist | ✓ | ✓ | ||
| D_Q27 | Explore topics or ideas that I was interested in | ||||
| D_Q29 | Talk about things in my childhood that were very painful | ✓ | |||
| D_Q35 | Explored my fears of being rejected by other people | ||||
| D_Q36 | Chose the issues we discussed in my therapy | ✓ | |||
Stronger items are bolded
aCorrelation with its own scale was not at least two standard errors higher than its correlation with other scales in the 30-item matrix
bCronbach’s alpha would increase if item were removed in 30-item model
The shortened instrument demonstrated good psychometric properties and an increased differentiation between factors. We conducted exploratory factor analyses using oblique (promax) rotation with varimax prerotation and squared multiple correlations as communality estimates priors to determine the initial composites (see Table 5). The number of factors was determined using eigenvalues and the interpretability of the rotated factor pattern matrix. There was only one eigenvalue over one (6.85, 0.93, 0.63, 0.19 so on) with the second eigenvalue approaching one (0.93); the average eigenvalue was 0.52. When items were assigned to a factor that had standardized regression coefficients of at least 0.30, the three factor solution was interpretable and all items loaded highly on the corresponding three therapy approaches.
Table 5.
Rotated factor loadings from a principal axis factor analysis of 15 item version from the PPS (N = 540)
| Items | Factors | |||
|---|---|---|---|---|
| 1 | 2 | 3 | ||
| C_Q26 | Come up with a plan for each session | 0.75 | 0.29 | 0.05 |
| C_Q22 | Develop a schedule of activities | 0.67 | 0.27 | 0.08 |
| C_Q28 | Try specific things to see if my thoughts were true | 0.63 | 0.30 | 0.32 |
| I_Q13 | Respond to my negative thoughts | 0.62 | 0.18 | 0.18 |
| C_Q25 | Explored how I could change my social activities | 0.61 | 0.33 | 0.21 |
| C_Q21 | Do things that I enjoy doing between sessions | 0.60 | 0.24 | 0.23 |
| I_Q32 | Feelings I had when I was with other people | 0.37 | 0.69 | 0.19 |
| I_Q38 | Discussed the good and bad parts of how I got along with others | 0.40 | 0.67 | 0.23 |
| I_Q30 | Discussed my ideas about how others should act | 0.19 | 0.67 | 0.20 |
| I_Q16 | Get along with other people in the future | 0.44 | 0.64 | 0.28 |
| I_Q24 | Understand how my problems might be related to my relationships with others | 0.40 | 0.63 | 0.25 |
| I_Q19 | Conversations that I had with other people | 0.23 | 0.60 | 0.38 |
| D_Q27 | Explore topics or ideas that I was interested in | 0.20 | 0.24 | 0.67 |
| D_Q36 | Chose the issues we discussed in my therapy | 0.06 | 0.18 | 0.64 |
| D_Q12 | Talk about issues as they came to my mind | 0.37 | 0.26 | 0.57 |
| Eigen value | 6.85 | 0.93 | 0.63 | |
| % of Variance | 87 | 12 | 8 | |
Bold values indicate eigen values above .50
Discussion
The psychometric and validity analyses of our Psychotherapy Practice Scale—Patient Depression Care Version offer evidence that patients are able to identify psychotherapeutic techniques and that developing a short patient measure of evidence-based care is promising. With further development and testing, this scale measuring the extent to which psychotherapy is evidence-based could be a vital tool in improving quality of psychotherapy in managed mental health care settings. This tool could be a cost effective method for identifying quality care and measuring improvements in providing evidence-based psychotherapy for depression.
This initial study is a solid first step towards developing a brief measure of the extent to which depression care is evidence-based. To develop this measure further, it should be used within ongoing studies of CBT, IPT and Psychodynamic Therapy where coded videotape validation is available to validate the measure. Alternative report formats might also be considered, including forced choose options, which might improve the reliability and validity of the measure. With clear validation, the measure could then be used to better measure care as usual in practice settings, as well as serve as a measure of evidence-based care in quality improvement interventions.
Although this first study of a patient measure of evidence-based care for depression provides useful information, several limitations should be noted. First, only 17% of our patient sample participated in this study. This number is quite typical of survey response rates in managed health care studies, but nonetheless raises questions about the representative nature of our sample. Second, providers in managed behavioral health care centers tend to be eclectic in their approach to psychotherapy. This was true of our clinicians. This prohibited us from being confident that our findings regarding the shortened scale would hold in settings where more rigorous adherence to a particular therapy occurs. Third, while our measure should evaluate if treatment is more consistent with evidence-based care, without including assessment of the intensity and sequencing of intervention approaches, the measure would not be an indication of the fidelity of care to a particular model. Finally, the scales in this study were highly related. The extent to which this represents an accurate representation of eclectic therapy as opposed to a measurement problem within the scale cannot be determined.
Despite limitations, this is the first study to our knowledge to propose a short patient-administered measured of evidence-based care for depression. A short patient-administered measure is particularly appealing because unlike providers, patients would not inherently have a bias towards reporting about evidence-based care as would clinicians who may feel they “should” be providing care according to guidelines. With further validation, this patient measure of evidence-based depression care could be a very promising tool for measuring and improving quality of psychotherapy.
Acknowledgments
We thank the MacArthur Network on Mental Health Policy Research for financial support and intellectual guidance. We would also like to thank the in-kind contribution of staff time, supplies, and the access to members and their providers by OptumHealth Behavioral Solutions. The authors wish to thank Drs. Ellen Frank and Holly Swartz for feedback on IPT items. We thank Joel Becker, PhD and Mark Oakley, PhD for allowing us access to their patients so we could conduct cognitive interviews with them about the measure. The first author’s writing time was funded by through three centers: Resource Centers for Minority Aging Research, Center for Health Improvement of Minority Elderly (RCMAR/CHIME) funded by National Institute of Aging (2P30AG021684-06), UCLA/Drew Project Export funded by the National Center for Minority Health and Health Disparities (NIH/2P20MD000182-06) and UCLA-RAND-USC Center for Partnered Quality Mental Health Care (P30MH082760—01).
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Contributor Information
Jeanne Miranda, Email: mirandaj@ucla.edu.
Kimberly A. Hepner, Email: hepner@rand.org
Francisca Azocar, Email: francisca_azocar@uhc.com.
Greg Greenwood, Email: gregory_l_greenwood@uhc.com.
Victoria Ngo, Email: vngo@mednet.ucla.edu.
M. Audrey Burnam, Email: aburnam@rand.org.
References
- Butler AC, Chapman JE, Forman EM, Beck AT. The empirical status of cognitive therapy: A review of meta-analyses. Clinical Psychology Review. 2006;26(1):7–31. doi: 10.1016/j.cpr.2005.07.003. [DOI] [PubMed] [Google Scholar]
- Chambless DL, Baker MJ, Baucom DH, Beutler LE, Calhoun KS, Crits-Christoph P, et al. Update on empirically validated therapies, II. The Clinical Psychologist. 1998;51(1):3–16. [Google Scholar]
- Chambless DL, Ollendick TH. Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology. 2001;52:685–716. doi: 10.1146/annurev.psych.52.1.685. [DOI] [PubMed] [Google Scholar]
- Cronbach LJ, Warrington WG. Time-limit tests: Estimating their reliability and degree of speeding. Psychometrika. 1951;16:167–188. doi: 10.1007/BF02289113. [DOI] [PubMed] [Google Scholar]
- Floyd FJ, Widaman KF. Factor analysis in the development and refinement of clinical instruments. Psychological Assessment. 1995;7:286–299. doi: 10.1037/1040-3590.7.3.286. [DOI] [Google Scholar]
- Frank E, Spanier C. Interpersonal psychotherapy for depression: Overview, clinical efficacy, and future directions. Clinical Psychology Science and Practice. 2006;2(4):349–369. doi: 10.1111/j.1468-2850.1995.tb00048.x. [DOI] [Google Scholar]
- Gilbody S, Whitty P, Grimshaw J, Thomas R. Educational and organizational interventions to improve the management of depression in primary care: A systematic review. Journal of the American Medical Association. 2003;289(23):3145–3151. doi: 10.1001/jama.289.23.3145. [DOI] [PubMed] [Google Scholar]
- IOM. (2001). Improving the quality of health care for mental and substance-use conditions. Washington, DC: National Academy Press. [PubMed]
- Katon W, Robinson P, Von Korff M, Lin E, Bush T, Ludman E, et al. A multifaceted intervention to improve treatment of depression in primary care. Archives of General Psychiatry. 1996;53(10):924–932. doi: 10.1001/archpsyc.1996.01830100072009. [DOI] [PubMed] [Google Scholar]
- Katon W, Von Korff M, Lin E, Walker E. Collaborative management to achieve treatment guidelines. Impact on depression in primary care. Journal of the American Medical Association. 1995;273(13):1026–1031. doi: 10.1001/jama.273.13.1026. [DOI] [PubMed] [Google Scholar]
- Lopez AD, Methers CD, Ezzati M, Jamison DT, Murray CJL. Global and regional burden of disease and risk factors, 2001: Systematic analysis of population health data. The Lancet. 2006;367(9524):1747–1757. doi: 10.1016/S0140-6736(06)68770-9. [DOI] [PubMed] [Google Scholar]
- Murray CI, Lopez A, editors. The global burden of disease: A comprehensive assessment of mortality and disability from disease, injuries and risk factors in 1990 and projected to 2020. Boston MA: Harvard University Press; 1996. [Google Scholar]
- Norcross JC, Hedges M, Castle PH. Psychologists conducting psychotherapy in 2001: A study of the division 29 membership. Psychotherapy. 2002;39(1):97–102. [Google Scholar]
- Nunnally JC, Bernstein IH. Psychometric theory. 3. New York: McGraw-Hill; 1994. [Google Scholar]
- Reise SP, Waller NG, Comrey AL. Factor analysis and scale revision. Psychological Assessment. 2000;12:287–297. doi: 10.1037/1040-3590.12.3.287. [DOI] [PubMed] [Google Scholar]
- Schoenwald, S. (2008). Toward evidence-based transport of evidence-based treatments: MST as an example. Journal of Child & Adolescent Substance Abuse, 1547-0652, 17(3), 69–91.
- U.S. Department of Health and Human Services . Mental health: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health; 1999. [Google Scholar]
- Wang PS, Berglund P, Kessler RC. Recent care for common mental disorders in the United States: Prevalence and conformance with evidence-based recommendations. Journal of General Internal Medicine. 2000;15(5):284–292. doi: 10.1046/j.1525-1497.2000.9908044.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Weersing VR, Weisz JR, Donenberg GR. Development of the therapy procedures checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child & Adolescent Psychology. 2002;31(2):168–180. doi: 10.1207/S15374424JCCP3102_03. [DOI] [PubMed] [Google Scholar]
- Wells KB, Sherbourne C, Schoenbaum M, Duan N, Meredith L, Unutzer J, et al. Impact of disseminating quality improvement programs for depression in managed primary care: A randomized controlled trial. Journal of the American Medical Association. 2000;283:212–220. doi: 10.1001/jama.283.2.212. [DOI] [PubMed] [Google Scholar]
- Young AS, Klap R, Sherbourne CD, Wells KB. The quality of care for depressive and anxiety disorders in the United States. Archives of General Psychiatry. 2001;1:55–61. doi: 10.1001/archpsyc.58.1.55. [DOI] [PubMed] [Google Scholar]