Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Apr 9.
Published in final edited form as: J Pers Assess. 2013 Sep 25;96(3):306–315. doi: 10.1080/00223891.2013.834439

Predicting Response to Intensive Multimodal Inpatient Treatment: A Comparison of Single and Multiple Class Latent Growth Curve Modeling Approaches

Justin D Smith 1, Mark J Van Ryzin 2, J Christopher Fowler 3, Leonard Handler 4
PMCID: PMC3965655  NIHMSID: NIHMS525056  PMID: 24066712

Abstract

In a modest body of research, personality functioning assessed via performance-based instruments has been found to validly predict treatment outcome and, to some extent, differential response to treatment. However, state-of-the-science longitudinal and mixture modeling techniques, which are common in many areas of clinical psychology, have rarely been used. In this article, we compare multilevel growth curve modeling (MLM) and latent class growth modeling (LCGM) approaches with the same dataset to illustrate the different research questions that can be addressed by each method. Global Assessment of Functioning (GAF) scores collected at six points during the course of a long-term multimodal inpatient treatment of 58 severely and persistently mentally ill adults were used to model the trajectory of treatment outcome. Pretreatment personality functioning and other markers of psychiatric severity were examined as covariates in each modeling approach. The results of both modeling approaches generally indicated that more psychologically impaired clients responded less favorably to treatment. The LCGM approach revealed two unique trajectories of improvement (a persistently low group and a higher starting, improving group). Personality functioning and baseline psychiatric variables significantly predicted group membership and the rate of change within the groups. A side-by-side examination of these two methods was found to be useful in predicting differential treatment response with personality functioning variables.

After nearly a half century of psychotherapy outcome research, data suggests that various treatments help people make significant gains and improvements in psychological symptoms and disorders (Lambert & Ogles, 2004). While the evidence base for specific psychotherapy modalities is rapidly expanding, there is a growing recognition that not all clients respond to treatments equally (Clarkin & Levy, 2004). This phenomenon of differential response to treatment is generally examined from three vantage points - that of therapist contributions, client contributions, and interaction factors. Reviews of the literature suggest that client factors, such as the clients’ general level of personality functioning, intelligence, motivation, age, educational level, and socioeconomic status, account for approximately 40% of variance related to the treatment outcome (Lambert, 1992) and may be the most influential of all factors related to change (for a review see Clarkin & Levy, 2004 and Luborsky, Chandler, Auerbach, Cohen, & Bachrach, 1971). While some single client variables are associated with treatment outcome, several research teams (e.g., Exner, 2003; Fowler, Ackerman, Speanburg, Blagys, & Conklin, 2004; Fowler, Smith, & Hilsenroth, 2009; Meyer & Handler, 1997) have attempted to explore the relationship between composite and multifactorial client variables and treatment outcome.

This study examines the way in which client characteristics contribute to response to an intensive multimodal treatment for treatment refractory inpatients. In line with the aims of this Special Series, we compare two special cases of growth mixture modeling: multilevel growth curve modeling (MLM) and latent class growth modeling (LCGM). The goal of the MLM approach is to characterize the mean trajectory of a sample and the individual variability around (1) the starting point (i.e., intercept) and (2) the rate of change over time (i.e., slope). Covariates such as demographic characteristics and psychological assessment scores can be introduced into an MLM to determine their relationship with these two parameters. The MLM approach is also referred to as individual growth curve modeling, hierarchical linear modeling, a mixed effects model, and other labels; yet they all share similar statistical assumptions and characteristics.

The LCGM approach allows for the possibility of distinct subgroups, also called classes, and for relationships among variables to vary across these subgroups (von Eye & Bogat, 2006). The LCGM approach is germane to situations in which modeling a mean trajectory does not accurately describe change in the sample. For example, a sample in which the vast majority of participants either increase or decrease steeply over time on a variable of interest, but very few trajectories occur in between. The LCGM approach is also applicable when the variability if trajectory parameters is nonnormally distributed, as might be the case in a study of treatment outcome where some clients experience little to no benefit while the rest experience widely varying trajectories of improvement. Adding covariates to LCGMs allows the researcher to examine their relationship with (1) membership in a particular subgroup and (2) the influence on the rate of change within the subgroups. An in-depth discussion of these approaches to longitudinal mixture modeling is beyond the scope of this manuscript. A cogent discussion in the context of personality assessment is presented in the introduction to this series by Wright and Hallquist (this issue).

Within a severely and persistently mentally ill sample, we examined the relationship between psychological severity characteristics, as assessed via the Rorschach along with various psychiatric history variables indicative of psychopathology and psychiatric disturbance, with the trajectory of improvement in treatment. Given the limitations of the small sample in this study (58 participants), the emphasis of this article is the presentation and comparison of the statistical methods and less on the results of the study, which we hope will inform inquiry in this area.

The Rorschach and Treatment Process and Outcome

Many authors have drawn parallels between the demand characteristics of the Rorschach and those of the psychotherapy process, particularly noting the commonalities of verbal disclosure of perceptions in a psychologically ambiguous setting that arouses anxiety, and the rapid development of implicit relationship patterns (e.g., Fowler, Hilsenroth, & Handler, 1996; Holt, 2009; Shafer, 1954). Several mental processes involved in completing the Rorschach task parallel the process of expressive-supportive psychotherapy, including challenging the client’s ability to spontaneously engage in the task and articulate perceptions, cognitions, and appraisals (Meyer, 1997). As such, verbal articulation, behavioral spontaneity, and general task engagement can all be influenced by the differential cognitive and emotional capacities of individual clients (Meyer, 1997). Viewing the Rorschach task as a prototype of thin slices of expressive behavior (Ambady & Rosenthal, 1992), it stands to reason that performance assessed during the Rorschach may be related to treatment response, particularly those treatment modalities predominantly utilizing expressive-supportive interventions.

The relationship between Rorschach assessed client characteristics and outcome might be most applicable to understanding long-term trends and change processes. McClelland, Koestner, and Weinberger (1989) found support for the hypothesis that implicitly measured client characteristics were the best predictors of long-term outcomes. Similarly, on the basis of a systematic review of the literature, Viglione (1999) concluded that the Rorschach is associated with behavioral and person-environment outcomes that emerge over time. The meta-analytic findings of Grønnerød (2004) support Viglione’s conclusions by empirically demonstrating that the Rorschach is a valid indicator of personality changes following psychotherapy. Further support for the use of the Rorschach in predicting subsequent treatment response is the validity of the prognostic rating scale (Klopfer, Kirkner, Wisham, & Baker, 1951). Meyer and Handler (1997) conducted a meta-analytic review of 20 effect sizes involving 752 clients and found a moderate effect size (ρ = 0.44) predicting treatment outcome. Similarly, Perry and Viglione (1991) found support for the predictive validity of the Ego Impairment Index (EII; Perry & Viglione, 1991) as a measure of treatment response with a sample of depressed clients treated with antidepressant medication, suggesting that this score assesses psychological elements instrumental to the change process.

Hypotheses

Given previous research in this area, we expect baseline markers of psychiatric severity and quality of representations to be predictive of treatment response. Generally speaking, we expect that greater psychiatric severity will be associated with lower levels of functioning at admission to treatment and lesser rates of improvement over time. We asses psychological disturbance and thought disorder using the EII-3 (Viglione, Perry, Giromini, & Meyer, 2011), object representation differentiation with the mutuality of autonomy scale (MOA; Urist, 1977), and aggressive ideation with Holt’s (Holt, 2009) method for scoring primary process aggression (AGG1). As discussed above, we will first run standard unconditional and conditional models of change (i.e., latent growth models), followed by group-based models (also known as latent class trajectory analysis). Since we have yet to analyze the data for distinct subgroups, we are unable to hypothesize relationships between these variables and group membership or within group change explicitly.

Methods

Participants

Participants in the current study are 58 of the original 77 subjects used in a phase model study of change, thus inclusion criteria, methods, and procedures are described in greater detail elsewhere (Fowler et al., 2004). Inclusion criteria consisted of sustained treatment for a minimum of 10 months, completion of psychological test battery at admission and at follow-up, and medical record data of typewritten therapist monthly progress notes for three contiguous months (before, during and after) administration of baseline and follow-up psychological test battery. The only exclusion criterion was the presence of a degenerative neurological disorder or Asperger’s syndrome (n = 2). The final sample consisted of 58 clients (35 women, 23 men) with an average treatment duration of 16 months (M = 16.0, SD = 4.2), an average age at admission of 28.9 years (SD = 9.6), and 14.8 years of education (SD = 2.0).

Diagnoses were assigned by a clinical team consisting of a board certified and licensed psychologist and psychiatrist based on the longitudinal expert evaluation using all data (LEAD) standard of diagnosis (Pilkonis, Heape, Ruddy, & Serrao, 1991). Clients’ diagnostic profiles and past psychiatric histories are indicative of severe psychopathology and high rates of treatment-resistant psychiatric disorders (Fowler, Plakun, & Shapiro, 2011; Munich & Allen, 2003; Oldham et al., 1995): 48 clients were diagnosed with comorbid Axis I and Axis II disorders, with an average number of Axis I diagnoses of 3.1 (SD = 1.4), 12 were diagnosed with a psychotic disorder; 22 had a comorbid major depression or dysthymia diagnosis; 21 had a polysubstance abuse disorder; 18 carried at least one anxiety disorder; and 14 had an active eating disorder. Other markers indicative of the severity of pathology included the high incidence of past suicide attempts (50%), self-mutilation (64%), previous hospitalizations (M= 2.2, SD = 2.3), psychotic symptoms present at admission (21%), and numerous previous outpatient psychotherapy and pharmacology trials. At admission, approximately 95% of clients were being treated with psychotropic medications, and the majority of these clients experienced limited or no benefit.

Procedure

Psychological assessment

Clients were administered a battery of psychological tests at admission and at an average of 16 months later as an aspect of clinical assessment. The psychological test battery included the Rorschach, Thematic Apperception Test (Morgan & Murray, 1935), and Wechsler Adult Intelligence Scale (Wechsler, 2008), among others. Administration and original scoring of the Rorschach followed the procedures articulated by (Exner, 1993). The third author, (name withheld for review), scored all protocols blind to client identity, diagnosis, and time of administration. To establish preliminary interrater reliability, 20 nonstudy Rorschach protocols were chosen at random from the treatment center archive and scored independently by the third author and an external research consultant who was also blind to all client data. The standardized Rorschach administration (Exner, 1993) assured that the records scored for the reliability trial were representative of the study sample.

Treatment

The treatment program consisted of psychotherapy, family therapy, and an intensive therapeutic milieu program similar to need-adapted treatment protocols (Alanen, Lehtinen, Räkköläinen, & Aaltonen, 1991), with step-down programs similar to that described by Cheisa & Fonagy (2003), including a day treatment program providing continuity of treatment with greater autonomy and lower daily cost.

The provision of psychotherapy consisted of four times weekly, expressive-supportive psychotherapy conducted by licensed and board certified psychologists and psychiatrists with clinical experience ranging from 3 to 30 years. Postdoctoral psychology and psychiatry fellows (those with 3 to 5 years experience) received a minimum of 1 hour of supervision per week for each individual case. The treatment approach was psychoanalytically oriented, broadly defined. Specific attributes of the treatment were not prescribed (i.e., manualized), and formal assessments of therapist adherence and competence were not available. Although all treatments were individualized, basic principles of Long-Term Psychodynamic Psychotherapy (LTPP) were similar to those described by Gabbard (2004) and included specific interventions such as: (a) negotiating and maintaining a therapeutic alliance, (b) approaching psychiatric symptoms as expressions of underlying conflicts related to the past, (c) attending to transference and countertransference phenomena, (d) helping clients articulate unconscious and heavily defended emotions, (e) judicious use of transference and defense interpretations, and (f) implementing a treatment plan that integrates medications, psychotherapy, and milieu modalities (Plakun, 2003).

Measures

Functioning

The primary outcome variable was clients’ global functioning. Information for rating global assessment of functioning (GAF) from the DSM–IV Axis V rating scales (American Psychiatric Association, 1994) was drawn from archival medical records consisting solely of the therapist monthly progress notes. To assure a broad sampling of functioning, six monthly progress notes were rated for baseline and follow- up. GAF scores were given during the month prior to, month of, and month after the administration of a psychological test battery, which occurred in the second month of treatment and the month prior to planned discharge. Thus, providing a baseline and follow-up assessment with three data points. Monthly notes included psychotherapy attendance, mental status exams, issues being addressed in psychotherapy, the nature of participation in hospital treatment programs, and therapist’s interactions with family members. These notes did not include therapist GAF ratings and no reference to this scale was made in the notes.

Prior to rating these scales for the study, five raters participated in 10 hours of group training in which scoring guidelines were reviewed for each of the Axis V scales and practice protocols were reviewed and discussed. For all cases, raters were blind to client identification, program level, and date of note. Random assignment of notes insured that individual raters had minimum repetition of client notes. For reliability, 82 randomly selected monthly progress notes were independently rated by each of the five raters, who were blind to client identity, time, and diagnosis. For these 82 notes, the average of all five raters scores were used in the analyses of change. To assess interrater reliability for all scales used in the study, one-way random effects model intraclass correlation coefficients (ICC [1,1]; Shrout & Fleiss, 1979) were computed. According to traditional benchmarks (Cicchetti, 1994; Cicchetti & Sparrow, 1981), the GAF interrater reliability of .79 is considered to be in the excellent range. This reliability is consistent with previously reported levels of reliability for Axis V scales examining clinician and observer ratings of psychotherapy session data (Hilsenroth et al., 2000).

Rorschach Variables

1. Thought disorder and psychological disturbance

We assessed thought disorder and psychological disturbance using the third iteration of the Ego Impairment Index (EII-3; Viglione et al., 2011). The EII-3 is a revised version of earlier Ego Impairment Indices that have been found to have high construct validity (e.g., Perry & Viglione, 1991; Viglione, Perry, & Meyer, 2003). The EII-3 has been updated to reflect changes in Rorschach scoring in the new Rorschach Performance Assessment System (Meyer, Viglione, Mihura, Erard, & Erdberg, 2011). The EII-3 is a weighted factor score comprised of the following Rorschach scores and variables: distorted form quality (FQ-), the weighted sum of cognitive processing errors (WSum6), representations of adaptive and problematic interactions involving people (Good and Poor Human Representation [GHR and PHR]), distorted perceptions of human movement (M-), and the Critical Contents score of crude and problematic imagery, which includes aggressive movement (Ag), anatomy (An), blood (Bl), explosions (Ex), fire (Fi), sex (Sx), and morbid content (MOR). The number of responses (R) in the protocol was statistically controlled for to reduce distortion based on protocol length. EII-3 was found to have high convergent validity with the EII-1 and EII-2 (Viglione et al., 2011). Internal consistency of the EII-3 in this study was in the excellent range at .87.

2. Object representation differentiation

The MOA (Urist, 1977) assesses the thematic content of stated or implied relationships between animals, inanimate objects, and humans in the Rorschach. The MOA scale assesses the degree of object representation differentiation focused primarily on the developmental progression of separation individuation. More specifically, MOA assesses the relative degree to which internal representations are characterized by engulfing and fused representations to highly differentiation self-other representations. The scale is rated from 1–7, in which scores of 1 and 2 reflect the capacity to construe self and other representations as structurally differentiated and engaged in mutually interactive and parallel activity, scores of 3 and 4 capture dependent and mirroring object relationships and often reveal an emerging loss of autonomy between objects, and points 5, 6, and 7 reflect the loss of separateness and increasing malevolence. The scale has demonstrated excellent reliability (Tuber, 1992), construct validity as a measure of object relation and psychopathology (Bombel, Mihura, & Meyer, 2009), and external validity (Blatt & Ford, 1994; Fowler et al., 2004; Fowler, Hilsenroth, & Handler, 1995; Urist, 1977). In this study, we chose to examine the mean score (MOAX). Interrater reliability was excellent (ICC = .85).

3. Aggressive ideation

We assessed aggressive ideation using Holt’s method for scoring primary process aggression on the Rorschach, referred to as A1 scores in Holt’s system and as AGG1 in this study. AGG1 scores are related to primitive aggressive themes, predominantly murderous or sadomasochistic aggression. Holt’s primary process scoring system has demonstrated high interrater reliability (Fowler et al., 1995) and construct validity in a number of studies (e.g., Blatt & Berman, 1984; Fowler et al., 2004; Hilsenroth, Hibbard, Nash, & Handler, 1993). In this study, interrater reliability was excellent (ICC = .87).

Treatment density

The treatment center implemented a utilization and performance improvement strategy, in which nursing staff members electronically recorded client behavior across multiple domains of functioning, including utilization of services. Nursing staff members were not aware of the purpose of this study, thereby mitigating any systematic bias. Client behavioral data (individual psychotherapy attendance, group therapy attendance, and participation in social activities and informal gatherings) were downloaded from an electronic database from daily activity logs for individual clients after being coded to eliminate identifying information. Treatment density represents a composite score of these sources and indicates the degree to which each client utilized or participated in the available treatment options: the scores do not, however, indicate the quality of clients’ engagement in treatment. Higher scores indicate greater participation. Each treatment modality was given equal weight in the formulation of the overall score.

Psychiatric characteristics

In the process of admission to the treatment facility and the conduct of the initial intake and psychological assessment a number of client characteristics indicative of psychiatric severity were recorded in the client’s medical record. The treatment team described above utilized medical records, information from the referral source, direct observation, and formal psychological assessment results to determine the (1) number of Axis I diagnoses, (2) presence of a diagnosable personality disorder, the determination of (3) double depression (co-occurring major depressive disorder and dysthymic disorder), whether the client had a documented (4) history of suicide attempts, and whether the client had active (5) psychotic symptoms at admission. These indices were based on the consensus of the treatment team and thus no formal reliability statistics are available.

Analysis Plan

To evaluate our hypotheses, we used two modeling approaches: MLM and LCGM. By conducting both modeling approaches, we can (a) examine predictors of response to treatment in the overall sample, and (b) explore the possibility that unique subgroups within the sample may demonstrate unique responses to treatment and that these unique responses may be contingent on different indices of functioning. We conducted the MLM using SAS Proc Mixed (Singer, 1998) and the LCGM using SAS Proc Traj (Jones & Nagin, 2007; Jones, Nagin, & Roeder, 2001) by first estimating a set of unconditional growth analyses for a linear and quadratic trajectory of change in functioning. Once a well-fitting curve was identified, we added covariates to determine the relationship with salient characteristics of each modeling approach (i.e., MLM: intercept, slope; LCGM: class membership, rate of change within class). Since the Rorschach variables were hypothesized predictors, we added EII-3, MOA, and AGG1 to the model to begin this process and continued to add indicators of the clients’ psychiatric severity (e.g., psychosis, number of Axis I diagnoses, history of suicide attempts) and participation in the treatment program, characterized by treatment density. Covariates were z-transformed in all analyses to provide a meaningful interpretation of change when discussing the results.

Results

Preliminary Analyses

Descriptive statistics and correlations with 2-tailed p-values between all the variables included in all the final analyses are presented in Table 1. There were significant correlations between the six measures of functioning across time using the GAF. The predictor variables showed few significant correlations with each other, except for EII-3 with AGG1 and MOA with psychotic symptoms. Similarly, only a few predictor variables were significantly correlated with the six measures of functioning, and those that were significantly correlated occurred mostly in the second and third months of treatment. There was some degree of missing data in our sample (see Table 1 for valid Ns of each variable), but the data were found to be missing completely at random (MCAR) [Little’s (1988) MCAR test, χ2(78) = 89.94, ns], so the missing data likely did not introduce bias into the analyses. Alpha (α) was set to .05.

Table 1.

Correlations and descriptive statistics of variables included in final global and latent trajectory analyses

N = 58
Variable 1 2 3 4 5 6 7 8 9 10 11 12 13
1. Ego Impairment Index (EII-3) - .26 .49** .02 .02 −.07 .25 .14 −.03 −.09 .15 −.06 .12
2. Mutuality of Autonomy (MOA) - .23 .33* −.16 .05 .05 −.20 −.36** −.32* −.25 −.16 −.04
3. Aggression (AGG) - −.06 .19 .19 .23 −.10 .04 −.34* −.01 −.10 .09
4. Psychotic Symptoms - −.18 .11 .04 −.20 −.36** −.12 −.32* −.17 −.21
5. History of Suicide Attempts - .16 −.12 .04 .14 −.19 .16 −.00 .07
6. Treatment Density - −.17 .10 .10 .08 .13 .04 −.02
7. Number of Axis I diagnoses - −.24 −.40** .05 −.15 −.20 −.19
DSM-IV Axis V Global Assessment of Functioning (GAF)
 8. Time 1a - .36** .33* .40** .40** .33*
 9. Time 2a - .26 .24 .28* .29*
 10. Time 3a - .36** .36** .20
 11. Time 4b - .53** .49**
 12. Time 5b - .49**
 13. Time 6b -
Mean 1.09 3.09 .96 .22 .48 3.05 3.14 41.36 42.78 45.37 49.99 50.69 49.73
Standard Deviation 1.34 .93 .139 .42 .50 .37 1.45 9.68 9.34 9.87 11.65 9.96 11.15
N 58 57 57 55 56 58 56 55 56 58 57 57 54
Open in a new tab

Note: correlations were calculated using a Pearson’s r. P-values are based on a two-tailed test of significance. p-values:

*

<.05,

**

<.01.

GAF:

a

measures of functioning at times 1, 2, and 3 were taken at months 1, 2, and 3 for all clients;

b

measures of functioning at times 4, 5, and 6 varied for each client, but were consistently the last three months of treatment.

Multilevel Growth Curve Modeling

We initially estimated an unconditional growth curve with only a linear term; the linear term was significant (B = 2.16, SE = .34, p < .001), suggesting that scores of functioning improved at an average rate of 2.16 points each month. The linear term demonstrated significant variance (1.67, SE = 1.20, p = .08) and thus the associated random effect was retained in the model. In MLM, a one-tailed test of significance is provided for the variance component (Berkhof & Snijders, 2001). Thus, halving the p-value yields a significant variance (p = .04). Model fit indices can be found in Table 2. We then estimated a model with a quadratic term, which was also found to be significant (B = −.41, SE = .19, p < .05), suggesting that the improvement in functioning (the positive linear slope) declined slightly each month in a rate that corresponded to the month of assessment squared. The variance of the quadratic term was essentially zero and thus the associated random effect was removed from the model. Model fit indices can be found in Table 2. We then added the three Rorschach variables (EII-3, MOA, and AGG1) and other covariates to the model to begin a process of identifying significant predictors of slope and intercept. We created interactions between these predictors and the model’s intercept and slope terms (we did not attempt to predict the quadratic term, since the variance was essentially zero and nonsignificant).

Table 2.

Model fit indices

N = 58
BIC AIC AICC
Individual growth model (Proc Mixed)
Unconditional linear model 2258.2 2250.0 2250.1
Unconditional quadratic model 2254.8 2246.5 2246.7
Conditional quadratic model 2136.2 2128.2 2128.3
Latent class growth model (Proj Traj)
Initial unconditional two class model 2287.3 2252.6 2259.5
Optimal unconditional two class model 2255.9 2238.5 2240.2
Unconditional two class model (linear test) 2259.8 2245.3 2246.5
Conditional two class model 2137.2 2103.3 2110.2
Open in a new tab

Note. BIC = Bayesion Information Criteria. AIC = Akaike Information Criteria. AICc = small-sample-size corrected version of AIC. AICc are not provided by Proc Traj; they were manually calculated following the procedures described by Burnham and Anderson (2004).

The model with the best fit included EII-3, MOA, AGG1, psychotic symptoms at admission, and treatment density as covariates. The results of the final model are presented in Table 3 and model fit is found in Table 2. Each of the predictors was significantly related to the intercept of the curve. In terms of the actual magnitude of the effect of each predictor variable and the intercept, one standard deviation in EII-3 and treatment density were related to 2.81 and 2.20 points higher in initial levels of functioning, respectively. Conversely, one standard deviation above the mean on MOA, AGG1, and psychotic symptoms at admission was related to lower initial levels of functioning at 2.73, 2.67, and 2.25 points, respectively. None of the variables tested were significantly associated with the slope parameter.

Table 3.

Results of individual growth model analysis

N = 58
Intercept Parameter Slope Parameter
Covariate B SE (B) β p 95% CI B SE (B) β p 95% CI
Ego Impairment Index (EII-3) 2.81 1.15 .32 .02 .50 | 5.11 −.32 .42 −.11 .45 −1.16 | .52
Mutuality of Autonomy (MOA) −2.73 1.11 −.30 .02 −4.94 | −.41 .64 .40 .20 .11 −.16 | 1.44
Aggression (AGG) −2.67 1.16 −.31 .02 −5.00 | −.35 .34 .41 .11 .41 −.48 | 1.16
Treatment Density 2.20 1.08 .24 .04 .03 | 4.36 −.21 .38 −.06 .59 −.99 | .57
Psychotic symptoms −2.25 1.05 −.24 .04 −4.36 | −.14 −.06 .37 −.02 .87 −.80 | .68
Open in a new tab

Note. SE = standard error. CI = confidence interval. Effect is considered significant if the 95% Confidence Interval does not contain zero.

Latent Class Growth Curve Modeling

We applied group-based trajectory modeling (Nagin, 2005) to determine whether our sample included unique trajectories of treatment response. We then conducted a risk-factor analysis to determine whether indices of functioning predicted membership in the various trajectory groups. This analysis would provide clinicians with broad indicators of how well a given individual may respond to the type of treatment described in this paper. Finally, we conducted a covariate analysis to evaluate whether indices of functioning impacted response to treatment differently within each trajectory. Significant findings here would suggest that the manner in which individuals respond to treatment might be different among the trajectory groups. One advantage of the SAS Proc Traj approach is the ability to model trajectories with differently distributed data. In this analysis, we used CNORM (i.e., censored normal data).

Number of groups

Our first step was to conduct group-based trajectory modeling to identify the appropriate number of groups and their associated trajectories for the response to treatment. As discussed in Nagin (2005), we started with a single group and added one group at a time until we achieved optimal model fit, which is defined as the lowest possible value for the model’s Bayesian Information Criterion (BIC) value. Once the number of groups was defined, we determined the optimal polynomial equation for each group’s trajectory (see Nagin, 2005).

We then evaluated the fit of the model to the data using diagnostics presented by Nagin (2005). First among these is the average posterior probability, which is calculated for each trajectory group j (AvePPj). The ideal situation is that each group has an AvePPj = 1, which would indicate that each individual belongs in a given group with absolute certainty. However, such certainty rarely exists, and as the certainty declines, the AvePPj also declines. Nagin’s (2005) rule of thumb is an AvePPj of at least 0.7 for all groups. The second diagnostic presented in Nagin (2005) is the Odds of Correct Classification, which is calculated for each group j (OCCj). If the model has no predictive power, then OCCj equals one. As the model becomes more predictive, OCCj increases. Nagin’s (2005) rule of thumb is an OCCj of at least 5.0 for all groups. The final diagnostic compares the probability of group assignment (Probj), which is calculated for each group j, with the actual proportion of individuals assigned to each group using the “maximum posterior probability classification rule” (Propj). If individuals were assigned to each group with absolute certainty (i.e., AvePPj = 1 for all groups j), then the probability would equal the proportion assigned using the classification rule. As the assignment error decreases, then the gap between the two figures will grow. According to Nagin (2005), the two figures should have a “reasonably close correspondence” (p. 89), which he defines as a difference of less than 50%.

The model with the lowest BIC value (see Table 2) contained two unique trajectories and demonstrated a good fit to the data (see Table 4). The two trajectories were (1) persistently low, and (2) higher starting, improving (see Figure 1). The higher starting, improving group was large (approximately 88.2% of the sample or about 51 participants) and was rated as experiencing serious symptoms at admission and demonstrated a significant degree of improvement over time (a significant and positive linear change) as well as a reduction or deceleration in improvement near the end of the treatment period (a significant and negative quadratic change). The low group was much smaller (approximately 11.8% of the sample or 7 participants) and was characterized by severe, debilitating symptoms at admission and a lack of treatment response; model terms describing linear and quadratic change in this curve were not significant, indicating a flat trajectory of improvement in functioning.

Table 4.

Diagnostics of model adequacy of latent class growth model

Global Assessment of Functioning
Group AvePPj OCCj Probj Propj |% Dif.|
Low starting, flat trajectory .909a 55.73a .152 .138 9.21a
Higher starting, improving .969a 5.60a .848 .862 1.65a
Open in a new tab

Note. Since probabilities and proportions only range between 0 and 1, they are presented to three decimal places.

a

Exceeds criteria presented in Nagin (2005) as evidence for a well-fitting model.

Figure 1.

Figure 1

Open in a new tab

Trajectories of latent class growth curve model

Risk-factor analysis

We then evaluated whether ego impairment, aggression, and psychotic symptoms predicted group membership (i.e., a “risk-factor analysis”; see Nagin, 2005). We included other variables that could be reasonably assumed to predict treatment outcome, such as the number of Axis I diagnoses, treatment density, and the client’s history of suicide attempts. When predictors of group membership are examined, group-based modeling provides the log odds of the impact of each predictor on the likelihood of membership in each trajectory group relative to a baseline or comparison group. The results from this analysis can be considered a set of binary logistic regression models that predict group membership in the designated comparison group versus each of the other groups in the model. A positive coefficient for a risk factor implies that higher levels of the factor increase the probability of group membership in the specified group relative to the comparison group, and a negative coefficient implies the opposite. When evaluating predictors of group membership, we designated the persistently lower functioning group as the reference. As seen in Table 5, our results suggest that the presence of psychotic symptoms at admission (OR = .18; p < .05) and higher AGG1 scores (OR = .11; p = .07) predict decreased likelihood of membership in the higher starting, improving group; in contrast, higher EII-3 scores (OR = 11.85; p = .05) predict increased likelihood of membership in the higher starting, improving group.

Table 5.

Predictions of latent class trajectory group membership

N = 58
B SE T p OR
Ego Impairment Index (EII-3) 2.48 1.27 1.95 .05 11.85
Aggressive ideation (AGG) −2.26 1.26 −1.79 .07 .11
Psychotic symptoms −1.70 .75 −2.26 .02 .18
Axis I diagnoses −1.18 .86 −1.37 .17 -
Treatment Density 1.21 .87 1.38 .17 -
History of suicide attempts 1.77 1.27 1.40 .16 -
Open in a new tab

Note. Results indicate the likelihood of membership in higher starting, improving group as compared to the persistently low group. SE = standard error. OR = odds ratio. OR is only reported for significant predictors.

Covariate analysis

Next, we evaluated whether MOA, EII-3, and AGG1 predicted deviations from the average treatment response within each trajectory group (i.e., a “covariate” analysis; see Nagin, 2005). A positive coefficient indicates that increased levels of the predictor imply a more positive trajectory of change (slope) within the group, and a negative coefficient implies the opposite. When evaluating the predictors of deviation in the mean curve within each trajectory group using Proc Traj, our results (Table 6) suggest that MOA predicts a significant negative deviation from the mean curve within both groups. Meaning, for each standard deviation increase in MOA, the expected deviation from the mean trajectory of the higher starting, improving group is 1.51 and 4.56 points for the persistently low group. Thus, even though MOA operates in the same manner for both groups, it appears to be a much more significant risk factor for individuals in the persistently low group because it predicts a much less steep trajectory of improvement. Similarly, one standard deviation above the mean on AGG1 predicts a significant and stable negative deviation from the mean curve (4.14 points), but in this case the risk applies only to the persistently low group.

Discussion

The results of the two modeling approaches described in this article result in different conclusions regarding the relationship between client characteristics and treatment response. Although the sample size of this study renders the findings and conclusions as tentative, we present an in-depth interpretation of each modeling approach. We ask the reader to be mindful of the limitations posed by the sample when interpreting and reporting the findings. The results of the MLM analysis were relatively limited in regard to our hypotheses proposing that thought disorder and psychological disturbance, object representation differentiation, and aggressive ideation would be related to clients’ response to treatment. None of the variables examined as covariates significantly predicted the rate of change in functioning over the course of treatment. This finding is likely due to the insignificant variance in the slope parameter of the model, meaning it was not significantly different across individuals, which makes it nearly impossible to identify statistically significant predictors of slope.

The results identifying covariates related to the intercept are largely consistent with theory and our hypothesis that higher scores would be related to lower levels of initial functioning. The surprising finding occurred for the EII-3, for which one standard deviation above the mean was significantly associated with a slightly higher (1.94 points) starting point in GAF score. While this finding contrasts with previous research (Perry & Viglione, 1991; Stokes et al., 2003) relating EII with outcome in less severely disturbed samples, the finding is consistent with research indicating greater psychological impairment, particularly in quality of representations, is associated with treatment adherence (Ackerman, Hilsenroth, Clemence, Weatherill, & Fowler, 2000; Fowler & DeFife, in press; Hilsenroth, Handler, Toman, & Padawer, 1995), and as a moderator of the psychotherapy alliance-outcome relationships (Piper, Ogrodniczuk, & Joyce, 2004).

A second finding worthy of mention is the significant relationship between initial starting value and treatment density. This finding suggests that the higher functioning clients are more likely to participate in the various elements of the treatment program, which could reasonably be assumed to increase the likelihood of benefitting from treatment (i.e., dose effect). Last, the presence of psychotic symptoms at the time of admission was associated with lower initial GAF ratings and seems to be a useful variable to control for when examining scores that assess thought disturbance on the Rorschach.

In this sample it proved fruitful to conduct a latent class growth analysis of our longitudinal data to determine (a) if distinct subgroups existed within the sample and (b) if our hypothesized variables were related to group membership and the rate of change within the group. The two subgroups identified by the analysis were distinct on both their initial levels of functioning and the rate at which they improved over the course of treatment. The results of this analysis suggest that 88.2% of the sample (51 clients) was at least on an upward trajectory, while only 11.8% (7 clients) of the sample showed little to no improvement. This is true at least for those clients who were rated as lower functioning (average GAF score of about 34) compared to those with higher levels of initial functioning (average GAF score of about 43).

Furthermore, the three Rorschach variables hypothesized to be predictive of treatment response, as indicated by both group membership and a significant relationship with the rate of change within group, were significantly associated with both aspects of the LCGM approach. Consistent with our hypothesis, greater aggressive ideation was significantly associated with a decreased likelihood of membership in the higher starting, improving group, as was psychotic symptoms at admission. The finding that thought disorder and psychological disturbance, as assessed with the EII-3, was associated with increased likelihood of membership in the higher starting, improving group suggests that this variable taps into unique client characteristics, which while controlling for other indicators of severe psychopathology, are indicative of a positive response to the type of treatment received by the clients in this sample. This finding was however in the unexpected direction and begs for further research. However, more severe psychopathology being predictive of a more positive response to treatment does not appear to be true for clients with higher degrees of primitive aggressive ideation and poorer object differentiation. These two latter factors may inhibit clients from actively engaging in the interpersonal and relational aspects of this treatment program; thus, reducing their therapeutic impact. An alternate hypothesis to consider is that the interpersonal and interpretive focus of the treatment program may not be suitable for patients with overt psychotic symptoms and high degrees of aggressive ideation.

The analysis of these covariates with slope change within the groups reveals a consistent picture: Greater object representation differentiation is associated with a lesser rate of improvement in both groups; aggressive ideation is predictive of the same trend, but only in the persistently low group; and thought disorder and psychological disturbance is associated with an increasingly greater rate of change over time, but only in the persistently low group. These findings indicate that some aspect or characteristics of the client captured by the EII-3 result in improved responsivity to this type of treatment.

Our results differ somewhat from the findings of Perry and Viglione (1991), which found that higher scores on the EII-1 were predictive of poorer response to the intervention. Thus, the results of this study do not support the interpretation of the EII-3 as an index of psychological impairment and thought disorder in the same way as previous research. However, the samples are very different (e.g., unipolar depression versus severe and persistently mentally ill) as is the treatment modality (e.g., antidepressant medication versus intensive multimodal inpatient) and the Rorschach scoring systems used to develop the different indices. The EII-3 was developed based on the administration and scoring procedures of the Rorschach Performance Assessment System (Meyer et al., 2011), while earlier iterations were derived from the Comprehensive System, as was done in this study. The mean score of this variable in our sample (1.09) is quite high compared to non-patient groups, supporting the validity of the EII-3 in the assessment of severe psychopathology. Meyer and Handler (1997) concluded that less developed ego strength and object representation differentiation assessed on the Rorschach using the prognostic rating scale was predictive of poorer outcomes overall. Our study controls for other factors that could conceivably be related to the EII-3, but have not been controlled for in previous research, which is a strength and could conceivable unveil the unique aspects of the EII-3 that are associated with treatment response. Specifically, AGG1 and psychotic symptoms, which could result in the unique variance accounted for by the EII-3 being largely based on the quality of representation.

Exploratory post-hoc analysis of group differences

In an attempt to begin to explain this somewhat unexpected finding, we conducted an exploratory post-hoc analysis of client diagnostic characteristics in each group. As one of the groups contained a small proportion of the overall sample, these analyses are included more as a means of illustrating this type of inquiry, rather than an attempt to draw conclusions about the characteristics of the two groups. After exporting the group membership probability from SAS Proc Traj, using the posterior probability of most likely group membership of each participant, and creating a group membership variable, we compared the groups on the variables included in the original analytic strategy and some additional variables. We selected new variables consistent with our belief that the diagnostic characteristics of the persistently low group were more severe and in the psychotic spectrum compared to the higher starting, improving group, which we believed would contain more personality disorder diagnoses. Given the small sample size overall, and the very small number of clients in the persistently low group, we decided to focus on the magnitude of the effect, rather than inferential mean comparisons (e.g., t-test, ANOVA), which are dependent on power and sample size to determine significance (see Cohen, 1988, 1994). The results of this analysis provide evidence to support our hypothesis. The lower trajectory group was more likely to have psychotic symptoms at admission (R2 = .04, Means: persistently low group = .43, higher starting, improving group = .19), more Axis I diagnoses (R2 = .10, M: 4.25, 2.96), and higher rates of double depression, defined as dysthymic disorder plus Major Depressive disorder (R2 = .04, M: .63, .35), while the higher starting, improving group was more likely to have a personality disorder diagnosis (R2 = .03, M: .63, .84).

These post-hoc findings suggest that perhaps the treatment strategy at the Austen Riggs Center is more successful at engaging clients whose problems center around quality of object representations, which is consistent with disorders of personality, and less so with clients with high degrees of interpersonal detachment, such as those with active psychotic symptoms. The lower group therefore might be less amenable to the heavy reliance on interpersonal skills used in this treatment model.

Limitations

The chief limitation of this study concerns the characteristics of the sample and the treatment program, both of which are relatively homogenous on a number of dimensions and therefore limit the generalizability of our findings to other samples. For example, this sample is comprised of 58 participants with severe and persistent mental illness that can be classified as treatment refractory, meaning prior treatment programs and modalities have had little to no enduring effect. Clients with this history require an intensive, multimodal approach of a residential treatment center that is, by design, geared toward treating clients who have heretofore been unable to benefit from treatment (Krikorian & Fowler, 2008). Examining these patterns among clients with severe and persistent mental illness may not generalize well to higher functioning clients; however, there is a growing demand for outcomes research on serious mental illness due to greater awareness of its prevalence, cost, and burden (Druss & Bornemann, 2010; World Health Organization (WHO), 2004). Last, the sample size of 58 is generally considered small in most research situations. However, a sample size of 58 is not uncommon for single-site treatment studies and is not outside the bounds of what multilevel modeling approaches are capable of analyzing (Snijders & Bosker, 1999), which indicates the statistical trustworthiness of the results. Still, the results of this study should be viewed as preliminary.

Conclusions

In recent years, a growing awareness that not all patients respond positively to interventions has led to greater interest in exploring moderators and mediators of change in order to better comprehend what treatment interventions work best for specific patient types (Roth & Fonagy, 1996). For decades researchers were primarily focused on identifying treatment interventions that work by focusing on average change in symptomatology within and between randomized clinical trials, with minimal regard for the individual differences. Group-based latent trajectory analysis provide researchers with powerful and flexible tools for exploring differential treatment response among individuals and in defining subgroups within larger outcome study samples. The comparison of these two analytic methods suggests that latent class growth modeling approaches are important when attempting to illuminate the subtle differences that result in treatment responsiveness. Our study results indicate that Rorschach variables indicative of more severe psychological disturbance significantly predict the level of functioning at admission and are related to the effectiveness of the treatment in improving functioning over time. Our findings are important for the interpretation of the EII-3, as our findings are not consistent with previous research, and the potential utility of the Rorschach in determining the appropriateness of a given treatment modality or treatment program for specific clients.

Acknowledgments

Justin Smith and Mark Van Ryzin received support from National Institute of Mental Health research training grant T32 MH20012, awarded to Elizabeth Stormshak.

Contributor Information

Justin D. Smith, Child and Family Center, University of Oregon

Mark J. Van Ryzin, Child and Family Center, University of Oregon

J. Christopher Fowler, The Menninger Clinic, Baylor College of Medicine.

Leonard Handler, University of Tennessee Knoxville.

References

  1. Ackerman SJ, Hilsenroth MJ, Clemence AJ, Weatherill R, Fowler JC. The effects of social cognition and object representation on psychotherapy continuation. Bulletin of the Menninger Clinic. 2000;64(3):386–408. [PubMed] [Google Scholar]
  2. Alanen YO, Lehtinen K, Räkköläinen V, Aaltonen J. Need-adapted treatment of new schizophrenic patients: Experiences and results of the Turku Project. Acta Psychiatrica Scandinavic. 1991;83(5):363–372. doi: 10.1111/j.1600-0447.1991.tb05557.x. [DOI] [PubMed] [Google Scholar]
  3. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 4. Washington, DC: Author; 1994. [Google Scholar]
  4. Berkhof J, Snijders TAB. Variance component testing in multilevel models. Journal of Educational and Behavioral Statistics. 2001;26(2):133–152. [Google Scholar]
  5. Blatt SJ, Berman WI. A methodology for the use of the Rorschach in clinical research. Journal of Personality Assessment. 1984;48(3):226–239. doi: 10.1207/s15327752jpa4803_1. [DOI] [PubMed] [Google Scholar]
  6. Blatt SJ, Ford RQ. Therapeutic change: An object relations perspective. New York: Plenum; 1994. [Google Scholar]
  7. Bombel G, Mihura JL, Meyer GJ. An examination of the construct validity of the Rorschach Mutuality of Autonomy (MOA) Scale. Journal of Personality Assessment. 2009;91(3):227–237. doi: 10.1080/00223890902794267. [DOI] [PubMed] [Google Scholar]
  8. Burnham KP, Anderson DR. Multimodel inference: Understanding AIC and BIC in model selection. Sociological Methods & Research. 2004;33(2):261–304. [Google Scholar]
  9. Chiesa M, Fonagy P. Psychosocial treatment for severe personality disorder: 36-month follow-up. British Journal of Psychiatry. 2003;183(4):356–362. doi: 10.1192/bjp.183.4.356. [DOI] [PubMed] [Google Scholar]
  10. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment. 1994;6:284–290. [Google Scholar]
  11. Cicchetti DV, Sparrow SS. Developing criteria for establishing the interrater reliability of specific items in an inventory. American Journal of Mental Deficiency. 1981;86:127–137. [PubMed] [Google Scholar]
  12. Clarkin JF, Levy KN. Research on client variables in psychotherapy. In: Lambert MJ, editor. Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change. 5. New York: John Wiley & Sons; 2004. pp. 194–226. [Google Scholar]
  13. Cohen J. Statistical power analysis for the bahavioral sciences. 2. Hillsdale, NJ: Erlbaum; 1988. [Google Scholar]
  14. Cohen J. The earth is round (p < .05) American Psychologist. 1994;49:997–1003. [Google Scholar]
  15. Druss BG, Bornemann TH. Improving health and health care for persons with serious mental illness. JAMA: The Journal of the American Medical Association. 2010;303(19):1972–1973. doi: 10.1001/jama.2010.615. [DOI] [PubMed] [Google Scholar]
  16. Exner JE., Jr . Basic foundations. 3. Vol. 1. New York: Wiley; 1993. The Rorschach: A Comprehensive System. [Google Scholar]
  17. Exner JE., Jr . The Rorschach: A comprehensive system. 4. New York: Wiley; 2003. [Google Scholar]
  18. Fowler JC, Ackerman SJ, Speanburg S, Blagys M, Conklin AC. Personality and symptom change in treatment-refractory inpatients: Evaluation of the phase model of change using Rorschach, TAT, and DSM-IV Axis V. Journal of Personality Assessment. 2004;83:306–322. doi: 10.1207/s15327752jpa8303_12. [DOI] [PubMed] [Google Scholar]
  19. Fowler JC, DeFife J. Quality of object representations predicting service utilization in long term residential treatment. Psychotherapy. doi: 10.1037/a0029566. (in press) [DOI] [PubMed] [Google Scholar]
  20. Fowler JC, Hilsenroth MJ, Handler L. Early memories: An exploration of theoretically derived queries and their clinical utility. Bulletin of the Menninger Clinic. 1995;59(1):79–98. [PubMed] [Google Scholar]
  21. Fowler JC, Hilsenroth MJ, Handler L. Two methods of early memories data collection: An empirical comparison of the projective yield. Assessment. 1996;3(1):63–71. [Google Scholar]
  22. Fowler JC, Plakun EM, Shapiro ER, editors. Treatment resistance. New York, NY: Norton Professional Books; 2011. [Google Scholar]
  23. Fowler JC, Smith JD, Hilsenroth MJ. Complex treatment resistant psychiatric disorders: Assessment of severity and impairment. Paper presented at the New England Society for Psychotherapy Research; Williamstown, MA. 2009. [Google Scholar]
  24. Gabbard GO. Long-term psychodynamic psychotherapy: A basic text. Arlington, VA: American Psychiatric Publishing, Inc; 2004. [Google Scholar]
  25. Grønnerød C. Rorschach assessment of changes following psychotherapy: A meta-analytic review. Journal of Personality Assessment. 2004;83(3):256–276. doi: 10.1207/s15327752jpa8303_09. [DOI] [PubMed] [Google Scholar]
  26. Hilsenroth MJ, Ackerman SJ, Blagys MD, Baumann BD, Baity MR, Smith SR, Holdwick DJ., Jr Reliability and validity of DSM-IV axis V. American Journal of Psychiatry. 2000;157(11):1858–1863. doi: 10.1176/appi.ajp.157.11.1858. [DOI] [PubMed] [Google Scholar]
  27. Hilsenroth MJ, Handler L, Toman KM, Padawer JR. Rorschach and MMPI-2 indices of early psychotherapy termination. Journal of Consulting and Clinical Psychology. 1995;63(6):956–965. doi: 10.1037//0022-006x.63.6.956. [DOI] [PubMed] [Google Scholar]
  28. Hilsenroth MJ, Hibbard SR, Nash MR, Handler L. A Rorschach study of narcissism defense, and aggression in borderline, narcissistic, and Cluster C personality disorders. Journal of Personality Assessment. 1993;60(2):346–361. doi: 10.1207/s15327752jpa6002_11. [DOI] [PubMed] [Google Scholar]
  29. Holt RR. Primary process thinking: Theory, measurement, and research. Lanham, MD: Jason Aronson; 2009. [Google Scholar]
  30. Jones BL, Nagin DS. Advances in group-based trajectory modeling and a SAS procedure for estimating them. Sociological Methods & Research. 2007;35:542–571. [Google Scholar]
  31. Jones BL, Nagin DS, Roeder K. A SAS procedure based upon mixture models for estimating developmental trajectories. Sociological Methods & Research. 2001;29:374–393. [Google Scholar]
  32. Klopfer B, Kirkner F, Wisham W, Baker G. Rorschach Prognostic Rating Scale. Journal of Projective Techniques and Personality Assessment. 1951;15:425–428. [PubMed] [Google Scholar]
  33. Krikorian SE, Fowler JC. A view from Riggs: Treatment resistance and patient authority—VII. A team approach to treatment resistance. Journal of the American Academy of Psychoanalysis and Dynamic Psychiatry. 2008;36(2):353–373. doi: 10.1521/jaap.2008.36.2.353. [DOI] [PubMed] [Google Scholar]
  34. Lambert MJ. Psychotherapy outcome research: Implications for integrative and eclectical therapists. In: Norcross JC, Goldfried MR, editors. Handbook of psychotherapy integration. New York: Basic Books; 1992. pp. 94–129. [Google Scholar]
  35. Lambert MJ, Ogles BM. The efficacy and effectiveness of psychotherapy. In: Lambert MJ, editor. Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change. 5. New York: John Wiley & Sons; 2004. pp. 139–193. [Google Scholar]
  36. Little RJA. A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association. 1988;83:1198–1202. [Google Scholar]
  37. Luborsky L, Chandler M, Auerbach AH, Cohen J, Bachrach HM. Factors influencing the outcome of psychotherapy: A review of quantitative research. Psychological Bulletin. 1971;75(3):145–185. doi: 10.1037/h0030480. [DOI] [PubMed] [Google Scholar]
  38. McClelland DC, Koestner R, Weinberger J. How do self-attributed and implicit motives differ? Psychological Review. 1989;96(4):690–702. [Google Scholar]
  39. Meyer GJ. On the integration of personality assessment methods: The Rorschach and the MMPI. Journal of Personality Assessment. 1997;68(2):297–330. doi: 10.1207/s15327752jpa6802_5. [DOI] [PubMed] [Google Scholar]
  40. Meyer GJ, Handler L. The ability of the Rorschach to predict subsequent outcome: A meta-analysis of the Rorschach Prognostic Rating Scale. Journal of Personality Assessment. 1997;69(1):1–38. [Google Scholar]
  41. Meyer GJ, Viglione DJ, Mihura J, Erard RE, Erdberg P. Rorschach Performance Assessment System: Administration, coding, interpretation, and technical manual. Toledo, OH: Rorschach Assessment System; 2011. [Google Scholar]
  42. Morgan CD, Murray HH. A method for investigating fantasies: the thematic apperception test. Archives of Neurology and Psychiatry. 1935;34:289–306. [Google Scholar]
  43. Munich RL, Allen JG. Psychiatric and sociotherapeutic perspectives on the difficult-to-treat patient. Psychiatry: Interpersonal and Biological Processes. 2003;66(4):346–357. doi: 10.1521/psyc.66.4.346.25438. [DOI] [PubMed] [Google Scholar]
  44. Nagin DS. Group-based modeling of development. Cambridge, MA: Harvard University Press; 2005. [Google Scholar]
  45. Oldham JM, Skodol JA, Kellman HD, Hyler SE, Doidge N, Rosnick L, et al. Co-morbidity of Axis I and Axis II disorders. American Journal of Psychiatry. 1995;152:571–578. doi: 10.1176/ajp.152.4.571. [DOI] [PubMed] [Google Scholar]
  46. Perry W, Viglione DJ. The Ego Impairment Index as a predictor of outcome in melancholic depressed patients treated with tricyclic antidepressants. Journal of Personality Assessment. 1991;56(3):487–501. doi: 10.1207/s15327752jpa5603_10. [DOI] [PubMed] [Google Scholar]
  47. Pilkonis P, Heape C, Ruddy J, Serrao P. Validity in the diagnosis of personality disorders: The use of the LEAD standard. Psychological Assessment. 1991;3:46–54. [Google Scholar]
  48. Piper WE, Ogrodniczuk JS, Joyce AS. quality of object relations as a moderator of the relationship between pattern of alliance and outcome in short-term individual psychotherapy. Journal of Personality Assessment. 2004;83(3):345–356. doi: 10.1207/s15327752jpa8303_15. [DOI] [PubMed] [Google Scholar]
  49. Plakun EM. Treatment-refractory mood disorders: A psychodynamic perspective. Journal of Psychiatric Practice. 2003;9(3):209–218. doi: 10.1097/00131746-200305000-00004. [DOI] [PubMed] [Google Scholar]
  50. Roth A, Fonagy P. What works for whom? A critical review of psychotherapy research. New York, NY: Guilford Press; 1996. [Google Scholar]
  51. Shafer R. Psychoanalytic interpretation in Rorschach testing: Theory and application. New York: Grune & Stratton; 1954. [Google Scholar]
  52. Shrout P, Fleiss J. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin. 1979;86:420–428. doi: 10.1037//0033-2909.86.2.420. [DOI] [PubMed] [Google Scholar]
  53. Singer JD. Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics. 1998;23(4):323–355. [Google Scholar]
  54. Snijders TAB, Bosker RJ. Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage; 1999. [Google Scholar]
  55. Stokes JM, Pogge DL, Powell-Lunder J, Ward AW, Bilginer L, DeLuca VA. The Rorschach Ego Impairment Index: Prediction of treatment outcome in a child psychiatric population. Journal of Personality Assessment. 2003;81(1):11–19. doi: 10.1207/S15327752JPA8101_02. [DOI] [PubMed] [Google Scholar]
  56. Tuber S. Empirical and clinical assessments of children’s object relations and object representations. Journal of Personality Assessment. 1992;58(1):179–197. doi: 10.1207/s15327752jpa5801_16. [DOI] [PubMed] [Google Scholar]
  57. Urist J. The Rorschach test and the assessment of object relations. Journal of Personality Assessment. 1977;41(1):3–9. doi: 10.1207/s15327752jpa4101_1. [DOI] [PubMed] [Google Scholar]
  58. Viglione DJ. A review of recent research addressing the utility of the Rorschach. Psychological Assessment. 1999;11:251–265. [Google Scholar]
  59. Viglione DJ, Perry W, Giromini L, Meyer GJ. Revising the Rorschach Ego Impairment Index to accommodate recent recommendations about improving Rorschach validity. International Journal of Testing. 2011;11(4):349–364. [Google Scholar]
  60. Viglione DJ, Perry W, Meyer GJ. Refinements in the Rorschach Ego Impairment Index incorporating the human representational variable. Journal of Personality Assessment. 2003;81(2):149–156. doi: 10.1207/S15327752JPA8102_06. [DOI] [PubMed] [Google Scholar]
  61. Wechsler D. Wechsler Adult Intelligence Scale. 4. New York, NY: Psychological Corporation; 2008. (WAIS-IV) [Google Scholar]
  62. World Health Organization (WHO) World report on knowledge for better health: Strengthening health systems. Geneva: World Health Organization; 2004. [Google Scholar]
  63. Wright AGC, Hallquist MN. Mixture modeling methods for the assessment of normal and abnormal personality part II: Longitudinal models. Journal of Personality Assessment. doi: 10.1080/00223891.2013.830262. (this issue) [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES