Accounting for group differences in study retention in a randomized trial of specialized treatment for first episode psychosis

Abstract

Background

Schizophrenia is a chronic disabling disorder for which current treatments are only partially effective. While the evaluation of novel interventions is a high priority, loss to followup is a major threat to validity.

Methods

Pattern mixture modeling is a statistical technique that incorporates information on patterns of retention that may bias comparisons between randomized treatment groups. This study used pattern mixture mixed model (PMMM) in the analysis of outcomes of a two-year cluster-randomized trial, the Recovery after an Initial Schizophrenia Episode-Early Treatment Program, which compared a coordinated specialty care intervention called NAVIGATE to usual community care (CC). PMM-adjusted outcome differences between NAVIGATE and CC were estimated by the weighted-average of effects across the retention patterns.

Results

Compared to the original analysis, PMMM improved model fit and the estimated effectiveness of NAVIGATE as compared to CC. On the Quality of Life Scale NAVIGATE effectiveness increased by 1.50 points (25.4%); on the Positive and Negative Syndrome Scale, by 1.72 points (39.8%), and on the Calgary Depression Scale by 0.49 points (62.1%). PMMM did not improve model fit for employment days, substance use days, or hospital days.

Conclusion

Use of PMMM improved model fit and increased the estimated differences between NAVIGATE and CC for major outcomes. Patients with differential retention patterns may have different outcome trajectories. PMMM is a useful tool for addressing potential biases arising from these differences.

INTRODUCTION

Randomized clinical trials are essential tools for evaluating the effectiveness of new treatments. A major impediment to the validity of randomized clinical trials is the differential retention or loss to follow-up between treatment groups. Such differences can bias estimates of the differential effectiveness of treatments in randomized trials because they undermine the assumption that treatment groups are equivalent since patients with better or worse prognosis may be more likely to drop out of one group as compared to the other. In addition, patients with different patterns of retention may have different outcome trajectories within or across treatment groups, regardless of differences in overall rates of retention.

The problem of differential retention or dropout may be especially important in studies of severe mental illness, and especially in cases of first episode psychosis, because patients with these disorders are often less likely to participate for the full duration of a study due to poor psychosocial functioning and impaired insight into their illness (Mohamed et al., 2009).

There has been particular interest in recent years in early intervention in psychosis and in first episode schizophrenia in particular. It has been hypothesized that early intervention can substantially improve both short and long-term outcomes because it prevents the deterioration in functioning that is believed to come with prolonged untreated or under-treated psychosis (Addington, 2007; Álvarez-Jiménez et al., 2011; Bird et al., 2010). Several recent trials of intensive early intervention in psychosis have shown promising results lending support for this hypothesis (Craig et al., 2004; Gafoor et al., 2010; Garety et al., 2006; Kane et al., 2015; Petersen et al., 2005; Srihari et al., 2015).

The NIMH-funded Recovery After an Initial Schizophrenia Episode-Early Treatment Program (RAISE-ETP) study (Kane et al., 2016) is currently the largest real-world study of specialized coordinated care for first episode psychosis yet conducted in the United States. This multisite, two-year study showed significant benefits for a coordinated specialty care intervention called NAVIGATE in quality of life and symptoms as compared to usual community care (CC). There were, however, substantial differences in retention patterns between the two conditions, presumably because NAVIGATE patientts were more engaged in treatment and less likely to drop out as they received more intensive and comprehensive services. While 129 of 223 NAVIGATE patients (57.8%) completed the 24-month assessment, only 76 of 181 (41.9%) of CC patients did so. Whether this differential follow up biased the results of this study, and whether modeling differences in retention would alter the results, has not been examined.

A major methodological advance in this area in recent decades has been the use of mixed models which allow the use of all available data even when some data are missing from some subjects (Gueorguieva and Krystal, 2004; Lavori et al., 1990). However, mixed models are based on the assumption that data are missing at random (MAR) given observed measurements (Little and Rubin, 2002) and may be of uncertain validity when there is extensive loss to follow-up. MAR is an untestable assumption which may well be violated (Fitzmaurice et al., 2008) since loss to follow-up may be dependent on the missing outcome. While improving retention through aggressive follow-up and outcome assessment is the best way to minimize dropout bias, statistical remedies may also be used.

For a brief review of the analytic approaches including pattern mixture model (PMM) for dealing with data that are Missing Not at Random (MNAR) in clinical trials (Little and Rubin, 2002), please see Dziura et al. (2013). Molenberghs et al. (2002) used imputation method in their PMM. Mixed-effects analysis has also been an appealing approach in PMM where discrete variables for dropout patterns are used in regular mixed-effects model (Little, 1993; Hedeker and Gibbons, 1997; Demirtas and Schafer, 2003). This paper uses the pattern mixture mixed model (PMMM) approach. In this approach, participants in a clinical trial are stratified post-hoc according to the discrete groups representing their observed pattern of retention, or missing data, and each retention pattern has its pattern-specific parameterization in its own mixed-effect model. The weighted average of estimated outcomes across retention patterns in such models can then be calculated. We used several pattern-specific mixed-effect models while the mixed-effect PMM used a common mixed-effect model with terms of dropout patterns included as predictor but the two PMMs are otherwise similar.

Using different parameters for different retention patterns in PMM is thus a type of missing not at random (MNAR) model in which missing responses depend on the missingness pattern and vice versa. The PMM therefore may correct some bias when the MAR assumption is violated, however, it may still suffer bias if the missing response depends on additional unobserved variables besides the missingness pattern.

In this study, we used PMMMs to characterize the differential retention patterns among the subjects in RAISE-ETP, and to explore whether such models improve the goodness of fit of the outcome analyses and modify the estimated magnitude of group differences.

METHODS

Sample

A total of 404 individuals aged 15 to 40 who presented for treatment for a first episode psychosis (FEP) and who had been prescribed antipsychotic medication for less than six months in lifetime, were enrolled between 2010–2012: 223 in NAVIGATE and 181 in CC. A CONSORT diagram of recruitment has been previously published (Kane et al., 2016). Thirty-four community mental health treatment centers were randomized to NAVIGATE or CC with equal probability following a national invitation and selection process. None of the centers withdrew after randomization.

Outcomes

Trained clinician interviewers who were masked to study participants’ treatment assignment assessed the primary outcome measure, the Quality of Life Scale (QLS) (Heinrichs et al., 1984), using two-way, live video conferencing at baseline and at 6, 12, 18, and 24 months. The Positive and Negative Syndrome Scale (PANSS) (Kay et al., 1987) and the Calgary Depression Scale for Schizophrenia (CDSS) (Addington et al., 1990) were also completed by the interviewers. Days of employment or school attendance, days in the hospital and days of alcohol or drug use were documented at monthly interviews using structured self-report questions.

Statistical Analysis

Identifying variables associated with dropout

Among patients who had a baseline visit, we first used a time-to-dropout model with frailty terms (Clayton, 1978) for each individual variable, to identify patient characteristics associated with dropping out of the study for each of the baseline and time-varying covariates listed in Table 1. Because the trial was a clustered randomized trial in which the two treatment conditions were randomized at the site-level rather than the patient-level, we used frailty terms to account for clustering of individual patients within sites. Particularly, in the time-to-dropout models with frailty terms, patients within a site share the same frailty value to account for correlation of data among patients within the same site. Frailty terms serve a similar role as random effects in linear mixed model regression analyses.

Table 1.

Patient characteristics associated with study dropout

Categorical Variables	Baseline Hazard Ratio (95% CI)	p-value
CC vs. NAVIGATE	1.55 (1.08, 2.22)	0.0165
DUP > 74	1.11 (0.83, 1.48)	0.47
Male	1.11 (0.81, 1.52)	0.53
Race		0.31
White	1
African American	1.25 (0.9, 1.74)
Other	1.31 (0.79, 2.18)
Hispanic ethnicity
Marital status		0.24
Presently married	0.62 (0.31, 1.24)
Widowed/divorced/separated	0.66 (0.32, 1.37)
Never married	1
Current residence		0.49
Independent living	1
Supported or structured	1.27 (0.54, 2.99)
Family, parents, grandparents, sibling	1.2 (0.8, 1.8)
Homeless, shelter, or other	1.59 (0.88, 2.88)
Patient’s education		0.77
Some college or higher	1
Completed high school	1.14 (0.79, 1.64)
Some high school	1.21 (0.84, 1.75)
Some or completed grade school	1.19 (0.63, 2.25)
Current student	1	0.99
Currently working	0.97 (0.64, 1.46)	0.87
Student or working	0.85 (0.61, 1.16)	0.30
Prescribed one or more antipsychotics at consent	1.3 (0.9, 1.87)	0.16
Number of prior hospitalizations		0.91
0	1
1	1.13 (0.78, 1.65)
2	1.05 (0.66, 1.67)
3 or more	1.02 (0.63, 1.64)
Medication compliance
Days not taking 1st antipsychotic		0.41
Few if any, <7	1
7 to 13	1.53 (0.85, 2.74)
14 to 20	0.88 (0.38, 2.02)
Most, >20	0.74 (0.34, 1.61)

Continuous Variable	Baseline Variable		Time-varying Variable
	Hazard Ratio (95% CI)	p-value	Hazard Ratio (95% CI)	p-value
Age	0.97 (0.935, 0.995)	0.024	N/A
Duration of untreated psychosis (weeks)	1.000 (0.999, 1.000)	0.85	N/A
Quality of Life Scale for Schizophrenia
Total score	0.994 (0.986, 1.002)	0.13	0.995 (0.988, 1.002)	0.14
Interpersonal relations	0.990 (0.974, 1.007)	0.23	0.991 (0.976, 1.006)	0.23
Instrumental role	0.994 (0.972, 1.016)	0.57	0.990 (0.970, 1.011)	0.36
Intrapsychic foundations	0.986 (0.966, 1.007)	0.19	0.991 (0.972, 1.010)	0.35
Common objects and activities	0.934 (0.878, 0.995)	0.03	0.928 (0.873, 0.986)	0.02
PANSS
Total score	1.006 (0.997, 1.016)	0.19	1.002 (0.993, 1.011)	0.62
Factor scores
Positive	0.986 (0.950, 1.023)	0.46	0.998 (0.963, 1.034)	0.92
Negative	1.009 (0.981, 1.038)	0.51	1.001 (0.975, 1.028)	0.93
Disorganized/concrete	1.018 (0.966, 1.071)	0.51	1.019 (0.966, 1.075)	0.50
Excited	1.044 (0.992, 1.098)	0.10	1.032 (0.982, 1.086)	0.22
Depressed	1.043 (0.998, 1.089)	0.06	1.002 (0.96, 1.046)	0.92
CDSS	1.033 (0.999, 1.067)	0.05	1.012 (0.979, 1.047)	0.47
CGI	0.998 (0.838, 1.188)	0.98	1.015 (0.869, 1.185)	0.85
Number of days of alcohol intoxication	0.941 (0.859, 1.031)	0.19	0.918 (0.823, 1.023)	0.12
Number of days of illegal drugs	1.003 (0.985, 1.022)	0.73	0.994 (0.975, 1.015)	0.58
How likely to complete study?	0.984 (0.908, 1.066)	0.69	0.928 (0.856, 1.006)	0.07
How likely to attend next assessment?	1.021 (0.928, 1.123)	0.68	0.905 (0.838, 0.977)	0.01

Analysis with pattern mixture mixed model

We groups individual study participants using three different approaches to the classification of retention patterns. The first approach was based on the number of follow-up visits out of a possible total of four (range=0 to 4). The second approach was based on the time of the last follow-up visit (0, 6, 12, 18, or 24 months). The third approach added an additional indicator variable to the second approach to represent the situation in which patients missed an assessment and then completed at least one subsequent assessment (i.e., who had intermittent missing data) and otherwise were set to zero. The three different approaches to characterizing retention patterns serve as sensitivity analyses to compare how treatment effects change under different missingness classifications.

For each outcome using PMMM, with the addition of a class variable distinguishing different retention patterns in each of the three classification approaches described above, a pattern-specific mixed model within each retention pattern was fitted. For all the patterns with at least one followup visit, the pattern-specific mixed models were similar to those mixed model in the original trial report (Kane et al., 2016). The mixed models included term of time, the interaction between treatment group and time, and baseline measures that had been determined to be significantly different between treatment groups and significant for predicting the outcome. Specifically, the baseline covariates included were male gender, student status, and PANSS baseline score (Kane et al., 2016). The term for time was the square root of months since randomization which resulted approximately linear relationship between time and outcome (Kane et al., 2016). The model also included individual-specific and site-specific random intercept and slope of time. For patients in Stratum 0 who had only baseline measurements without followup visit, no model term involving time was included in their pattern-specific mixed models. For the outcome measures assessed monthly (days of work, substance use and hospitalization), mixed negative binomial regressions were conducted with terms analogous to the linear mixed models except with only random intercepts for individuals and sites. As there was no statistically significant difference between the two treatment groups for baseline measures of the outcomes except for the PANSS, baseline values were used as outcomes. Since there was a statistically significant difference in the baseline PANSS between the two treatment groups, the baseline PANSS was not used as an outcome but instead the treatment indicator was used as covariate for PANSS to account for baseline difference.

In the pattern mixture mixed models, additional indicator variables were included to represent the patterns of missing data. In addition, terms representing the interaction of retention pattern by treatment group, retention pattern by time, and the three-way interaction of retention pattern by treatment by time were also included.

Comparing treatment effects between pattern mixture mixed model and linear mixed model

For each outcome in each of the three approaches, the model-fitted difference in outcomes between treatment groups was estimated for each stratified retention pattern. The retention pattern-specific effect is the model-estimated difference in outcome between the NAVIGATE group and the CC group at the end of study (24 months). Across all the three approaches in our PMMM, for patients in the Stratum 0 who had only baseline measurements, no pattern-specific effect was calculated as no term involving time was included in the mixed model for the baseline only pattern. The difference in the pattern-specific effect between a retention pattern and the referent stratum was evaluated with a Wald test of the difference. The referent pattern was the stratum in which the patients had all four follow-up visits under Approach 1, or in which the patients had a last visit at month 24 under Approaches 2 and 3. By comparing to the referent pattern, we can understand how the effectiveness of NAVIGATE among the dropouts differed from the outcome among those who had the most follow-up visits in the trial.

Next, the overall treatment effect in a given PMM for each outcome was obtained by averaging the estimated pattern-specific effect across the retention patterns, weighting by the proportion of patients in each retention pattern. The standard errors of these averages were calculated using the delta method (Casella and Berger, 2002). The magnitude of the resultant treatment effect could then be compared to that in the original linear mixed model.

In addition, the model fits of the regular mixed model and the three PMMMs for each outcome were typically compared using the Bayesian Information Criterion (BIC) (Schwarz 1978) which serves a method for model selection that takes into account of both fit and parsimony among candidate models. The PMMMs are nonnested models. BIC applied to both nested and nonnested models. A model with a smaller BIC is preferred.

RESULTS

Sample

Characteristics of the sample have been previously published (Kane et al., 2016). Frailty models showed CC patients were more likely to drop out than NAVIGATE patients (HR 1.55, p-value 0.02). Older patients were less likely to drop out (HR 0.97 per year, p-value 0.02) as were patients with a higher baseline and time-varying scores on the common objects and activities subscale of QLS (HR 0.93, p-value 0.04). Patients with a higher self-rated likelihood for their next visit were also less likely to drop out (HR 0.91, 0.84, 0.98, p-value=0.01). No other baseline or time-varying measures were predictive of dropout at p<.05 (Table 1).

Retention Patterns and Model Fit of the PMMMs

The number and proportion of patients in each pattern are shown in Table 2 for the three approaches. Of the patients that were enrolled and completed baseline assessment, 83 (20.5%) left the study before the first post-baseline visit at 6 months: 46 (25.4%) in CC and 37 (16.6%) in NAVIGATE. Since outcomes for these patients are not obtainable, their outcomes were not modeled as a separate category, and thus they were de facto treated as if their outcomes were similar to those of patients in the other retention patterns taken together.

Table 2.

The number and proportion of patients in three types of patterns

a. Number of follow-up visits
	All patients		Community Care		NAVIGATE
# follow- up visits	N	%	N	%	N	%
0	83 (20.5%)	---	46 (25.4%)	---	37 (16.6%)	---
1	50	15.6	24	17.8	26	14.0
2	49	15.3	21	15.6	28	15.1
3	67	20.9	30	22.2	37	19.9
4	155	48.3	60	44.4	95	51.1

b. Last follow-up visit with intermittent follow-up visits accounted in the visit
	All patients		Community Care		NAVIGATE
Last visit	N	%	N	%	N	%
Month 0	83 (20.5%)	---	46 (25.4%)	---	37 (16.6%)	---
Month 6	40	12.5	20	14.8	20	10.8
Month 12	37	11.5	20	14.8	17	9.1
Month 18	39	12.1	19	14.1	20	10.8
Month 24	205	63.9	76	56.3	129	69.4

c. Last follow-up visit with intermittent follow-up visits as a separate category
	All patients		Community Care		NAVIGATE
Last visit	N	%	N	%	N	%
Month 0	83 (20.5%)	---	46 (25.4%)	---	37 (16.6%)	---
Month 6	40	12.5	20	14.8	20	10.8
Month 12	34	10.6	18	13.3	16	8.6
Month 18	30	9.3	17	12.6	13	7.0
Month 24	155	48.3	60	44.4	95	51.1
Intermittent	62	19.3	20	14.8	42	22.6

Including patients with no follow-up visits: χ² = 10.70, df=4, p=0.03

Excluding patients with no follow-up visits: χ² = 1.65, df=3, p=0.65

Including patients with no follow-up visits: χ² = 6.39, df=4, p=0.17

Excluding patients with no follow-up visits: χ² = 6.02, df=3, p=0.11

Including patients with no follow-up visits: χ² = 13.11, df=5, p=0.02

Excluding patients with no follow-up visits: χ² = 8.47, df=4, p=0.08

For the approach based on the number of follow-up assessments, 50 (15.6%) had one follow-up assessment, 49 (15.3%) two follow-up assessments, 67 (20.9%) three follow-up assessments, and 155 (48.3%) completed all four follow-up assessments (Table 2, upper panel). There was no significant difference in the distribution of these patterns between the two treatment groups (χ² = 1.65, df=3, p=0.65). Data for the other two approaches are presented in the other panels in Table 2.

For each of the QLS, the PANSS total score, and the CDSS, the three approaches of retention classifications all exhibited a better model fit than the original mixed model, as demonstrated by lower BICs than the model with the regular mixed model (Table 3). Furthermore, the best-fit PMMM was the one under Approach 3 of using the last visit time and intermittent missingness for pattern distinction. (Table 3). Model fit was not improved by the PMMMs for days of attending work or school, an index of daily use of alcohol or drugs during the past month, or for days of hospitalization in the past month presumably (Supplementary Table 1).

Table 3.

Mean Change at 24 Months from Baseline between CC and NAVIGATE

		Standard Mixed Model (MM)		PMMM based on number of visits			PMMM based on time of last visit			PMM based on the time of last visit and intermittent missingness
		Mean Change	SE	Mean Change	SE	% diff. from MM	Mean Change	SE	% diff. from MM	Mean Change	SE	% diff. from MM
	CC	9.89	1.92	8.54	2.05	−13.7%	9.10	2.18	−8.0%	8.76	2.16	−11.4%
	NAV	15.79	1.62	15.57	1.67	−1.4%	16.19	1.80	2.5%	16.15	1.78	2.3%
QLS total score	Diff. between 2 groups	5.90	2.41	7.04	2.52	19.3%	7.09	2.69	20.2%	7.40	2.66	25.4%
	p-val.	0.015		0.0055			0.0085			0.0056
	BIC	11411.2		11367.3			11366.0			11353.8
	CC	−9.99	1.38	−10.15	1.51	1.6%	−9.70	1.69	−2.9%	−9.47	1.72	−5.2%
	NAV	−14.31	1.14	−14.70	1.16	2.7%	−15.57	1.33	8.8%	−15.51	1.37	8.4%
PANSS total score	Diff. between 2 groups	−4.32	1.79	−4.55	1.90	5.3%	−5.87	2.15	35.8%	−6.04	2.20	39.8%
	p-val.	0.016					0.0065			0.0061
	BIC	11118.0		11055.4			11038.9			11033.8
	CC	−1.20	0.33	−0.93	0.39	−22.5%	−1.07	0.43	−10.8%	−0.95	0.43	−20.8%
	NAV	−1.98	0.28	−1.89	0.32	−4.5%	−2.22	0.35	12.1%	−2.23	0.35	12.6%
CDSS total score	Diff. between 2 groups	−0.79	0.37	−0.97	0.42	22.8%	−1.15	0.48	45.6%	−1.28	0.48	62.0%
	p-val.	0.032		0.0226			0.0177			0.0078
	BIC	7344.7		7333.5			7329.7			7323.6

Outcome Difference between Retention Patterns

To evaluate how differences in the outcomes between the two treatment groups vary across retention patterns, Tables 4 showed the differences in treatment group outcomes on the QLS, PANSS and CDSS for each retention pattern based on the best-fit PMMM, the ones under Approach 3 of using the last visit time and intermittent missingness for pattern classification. For each outcome group differences between NAVIGATE and CC were greater in the retention patterns with earlier dropout, although these differences were statistically significant only for the PANSS at month 6 (Table 4a, 4b, and 4c). Examination of outcomes specifically at 6 months, for example, showed that among patients in the retention pattern that dropped out at 6-months the NAVIGATE group showed its greatest magnitude of superior improvement to CC on all three outcomes: the QLS (17.36, SE = 11.52) (top row, Table 4a); the PANSS (−29.88, SE = 10.85) (top row, Table 4b) and the CDSS (−5.38, SE = 2.53) (top row, Table 4c). Within the retention pattern with earliest dropout, the CC group showed less improvement than in the other four retention patterns, while the NAVIGATE group showed greater improvement than in the four retention patterns with later dropout.

Table 4.

Mean Change at End of Study from Baseline by Retention Pattern based on Time of Last Visit and Intermittent Missingness (Approach 3)

a. Quality of life (QLS)
	Community Care			NAVIGATE			Difference
Retention Pattern	Mean Change	SE	p*	Mean Change	SE	p*	Mean Change	SE	p*
Month 6	2.92	8.56	0.43	20.28	8.57	0.64	17.36	11.52	2.92
Month 12	10.29	6.27	0.96	26.99	6.61	0.11	16.71	8.58	10.29
Month 18	11.18	5.26	0.83	4.43	5.96	0.06	−6.76	7.63	11.18
Month 24	9.93	2.46	Ref.	16.13	2.03	Ref.	6.20	3.06	9.93
Intermittent	7.62	4.34	0.64	13.73	3.08	0.50	6.11	5.15	7.62
Overall PMMM	8.76	2.06		16.25	1.88		7.49	2.65	8.76
Mixed Model	9.89	1.92		15.79	1.62		5.90	2.41	9.89

b. PANSS
	Community Care			NAVIGATE			Difference
Retention Pattern	Mean Change	SE	p*	Mean Change	SE	p*	Mean Change	SE	p*
Month 6	−2.47	7.67	0.31	−32.35	7.67	0.0174	−29.88	10.85	0.0168
Month 12	−12.55	4.91	0.69	−20.09	5.21	0.24	−7.54	7.16	0.57
Month 18	−12.81	4.50	0.63	−11.37	5.13	0.66	1.44	6.83	0.51
Month 24	−10.45	1.85	Ref.	−13.72	1.47	Ref.	−3.27	2.36	Ref.
Intermittent	−7.91	3.44	0.52	−11.05	2.39	0.34	−3.14	4.19	0.98
Overall PMMM	−9.47	1.72		−15.51	1.37		−6.04	2.20
Mixed Model	−9.99	1.38		−14.31	1.14		−4.32	1.79

c. CDSS
	Community Care			NAVIGATE			Difference
Retention Pattern	Mean Change	SE	p*	Mean Change	SE	p*	Mean Change	SE	p*
Month 6	0.08	1.96	0.74	−5.30	1.96	0.08	−5.38	2.53	0.11
Month 12	−1.48	1.24	0.50	−4.29	1.32	0.07	−2.81	1.51	0.32
Month 18	−2.32	1.07	0.14	−0.70	1.17	0.37	1.62	1.31	0.0402
Month 24	−0.59	0.44	Ref.	−1.82	0.38	Ref.	−1.23	0.46	Ref.
Intermittent	−1.40	0.80	0.38	−1.38	0.62	0.55	0.02	0.84	0.20
Overall PMMM	−0.95	0.43		−2.23	0.35		−1.28	0.48
Mixed Model	−1.20	0.33		−1.98	0.28		−0.79	0.37

^*p-values for the difference in the mean change comparing to the referent pattern.

^*p-values for the difference in the mean change comparing to the referent pattern.

^*p-values for the difference in the mean change comparing to the referent pattern.

Treatment Effectiveness in Outcomes in PMMM

We next examine the change in magnitude of treatment effects for NAVIGATE vs. CC, on each of the three major outcomes, between the original mixed model and the best fit PMMM, the one based on the last visit but including those with intermittent missing data as a separate retention pattern. For the primary outcome, the total QLS score, the treatment effect of the NAVIGATE group over CC using PMMM was 7.40 (SE 2.66) points, compared to 5.90 (SE 2.41) using the mixed model alone (Table 3), representing an increase in effectiveness of 25.4% (Table 3). In addition, the p-value is much lower than in the original mixed model analysis falling to p=.006 (last column, Table 3).

For the total PANSS score, the treatment effect of the NAVIGATE group over CC was −6.04 (SE 2.20) points using PMMM, compared to only −4.32 (SE 1.79) points using the mixed model, representing an increase in effectiveness of 39.8% (last column, Table 3). The p-value is also much lower than in the original mixed model analysis falling to .006. For the CDSS, the treatment effect of the NAVIGATE group over CC using PMMM was −1.28 (SE 0.48) points, compared to only −0.79 (SE 0.37) using the mixed model, an increase in improvement of 62.0% (last column, Table 3).

DISCUSSION

This study demonstrates an approach to the problem of differential retention between treatment groups using PMMMs in an examination of the outcomes of a cluster-randomized trial of treatment for first episode psychosis. Using PMMM, the benefits of NAVIGATE appeared to be notably larger for the analysis of three standardized scales (QLS, PANSS, and CDSS) than in the standard mixed model analysis without PMM, with a better goodness of fit. The benefits of NAVIGATE over CC were estimated to be greater by 26% to 65% than previously estimated.

We found substantial differences in outcome trajectories between retention patterns even though there were very few differences in baseline characteristics associated with retention. A key finding was that there were differences in outcome trajectories between different retention patterns that were only revealed using the pattern mixture model. Because patients in CC were more likely to dropout (Table 1 Row 1) and patients who dropped out earlier were worse off, in the retention patterns with earlier dropouts (Table 4), the estimated pattern-specific treatment effects between NAVIGATE and the CC were greater, and this greater difference was identified in the PMMM but not in regular mixed models. A plausible interpretation of this finding is that study participants may have dropped out of the two treatment groups for different reasons. Early dropouts in the NAVIGATE group may have benefited more during the early phases of the trial when the NAVIGATE intervention provided much greater intensity of service delivery, and may have dropped out when they felt they no longer needed the treatment program. In contrast, participants in CC may have been discouraged by the limited treatment they received. Poorer functioning participants who failed to improve may have drop out early in treatment. These differences in reasons for dropping out may have been most marked early in the study when the intensity of service delivery between the two programs differed most, with similar reasons for dropping out emerging over time for both groups. The substantial differences in pattern-specific effect between groups among these early dropouts appears to account for the increased estimates of the relative effectiveness of NAVIGATE in the PMM analysis as compared to the original mixed models. These differences in pattern-specific effects between retention patterns did not achieve statistical significance most likely because of the small number of patients in each retention stratum. Nevertheless, these findings demonstrate that there are differences in estimated benefits between retention patterns and that taking these differences into account gives a more accurate picture of the trial results, as indicated by better model fit indices, and one that shows the benefits of NAVIGATE in a more positive light than the original analyses.

The results of this analysis of the RAISE-ETP trial using PMMM strengthen the positive findings of the original RAISE-ETP study analyses (Kane et al., 2016), and reinforces the findings of several previous studies showing that coordinated specialty care programs improve the outcomes of first episode psychosis (Heinssen et al., 2014).

From the research methods perspective, this is only the seventh randomized trial of schizophrenia treatment outcomes that we could identify that used the PMM method to address differential study retention (Harris et al., 2006; Hedeker and Gibbons, 1997; Hill et al., 2004; Kong and Chen, 2016; Reilly et al., 2005; Rybin et al., 2015). Application of the PMM method to two previous randomized trials for schizophrenia was similar to the analysis presented here and compared the PMM results to standard mixed model results, but found no indication of better goodness of fit nor any substantial differences in effectiveness estimates between randomized treatment groups (Harris et al., 2006; Hedeker and Gibbons, 1997). Two other studies that used PMM in schizophrenia treatment trials did not compare the results between PMM and standard mixed models (Hill et al., 2004; Rybin et al., 2015). Rybin et al. (2015) performed a sensitivity analysis of a schizophrenia trial examining the effect of various departures from the missing at random assumption in a simulation analysis and found that such departures did not threaten the validity of the original findings. Kong and Chen (2016) compared a latent growth mixture model to a pattern mixture model and expressed preference for the former due to its greater flexibility since subgroups with differing outcome trajectories are not necessarily defined by the missing data patterns.

The lack of evidence that PMM changes outcome evaluations in schizophrenia trials, even though they reported dropout rates ranging from 21% to 68%, may explain why the effective method for incomplete data has been used so few times, although it has been available to researchers for over 20 years. While the reasons for this limited use are not clear, the results presented here should encourage further use of this approach in studies that are characterized by substantial rates of attrition even when treatment groups do not have significantly different rates of attrition. One of the striking findings of this study is that even though there was no difference in attrition rate after the first follow-up assessment at 6 months, subgroups with different patterns of attrition showed different outcome trajectories, possibly due to different reasons of dropout between the two treatment conditions. It is thus advisable to evaluate the relevance of PMM whenever there is substantial attrition to see if there are differences in outcomes between groups with different patterns of attrition. The examination of a larger number of retention patterns with pattern-specific mixed models may provide an explanation of the fact that we found differences in outcome trajectories with PMMs in this study while other studies did not find such differences.

Several methodological limitations of this study require comment. First, statistical power in this study is limited, due to small sample size and clustering of patients within sites, which may have impaired our ability to finding statistically significant differences between outcomes across retention patterns or between PMMM analyses and standard mixed model analyses.

A second limitation is that while the use of pattern mixture models in a mixed effects model is a relatively simple extension of the mixed effects model, it does not improve model fit for all outcomes. We found the stratification based on the last visit with intermittent data as a separate retention pattern achieved the best fit for the QLS, the PANSS, and the CDSS, but it did not yield a better fit for other outcomes.

Third, since 83 patients (20.5%) left the study before the first post-baseline visit at 6 months, for whom, an intercept only mixed model was fitted in their own stratum and model-predicted outcomes for these patients were not obtainable at the end of the 24 months in PMMM as there was no slope of time for them and we did not make further assumption for these patients on their 24-month’ outcomes.

This study suggests both substantive and methodological conclusions. It shows that coordinated specialty care for first episode psychosis is effective and may have been even more effective in the RAISE-ETP trial than had previously been estimated. Methodologically, it shows that pattern mixture models can give a more accurate perspective on the outcomes and the treatment effect of randomized clinical trials even when there are limited differences in baseline characteristics associated with retention. PMM should be more widely employed in the analysis of randomized clinical trials with substantial missing data as part of a standard sensitivity analysis.

Appendix

SAS code for fitting pattern mixture mixed-effects model:

• * Standard model;

• proc mixed data=raise2;

• class site subject_id time6c;

• model qlsts = sqrttime1m trt*sqrttime1m male studntyn_bl pants_bl / solution;

• random intercept sqrttime1m / subject=site type=un;

• random intercept sqrttime1m / subject=subject_id(site) type=un;

• repeated time6c / subject=subject_id(site) type=un;

• run;

• * Code for last visit pattern in PMMM;

• %macro patmix(outcome, covariates);

• title “&outcome.”

• proc mixed data=raise2;

• class site subject_id time6c;

• model &outcome. = sqrttime1m trt*sqrttime1m &covariates.

• lastvisit1 trt*lastvisit1 lastvisit1*sqrttime1m trt*lastvisit1*sqrttime1m

• lastvisit2 trt*lastvisit2 lastvisit2*sqrttime1m trt*lastvisit2*sqrttime1m

• lastvisit3 trt*lastvisit3 lastvisit3*sqrttime1m trt*lastvisit3*sqrttime1m

• / solution;

• random intercept sqrttime1m / subject=site type=un;

• random intercept sqrttime1m / subject=subject_id(site) type=un;

• repeated time6c / subject=subject_id(site) type=un;

• contrast ‘test pattern terms’

• lastvisitl 1,

• lastvisit2 1,

• lastvisit3 1,

• lastvisit1*sqrttime1m 1,

• lastvisit2*sqrttime1m 1,

• lastvisit3*sqrttime1m 1,

• trt*lastvisit1*sqrttime1m 1,

• trt*lastvisit2*sqrttime1m 1,

• trt*lastvisit3*sqrttime1m 1;

• estimate ‘g4 - change from BL at 24m CC’

• sqrttime1m 4.899;

• estimate ‘g4 - change from BL at 24m NAV’

• sqrttime1m 4.899 trt*sqrttime1m 4.899;

• estimate ‘g4 - difference in change at 24m’

• trt*sqrttime1m 4.899;

• ***;

• estimate ‘g1 - change from BL at 24m CC’

• sqrttime1m 4.899

• lastvisit1*sqrttime1m 4.899;

• estimate ‘g1 - change from BL at 24m NAV’

• sqrttime1m 4.899 trt*sqrttime1m 4.899

• lastvisit1*sqrttime1m 4.899 trt*lastvisit1*sqrttime1m 4.899;

• estimate ‘g1 - difference in change at 24m’

• trt*sqrttime1m 4.899

• trt*lastvisit1*sqrttime1m 4.899;

• ***;

• estimate ‘g2 - change from BL at 24m CC’

• sqrttime1m 4.899

• lastvisit2*sqrttime1m 4.899;

• estimate ‘g2 - change from BL at 24m NAV’

• sqrttime1m 4.899 trt*sqrttime1m 4.899

• lastvisit2*sqrttime1m 4.899 trt*lastvisit2*sqrttime1m 4.899;

• estimate ‘g2 - difference in change at 24m’

• trt*sqrttime1m 4.899

• trt*lastvisit2*sqrttime1m 4.899;

• ***;

• estimate ‘g3 - change from BL at 24m CC’

• sqrttime1m 4.899

• lastvisit3*sqrttime1m 4.899;

• estimate ‘g3 - change from BL at 24m NAV’

• sqrttime1m 4.899 trt*sqrttime1m 4.899

• lastvisit3*sqrttime1m 4.899 trt*lastvisit3*sqrttime1m 4.899;

• estimate ‘g3 - difference in change at 24m’

• trt*sqrttime1m 4.899

• trt*lastvisit3*sqrttime1m 4.899;

• ***;

• estimate ‘change from BL at 24m CC’

• sqrttime1m 4.899

• lastvisit1*sqrttime1m 0.612 /* 4.899*0.125 */

• lastvisit2*sqrttime1m 0.563 /* 4.899*0.115 */

• lastvisit3*sqrttime1m 0.593 /* 4.899*0.121 */;

• estimate ‘change from BL at 24m NAV’

• sqrttime1m 4.899 trt*sqrttime1m 4.899

• lastvisit1*sqrttime1m 0.612 trt*lastvisit1*sqrttime1m 0.612

• lastvisit2*sqrttime1m 0.563 trt*lastvisit2*sqrttime1m 0.563

• lastvisit3*sqrttime1m 0.593 trt*lastvisit3*sqrttime1m 0.593;

• estimate ‘difference in change at 24m’

• trt*sqrttime1m 4.899

• trt*lastvisit1*sqrttime1m 0.612

• trt*lastvisit2*sqrttime1m 0.563

• trt*lastvisit3*sqrttime1m 0.593;

• estimate ‘change from BL at 24m CC - based on within trt’

• sqrttime1m 4.899

• lastvisit1*sqrttime1m 0.725 /* 4.899*0.148 = 0.725 */

• lastvisit2*sqrttime1m 0.725 /* 4.899*0.148 = 0.725 */

• lastvisit3*sqrttime1m 0.691 /* 4.899*0.141 = 0.691 */;

• estimate ‘change from BL at 24m NAV - based on within trt’

• /*

• 4.899*0.108 = 0.529

• 4.899*0.091 = 0.446

• 4.899*0.108 = 0.529

• */

• sqrttime1m 4.899 trt*sqrttime1m 4.899

• lastvisit1*sqrttime1m 0.529 trt*lastvisit1*sqrttime1m 0.529

• lastvisit2*sqrttime1m 0.446 trt*lastvisit2*sqrttime1m 0.446

• lastvisit3*sqrttime1m 0.529 trt*lastvisit3*sqrttime1m 0.529;

• estimate ‘difference in change at 24m - based on within trt’

• /*

• 0.529 – 0.725 = −0.196

• 0.446 – 0.725 = −0.279

• 0.529 – 0.691 = −0.162

• */

• trt*sqrttime1m 4.899

• lastvisit1*sqrttime1m −0.196 trt*lastvisit1*sqrttime1m 0.529

• lastvisit2*sqrttime1m −0.279 trt*lastvisit2*sqrttime1m 0.446

• lastvisit3*sqrttime1m −0.162 trt*lastvisit3*sqrttime1m 0.529;

• run;

• title ““;

• %mend;

• %patmix(qlsts, male studntyn_bl pants_bl);