Skip to main content
NCBI home page
Elsevier Sponsored Documents logoLink to Elsevier Sponsored Documents
. 2022 Dec;250:1–9. doi: 10.1016/j.schres.2022.09.009

Clinical predictors of antipsychotic treatment resistance: Development and internal validation of a prognostic prediction model by the STRATA-G consortium

Sophie E Smart a,b,⁎, Deborah Agbedjro c, Antonio F Pardiñas a, Olesya Ajnakina c,d, Luis Alameda b,e,f,g, Ole A Andreassen h,i, Thomas RE Barnes j, Domenico Berardi k, Sara Camporesi g,l, Martine Cleusix l, Philippe Conus g, Benedicto Crespo-Facorro e,f, Giuseppe D'Andrea k, Arsime Demjaha b, Marta Di Forti m,n, Kim Do l, Gillian Doody o, Chin B Eap p,q,r,s, Aziz Ferchiou t,u, Lorenzo Guidi v, Lina Homman w, Raoul Jenni l, Eileen Joyce x, Laura Kassoumeri b, Ornella Lastrina v, Ingrid Melle h,i, Craig Morgan y,z, Francis A O'Neill aa, Baptiste Pignon t,u, Romeo Restellini g,l, Jean-Romain Richard t, Carmen Simonsen h,ab, Filip Španiel ac,ad, Andrei Szöke t,u, Ilaria Tarricone v, Andrea Tortelli t,ae, Alp Üçok af, Javier Vázquez-Bourgon e,ag,ah, Robin M Murray b, James TR Walters a, Daniel Stahl c, James H MacCabe b,n
PMCID: PMC9834064  PMID: 36242784

Abstract

Introduction

Our aim was to, firstly, identify characteristics at first-episode of psychosis that are associated with later antipsychotic treatment resistance (TR) and, secondly, to develop a parsimonious prediction model for TR.

Methods

We combined data from ten prospective, first-episode psychosis cohorts from across Europe and categorised patients as TR or non-treatment resistant (NTR) after a mean follow up of 4.18 years (s.d. = 3.20) for secondary data analysis. We identified a list of potential predictors from clinical and demographic data recorded at first-episode. These potential predictors were entered in two models: a multivariable logistic regression to identify which were independently associated with TR and a penalised logistic regression, which performed variable selection, to produce a parsimonious prediction model. This model was internally validated using a 5-fold, 50-repeat cross-validation optimism-correction.

Results

Our sample consisted of N = 2216 participants of which 385 (17 %) developed TR. Younger age of psychosis onset and fewer years in education were independently associated with increased odds of developing TR. The prediction model selected 7 out of 17 variables that, when combined, could quantify the risk of being TR better than chance. These included age of onset, years in education, gender, BMI, relationship status, alcohol use, and positive symptoms. The optimism-corrected area under the curve was 0.59 (accuracy = 64 %, sensitivity = 48 %, and specificity = 76 %).

Implications

Our findings show that treatment resistance can be predicted, at first-episode of psychosis. Pending a model update and external validation, we demonstrate the potential value of prediction models for TR.

Keywords: Treatment resistant schizophrenia, First episode psychosis, Prospective longitudinal cohort, Prediction modelling, Stratification, Machine learning

1. Introduction

Approximately 20–30 % of patients with schizophrenia or a related psychotic disorder continue to experience disabling psychotic symptoms, despite two adequate trials of antipsychotic medication (Mørup et al., 2020; NICE, 2014; Siskind et al., 2022; Stokes et al., 2020). These patients are termed treatment resistant (TR). Patients with TR have a poorer quality of life, poor physical health, and their level of disability often prevents them from working (Brain et al., 2018; Iasevoli et al., 2016; Nordstroem et al., 2017). This, in combination with the cost of medical treatment and hospitalisation, increases the societal economic burden of TR (Andrews et al., 2012; Kennedy et al., 2014). However, many patients with TR can be successfully treated: clozapine is highly effective in reducing symptom severity and mortality rates (Siskind et al., 2017; Warnez and Alessi-Severini, 2014) and there is some evidence that cognitive behavioural therapy (CBT) as an adjunct treatment may be effective (Polese et al., 2019). The problem lies in how long it takes patients to access that treatment. Delays to successful treatment are associated with poorer symptomatic response throughout the course of psychosis (Malla et al., 2006; Murru and Carpiniello, 2018), and a delay in clozapine prescription for patients with TR is associated with poor response to clozapine (Griffiths et al., 2021). One study estimated that clozapine is delayed by an average of four years (Howes et al., 2012). A prediction model capable of identifying TR earlier in the course of illness could help to reduce this delay by identifying patients at higher risk of TR and offering them clozapine earlier in the disease.

Prognostic prediction models are mathematical equations which quantify the predictive power of data at a given time point to predict a future outcome among individuals at risk for that outcome (Hemingway et al., 2013). The equation is optimised to calculate an individual's probability of developing the outcome rather than a group-level association. As a consequence, prediction models can be used to inform clinical decision making (Lee et al., 2016). A single predictor is easy to collect but often has poor predictive power. Conversely, models that require many variables will have better predictive validity but are more cumbersome and difficult to apply across samples with different data; there is thus a trade-off between predictive power and usefulness in real-world settings. Prediction models should therefore include a variable selection step, where the most parsimonious combination of variables is selected. They then undergo model development, internal and external validation, and are in many cases refined, before being implemented in clinical settings. Prognostic prediction models are a useful aid in planning treatment decisions but can also be used to stratify participants by risk in clinical trials (Collins et al., 2015).

Since prediction models require different optimization criteria they thus require different methodologies to classical statistical models (the latter are sometimes referred to as explanatory models) (Hastie et al., 2009; Shmueli, 2010). Machine learning and penalised maximum likelihood methods are often used to develop prediction models (Bzdok and Meyer-Lindenberg, 2018); penalised regression methods are well suited to predicting outcomes with a relatively low prevalence, such as TR (Pavlou et al., 2016).

In a systematic review of predictors of TR, identified in prospective observational cohort studies, only two studies reported statistics that measure predictive validity and all but one used statistical methods designed to identify explanatory variables of TR rather than a prediction model (Smart et al., 2021). Since this review, two further studies have used first-episode cohort data to predict TR. One used a machine learning method, but focused on evaluating individual predictors, not on proposing a model capable of predicting an individual's risk (Legge et al., 2020). The second developed two separate models; one to predict early-onset TR (never experiencing symptomatic remission) and the other to predict late-onset TR (experiencing symptomatic remission before later developing TR) (Ajnakina et al., 2020).

Here we, firstly, report an explanatory model of TR which quantifies the associations between clinical and demographic data collected at first-episode and TR. Secondly, we report the development and internal validation of a prognostic prediction model for TR, using the same set of clinical and demographic variables.

2. Methods

We followed guidance set out in the ‘Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis’ (TRIPOD) statement to report this study (Collins et al., 2015). The TRIPOD checklist can be found in the Supplementary Material.

2.1. Study design and participants

The ‘Schizophrenia: Treatment Resistance and Therapeutic Advances’ (STRATA) Consortium is a UK Medical Research Council-funded, multi-disciplinary group of researchers working together on stratified medicine for schizophrenia and related psychoses. As part of the STRATA-Genetics workstream (STRATA-G), genetic and phenotypic data was collected from existing observational, prospective, first-episode psychosis cohorts to perform secondary data analysis. To be included in STRATA-G, cohorts were required to have data on individuals recruited at first episode and prospectively followed for a minimum of 12 months, and to have included DNA collection in the study protocol. Analysis of genetic data is not reported here, and all participants were included in this analysis regardless of whether their genetic data were available. Details of recruitment, follow up, and where data has been previous published for each cohort can be found in the Supplementary Material. For this study, we removed cohorts where the number of TR cases was less than five (see Supplementary Material). Ten STRATA-G cohorts were included, with data collected in the Czech Republic (Spaniel et al., 2016) (1), France (Jongsma et al., 2018) (1), Italy (Jongsma et al., 2018; Tarricone et al., 2012) (1), Spain (Crespo-Facorro et al., 2007) (1), Switzerland (Baumann et al., 2013) (1), Turkey (Üçok et al., 2011) (1), and the UK (Barnes et al., 2008; Demjaha et al., 2017; Lally et al., 2016; Turkington et al., 2018) (4). All cohorts had ethical approval and participants provided written informed consent. The study was approved by an NHS Research Ethics Committee (15/SC/0021).

2.2. Treatment resistance (TR)

Participants were defined as TR if they met any of the following three definitions of TR (criteria varied across cohorts depending on the data available): either (1) prescription of or treatment with clozapine, or (2) treatment with two different antipsychotics at a therapeutic dose for a specified duration (each of at least 6 weeks' duration, with dosages in at least the mid-point of the licensed therapeutic range; data on the reason for switching was not available and therefore not included in the definition of TR), or (3) persistent psychotic symptoms and moderate functional impairment despite treatment with two different antipsychotics at a therapeutic dose for a specified duration of time. Participants not meeting any of these criteria were classified as non-treatment resistant (NTR). Further details on these criteria, broken down by cohort, can be found in Supplementary Table 6.

2.3. Predictors

Any variables recorded in more than one cohort, at baseline, were considered as potential predictors. We removed variables where >80 % of the data were missing (Supplementary Table 1), one variable from every pair with a Pearson's correlation >0.8 (Supplementary Table 2) to prevent multicollinearity (retaining the most clinically informative), and categorical variables where the number of events or non-events per category was less than ten.

2.4. Statistical analysis

2.4.1. Explanatory model

We imputed missing data using the mice package (van Buuren and Groothuis-Oudshoorn, 2011) for multiple imputation by chained equations (MICE). TR status, length of follow up, and an indicator for cohort were included. We ran MICE using 100 imputations, with ten iterations in the burn-in period. Performance was assessed by plotting the residuals and convergence of chains, exploring the range of imputed values, and comparing observed and imputed data (Nguyen et al., 2017).

We conducted logistic regressions to examine the associations between predictors and TR. Each of the 100 imputed datasets were analysed separately and then the results were pooled together based on Rubin's rules (Azur et al., 2011; Rubin, 1987). Regression coefficients were pooled by taking the average coefficient from all the imputed datasets and standard errors by combining the within imputation variance and the between imputation variance. We report the pooled estimates from univariable analyses and the multivariable model. Length of follow up and cohort were included as covariates in the multivariable model. As a measure of overall model fit, we report the pooled Nagelkerke R2 value (Harel, 2009) for the multivariable model.

2.4.2. Prediction model

We imputed missing data using the missForest package (Stekhoven and Buhlmann, 2012) for imputation by random forests (Breiman, 2001). Imputation performance was assessed using the out-of-bag imputation normalized root mean squared error (NRMSE) for continuous imputed data, and the proportion of falsely classified entries (PFC) for the categorical imputed data set (in both cases, a value close to 0 % indicates good performance and a value close to 100 % poor performance; we considered a value around 50 % to indicate a moderate performance) (Stekhoven and Buhlmann, 2012).

We conducted a ‘Least Absolute Shrinkage and regression Operator’ (LASSO) logistic regression to identify a model capable of predicting TR. LASSO is a form of regularised regression which can reduce overfitting when there are a large number of predictors (Hastie et al., 2009). We preferred lasso over other regularization methods (ridge or elastic net) because it tends to select a sparse and, therefore, more clinically practical model (Steyerberg, 2008). The model coefficients are shrunk towards zero, with some coefficients being assigned exactly zero to perform variable selection. The model was fitted using the penalised maximum likelihood procedure implemented in ‘glmnet’. The tuning parameter lambda (λ) was estimated on the imputed data using a grid of 100 tuning parameters. After repeated cross-validation (5 folds, 50 repeats), lambda was selected as the value which maximised the cross-validated area under the receiver operating characteristic curve (AUC) while being within one standard error of the maximum AUC (also known as the “1SE rule”). Simulations suggest that the 1SE model provides the best compromise between reliable variable selection and good prediction accuracy (Hastie et al., 2009). When the number of individuals in one group is substantially larger than the other (in our case, the NTR group), choosing the threshold that maximises accuracy will often produce a model that achieves good overall accuracy by simply classifying all participants into the larger group. Therefore, we chose the ‘best’ threshold, defined as the threshold that maximised the sum of the sensitivity and specificity (Perkins and Schisterman, 2006; Youden, 1950). The apparent performance of the model is presented in the Supplementary Material: we assessed the model performance using calibration (calibration slope and calibration-in-the-large) and discrimination (AUC) and report the model sensitivity (proportion of TR cases correctly predicted), specificity (proportion of NTR cases correctly predicted), positive predictive value (PPV; proportion of predicted TR cases that are indeed TR), negative predictive value (NPV; proportion of predicted NTR cases that are indeed NTR) at the optimal cut-off point for the predicted probability. The optimal cut-off point was defined as the threshold which maximised overall correct classification rates and minimised misclassification rates, while choosing the point on the receiver operating characteristic curve farthest from chance.

In the absence of an independent dataset with which to externally validate the model, we conducted internal, repeated cross-validation (5 folds, 50 repeats) to estimate the optimism of the model (the degree to which the accuracy of the model is inflated by overfitting) and recalibrated the beta coefficients (Steyerberg, 2008). The difference between the training and test performance estimates of each fold were averaged to obtain an estimate of the optimism. The optimism was then subtracted from the apparent performance measures to obtain the corrected (internally validated) performance measures. We report the corrected performance measures and the recalibrated regression coefficients.

2.4.3. Prediction model (LASSO logistic regression) – sensitivity analyses

We refitted the model to examine the stability of the discrimination performance measures across cohorts by using repeated 5-fold cross-validation (5 folds, 50 repeats). In addition, we repeated the model development using (i) a subsample of participants where TR was defined using only clozapine prescription, and (ii) a subsample with a diagnosis of schizophrenia (at the last known follow-up visit). Our main analysis included participants with a range of diagnoses and did not require clozapine treatment as a definition of TR because we wished to maximise our sample size. Our clozapine model uses a subgroup of participants where we are more confident in their classification as TR and our schizophrenia model reflects clinical guidelines which specify a diagnosis of schizophrenia as one of the criteria for TR.

2.5. Software

Pre-processing of individual cohorts was conducted in STATA version 14 (Stata Statistical Software: Release 14, StataCorp LP., College Station, TX). Analyses were performed in RStudio (v1.3.1073) using R (v4.0.2) with the exception of MICE and optimism-correction which used R (v4.0.0) in the Hawk cluster of Supercomputing Wales. Scripts used for analysis can be found at: https://github.com/sophiesmart/stratagprediction.

3. Results

The final dataset consisted of N = 2216 participants, from ten cohorts, of which 385 (17 %) were TR (Table 1). Data preparation is described in the Supplementary Material, after which we retained 19 predictors. The descriptive statistics for these predictors, as well as length of follow up, clozapine use, and schizophrenia diagnosis, are presented in Table 2. The mean proportion of missing data was 45.68 % (ranging from 0 % to 75.30 %; see Supplementary Table 1).

Table 1.

STRATA-G cohorts included in this study, stratified by treatment resistant status.

Non-TR (%) TR (%) Total
AESOP London 210 (73.43) 76 (26.57) 286
Belfast 138 (90.20) 15 (9.80) 153
Bologna 43 (86.00) 7 (14.00) 50
GAP London 216 (75.26) 71 (24.74) 287
Istanbul 98 (72.59) 37 (27.41) 135
Lausanne 250 (89.29) 30 (10.71) 280
Paris 26 (81.25) 6 (18.75) 32
Prague 76 (54.29) 64 (45.71) 140
Santander 410 (86.32) 65 (13.68) 475
West London 364 (96.30) 14 (3.70) 378
Total 1831 (82.63) 385 (17.37) 2216
Open in a new tab

Table 2.

Summary statistics for the first-episode variables, covariates, and the data used to define subsamples of the data, stratified by treatment resistant status.

Overall Non-TR TR Statistica P-value
N 2216 1831 385
Clozapine (yes; %) 238 (12.00) 0 (0.00) 238 (68.40)
Length of follow up (years; mean (SD)) 4.41 (3.21) 4.17 (2.93) 5.53 (4.12) 6.17 <0.001
Age at onset (years; mean (SD)) 26.87 (9.26) 27.35 (9.48) 24.41 (7.56) −6.02 <0.001
Duration of untreated psychosis (days; mean (SD)) 232.41 (619.55) 224.62 (611.71) 276.37 (661.64) 1.20 0.231
Gender (female; %) 820 (39.10) 713 (40.50) 107 (31.70) 9.38 0.002
BMI (mean (SD)) 23.87 (4.63) 23.76 (4.71) 24.50 (4.17) 1.52 0.129
Current relationship (yes; %) 300 (19.50) 269 (21.50) 31 (10.90) 16.67 <0.001
Living situation (%) 2.75 0.251
 Alone 258 (25.2) 208 (24.9) 50 (26.7)
 With family 667 (65.3) 553 (66.2) 114 (61.0)
 With others 97 (9.5) 74 (8.9) 23 (12.3)
Employment (employed; %) 213 (39.9) 174 (40.7) 39 (36.4) 0.66 0.461
Education (years; mean (SD)) 12.52 (3.17) 12.69 (3.19) 11.73 (2.93) −2.91 0.004
Cannabis (yes; %) 489 (45.9) 415 (45.6) 74 (48.1) 0.33 0.601
Tobacco (no; %) 423 (43.7) 381 (45.0) 42 (34.7) 4.59 0.043
Alcohol (no; %) 370 (40.5) 339 (42.1) 31 (29.0) 6.71 0.011
PANSS positive subscale (mean (SD)) 14.73 (6.04) 14.46 (6.03) 16.00 (5.97) 2.04 0.042
SAPS (mean (SD)) 11.39 (3.97) 11.30 (3.95) 12.09 (4.06) 2.02 0.044
SANS (mean (SD)) 8.09 (6.51) 7.97 (6.44) 9.07 (7.01) 1.70 0.090
BPRS (mean (SD)) 63.02 (13.88) 62.69 (13.79) 64.70 (14.29) 1.32 0.186
GAF (mean (SD)) 50.32 (17.07) 51.68 (17.45) 45.46 (14.73) −3.45 <0.001
Ethnicity (%) 4.67 0.098
 European 1071 (70.5) 873 (71.3) 198 (67.1)
 African 308 (20.3) 235 (19.2) 73 (24.7)
 Asian/mixed/other 140 (9.2) 116 (9.5) 24 (8.1)
Schizophrenia diagnosis (schizophrenia; %) 580 (26.2) 470 (25.7) 110 (28.6) 1.39 0.262
Open in a new tab

Abbreviations: BMI, Body Mass Index; BPRS, Brief Psychiatric Rating Scale; GAF, Global Assessment of Functioning; PANSS, Positive and Negative Syndrome Scale; SANS, Scale for the Assessment of Negative Symptoms; SAPS, Scale for the Assessment of Positive Symptoms; SD, standard deviation.

a

Continuous variables were compared across groups using independent samples t-tests and categorical variables were compared across groups using Chi-squared tests (with a Monte Carlo test, with 2000 repeats, to simulate p-values). For continuous variables, a F-test for homogeneity in the variances was performed and, consequently, a Welch Two Sample t-test assuming unequal variances was used for length of follow up and age of onset.

3.1. Explanatory model

Missing data imputation performed well: chains of trace plots showed convergence, the distributions of the imputed values were consistent with the distributions of the non-imputed values, and the residuals of each imputed dataset showed that the assumptions were met (see Supplementary Material). In univariable logistic regression models, eight predictors were associated with TR. Younger age of onset, being male, higher BMI, not being in a relationship, fewer years in education, higher SAPS score, lower GAF score (worse functioning), and being of African ethnicity (compared to European ethnicity) all increased the odds of being TR (see Supplementary Table 3).

However, in the multivariable logistic regression, controlling for all other predictors as well as cohort and length of follow up, only younger age of onset and fewer years in education increased the odds of being TR (Table 3; Supplementary Table 4). The R2 for this model was 26.83 %.

Table 3.

Results of the multivariable logistic regression (explanatory model).

Odds ratio 95 % CI P-value
Age at onset (years) 0.96 0.94 0.98 <0.001***
Duration of untreated psychosis (days) 1.00 1.00 1.00 0.821
Gender (female) 0.89 0.53 1.25 0.518
BMI 1.05 0.99 1.11 0.089
Current relationship (yes) 0.64 0.12 1.17 0.098
Living situation (alone) 0.92 0.43 1.42 0.754
Living situation (with others) 1.52 0.94 2.09 0.160
Employment (employed) 1.03 0.43 1.63 0.922
Education (years) 0.87 0.79 0.95 0.001***
Cannabis (yes) 1.01 0.45 1.57 0.981
Tobacco (no) 1.07 0.35 1.79 0.858
Alcohol (no) 0.87 0.17 1.56 0.686
PANSS positive subscale 1.04 0.95 1.12 0.435
SAPS 1.09 0.98 1.19 0.134
SANS 1.00 0.94 1.07 0.887
BPRS 0.98 0.95 1.02 0.297
GAF 0.98 0.95 1.00 0.059
Ethnicity (African) 1.00 0.15 1.84 0.991
Ethnicity (Asian/Mixed/Other) 0.85 0.19 1.51 0.630
Open in a new tab

NB: Cohort and length of follow up were included as covariates in the model.

Beta coefficients and standard errors for all predictors and covariates are in Supplementary Table 3 (univariable) and Supplementary Table 4 (multivariable).

The reference category for living situation was ‘living with family’ and for ethnicity the reference category was ‘European’.

Abbreviations: BMI, Body Mass Index; BPRS, Brief Psychiatric Rating Scale; GAF, Global Assessment of Functioning; PANSS, Positive and Negative Syndrome Scale; SANS, Scale for the Assessment of Negative Symptoms; SAPS, Scale for the Assessment of Positive Symptoms.

3.2. Prediction model

Missing data imputation performed moderately well: the NRMSE was 52.25 % and PFC was 19.15 %. In the 1SE LASSO regression model, seven variables were selected (Table 4): age of onset, years in education, gender, BMI, relationship status, alcohol use, and positive symptoms. The model predicted TR with an apparent AUC of 0.65 (apparent performance measures for the prediction model and all sensitivity analyses are in the Supplementary Material while the model coefficients are in Supplementary Table 5). The optimal cut-off for the predicted probabilities was 20.21 %. After correcting for optimism, the model AUC was 0.59 (accuracy = 64.33 %, sensitivity = 48.31 %, specificity = 76.19 %, PPV = 27.49 %, and NPV = 87.55 %). When looking across cohorts, the apparent AUC ranged between 0.59 and 0.77 (see Supplementary Material).

Table 4.

Results of the multivariable LASSO regressions (prediction models).

 Original Model (N = 2216)
 Clozapine-only Model (N = 1986)
 Schizophrenia-only Model (N = 580)
Log Odds Effect Dir. Log Odds Effect Dir. Log Odds Effect Dir.
Age at onset (years) −0.03 − −0.04 − −0.02 −
Duration of untreated psychosis (days)
Gender (female) −0.09 − −0.13 −
BMI 0.03 +
Current relationship (yes) −0.37 − −0.06 − −0.76 −
Living (alone)
Living (with others)
Employment (employed) −0.01 − −0.35 −
Education (years) −0.06 −
Cannabis (yes)
Tobacco (no) −0.24 −
Alcohol (no) −0.48 − −0.11 −
PANSS positive subscale −0.002 −
SAPS
SANS 0.03 +
BPRS
GAF
Ethnicity (African) 0.25 +
Ethnicity (Asian/Mixed/Other)
Open in a new tab

NB: The clozapine-only model was developed using a subsample of participants where TR was defined using only clozapine prescription. The schizophrenia-only model was developed using a subsample of participants with a diagnosis of schizophrenia (at the last known follow-up visit). Apparent and recalibrated regression beta coefficients are presented side by side in Supplementary Table 5.

The reference category for living situation was ‘living with family’ and for ethnicity the reference category was ‘European’.

Abbreviations: BMI, Body Mass Index; BPRS, Brief Psychiatric Rating Scale; GAF, Global Assessment of Functioning; PANSS, Positive and Negative Syndrome Scale; SANS, Scale for the Assessment of Negative Symptoms; SAPS, Scale for the Assessment of Positive Symptoms.

After correcting for optimism, the clozapine-model AUC was 0.61 (accuracy = 55.79 %, sensitivity = 65.97 %, specificity = 55.61 %, PPV = 16.83 %, and NPV = 92.41 %), while the schizophrenia-model AUC was 0.55 (accuracy = 48.97 %, sensitivity = 75.12 %, specificity = 49.80 %, PPV = 21.73 %, and NPV = 87.13 %). It is hard to draw direct comparisons with the original model due to the different sample sizes, but when the same predictors were selected by more than one model, the direction of effect was consistent (Table 4).

4. Discussion

The results of this study show that treatment resistance can be predicted with modest accuracy, using only clinical and demographic measures, when a patient with psychosis first presents to clinical services at the start of their illness. After correcting for optimism, the model was able to correctly identify 48 % of TR patients and 76 % of responders. 28 % of TR patients were correctly classified while 88 % of responders were correctly classified. This is the largest analysis of TR conducted to date; combining data from many research groups and driven by an international collaboration.

In our explanatory model, for each one-year decrease in age at onset there was an increase in the odds of developing TR. Previous research has found that younger age of onset is associated with both TR (Legge et al., 2020; Wimberley et al., 2016), a broad range of poor outcomes in schizophrenia (Immonen et al., 2017), and poor response to treatment in other disorders, including depression (Perlman et al., 2019) and bipolar (Hui et al., 2019). Immonen et al. (2017) suggest that younger age of onset results in poor outcomes because it disrupts social and cognitive development. There is evidence to support this claim: in a recent meta-analysis (Rajji et al., 2009), adult and adolescent-onset schizophrenia were both associated with cognitive deficits across multiple domains, but those with adolescent-onset schizophrenia had greater deficits on domains including IQ, executive functioning, and verbal memory. Adolescent-onset psychosis has also been associated with more negative symptoms at first-presentation (Downs et al., 2019) and worse premorbid functioning between the ages of 16 and 18 (Ballageer et al., 2005). Furthermore, cognitive deficits, particularly poor verbal memory (de Bartolomeis et al., 2013; Joober et al., 2002), an increased burden of negative symptoms (Iasevoli et al., 2018b), and worse premorbid functioning (Albert et al., 2011; Levine and Rabinowitz, 2010) have all been associated with non-response to antipsychotic treatment. In contrast, a small study of age of onset and TR found that TR explained most associations with symptom severity and functional impairment; only a few independent effects of age of onset were identified (one being poorer visuospatial memory in early onset TR participants) and the authors suggest that the associations observed between earlier age of onset and worse clinical outcomes could be an indirect effect of a TR subtype of schizophrenia (Iasevoli et al., 2022). Our finding that only younger age of onset and fewer years in education are associated with TR, after controlling for other factors, is in line with the hypothesis that TR has a neurodevelopmental aetiology. However, given that age of onset and years of education are independently associated with TR, there may be more than one path from disrupted cognitive and social development to TR.

Our prediction model automatically selected age of onset and years in education for inclusion in the model. Some of the other five predictors selected by the model (gender, BMI, relationship status, alcohol use, positive symptoms) have been previously associated with TR, but for the others their selection in the prediction model justifies further work in understanding their role in TR. Explanatory models of TR using longitudinal data have identified younger age of onset (Chan et al., 2021; Wimberley et al., 2016), being male (Szymanski et al., 1995), and never marrying (Emsley et al., 2006) as factors associated with TR. Previous studies comparing TR and NTR patients have failed to find a difference in the number of years in education (Frydecka et al., 2016; Iasevoli et al., 2018c), alcohol use (Chen et al., 2018), and positive symptoms (Freitas et al., 2019; Iasevoli et al., 2018a), although one study of treatment non-responders vs responders found a difference in BMI (Chiliza et al., 2015). It is important to note, however, that our model coefficients can be only interpreted in the context of their contribution to the prediction of TR. The coefficients are biased due to the penalization and are hence not an estimate of a population parameter.

The sensitivity and specificity of our prediction model suggests that, after external validation, it would not be suited for use in clinical settings due to the high misclassification rate, at least not as the sole decision aid for antipsychotic treatment which can have burdensome side effects. Nevertheless, the model as it stands would be more cost effective than current treatment practice in the UK. Jin et al. (2019) calculated the cost of implementing a prediction model for TR in clinical practice. They considered a test that could be used after patients had failed to respond to one antipsychotic. Those who were unlikely to respond to a second antipsychotic, according to the test, would be offered clozapine as a second-line treatment. Compared to treatment as usual (i.e. clozapine as a third-line treatment for all patients), and taking into consideration the side effects of first-line antipsychotics and clozapine, using a prediction model in this scenario would result in an improvement of 0.10 quality-adjusted life years (QALYs) and reduce healthcare costs by £7363 per person. Their analysis suggests that a predictive model would be more cost effective than treatment as usual even if it only accurately identified 6 % of TR patients and 50 % of NTR patients.

Although an AUC of 0.59 means the prediction model may not be of clinical use, it provides a base on which to improve stratified medicine for TR. The addition of other predictors may improve model performance and likely candidates include additional clinical data (Ajnakina et al., 2020; Legge et al., 2020), genetic risk for TR (Pardiñas et al., 2022), multiple polygenic risk scores for other traits (Krapohl et al., 2018), anterior cingulate cortex glutamate levels (Egerton et al., 2021), and lack of response in the first two weeks of treatment (Samara et al., 2015). It could also be useful in the context of randomized controlled trials; used to recruit samples enriched for TR (Vickers et al., 2006), added as a covariate (Hernández et al., 2004), or to stratify existing data for further analysis.

4.1. Strengths and limitations

Our study uses only data collected at first-episode, encompassing the full range of first episode presentations of psychosis. Many studies rely on cross-sectional or retrospective case-control designs which suffer from selection bias since they tend to include only patients who remain in mental health services and exclude those with good outcomes. Furthermore, cross-sectional or retrospective studies cannot exclude reverse causation. We cannot therefore assume that findings from these studies will generalise to first-episode patients.

One of the strengths of this analysis is that we have estimated, separately, the strength of associations, between characteristics at first-episode and TR, and the predictive accuracy of these same characteristics (Shmueli, 2010). A LASSO regression is an optimal method for prediction in this context. Variable selection is an automatic part of the model, which produces a parsimonious model (Kuhn and Johnson, 2013); although there is always a possibility that an important predictor is not selected, models with fewer predictors are easier to apply to new samples, since the amount of information required from participants is reduced, as well as to existing samples, as there is a higher chance that all the required data has been collected. LASSO regression combined with random forests imputation also prevents predictors selected by chance from having excess importance; this is important in our analysis since a limitation of the random forests imputation method is that it can give more weight to these false predictors when they have a large proportion of missingness and when the percentage of missingness varies across predictors (Lu and Petkova, 2014). LASSO handles multicollinearity well, a consistent occurrence in clinical samples, and we took the further step of removing highly correlated variables before any analyses. LASSO regressions have been shown to work well in clinical data sets (Kuhn and Johnson, 2013), and have good predictive accuracy when determining psychosis onset (Ciarleglio et al., 2018), schizophrenia diagnosis (Salvador et al., 2017), and response to clozapine (Fonseca de Freitas et al., 2022).

After acknowledging that our sample size may have limited the power of our analyses, the main limitations of our statistical methodology are that we have not been able to externally validate our prediction model (and split-sample validation is not recommended; Austin and Steyerberg, 2017), an important step in evaluating the clinical utility of a model (Kapur et al., 2012). An equally important limitation is the high proportion of missingness in our data; a consequence of combining multiple pre-existing cohorts. The percentage of missingness may have affected the imputations and introduced some bias to the model regression coefficients (Seijo-Pardo et al., 2019). In addition, variable selection for prediction models can be unreliable for predictors with a high proportion of missingness. For predictors with a high proportion of missingness, we cannot generalise that selected predictors are important or that non-selected predictors are not important; we can only conclude that they are important and not important for our specific prediction model after accounting for random forests imputation. A final limitation of our statistical methodology is that we did not allow for nonlinear predictors.

There are two key limitations of our participant sample: this is a sample of first-episode psychosis patients – not all of whom were later diagnosed with schizophrenia – and the rates of TR vary between the cohorts. We did not restrict the main analysis to those with a schizophrenia diagnosis to ensure we had the largest sample size possible for model development (not all cohorts recorded diagnosis at follow up). However, this is comparable to other work predicting TR (Legge et al., 2020) and may be more clinically useful since diagnosis is often unclear at first-episode; a recent meta-analysis found that patients eventually diagnosed with schizophrenia are often diagnosed with other psychotic disorders at first-episode (Fusar-Poli et al., 2016). Regardless, this may still have biased our sample and, as a consequence, we were not able to use the Treatment Response and Resistance in Psychosis (TRRIP) working group criteria for TR (Howes et al., 2017). As a result, different definitions of TR were used across different cohorts (see Supplementary Table 6), which may have led to the substantial variation in the rates of TR. According to a recent meta-analysis, the rate of treatment resistant schizophrenia – as defined by the TRRIP criteria – is estimated to be 22.8 % (95 % CI 19.1–27.0 %) among all first-episode cohorts and 24.4 % (95 % CI 19.5–30.0 %) among first-episode schizophrenia cohorts (Siskind et al., 2022). The Istanbul cohort was the only cohort to exclusively include participants with a diagnosis of schizophrenia, and its rate of TR (27.41 %) fell within the estimated 95 % CI for first-episode schizophrenia cohorts. Of the first-episode psychosis cohorts, only two cohorts were within the estimated 95 % CI (GAP London, 24.74 %, and AESOP London, 26.57 %). The Prague cohort had a substantially higher rate of TR (45.71 %), but notably this cohort was the only one where a measure of clinical response was not used. The West London cohort had the lowest rate of TR (3.70 %) and was the only cohort where TR was defined using clozapine only. The other five cohorts with lower rates of TR, than estimated in the meta-analysis, were the only cohorts to include a measure of functional response in the definition of TR (at least moderate functional impairment measured using the GAF or CGI scale), which may have proved too stringent a threshold when applied to secondary data analysis. Our prediction model may have had better accuracy if there was less measurement error in our outcome. The differing rates of TR reflect one of the statistical problems of international research consortia (Budin-Ljøsne et al., 2014): the violation of the assumption of homogeneity. Due to the proportion of missing data in our sample, we were not able to use methods such as leave-one-cohort-out cross validation or meta-analyses to estimate the heterogeneity. However, Bühlmann and van de Geer (2018) argue that sample heterogeneity can produce more robust estimates in causal modelling and while heterogeneity is a problem for robust prediction, our model still discriminated between TR and NTR cases better than chance given all the described problems.

4.2. Conclusion

Our results show that younger age of onset and fewer years in education at first-episode of psychosis are independently associated with TR. We have developed and internally validated a parsimonious prediction model for TR. This model can quantify future individuals' probability of developing TR with modest predictive accuracy and using only clinical and demographic information recorded at first-episode. We have identified novel predictors of TR (BMI, alcohol use, years in education, positive symptoms) and confirmed existing predictors (age of onset, gender, relationship status). At present, this is the largest sample of prospective psychosis patients where TR status is known. Further work is required to build a clinically useful model for predicting TR; but there is scope to develop a new model using our identified predictors and other known predictors from the wider literature.

Data availability

Data can be requested by contacting james.maccabe@kcl.ac.uk and completing a STRATA secondary analysis proposal form which will then be circulated to all consortium members for review.

Twitter

NEW prediction model for treatment resistance in schizophrenia, secondary data-analysis of prospective first-episode cohorts, good with lasso, seeking like-minded researchers to replicate and update. GSOH needed to read this tweet.

Funding

This work was supported by a Stratified Medicine Programme grant to JHM from the Medical Research Council (grant number MR/L011794/1 which funded the research and supported S.E.S., D.A., A.F.P, L.K., R.M.M., D.S., J.T.R.W, & J.H.M.); funding from the National Institute for Health Research Biomedical Research Centre at South London and Maudsley National Health Service Foundation Trust and King's College London to D.A. and D.S; and funding from the Collaboration for Leadership in Applied Health Research and Care (CLAHRC) South London at King's College Hospital National Health Service Foundation Trust to S.E.S.

The views expressed are those of the author(s) and not necessarily those of the Medical Research Council, National Health Service, the National Institute for Health Research, or the Department of Health.

The AESOP (London, UK) cohort was funded by the UK Medical Research Council (Ref: G0500817). The Belfast (UK) cohort was funded by the Research and Development Office of Northern Ireland. The Bologna (Italy) cohort was funded by the European Community's Seventh Framework Program under grant agreement (agreement No. HEALTH-F2-2010–241909, Project EU-GEI). The GAP (London, UK) cohort was funded by the UK National Institute of Health Research (NIHR) Specialist Biomedical Research Centre for Mental Health, South London and Maudsley NHS Mental Health Foundation Trust (SLaM) and the Institute of Psychiatry, Psychology, and Neuroscience at King's College London; Psychiatry Research Trust; Maudsley Charity Research Fund; and the European Community's Seventh Framework Program grant (agreement No. HEALTH-F2-2009-241909, Project EU-GEI). The Lausanne (Switzerland) cohort was funded by the Swiss National Science Foundation (no. 320030_135736/1 to P.C. and K.Q.D., no 320030-120686, 324730-144064 and 320030-173211 to C.B.E and P.C., and no 171804 to LA); National Center of Competence in Research (NCCR) “SYNAPSY - The Synaptic Bases of Mental Diseases” from the Swiss National Science Foundation (no 51AU40_125759 to PC and KQD); and Fondation Alamaya (to KQD). The Oslo (Norway) cohort was funded by the Research Council of Norway (#223273/F50, under the Centers of Excellence funding scheme, #300309, #283798) and the South-Eastern Norway Regional Health Authority (#2006233, #2006258, #2011085, #2014102, #2015088 to IM, #2017-112). The Paris (France) cohort was funded by European Community's Seventh Framework Program grant (agreement No. HEALTH-F2-2010–241909, Project EU-GEI). The Prague (Czech Republic) cohort was funded by the Ministry of Health of the Czech Republic (Grant Number: NU20-04-00393). The Santander (Spain) cohort was funded by the following grants (to B.C.F): Instituto de Salud Carlos III, FIS 00/3095, PI020499, PI050427, PI060507, Plan Nacional de Drogas Research Grant 2005-Orden sco/3246/2004, and SENY Fundatio Research Grant CI 2005-0308007, Fundacion Marques de Valdecilla A/02/07 and API07/011. SAF2016-76046-R and SAF2013-46292-R (MINECO and FEDER). The West London (UK) cohort was funded The Wellcome Trust (Grant Number: 042025; 052247; 064607).

CRediT authorship contribution statement

S.E.S., D.A., A.F·P., R.M.M., J.T.R.W., D·S., and J.M. designed the study, interpreted the data, and revised the manuscript. S.E.S. acquired the data for secondary data analysis. S.E.S. and D.A. conducted the analysis. S.E.S. drafted the manuscript. All other authors contributed to the interpretation of the data and revised the manuscript.

Declaration of competing interest

The Authors declare no Competing Non-Financial Interests but the following Competing Financial Interests: J.T.R.W. is an investigator on a grant from Takeda Pharmaceuticals Ltd. to Cardiff University, for a project unrelated to the work presented here. S.E.S. is employed on this grant. M.D.F. has received a fee for educational seminars from Lundbeck and Janssen. O.A.A. is a consultant to HealthLytix and has received speakers honorarium from Lundbeck and Sunovion. T.R.E.B. has been a member of an advisory board for Gedeon Richter. C.B.E. received honoraria for conferences from Forum pour la formation médicale, Janssen-Cilag, Lundbeck, Otsuka, Sandoz, Servier, Sunovion, Sysmex Suisse AG, Takeda, Vifor-Pharma, and Zeller in the past 3 years. B.C.F. has received honoraria for participation as a consultant and/or as a speaker at educational events from ADAMED, Mylan, Angelini, Janssen Johnson & Johnson, Lundbeck, and Otsuka Pharmaceuticals. R.M.M. has received payments for non-promotional lectures from Janssen, Otsuka, Sunovian, and Lundbeck. J.H.M. has received research funding from H Lundbeck.

Acknowledgments

We acknowledge the support of the Supercomputing Wales project, which is part-funded by the European Regional Development Fund (ERDF) via Welsh Government. We would like to thank the wider STRATA-G Consortium.

Footnotes

Appendix A

Supplementary data to this article can be found online at https://doi.org/10.1016/j.schres.2022.09.009.

Appendix A. Supplementary data

Supplementary material

mmc1.docx (18.1MB, docx)

References

  1. Ajnakina O., Agbedjro D., Lally J., Forti M.D., Trotta A., Mondelli V., Pariante C., Dazzan P., Gaughran F., Fisher H.L., David A., Murray R.M., Stahl D. Predicting onset of early- and late-treatment resistance in first-episode schizophrenia patients using advanced shrinkage statistical methods in a small sample. Psychiatry Res. 2020;294 doi: 10.1016/j.psychres.2020.113527. [DOI] [PubMed] [Google Scholar]
  2. Albert N., Bertelsen M., Thorup A., Petersen L., Jeppesen P., Le Quack P., Krarup G., Jorgensen P., Nordentoft M. Predictors of recovery from psychosis analyses of clinical and social factors associated with recovery among patients with first-episode psychosis after 5 years. Schizophr. Res. 2011;125(2–3):257–266. doi: 10.1016/j.schres.2010.10.013. [DOI] [PubMed] [Google Scholar]
  3. Andrews A., Knapp M., McCrone P., Parsonage M., Trachtenberg M. 2012. Effective Interventions in Schizophrenia: The Economic Case. A Report Prepared for the Schizophrenia Commission. Rethink Mental Illness, London. [Google Scholar]
  4. Austin P.C., Steyerberg E.W. Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models. Stat. Methods Med. Res. 2017;26(2):796–808. doi: 10.1177/0962280214558972. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Azur M.J., Stuart E.A., Frangakis C., Leaf P.J. Multiple imputation by chained equations: what is it and how does it work? Int. J. Methods Psychiatr. Res. 2011;20(1):40–49. doi: 10.1002/mpr.329. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Ballageer T., Malla A., Manchanda R., Takhar J., Haricharan R. Is adolescent-onset first-episode psychosis different from adult onset? J. Am. Acad. Child Adolesc. Psychiatry. 2005;44(8):782–789. doi: 10.1097/01.chi.0000164591.55942.ea. [DOI] [PubMed] [Google Scholar]
  7. Barnes T.R.E., Leeson V.C., Mutsatsa S.H., Watt H.C., Hutton S.B., Joyce E.M. Duration of untreated psychosis and social function: 1-year follow-up study of first-episode schizophrenia. Br. J. Psychiatry. 2008;193(3):203–209. doi: 10.1192/bjp.bp.108.049718. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Baumann P.S., Crespi S., Marion-Veyron R., Solida A., Thonney J., Favrod J., Bonsack C., Do K.Q., Conus P. Treatment and early intervention in psychosis program (TIPP-Lausanne): implementation of an early intervention programme for psychosis in Switzerland. Early Interv. Psychiatry. 2013;7(3):322–328. doi: 10.1111/eip.12037. [DOI] [PubMed] [Google Scholar]
  9. Brain C., Kymes S., DiBenedetti D.B., Brevig T., Velligan D.I. Experiences, attitudes, and perceptions of caregivers of individuals with treatment-resistant schizophrenia: a qualitative study. BMC Psychiatry. 2018;18(1):253. doi: 10.1186/s12888-018-1833-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Breiman L. Random forests. Mach. Learn. 2001;45(1):5–32. [Google Scholar]
  11. Budin-Ljøsne I., Isaeva J., Maria Knoppers B., Marie Tassé A., Shen H.-Y., McCarthy M.I., Harris J.R., Consortium E. Data sharing in large research consortia: experiences and recommendations from ENGAGE. Eur. J. Hum. Genet. 2014;22(3):317–321. doi: 10.1038/ejhg.2013.131. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Bühlmann P., van de Geer S. Statistics for big data: a perspective. Statist. Probab. Lett. 2018;136:37–41. [Google Scholar]
  13. Bzdok D., Meyer-Lindenberg A. Machine learning for precision psychiatry: opportunities and challenges. Biol. Psychiatry. 2018;3(3):223–230. doi: 10.1016/j.bpsc.2017.11.007. [DOI] [PubMed] [Google Scholar]
  14. Chan S.K.W., Chan H.Y.V., Honer W.G., Bastiampillai T., Suen Y.N., Yeung W.S., Lam M., Lee W.K., Ng R.M.K., Hui C.L.M., Chang W.C., Lee E.H.M., Chen E.Y.H. Predictors of treatment-resistant and clozapine-resistant schizophrenia: a 12-year follow-up study of first-episode schizophrenia-spectrum disorders. Schizophr. Bull. 2021;47(2):485–494. doi: 10.1093/schbul/sbaa145. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Chen Y.-L., Chen K.-P., Chiu C.-C., Tai M.-H., Lung F.-W. Early predictors of poor treatment response in patients with schizophrenia treated with atypical antipsychotics. BMC Psychiatry. 2018;18(1):376. doi: 10.1186/s12888-018-1950-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Chiliza B., Asmal L., Kilian S., Phahladira L., Emsley R. Rate and predictors of non-response to first-line antipsychotic treatment in first-episode schizophrenia. Hum. Psychopharmacol. Clin. Exp. 2015;30(3):173–182. doi: 10.1002/hup.2469. [DOI] [PubMed] [Google Scholar]
  17. Ciarleglio A.J., Brucato G., Masucci M.D., Altschuler R., Colibazzi T., Corcoran C.M., Crump F.M., Horga G., Lehembre-Shiah E., Leong W., Schobel S.A., Wall M.M., Yang L.H., Lieberman J.A., Girgis R.R. A predictive model for conversion to psychosis in clinical high-risk patients. Psychol. Med. 2018:1–10. doi: 10.1017/S003329171800171X. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Collins G.S., Reitsma J.B., Altman D.G., Moons K.G. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. J. Clin. Epidemiol. 2015;68:134–143. doi: 10.1016/j.jclinepi.2014.11.010. [DOI] [PubMed] [Google Scholar]
  19. Crespo-Facorro B., Pelayo-Teran J.M., Perez-Iglesias R., Ramirez-Bonilla M., Martinez-Garcia O., Pardo-Garcia G., Vazquez-Barquero J.L. Predictors of acute treatment response in patients with a first episode of non-affective psychosis: sociodemographics, premorbid and clinical variables. J. Psychiatr. Res. 2007;41(8):659–666. doi: 10.1016/j.jpsychires.2006.05.002. [DOI] [PubMed] [Google Scholar]
  20. de Bartolomeis A., Balletta R., Giordano S., Buonaguro E.F., Latte G., Iasevoli F. Differential cognitive performances between schizophrenic responders and non-responders to antipsychotics: correlation with course of the illness, psychopathology, attitude to the treatment and antipsychotics doses. Psychiatry Res. 2013;210(2):387–395. doi: 10.1016/j.psychres.2013.06.042. [DOI] [PubMed] [Google Scholar]
  21. Demjaha A., Lappin J.M., Stahl D., Patel M.X., MacCabe J.H., Howes O.D., Heslin M., Reininghaus U.A., Donoghue K., Lomas B., Charalambides M., Onyejiaka A., Fearon P., Jones P., Doody G., Morgan C., Dazzan P., Murray R.M. Antipsychotic treatment resistance in first-episode psychosis: prevalence, subtypes and predictors. Biol. Psychiatry. 2017;47(11):1981–1989. doi: 10.1017/S0033291717000435. [DOI] [PubMed] [Google Scholar]
  22. Downs J., Dean H., Lechler S., Sears N., Patel R., Shetty H., Hotopf M., Ford T., Kyriakopoulos M., Diaz-Caneja C.M., Arango C., MacCabe J.H., Hayes R.D., Pina-Camacho L. Negative symptoms in early-onset psychosis and their association with antipsychotic treatment failure. Schizophr. Bull. 2019;45(1):69–79. doi: 10.1093/schbul/sbx197. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Egerton A., Murphy A., Donocik J., Anton A., Barker G.J., Collier T., Deakin B., Drake R., Eliasson E., Emsley R., Gregory C.J., Griffiths K., Kapur S., Kassoumeri L., Knight L., Lambe E.J.B., Lawrie S.M., Lees J., Lewis S., Lythgoe D.J., Matthews J., McGuire P., McNamee L., Semple S., Shaw A.D., Singh K.D., Stockton-Powdrell C., Talbot P.S., Veronese M., Wagner E., Walters J.T.R., Williams S.R., MacCabe J.H., Howes O.D. Dopamine and glutamate in antipsychotic-responsive compared with antipsychotic-nonresponsive psychosis: a multicenter positron emission tomography and magnetic resonance spectroscopy study (STRATA) Schizophr. Bull. 2021;47(2):505–516. doi: 10.1093/schbul/sbaa128. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Emsley R., Oosthuizen P.P., Kidd M., Koen L., Niehaus D.J., Turner H.J. Remission in first-episode psychosis: predictor variables and symptom improvement patterns. J. Clin. Psychiatry. 2006;67(11):1707–1712. doi: 10.4088/jcp.v67n1106. [DOI] [PubMed] [Google Scholar]
  25. Fonseca de Freitas D., Kadra-Scalzo G., Agbedjro D., Francis E., Ridler I., Pritchard M., Sheety H., Segev A., Casetta C., Smart S., Downs J., Christensen S.R., Bak N., Kinon B.J., Stahl D., Maccabe J., Hayes R. Using a statistical learning approach to identify sociodemographic and clinical predictors of response to clozapine. J. Psychopharm. 2022;36(4):498–506. doi: 10.1177/02698811221078746. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Freitas R., Dos Santos B., Altamura C., Bernasconi C., Corral R., Evans J., Malla A., Krebs M.O., Nordstroem A.L., Zink M., Haro J.M., Elkis H. Can the positive and negative syndrome scale (PANSS) differentiate treatment-resistant from non-treatment-resistant schizophrenia? A factor analytic investigation based on data from the pattern cohort study. Psychiatry Res. 2019;276:210–217. doi: 10.1016/j.psychres.2019.05.002. [DOI] [PubMed] [Google Scholar]
  27. Frydecka D., Beszłej J.A., Gościmski P., Kiejna A., Misiak B. Profiling cognitive impairment in treatment-resistant schizophrenia patients. Psychiatry Res. 2016;235:133–138. doi: 10.1016/j.psychres.2015.11.028. [DOI] [PubMed] [Google Scholar]
  28. Fusar-Poli P., Cappucciati M., Rutigliano G., Heslin M., Stahl D., Brittenden Z., Caverzasi E., McGuire P., Carpenter W.T. Diagnostic stability of ICD/DSM first episode psychosis diagnoses: meta-analysis. Schizophr. Bull. 2016;42(6):1395–1406. doi: 10.1093/schbul/sbw020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Griffiths K., Millgate E., Egerton A., MacCabe J.H. Demographic and clinical variables associated with response to clozapine in schizophrenia: a systematic review and meta-analysis. Psychol. Med. 2021:1–11. doi: 10.1017/S0033291721000246. [DOI] [PubMed] [Google Scholar]
  30. Harel O. The estimation of R 2 and adjusted R 2 in incomplete data sets using multiple imputation. J. Appl. Stat. 2009;36(10):1109–1118. [Google Scholar]
  31. Hastie T., Tibshirani R., Friedman J. Springer; New York, NY: 2009. The Elements of Statistical Learning. [Google Scholar]
  32. Hemingway H., Croft P., Perel P., Hayden J.A., Abrams K., Timmis A., Briggs A., Udumyan R., Moons K.G.M., Steyerberg E.W., Roberts I., Schroter S., Altman D.G., Riley R.D. Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes. BMJ. 2013;346 doi: 10.1136/bmj.e5595. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Hernández A.V., Steyerberg E.W., Habbema J.D.F. Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements. J. Clin. Epidemiol. 2004;57(5):454–460. doi: 10.1016/j.jclinepi.2003.09.014. [DOI] [PubMed] [Google Scholar]
  34. Howes O.D., Vergunst F., Gee S., McGuire P., Kapur S., Taylor D. Adherence to treatment guidelines in clinical practice: study of antipsychotic treatment prior to clozapine initiation. Br. J. Psychiatry. 2012;201(6):481–485. doi: 10.1192/bjp.bp.111.105833. [DOI] [PubMed] [Google Scholar]
  35. Howes O.D., McCutcheon R., Agid O., et al. Treatment-resistant schizophrenia: treatment response and resistance in psychosis (TRRIP) working group consensus guidelines on diagnosis and terminology. Am. J. Psychiatry. 2017;174(3):216–229. doi: 10.1176/appi.ajp.2016.16050503. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Hui T.P., Kandola A., Shen L., Lewis G., Osborn D.P.J., Geddes J.R., Hayes J.F. A systematic review and meta-analysis of clinical predictors of lithium response in bipolar disorder. Acta Psychiatr. Scand. 2019;140(2):94–115. doi: 10.1111/acps.13062. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Iasevoli F., Giordano S., Balletta R., Latte G., Formato M.V., Prinzivalli E., De Berardis D., Tomasetti C., de Bartolomeis A. Treatment resistant schizophrenia is associated with the worst community functioning among severely-ill highly-disabling psychiatric conditions and is the most relevant predictor of poorer achievements in functional milestones. Prog. Neuro-Psychopharmacol. Biol. Psychiatry. 2016;65:34–48. doi: 10.1016/j.pnpbp.2015.08.010. [DOI] [PubMed] [Google Scholar]
  38. Iasevoli F., Avagliano C., Altavilla B., Barone A., Ciccarelli M., D'Ambrosio L., Notar Francesco D., Razzino E., Fornaro M., de Bartolomeis A. Evaluation of a few discrete clinical markers may predict categorization of actively symptomatic non-acute schizophrenia patients as treatment resistant or responders: a study by ROC curve analysis and multivariate analyses. Psychiatry Res. 2018;269:481–493. doi: 10.1016/j.psychres.2018.08.109. [DOI] [PubMed] [Google Scholar]
  39. Iasevoli F., Avagliano C., Altavilla B., Barone A., D'Ambrosio L., Matrone M., Notar Francesco D., Razzino E., de Bartolomeis A. Disease severity in treatment resistant schizophrenia patients is mainly affected by negative symptoms, which mediate the effects of cognitive dysfunctions and neurological soft signs. Front. Psychiatry. 2018;9(553) doi: 10.3389/fpsyt.2018.00553. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Iasevoli F., D'Ambrosio L., Notar Francesco D., Razzino E., Buonaguro E.F., Giordano S., Patterson T.L., de Bartolomeis A. Clinical evaluation of functional capacity in treatment resistant schizophrenia patients: comparison and differences with non-resistant schizophrenia patients. Schizophr. Res. 2018;202:217–225. doi: 10.1016/j.schres.2018.06.030. [DOI] [PubMed] [Google Scholar]
  41. Iasevoli F., Razzino E., Altavilla B., Avagliano C., Barone A., Ciccarelli M., D'Ambrosio L., Matrone M., Milandri F., Notar Francesco D., Fornaro M., de Bartolomeis A. Relationships between early age at onset of psychotic symptoms and treatment resistant schizophrenia. Early Interv. Psychiatry. 2022;16:352–362. doi: 10.1111/eip.13174. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Immonen J., Jääskeläinen E., Korpela H., Miettunen J. Age at onset and the outcomes of schizophrenia: a systematic review and meta-analysis. Early Interv. Psychiatry. 2017;11(6):453–460. doi: 10.1111/eip.12412. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Jin H., McCrone P., MacCabe J.H. Stratified medicine in schizophrenia: how accurate would a test of drug response need to be to achieve cost-effective improvements in quality of life? Eur. J. Health Econ. 2019;20(9):1425–1435. doi: 10.1007/s10198-019-01108-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Jongsma H.E., Gayer-Anderson C., Lasalvia A., Quattrone D., Mule A., Szoke A., Selten J.P., Turner C., Arango C., Tarricone I., Berardi D., Tortelli A., Llorca P.M., de Haan L., Bobes J., Bernardo M., Sanjuan J., Santos J.L., Arrojo M., Del-Ben C.M., Menezes P.R., Velthorst E., Murray R.M., Rutten B.P., Jones P.B., van Os J., Morgan C., Kirkbride J.B. Treated incidence of psychotic disorders in the multinational EU-GEI study. JAMA Psychiatry. 2018;75(1):36–46. doi: 10.1001/jamapsychiatry.2017.3554. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Joober R., Rouleau G.A., Lal S., Dixon M., O'Driscoll G., Palmour R., Annable L., Bloom D., Lalonde P., Labelle A., Benkelfat C. Neuropsychological impairments in neuroleptic-responder vs. -nonresponder schizophrenic patients and healthy volunteers. Schizophr. Res. 2002;53(3):229–238. doi: 10.1016/s0920-9964(01)00279-1. [DOI] [PubMed] [Google Scholar]
  46. Kapur S., Phillips A.G., Insel T.R. Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it? Mol. Psychiatry. 2012;17(12):1174–1179. doi: 10.1038/mp.2012.105. [DOI] [PubMed] [Google Scholar]
  47. Kennedy J.L., Altar C.A., Taylor D.L., Degtiar I., Hornberger J.C. The social and economic burden of treatment-resistant schizophrenia: a systematic literature review. Int. Clin. Psychopharmacol. 2014;29(2):63–76. doi: 10.1097/YIC.0b013e32836508e6. [DOI] [PubMed] [Google Scholar]
  48. Krapohl E., Patel H., Newhouse S., Curtis C.J., von Stumm S., Dale P.S., Zabaneh D., Breen G., O'Reilly P.F., Plomin R. Multi-polygenic score approach to trait prediction. Mol. Psychiatry. 2018;23(5):1368. doi: 10.1038/mp.2017.163. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Kuhn M., Johnson K. Springer; 2013. Applied Predictive Modeling. [Google Scholar]
  50. Lally J., Ajnakina O., Di Forti M., Trotta A., Demjaha A., Kolliakou A., Mondelli V., Reis Marques T., Pariante C., Dazzan P., Shergil S.S., Howes O.D., David A.S., MacCabe J.H., Gaughran F., Murray R.M. Two distinct patterns of treatment resistance: clinical predictors of treatment resistance in first-episode schizophrenia spectrum psychoses. Psychol. Med. 2016;46(15):3231–3240. doi: 10.1017/S0033291716002014. [DOI] [PubMed] [Google Scholar]
  51. Lee Y.H., Bang H., Kim D.J. How to establish clinical prediction models. Endocrinol. Metab. 2016;31(1):38–44. doi: 10.3803/EnM.2016.31.1.38. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Legge S.E., Dennison C.A., Pardiñas A.F., Rees E., Lynham A.J., Hopkins L., Bates L., Kirov G., Owen M.J., O'Donovan M.C., Walters J.T.R. Clinical indicators of treatment-resistant psychosis. Br. J. Psychiatry. 2020:1–8. doi: 10.1192/bjp.2019.120. [DOI] [PubMed] [Google Scholar]
  53. Levine S.Z., Rabinowitz J. Trajectories and antecedents of treatment response over time in early-episode psychosis. Schizophr. Bull. 2010;36(3):624–632. doi: 10.1093/schbul/sbn120. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Lu F., Petkova E. A comparative study of variable selection methods in the context of developing psychiatric screening instruments. Stat. Med. 2014;33(3):401–421. doi: 10.1002/sim.5937. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Malla A., Norman R., Schmitz N., Manchanda R., BÉChard-Evans L., Takhar J., Haricharan R.A.J. Predictors of rate and time to remission in first-episode psychosis: a two-year outcome study. Psychol. Med. 2006;36(5):649–658. doi: 10.1017/S0033291706007379. [DOI] [PubMed] [Google Scholar]
  56. Mørup M.F., Kymes S.M., Oudin Åström D. A modelling approach to estimate the prevalence of treatment-resistant schizophrenia in the United States. PLoS One. 2020;15(6) doi: 10.1371/journal.pone.0234121. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Murru A., Carpiniello B. Duration of untreated illness as a key to early intervention in schizophrenia: a review. Neurosci. Lett. 2018;669:59–67. doi: 10.1016/j.neulet.2016.10.003. [DOI] [PubMed] [Google Scholar]
  58. Nguyen C.D., Carlin J.B., Lee K.J. Model checking in multiple imputation: an overview and case study. Emerg. Themes Epidemiol. 2017;14:8. doi: 10.1186/s12982-017-0062-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. NICE . National Institute for Health and Clinical Excellence: Guidance National Collaborating Centre for Mental Health; London: 2014. Psychosis and Schizophrenia in Adults: Treatment and Management: Updated Edition 2014. [PubMed] [Google Scholar]
  60. Nordstroem A.-L., Talbot D., Bernasconi C., Berardo C.G., Lalonde J. Burden of illness of people with persistent symptoms of schizophrenia: a multinational cross-sectional study. Int. J. Soc. Psychiatry. 2017;63(2):139–150. doi: 10.1177/0020764016688040. [DOI] [PubMed] [Google Scholar]
  61. Pardiñas A.F., Smart S.E., Willcocks I.R., Holmans P.A., Dennison C.A., Lynham A.J., Legge S.E., Baune B.T., Bigdeli T.B., Cairns M.J., Corvin A., Fanous A.H., Frank J., Kelly B., McQuillin A., Melle I., Mortensen P.B., Mowry B.J., Pato C.N., Periyasamy S., Rietschel M., Rujescu D., Simonsen C., St Clair D., Tooney P., Wu J.Q., Andreassen O.A., Kowalec K., Sullivan P.F., Murray R.M., Owen M.J., MacCabe J.H., O’Donovan M.C., Walters J.T.R., Genetics Workstream of the Schizophrenia Treatment Resistance and Therapeutic Advances (STRATA) Consortium. Schizophrenia Working Group of the Psychiatric Genomics Consortium (PGC) Interaction testing and polygenic risk scoring to estimate the association of common genetic variants with treatment resistance in schizophrenia. JAMA Psychiatry. 2022;79(3):260–269. doi: 10.1001/jamapsychiatry.2021.3799. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Pavlou M., Ambler G., Seaman S., De Iorio M., Omar R.Z. Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. Stat. Med. 2016;35(7):1159–1177. doi: 10.1002/sim.6782. [DOI] [PMC free article] [PubMed] [Google Scholar]
  63. Perkins N.J., Schisterman E.F. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am. J. Epidemiol. 2006;163(7):670–675. doi: 10.1093/aje/kwj063. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Perlman K., Benrimoh D., Israel S., Rollins C., Brown E., Tunteng J.F., You R., You E., Tanguay-Sela M., Snook E., Miresco M., Berlim M.T. A systematic meta-review of predictors of antidepressant treatment outcome in major depressive disorder. J. Affect. Disord. 2019;243:503–515. doi: 10.1016/j.jad.2018.09.067. [DOI] [PubMed] [Google Scholar]
  65. Polese D., Fornaro M., Palermo M., De Luca V., de Bartolomeis A. Treatment-resistant to antipsychotics: a resistance to everything? Psychotherapy in treatment-resistant schizophrenia and nonaffective psychosis: a 25-year systematic review and exploratory meta-analysis. Front Psychiatry. 2019;10(210) doi: 10.3389/fpsyt.2019.00210. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Rajji T.K., Ismail Z., Mulsant B.H. Age at onset and cognition in schizophrenia: meta-analysis. Br. J. Psychiatry. 2009;195(4):286–293. doi: 10.1192/bjp.bp.108.060723. [DOI] [PubMed] [Google Scholar]
  67. Rubin D.B. John Wiley & Sons; New York, NY: 1987. Multiple Imputation for Nonresponse in Surveys. [Google Scholar]
  68. Salvador R., Radua J., Canales-Rodriguez E.J., Solanes A., Sarro S., Goikolea J.M., Valiente A., Monte G.C., Natividad M.D.C., Guerrero-Pedraza A., Moro N., Fernandez-Corcuera P., Amann B.L., Maristany T., Vieta E., McKenna P.J., Pomarol-Clotet E. Evaluation of machine learning algorithms and structural features for optimal MRI-based diagnostic prediction in psychosis. PLoS One. 2017;12(4) doi: 10.1371/journal.pone.0175683. [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Samara M.T., Leucht C., Leeflang M.M., Anghelescu I.-G., Chung Y.-C., Crespo-Facorro B., Elkis H., Hatta K., Giegling I., Kane J.M. Early improvement as a predictor of later response to antipsychotics in schizophrenia: a diagnostic test review. Am. J. Psychiatry. 2015;172(7):617–629. doi: 10.1176/appi.ajp.2015.14101329. [DOI] [PubMed] [Google Scholar]
  70. Seijo-Pardo B., Alonso-Betanzos A., Bennett K.P., Bolón-Canedo V., Josse J., Saeed M., Guyon I. Biases in feature selection with missing data. Neurocomputing. 2019;342:97–112. [Google Scholar]
  71. Shmueli G. To explain or to predict? Stat. Sci. 2010;25(3):289–310. [Google Scholar]
  72. Siskind D., Siskind V., Kisely S. Clozapine response rates among people with treatment-resistant schizophrenia: data from a systematic review and meta-analysis. Can. J. Psychiatr. 2017;62(11):772–777. doi: 10.1177/0706743717718167. [DOI] [PMC free article] [PubMed] [Google Scholar]
  73. Siskind D., Orr S., Sinha S., Yu O., Brijball B., Warren N., MacCabe J.H., Smart S.E., Kisely S. Rates of treatment-resistant schizophrenia from first-episode cohorts: systematic review and meta-analysis. Br. J. Psychiatry. 2022;220(3):115–120. doi: 10.1192/bjp.2021.61. [DOI] [PubMed] [Google Scholar]
  74. Smart S.E., Kępińska A.P., Murray R.M., MacCabe J.H. Predictors of treatment resistant schizophrenia: a systematic review of prospective observational studies. Psychol. Med. 2021;51(1):44–53. doi: 10.1017/S0033291719002083. [DOI] [PMC free article] [PubMed] [Google Scholar]
  75. Spaniel F., Tintera J., Rydlo J., Ibrahim I., Kasparek T., Horacek J., Zaytseva Y., Matejka M., Fialova M., Slovakova A., Mikolas P., Melicher T., Gornerova N., Hoschl C., Hajek T. Altered neural correlate of the self-agency experience in first-episode schizophrenia-Spectrum patients: an fMRI study. Schizophr. Bull. 2016;42(4):916–925. doi: 10.1093/schbul/sbv188. [DOI] [PMC free article] [PubMed] [Google Scholar]
  76. Stekhoven D.J., Buhlmann P. MissForest–non-parametric missing value imputation for mixed-type data. Bioinformatics. 2012;28(1):112–118. doi: 10.1093/bioinformatics/btr597. [DOI] [PubMed] [Google Scholar]
  77. Steyerberg E.W. 2nd ed. Springer Science & Business Media; 2008. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. [Google Scholar]
  78. Stokes I., Griffiths S.L., Jones R., Everard L., Jones P.B., Fowler D., Hodgekins J., Amos T., Freemantle N., Sharma V., Marshall M., Singh S.P., Birchwood M., Upthegrove R. Prevalence of treatment resistance and clozapine use in early intervention services. BJPsych Open. 2020;6(5) doi: 10.1192/bjo.2020.89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  79. Szymanski S., Lieberman J.A., Alvir J.M., Mayerhoff D., Loebel A., Geisler S., Chakos M., Koreen A., Jody D., Kane J., et al. Gender differences in onset of illness, treatment response, course, and biologic indexes in first-episode schizophrenic patients. Am. J. Psychiatry. 1995;152(5):698–703. doi: 10.1176/ajp.152.5.698. [DOI] [PubMed] [Google Scholar]
  80. Tarricone I., Mimmi S., Paparelli A., Rossi E., Mori E., Panigada S., Carchia G., Bandieri V., Michetti R., Minenna G., Boydell J., Morgan C., Berardi D. First-episode psychosis at the West Bologna community mental health Centre: results of an 8-year prospective study. Psychol. Med. 2012;42(11):2255–2264. doi: 10.1017/S0033291712000335. [DOI] [PubMed] [Google Scholar]
  81. Turkington A., Mulholland C.C., Rushe T.M., Anderson R., McCaul R., Barrett S.L., Barr R.S., Cooper S.J. Impact of persistent substance misuse on 1-year outcome in first-episode psychosis. Br. J. Psychiatry. 2018;195(3):242–248. doi: 10.1192/bjp.bp.108.057471. [DOI] [PubMed] [Google Scholar]
  82. Üçok A., Serbest S., Kandemir P.E. Remission after first-episode schizophrenia: results of a long-term follow-up. Psychiatry Res. 2011;189(1):33–37. doi: 10.1016/j.psychres.2010.11.013. [DOI] [PubMed] [Google Scholar]
  83. van Buuren S., Groothuis-Oudshoorn K. mice: Multivariate Imputation by Chained Equations in R. 45(3) 2011. p. 67. [Google Scholar]
  84. Vickers A.J., Kramer B.S., Baker S.G. Selecting patients for randomized trials: a systematic approach based on risk group. Trials. 2006;7(1):30. doi: 10.1186/1745-6215-7-30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  85. Warnez S., Alessi-Severini S. Clozapine: a review of clinical practice guidelines and prescribing trends. BMC Psychiatry. 2014;14(1):102. doi: 10.1186/1471-244X-14-102. [DOI] [PMC free article] [PubMed] [Google Scholar]
  86. Wimberley T., Stovring H., Sorensen H.J., Horsdal H.T., MacCabe J.H., Gasse C. Predictors of treatment resistance in patients with schizophrenia: a population-based cohort study. Lancet Psychiatry. 2016;3(4):358–366. doi: 10.1016/S2215-0366(15)00575-1. [DOI] [PubMed] [Google Scholar]
  87. Youden W.J. Index for rating diagnostic tests. Cancer. 1950;3(1):32–35. doi: 10.1002/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary material

mmc1.docx (18.1MB, docx)

Data Availability Statement

Data can be requested by contacting james.maccabe@kcl.ac.uk and completing a STRATA secondary analysis proposal form which will then be circulated to all consortium members for review.

RESOURCES