A predictive model for conversion to psychosis in clinical high-risk patients

Abstract

Background.

The authors developed a practical and clinically useful model to predict the risk of psychosis that utilizes clinical characteristics empirically demonstrated to be strong predictors of conversion to psychosis in clinical high-risk (CHR) individuals. The model is based upon the Structured Interview for Psychosis Risk Syndromes (SIPS) and accompanying clinical interview, and yields scores indicating one’s risk of conversion.

Methods.

Baseline data, including demographic and clinical characteristics measured by the SIPS, were obtained on 199 CHR individuals seeking evaluation in the early detection and intervention for mental disorders program at the New York State Psychiatric Institute at Columbia University Medical Center. Each patient was followed for up to 2 years or until they developed a syndromal DSM-4 disorder. A LASSO logistic fitting procedure was used to construct a model for conversion specifically to a psychotic disorder.

Results.

At 2 years, 64 patients (32.2%) converted to a psychotic disorder. The top five variables with relatively large standardized effect sizes included SIPS subscales of visual perceptual abnormalities, dysphoric mood, unusual thought content, disorganized communication, and violent ideation. The concordance index (c-index) was 0.73, indicating a moderately strong ability to discriminate between converters and non-converters.

Conclusions.

The prediction model performed well in classifying converters and non-converters and revealed SIPS measures that are relatively strong predictors of conversion, comparable with the risk calculator published by NAPLS (c-index = 0.71), but requiring only a structured clinical interview. Future work will seek to externally validate the model and enhance its performance with the incorporation of relevant biomarkers.

Introduction

Given the often progressive nature of schizophrenia and related psychoses, as well as the abundant evidence supporting the benefits of early detection and intervention in improving outcomes and prognosis (Marshall et al., 2005; Perkins et al., 2005), researchers have sought to develop means of identifying individuals in the pre-psychotic or clinical high-risk (CHR) phase, characterized by attenuated positive symptoms, which typically antedate full-blown psychotic illness (Correll et al., 2010; Fusar-Poli et al., 2013). Initial studies found that approximately 30% of CHR individuals convert to a psychotic disorder, usually within 2 years, while more recent studies by the NAPLS group and others have reported declining rates of conversion (Cannon et al., 2016; Hartmann et al., 2016).

In North America, CHR is defined primarily by the Structured Interview for Psychosis-Risk Syndromes (SIPS), a semi-structured interview (McGlashan et al., 2001; Miller et al., 2002; Rosen et al., 2002) that probes for past and current attenuated v. threshold psychotic symptoms. The Attenuated Positive Symptom Psychosis-Risk Syndrome (APSS), delineated in the SIPS, served as the basis for Attenuated Psychosis Syndrome (APS) included in the appendix section of proposed diagnoses in the DSM-5, with the sole additional requirement that CHR individuals be help-seeking.

The major reasons that prevented APS from being approved as an official diagnostic category, were the high false-positive rates (Haroun et al., 2006; Corcoran et al., 2010; Fusar-Poli et al., 2015); the small sample sizes of studies (Woods et al., 2009; Van der Gaag et al., 2013); the low annual incidence of CHR cases (less than 1/10 000 persons) (Addington et al., 2007); and the variable and declining conversion rates (or increasing false positive rates) (Yung et al., 2007; Simon et al., 2014). Further, at the present time, there are few treatments available that are specific to the CHR phase beyond treating concomitant symptoms (e.g. anxiety, depression) (Fusar-Poli et al., 2013).

Despite these limitations, there is an emerging consensus about the clinical characteristics that predict conversion to psychosis among CHR individuals, including specific positive symptoms from the SIPS (e.g. P.1. Unusual Thought Content) as well as other clinical/demographic factors (Brucato et al., 2017; Fusar-Poli et al., 2013; Addington et al., 2015). Based upon these and other data, Cannon et al. developed a risk calculator that was published online and allows a clinician to obtain individualized 2-year conversion risks by way of an assessment battery that includes the SIPS, neurocognitive tests, and several clinical measures inclusive of stressful life events (Carrión et al., 2016). In addition, Fusar-Poli et al. published on a transdiagnostic risk calculator that utilizes basic demographic and diagnostic information to identify individuals who are at highest risk of becoming psychotic over 6 years (Fusar-Poli et al., 2017). These efforts seek to emulate, in psychiatry, a paradigm that was successfully implemented in other areas of medicine, such as cardiovascular disease (D’Agostino et al., 2008) and oncology (Nam et al., 2011).

In light of these efforts, we examined our data from previously published articles on a CHR sample to determine whether clinical characteristics readily obtained from only the SIPS and accompanying clinical interview, could predict conversion to psychosis. Importantly, we recently identified SIPS-based measures that are highly associated with conversion to psychosis but have not traditionally been scored on the SIPS by other groups - namely, specific perceptual abnormalities (i.e. auditory and visual perceptual abnormalities) (Lehembre-Shiah et al., 2017) and violent ideation and violent behavior (Brucato et al., 2018). Therefore, our goal was to examine whether our SIPS data, enhanced with SIPS-based measures of violent ideation, violent behavior, and auditory and visual perceptual abnormalities, could predict conversion to psychosis, and hypothesized that these measures would be particularly informative. To construct a predictive model based on these measures, we employed a modern machine-learning algorithm known as LASSO-logistic regression that simultaneously selects relevant predictors and estimates their joint effects on the outcome (i.e. conversion to psychosis). The model can be used to compute subject-specific risk estimates for conversion to psychosis in CHR individuals. We assess the relative importance of the predictor variables that are included in the model and assess the utility of our predictive model with respect to overall accuracy by using evaluation methods that reduce the risk of obtaining overly optimistic performance metrics.

Methods

Subjects

We have previously reported on our procedures for subject recruitment/enrollment as well as data collection (Brucato et al., 2017). In summary, we recruited help-seeking individuals age 13–30 participating at the early detection and intervention for mental disorders program at the Center of Prevention and Evaluation (COPE) at the New York State Psychiatric Institute (NYSPI) at Columbia University Medical Center, all meeting criteria for both the APSS syndrome defined in the SIPS and the APS defined in the DSM-5. Written informed consent was provided by those aged ≥18. Minors gave written assent, with written informed consent provided by a parent/legal guardian. All research procedures were approved by NYSPI’s Institutional Review Board.

Exclusion criteria included lack of proficiency in English; any history of full-blown psychotic illness; a DSM disorder better accounting for the clinical presentation; I.Q. < 70; medical conditions affecting the central nervous system; imminent risk of harm to self or others; unwillingness to participate in research; geographic distance; or active current substance use disorder.

Clinical assessments

The SIPS (McGlashan et al., 2001; Miller et al., 2002; Rosen et al., 2002) includes a semi-structured interview which probes for past and current signs and symptoms of attenuated v. threshold psychotic states and contains the Scale of Prodromal Symptoms which includes 19 items, divided into Positive, Negative, Disorganization, and General Symptom subsections. CHR criteria are met if one meets criteria for the APSS, the Genetic Risk and Deterioration Psychosis-Risk Syndrome, or the Brief Intermittent Psychotic Psychosis-Risk Syndrome, but, notably, all COPE patients have met at least APSS criteria. The SIPS also includes a checklist of DSM-IV symptoms of Schizotypal Personality Disorder (American Psychiatric Association, 2000), a survey of family history of mental illness (Andreasen et al., 1977), and a modified version of the DSM-IV Global Assessment of Functioning scale (GAF) (Hall, 1995).

At baseline, participants also completed either the Diagnostic Interview for Genetic Studies (Nurnberger et al., 1994) or Structured Clinical Interview for DSM-IV Axis-I Disorders, Patient Edition (SCID-I/P) (First et al., 2002). Those aged <16 completed the Kiddie Schedule for Affective Disorders and Schizophrenia, Present and Lifetime Version (K-SADS-PL) (Kaufman et al., 1996).

Participants were seen for follow-up with the SIPS every 3 months for 2 years when possible, or else whenever conversion was suspected. For some patients who would not come in for SIPS, we used clinical notes, family or physician report, and/or telephone notes to determine the outcome. Post-conversion diagnoses were established by COPE psychologists and/or psychiatrists. Administrators were certified in the measures and established scoring consensus.

The Global Functioning Scale: Social (GF: Social) and Global Functioning Scale: Role (GF: Role) were assessed at baseline and each follow-up visit (Cornblatt et al., 2007). Violent behavior and violent ideation (Brucato et al., 2018), suicidal ideation and behavior and history of trauma (Grivel et al.,2018), the SIPS P.4. subcomponents of auditory and visual perceptual abnormalities (Lehembre-Shiah et al., 2017), and the SIPS P.1. subcomponents (Crump et al., 2017) were all extracted from the baseline SIPS and accompanying the clinical interview, as previously described.

Statistical methods

All analyses were conducted in R version 3.3.2. Demographic and clinical characteristics of the entire analytic sample, as well as those for only converters and only non-converters, were summarized using means, standard deviations, and ranges for continuous measures, and frequencies and percentages for categorical measures. Baseline characteristics of converters and the non-converters were compared using two-sided two-sample t tests for continuous measures and Fisher’s exact test for categorical measures. The criteria used for statistical significance is a p value <0.05.

In constructing the predictive model for conversion, we sought to use all of the baseline clinical and demographic measures available. Some subjects were missing one or more of these baseline measures, though the amount of missing data was small. We used the k-nearest neighbors (kNN) algorithm, with k = 5 implemented using the VIM package in R, to ‘fill in’ the values for those with missing baseline measures prior to fitting the predictive model (Kowarik and Templ, 2016). We constructed the predictive model using binary logistic regression with a LASSO penalty (Tibshirani, 1996) where the outcome was a binary indicator of conversion status within the 2 years of follow-up and the predictors consisted of the baseline clinical and demographic characteristics. LASSO-logistic regression is ideal in this setting because it allows us to perform both estimations of the logistic model parameters and selection of relevant baseline variables simultaneously (i.e. it can screen out those baseline variables that are not predictive of conversion status by setting their corresponding parameter estimates to zero). Estimating the predictive model parameters using the LASSO penalty requires the selection of a tuning parameter, λ, which controls the number of model coefficients that are estimated to be non-zero. We selected an optimal λ from a range of plausible values using 5-fold cross-validation, selecting the λ value that maximized the average cross-validated area under the receiver operating characteristic (ROC) curve (AUC). The folds for the cross-validation procedure were selected via stratified random sampling to reflect the approximate 30% conversion rate in the entire sample. Selection of λ and fitting of the LASSO logistic model were carried out using the GLMNET package in R (Friedman et al., 2010). The final predictive model was internally validated using 1000 bootstrap resamples (also sampled to reflect the approximate 30% conversion rate) to compute an optimism-adjusted performance measure (Harrell et al., 1996) of AUC, or as it is also known, concordance index (c-index) (ranging from 0.5 for no ability to discriminate converters from non-converters to 1 for perfect ability to discriminate them).

Results

The analytic sample consisted of 199 subjects who had conversion status (converted to psychosis v. did not convert to psychosis) available in the 2-year follow-up period. In the entire sample, 64 (32.16%) converted to psychosis. Their mean age is 20.08 (S.D. = 3.82). Fifty-three (27.00%) are female (three subjects were missing gender information) and 106 (54.90%) are non-Caucasian (six subjects were missing information on race). Table 1 shows the descriptive statistics for all baseline demographic and clinical variables that are considered in the predictive model.

Table 1.

Characteristics of the full sample, converters, and non-converters

Variable Names	N	Range	Full mean (S.D.) or N (%)	Non-converter mean (S.D.) or N (%)	Converter mean (S.D.) or N (%)	ES*	p**
			(n = 199)	(n = 135)	(n = 64)
Age	198	13–29	20.08 (3.82)	20.11 (3.86)	20.02 (3.76)	−0.02	0.871
Positive symptoms
P.1. Unusual thought content	199	0–5	3.58 (1.03)	3.43 (1.01)	3.91 (0.99)	0.47	0.002
P.2. Paranoia	199	0–5	3.34 (1.24)	3.27 (1.22)	3.48 (1.27)	0.17	0.264
P.3. Grandiosity	199	0–5	2.09 (1.59)	2.04 (1.57)	2.20 (1.64)	0.10	0.493
P.4. Overall hallucinations	199	0–5	2.85 (1.43)	2.87 (1.40)	2.83 (1.51)	−0.03	0.860
P.5. Disorganization	199	0–5	2.70 (1.31)	2.50 (1.29)	3.11 (1.27)	0.47	0.002
Total positive symptoms	199	4–22	14.57 (4.00)	14.11 (3.99)	15.53 (3.87)	0.36	0.019
P.1. Sub-symptoms
P.1.PD Perplexity and delusional mood	183	0–5	2.70 (1.12)	2.64 (1.11)	2.84 (1.15)	0.18	0.266
P.1.FR First rank symptoms	183	0–5	1.73 (1.73)	1.70 (1.72)	1.81 (1.75)	0.06	0.694
P.1.OB Overvalued beliefs	183	0–4	2.14 (1.16)	2.11 (1.08)	2.19 (1.32)	0.07	0.659
P.1.SNG Somatic nihilistic/very guilty ideas	183	0–5	1.96 (1.29)	1.98 (1.27)	1.93 (1.36)	−0.04	0.823
P.1.NP Non-persecutory ideas of reference	181	0–5	1.27 (1.35)	1.29 (1.36)	1.21 (1.35)	−0.06	0.713
P.4. Sub-symptoms
P.4. Auditory hallucinations	184	0–5	2.48 (1.59)	2.54 (1.58)	2.36 (1.61)	−0.11	0.482
P.4. Visual hallucinations	184	0–4	1.89 (1.49)	2.12 (1.47)	1.40 (1.41)	−0.50	0.002
Negative symptoms
N.1. Social anhedonia	199	0–6	3.54 (1.57)	3.30 (1.54)	4.03 (1.53)	0.47	0.002
N.2. Avolition	199	0–6	3.38 (1.63)	3.28 (1.66)	3.59 (1.56)	0.19	0.208
N.3. Expression of emotion	199	0–6	2.09 (1.72)	1.87 (1.64)	2.55 (1.83)	0.40	0.010
N.4. Experience of emotions and self	199	0–6	2.49 (1.89)	2.40 (1.94)	2.67 (1.77)	0.14	0.343
N.5. Ideational richness	199	0–5	1.85 (1.42)	1.67 (1.30)	2.25 (1.56)	0.42	0.006
N.6. Occupational functioning	199	0–6	3.79 (1.70)	3.69 (1.73)	4.00 (1.61)	0.18	0.228
Total negative symptoms	199	0–31	17.14 (6.65)	16.21 (6.39)	19.09 (6.82)	0.44	0.004
Disorganization symptoms
D.1. Odd behavior	199	0–5	2.57 (1.36)	2.36 (1.34)	3.02 (1.32)	0.50	0.001
D.2. Bizarre thinking	199	0–5	2.53 (1.46)	2.44 (1.42)	2.72 (1.55)	0.19	0.205
D.3. Trouble with focus and attention	199	0–6	3.17 (1.20)	3.01 (1.16)	3.48 (1.23)	0.40	0.010
D.4. Impairments in hygiene	199	0–6	1.62 (1.63)	1.59 (1.65)	1.69 (1.59)	0.06	0.702
Total disorganization symptoms	199	1–18	9.88 (3.82)	9.40 (3.76)	10.91 (3.77)	0.40	0.009
General symptoms
G.1. Sleep disturbance	197	0–6	2.69 (1.74)	2.65 (1.75)	2.77 (1.73)	0.06	0.674
G.2. Dysphoric mood	199	0–6	3.23 (1.55)	3.40 (1.53)	2.86 (1.55)	−0.35	0.021
G.3. Motor disturbance	199	0–6	1.92 (1.58)	1.70 (1.47)	2.39 (1.72)	0.45	0.004
	199	0–6	3.91 (1.83)	3.91 (1.84)	3.91 (1.83)	0.00	0.986
G.4. Impaired tolerance to stress
Total general symptoms	197	0–20	11.74 (4.25)	11.65 (4.47)	11.92 (3.77)	0.06	0.672
SIPS Sum of all total scores	199	16–82	53.53 (13.75)	51.67 (14.14)	57.45 (12.07)	0.43	0.005
Global assessment of functioning	197	31–60	45.18 (6.84)	46.10 (7.08)	43.18 (5.86)	−0.44	0.005
Social functioning	158	1–10	5.35 (1.77)	5.35 (1.76)	5.37 (1.82)	0.01	0.945
Role functioning	158	1–10	5.16 (2.35)	5.12 (2.29)	5.28 (2.49)	0.07	0.697
Violent Ideation (yes)	193		56 (29.0)	28 (21.5)	28 (44.4)	2.91	0.002
Violent Behavior (yes)	193		12 (6.2)	4 (3.1)	8 (12.7)	4.58	0.023
Suicidal ideation (yes)	193		13 (6.7)	7 (5.4)	6 (9.5)	1.85	0.442
Suicidal behavior (yes)	193		5 (2.6)	2 (1.5)	3 (4.8)	3.20	0.402
History of nonsexual trauma (yes)	193		29 (15.0)	20 (15.4)	9 (14.3)	0.92	1.000
History of sexual trauma (yes)	193		21 (10.9)	12 (9.2)	9 (14.3)	1.64	0.417
Race (Non-Caucasian)	193		106 (54.9)	63 (48.5)	43 (68.3)	2.29	0.015
Schizotypal (yes)	193		111 (57.5)	70 (53.8)	41 (65.1)	1.60	0.185
Genetic risk and deterioration syn. (yes)	184		8 (4.3)	5 (4.0)	3 (5.0)	1.25	1.000
Family history of psychosis (yes)	165		61 (37.0)	41 (38.3)	20 (34.5)	0.85	0.750
Sex (female)	196		53 (27.0)	41 (30.6)	12 (19.4)	0.54	0.140

^*ES column provides Cohen’s d statistic value comparing Converters and non-Converters for continuous measures or odds ratios for binary measures.

^**P column provides p values from t tests for continuous measures or Fisher’s Exact test for binary measures.

Table 1 also shows descriptive measures and statistical tests for the converters and non-converters. At baseline, converters appear to score higher on average than non-converters on Total Positive Symptoms (Cohen’s d (d) = 0.36, p = 0.02) and the corresponding subscales of P.1. Unusual Thought Content (d = 0.47, p <0.01) and P.5. Disorganized Communication (d = 0.47, p <0.01); Total Negative Symptoms (d = 0.44, p <0.01) and the corresponding subscales of N.1. Social Anhedonia (d = 0.47, p <0.01), N.3. Expression of Emotion (d = 0.40, p = 0.01), and N.5. Ideational Richness (d = 0.42, p = 0.01); total Disorganized Symptoms (d = 0.40, p = 0.01) and the corresponding subscales of D.1. Odd Behavior (d = 0.50, p <0.01) and D.3. Trouble with Focus and Attention (d = 0.40, p = 0.01); G.3. Motor Disturbance (d = 0.45, p <0.01) and SIPS Total Scores (d = 0.43, p = 0.01). Converters appear to score lower on average than non-converters with respect to P.4. Visual Perceptual Abnormalities (d = −0.50, p < 0.01) and G.2. Dysphoric Mood (d = −0.35, p = 0.02), as well as the GAF (d = −0.44, p = 0.01). A higher proportion of converters described violent ideation at baseline (44.40% v. 21.50%, p < 0.01), endorsed violent behavior at baseline (12.70% v. 3.10%, p = 0.02), and were non-Caucasian (68.30% v. 48.5%, p = 0.02). Converters and nonconverters did not differ on any other measured characteristics.

Only 3.21% of the overall data were missing from the analytic sample. Table 1 shows the number of observations (N) of each variable that was available in the data set. Social and role functioning scores had the highest amount of missing data, with each having 20.60% of the values missing. Family history was missing for 17.10% of the sample while all other variables had less than 10% missing values. Missing values were imputed using the kNN algorithm with k = 5. This allowed us to use all 199 observations to construct and validate the predictive model.

All of the clinical and demographic variables shown in Table 1 were considered as potential predictors of conversion status, except the Total Sum Scores (i.e. Total Positive, Negative, and Disorganized Symptoms, as well as the SIPS Sum of Total Scores) as they were linear combinations of other measures. We eliminated the sum scores to allow for better ability to identify sub-symptom scores that are predictive of conversion status.

The LASSO logistic prediction model identified 17 of the 40 baseline variables as having predictive value based on cross-validation. Table 2 lists these predictors in order of the magnitude of their standardized coefficient estimates (log odds ratios). The last column of Table 2 shows the proportion of times that the corresponding variable was selected to remain in the model across all 1000 bootstrap resamples. A high value provides evidence that the corresponding variable is predictive of the outcome. The P.4. subsymptom score for P.4. Visual Perceptual Abnormalities had the largest predictive value, reflected by the magnitude of the standardized coefficient and was also selected to remain in the largest proportion of LASSO logistic models (99%) across all 1000 bootstrap resamples. Other predictors with relatively large magnitude standardized coefficient estimates that were also selected in 90% or more of the LASSO logistic models across all 1000 bootstrap resamples were G.2. Dysphoric Mood, P.1. Unusual Thought Content, P.5. Disorganized Communication, Violent Ideation, race, N.1. Social Anhedonia, and Violent Behavior.

Table 2.

Estimated unstandardized and standardized model parameters (ORs) from the LASSO logistic model

Predictor	Coefficient estimatea (OR)	Standardized coefficient estimate (OR)	Proportion selectedb
P.4. Visual perceptual abnormalities	−0.30 (0.74)	−0.43 (0.65)	0.99
G.2. Dysphoric mood	−0.18 (0.84)	−0.29 (0.74)	0.95
P.1. Unusual thought content	0.28 (1.32)	0.28 (1.32)	0.90
P.5. Disorganization	0.20 (1.22)	0.26 (1.30)	0.90
Violent Ideation (yes)	0.57 (1.76)	0.26 (1.30)	0.95
Race (Non-Caucasian)	0.49 (1.63)	0.24 (1.27)	0.92
N.1. Social anhedonia	0.14 (1.15)	0.23 (1.26)	0.90
Violent Behavior (yes)	0.82 (2.27)	0.20 (1.22)	0.90
Global assessment of functioning	−0.03 (0.97)	−0.18 (0.83)	0.86
P.1.PD Perplexity and delusional mood	0.08 (1.08)	0.09 (1.09)	0.73
G.3. Motor disturbance	0.05 (1.05)	0.08 (1.08)	0.73
Suicidal ideation (yes)	0.17 (1.19)	0.04 (1.04)	0.67
History of sexual trauma (yes)	0.11 (1.12)	0.03 (1.03)	0.74
N.5. Ideational richness	0.02 (1.02)	0.03 (1.03)	0.66
D.3. Trouble with focus and attention	0.02 (1.02)	0.03 (1.03)	0.69
Social functioning	0.02 (1.02)	0.03 (1.03)	0.66
Suicidal behavior (yes)	0.09 (1.09)	0.02 (1.02)	0.57

Coefficients are ordered by magnitude of the standardized coefficient estimate.

^aThe value for the intercept is −1.65.

^bProportion of times that each variable was selected to remain in the model across all 1000 bootstrap resamples.

Figure 1 displays the frequency distributions of the model-based predicted risks for converters and non-converters in the sample. There tends to be a higher proportion of converters than non-converters in each risk class corresponding to risk scores above 44%, with the exception of the 54–59% risk class. Table 3 shows the in-sample statistics for prediction of conversion to psychosis for increasing thresholds of predicted risk. The in-sample values in the table are computed using predicted risk scores for the full sample derived from the fitted model and are therefore overly optimistic as they are essentially using the same data twice, but are included for completeness and for comparison with the similar published values from the North American Prodrome Longitudinal Study (NAPLS) (Cannon et al., 2016). The bootstrap validation procedure produced an optimism-adjusted AUC of 0.73 for our model, in comparison with a similarly adjusted AUC of 0.71 provided by NAPLS.

Fig. 1.

Frequency distributions of the model-based predicted risks for the converters and non-converters.

Table 3.

In-sample prediction statistics for conversion to psychosis for different levels of model-based risk scores

Risk class predicting conversion	Proportion in risk class	Positive predictive value	Negative predicative value	Sensitivity	Specificity
0.05–1.00	98.49	32.49	100	100	2.22
0.10–1.00	92.96	34.43	100	100	10.37
0.15–1.00	80.90	37.71	92.16	95.31	25.93
0.20–1.00	68.34	44.67	95.27	95.31	44.44
0.25–1.00	58.29	49.82	92.82	90.62	57.04
0.30–1.00	47.24	55.14	88.65	81.25	68.89
0.35–1.00	38.69	62.16	86.97	75.00	78.52
0.40–1.00	32.66	65.99	84.43	67.19	83.70
0.45–1.00	26.63	71.55	82.30	59.38	88.89
0.50–1.00	18.09	72.07	76.82	40.62	92.59
0.55–1.00	12.06	83.23	75.00	31.25	97.04
0.60–1.00	9.55	94.70	74.58	28.12	99.26
0.65–1.00	5.03	100	71.58	15.62	100
0.70–1.00	3.02	100	70.10	9.38	100
0.75–1.00	2.01	100	69.39	6.25	100
0.80–1.00	1.51	100	69.04	4.69	100

Figure 2 provides ROC curves for the present model (in-sample) and from published (in-sample) values from the NAPLS study [Table 3 in the original publication (Cannon et al., 2016)]. Figure 2 shows our prediction model to have superior sensitivity and specificity at every level of risk. These in-sample ROC curves are overly optimistic though, as they do not take into account that the data have been used to both fit the models and evaluate their performance. The in-sample AUC of our model is 0.84 in comparison with 0.74 for the NAPLS model (obtained by calculation from tabulated numbers in the original publication). Additionally, we calculated an in-sample Brier score of 0.15 for our model.

Fig. 2.

ROC curves for New LASSO-logistic model (in-sample) and NAPLS model (in-sample).

Discussion

The goal of this study was to construct a practical and clinically useful predictive model for conversion to psychosis among individuals at CHR for schizophrenia and related psychotic disorders as per the SIPS research measure and the APS diagnosis proposed in the DSM-5 utilizing only clinical and demographic characteristics as obtained using the SIPS and accompanying baseline clinical interview. This technique was intended to facilitate assessment of risk for psychotic illness in clinical settings, where the use of the various batteries requiring specialized expertise, such as required by the NAPLS calculator, might be impractical. We applied LASSO-logistic regression with 5-fold cross-validation to data from 199 CHR patients, (135 non-converting and 64 converting to psychosis within the 2-year follow-up period) to construct our predictive model and then performed a bootstrap validation procedure to assess model performance.

The estimated predictive model performed well, with an optimism-adjusted AUC of 0.73. Our predictive model bears some similarities with other published predictive models for conversion to psychosis, namely the recently published NAPLS model (Cannon et al., 2016). The NAPLS model includes unusual thought content (SIPS P.1.) and Suspiciousness (SIPS P.2.), decline in social functioning, lower verbal learning and memory performance, slower speed of processing, age, stressful life events, trauma, and family history of psychosis as predictors of conversion, with the combined factor of P1 + P2 being the strongest predictor. By comparison with respect to the ROC curves, the in-sample performance of our proposed predictive model is superior to the in-sample performance of the NAPLS model. Our optimism-adjusted AUC of 0.73 was also comparable with that from the NAPLS model, which had a bootstrap-validated AUC of 0.71. Similar to the NAPLS model and to other literature, SIPS symptoms P.1. and P.5. have been shown to be reliable predictors of conversion to psychosis (Addington et al., 2015). A finding unique to our model is that the subcomponent of P.4. Visual Perceptual Abnormalities had the largest magnitude standardized coefficient indicating non-risk for conversion, and that Violent Behavior and Violent Ideation were also strong predictors of conversion. Importantly, these are variables that are already collected by the SIPS and accompanying clinical interview but have not been assigned their own scores as they are subcomponents of other scores and therefore had not, in this form, been examined for their predictive abilities until recently by our group (Brucato et al., 2018; Lehembre-Shiah et al., 2017). It is possible that these variables conferred the robust predictive ability of our model and are supported by previous studies which examine similar variables and methodologies (Perkins et al., 2015; Marshall et al., 2016).

We could not directly apply the NAPLS predictive model to our sample because our patients were not assessed with the Hopkins Verbal Learning Test nor the BACS symbol coding as part of standard admission, nor do we routinely perform the specific life events and trauma scales performed by NAPLS. We believe that our predictive model has potential as an aid for clinicians to assess prognosis in CHR patients and to assist in guiding clinicians in the selection of treatment. However, before these clinical applications can be realized, the proposed model needs to be applied to and validated in a large independent set of CHR individuals, by our group as well as in other independent cohorts. The potential effectiveness of this model could also be enhanced by incorporating specific, biological measures of conversion that are readily available, inexpensive, and highly replicated.

It is also possible that the reasons for our findings and differences with other predictor models like NAPLS are due to the patient samples as reflected by the conversion rates. The NAPLS calculator was based on data derived from a patient sample with a substantially lower conversion rate, indicating possibly greater heterogeneity in the disturbances for which they sought treatment or who had milder forms or were at an earlier stage of their onset of illness. Alternatively, it could be that our sample was more atypical than other CHR cohorts in terms of its high conversion rate, in comparison with many recently published CHR samples, particularly those in recent clinical trials (McGorry et al., 2017). For example, NAPLS-2 (Addington et al., 2015) noted a conversion rate of 15.2%, or 25.3% of 367 participants who either converted or completed 2-year follow-up.

There are also differences between our prediction model and the risk calculator of Fusar-Poli et al. (2017), which disallow direct comparisons. While the study of Fusar-Poli et al., utilized individuals with non-organic, non-psychotic disorders, we utilized a purely CHR sample. In addition, the study of Fusar-Poli et al. calculated 6-year conversion rates, as opposed to two for our model. Given the robust accuracy of the Fusar-Poli et al. calculator as well as the NAPLS calculator, we would suggest that these calculators/prediction models serve to complement each other and together provide a more complete assessment/study of the variables that predict conversion to psychosis in different populations.

One criticism of using the LASSO, as with many other regression fitting procedures, is that if the predictors are correlated, then their individual effects on the outcome may be poorly estimated. However, the predictions derived from the estimated model need not be poor. Simulation studies have shown that LASSO-logistic regression models can perform well with respect to prediction of the outcome, even when the predictors are highly correlated (Lu and Petkova, 2014). Since the primary goal of this study is to construct a prediction model rather than to derive estimates of individual predictor effects, we believe that it is appropriate to apply the LASSO logistic model in the present context.

There are several limitations of this study. First, we noted that some of the subjects were missing information on one or more of the baseline predictors. Though the amount of missing data is very small, there is a possibility that the estimated prediction model, based on the combined complete and imputed data, is different from what the estimated prediction model would have been, had we had complete data on all subjects. We conducted several sensitivity analyses to assess the stability of the estimated model under various patterns for the missing covariates and found the estimated model was robust to the substitution of a range of realistic values for the missing data. Hence, we are not particularly concerned with the effects of missing data in this setting. Second, we used the COPE data to perform both model selection and estimation. Though we used LASSO-logistic regression with 5-fold cross-validation to avoid overfitting and a bootstrap procedure to compute optimism-adjusted measures of model performance so as not to suggest that the model performs better than it actually does, the real test of our model’s performance will entail conducting a large study to assess how well the model classifies new CHR patients as converters or non-converters within 2 years of follow-up. Generalizability of the model would be supported if the CHR patients in the aforementioned study come from a diverse collection of subjects who represent the population of all CHR patients, not just those with characteristics represented in the present sample of 199 patients from NYSPI. Third, using LASSO-logistic regression to estimate the prediction model does not allow for a precision estimate of a single predicted risk score for a new CHR individual to whom we wish to apply our prediction model. Though, as pointed out by Cannon et al. (2016), the individual predicted risk scores generated from the predictive model are more meaningful to the individual patient whereas the confidence and prediction intervals may not be of specific interest since they relate to likelihoods under future sampling.

In summary, this study demonstrates a practical and potentially clinically useful model that uses baseline data obtained exclusively from the SIPS and accompanying clinical interview to predict conversion to psychosis within 2 years of follow-up for CHR patients. The primary advantage of the proposed predictive model is that all of the information can be obtained by a single, 1 −2 h interview, and does not require other lengthy assessments (including cognitive assessments) requiring special expertise (Cannon et al., 2016). We believe that our predictive model has potential as an aid for clinicians to assess prognosis in CHR patients and to assist in guiding clinicians in the selection of treatment. Our goal is to make our model available online in the form of a ‘risk calculator.’ However, before these clinical applications can be realized, the proposed model needs to be applied to and validated in a large independent set of CHR individuals, by our group as well as in other independent cohorts. Future work should focus on replicating this model in independent samples as well as determining if its effectiveness/accuracy can be enhanced by incorporating biological measures that are predictive of conversion to psychosis.