External Validation and Extension of the NAPLS-2 and SIPS-RC Personalized Risk Calculators in an Independent Clinical High-Risk Sample

Abstract

Early identification of individuals likely to develop psychosis is a priority for the field, resulting in the development of risk calculators that provide personalized estimates that an individual at clinical high-risk (CHR) will develop psychosis. The North American Prodrome Longitudinal Study (NAPLS) consortium and Shanghai At-Risk for Psychosis program have recently developed such calculators (NAPLS-2/SIPS-RC, respectively), but their discrimination performance has never been examined within the same sample. Moreover, validation studies of NAPLS-2 are limited in number and the SIPS-RC has not been cross-validated in a North American sample. The present research (N = 68) used the area under the receiver operating characteristic curve (AUC) to examine the accuracy of the NAPLS-2 and SIPS-RC calculators for discriminating CHR converters and non-converters, as well as extend their use by examining their ability to predict illness progression over a two-year period. For conversion, the NAPLS-2 and SIPS-RC risk calculators demonstrated moderate (AUC = .71) and fair (AUC = .65) discrimination performance, respectively. Both calculators provided moderate accuracy for discriminating illness progression over two-years (NAPLS-2 AUC = .71/ SIPS-RC AUC =. 76). We discuss implications for researchers and practitioners interested in using the NAPLS-2 and/or SIPS-RC and identify important steps for future research.

1. Introduction

Despite recent improvements in interventions for individuals with schizophrenia spectrum disorders, evidence indicates that recovery after illness onset remains low (1 in 7 patients; Jääskeläinen et al., 2012), making early identification and intervention for those most likely to develop psychosis a priority for the field (Fusar-Poli et al., 2017). However, current methods for identifying youth at high risk for developing a psychotic disorder employ dichotomous diagnostic criteria (i.e., high-risk/not-high risk) that have resulted in high false positive rates (65-80%; Cannon et al., 2016; Fusar-Poli et al., 2013), leading to recent efforts focusing on continuous and individualized risk prediction models (i.e., risk calculators). Notably, both the North American Prodrome Longitudinal Study (NAPLS) consortium and Shanghai At-Risk for Psychosis (SHARP) program at the Shanghai Mental Health Center have recently developed risk calculators that provide a personalized probability estimate that an individual at clinical high-risk (CHR) for psychosis (i.e., a person that meets criteria for a psychosis-risk syndrome) will transition to a psychotic disorder over a two-year period (NAPLS-2 and SIPS-RC, respectively). Although both risk calculators have been validated in well-powered and independent samples, to date, the performance of the NAPLS-2 and SIPS-RC calculators has not been examined within the same sample and the SHARP risk calculator has not been cross-validated in a North American sample. The present research examines the performance of both risk calculators for discriminating CHR converters from non-converters as well as investigates their potential use for discriminating between CHR youth with and without illness progression over a two-year period.

The CHR syndrome is characterized by subtle positive (e.g., loosely held delusions, brief hallucinations) and negative (e.g., blunted affect, social anhedonia) symptoms that are often accompanied by a decline in socio-occupational functioning (McGlashan et al., 2010). A proportion of those meeting criteria for a CHR syndrome will eventually convert to a psychotic disorder and, from a prospective perspective, this subgroup is currently hidden. These individuals are the only CHR syndrome group members that can be labeled as truly “prodromal” or in a “prodromal stage”, as a prodromal diagnosis can only truly be made retrospectively (i.e., after transition determination is completed) (Yung and McGorry, 1996). As noted, distilling this smaller and critical group from the larger pool of CHR syndrome individuals is a pressing priority for this area and much research has been conducted on identifying the key factors that distinguish this group and/or predict transition (see Addington and Heinssen, 2012; Fusar-Poli et al., 2013 for reviews). For example, disordered thought content (e.g., unusual thought content, suspiciousness, bizarre thoughts) (Addington et al., 2015; Ciarleglio et al., 2018; Crump et al., 2018; Perkins et al., 2015), social dysfunction (Cornblatt et al., 2011; Fusar-Poli et al., 2010; Jang et al., 2011), neurocognitive deficits (Pukrop et al., 2007; Schultze-Lutter et al., 2014), and negative symptom severity (Piskulic et al., 2012; Zhang et al., 2018b) are just a few of the clinical characteristics that differentiate CHR converters from non-converters. Similar to other areas of medicine (Bilimoria et al., 2013; D’agostino et al., 2008), this information is now being leveraged to develop personalized risk estimates for CHR individuals through the use of multivariate models to identify the optimal set of factors for predicting transition. These personalized risk estimates provide several benefits for early intervention. In particular, risk estimates for CHR individuals provide clinicians with previously unavailable personalized prognostic information for patients that aid in treatment planning and monitoring.

As previously indicated, consistent with these goals, both the NAPLS consortium and SHARP program at the Shanghai Mental Health Center have recently developed risk calculators that provide a personalized probability estimate for a CHR individual’s risk for transitioning to a psychotic disorder over a two-year period (Cannon et al., 2016; Zhang et al., 2018b). The NAPLS-2 and SIPS-RC calculators were designed specifically for use in individuals diagnosed as CHR based on the Structured Interview for Psychosis-Risk Syndromes criteria (SIPS; McGlashan et al., 2010), the gold standard for “high-risk” diagnoses in North America and much of Europe. The NAPLS-2 derives a single risk estimate by integrating demographic, clinical, and neurocognitive information obtained through interviews and neurocognitive testing. In contrast, the SIPS-RC was designed to only use information obtained from the SIPS interview, which provides a notable benefit in regards to both time and efficiency. Specifically, personalized probability estimates for the SIPS-RC are quantified using four dimensions derived from the following SIPS structures: functional deterioration within the past year, positive and negative symptom severity, and low levels of dysphoric mood. It was designed to provide a practical and individualized risk score for psychosis conversion that can be used by clinicians for treatment planning. Both the NAPLS-2 and SIPS-RC have been validated in independent samples with accuracies of 74% in the SIPS-RC sample (Zhang et al., 2018b) and 71% and 79% in the NAPLS-2 samples (Cannon et al., 2016; Carrión et al., 2016).

Although formal psychosis status is the principal endpoint of interest for research on predictors and mechanisms of transition to psychosis in CHR samples, conversion rates in CHR research are generally low, which typically precludes transition as a modeled outcome for studies at the majority of research centers within a reasonable time frame (Addington et al., 2015). Given this low conversion rate, examining progression provides a novel means for investigating potential illness course, as increased symptom severity may indicate greater risk for conversion past typical longitudinal study periods. Moreover, an increasing amount of evidence suggests that there is considerable heterogeneity in symptom progression in CHR individuals. For instance, anywhere from 30% to 67% of individuals diagnosed at CHR for psychosis remit from CHR status over a 2-year period, with varying rates dependent on remission criteria (Addington et al., 2011; Addington et al., 2015; Addington et al., 2018; Lee et al., 2014; Schlosser et al., 2011; Simon et al., 2013). Given that many of the clinical characteristics (e.g., unusual thought content, suspiciousness, functioning) identified as the optimal predictors of transition for the NAPLS-2 and SIPS-RC risk calculators also differentiate CHR individuals with and without illness progression (Addington et al., 2015; Addington et al., 2018; Tessner et al., 2009), both risk calculators may have utility for predicting increases in symptom severity, which would potentially provide a useful metric for predicting progression. Furthermore, predicting progression is important even in the absence of conversion, as individuals diagnosed as CHR continue to be at increased risk for a myriad of negative outcomes (e.g., Axis-1 diagnoses, social/role dysfunction) in addition to persistent attenuated psychosis symptoms (Hengartner et al., 2017; Michel et al., 2018), indicating they are an important clinical group regardless of transition status. Thus, the present research will also examine the efficacy of the NAPLS-2 and SIPS-RC for discriminating between individuals with and without illness progression.

The present study used an independent sample of individuals at CHR for psychosis to examine the discrimination performance of the NAPLS-2 and SIPS-RC risk calculators for distinguishing CHR converters from non-converters and individuals with and without illness progression over a two-year period. In line with previous research (Cannon et al., 2016; Carrión et al., 2016; Zhang et al., 2018b), we predicted that discrimination performance for distinguishing between converters and non-converters for both the NAPLS-2 and SIPS-RC risk calculators would be comparable to those of the original validation samples (i.e., .70 – .80). As noted earlier, because many of the variables used in the risk calculators are also sensitive to illness progression, we predicted that both risk calculators would have acceptable to moderate discrimination accuracy for classifying CHR individuals with and without illness progression over the one- and two-year follow-ups.

2. Methods

2.1. Participants

Data for the present study were obtained from 68 CHR youth (ages 13 – 21; M = 18.59, SD = 1.76; 41.2% female) during their participation in a longitudinal study of motor behavior and psychosis risk at the Adolescent Development and Preventative Treatment (ADAPT) program. Participants were recruited via newspaper ads, Craigslist, email postings, and community professional referrals. Exclusion criteria included any neurological disorder, history of head injury, life-time substance dependence, or any current or past psychotic disorder. In order to be included in the study, participants were required to meet the Criteria of Prodromal Syndromes (COPS; McGlashan et al., 2010) for a psychosis-risk syndrome (i.e., clinical high-risk) which included one or more of the following: (1) progression or recent onset of attenuated positive symptoms, (2) the presence of schizotypal personality disorder and/or a family history of a psychotic disorder accompanied by a recent decline in global functioning (McGlashan et al., 2010). Brief intermittent psychotic symptom syndrome was not used as an inclusion criterion in the current sample. Informed consent was obtained in accordance with the protocol approved by the Institutional Review Board. Assent was obtained for individuals younger than 18 with informed consent obtained from their legal guardian.

2.2. Clinical Outcomes

Of the 68 CHR participants assessed at baseline, 62 (91.2%) had at least one follow-up assessment over 2 years of follow-up. Seven of the 62 CHR participants (11.3%) with follow-up assessments transitioned to a psychotic disorder over the two-year follow-up period. For the purposes of the current study, illness progression was defined as an increase of 2 points on any of the positive symptom dimension scales of the SIPS. Positive symptomology was chosen to reflect illness progression in order to adhere to SIPS conceptualization of CHR status, and progression on any positive symptom dimension (rather than the total sum score across dimensions) was selected to avoid potential fluctuations in positive symptom dimensions either masking increases in symptom severity or creating spurious increases. Furthermore, progression status was limited to individuals that met COPS criteria for a psychosis-risk syndrome at follow-ups to ensure participants met COPS criteria for progression and not persistence (Woods et al., 2014). Eighteen (29.0%) of the 62 CHR participants with at least one follow-up assessment over the two-year period experienced illness progression. Of the 68 CHR participants assessed at baseline, 53 returned for a one-year follow-up session and 43 completed a two-year follow-up session. Thus, we experienced 22.0% and 36.7% attrition over the one-year and two-year follow-up periods, respectively. These attrition rates are comparable to other studies of CHR samples (Addington et al., 2015; Bernard et al., 2017; Cannon et al., 2016; Mittal et al., 2013; Mittal et al., 2008; Zhang et al., 2018b).

2.3. Clinical Interviews

The Structured Interview for Psychosis-Risk Syndromes (SIPS; McGlashan et al., 2010) was administered to determine if a psychosis-risk syndrome was present (as noted above) and to determine illness progression. The SIPS assesses several dimensions of attenuated positive, negative, disorganized, and general symptomology. Positive symptomology is comprised of dimensions reflecting unusual thought content, suspiciousness, perceptual abnormalities, grandiosity, and disorganized communication. Negative symptom dimensions include social anhedonia, avolition, ideational richness, emotional expressiveness, blunted affect, and occupational functioning. Disorganized symptom dimensions include odd behavior of appearance, bizarre thinking, trouble with focus and attention, and impairment in personal hygiene. General symptomology includes sleep disturbances, dysphoric mood, motor disturbances, and impaired tolerance of normal stress. All symptom dimensions are rated on 0 to 6 scales with positive symptom ratings ranging from absent (0) to psychotic (6) and all other symptom dimensions using an absent (0) to extreme (6) scale. The Structured Clinical Interview for Axis I DSM-IV Disorders (SCID; First et al., 1995) was administered in order to rule out an Axis I psychotic disorder at baseline and to confirm a SIPS conversion diagnosis (i.e., a score of 6 on the positive symptom scale) up to 2 years following the baseline period. Clinical interviews were administered by advanced doctoral students that were trained with didactic sessions, videos, and then live cases until a high level of inter-rater reliability was met (Kappa ≥ .80). High inter-rater reliability was then regularly maintained in group consensus meetings and regularly held training meetings.

2.4. Cognitive Assessment

Participants were administered the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) cognitive battery (Green and Nuechterlein, 2004). The MATRICS battery consists of 10 standardized cognitive measures that comprise seven domains including working memory, social cognition, verbal and visual learning, speed of processing, reasoning/problem solving, and attention. These scores were used as part of the NAPLS-2 risk calculator (see below). The MATRICS was administered by trained graduate students.

2.5. NAPLS-2 Calculator

The NAPLS-2 risk calculator (http://riskcalc.org/napls/) was developed by the North American Prodrome Longitudinal Study (NAPLS; Cannon et al., 2016). The online calculator provides a continuous, individualized risk score that indexes the probability of transitioning to formal psychosis and was designed specifically for individuals diagnosed as CHR based on the aforementioned SIPS criteria. The calculations are based on a multivariate proportional hazards regression model that includes several demographic, cognitive, and psychosocial variables including: age, Brief Assessment of Cognition in Schizophrenia (BACS): Symbol-Coding score (Keefe et al., 2004), the Hopkins Verbal Learning Test-Revised (HVLT-R) sum score (trials 1-3; (Brandt and Benedict, 2001), negative life event sum scores from the Research Interview Life Events Scale (Dohrenwend et al., 1978), difference score between the highest Global Functioning Scale: Social (GFS-S; Carrión et al., 2018; Cornblatt et al., 2007) score from 1 year prior to baseline and current score, severity of unusual thought content and suspiciousness (rescaled from SIPS scores), family history of psychosis, and the total number of life-time experienced traumas (assessed during SCID interviews). The current study obtained each of the measures required for the online calculator through the initial baseline clinical interviews and cognitive assessment. Specifically, all demographics and clinical characteristics (e.g., negative life events, GFS-S scores, etc.) were acquired through the initial baseline interview while the BACS and HVLT-R raw scores were obtained from the MATRICS battery. Discriminant validity of the NAPLS-2 risk calculator for classifying CHR individuals who did and did not transition to psychosis was demonstrated in a well powered study (Cannon et al., 2016) that was replicated in an independent sample (Carrión et al., 2016). NAPLS-2 risk scores are provided for both one- and two-year probabilities for developing psychosis and both were computed for the present study.

2.6. SIPS-RC Calculator

The SIPS-RC was developed by the Shanghai-At-Risk-for-Psychosis (SHARP) program (Zhang et al., 2018b) to be a simple, SIPS-based risk calculator for use in clinical settings. It was designed to provide a practical and individualized risk score for psychosis conversion that can be used by clinicians for treatment planning. Probability estimates for the SIPS-RC are quantified using four dimensions derived from the following SIPS structures: functional deterioration within the past year, positive and negative symptom severity, and low levels of dysphoric mood. Functional decline is measured using the Global Assessment of Functioning scale (Hall, 1995), which is included in the SIPS to assess for recent psychosocial occupational deterioration. Positive and negative symptom severity are both quantified using sum scores from three domains from the positive (unusual thought content, suspiciousness, and disorganized communication) and negative (social anhedonia, expression of emotion, and ideational richness) SIPS dimensions, respectively. Low levels of dysphoric mood are measured using the dysphoric mood dimension from the general symptoms SIPS domain. These four dimensions were empirically selected using logistic regression wherein they were entered into a model predicting conversion in a well-powered CHR sample (Zhang et al., 2018b). Each of these four SIPS-RC components are then converted into simple binary variables (Yes/No) that can be used to determine the individualized risk estimate from the flowchart of calculated risk for CHR individuals provided in Zhang and colleagues (2018b).

2.7. Statistical Approach

All analyses were conducted using SPSS 25.0. The area under the receiver operating characteristic curve (AUC) was used to examine the discrimination performance of the NAPLS-2 and SIPS-RC risk calculators for correctly classifying CHR converters from non-converters and CHR youth with and without illness progression. Relevant demographic, cognitive, psychosocial, and SIPS variables were manually entered into the risk calculator to obtain individualized risk scores for each participant. These risk scores were then submitted to AUC analyses. The AUC provides a single value (ranging from 0 to 1) that reflects the overall ability of a parameter (e.g., risk score) to distinguish between outcomes (i.e., conversion/illness progression) with an AUC of .5 reflecting no better than chance discrimination and an AUC of 1.0 indicating perfect discrimination (i.e., no false positives or negatives) (Hajian-Tilaki, 2013). Use of the AUC statistic to measure discrimination performance was chosen to maintain consistency and facilitate comparability with the original NAPLS-2 and SIPS-RC validation studies (Carrión et al., 2016; Zhang et al., 2018b). On the basis of previous work showing good discrimination performance for both the NAPLS-2 and SIPS-RC in CHR samples (Cannon et al., 2016; Carrión et al., 2016; Zhang et al., 2018b), and discrimination performance less than chance (AUC < .5) indicating an inverse relationship (i.e., risk scores predicting better outcomes), we used one-tailed tests to evaluate the ability of the two risk calculators for classifying converters from non-converters and those with and without symptom progression. All AUC analyses related to conversion were performed on the full sample (N = 68) that had at least one follow-up assessment over the two-year study period. For AUC analyses examining illness progression, analyses were conducted for both one-year follow-up (N = 53) and two-year follow-up (N = 43). We used similar terminology to the original validation articles to describe AUC effect sizes. Specifically, for the purposes of this article, an AUC of .80 or greater is considered good discrimination performance, .70 to .80 moderate, and .60 to .70 fair, and .60 and below considered poor. For comparison with conventional interpretations of Cohen’s d (Cohen, 1988), an AUC score of .80 reflects a Cohen’s d of 1.20 (a large effect size), an AUC of .70 being a d of .75 (a medium effect size), an AUC of .60 being a d of .38 (a small effect size), and an AUC of .50 being a d of 0 (Rice and Harris, 2005).

3. Results

Baseline demographics and clinical characteristics of the CHR sample are provided in Table 1.

Table 1.

Baseline demographics and clinical characteristics of the CHR sample

	Mean	SD
Demographics
Age	18.59	1.76
Gender
Male	40
Female	28
Education (yr.)	12.39	1.80
Parent education (yr.)	15.38	2.66
SIPS Symptoms
Positive symptoms	11.91	4.61
Negative	10.06	7.03
Disorganized	5.35	3.67
General	7.08	4.30
Risk Calculator Scores
NAPLS-2	14.58	9.68
SIPS-RC	5.29	3.12

Note: Symptoms are derived from the Structure Interview from Psychosis-Risk Syndromes (SIPS).

3.1. Conversion

The NAPLS-2 risk calculator demonstrated moderate discrimination for distinguishing between CHR converters and non-converters with an AUC of .71 (p = .038). The SIPS-RC calculator showed fair discrimination for distinguishing between CHR converters and non-converters with an AUC of .65 (p = .10). See Figure 1a for receiver operating curves (ROC) for both calculators along with their confidence intervals (CIs) and standard errors.

Figure 1.

Panel A: Receiver operator characteristics curves (ROC) for the NAPLS-2 and SIPS-RC risk calculators discriminating subsequent conversion. Panel B: ROC curves for the NAPLS-2 and SIPS-RC risk calculator discriminating CHR youth with and without illness progression over a one-year period. Panel C: ROC curves for the NAPLS-2 and SIPS-RC risk calculators discriminating CHR youth with and without illness progression over a two-year period. Note: AUC = area under the curve; SE = standard error; CI = confidence interval.

3.2. Illness Progression over One-year Follow-up

The NAPLS-2 risk calculator demonstrated generally poor discrimination performance for distinguishing between CHR youth with and without illness progression over a one-year period (AUC = .57, p = .20). The SIPS-RC calculator showed fair discrimination performance for distinguishing between CHR youth with and without illness progression over a one-year period (AUC of .64, p = .046). See Figure 1b for ROCs.

3.3. Illness Progression over Two-year Follow-up

The NAPLS-2 risk calculator demonstrated moderate discrimination performance for distinguishing between CHR youth with and without illness progression at two-year follow-up (AUC = .71, p = .033). The SIPS-RC calculator also showed moderate discrimination performance for distinguishing between CHR youth with and without illness progression over a two-year period (AUC of .76, p = .012). See Figure 1c for ROCs.

4. Discussion

Early identification of individuals most likely to develop psychosis is a priority for the field, but we are only now improving traditional dichotomous risk approaches through the development of continuous and individualized risk estimates. The purpose of this study was to examine and extend the utility of the NAPLS-2 and SIPS-RC risk calculators for predicting subsequent transition to psychosis and illness progression over a two-year period in an independent sample of individuals at CHR for psychosis. As predicted, the NAPLS-2 risk calculator demonstrated moderate discrimination performance for distinguishing CHR converters and non-converters (AUC = .71). Counter to our prediction, the SIPS-RC risk calculator demonstrated fair discrimination performance for predicting conversion in the current sample (AUC = .65). Regarding illness progression from baseline to one-year follow-up, the NAPLS-2 and SIPS-RC demonstrated poor and fair discrimination performance respectively (NAPLS-2 AUC = .57; SIPS-RC AUC = .64) when classifying CHR individuals with and without positive symptom progression. However, both calculators’ prediction of positive symptom progression over the full two-year study period (compared to one year) was substantially better (NAPLS-2 AUC = .71; SIPS-RC AUC = .76).

When examining discrimination performance of the NAPLS-2 risk calculator for classifying CHR converters and non-converters in the present sample, we observed similar accuracies to those reported in both the NAPLS-2 development (71%) and external validation (79%) samples (Cannon et al., 2016; Carrión et al., 2016). furthermore, a NAPLS-2 predicted risk of 18% provided the best balance between sensitivity and specificity within the current sample, which was near identical to the 20% predicted risk reported in the development and validation samples. Specifically, an 18% predicted risk resulted in sensitivity and specificity levels of 71% and 77%, respectively, which is consistent with both the development and external samples (development: 66.7% and 72.1%; validation: 58.3% and 72.6%). Taken together, the present findings replicate the diagnostic accuracy reported in the original samples, and further support the utility of the NAPLS-2 risk calculator for predicting conversion.

Counter to our prediction, the SIPS-RC risk calculator demonstrated fair discrimination performance for predicting conversion in the current sample (AUC = .65). This is in contrast to the moderate discrimination performance observed in the development sample, which reported an accuracy of .74 (Zhang et al., 2018b). Nonetheless, the simplicity and efficiency of SIPS-RC is appealing, and it will be important for future research to further cross-validate the current findings in more well-powered samples. Interestingly, when cross-validating the NAPLS-2 risk calculator in the Chinese SHARP sample, Zhang and colleagues (2018a) observed an AUC of .63 when classifying conversion and suggested lower accuracies compared to the validation samples may be the result of statistical shrinkage wherein regression models exhibit reduced fit in new samples. However, given comparable accuracies between the NAPLS-2 development and validation samples and the present independent sample, lower accuracy scores for the crossvalidations (NAPLS-2 in the Chinese SHARP sample and SIPS-RC in the present sample) may reflect differences in risk prediction cross-culturally. Consistent with this notion, it is possible that the best predictors of risk for conversion differ within Chinese and North American CHR samples and similarity between validation and development samples may affect accuracies observed in any given study.

The discrimination accuracy for distinguishing amongst CHR individuals with and without illness progression over a one-year period was poor for the NAPLS-2 risk calculator (AUC = .57), whereas the SIPS-RC risk calculator demonstrated fair classification accuracy (AUC = .64). Furthermore, the discrimination performance improved for both risk calculators when classifying illness progression over the full two-year study period, with both the NAPLS-2 (AUC = .71) and the SIPS-RC (AUC = .76) showing moderate discrimination. Thus, although both the SIPS-RC and NAPLS-2 risk calculators only demonstrated poor to fair accuracy when illness progression was the outcome of interest over a one-year period, both calculators performed moderately well over the full two-year study period. Examining progression provides a novel means for investigating potential illness course as increased symptom severity may indicate greater risk for conversion past typical longitudinal study periods. However, it is important to note that this interpretation should be considered tentative until replications with larger sample sizes are conducted.

There are limitations to the current study that warrant discussion. Due to the relatively small sample size in the current study, we were unable to statistically compare AUC values. Although qualitative comparisons of AUC scores are more common (see Carrión et al., 2016; Zhang et al., 2018a; Zhang et al., 2018b), quantitative techniques are available (see DeLong et al., 1988) but the current sample was not powered to statistically test for AUC differences between NAPLS-2 and SIPS-RC calculators. In line with this, although AUCs for the risk calculators observed in the current sample are consistent with previous research (Carrión et al., 2016; Zhang et al., 2018a; Zhang et al., 2018b), it is important to note that differences amongst the AUC scores of the two calculators may be due to low power and it will be important for future work in well-powered samples to replicate the current findings. Further, as other psychosis risk calculators are in development and continue to utilize different predictors in multivariate models (Ciarleglio et al., 2018), direct statistical comparisons in well-powered samples will be informative for both clinicians and researchers to make appropriate decisions regarding the cost-benefits of obtaining the required information for risk calculation. However, we also believe it is highly useful to test how well these calculators perform in smaller samples, as outside large consortiums, this is where most prodromal research takes place. Relatedly, the calculators are ultimately designed for application in an N of 1. Additionally, the cross-validation of the SIPS-RC was restricted to a North American sample. Thus, cross-cultural inferences are speculative and rely on comparisons to other studies. Lastly, it is important to note that both the NAPLS-2 and SIPS-RC calculators were only developed and validated for transition outcomes. Thus, findings related to symptom progression should be viewed as preliminary until studies with larger samples replicate the current findings.

Highlights.

• Replicated discrimination performance of NAPLS-2 risk calculator for high-risk syndromes in an independent sample

• Cross-validated the SIPS-RC risk calculator for high-risk syndromes in an independent North American sample

• Both risk calculators demonstrated longitudinal utility for predicting illness progression