Development and Validation of Predictive Model for a Diagnosis of First Episode Psychosis Using the Multinational EU-GEI Case–control Study and Modern Statistical Learning Methods

Abstract

Background and Hypothesis

It is argued that availability of diagnostic models will facilitate a more rapid identification of individuals who are at a higher risk of first episode psychosis (FEP). Therefore, we developed, evaluated, and validated a diagnostic risk estimation model to classify individual with FEP and controls across six countries.

Study Design

We used data from a large multi-center study encompassing 2627 phenotypically well-defined participants (aged 18–64 years) recruited from six countries spanning 17 research sites, as part of the European Network of National Schizophrenia Networks Studying Gene-Environment Interactions study. To build the diagnostic model and identify which of important factors for estimating an individual risk of FEP, we applied a binary logistic model with regularization by the least absolute shrinkage and selection operator. The model was validated employing the internal-external cross-validation approach. The model performance was assessed with the area under the receiver operating characteristic curve (AUROC), calibration, sensitivity, and specificity.

Study Results

Having included preselected 22 predictor variables, the model was able to discriminate adults with FEP and controls with high accuracy across all six countries (ranges_AUROC = 0.84–0.86). Specificity (range = 73.9–78.0%) and sensitivity (range = 75.6–79.3%) were equally good, cumulatively indicating an excellent model accuracy; though, calibration slope for the diagnostic model showed a presence of some overfitting when applied specifically to participants from France, the UK, and The Netherlands.

Conclusions

The new FEP model achieved a good discrimination and good calibration across six countries with different ethnic contributions supporting its robustness and good generalizability.

Introduction

First episode psychosis (FEP), which affects approximately 3% of the adult population, is an umbrella term used to refer to schizophrenia spectrum disorders or related psychotic disorders.¹ Although schizophrenia was initially conceptualized as a chronic, progressive deteriorating condition,² accumulating evidence suggests that people with a diagnosis of schizophrenia can experience symptomatic improvements and regain a degree of social and occupational functioning,^3–5 especially when early intervention services intervene at the onset of the very first psychosis episode.^6,7 This ignited an increased focus on specialist early intervention services for FEP,^8,9 the aim of which is to reduce treatment delay, increase chance for recovery and improve overall prognosis of psychosis¹⁰; however, the detection of those individuals who are at risk for developing FEP is currently limited.¹¹ The reasons for the difficulties in detecting people who are at a greater risk for FEP are diverse including lack of financial recourses, high work-load and reliance on help-seeking behaviors. Indeed, most psychiatric services cannot offer a prompt assessment of the person at risk after the referral was made.¹²

This recognition ignited development of individualized diagnostic prediction modeling for disease diagnosis that considers individual variability in characteristics and lifestyle of each person.¹³ Thus, diagnostic models, which having been built on combined effects of thoroughly selected predictors, can be used to forecast the probability of a certain condition being present at the individual level.¹⁴ This is particularly potent considering that detection of people who are at risk for FEP using a validated diagnostic model does not rely on help-seeking and can be implemented at differences services. For example, it has been shown that while 45% of the people who were unlimitedly diagnosed with FEP were referred to early intervention services by the emergency medical services with another 17.9% of the FEP patients came in contact with mental health services via the criminal justice agencies.^15,16 These demonstrate that having a reliable tool to identify persons at risk of having FEP across these services will make the identification of people who are at risk for FEP much easier making the referrals to early interventions services more promptly. Therefore, it is hoped that availability of diagnostic models will facilitate a more rapid identification of individuals who are at a higher risk of FEP.¹⁷ This in turn would reduce time to treatment initiation, which is currently delayed for up to 3 years,¹⁸ subsequently minimizing the social and functional disability that results from prolonged untreated psychoses.^3–5

To-date, several studies, having employed either neuroimaging methods^19,20 or proteomic data²¹ to relatively small samples, aimed to develop a diagnostic model to classify individuals with schizophrenia compared to healthy controls. However, the implementation of the models built on such complex data is likely to be constrained by logistical and financial challenges. Currently, there is no study that has developed, evaluated, and validated a diagnostic model for FEP using data reflecting the real-life clinical information available to a physician and a patient at the time of the assessment. Imperatively, it remains unknown if a diagnostic model for classifying individual with FEP trained on data acquired at one site will perform similarly well on data acquired from a new site that was not included in the model development.¹⁹

Of course, this lack of progress in individualized diagnostic for FEP could be due to the complex etiology of FEP disorders, which may be intrinsically difficult to predict at an individual level. It is, nonetheless, equally feasible that the lack of progress can, at least in part, be attributed to significant methodological shortcomings that have engulfed the field of prediction modeling in psychiatry.¹⁴ These include relying on small samples, utilizing substantially lower numbers of cases relative to the number of considered predictor variables, employing unreliable methods to select predictor variables for inclusion into the model, not properly assessing the accuracy of the model, and not efficiently dealing with missing data.^22–24 In fact, a recent systematic review showed that all current diagnostic models in FEP were at high risk of bias,¹⁴ making it highly unlikely for these models to be of any use.

In the era of precision medicine, computationally demanding modern statistical learning algorithms, particularly regularized regression methods (RRMs),²⁵ promise to provide a useful tool for diagnostic modeling. Through an introduction of a penalty for overfitting, which occurs when the developed model provides an over-optimistic assessment of the predictive performance,²¹ RRMs produce a model with good interpretability,²⁶ which is especially portent for clinical application. Therefore, in the present study using a large multi-center phenotypically well-defined sample of FEP,²⁷ we employed RRMs to develop, evaluate, and validate a diagnostic risk estimation model to classify individual with FEP based on an individual profile of sociodemographic characteristics and environmental circumatnces.²⁸ The model was developed following the current guidelines.^29,30 To ensure our model is appropriate for routine use in clinical practice,³¹ we used the internal-external validation in multi-site settings highlighting the extent to which the developed model can be generalized to the data from plausibly related settings.³²

Methods

Study Design and Participants

Participants were recruited and assessed as part of the incidence and first episode case-control study, conducted as part of the EUropean network of national schizophrenia networks investigating Gene-Environment Interactions (EU-GEI) study,²⁷ which comprises the largest multi-site study of psychotic disorders ever conducted. EU-GEI study was established between May 2010 and April 2015 in tightly defined catchment areas in 17 sites across 6 countries, which were UK, The Netherlands, France, Spain, Italy and Brazil.³³ The research sites within each country were purposefully selected to include a mix of urban and rural areas.^27,33 All participants provided informed, written consent following full explanation of the study. It is noteworthy that the combined incidence and case-control methodology allowed us to account for any potential selection biases amongst the recruited and assessed cases.

Ethical Approval.

All participants who agreed to take part in the case–control study provided informed, written consent following full explanation of the study. Ethical approval for the study was provided by relevant local research ethics committees in each of the study sites.^27,33

Ascertainment of Cases.

The inclusion criteria for FEP cases were: (1) presentation with a clinical diagnosis for an untreated FEP as defined by International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10) criteria²⁷ (codes F20-F33) within the timeframe of the study; (2) aged between 18 and 64 years (inclusive); and (3) resident within one of the 17 defined catchment area at the time of their first presentation to psychiatric services for psychosis. Exclusion criteria were: (1) a previous contact with specialist mental health services for psychotic symptoms outside of the study period at each site; (2) evidence of psychotic symptoms precipitated by an organic cause (ICD-10: F09); (3) transient psychotic symptoms resulting from acute intoxication (F1x.5); (4) severe learning disabilities, defined by an IQ less than 50 or diagnosis of intellectual disability (F70–F79); and (5) insufficient fluency of the primary language at each site to complete assessments.²⁷

Ascertainment of Controls.

To better reflect the source population from which the cases arose, controls were recruited based on random and quota sampling that considered the distribution of age, sex, and ethnicity in each region. Inclusion criteria for controls were (1) age 18–64 years; (2) resident in the distinctly defined catchment region; (3) adequate fluency of the primary language used in each site; (4) no history of current or past psychiatric disorders.^27,33 The individuals who were recruited as controls for the study were broadly representative of local populations in relation to age, gender, and ethnicity.²⁷ Individuals who agreed to take part were screened for a history of psychosis. Those who reported previous or current treatment for psychosis were excluded.²⁷

Predictors.

Following a previous research protocol,³⁴ we excluded variables which had a high collinearity with other variables, and/or had > 50% missing values. Overall, 96 predictors related to participants’ sociodemographic circumstances, childhood adversity, life events experienced in adulthood and substance use were included in the model development (Supplementary table 1). Information on these predictors was collected using previously validated tools with a good inter-rater reliability and structured, standardized format across sites.^27,33,35

Statistical Analysis

The process of model development, evaluation and validation was carried out according to methodological standards outlined by Steyerberg et al²⁵; results were reported according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines³⁶; the completed checklist is provided in Supplementary table 2. All analyses were performed in RStudio release 4.02.³⁷

Sample Size Calculations.

To calculate the sample size in the present study, we utilized the guidelines for sample size calculations when developing prediction models for binary outcomes.³⁸ These guidelines require to account for not only on the number of events relative to the number of candidate predictor, which is a well-known rule of thumb for the required sample size,³⁹ but also on the total number of participants, the outcome proportion (incidence) in the study population, and the expected predictive performance of the model. These information is used to tailor sample size requirements to the specific setting of interest, with the aim of minimizing the potential for model overfitting while targeting precise estimates of key parameters.³⁸ Accordingly, to assess if our sample size was large enough to develop a robust prediction diagnostic model, we calculated the required sample size according to these guidelines³⁸ considering several parameters (Supplementary Material). Assuming the value of R² corresponds to an R²_Nagelkerke of 0.15, that is, R²_CS = 0.15 × max([R²_CS]),³⁸ the sample size required for the model development was n = 2816 corresponding to 14.67 events per predictor. Consequently, our sample size of n = 2627 was slightly below the requirement; though, this sample size calculation did not consider regularization, which reduces the risk of overfitting.

Imputation of Missing Values.

In the present study, some variables had missing values (Supplementary table 4) with an average missingness in the entire sample was approximately 12.9%. To avoid using unrepresentative sample of complete cases that may result in incorrect risk predictions,^40,41 we imputed missing values employing missForest, which is an imputation method based on random forest that handles continuous and categorical variables equally well and accommodates non-linear relation structures.^42,43 As recommended for prediction models, the outcome was included in the imputation process.⁴⁴ The distribution of the variables included in the analyses before and after the imputation are presented in Supplementary table 3 showing that the imputed values were very closely aligned with the observed values across all variables used in the analyses.

Variable Selection and Model Fitting.

To build the diagnostic model and identify which of included predictors were important for classifying individual with FEP and controls, we applied a binary logistic model with regularization by the least absolute shrinkage and selection operator (LASSO).⁴⁵ LASSO entails fitting a model, which, by imposing penalty (λ) on the size of regression parameter estimates shrink them towards zero,^46,47 simultaneously selects predictors, estimates their effects, and introduces parsimony. Therefore, if a suitable λ is chosen, LASSO intrinsically performs predictor selection and deals with collinearity. Selection of the optimal tuning parameter λ optimizing the model performance is described below.

Model Estimation.

The tuning parameter λ optimizing optimized the area under the receiver operating characteristic curve (AUROC) was chosen from a grid of 100 λ values through 10-fold repeated cross-validation (CV).⁴⁵ 10-fold CV divided data randomly into 10 non-overlapping data partitions; participants included in the first 9 partitions were considered as the training sample, and the remaining individuals as the test sample. To reduce the variance of CV, 10-fold CV was repeated 10 times computing the AUROC for each λ value. As a parsimonious model is desirable for practice⁴⁸ and may generalize better to different populations,⁴⁹ though often at the expense of a lower predictive performance, the model that corresponded to 3% tolerance of the maximum AUROC yielding more parsimony with fewer irrelevant variables compared to standard minimum lambda,⁴⁶ was chosen as the final model.

Model Performance.

Model’s accuracy was measured with discrimination and calibration. Discrimination indicates how well a model separates individuals who experienced an event from those who did not; we assessed discrimination using AUROC, where a value of 0.5 indicates that a model does not discriminate better than chance, while 1 indicates that a model discriminates perfectly.⁵⁰ Calibration, assessed via calibration slope β, which should ideally be 1, and the calibration-in-the-large α, which ideally should be zero, describes how well the predicted risk corresponds to the risk from the observed data^51,52 and can be described as a measure of bias in a model.⁵³ We present calibration graphically by placing the estimated and actual outcome risk on the horizontal and vertical axes, respectively. We further measured the prediction accuracy of our models with sensitivity and specificity. Unlike the traditional 50%, which follows often incorrect assumption that the false-positive and false-negative are equally important,²⁵ to classify an individual as high or low risk based on a prediction model, a cut-off for the predicted probability (ie, “decision threshold”)²⁵ was selected by maximizing the sum of the model’s sensitivity and specificity to minimize the false positives, which are unavoidable.⁵⁴ This entailed selecting the decision threshold that maximized the overall correct classification rates, while choosing the point on the receiver operating characteristic curve farthest from chance.⁵⁵

Model Validation.

We validated the developed model using internal-external cross-country validation, which allows quantifying the generalizability of a prediction model across different settings and populations.⁵⁶ Specifically, having six countries, we grouped participants by country and redeveloped the model, repeating every step of estimating and selecting candidate predictors, in five of the six countries. We then evaluated the resulting model using the data from the remaining country measuring discrimination and calibration as described above. We repeated this validation algorithm six times until each country was used as a validation sample and reported the mean as general performance estimates. The full model equation is presented in Supplementary materials to enable the essential independent external validation.⁵⁷

Results

Study Participants

The sample characteristics are provided in table 1. The sample comprised 2627 participants; of these, 43.0% (n = 1130) had a diagnosis of FEP and 57.9% (n = 1497) were participants without FEP disorders. The average age of participants with a diagnosis of FEP at the time of the assessment was 31.3 years (SD = 10.6), 61.7% (n = 697) were men and 63.3% (n = 715) were of white ethnicity. Compared to FEP group, participants without FEP were older (mean = 36.3 years, SD = 12.9) of whom 47.2% (n = 706) were men and 78.7% (n = 1178) were of white ethnicity. Of the entire sample, 22.2% (n = 582) were recruited from the UK, 15.5% (n = 406) were recruited from The Netherlands, 16.2% (n = 426) came from Spain, 9.6% (n = 252) came from France, 17.8% (n = 467) were recruited from Italy, and 18.8% (n = 494) came from Brazil. The participants recruited across all countries were comparable in terms of age, gender, and relationship status at the time of the assessment (table 2).

Table 1.

Characteristics of Participants in the Study Population

Characteristic	Overall n = 2627		Case status
			FEP n = 1130 (43.0%)		Control n = 1497 (57.9%)
	n/mean	%/SD	n/mean	%/SD	n/mean	%/SD
Age at assessment (years)	34.0	12.2	31.3	10.6	36.1	12.9
Gender
Male	1403	53.4	697	61.7	706	47.2
Female	1224	46.6	433	38.3	791	52.8
Education (years)	14.0	4.3	12.9	4.2	14.7	4.2
Ever been employed	2402	91.6	999	88.6	1403	93.8
Never been in a long-term relationship	507	19.3	345	30.5	162	10.8
Ethnicity
White	1893	72.1	715	63.3	1178	78.7
Black	380	14.5	235	20.8	145	9.7
Other	353	13.4	180	15.9	173	11.6
Country
The UK	582	22.2	246	21.8	336	22.4
The Netherlands	406	15.5	196	17.3	210	14.0
Spain	426	16.2	204	18.1	222	14.8
France	252	9.6	105	9.3	147	9.8
Italy	467	17.8	187	16.5	280	18.7
Brazil	494	18.8	192	17.0	302	20.2

Note: sd, standard deviation.

Table 2.

Characteristics of Participants in the Study Population, by Country

Recruitment countries	The UK n = 582 (22.2%)		The Netherlands n = 406 (15.5%)		Spain n = 426 (16.2%)		France n = 252 (9.6%)		Italy n = 467 (17.8%)		Brazil n = 494 (18.8%)
	n/mean	%/SD	n/mean	%/SD	n/mean	%/SD	n/mean	%/SD	n/mean	%/SD	n/mean	%/SD
Case status
FEP	246	42.3	196	48.3	196	48.3	105	41.7	187	40.0	192	38.9
Health control	336	57.7	210	51.7	210	51.7	147	58.3	280	60.0	302	61.1
Age at assessment (years)	33.5	11.9	34.4	13.3	34.9	11.5	35.9	13.5	33.1	11.4	33.3	12.1
Median (IQR)	31	17	31	22	34	18	33	22	31	19	30	14
Gender
Male	315	54.1	235	57.9	188	44.1	134	53.2	228	48.8	253	51.2
Female	267	45.9	171	42.1	238	55.9	118	46.8	239	51.2	241	48.8
Education (years)	15.3	3.4	16.3	3.8	13.4	4.5	13.3	3.7	14.0	3.9	11.2	4.3
Ever been employed	539	92.6	398	98.0	385	90.8	229	91.2	396	85.2	455	92.1
Never been in a long-term relationship	110	18.9	94	23.2	91	21.4	35	13.9	89	19.1	88	17.8
Ethnicity
White	348	59.8	292	72.1	382	89.7	143	56.7	434	92.9	294	59.5
Black	170	29.2	57	14.1	9	2.1	81	32.1	12	2.6	51	10.3
Other	64	11.0	56	13.8	35	8.2	28	11.1	21	4.5	149	30.2

Note: FEP, first-episode psychosis; N, number of participants; IQR, interquartile range; df, degrees of freedom.

Diagnostic Model

Internally-externally validated performance of our model is presented in table 3. The model included 22 (22.9% out of n = 96) predictor variables (Supplementary table 4). The model’s apparent performance is presented in Supplementary table 5. Following model’s validation, a very good discrimination was observed across all countries (range_AUROC = 0.84–0.86). The calibration intercept (α) for all, but The Netherlands (calibration intercept [α] = 0.56), countries was slightly larger than 0 (range = −0.24 to 0.13). Calibration slope (β) was the lowest for the sample obtained in Brazil (calibration slope [β] = 1.11) and Spain (calibration slope [β] = 1.22) indicating an excellent model predication accuracy; though, calibration slope (β) for France, the UK and The Netherlands were relatively high (1.77, 1.71, and 1.61, respectively) suggesting that the model for these sites slightly overestimate risks. Calibration plots showed good agreement between observed and expected risk at predicted probabilities across all countries (figure 1). Using a cut-off point of 39.8%, our model was able to discriminate adults FEP from controls with good sensitivity (range = 73.9–78.0%) and specificity (range = 75.6–79.3%).

Table 3.

Internally–Externally Validated the Model’s Performance

Internally-externally validated performance	Prediction model of FEP
	The UK	The Netherlands	Spain	France	Italy	Brazil
Sample size, n	582	406	426	252	467	494
Number of outcome events, n	246	196	204	105	187	192
Proportion of outcome events	0.42	0.48	0.48	0.42	0.40	0.39
AUROC	0.84 (95% CI = 0.82–0.85)	0.84 (95% CI = 0.82–0.85)	0.84 (95% CI = 0.83–0.86)	0.84 (95% CI = 0.82–0.85)	0.86 (95% CI = 0.84–0.87)	0.85 (95% CI = 0.83–0.87)
Calibration intercept (α)	0.23	0.56	0.14	0.13	−0.24	−0.19
Calibration slope (β)	1.71	1.61	1.22	1.77	0.85	1.11
Sensitivity	75.0%	73.9%	74.0%	74.8%	77.6%	78.0%
Specificity	75.6%	79.3%	78.1%	77.0%	76.7%	76.8%

Note: AUROC, area under the receiver operating characteristic curve; CI, confidence intervals.

Fig. 1.

Internally–externally validated calibration plots for the prediction model of FEP in the United Kingdom, the Netherlands, France, Spain, Italy, and Brazil. Squares illustrate risk groups by fourths of equally spaced model-estimated risks, through which the linear line was fitted (red). The smoothed loess curve (orange) was fitted based on individual data points. The 45° line (gray) represents perfect calibration where model-estimated equal actual risk. The histogram on the upper margin represents the distribution of model-estimated risks.

Predictor Variables.

Several predictor variables selected in the model (Supplementary table 4), such as educational attainment (unstandardized β = 0.035), being foreign-born (unstandardized β = 0.113), and childhood experiences including not having peers to go to (unstandardized β = 0.264), experiences of prolonged loneliness (unstandardized β = 0.152) and running away from home (unstandardized β = 0.052) existed before FEP onset. Thus, these variables may be seen as potentially causative predictors for developing FEP. Other important contributing factors for diagnostic risk for FEP were being unemployed (unstandardized β = 0.671), being single (unstandardized β = 0.577), having problems with the police (unstandardized β = 0.484) or having difficulties at work (unstandardized β = 0.528), using more cannabis than intended (unstandardized β = 0.523), daily cigarette smoking (unstandardized β = 0.590), and using other substances, such as cocaine (unstandardized β = 0.227). A worked example of calculating an individualized risk for FEP is provided in Supplementary Material.

Discussion

Having utilized data across 17 research sites from 6 countries to our knowledge, this is the first study to develop, fully evaluate and validate a diagnostic model for classifying FEP based on a personal profile of 22 personal characteristics and lifestyle. We followed the current guidelines for model development, evaluation and validiton,^29,30 our results indicate that classification of FEP and controls is possible with high predictive accuracy across the UK, The Netherlands, Spain France Italy and even Brazil. To maximize the predictive accuracy,⁵⁸ we catered for incomplete data, which is a common but serious limitation in psychiatric research but generally not addressed sufficiently.^40,41 Given that the data ascertainment for this study was carried out in major urban and rural sites with heterogeneous populations²⁷ suggests that the validity of our model may extend to other centers with similar population profiles. Sensitivity and specificity are tests of accuracy of a model and are among the fundamental measures to understanding the utility of clinical tests. Sensitivity refers to how good a test is at correctly identifying people who have the disease; whereas the specificity of a clinical test refers to the ability of the test to correctly identify those patients without the disease. Our results demonstrate that our model has a high sensitivity and high specificity implying that it will detect accurately many adults who are disease free as low risk without recurring further investigation. Because the model does not require any laboratory testing or clinical measurements, it could be easily integrated into electronic case-registers to facilitate the automatic and individualized diagnostic identification of FEP based on electronic or clinical records.

The predictor variables that were selected by our model, such as lower level of educational attainment, childhood adversity and stressful life events in adulthood,^16,59–61 and cannabis use^62–65 have previously been linked to FEP risk and probably occurred before the onset of psychoses. Accordingly, our results reiterate the important role these experiences play in increasing risk for FEP providing avenues for prevention strategies.⁶⁶ Importantly, some of these factors, such as cannabis use, childhood adversities and educational attainment are potentially preventable with the right interventions.⁶⁶ For example, using the same data as in the present study, it was shown that 24% of FEP cases would have been prevented if none in the population consumed cannabis of high potency.⁶³ Our results further reiterate that better educational attainment may protect from FEP risk perhaps via more effective coping strategies, healthier behaviors and social relationships.^67–69 For many, however, FEP develops during a period critical to the consolidation of life skills,⁷⁰ which may result in an individual being unable to obtain qualifications after illness onset.⁷¹ It is, therefore, imperative to provide people with FEP access to supported education programs to (re)-engage them in the workforce.⁷² The confirmation of these factors as pivotal in development of FEP further supports the long-term benefits of reducing an exposure to these risk factors for psychosis; though this likely to be challenging considering the pathogenic mechanism underlying the link between some of these risk factors and psychosis is not fully understood.¹⁵ Furthermore, it may be very difficult to diminish exposure to some risk factors, for example, child abuse or migration; though, an obvious place to start is by attempting to reduce society’s consumption of high-potency cannabis through public education.⁶²

Nonetheless, because all individuals with a diagnosis of FEP in the present study were already under the care of mental health services upon recruitment, it may be argued that there is a window of missed opportunity for detection of FEP before the illness onset using this model. An alternative approach is to develop a prognostic model that will aim to estimate an individual risk for FEP onset among young help-seeking people who have been identified as at clinical high risk, which is a state characterized by either “attenuated” psychotic symptoms, or full-blown psychotic symptoms that are brief and self-limiting. While these prognostic risk estimation studies offer a promise for detecting young people who are at high risk for converting to FEP from experiencing suboptimal symptoms, those young adults who have been classified as “at clinical high risk” are not representative of those who develop FEP in terms of socio-economic status, life-experiences and ethnical composition.^11,15,66 In contract, our model was developed specifically for true FEP cases, which ensures its generalizability to the wider FEP communities.

Calibration slope for our prediction diagnostic model, however, showed a presence of some overfitting when applied specifically to participants from France, the UK, and The Netherlands suggesting that for the patients recruited in these countries estimated risks may be too high for those who are at high risk and too low for those who are at low risk; though, the observed calibration slope estimates were still within a range of previously reported models.^30,73 It may be argued that calibration slope would have been better if more complex machine learning methods, such as support vector machines, had been used,²⁶ or by employing more complex predictors, such as neuroanatomical biomarkers.¹⁴ Nonetheless, there is no evidence to suggest that more complex models or encompassing biomarkers lead to significant improvements in prediction accuracy even at the expense of reduced interpretability compared with simpler statistical models.^14,74,75

The present study is not without limitations. Because we developed the model on a case-control sample, the true prevalence of FEP in the general population differs considerably. Although it may be argued that the percentage of missing values across variables might have affected the imputations and induced some bias in the estimates of the effects in the model, the proportion of missingness in the present study was comparable to many longitudinal datasets^3–5,60 and within the range for missForest to handle it efficiently.⁷⁶ As with many risk models, we only accounted for baseline variables, although for many time-varying factors, exposure status may change over time.⁷⁷ However, using baseline variables reflects the real-life clinical information available to a physician and a participant when they need to make decisions on the likely risk of developing FEP disorders. Even though the average age between participants without a diagnosis of FEP was older than FEP participants, the age of onset of FEP in out sample was consistent with many studies on FEP conducted across Europe, and other continents.^78–85 A higher proportion of our participants with FEP were from ethnic minorities when compared to health controls. However, psychiatric epidemiology has consistently demonstrated that the incidence rates of psychotic disorders are considerably elevated among those of Black ethnicity residing in the UK compared to the host population.^80,86–88 Therefore, it is expected that individuals with FEP will be different greatly in ethnicity compared to adults without FEP diagnosis. It may be argued that many diagnostic categories assigned to patients on first contact with mental health services may either be provisional or likely to change over the illness course.⁸⁹ Nevertheless, in the present study we focused on the baseline diagnosis to emulate the naturalistic setting for all patients with FEP when predicting their onset. There is further robust meta-analytical evidence for high prospective diagnostic stability in schizophrenia spectrum and affective spectrum psychoses in the due course of the illnesses.^90,91 Nonetheless, it may be feasible to assume that there may be different predictors for affective versus non-affective psychosis. Thus, further modeling approaches may be necessary to investigate this in the possibility future. Finally, in the present study we have developed the model for FEP rather than individual diagnoses, such bipolar disorder, depression with psychotic features, because many diagnostic categories assigned to patients on first contact with mental health services may either be provisional or likely to change over the illness course.⁸⁹

Conclusions

Having employed modern statistical learning algorithms, we developed, evaluated, and validated a diagnostic model for classifying FEP that achieved a good discrimination and calibration across six European countries and Brazil supporting its robustness and good generalizability across FEP programs in different countries. This study, therefore, bears important implications for the development of affordable and easy-to-administer standardized assessment batteries that can evaluate individuals’ risk for FEP in clinical settings across countries with similar characteristics of adults with FEP.

Supplementary Material

Supplementary data are available at Schizophrenia Bulletin Open online.

^§EU-GEI collaborators

Kathryn Hubbard¹, Stephanie Beards¹, Doriana Cristofalo², Mara Parellada³, Pedro Cuadrado⁴, José Juan Rodríguez Solano⁵, David Fraguas⁵, Álvaro Andreu-Bernabeu⁵, Angel Carracedo⁶, Enrique García Bernardo⁷, Laura Roldán³, Gonzalo López³, Silvia Amoretti⁸, Juan Nacher⁹, Paz Garcia-Portilla¹⁰, Javier Costas⁶, Estela Jiménez-López¹¹, Mario Matteis³, Marta Rapado Castro³, Emiliano González³, Covadonga Martínez³, Emilio Sánchez⁷, Manuel Durán-Cutilla⁷, Nathalie Franke¹², Fabian Termorshuizen^13,14, Daniella van Dam¹², Elsje van der Ven^13,14, Elles Messchaart¹⁴, Marion Leboyer^15,16,17,18, Franck Schürhoff^15,16,17,18, Stéphane Jamain^16,17,18, Grégoire Baudin^15,16, Aziz Ferchiou^15,16, Baptiste Pignon^15,16,18, Jean- Romain Richard^16,18, Thomas Charpeaud^18,19,21, Anne-Marie Tronche^18,19,21, Flora Frijda²², Daniele La Barbera^22,23, Giovanna Marrazzo²³, Lucia Sideli²², Crocettarachele Sartorio^22,23, Laura Ferraro²²,Fabio Seminerio²², Camila Marcelino Loureiro^24,25, Rosana Shuhama^24,25, Mirella Ruggeri², Antonio LaSalvia², Chiara Bonetto²

¹Department of Health Service and Population Research, Institute of Psychiatry, King’s College London, De Crespigny Park, Denmark Hill, London, SE5 8AF, United Kingdom; ²Section of Psychiatry, Department of Neuroscience, Biomedicine and Movement, University of Verona, Piazzale L.A. Scuro 10, 37134 Verona, Italy; ³Department of Child and Adolescent Psychiatry, Hospital General Universitario Gregorio Marañón, School of Medicine, Universidad Complutense, IiSGM (CIBERSAM), C/Doctor Esquerdo 46, 28007 Madrid, Spain; ⁴Villa de Vallecas Mental Health Department, Villa de Vallecas Mental Health Centre, Hospital Universitario Infanta Leonor/ Hospital Virgen de la Torre, C/San Claudio 154, 28038 Madrid, Spain; ⁵Puente de Vallecas Mental Health Department, Hospital Universitario Infanta Leonor/ Hospital Virgen de la Torre, Centro de Salud Mental Puente de Vallecas, C/Peña Gorbea 4, 28018 Madrid, Spain; ⁶Fundación Pública Galega de Medicina Xenómica, Hospital Clínico Universitario, Choupana s/n, 15782 Santiago de Compostela, Spain; ⁷Department of Psychiatry, Hospital General Universitario Gregorio Marañón, School of Medicine, Universidad Complutense, IiSGM (CIBERSAM), C/Doctor Esquerdo 46, 28007 Madrid, Spain; ⁸Department of Psychiatry, Hospital Clinic, IDIBAPS, Universidad de Barcelona, C/Villarroel 170, 08036 Barcelona, Spain; Department of Psychiatry, Hospital Universitari Vall d’Hebron; Group of Psychiatry, Mental Health and Addictions, Psychiatric Genetics Unit, Vall d’Hebron Research Institute (VHIR), Barcelona, Spain; Biomedical Network Research Centre on Mental Health (CIBERSAM), Barcelona, Spain; ⁹Neurobiology Unit, Program in Neurosciences and Interdisciplinary Research Structure for Biotechnology and Biomedicine (BIOTECMED), Universitat de València, Burjassot, Spain. Biomedical Research Networking Centre in Mental Health (CIBERSAM), Madrid, Spain. Biomedical Research Institute INCLIVA, Valencia, Spain; ¹⁰Department of Medicine, Psychiatry Area, School of Medicine, Universidad de Oviedo, Centro de Investigación Biomédica en Red de Salud Mental (CIBERSAM), C/Julián Clavería s/n, 33006 Oviedo, Spain; ¹¹Department of Psychiatry, Servicio de Psiquiatría Hospital “Virgen de la Luz,” C/Hermandad de Donantes de Sangre, 16002 Cuenca, Spain; ¹²Department of Psychiatry, Early Psychosis Section, Academic Medical Centre, University of Amsterdam, Meibergdreef 5, 1105 AZ Amsterdam, The Netherlands; ¹³Rivierduinen Centre for Mental Health, Leiden, Sandifortdreef 19, 2333 ZZ Leiden, The Netherlands; ¹⁴Department of Psychiatry and Neuropsychology, School for Mental Health and Neuroscience, South Limburg Mental Health Research and Teaching Network, Maastricht University Medical Centre, P.O. Box 616, 6200 MD Maastricht, The Netherlands; ¹⁵AP-HP, Groupe Hospitalier “Mondor,” Pôle de Psychiatrie, 51 Avenue de Maréchal de Lattre de Tassigny, 94010 Créteil, France; ¹⁶INSERM, U955, Equipe 15, 51 Avenue de Maréchal de Lattre de Tassigny, 94010 Créteil, France; ¹⁷Faculté de Médecine, Université Paris-Est, 51 Avenue de Maréchal de Lattre de Tassigny, 94010 Créteil, France; ¹⁸Fondation Fondamental, 40 Rue de Mesly, 94000 Créteil, France; ¹⁹CMP B CHU, BP 69, 63003 Clermont Ferrand, Cedex 1, France; ²⁰EPS Maison Blanche, Paris 75020 France; ²¹Université Clermont Auvergne, EA 7280, Clermont-Ferrand, 63000, France; ²²Department of Experimental Biomedicine and Clinical Neuroscience, Section of Psychiatry, University of Palermo, Via G. La Loggia n.1, 90129 Palermo, Italy; ²³Unit of Psychiatry, “P. Giaccone” General Hospital, Via G. La Loggia n.1, 90129 Palermo, Italy; ²⁴Departamento de Neurociências e Ciencias do Comportamento, Faculdade de Medicina de Ribeirão Preto, Universidade de São Paulo, Av. Bandeirantes, 3900 - Monte Alegre- CEP 14049-900, Ribeirão Preto, SP, Brasil; ²⁵Núcleo de Pesquina em Saúde Mental Populacional, Universidade de São Paulo, Avenida Doutor Arnaldo 455, CEP 01246-903, SP, Brasil

Funding

OA is funded by the National Institute for Health Research (NIHR) (NIHR Post-Doctoral Fellowship—PDF-2018-11-ST2-020). IF is funded by NIHR Predoctoral Fellowship (NIHR300493). MDF is funded by Clinician Scientist Medical Research Council fellowship (project reference MR/M008436/1). DQ is funded by Post-Doctoral Guarantors of Brain Clinical Fellowship. DS is funded part funded by the National Institute for Health Research (NIHR) Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London. JBK is supported by National Institute for Health Research, University College London Hospital, Biomedical Research Centre. The EU-GEI Project is funded by the European Community’s Seventh Framework Programme under grant agreement No. HEALTH-F2-2010-241909 (Project EU-GEI). The Brazilian study was funded by the São Paulo Research Foundation under grant number 2012/0417-0. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the National Institute for Health Research or the Department of Health and Social Care.

Conflict of Interest

R.M.M. has received honoraria from Janssen, Sunovian, Lundbeck and Otsuka. M.B. has been a consultant for, received grant/research support and honoraria from, and been on the speakers/advisory board of ABBiotics, Adamed, Angelini, Casen Recordati, Janssen-Cilag, Menarini, Rovi and Takeda. Other authors declare that they have no conflict of interest. All other authors declare no conflict of interest.