At risk or not at risk? A meta-analysis of the prognostic accuracy of psychometric interviews for psychosis prediction

Abstract

An accurate detection of individuals at clinical high risk (CHR) for psychosis is a prerequisite for effective preventive interventions. Several psychometric interviews are available, but their prognostic accuracy is unknown. We conducted a prognostic accuracy meta-analysis of psychometric interviews used to examine referrals to high risk services. The index test was an established CHR psychometric instrument used to identify subjects with and without CHR (CHR+ and CHR−). The reference index was psychosis onset over time in both CHR+ and CHR− subjects. Data were analyzed with MIDAS (STATA13). Area under the curve (AUC), summary receiver operating characteristic curves, quality assessment, likelihood ratios, Fagan’s nomogram and probability modified plots were computed. Eleven independent studies were included, with a total of 2,519 help-seeking, predominately adult subjects (CHR+: N=1,359; CHR−: N=1,160) referred to high risk services. The mean follow-up duration was 38 months. The AUC was excellent (0.90; 95% CI: 0.87-0.93), and comparable to other tests in preventive medicine, suggesting clinical utility in subjects referred to high risk services. Meta-regression analyses revealed an effect for exposure to antipsychotics and no effects for type of instrument, age, gender, follow-up time, sample size, quality assessment, proportion of CHR+ subjects in the total sample. Fagan’s nomogram indicated a low positive predictive value (5.74%) in the general non-help-seeking population. Albeit the clear need to further improve prediction of psychosis, these findings support the use of psychometric prognostic interviews for CHR as clinical tools for an indicated prevention in subjects seeking help at high risk services worldwide.

Treatments for psychosis have been in wide use for nearly half a century, yet there is little evidence that they have substantially improved outcomes (1). Therefore, indicated preventive intervention is the main paradigm yielding new hope for impacting the course of psychosis (2). However, this intervention requires an accurate identification of individuals at clinical high risk (CHR), that relies on the use of accurate prognostic tools to detect psychosis as early as possible, so that its progress can be arrested and, if possible, reversed.

Prognostic testing is commonly used in preventive medicine (3). While a screening test should identify all individuals who may develop the disease (4), a prognostic test is used to predict the development or not of the future disease when a patient shows some heralding signs or symptoms. Examples of predictive testing in somatic medicine include fasting glucose and oral glucose tolerance test and glycated haemoglobin to detect subjects at high risk for diabetes (pre-diabetes or intermediate hyperglycaemia) (5). Pre-diabetes closely resembles the CHR state, in that only about 5-10% of people per year will progress to diabetes, with the same proportion converting back to normoglycaemia (5).

No biological tests such as those used to detect pre-diabetes are available in clinical psychiatry (6). Therefore, for an indicated prevention of psychosis, prognostic testing is usually accomplished by administration of specific psychometric interviews, which assess validated CHR criteria (7). These instruments include: the Comprehensive Assessment of At Risk Mental State (CAARMS, 8,9), the Structured Interview for Psychosis-Risk Syndrome (SIPS, 10) and the Basel Screening Instrument for Psychosis (BSIP, 11) for the assessment of “ultra-high risk” (UHR) criteria (12); and the Bonn Scale for the Assessment of Basic Symptoms (BSABS, 13) and the Schizophrenia Proneness Instruments (Adult version, SPI-A, 14, and Child & Youth version, SPI-CY, 15) for the assessment of basic symptom (BS) criteria (16).

The UHR criteria include attenuated psychotic symptoms (APS), brief limited intermittent psychotic symptoms (BLIPS) and trait vulnerability plus a marked decline in psychosocial functioning (genetic risk and functional deterioration syndrome: GRFD). The two partially overlapping BS criteria rely on subjectively experienced disturbances of perception, thinking, language and attention (17).

These CHR instruments show excellent reliability when used by trained raters: the overall inter-rater agreement is 0.95 for the SIPS (18), 0.85 for the CAARMS (12) and 0.91 for the SPI-A (19). Yet, their prognostic accuracy is still uncertain. For an ideal instrument, all subjects actually about to develop psychosis should be classified as “at risk” (CHR+) while those suffering from other complaints not leading to frank psychosis should be classified as “not at risk” (CHR−).

The prognostic accuracy of a test can be quantified by different measures – sensitivity (Se), specificity (Sp), summary receiver operating characteristic (SROC) curves, area under the curve (AUC) – whose evaluation requires follow-up not only of CHR+ but also of CHR− subjects. So far, no robust meta-analysis has addressed the consistency and magnitude of the prognostic accuracy of psychometric CHR testing, and the few available studies reported inconsistent prognostic accuracy findings (18,20). Because of this, the overall clinical utility (i.e., predictive value) of psychometric interviews in help-seeking and non-help-seeking subjects is still unknown.

Predictive values are not fixed indicators of a test performance, but are affected by the prevalence of the condition (4). Within help-seeking CHR+ samples, the ability of the above psychometric instruments to identify true positives is accumulating to 29% at 2-year follow-up (21,22) – a finding comparable to other preventive approaches in medicine (23). On the contrary, the predictive value and potential clinical utility of these instruments in samples with a lower prevalence of the condition, such as the general population, still await results from follow-ups (24–26). Similarly, the predictive value in other samples with a variable psychosis risk, such as unselected adolescents with psychiatric problems (27), subjects accessing public treatment services, psychiatric patients in forensic units (28), primary care patients, genetic high risk samples, prisoners, post-partum women, people with 22q11.2 deletion syndrome, users of high potency cannabis, military, black ethnic minorities, refugees, people with borderline personality disorders or epilepsy, is still largely unknown.

To overcome this lack of knowledge, we conducted the first robust meta-analysis to examine the consistency and magnitude of the prognostic accuracy of instruments used for psychosis prediction, while at the same time investigating their potential clinical utility in help-seeking samples of high risk services, in the general population and across other groups.

METHODS

Search strategy

Two investigators (MC, GR) conducted a two-step literature search. At a first step, the Web of Knowledge database was searched, incorporating both the Web of Science and MEDLINE. The search was extended until March 2015, including only abstracts in English. The electronic research adopted several combinations of the following keywords: “at risk mental state”, “psychosis risk”, “prodrome”, “prodromal psychosis”, “ultra-high risk”, “high risk”, “help-seeking”, “diagnostic accuracy”, “sensitivity”, “specificity”, “psychosis prediction”, “psychosis onset”, and name of the CHR assessment instruments. The second step involved the use of Scopus to investigate citations of previous systematic reviews on transition outcomes in CHR subjects and a manual search of the reference lists of the retrieved articles.

Articles identified through these two steps were then screened for the selection criteria on the basis of abstract reading. The articles surviving this selection were assessed for eligibility on the basis of full text reading. To achieve a high standard of reporting, we adopted the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) checklist (29).

Selection criteria

Studies were eligible for inclusion if: a) they were reported in original articles, written in English or in German; b) they had used in the same pool of referrals an established CHR psychometric instrument (index test); c) they had followed up both CHR+ and CHR− subjects for psychosis onset (reference index) using established international diagnostic manuals (ICD or DSM); d) they had reported sufficient prognostic accuracy data. With respect to this last point, when data were not directly presented, they were indirectly extracted from associated data. Additionally, we contacted all corresponding authors to request additional data when needed.

We excluded: a) abstracts, pilot datasets, reviews, articles in a language other than English or German; b) studies in which interviews were not conducted in the same pool of referrals or that used an external CHR− group of healthy controls; c) studies with overlapping datasets. In case of multiple publications deriving from the same study population, we selected the article reporting the largest and most recent data set. The literature search was summarized according to PRISMA guidelines (30).

Recorded variables

Data extraction was independently performed by two investigators (MC, GR). Data included author, year of publication, characteristics of subject samples (baseline sample sizes, mean age and age range, proportion of females), the CHR psychometric instrument used, exposure to antipsychotics, diagnostic criteria used at follow-ups to assess the psychotic outcome, follow-up time, prognostic accuracy data (number of true and false positives, true and false negatives or associated data) and quality assessment conducted with the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) checklist (31).

Statistical analysis

The statistical analysis followed the Cochrane Guidelines for Systematic Reviews of Diagnostic Test Accuracy, Version 1.0 (32) and the Methods Guide for Authors of Systematic Reviews of Medical Tests by the Agency for Healthcare Research and Quality (chapter 8) (33). Evaluating test accuracy requires knowledge of two quantities: the test’s Se and Sp. Meta-analysis methods for diagnostic test accuracy thus have to deal with two summary statistics simultaneously rather than one (32). Methods for undertaking analyses which account for both Se and Sp, the relationship between them, and the heterogeneity in test accuracy, require fitting advanced hierarchical random effects models (32).

For each study we constructed a two-by-two table, which included true positive, false positive, true negative, and false negative values. When studies reported different data at different follow-up times, we used data from the longest follow-up. The baseline sample size was conservatively used as the base reference to avoid a bias towards overly high transition risks at longer follow-ups and related higher drop-out rates of transition negatives.

Data were then analyzed with MIDAS (Meta-analytical Integration of Diagnostic Accuracy Studies) (34), a comprehensive program of statistical and graphical routines for undertaking meta-analysis of diagnostic/prognostic test performance in STATA 13 software. The index tests of CHR status (CHR+ or CHR−) and reference tests of transition to psychosis according to international diagnostic manuals (ICD or DSM as gold standard) were dichotomous.

Primary data synthesis was performed within the bivariate mixed-effects regression framework for the logit transforms of Se and Sp (34). In addition to accounting for study size, the bivariate model estimates and incorporates the intrinsic negative correlation that may arise between Se and Sp within studies (threshold effect) (35), as a result of differences in the test threshold between studies (36). The bivariate model allows for heterogeneity beyond chance as a result of clinical and methodological differences between studies (36).

We estimated the summary Se and Sp and the estimated hierarchical SROC curves (32). A SROC graph across each predictor, with the y-axis representing the predictor’s Se and the x-axis representing 1-specificity, was used to plot around the summary estimates a 95% confidence region and a 95% prediction region to illustrate the precision with which the summary values were estimated (confidence ellipse of a mean), and to show the amount of between-study variation (prediction ellipse; the likely range of values for a new study). We also estimated the AUC. Finally, for sensitivity analyses of the impact of follow-up times, supplementary analyses were conducted by grouping the data at each specific time point of 6, 12, 24 and ≥30 months.

Heterogeneity across studies was assessed using the I², with values of 25%, 50% and 75% representing mild, moderate and severe inconsistency, respectively (37). Within MIDAS, forest plots and heterogeneity statistics can be created for each test performance parameter individually or may be displayed as paired plots. Subgroups analyses and meta-regressions were used to examine the influence of CHR instruments used, mean age, gender (% females), follow-up time, sample size, exposure to antipsychotics, and quality assessment (QUADAS) on meta-analytical estimates. To control for biases associated with imbalanced datasets (38), we further tested the impact of the proportion of CHR+ subjects in the overall samples. The meta-regressions were used if there was substantial heterogeneity (I² >50%) (39).

Model diagnostic analyses included quantile plot of residual based goodness-of-fit; chi-squared probability plot of squared Mahalanobis distances for assessment of the bivariate normality assumption; spike plot for checking for particularly influential observations using Cook’s distance; a scatter plot for checking for outliers using standardized predicted random effects (standardized level-2 residuals) (34). Sensitivity analyses (i.e., exclusion of outliers and rerunning of the model) were conducted to further explore heterogeneity. We did not test publication bias (40), because no proven statistical method exists for this type of meta-analysis (41).

In a second step, we employed the probability-modifying plot and the Fagan’s nomogram to estimate the clinical or patient-relevant utility of the CHR interview in subjects seeking help at early detection services, in the general population, as well as in other samples (i.e., genetic high risk samples, prisoners, post-partum women, people with 22q11.2 deletion syndrome, users of high potency cannabis, military, black ethnic minorities, people with borderline personality disorders, and unselected psychiatric samples).

The clinical utility was evaluated using the positive and negative likelihood ratios (LR+ and LR−) to calculate post-test probability (PostTP) based on Bayes’ theorem (with pre-test probability, PrePT, being the prevalence of the condition in the target population), as follows: PostTP=LR × PreTP/[(1−PreTP) + (PreTP × LR)] (35). Specifically, the probability-modifying plot (34) is a graphical sensitivity analysis of the test’s predictive values across a baseline psychosis risk continuum in people seeking help at early detection services. It depicts separate curves for positive and negative tests and uses general summary statistics (i.e., unconditional positive and negative predictive values, NPV and PPV, which permit underlying psychosis risk heterogeneity) to evaluate the effect of the CHR assessment on predictive values (42). The PreTP probability of psychosis risk in subjects seeking help at early detection services was computed in the current dataset as the proportion of subjects developing psychosis on the total baseline sample (CHR+ plus CHR−) (34).

Fagan’s nomogram, a two-dimensional graphical tool for estimating how much the result of a test changes the pre-test probability that a patient will develop psychosis, was used to estimate the clinical value of psychometric CHR interview in the general population and in the other samples. Again, the clinical value is calculated on the LR+ and LR− obtained from the current meta-analysis (43) and using the pre-test psychosis risk in the different samples as estimated from the available literature.

Statistical tests were two-sided and statistical significance was defined as p values <0.05.

RESULTS

Database

The literature review (PRISMA flow chart available from the authors upon request) produced eleven independent studies that met the inclusion criteria, for a total of 2,519 subjects (CHR+: N=1,359; CHR−: N=1,160) referred to high risk services (Table1). The proportion of CHR+ subjects in the total sample was 0.54%, revealing an overall balanced dataset.

Table 1.

Studies included in the meta-analysis

Study	QUADAS score (max. 14); exposure to antipsychotics at baseline	Predictor (index test)	Psychosis diagnosis (reference standard)	Age (years, mean±SD, range)	Gender (% females)	Follow-up (months)	CHR+ subjects (baseline)	CHR− subjects (baseline)
Klosterkötter et al ((52))	14; No	BSABS (BS)	DSM-IV	29.3±10.0 ((15–53))	47.5	0, ≥30	110	50
Yung et al ((45))	12; Yes (% NA)	CAARMS (UHR)	CAARMS	18.1 ((15–24))	51.0	0, 6, 24	119	173
Riecher-Rössler et al ((11))	13.5; No	BSIP (UHR plus 4th criterion)	BPRS	26.8±8.9 ((18–60))	41.4	0, 6, 12, 24, ≥30	58	32
Woods et al ((20))	13.5; Yes (11.6%)	SIPS (UHR)	DSM-IV or medical records	17.8±4.4 ((12–36))	39.5	0, 6, 12, 24	259	111
Addington et al ((48))	13.5; Yes (1.8%)	SIPS (UHR)	DSM-IV	19.8±4.5 ((12–31))	47.8	0, 6, 12, 24	172	100
Liu et al ((49))	2.5; Yes (79.7%)	SIPS (UHR)	DSM-IV	21.4±4.0 ((16–24))	47.7	0, 24	59	48
Simon et al ((50))	6; No	SIPS/SPI-A (BS/UHR)	DSM-IV	21.0 ((14–40))	32.4	0, 12, 24	99	49
Lee et al ((44))	13; No	CAARMS (UHR)	DSM-IV	21.6±3.5 ((14–29))	39.9	0, 6, 12, 24, ≥30	173	494
Schultze-Lutter et al ((46))	13; Yes (13.8%)	SPI-A/SIPS (BS/UHR)	DSM-IV	24.9±6.0 ((15–39))	37.0	0, 6, 12, 24, ≥30	194	52
Kotlicka-Antczak et al ((47))	11.5; Yes (10.2%)	CAARMS (UHR)	ICD-10	19.0±3.6 ((15–29))	51.1	≥30	94	33
Spada et al ((51))	11; No	CAARMS (UHR)	DSM-IV	15.8±1.7 ((12–17))	47.5	0, 6	22	18

QUADAS – Quality Assessment of Diagnostic Accuracy Studies checklist, CHR – clinical high risk, UHR – ultra-high risk, BS – basic symptoms, BSABS – Bonn Scale for the Assessment of Basic Symptoms, BPRS – Brief Psychiatric Rating Scale, BSIP – Basel Screening Instrument for Psychosis, CAARMS – Comprehensive Assessment of At Risk Mental State, SIPS – Structured Interview for Prodromal Syndromes, SPI-A – Schizophrenia Proneness Instrument, NA – not available

Four studies employed the CAARMS, three the SIPS, one the BSIP, one the BSABS, and two both the SIPS and the SPI-A. The mean follow-up time was 37.72 months (SD 27.81, median=33). QUADAS ratings ranged from 2.5 to 14 (the latter is the highest possible score). The main reasons for a non-optimal rating were (partial) exposure to antipsychotics and unsatisfactory reporting of results.

Prognostic accuracy of CHR interview

Across the eleven studies interviewing help-seeking subjects for CHR symptoms, the summary meta-analytical estimate of Se and the AUC were outstanding, while the estimate of Sp was poor (Figure 1). There was moderate to substantial heterogeneity for Se (I²=51, p=0.02) and severe heterogeneity for Sp (I²=95, p<0.001), 17% of which was due to threshold effects.

Figure 1.

Meta-analytical summary receiver operating characteristic (SROC) curve of clinical high risk (CHR) psychometric interviews. Se – sensitivity, Sp – specificity, AUC – area under the curve, 1 – Klosterkötter et al (52), 2 – Yung et al (45), 3 – Riecher-Rössler et al (11), 4 – Woods et al (20), 5 – Addington et al (48), 6 – Liu et al (49); 7 – Simon et al (50), 8 – Lee et al (44), 9 – Schultze-Lutter et al (46), 10 – Kotlicka-Antczak et al (47), 11 – Spada et al (51)

Sensitivity analyses revealed that the two studies with the highest proportion of CHR− subjects in the total sample had the highest Sp (44,45), while the two studies with the lowest proportion of CHR− subjects had the lowest Sp (46,47). However, meta-regression analyses showed that the proportion of CHR+ subjects in the total sample had no impact on the overall AUC (38).

Across SIPS samples (20,46,48–50), Se was 0.96 (95% CI: 0.88-0.99) and Sp was 0.39 (95% CI: 0.32-0.46). Across CAARMS samples (44,45,47,51), Se was 0.96 (95% CI: 0.82-0.99) and Sp was 0.56 (95% CI: 0.38-0.73). There were not enough data to perform subgroups meta-analyses in BSIP samples (11), BSABS/SPI-A samples (46,50,52) and samples combining the SIPS and SPI-A (46).

Meta-regression analyses revealed no significant effects for mean age, gender, follow-up time, sample size and quality assessment (QUADAS), but there was a significant effect for exposure to antipsychotics at baseline (p=0.04). This effect was driven by a significant decrease of Se (0.94) in the five studies where subjects were exposed to antipsychotics as compared to the six studies where subjects were not exposed (Se=0.98).

Model diagnostics revealed a good fit of the model and indicated that one study was close to the outlier threshold (44). Sensitivity analyses confirmed a very good AUC (0.84) after this study was removed from the dataset.

Supplementary analyses were conducted grouping the available samples at specific time points of 6, 12, 24 and ≥30 months. The AUCs were outstanding at each time point: at 6 months (seven samples, AUC=0.97, 95% CI: 0.95-0.98), at 12 months (six samples, AUC=0.94, 95% CI: 0.92-0.96), at 24 months (eight samples, AUC=0.94, 95% CI: 0.92-0.96), and at ≥30 months (seven samples, AUC=0.91, 95% CI: 0.88-0.93).

Clinical utility of psychometric CHR interviews in subjects seeking help at high risk services

The 38-month psychosis risk in the 2,519 help-seeking subjects was 15% (95% CI: 0.9%-24%). On the basis of this prior distribution, the continuous relationship between PreTP and PostTP probability is summarized in Figure 2. Being CHR+ was associated with a 26% (95% CI: 23%-30%) risk of developing psychosis within 38 months, yet a small LR+ of just 1.82 (95% CI: 1.52-2.18), while being CHR− was associated with a 1.56% (95% CI: 0.7%-2.42%) risk of developing psychosis and a large LR− of 0.09 (CI 95%: 0.04-0.18) (Figure 3).

Figure 2.

Meta-analytical probability modifying plot, illustrating the relationship between pre-test probability (PreTP) (9 to 24% psychosis risk at 38 months in subjects seeking help at early detection services) and post-test probability (PostTP) (psychosis risk at 38 months in help-seeking subjects based on clinical high risk psychometric interviews), computed as the likelihood of a positive (above diagonal line; LR+) or negative (below diagonal line, LR−) test result over the 0-1 range of PreTP

Figure 3.

Fagan’s nomogram illustrating the meta-analytical clinical value (post-test probability) of clinical high risk (CHR) psychometric interviews in the general population in order to predict risk of psychosis at 38 months, given an assumed psychosis risk (pre-test probability) of 3.27%, as reported in a nationally representative sample of general population subjects aged 30-44 years (see 53)

Estimated clinical utility of psychometric CHR interviews in the general population and in other samples

Based on a lifetime prevalence of all non-organic psychotic disorders of 3.27% (53) and the above LRs, Fagan’s nomogram revealed only limited clinical utility for CHR instruments in the general population. Testing positive for CHR was associated with a 5.74% lifetime risk of developing psychosis, while testing negative was associated with hardly any such risk (0.26%). Corresponding figures for other clinical and non-clinical samples are displayed in Table2.

Table 2.

Estimated clinical utility of clinical high risk psychometric instruments for psychosis prediction in various populations

Sample	Psychosis risk	Positive test result	Negative test result
Unselected psychiatric adolescents (27)	3.13% (12 mo.)	3.13%	0.29%
Subjects in contact with public treatment services (54)	0.35% (lifetime)	0.63%	<0.001%
Psychiatric patients in forensic units (55)	74% (lifetime)	83.38%	20.39%
Primary care patients (56)	0.045% (per year)	<0.001%	<0.001%
Prisoners (57)	3.90% (lifetime)	6.87%	0.36%
Post-partum women (58)	4% (12 mo.)	7.04%	0.37%
22q11.2 deletion syndrome (59)	16% (48 mo.)	25.74%	1.68%
Young adults at familial risk for psychosis (60)	12% (30 mo.)	19.88%	1.21%
Users of high potency cannabis (61)	24% (lifetime)	36.49%	2.76%
Military (62)	0.014% (per year)	<0.001%	<0.001%
Black ethnic minority (63)	1.45% (lifetime)	2.60%	0.13%
Refugees (64)	3.3% (lifetime)	5.84%	0.31%
Epilepsy (65)	5.6% (lifetime)	9.74%	0.53%

DISCUSSION

This is the first study to present a robust and elaborated meta-analytical estimate of the prognostic accuracy of CHR psychometric interviews for psychosis prediction. Assessing help-seekers referred to a high risk service with a CHR interview generally revealed an excellent overall prognostic performance in terms of the AUC at 38-month follow-up (values of 0.9-1.0 are considered outstanding, of 0.8-0.9 excellent and of 0.7-0.8 acceptable) (66), which is comparable to other preventive approaches in medicine. However, excellent AUC values were mainly mediated by an outstanding ability of the instruments to rule out psychosis (i.e., very satisfyingly low LR− and high Se), at an expense of their ability to rule in psychosis (i.e., unsatisfactory low LR+ and only moderate overall Sp), which indicates some need to further improve prediction. On the contrary, the clinical utility of current CHR instruments in non-help-seeking subjects in the general population was estimated to be low.

Our first aim was to investigate at meta-analytical level the overall prognostic accuracy of CHR instruments in determining the risk of developing psychosis at 38 months in young help-seeking subjects referred to high risk services. We first estimated the AUC, which serves as a global measure of test performance and indicates the overall goodness of a diagnostic tests. Thereby, we adopted a robust methodological approach following international guidelines for diagnostic/prognostic accuracy meta-analysis, to avoid the serious flaws observed in a previous meta-analytical attempt, such as overlapping samples, missing studies and lack of control for several moderators (67,68). Our finding of consistent prognostic accuracy across CHR instruments is particularly important, given the significant differences in their criteria (69). This evidence of a negligible role of the CHR assessment instrument (i.e., CAARMS vs. SIPS) is in line with our previous meta-analysis, which found no differences in pooled annual transition risks between these instruments (21). This finding was also confirmed by a second independent meta-analysis (22).

We further revealed that, despite an excellent overall prognostic accuracy, there is a need to specifically improve the ability to rule in subsequent psychosis, i.e., to improve LR+ and Sp, while preserving the outstanding ability to rule it out. This is particularly relevant given that interviewing subjects seeking help at high risk services is particularly difficult: these individuals are assumed to lay on an upper mid-range of a symptomatic continuum by showing mild and often infrequent symptoms of yet some clinical significance already (24).

However, differentiating between such gradual symptoms with specific tests or interviews is not a problem specific to psychosis prevention or other preventive approaches in psychiatry. For example, in case of the at-risk state of diabetes, the World Health Organization (WHO) proposed the use of the term “intermediate hyperglycaemia” (i.e., pre-diabetes) to accurately reflect the observation that glycaemia is a continuous variable and that their defined categories are based on somewhat arbitrary decisions on where to draw a line between normality and abnormality (70). Similarly to the different cut-offs and criteria used to identify CHR subjects, the definition of pre-diabetes is based on cut-off points for glycaemia (5) for which there are different operationalizations (e.g., by WHO and by the American Diabetes Association) (5). Furthermore, as for the CHR state (7), progression to diabetes is not inevitable in pre-diabetes; some individuals, in the absence of any intervention, may remain in that state or even revert to normoglycaemia (5). Because of this, various risk assessment tools based on socio-demographic or questionnaire data are available to identify subjects with pre-diabetes, and their overall prognostic accuracy is comparable to our meta-analytical estimates, such as the AUC=0.76 reported for the Cambridge risk score (71). More broadly, the overall prognostic accuracy of the CHR instruments was comparable if not superior to various other medical tests used for an indicated prevention (Table3).

Table 3.

Prognostic accuracy of indicated prevention tests in clinical medicine

At-risk population	Outcome	Diagnostic test	Sensitivity (follow-up)	Specificity (follow-up)	AUC (follow-up)
Patients presenting for CHR evaluation	Psychosis	CHR interview	0.96 (2 yrs.)	0.47 (2 yrs.)	0.89 (2 yrs.)
Men at risk for prostate cancer	Prostate cancer	PSA (72,73)	0.69 (5 yrs.)	0.89 (5 yrs.)	0.88 (5 yrs.)
Men at risk for colorectal cancer	Colorectal cancer	Risk prediction model (74)	NA (5 yrs.)	NA (5 yrs.)	0.80 (5 yrs.)
Women at risk for colorectal cancer	Colorectal cancer	Risk prediction model (74)	NA (5 yrs.)	NA (5 yrs.)	0.73 (5 yrs.)
Patients with transient ischemic attack	Stroke	ABCD2 score (75,76)	0.57 (30 days)	0.32 (30 days)	0.72 (7 days)
Patients with stable coronary disease	Coronary event	Framingham risk score + number of diseased vessels (77)	NA (8.5 yrs.)	NA (8.5 yrs.)	0.67 (77) (8.5 yrs.)
Pre-diabetes	Diabetes	30-min plasma glucose (78)	0.91 (9 yrs.)	0.39 (9 yr.)	0.67 (9 yrs.)
Mild cognitive impairment	Alzheimer’s disease	ADAS-cog subscale (79)	0.62 (1 yr.)	0.73 (1 yr.)	0.67 (1 yr.)
Women at risk for breast cancer	ER-positive invasive breast cancer	Gail model (80)	0.50 (5 yrs.)	0.65 (5 yrs.)	0.60 (5 yrs.)

CHR – clinical high risk, AUC – area under the curve, PSA – prostate specific antigen, ER – estrogen receptor, NA – not available, ADAS-cog – Alzheimer Disease Assessment Scale-cognitive part

However, it is important to highlight that the high AUC of CHR instruments is secondary to an accurate training of raters and ongoing close supervision provided by expert clinicians (7). Thus, a recent guidance on the early detection of psychosis explicitly recommends CHR assessment to be conducted in specialized centres by well-trained raters and/or clinical supervision by such raters (22).

The imbalance between an excellent Se (0.96) and an only modest Sp (0.47) may have some relevant clinical implications, when considering that we have selectively included only studies discriminating CHR+ from CHR− within the same pool of help-seeking subjects. Since these patients were seeking help at or were subsequently referred to early detection services and frequently presented also with psychosocial and functional impairment (81) and other non-psychotic symptoms (82) and disorders (83), the use of CHR assessments should not be thought of as identifying and treating an unselected and asymptomatic group at risk of a poor outcome (universal prevention) (84). Rather, the use of CHR assessment follows the approach of an indicated prevention, which is concerned with detecting a disease in its earliest stages, before frank symptoms appear, and with intervening to slow or stop its progression into the full-blown medical picture. Therefore, the above-mentioned recent guidance explicitly restricts CHR assessment to the clients of mental health services (22).

With regard to the potential CHR+ misdiagnosis of persons who do not in fact develop psychosis, or the potential CHR− misdiagnosis of persons who will develop psychosis, the low Sp suggests a stepped and multi-component strategy. In a first sensitivity-preserving step, CHR instruments could be used to rule out true negatives, i.e. subjects who are unlikely to develop psychosis. In a second step, additional clinical, neurocognitive, biological or combined models of risk stratification could be applied to the CHR+ group, with the aim of increasing Sp and prognostic reliability. This would enable risk stratification and personalized treatments accordingly (85,86).

We further estimated the clinical utility of CHR assessments in other clinical and non-clinical populations, as clinical utility is affected by the underlying psychosis risk in a population. We found that testing positive for CHR was associated with a 26% risk of developing psychosis within 38 months, a proportion comparable with our previous meta-analysis (95% CI: 23-35) (21) of transition risks in CHR+ subjects. This was due to a small LR+ of 1.82. We could also show here for the first time that being CHR− was associated with only a 1.56% risk of developing the illness, corresponding to a large LR− of 0.09. It is important to note that the PostTP, as estimated from the likelihood ratio and PreTP, is generally more accurate than if estimated from the PPV of the test. In fact, with the help of these two measures (LR+ and LR−), it was possible to estimate the PostTP in different settings characterized by a variable PrePT of psychosis risk, which however will still require empirical studies.

We clearly estimated for the first time a limited clinical utility of CHR interviews in the general population, revealing only a small and inadequate PPV of 5.74%. This estimate is in line with meta-analytical results indicating that self-reported psychotic-like experiences in the young non-help-seeking general population are associated with a negligible risk of transitioning to psychotic disorders over time (87). Yet, as self-reported psychotic experiences are only a poor estimate of clinician-assessed CHR symptoms, these findings might not reflect the true predictive power of CHR criteria in the community. Similarly, it appears there is no scope to use psychometric CHR interviews in unselected psychiatric adolescent samples, patients accessing public treatment or primary care services, patients admitted to forensic units, post-partum women, ethnic minorities, military, refugees, patients with epilepsy and prisoners. The latter finding is in line with a recent study indicating that the CHR state does not predict psychosis in adolescent delinquent samples (28). On the other hand, our estimates provide some support for the clinical utility of CHR assessments in subjects with two psychotic relatives, in patients with 22q11.2 deletion syndrome and in subjects using high potency cannabis, as well as for preventive trials already proposed in some of these clinical samples (88).

The additional novel finding is that our probability-modifying plot allows future power calculation studies in samples characterized by an underlying variable psychosis risk that is ranging from 0 to 1. For example, with our plot available, researchers may draw a vertical line from the selected pre-test probability of the sample to the appropriate likelihood ratio line and then read the post-test probability off the vertical scale.

Some limitations of this meta-analysis should be acknowledged. First, because of the limited statistical power, we were unable to directly compare the prognostic accuracy of different psychometric instruments. However, subgroups analyses revealed comparable SIPS vs. CAARMS AUCs. Furthermore, two independent meta-analyses (21,22) did not reveal any significant impact of the type of psychometric instrument employed on risk estimates. Also, we were unable to explain all the observed heterogeneity across individual studies. However, some of this was accounted for by threshold effects and the effect of antipsychotics exposure on Se. An effect of age, with lower transition risks in younger CHR+ subjects, was observed in our first meta-analysis (21) and recently confirmed in another re-analysis (22). Such an age effect might have been missed in our analyses, as only the by far smallest of the included studies, with an only 6-month follow-up (69), was on minors only.

Furthermore, the individual studies included here varied with respect to follow-up time, although meta-regression did not reveal any significant effect of this variable. We additionally conducted supplementary analyses at each specific time point, and these analyses confirmed excellent AUCs. Furthermore, there is new meta-analytical evidence that, in UHR samples, transition to psychosis is most likely to occur within the first 2 years after presentation to clinical services, with a stable plateau after 36 months (89). Since our mean follow-up time (38 months) falls in this plateau period, follow-up had no significant impact on the meta-analytical estimates across samples mainly at risk for UHR criteria.

CONCLUSIONS

The present prognostic accuracy meta-analysis indicated that currently used interviews for psychosis prediction have an excellent overall prognostic performance. This supports their use as clinical tools for an indicated prevention in subjects seeking help at mental health services worldwide, provided raters have undergone adequate training, while discouraging their use for prevention in non-help-seeking subjects in the general population.