Why ultra high risk criteria for psychosis prediction do not work well outside clinical samples and what to do about it

The use of ultra high risk (UHR) criteria in selected help‐seeking samples is the only clinical possibility to alter the course of psychosis by preventing its onset. The UHR paradigm can additionally reduce the duration of untreated psychosis1 and provide extended benefits to patients who are experiencing a first episode of psychosis2.

Because of these potentials, there is a great interest in the use of UHR outside clinical samples, such as in the general population. The first epidemiological study investigating the significance of UHR criteria in the non‐help‐seeking general population aged 8‐40 was published in this journal3. It indicated that only 1.3% of the general population met the UHR criteria of the Structured Interview for Psychosis‐Risk Syndromes (SIPS)3. The longitudinal fate of these individuals has just been released4: 143 UHR and 131 controls were followed up for an average of 2.5 years, with three transitions to psychosis in the UHR group (psychosis risk = 2.09%) and no transition in the control group.

These results are of great interest, as they may support the epidemiological validity of the UHR paradigm, although they are likely to be underpowered (assuming a 0.001% risk in the control group as continuity correction and an alpha = 0.05, the resulting power would be of 38% only). Beyond these limitations, the key finding of 2.09% psychosis risk (at 2.5 years) in people meeting UHR from the general population is of crucial clinical relevance. It is strikingly lower than the annualized 2‐year 20% (95% CI: 17%‐25%)5 transition risk in help‐seeking UHR samples, that are characterized by frequent comorbid affective disorders and functional impairments6.

These findings clearly confirm that the prognostic accuracy of the UHR criteria strictly depends on the sample to which they are being applied. Indeed, clinical help‐seeking samples of individuals undergoing UHR assessment are characterized by a substantial pre‐test risk enrichment (pre‐test risk for psychosis)7 of up to 15% at 38 months8. As demonstrated in a previous paper in this journal9, the use of UHR assessment is associated with a small positive likelihood ratio of 1.82 at 38 months and a modest ability to rule in psychosis9. Therefore, to reach some prognostic accuracy of clinical utility in individuals meeting UHR criteria, it is necessary to apply them to samples that are already enriched in psychosis risk, i.e., with a significant pre‐test risk. For example, a recent study published in this journal10 has shown that meeting the UHR criteria given an underlying 22q11.2 deletion syndrome, a condition that is characterized by a substantial pre‐test risk for psychosis, is associated with a 27.3% risk of psychosis at 32 months.

These considerations clearly limit the practical utility of the UHR outside of clinical samples, as recently recognized by the recommendation no. 4 of the European Psychiatric Association, which suggests that the UHR assessment should be primarily offered to selected samples of subjects “already distressed by mental problems and seeking help for them”.

At the same time, because of the potential benefits yielded by the UHR paradigm, it seems important to continue exploring the usefulness of an extended application of UHR assessment in several different samples. A first pragmatic approach to estimating the prognostic accuracy of the UHR assessment in several scenarios would be to use the meta‐analytical Fagan's nomogram that we presented in a previous paper in this journal9. This nomogram is based on the intrinsic properties of the UHR assessment (such as the positive and negative likelihood ratios7) and can be applied to different populations with a given pre‐test risk of psychosis onset to estimate their post‐test risk of psychosis at 38 months.

Importantly, our nomogram has now been externally validated. In fact, with that nomogram, we had estimated a small post‐test psychosis risk (less than 5% at 38 months) in the general population, a value that is similar to the real value observed in the epidemiological study discussed above4. Similarly, with our nomogram, we had estimated a post‐test psychosis risk of 26% for patients affected with the 22q11.2 deletion syndrome9, which exactly matches to the real value recently reported in this journal10.

The use of our nomogram can thus provide reliable estimates (along with 95% CIs) for post‐test risk of psychosis in individuals meeting UHR criteria, given a determined pre‐test risk. Using the nomogram, researchers can simulate the expected prognostic accuracy, and estimate the required sample size needed to test their hypotheses.

Since the use of the UHR assessment outside clinical samples is likely to be associated with low predictive power, it is fundamental to perform accurate power calculations. In this scenario and considering the probability of infrequent events, a second approach could involve using sequential testing methods11, for example by using the SIPS in samples already enriched for psychosis risk, as shown in this journal12. Sequential testing is traditionally adopted in medicine to enrich the risk of samples that are selected to undergo different diagnostic or prognostic tests.

A third practical approach could be to better investigate the factors that modulate pre‐test risk enrichment in samples undergoing a UHR assessment. We have recently shown that it may be possible to stratify help‐seeking individuals undergoing UHR assessment through the use of simple socio‐demographic and clinical variables13. The predictive model has been externally validated and can be used to inform future research in the field, with the scope to improve prognostic accuracy of psychosis prediction.

Paolo Fusar‐Poli
King's College London, Institute of Psychiatry, Psychology and Neuroscience; OASIS Service, South London and Maudsley NHS Foundation Trust, London, UK