Validation of a modified version of the PRIME screen for psychosis-risk symptoms in a non-clinical Kenyan youth sample

Abstract

Background

The PRIME screen is a self-administered questionnaire designed to quickly assess individuals at risk for developing a psychotic disorder. It is shorter in both length and administration time compared to the Structured Interview for Psychosis-Risk Syndromes (SIPS)—a standard instrument for psychosis prodromal risk assessment. Validation of the PRIME against the SIPS has not been reported in large non-clinical populations.

Methods

A culturally modified version of the PRIME screen (mPRIME) was administered to Kenyan youth between the ages of 14 and 29. 182 completed both the SIPS and mPRIME. Validation measures (sensitivity, specificity, positive predictive value, negative predictive value) were calculated and the study sample was then broken down into true positives, false positives, and false negatives for comparison on different quantitative measures.

Results

Using previously suggested thresholds for a positive screen, the mPRIME had a sensitivity of 40% and a specificity of 64.8% for our entire sample. Positive predictive value (PPV) and negative predictive value (NPV) were 12.3% and 89.7%, respectively. Breaking the sample down by questionnaire outcome showed that true-positive individuals scored higher on average rate and intensity of endorsement of mPRIME items compared to false-positive and false-negatives, while false-negatives on average registered disagreement on all mPRIME questionnaire items.

Conclusions

The mPRIME does not appear to be an effective screener of at-risk individuals for psychosis in our non-clinical sample. Further validation efforts in other general populations are warranted.

INTRODUCTION

Schizophrenia is increasingly being understood as a developmental syndrome that often has manifestations before the full-blown disorder symptomatology presents [1-4]. While clinicians often identify and begin to treat individuals with psychotic illness after the disorder has already manifested, an increasing amount of effort is being placed on identifying individuals at risk for psychosis before they develop a psychotic disorder, thereby improving functioning and the chances of preventing illness [5,6]. Past research has shown that attenuated symptoms of psychosis are a major characteristic of the prodromal phase of psychotic disorder and that early attention to premorbid dysfunction states can be critical to potentially changing the course of later illness. For these reasons, efforts to focus on early intervention strategies become even more critical for treatment development [5,7,8]. As part of these efforts, many research programs around the world have developed criteria to identify individuals who are at risk for psychotic disorder development but who have not yet met criteria for actual psychotic syndromes (e.g., prodromal, clinical high-risk, ultra-high risk) [5,7,9-11]. A prominent example of such efforts is the ultra-high risk (UHR) criteria which involves identification of at-risk individuals based on combinations of attenuated or self-limited psychotic symptoms, family history, and recent functional decline amongst other characteristics [12,13]. A major focus in identified high-risk groups has been on the rates of conversion to psychotic syndromes in the first few years after initial identification. Past longitudinal studies lasting 12 to 24 months have shown psychosis conversion rates ranging from 15% to around 40% [13-17]. Ongoing work has tried to improve the ability of high-risk criteria to predict conversion rate by both incorporating relevant genetic and environmental factors with improved objective assessment measures.

The Structured Interview for Psychosis-Risk Syndromes (SIPS) is a screening tool designed to identify early symptoms of the psychosis prodrome and to identify individuals with psychosis–risk syndromes [18]. The primary portion of the SIPS that is used to classify those at risk for later psychotic illness development inquires about the past occurrence of positive symptoms such as perceptual abnormalities, delusional ideation and other experiences. If the examined individual is determined by the test administrator to have had a previous experience of a positive symptom, the severity of that occurrence is then scored on a 0 to 6 Scale of Psychosis-risk Symptoms (SOPS) where a score of 3 represents the threshold for having an Attenuated Positive Symptom Syndrome (APSS). The SIPS is performed by a trained individual and requires several hours of background and education about the procedure before it can be administered—which itself takes over an hour.

In an effort to simplify the assessment process, a variety of screening instruments have been developed to serve as more efficient ways of identifying at-risk individuals. One such example was developed by the Prevention through Risk Identification, Management, and Education (PRIME) group at Yale University (the PRIME screen). The PRIME screen is a short self-administered questionnaire based on the positive symptom portion of the SIPS and requires minutes to complete [19]. Its 12 questions ask about the occurrence of positive symptom experiences over the last year with responses measured on a Likert-scale of 0 (definitely disagree) to 6 (definitely agree) with a response of ‘not sure’ being 3. Validation studies in general non-help seeking populations have been lacking. Early validation measurements for the PRIME in a patient sample against the SIPS showed a sensitivity of 0.90 and a specificity of 1.0 [19]. Later validation studies of the PRIME screen in clinical samples have found a specificity of 0.74 and sensitivity of 1.00 in a Japanese youth sample [20] and a specificity and sensitivity of 0.66 and 0.75, respectively in a U.S. sample [21]. Other validation studies for the PRIME screen as well as other psychosis-risk screeners that have occurred in clinical or help-seeking populations have generally found fair to strong measures of validity as well [22-24].

Our group previously examined psychosis risk in young Kenyan populations using a slightly modified version of the PRIME screen to account for cultural differences in screen item interpretation (mPRIME), and found that 45.5% of participants aged 14 to 29 reported having had a psychosis risk symptom (i.e., a score of ‘6’) on the screen [25]. The main goal for the current study was to evaluate the validity of the mPRIME as a screening tool for general youth populations, using the SIPS as the reference standard. To our knowledge, there have been no published studies on PRIME validation in a non-clinical population. Our objective was to obtain a large sample of general population Kenyan youth for administration of both the mPRIME and SIPS and to determine validity measures for the mPRIME. We hypothesized that major validity measures would be less robust than those found in studies of clinical populations.

METHODS

Participants

Participants were recruited in the summer of 2010 from the Kangemi neighborhood in Nairobi, Kenya. All were proficient in reading and writing English. Written and signed consent was obtained from the study participants and the study was approved by the institutional review board of Washington University Medical School, the Kenyan Medical Research Institute, and the Ministry of Education, Science, and Technology, Kenya. A total of 2758 individuals between the ages of 14 and 29 were recruited for our prior survey using mPRIME, as detailed in Mamah et al [25].

Assessment

Ten of the twelve mPRIME questions were identical to the original PRIME screen questionnaire with the only differences being replacement of items 9 (“I think I might feel like my mind is ‘playing tricks on me’”) and 12 (“I have been concerned that I might be ‘going crazy’”) with modified items (“I feel that my ability to properly think or mentally function has seriously worsened in the last month” and “I have been concerned that I might be ‘going mad’” for items 9 and 12 respectively) (See Figure 1 for full mPRIME screen). The response choices for the mPRIME items are on a Likert-scale as follows: 0-definitely disagree; 1-somewhat disagree; 2-slightly disagree; 3-not sure; 4-slightly agree; 5-somewhat agree; 6-definitely agree.

Figure 1.

Items from the administered mPRIME screen

200 individuals who completed the mPRIME were randomly selected via stratified sampling to receive the SIPS interview, such that the highest mPRIME scores in any item were 0-3 in 50 individuals, 4-5 in 100 individuals, and 6 in 50 individuals. Of the 200 individuals selected, 182 completed the SIPS. Each participant was administered the SIPS by research assistants who were health science students enrolled at the University of Nairobi. Research assistants underwent an extensive 2-day training in the SIPS background and administration by a psychiatrist (D.M.) with significant prior experience in conducting the SIPS interview, formal certified training in the SIPS and PRIME by the developers of the assessments at Yale University, and extensive clinical and research experience in psychotic disorders. Training also included sessions on psychotic illness and symptoms and discussion of illness presentations by D.M. and other expert members of the research team. The ratings of eight SIPS raters were also initially measured for level of agreement after observing one individual being administered the SIPS. Using a computerized kappa coefficient calculator [26] and each item on the SIPS as a “case” a Fleiss kappa value of 0.28 was obtained for the positive symptom portion of the SIPS and 0.43 for the entire SIPS examination, corresponding to roughly “fair” or “moderate” levels of inter-rater agreement for the various SIPS items [27]. As this was not an ideal measure of agreement considering there was only one case being rated, similar analysis was performed with a different set of research assistants who came from the exact same student population, receiving the exact same training from the same individuals at a similar time frame; this set of research assistants was calibrated with rating six different individuals on the SIPS. Inter-rater reliability correlations for the positive section of the SIPS as measured using the method detailed by Shrout and Fleiss in the seminal 1979 paper on the subject [28] revealed ICCs of .512, .537, .691, .736, and .792 for the individual SIPS items—indicating moderate to strong agreement between trained raters on the SIPS [unpublished data]. Only five cases were measured for agreement as the sixth case showed almost complete unanimity and too little variation in rater’s responses on the items to calculate an ICC.

Data Analysis

General statistical analyses were completed with IBM SPSS Statistics 19.0.1 (IBM Corp., Armonk, NY). Validity measures (sensitivity, specificity, positive predictive value [PPV], and negative predictive value [NPV]) for the mPRIME against the SIPS were also calculated. A positive screen on the mPRIME is classified as reporting a ‘5’ (“somewhat agree”) on at least three different mPRIME questions or a ‘6’ (“definitely agree”) on one mPRIME screen question. A positive SIPS examination (APSS) is defined as scoring a ‘3’ (“moderate”) on at least one item in the positive symptom section of the assessment. For purposes of our analysis, we used the APSS as the psychosis risk syndrome of comparison to the mPRIME screen as the latter involves questioning of positive psychotic experiences.

Different portions of the study sample (true positive mPRIME cases, false positive mPRIME cases, and false negative—but SIPS-positive—cases) were also compared on average scores on questionnaire items using ANOVAs with subsequent Tukey’s for post-hoc analysis. For mPRIME item analysis, “not sure” responses were removed and the remaining six possible scores were coded from 1 (definitely disagree) to 6 (definitely agree). Fisher Exact Test analyses were utilized to determine significant differences in group response rates for agreement on each mPRIME item. Sample data were also examined to look at different mPRIME screen thresholds to determine validity measures. Receiver operated characteristic (ROC) curves were generated for positive thresholds for ‘4’ and ‘5’ responses on the mPRIME; different combinations of these positive responses were also explored for validity.

RESULTS

Demographic Information

Among the 182 participants that were assessed using the SIPS, there were 74 females and 108 males. The average age for this group was 19.0 years with a standard deviation of 3.33 years (females: 18.7 years (2.89); males: 19.1 years (3.64)).

Validity Measures

Validity measurements for different combinations of threshold cutoffs for the mPRIME are provided in Table 1. The recommended threshold for a positive mPRIME screen (at least three ‘5’s or one ‘6’ on mPRIME items [19,20] showed the most favorable validity characteristics (sensitivity of 40% and specificity of near 65%) as judged by location on a corresponding ROC curve (see Figure 2). The other examined combinations of mPRIME positive item thresholds fell below the no-discrimination line. ROC curves for thresholds involving different numbers of mPRIME ‘4’ and ‘5’ responses are presented in Figure 3.

Table 1.

Validity measures for different thresholds for a positive mPRIME screen

positive mPRIME threshold	Sensitivity	Specificity	PPV	NPV
at least one item at 4	65%	21.6%	9.3%	83.3%
at least three items at 4	45%	51.9%	10.1%	88.6%
at least one item at 5	50%	49.4%	10.9%	88.9%
at least three items at 5 or one 6	40%	64.8%	12.3%	89.7%
at least one item at 6	25%	70.4%	9.4%	88.4%

Figure 2.

ROC curve for previously recommended threshold for positive mPRIME screen

Figure 3.

ROC curves for positive mPRIME thresholds utilizing different amounts of ‘4’ and ‘5’ responses

mPRIME Subgroup Item Score Comparisons

We compared questionnaire item scores of the mPRIME true positive cases (n=8), false negative cases (n=12), and false positive cases (n=56). For this analysis, a positive mPRIME was defined as at least three ‘5’s or one ‘6’ on screener item scoring. mPRIME scores are presented graphically in Figure 4. For all assessment items on the mPRIME, true positive case scores were not significantly different from those of false positive cases (though true positives scored on average higher than false positives on all mPRIME items except “special or supernatural gifts”). In contrast, there was a marked statistically significant difference in average scores between the false negative cases and those for both other participant subgroups on multiple items in which false negatives showed lower scores (for “people talking” only the true positive and false negative groups differed significantly in average scores; the items of “odd or unusual things”, “worsened functioning”, and “thoughts out loud” were not different across groups). Statistics across groups are presented in Table 2.

Figure 4.

mPRIME item scores of the different validity groups

Table 2.

Measure of statistical difference across groups (TP, FN, FP) for mPRIME items

mPRIME item	df	F-statistic	Sig. (p)
P1	2	2.762	.070
P2	2	7.609	.001**
P3	2	4.550	.014*
P4	2	6.292	.003**
P5	2	8.220	.001**
P6	2	7.560	.001**
P7	2	6.466	.003**
P8	2	10.113	.000**
P9	2	1.102	.338
P10	2	3.781	.027*
P11	2	2.009	.142

^*p<0.05;

^**p<0.01

TP: true positive; FN: false negative; FP: false positive

mPRIME Item Agreement

Across all items, the true positive group generally scored higher for amount of “agree” responses on mPRIME questions while the agreement response rates for false negative individuals were markedly lower than both true positive and false positive groups (Table 3). For seven of the screen items, the differences across groups in response rates for “agree” were found to be statistically significant (see Table 3).

Table 3.

Comparison of “agree” responses on different mPRIME screen items between groups

mPRIME Item	Percentage Response “agree”			Group Comparison
	True positive	False negative	False positive	Significance (p)
P1	62.5	16.7	51.8	0.053
P2	62.5	8.33	58.9	0.003**
P3	62.5	16.7	51.8	0.044*
P4	74.0	8.33	41.1	0.007*
P5	87.5	0	60.7	<0.001**
P6	87.5	8.33	57.1	<0.001**
P7	75.0	0	42.9	<0.001**
P8	62.5	8.33	64.3	0.001**
P9	37.5	0	10.7	0.060
P10	50.0	8.33	17.9	0.074
P11	37.5	8.33	21.8	0.262

^*p<0.05;

^**p<0.01

Comparison Between mPRIME Responses and Corresponding Items on SIPS

We finally looked at correlations between positive and negative responses on individual mPRIME items and the corresponding questions on the PRIME screen, as the screener is based on the positive symptom portion of the SIPS. For example, mPRIME item P1 (“I think that I have felt that there are odd or unusual things going on that I can’t explain”) corresponds with the first question of the Perplexity and Delusional Mood portion of the Positive Symptoms section of the SIPS (“Have you had the feeling that something odd is going on or that something is wrong that you can’t explain?”). Levels of correlations were measured as phi correlations (Pearson’s) and are reported in Table 4.

Table 4.

Correlations between mPRIME screen items and corresponding questions on SIPS

mPRIME Item	Correlation with Corresponding SIPS Question (ϕ)	Significance (p)
P1 (odd or unusual things)	0.143	0.056
P2 (predict the future)	0.185	0.016
P3 (controlling thoughts)	0.202	0.008
P4 (supersitions)	0.017	0.824
P5 (real or imagination)	0.079	0.289
P6 (read minds)	0.107	0.159
P7 (plans to hurt me)	0.158	0.038
P8 (special or supernatural gifts)	0.135	0.075
P9 (worsened functioning)	*
P10 (people talking)	0.148	0.053
P11 (thoughts out loud)	0.152	0.541
P12 (going mad)	*

^*no corresponding SIPS question for mPRIME item

DISCUSSION

In this study, our goal was to validate a slightly modified version of the PRIME screen in a non-clinical Kenyan population. When comparing mPRIME items to corresponding questions in the standard SIPS, some item-question pairs showed statistically significant (though relatively weak) correlations (e.g., mPRIME items P2, P3, P7) while the other items did not demonstrate significant correlation with SIPS questions. Additionally, the observed validity measurements (sensitivity 40%, specificity 64.8%, PPV 12.3%, NPV 89.7%) were lower than that obtained in previous study populations [20,21] and may have been due to a variety of reasons. Most importantly, as a non-clinical and non-help seeking cohort, the prevalence of illness in our sample was lower than that seen in other validation studies of psychosis screeners, including the PRIME screen. As such, the amount of expected positive SIPS assessments is much lower in relation to the overall study size, rendering sensitivity and predictive value data particularly weak in comparison to clinical cohorts. Of note, the suggested positive PRIME screen threshold of at least three ‘5’ responses or one ‘6’ response did yield the best overall measures on the ROC curve but the screener as a whole in our sample did not show robust validity. Interestingly, when compared to the SIPS, using mPRIME scores of ‘6’ only as a positive result reveals a positive predictive value of 9.4%; this, when coupled with an mPRIME prevalence of 45.5% on any item in our original Kenyan population sample study (which counted ‘6’ responses as positive) yields an estimated actual psychosis-risk symptom prevalence value of 4.2%--similar to general population estimates in a large-scale Finnish study of around 3% [29]. Careful examination of the validity measures for assessment tools in such non-clinical populations may provide further insights for general population assessment efforts.

Our data did show that when dividing the sample group into true positives, false negatives, and false positives, a picture emerged in which those individuals who scored the highest on the mPRIME tended to be the ones that were more likely to be true positives. While the false positive individuals on the mPRIME expectedly scored significantly higher on the majority of screener items compared to the false negative group, on average the false negative group scored ‘2’ or below on all mPRIME items— indicating more than just a simple level of item disagreement. This discordance between positive assessment on the SIPS and such negative results on the mPRIME may be suggestive of inherent differences between self and observer-based assessment tools for non-clinical/non-help seeking populations. A recent study by Kobayashi and colleagues found that in comparison to young individuals with depressive symptomatology, those with attenuated psychotic symptoms were far less likely to report having problems and were not very likely to seek help on their own [30]. In a study specifically looking at non-help seeking individuals, the at-risk youth identified by the SIPS may represent a special type of risk profile that is different from an individual who seeks out or is brought to clinical attention earlier on. Additionally, there may be something about the way in which question items are approached by different individuals based on level of functioning/prodromal illness. For many of the mPRIME items, questions regarding “special” abilities, prediction of the future, or presence of “superstitions” may be looked upon as somewhat normative depending on the population or the setting; as such, high scores on such items from a cohort that is not help-seeking may be less informative (the false positive group in our study in fact scored highest among the entire sample on the mPRIME item of “special or supernatural gifts”). Indeed, what is considered a sign of definite or possible pathology verses a characteristic of a culturally normative worldview or outlook varies between groups and has been found to be variable both within Africa as well as compared to cultures outside the continent [31-33]. However, an individual who is at risk for possible psychosis may not approach such questions as openly and may take a more concrete view of them. A study from 2010 looked at the relationship between different individual factors and length of time of untreated illness in young populations and found an inverse relationship between level of premorbid adjustment and length of time before coming to clinical attention; conversely, a direct relationship was seen between negative symptom level and time to clinical attention [34]. Things such as low premorbid intelligence, lack of abstraction ability, and increased negativism in cognition may influence the way in which an individual takes a screening assessment. More research is necessary to clarify these potential relationships in the data.

Limitations of our study include the fact that the original PRIME was not used for the validation secondary to concerns regarding cultural understanding of two of the screening question items. However, the two questions that were removed from the original PRIME for analysis did not have any analogous items in the SIPS for comparison. Group size in our sample is also a limitation. While the overall number of participants was not insignificant, the number of individuals who were found to be positive on the SIPS (20) was somewhat small and may have consequently affected the overall quantitative measures of the study. An even larger sample size with increased opportunity to pick up potential prodromal pathology in the population would improve the power of our calculations and observations. An additional limitation was the fact that only one patient was used for inter-rater reliability analysis for the SIPS by our raters; using more patients for inter-rater reliability measurements and calibration would have been ideal. There is also the possibility that SIPS ratings may not have been completely precisely measured by raters as reliability of raters’ assessments was not fully established for this particular group of raters. However, inter-rater reliability analysis from a group similar in all relevant aspects showed a good level of agreement on positive SIPS items—indicating that calibration amongst the raters was likely achieved. Additional work with rater staff to improve even further the agreement level on SIPS items would only serve to improve the potential robustness of the observer evaluations. That being said, this study takes a step in evaluating the wider usability of the PRIME and similar screeners in non-clinical populations and serves as a starting point for further development of these kinds of tools for greater implementation in young populations as part of potential primary prevention strategies.