Item-level endorsement on the Prodromal Questionnaire in a large non-clinical sample

Abstract

Background:

Brief questionnaires, such as the Prodromal Questionnaire (PQ) positive scale, have been used to pre-screen individuals who may be at clinical high-risk (CHR) for psychosis. Despite the apparent utility of the PQ, few studies have examined response styles in non-clinical settings, which this study aimed to assess.

Methods:

Response frequencies were examined for PQ positive subscale items in 3584 students (ages 18–35) from a nationally representative, semi-public undergraduate institution. Highly endorsed items were evaluated further in conjunction with established cutoffs and associated symptom ratings from the Structured Interview for Psychosis-risk Syndromes (SIPS) in a smaller subset of participants (n = 162). Positive subscale and distressing item responses were also evaluated by gender, race, and ethnicity using measurement invariance analyses and by comparing the relative proportion of individuals above established cutoffs.

Results:

Fifteen symptoms were endorsed by over 20 % of the sample with as high as 71 % of respondents endorsing them. Responses to 12 of these items were not associated with ratings on the SIPS. The PQ functioned similarly across demographic characteristics with strong evidence found for gender and race invariance across items and strong ethnicity invariance and partial invariance for positive subscale items and distressing items, respectively.

Conclusions:

These findings suggest that a commonly used psychosis-risk questionnaire may not be appropriate for non-clinical samples, with the possibility of high false positive rates of those at CHR for psychosis. Future large-scale epidemiological studies should evaluate if psychosis-risk screeners can be improved to identify CHR individuals in community settings.

1. Introduction

Psychotic disorders, such as schizophrenia, are associated with substantial emotional distress and socio-occupational impairment (Hegarty et al., 1994; Salomon et al., 2012). Intervention early in the course of psychosis has been found to positively influence long-term illness trajectories, while delays in receiving sufficient treatment are associated with negative cognitive and functional outcomes and poorer treatment responses (Addington et al., 2004; Cuesta et al., 2012; Correll et al., 2018; Perkins et al., 2005). As such, the identification of individuals experiencing risk-related symptoms for psychosis is a priority with widely available screeners being an essential tool in early identification and treatment (Ellman et al., 2020; Schiffman et al., 2019).

To date, semi-structured interviews such as the Structured Interview for Psychosis-Risk Syndromes (SIPS) and the Comprehensive Assessment of At-Risk Mental States (CAARMS) have primarily been used to identify individuals at clinical high risk (CHR) or ultra-high risk (UHR) for psychosis based on the presence of attenuated psychotic symptoms, brief episodes of full psychosis, or genetic risk accompanied by a recent decline in functioning (Miller et al., 2003; Yung et al., 2005). The majority of CHR/UHR individuals fall into the first category with attenuated psychotic symptoms that do not meet the severity required for full diagnosis of a psychotic disorder (Miller et al., 2003). Unfortunately, specialty assessment clinics in the United States are rare, and risk interview administration requires substantial training for assessors and is resource intensive. Additionally, the assessment of CHR individuals in non-help seeking community members primarily through interview format would overload clinics due to low base-rates of risk (Kelleher et al., 2012b). Therefore, valid, brief screening tools, such as questionnaires, are needed to determine who would benefit from further clinical assessments (Ellman et al., 2020; Gold et al., 2020; Schiffman et al., 2019).

Several self-report measures have been developed to quickly assess for the presence psychosis-risk symptoms (Heinimaa et al., 2003; Loewy et al., 2005; Miller et al., 2004). Of these, the Prodromal Questionnaire (PQ) has been the most widely used and assesses general endorsement of psychotic-like experiences and associated distress (Kline and Schiffman, 2014; Savill et al., 2018). Endorsement of items on the PQ positive subscale, specifically, has been consistently found to be associated with receiving a concurrent CHR status by full clinical interview in adolescent psychiatry and other general mental health clinics (Loewy et al., 2007; Savill et al., 2018).

Although the PQ positive subscale has shown utility at identifying individuals who may be CHR, little work has examined the PQ in the context of non-clinical populations such as with undergraduate students. Understanding the prevalence of self-reported psychosis-risk symptoms in non-clinical settings is necessary for establishing accurate cutoff scores and tailoring abbreviated versions given that presentations may differ from help-seeking individuals. Proper identification of CHR individuals in non-help-seeking populations may help to engage individuals in health services who may not have otherwise encountered them, consequently reducing durations of untreated psychosis. One study which examined the positive subscale of the PQ in a sample of University of California, Los Angeles undergraduate students found that 43 % of the sample endorsed 8 or more positive items, however only 18 % endorsed over 14 items (Loewy et al., 2007). Additionally, only 2 % of students endorsed 8 or more items as distressing, which is commensurate with the prevalence of psychosis in the general population. By eliminating the commonly endorsed symptoms of this UCLA undergraduate sample, the same research group also later developed an abbreviated 21-item PQ-B which has since been widely used in clinical and research settings (Loewy et al., 2011). Items which are endorsed by a large proportion of non-help-seeking samples are thought to capture more normative experience rather than psychosis-risk symptomatology. As such, studying item-by-item response frequencies in non-clinical samples may help to better tailor existing measures and improve sensitivity and specificity. While the PQ and its abbreviated versions have shown predictive validity in specialty care and general mental health clinics for the identification of potential CHR individuals, it is important to confirm the response characteristics in other non-clinical settings before administering it to the general population (Savill et al., 2018).

Findings are largely mixed regarding gender and psychotic-like experience endorsement in non-clinical populations (Calkins et al., 2014; Kelleher et al., 2012a; Johns et al., 2004; Dhossche et al., 2002; Ndetei et al., 2012; Armando et al., 2012). Studies examining psychotic-like experiences in the context of race and ethnicity commonly find that individuals identify as Black or African American endorse more symptoms and endorse them as distressing more frequently than White or Latino/a individuals, which may be explained by differential experience of discrimination (Anglin et al., 2016; Anglin and Lui, 2021; Anglin et al., 2021). Other research indicates that White and Hispanic individuals endorsed PQ-B symptoms more frequently than Asian or multiracial individuals; however, this relationship is reversed when considering level of distress (Cicero et al., 2019). Prior to validly administering psychosis-risk screeners in community samples and comparing endorsement by demographic characteristics, it is necessary to first demonstrate that the items function similarly across gender, race, and ethnicity and thus demonstrate measurement invariance. Gender and ethnoracial invariance have been found using the PQ-B in studies of undergraduates, adolescents, and school-aged children (Cicero et al., 2019; Fonseca-Pedrero et al., 2018; Lång et al., 2020; Karcher et al., 2018). However, no studies, to our knowledge, have examined measurement invariance with the full PQ positive subscale thus limiting confidence in findings associated with gender, race, and ethnicity.

Therefore, we examined response frequencies on the PQ positive subscale in a large and diverse undergraduate sample and investigated item relationships with ratings by clinician-administered symptom interviews. We hypothesized that the most commonly endorsed PQ items, which putatively represent more normative experience, would be unrelated to corresponding interview ratings. Additionally, we tested for measurement invariance (a method often used to examine the behavior of items across certain respondent characteristics) between (a) males and females, (b) between individuals identifying as Asian or Asian American, Black or African American, White, or multiracial, and (c) Hispanic and non-Hispanic individuals. Given previous findings with the abbreviated PQ-B, we hypothesized that the scale would function similarly across groups.

2. Materials and methods

2.1. Sample and procedure

This study was approved by the Institutional Review Board at Temple University. Students attending a large urban state-funded university in the Philadelphia area over the age of 18 were invited to participate in this study through an online subject portal. The university maintains a diverse student body in terms of race, ethnicity, student income level, and the number of admitted transfer students from community colleges. Demographic diversity is also supported by the school's relatively high acceptance rates and its omittance of mandatory standardizing testing as part of the application process. All participants were recruited and selected from a non-clinical undergraduate setting; however, many participants did endorse general psychiatric symptoms and receiving associated treatment similar to other community samples.

Participants received course credit for their participation. Three thousand six hundred sixteen students completed consent and were administered the PQ and other questionnaires on a lab computer; however, one participant did not complete a portion of the positive symptom subscale and was removed from further analyses. Additionally, subjects over the age of 35 (n = 18) and subjects with missing data on age (n = 13) were removed from further analyses. Participants with incomplete PQ data did not noticeably alter the results whether included or not in the analyses. Therefore, 3584 participants were included in the primary analyses below. Additionally, a subset of participants (n = 162) who endorsed 8 or more distressing PQ positive subscale items or 3 or less distressing items and 8 or fewer total PQ positive items (based on the means at the time of recruitment) participated in a second small pilot study where they were interviewed with the SIPS. This subset did not differ from the full sample in any of the primary demographic variables collected (age: U = 121,962, p = 0.09; gender: X²(1, N = 3584) = 1.46, p = 0.23; race: X²(6, N = 3584) = 9.22, p = 0.42; ethnicity: X²(1, N = 3584) = 0.17, p = 0.68) (Table 1).

Table 1.

Demographics.

	Overall Sample (n = 3584)
Age, mean (SD) [range]	20.25 (2.41) [18–35]
Female, n (%)	2653 (74.02)
Race, n (%)1
White	2104 (59.00)
Black or African American	556 (15.6)
Asian or Asian American	526 (14.70)
Native American or Alaskan Native	9 (0.25)
Native Hawaiian or other Pacific Islander	6 (0.17)
More than one race	172 (4.82)
Unknown	195 (5.47)
Ethnicity, n (%)
Hispanic	215 (6.00)
Non-Hispanic	3369 (94.00)

¹Note: N = 16 chose not to respond.

2.2. Measures

2.2.1. The Prodromal Questionnaire (PQ)

The PQ is a self-report questionnaire designed to identify individuals who may be at risk for psychosis and is organized into positive, negative, disorganized, and general symptom subscales (Loewy et al., 2007). Participants completed the 45-item positive subscale and were asked to indicate how frequently they have experienced each symptom in the last month in the absence of any substance use. For the purposes of this study, we only examined yes/no endorsement. Respondents were also asked to indicate the presence of distress if they endorsed a given item.

Cutoff scores of 18 positive items and/or 8 positive items endorsed as distressing were used in the present investigation and have been associated with a concurrent CHR status on the SIPS with high sensitivity and moderate specificity in mental health help-seeking samples (Chen et al., 2014, 2016; Ising et al., 2012; Loewy et al., 2012; Okewole et al., 2015).

2.2.2. The Structured Interview for Psychosis-Risk Syndromes (SIPS)

The SIPS is a semi-structured interview intended to identify individuals who are at elevated risk for transition to a psychotic disorder (Miller et al., 2003). Individuals are rated on a scale of 0 to 6 where scores of 3 and above represent clinically relevant symptomatology and may warrant CHR or psychotic spectrum diagnoses. Interviews were conducted by SIPS-certified doctoral students and lab staff, and ratings were confirmed in consensus review with at least two advanced interviewers and a Ph.D.-level supervisor.

2.3. Statistical analysis

Endorsement frequencies were calculated for (a) each PQ positive symptom experienced and (b) each PQ positive symptom experienced as distressing. Items which were endorsed by over 20 % of the sample were also examined in relation to the SIPS clinical interview to test the hypothesis that highly endorsed items would be unrelated to interview ratings. Given that only a small number of individuals were identified as CHR by the SIPS (n = 18), CHR status was not included in analyses. Chi-square tests, positive predictive values, and measures of sensitivity and specificity were calculated by comparing PQ item endorsement with the proportion of individuals rated a three or higher on the corresponding SIPS symptom domain. Additionally, the proportion of individuals above established clinical PQ cutoffs were calculated using the full PQ positive subscale and after removing items endorsed by over 20 % of the sample. Twenty percent was chosen for exploratory purposes given expected rates of psychotic spectrum disorders and psychosis-risk symptoms in the general population (Fusar-Poli et al., 2012; Moreno-Küstner et al., 2018; Perälä et al., 2007). A clinical cutoff score on the PQ of 18 positive items was selected given previous validation studies in general mental health clinics (Ising et al., 2012; Loewy et al., 2012; Savill et al., 2018). Additionally, in line with a previous study which evaluated the PQ in a college sample, a cutoff of 8 distressing items was used (Loewy et al., 2007).

Measurement invariance was also examined in the context of gender, race, and ethnicity for positive subscale items and distressing items. Measurement invariance analyses investigate whether questions function similarly across individuals despite the presence of other variables (van de Schoot et al., 2012). This was conducted using the Lavaan package in R and by estimating a series of consecutively more restrictive confirmatory factor analyses (CFAs) using a single factor and the demographic variable as the grouping variable (Rosseel, 2012). Single factor CFAs were chosen given past research demonstrating good fit for the positive subscale which was not better explained by other multigroup approaches (Azis et al., 2021). Weighted least square mean and variance adjusted (WLSMV) estimators were to account for non-continuous binary indicators (Brown, 2015). A configural invariance model without any equality constraints was tested first to examine if a complementary latent structure existed within each demographic grouping. As described by Pagliaccio et al. (2016), metric, scalar, and strict invariance were then examined by iteratively introducing additional restrictions by fixing item loadings, thresholds, and residual item variances, respectively. Goodness of fit was evaluated according to thresholds described by Hu and Bentler (comparative fit index or CFI > 0.95, root mean square error of approximation or RMSEA≤0.06, RMSEA upper CI bound≤0.08; 1999). Non-invariance was identified in constrained models which demonstrated worse model fit compared to the prior models by examining change in CFI and RMSEA. More restrictive models which demonstrated decrease in CFI of less than or equal to 0.01 and an increase in RMSEA of less than or equal to 0.015 were considered not significantly different from the less constrained model (Cheung and Rensvold, 2002; Chen, 2007). Chi-squared tests used to examine relative model fit were not examined given that they may be less valid when used with sample sizes above n = 300 (Brannick, 1995). For models demonstrating poorer fit in comparison to the previous less-constrained model, significant factor loadings and/or thresholds were released from constraints and the models were re-examined for partial invariance. For model factor loadings and thresholds, see Supplementary Materials.

The relationships between gender, race, and ethnicity and clinical cutoffs were also evaluated for both the primary and distressing items using chi-square tests. Subsets of the sample were excluded from measurement invariance analyses or cutoff comparisons if they did not have sufficient variance across items (Native American/Alaska Native, n = 9; Native Hawaiian/Other Pacific Islander, n = 6). For the full frequency data by gender, race, and ethnicity, see the Supplementary Materials.

3. Results

Fifteen items were endorsed by 20 % or more of participants in the sample (Table 2 and Fig. 1). Eight items were endorsed by over 30 % of the sample including 3. I have had difficulty organizing my thoughts or finding the right words which was endorsed by nearly 72 % of individuals. This item was also the most highly reported distressing item with nearly 35 % of the sample indicating distress (Table 2 and Fig. 2). See Supplementary Table for the full list of item frequencies.

Table 2.

Commonly endorsed items on the PQ Positive Symptom Subscale.

No.	Frequency item was endorsed (%)	Frequency Item Endorsed as Distressing (%)	Item
3	71.62	34.79	I have had difficulty organizing my thoughts or finding the right words.
2	64.4	19.45	The passage of time has felt unnaturally faster or slower than usual.
37	55.75	27.12	I have had trouble focusing on one thought at a time.
23	54.35	7.06	I have wandered off the topic or rambled on too much when I was speaking.
8	53.32	4.19	I seemed to live through events exactly as they happened before (déjà vu).
38	41.49	23.86	I have felt that other people were watching me or talking about me.
49	33.23	24.02	My thinking has felt confused, muddled, or disturbed in some way.
77	32.76	24.55	I have been concerned that my closest friends and co-workers were not really loyal or trustworthy.
55	27.15	20.79	I have been worried that something may be wrong with my mind.
35	24.72	5.39	I have had superstitious thoughts.
90	24.30	10.04	People have found it hard to understand what I say.
60	24.11	12.70	I have experienced unusual bodily sensations such as tingling, pulling, pressure, aches, burning, cold, numbness, shooting pains, vibrations or electricity.
30	23.02	1.59	I have thought that I am very important or have abilities that are out of the ordinary.
9	22.85	1.09	I have smelled or tasted things that other people didn't notice.
61	20.87	2.20	I have thought about beliefs that other people would find unusual or bizarre.

Note: Full list of item frequencies can be found in Supplementary Material.

Fig. 1.

Prodromal Questionnaire Positive Subscale endorsement frequencies.

Fig. 2.

Prodromal Questionnaire Positive Subscale distressing item frequencies.

Almost 14 % of the sample was above the cutoff of 18 endorsed items (Fig. 3). Removing items endorsed by over 20 % of the sample reduced the proportions of individuals above the cutoff to 1.53 %. Additionally, over 15 % of respondents reached a cutoff of 8 or more items endorsed as distressing, and this was reduced to 7.09 % after removing most commonly endorsed items.

Fig. 3.

Proportion of sample above clinical cutoffs after removing items endorsed by over 20 % of individuals.

Overall, commonly endorsed items were not found to be significantly associated with clinically relevant ratings on the corresponding SIPS symptom domain (Table 3). Still, three commonly endorsed items were associated with SIPS symptoms including 2. The passage of time has felt unnaturally faster or slower than usual 9. I have smelled or tasted things that other people didn't notice, and 30. I have thought that I am very important or have abilities that are out of the ordinary. Despite showing association with the SIPS unusual thought content domain, item 2 demonstrated a relatively low positive predictive value (18.64 %) and specificity level (30.94 %).

Table 3.

Commonly endorsed item association with the SIPS.

Item	Corresponding SIPS Symptom Domain	SIPS+ / PQ+	SIPS− / PQ+	Chi-square test	Sensitivity	Specificity	Positive Predictive Value
		SIPS+ / PQ−	SIPS− / PQ−
3. I have had difficulty organizing my thoughts or finding the right words.	Disorganized communication	7	127	X2(1, N = 162) =0.53, p = 0.47	100	18.06	5.22
		0	28
2. The passage of time has felt unnaturally faster or slower than usual.	Unusual thought content / Delusional ideas	22	96	X2(1,N = 162) = 5.77, p = 0.02	95.65	30.94	18.64
		1	43
37. I have had trouble focusing on one thought at a time.	Disorganized communication	5	115	X2(1,N = 162) < 0.001, p = 1	71.43	25.81	4.17
		2	40
23. I have wandered off the topic or rambled on too much when I was speaking.	Disorganized communication	6	103	X2(1,N = 162) =0.42, p = 0.52	85.71	33.55	5.5
		1	52
8. I seemed to live through events exactly as they happened before (déjà vu).	Unusual thought content / Delusional ideas	18	92	X2(1,N = 162) =0.82, p = 0.36	78.26	33.81	16.36
		5	47
38. I have felt that other people were watching me or talking about me.	Suspiciousness/ Persecutory ideas	20	74	X2(1,N = 162) = 3.66, p = 0.06	76.92	45.59	21.28
		6	62
49. My thinking has felt confused, muddled, or disturbed in some way.	Disorganized communication	5	98	X2(1,N = 162) = 0.002, p = 0.97	71.43	36.77	4.85
		2	57
77. I have been concerned that my closest friends and co-workers were not really loyal or trustworthy.	Suspiciousness/ Persecutory ideas	17	69	X2(1,N = 162) = 1.34, p = 0.25	65.38	49.26	19.77
		9	67
55. I have been worried that something may be wrong with my mind.	Disorganized communication	5	85	X2(1,N = 162) = 0.23, p = 0.63	71.43	45.16	5.56
		2	70
35. I have had superstitious thoughts.	Unusual thought content / Delusional ideas	9	45	X2(1,N = 162) = 0.16, p = 0.69	39.13	67.63	16.67
		14	94
90. People have found it hard to understand what I say.	Disorganized communication	3	61	X2(1,N = 162) < 0.001, p = 1	42.86	60.65	4.69
		4	94
60. I have experienced unusual bodily sensations such as tingling, pulling, pressure, aches, burning, cold, numbness, shooting pains, vibrations or electricity.	Perceptual abnormalities / Hallucinations	8	49	X2(1,N = 162) = 2.27, p = 0.13	57.14	66.89	14.04
		6	99
30. I have thought that I am very important or have abilities that are out of the ordinary.	Grandiose ideas	5	52	X2(1,N = 162) = 4.33, p = 0.04	83.33	66.67	8.77
		1	104
9. I have smelled or tasted things that other people didn't notice.	Perceptual abnormalities / Hallucinations	10	41	X2(1,N = 162) = 9.40, p = 0.002	71.43	72.30	19.61
		4	107
61. I have thought about beliefs that other people would find unusual or bizarre.	Unusual thought content / Delusional ideas	13	53	X2(1,N = 162) = 2.06, p = 0.15	56.52	61.87	19.70
		10	86

Note: Chi-squares testing relationship between the proportion of PQ item endorsement and a clinically relevant rating of 3 or above on the corresponding SIPS positive symptom.

Gender, race, and ethnicity invariance findings are displayed in Tables 4, 5, and 6. The unidimensional model fit the data well for the primary positive items and distressing items (Endorsed_Gender: CFI = 0.979, RMSEA = 0.030, RMSEA CI = 0.029, 0.031; Distressing_Gender: CFI = 0.985, RMSEA = 0.019, RMSEA CI = 0.018, 0.021; Endorsed_Race: CFI = 0.983, RMSEA = 0.028, RMSEA CI = 0.027, 0.029; Distressing_Race: CFI = 0.978, RMSEA = 0.026, RMSEA CI = 0.025, 0.028; Endorsed_Ethnicity: CFI = 0.979, RMSEA = 0.030, RMSEA CI = 0.029, 0.031; Distressing_Ethnicity: CFI = 0.981, RMSEA = 0.026, RMSEA CI = 0.025, 0.027). The configural, metric, scalar, and strict invariance models for the primary endorsed items showed good fit for each demographic variable according to conventions described by Hu and Bentler (1999). Additionally, the changes in CFI and RMSEA between each model and its successively restrictive model were non-significant providing evidence for strict demographic invariance.

Table 4.

Measurement invariance model fit statistics for the Prodromal Questionnaire positive subscale and gender.

Model	X2 (df)	CFI	RMSEA (90 % CI)	Change in CFI	Change in RMSEA
Endorsed Items
Configural	4848.23 (1890)	0.979	0.030 (0.029, 0.031)	NA	NA
Metric	5113.75 (1934)	0.978	0.030 (0.029, 0.031)	−0.001	0.000
Scalar	5391.11 (1978)	0.976	0.031 (0.030, 0.032)	−0.002	0.001
Strict	5127.39 (1933)	0.978	0.030 (0.029, 0.031)	0.002	−0.001
Distressing Items
Configural	3168.19 (1890)	0.985	0.019 (0.018, 0.021)	NA	NA
Metric	3640.72 (1934)	0.980	0.022 (0.021, 0.023)	−0.005	0.003
Scalar	3750.28 (1978)	0.979	0.022 (0.021, 0.023)	−0.001	0.000
Strict	3464.62 (1933)	0.982	0.021 (0.020, 0.022)	0.003	−0.001

Note: Abbreviations: X² = chi square, df = degrees of freedom, CFI = comparative fit index, RMSEA = root mean squared error of approximation, CI = confidence interval; goodness of fit thresholds: CFI > 0.95, RMSEA ≤0.06, RMSEA upper CI bound ≤0.08, change in CFI decrease ≤0.010, change in RMSEA increase ≤0.015. Change in CFI and RMSEA derived by taking the difference with the previous less restricted model.

Table 5.

Measurement invariance model fit statistics for the Prodromal Questionnaire positive subscale and race.

Model	X2 (df)	CFI	RMSEA (90 % CI)	Change in CFI	Change in RMSEA
Endorsed Items
Configural	6211.37 (3780)	0.983	0.028 (0.027, 0.029)	NA	NA
Metric	6899.83 (3912)	0.979	0.030 (0.029, 0.031)	−0.004	0.002
Scalar	7300.10 (4044)	0.977	0.031 (0.030, 0.032)	−0.002	0.001
Strict	6691.41 (3909)	0.980	0.029 (0.028, 0.030)	0.003	−0.002
Distressing Items
Configural	5912.92 (3780)	0.978	0.026 (0.025, 0.028)	NA	NA
Metric	7380.32 (3912)	0.964	0.033 (0.031, 0.034)	−0.014	0.007
Scalar	7599.56 (4044)	0.963	0.032 (0.031, 0.034)	−0.001	−0.001
Strict	6560.74 (3909)	0.972	0.028 (0.027, 0.030)	0.009	−0.004

Note: Abbreviations: X² = chi square, df = degrees of freedom, CFI = comparative fit index, RMSEA = root mean squared error of approximation, CI = confidence interval; goodness of fit thresholds: CFI > 0.95, RMSEA ≤0.06, RMSEA upper CI bound ≤0.08, change in CFI decrease ≤0.010, change in RMSEA increase ≤0.015. Change in CFI and RMSEA derived by taking the difference with the previous less restricted model.

Table 6.

Measurement invariance model fit statistics for the Prodromal Questionnaire positive subscale and ethnicity.

Model	X2 (df)	CFI	RMSEA (90 % CI)	Change in CFI	Change in RMSEA
Endorsed Items
Configural	4888.74 (1890)	0.979	0.030 (0.029, 0.031)	NA	NA
Metric	5114.68 (1934)	0.978	0.030 (0.029, 0.031)	−0.001	0.000
Scalar	5159.45 (1978)	0.978	0.030 (0.029, 0.031)	0.000	0.000
Strict	5007.02 (1933)	0.979	0.030 (0.029, 0.031)	0.001	0.000
Distressing Items
Configural	4236.64 (1890)	0.981	0.026 (0.025, 0.027)	NA	NA
Metric	8522.85 (1934)	0.948	0.044 (0.043, 0.045)	−0.033	0.018
Partial Metrica	4310.07 (1904)	0.981	0.027 (0.026, 0.028)	0.000	0.001
Partial Scalar	5254.97 (1948)	0.974	0.031 (0.030, 0.032)	−0.007	0.004
Partial Strict	4247.62 (1903)	0.981	0.026 (0.025, 0.027)	0.007	−0.005

Note: Abbreviations: X² = chi square, df = degrees of freedom, CFI = comparative fit index, RMSEA = root mean squared error of approximation, CI = confidence interval; goodness of fit thresholds: CFI > 0.95, RMSEA ≤0.06, RMSEA upper CI bound ≤0.08, change in CFI decrease ≤0.010, change in RMSEA increase ≤0.015. Change in CFI and RMSEA derived by taking the difference with the previous less restricted model.

^aCompared to configural model.

All models and nested comparisons for responses to distressing items by gender and race showed good indications of fit and evidence for strong invariance. The CFI decreased by 0.014 from the configural to metric models for distressing items examined by race; however, the change in RMSEA was low supporting metric invariance. The metric model with constrained factor loadings for ethnicity was significantly different than the base configural model (ΔCFI = −0.033, ΔRMSEA = 0.018). Thirty of the 45 configural factor item loadings were found to be significant and released from the metrical model to achieve partial metric (ΔCFI = −0.00, ΔRMSEA = 0.001), partial scalar (ΔCFI = −0.007, ΔRMSEA = 0.004), and partial strict invariance (ΔCFI = 0.007, ΔRMSEA = −0.005). Items surveying unusual perceptual experiences generally performed well irrespective of ethnicity, although items asking about experiences with persecutory ideation and disorganized performed poorly (see Supplementary Materials).

Twenty items were endorsed by 20 % or more among male participants, while only 13 items were endorsed by 20 % or more among female participants. Six items were endorsed as distressing in 20 % or more of males, and 7 were endorsed as distressing in over 20 % of females. The number of items endorsed by over 20 % of individuals identifying as Asian or Asian American, Black or African American, White, or more than one race were 20, 18, 17, and 23 respectively and 6, 4, 6, and 7 for distressing items. A similar number of items were endorsed by over 20 % of the Hispanic and Latino/a and non-Hispanic and Latino/a sample (Endorsed items: 16 and 15, respectively; Distressing items: 6 and 6, respectively). While the rates of endorsement differed somewhat across gender and race, items associated with disorganized thinking and unusual thoughts were highly endorsed across all individuals. See Supplementary Material for full frequencies by demographics.

A significantly larger proportion of males reached a cutoff of 18 or more endorsed items than females (x⁻_male = 16.97 %, x⁻_female =12.85 %, X²(1, N = 3584) = 9.41, p = 0.002; Fig. 4); however, the proportions of females and males above the 8 distressing item cutoffs were not significantly different from one another (x⁻_male = 13.64 %, x⁻_female = 16.02 %, X²(1, N = 3584) = 2.81, p = 0.09). Additionally, within females a larger proportion of individuals reached a cutoff of 8 distressing symptoms than they did a cutoff of 18 endorsed symptoms (x⁻_distressing, = 16.02 %, x⁻_endorsed = 12.85 %, X²(1, N = 5306) = 10.51, p = 0.001). Although fewer males reached a cutoff of 8 distressing items than the cutoff of 18 endorsed items, this decrease was non-significant (x⁻_distressing, = 13.64 %, x⁻_endorsed = 16.97 %, X²(1, N = 1862) = 3.73, p = 0.053). The proportion of individuals above clinical cutoffs were not significantly different by race and ethnicity with the exception of individuals identifying as white who reached the distressing item cutoff more frequently than they did the endorsed item cutoff (x⁻_distressing, = 16.25 %, x⁻_endorsed = 13.36 %, X²(1, N = 4208) = 7, p = 0.009) (Figs. 5 and 6).

Fig. 4.

Proportion of sample above established clinical cutoffs by gender

n.s. = not significant, *0.01 < p < 0.05, **0.001 < p < 0.01.

Fig. 5.

Proportion of sample above established clinic cutoffs by race

The proportions of individuals above cutoffs across race were also examined and did not differ for endorsed items (X² (3, 3358) = 4.47, p = 0.21) and distressing items (X² (3, 2924) = 7.19, p = 0.07.

Fig. 6.

Proportion of sample above established clinical cutoffs by ethnicity.

4. Discussion

This study identified a series of items on the positive subscale of the PQ which were highly endorsed in an undergraduate sample, raising the possibility that the PQ may operate uniquely in non-clinical settings. Certain items were endorsed by as much as 72 % of the sample. Many of these overly endorsed items were not associated with SIPS ratings, affirming recent research suggesting that psychosis-risk items on questionnaires may be less valid among non-help-seeking populations (Ellman et al., 2020). Measurement invariance analyses indicated that PQ endorsed items performed well across gender, race and ethnicity and distressing items performed well for gender and race. Only partial metric invariance was achieved between Hispanic and non-Hispanic individuals. Gender differences were observed, however, with item response frequencies and the percentage of individuals above clinical cutoffs. Overall, results from this study suggest that psychosis-risk questionnaires, such as the PQ, may not operate similarly in non-clinical populations as in psychiatric settings; therefore, to improve screening additional studies are needed in community settings.

Findings indicate that a total of 15 items were endorsed by over 20 % of participants. Fourteen percent of respondents reached a cutoff of 18 symptoms and as 15.4 % endorsed 8 or more distressing items. Interestingly, a similar study conducted in UCLA college students found that only 2 % of the sample endorsed over 8 items as distressing (Loewy et al., 2007). It is possible that this discrepancy may be due to demographic differences between the samples, with the current sample being much more representative of the US population than other institutions in terms of socioeconomic, ethnoracial, and achievement. Particularly, the UCLA sample's largest racial groupings were Asian (43 %), Caucasian (24 %), and Hispanic or Latino/a (14 %), whereas our sample's largest proportions were White (59 %), Black or African American (16 %), and Asian or Asian American (15 %).

Participants who experienced commonly endorsed items (i.e., items experienced by over 20 % of the sample) were not found to be more likely to receive a clinically relevant risk rating on the associated SIPS symptom domain in 12 out of the 15 items. These findings suggest that a subset of questions may represent normative experience and may contribute to potential false positives if used to screen for CHR in non-clinical settings. Previous research suggests the specificity of the PQ positive subscale may be as low as 40 % in general mental health clinics (Savill et al., 2018). Screening individuals from the greater community will likely increase the number of high-risk individuals identified; however, it will also elevate the number of false positive cases. Removing commonly endorsed items such as the ones identified in this study may remedy this problem and reduce burden put on early psychosis intervention clinics.

A 21-item version of the PQ, the Prodromal Questionnaire-Brief (PQ-B) was developed after removing frequently endorsed items endorsed in a UCLA undergraduate sample (Loewy et al., 2011). This abbreviated questionnaire has shown high sensitivity in identifying CHR individuals screened with full clinical interviews, although specificity has been low to moderate (Jarrett et al., 2012; Kline and Schiffman, 2014; Okewole et al., 2015). Of the items included on the PQ-B, 6 items which corresponded with the original PQ were endorsed by over 20 % of individuals in the current study, but not the original norming sample. Of these items, 54.35 % of participants in the current study experienced 23. I have wandered off the topic or rambled on too much when I was speaking, and 41.49 % experienced 38. I have felt that other people were watching me or talking about me. This discrepancy between studies indicates that there is potential variability in responding to psychosis-risk questionnaires that is essential to investigate further, in order for valid screeners to be used widely.

Our finding of strict gender and race invariance in the PQ positive subscale is in line with recent studies examining invariance in the PQ-B. Two undergraduate studies found good model fit and evidence for scalar invariance for total endorsed PQ-B items and partial scalar invariance for distressing items (Cicero et al., 2019; Lång et al., 2020). Karcher et al. (2018) similarly identified strict sex (notably, different from gender as examined in the present study) and race invariance in PQ-B items in a large community sample of school-aged children. Although partial ethnic strict variance was achieved for distressing items, 30 of the 45 item constraints were released indicating that the PQ positive subscale distressing items may not function similarly across Hispanic and non-Hispanic individuals and therefore, group comparisons should be interpreted cautiously. Particularly, we found that items designed to assess persecutory ideation and disorganized thinking performed poorly. This is, to our knowledge, the first study to examine measurement invariance in the full positive subscale of the PQ. Our findings suggests that PQ items behave similarly across gender, race, and ethnicity, although there is some evidence for differential response styles between Hispanic and non-Hispanic individuals when answering distressing items.

Our finding that male undergraduates endorsed more items and they reached clinical cutoffs more frequently is consistent with prior research indicating gender differences in psychotic disorders, as well as those at CHR (Cocchi et al., 2014; Mcgrath et al., n.d.). Past research has also found that males transition into psychosis at earlier ages than females, which may partially explain why our sample consisting of primarily college-age students had higher rates of males endorsing positive PQ symptoms. Additionally, our results lend support to further investigation of using differential scoring methods by gender. Males reached cutoffs of 18 endorsed items more commonly than females, and females more commonly crossed distressing item cutoffs than they did symptom endorsement cutoffs. Still, more work is needed examining gender differences and other individual factors in responding to psychosis-risk screeners (Ellman et al., 2020; Schiffman et al., 2019).

Despite previous research indicating different rates of psychotic-like experiences in ethnoracial groupings using the PQ-B, there were no racial or ethnic differences found in the proportion of individuals above clinical cutoffs in our sample (Anglin et al., 2016; Anglin et al., 2021; Cicero et al., 2019). Both the number of items endorsed by over 20 % of the sample and the content of these items were similar across ethnoracial groups. Removing these frequently endorsed items may allow for more targeted group comparisons by race.

While this study benefits from a large, diverse, representative sample, some limitations remain. Almost 75 % of the sample was female, limiting the interpretability of the primary study results. Still, commonly endorsed items and their relationship to clinical cutoffs were investigated by gender. This study also did not evaluate how commonly endorsed items influence sensitivity and specificity of the PQ for predicting clinical risk syndromes as identified by established psychosis risk interviews such as the SIPS and the CAARMS, nor did it follow participants longitudinally to track transition to full-blown psychosis. While this study evaluated college students within prime age periods for prodromal symptoms and psychosis onset, the PQ should also be evaluated in normative adolescent samples to determine if response characteristics vary by age (Beiser and Erickson, 1993; Wang et al., 2013). Vulnerable adolescents and young adults in school settings, such as those seeking support from school psychologists/counselors, may show higher levels of endorsement and therefore may benefit from using the PQ in its present form; however, this should be investigated further. While a variety of questionnaires tapping into other current psychiatric symptoms were collected in this study, examining the overlap between these symptoms was beyond the scope of this study. Future work involving these variables may help to better understand whether certain commonly endorsed PLEs are more suggestive of other psychiatric experiences such as anxiety or obsessive-compulsive symptoms as has been demonstrated in other research involving negative psychotic symptoms (Pierce et al., 2021). Importantly, future studies should also aim to sample directly from the general community to understand maximum generalizability, which we are currently doing in an ongoing multi-site study (Ellman et al., 2020).

5. Conclusions

To our knowledge, this is the first study to directly evaluate item-specific endorsement frequencies on the PQ positive subscale and to compare them with associated ratings by clinical interview. We identified a set of items in an undergraduate sample which were endorsed at a relatively high rate and were not associated with clinically meaningful symptoms on the SIPS. Additionally, gender-specific response styles indicated differently patterns of responding between males and females on some items. The PQ positive subscale generally performed well across gender, race, and ethnicity in terms of measurement invariance. Overall, these findings suggest that in order to reach the most individuals at CHR for psychosis, psychosis-risk questionnaires need to be validated in non-clinical samples and potentially different screening tools are necessary in clinical and non-clinical settings. Effective, brief and valid screening is needed, given the high cost, specialized training, and resources needed to conduct interview-based assessments; therefore, the findings from this study have potentially large implications for future efforts to improve identification and early intervention of those at CHR for psychosis (Ellman et al., 2020; Schiffman et al., 2019). Future work evaluating frequent items on the PQ should be conducted in other general community populations and also with adolescents to further investigate item-by-item contribution to risk prediction, as well as to include items other than positive symptoms to determine if additional constructs can improve predicting.