Preliminary support for using the Atypicality Scale from the Behavior Assessment System for Children, Second Edition, to screen for psychosis-spectrum disorders within a college counseling center

Abstract

Aim:

Evidence supports the use of brief psychosis-spectrum screeners for identifying individuals at risk for psychosis. Screening has not been well-studied in help-seeking college samples. This study investigated the use of the Behavior Assessment System for Children, Second Edition (BASC-2) Atypicality Scale as a psychosis-spectrum screening tool within a university counseling center.

Methods:

Atypicality scores from the BASC-2 were compared to interview-based assessment, the Structured Interview for Psychosis-risk Syndromes (SIPS), to explore associations across the measures and evaluate the scale’s ability to identify individuals who meet criteria for a psychosis-spectrum diagnosis.

Results:

Forty-three participants completed the BASC-2 and SIPS, and 23 were SIPS-positive. Compared to the SIPS-negative group, the SIPS-positive group had significantly higher Atypicality scores. Exploratory results indicated that Atypicality scores identified SIPS-positive individuals with an overall accuracy of 72% (78% sensitivity, 65% specificity).

Conclusion:

The Atypicality Scale may be an appropriate first-line psychosis-spectrum screening tool in college counseling centers.

Accurate identification of individuals experiencing emerging psychosis, including those at clinical high risk (CHR), is critical for initiating early treatment. University counseling centers offer an important setting for psychosis-risk screening in the U.S. given that 1) a large proportion (~40%) of young adults attend college (Snyder, de Brey, & Dillow, 2019), 2) these clinics serve young adults in the age range associated with highest risk for psychosis, 3) national trends show increases in students reporting mental health (MH) symptoms and seeking on-campus services (Lipson, Lattie, & Eisenberg, 2019; Oswalt et al., 2019), which indicates a growing need for appropriate triage and referral, 4) a large percentage of intakes to college clinics report psychosis-spectrum experiences (Thompson et al., 2020), and 5) early detection can potentially improve outcomes during a critical period of academic/social development (Schiffman & Dixon, 2011).

Given time pressures and challenges to engaging students in MH care, there is a strong need for efficient and accurate screening tools that can be disseminated in college settings. Although evidence supports psychosis-risk screens as a first step in risk assessment (Kline & Schiffman, 2014), these tools have been primarily developed and tested in specialized settings serving psychosis-specific referrals, or in non-clinical settings where the utility may be limited (van Os et al., 2009). Exploration of screening in age-appropriate, general MH settings is needed to bring early identification initiatives to scale.

The Behavior Assessment System for Children, Second Edition (BASC-2) Atypicality Scale has been shown to be a valid screen for CHR, holding promise for widespread adoption (Thompson et al., 2013). The BASC-2 college-age form is a reliable, developmentally-validated, and easy-to-administer questionnaire that is recommended for widespread use in college settings to identify MH concerns (Nowinski et al., 2008). The college-age form has not been evaluated as a psychosis-risk screen.

In this study, Atypicality scores obtained from a help-seeking university clinic sample were compared to interview-based assessment to investigate the scale’s ability to identify individuals meeting criteria for psychosis-spectrum diagnoses. We hypothesized that the Atypicality Scale would demonstrate acceptable predictive validity as a screening tool.

Method

Participants

Students who presented to a college MH center were eligible for the study based on endorsing potential psychosis-spectrum symptoms at intake (via self-report questionnaires or to their intake clinician). Forty-four students agreed to a psychosis-risk evaluation and were included in this sample (for full description, see Thompson et al., 2020).

Measures

The BASC-2 is a 185-item behavior checklist (Reynolds & Kamphaus, 2004). The Atypicality Scale consists of ten items designed to assess experiences linked to psychosis: odd behaviors, delusions, hallucinations (Table 1). The BASC-2 is validated through age 25, however, our sample included five older individuals (26–33 years). For this reason, we used raw scores (0–15) instead of age-normed T scores for our main analyses. For comparison, age-normed T scores were also explored in relation to psychosis-spectrum diagnoses.

TABLE 1.

Sample characteristics in relation to and across SIPS groups

	Full Sample [n = 43]	SIPS-Negative [n = 20]	SIPS-Positive [n = 23]	Independent T-test		Fisher’s exact test p
				t	p
Age (18–33 years): Mean (SD)	21.05 (3.27)	20.95 (3.66)	21.13 (2.97)	−0.18	.859
Gender: n (%)
Female	22 (51.2%)	11 (55.0%)	11 (47.8%)	---		n.s.
Male	13 (30.2%)	5 (25.0%)	8 (34.8%)	---		n.s.
Other	8 (18.6%)	4 (20.0%)	4 (17.4%)	---		n.s.
Race: n (%)
White	19 (44.2%)	10 (50.0%)	9 (39.1%)	---		n.s.
Black	8 (18.6%)	4 (20.0%)	4 (17.4%)	---		n.s.
Multiracial	8 (18.6%)	3 (15.0%)	5 (21.7%)	---		n.s.
Other or not reported	8 (18.6%)	3 (15.0%)	5 (21.7%)	---		n.s.
Atypicality Items	Correlation with SIPS Group	Frequency (%) Mean (SD)	Frequency (%) Mean (SD)	Independent T-test		Fisher’s exact test p
				t	p
1. Hears own name when alone	.51**	3 (15.0%) 0.30 (0.73)	15 (62.5%) 1.30 (0.97)	−3.85	< .001	.002
2. Unable to stop self from doing bad things	−.37*	8 (40.0%) 0.80 (1.01)	2 (8.7%) 0.17 (0.58)	2.46	.020	.028
3. Feels like people out to get them	.34*	11 (55.5%) 0.60 (0.60)	19 (82.6%) 1.04 (0.64)	−2.34	.024	.094
4. Believes someone else controls thoughts	.05	4 (20.0%) 0.25 (0.55)	7 (30.4%) 0.30 (0.47)	−0.35	.729	n.s.
5. Does things over and over and can’t stop	−.30*	14 (70.0%) 1.45 (1.23)	14 (60.9%) 0.83 (0.78)	1.95	.060	n.s.
6. Believes someone wants to hurt them	.13	4 (20.0%) 0.25 (0.55)	8 (34.8%) 0.39 (0.58)	−0.81	.421	n.s.
7. Hears things that others cannot	.36*	6 (30.0%) 0.45 (0.83)	17 (73.9%) 1.04 (0.77)	−2.44	.019	.006
8. When alone, feels like someone is watching	.64**	9 (45.0%) 0.50 (0.61)	22 (95.7%) 1.74 (0.86)	−5.36	< .001	< .001
9. Sees weird things	.52**	2 (10.0%) 0.10 (0.31)	14 (60.9%) 0.65 (0.57)	−4.01	< .001	.001
10. Hears voices in head that no one else hears	.34*	4 (20.0%) 0.30 (0.73)	14 (60.9%) 0.82 (0.73)	−2.29	.027	.012
Total raw score	.47**	--- 5.00 (3.15)	--- 8.26 (3.09)	−3.42	.001	---

Notes. SIPS: Structured interview for psychosis-risk syndromes. Individuals in the “other” gender category include students who identify as transgender or non-binary. Frequency refers to frequency of item endorsement; any ratings above “0” (false or never) were considered endorsements. Atypicality items 1 and 2 were scored “0” for “false”, and “2” for “true”. Phi coefficient was used for two dichotomous variables (i.e. SIPS group [positive/negative] and true/false items [1–2]). Point biserial correlation was used for one dichotomous variable and one continuous variable (i.e. SIPS group and Likert scale variables [items 3–10, total score]).

^*p <.05;

^**p <.01;

^n.s.-nonsignificant.

The Structured Interview for Psychosis-risk Syndromes (SIPS; McGlashan, Walsh, & Woods, 2010) was used to categorize participants as low- or high-risk, or meeting criteria for psychosis, based on five positive symptoms: unusual thoughts/delusions, suspiciousness/paranoia, grandiosity, perceptual abnormalities/hallucinations, disorganized communication. Positive symptoms are rated from 0 (absent) to 6 (severe/psychotic), with ratings of 3–5 indicating CHR symptomatology. This study used a continuous sum of scores from the positive symptom scale (“P-sum,” range 0–30), as well as SIPS diagnostic status (SIPS-negative or SIPS-positive). Following SIPS convention, to be categorized as SIPS-positive, individuals must endorse positive symptoms that meet threshold criteria for one of four defined psychosis-spectrum diagnoses (psychotic syndrome, brief intermittent psychotic syndrome, attenuated positive symptom syndrome, or genetic risk and functional decline syndrome). Criteria is based on SIPS-defined cutoffs for frequency, duration, and clinical significance, and symptoms must not be better accounted for by another diagnosis (e.g., substance use, trauma). The Structured Clinical Interview for DSM-5 Disorders (First, Williams, Karg, & Spitzer, 2014) was administered prior to the SIPS, to inform clinical judgment related to symptom presentation and diagnostic considerations. Staff completed rigorous training from a SIPS-certified trainer and achieved symptom-level reliability > .80.

Procedure

This study was approved by the university institutional review board and all participants provided informed consent. Participants completed the BASC-2 and a diagnostic interview including the SIPS, administered by research staff (graduate-level psychology students supervised by a licensed psychologist). Via consensus meetings, participants were categorized into two groups: SIPS-Negative (no diagnosis) or SIPS-Positive (CHR or psychosis).

Results

The final sample included 43 individuals. Demographic variables did not significantly differ across SIPS groups (Table 1). Age was not correlated with P-sum or Atypicality raw scores, and T-tests demonstrated that these symptom scores did not significantly differ across gender (male, female, other) or racial minority status (yes/no).

Twenty participants were SIPS-Negative, and 23 were SIPS-Positive. Seventeen participants met CHR criteria, four met criteria for a psychotic disorder, and two met criteria for bipolar disorder with psychotic features. The six individuals who met criteria for full threshold symptoms were considered “newly identified,” as symptoms were undiagnosed and untreated prior to study participation. Given the study’s aim to identify clinically significant, undetected symptoms, these individuals were included in main analyses.

Linear regression indicated that Atypicality raw scores statistically predicted P-sum (F(1,41)=21.16, p<.001; R²=.34). T-tests demonstrated that compared to SIPS-Positive, the SIPS-Negative group had significantly lower scores (less symptomatology) on six Atypicality items and mean total raw score (Table 1). Notably, the results were consistent when analyses were run using age-normed T scores (n = 38).

Exploratory analyses were used to examine the accuracy of the Atypicality Scale in predicting SIPS groups. Receiver operating characteristic curves were used to determine empirically-derived optimal cutoffs for this sample (based on the shortest distance to (0,1)). The optimal raw score cutoff (>/= 7) yielded high sensitivity, high accuracy, and adequate specificity (Table 2).

TABLE 2.

Performance of BASC-2 Atypicality scores for predicting SIPS group status

	Specificity	Sensitivity	PPV	NPV	Accuracy	AUC
Full sample (n = 43):
Full Atypicality Scale raw score optimal cut (sum >/= 7)	65.0	78.3	72.0	72.2	72.1	0.72
Modified Atypicality score optimal cut (sum >/= 4)	70.0	91.3	77.8	87.5	81.4	0.81
CHR/Low-risk sample (n = 37)
Full Atypicality Scale raw score optimal cut (sum >/= 7)	65.0	70.6	63.2	72.2	67.6	0.68
Modified Atypicality score optimal cut (sum >/= 4)	70.0	88.2	71.4	87.5	78.4	0.79
Age-normed sample (n = 38):
Atypicality Scale T score clinical cutoff (>/= 60)	52.9	81.0	68.0	69.2	68.4	0.67

Notes. BASC-2: Behavior Assessment System for Children, Second Edition; SIPS: Structured interview for psychosis-risk syndromes; PPV: positive predictive value; NPV: negative predictive value; AUC: area under the curve; CHR: Clinical high risk. The Modified Atypicality score is a sum of all item scores, except for two items that were negatively associated with SIPS-Positive status (unable to stop self from doing bad things and does things over and over and can’t stop). Full sample: n = 43, including 20 SIPS-Negative, 23 SIPS-Positive. CHR/low-risk sample (with full-psychosis cases removed): n = 37, including 20 SIPS-Negative, 17 SIPS-Positive. Age-normed sample (including individuals </= 25 years old): n = 38, including 17 SIPS-Negative, 21 SIPS-Positive.

Given negative associations between SIPS status and two Atypicality items (#2 and #5 in Table 1), these items were dropped from the scale to explore the accuracy of a modified Atypicality raw score in predicting SIPS status. The optimal cutoff (>/= 4) for this modified score yielded strong performance statistics across all indices of accuracy (Table 2). For comparison, parallel analyses with the full and modified Atypicality raw scores were run, excluding participants with full threshold symptoms (n = 37; Table 2).

Finally, the accuracy of Atypicality T scores at or above 60 (i.e. indicating clinical risk, per BASC-2 developers) were also explored in relation to SIPS groups (Table 2).

Discussion

Results suggest that screening and assessing risk among help-seeking university students offers the promise of early identification and potentially intervention to a larger population relative to psychosis specialty clinics. The BASC-2 is a multidimensional tool that can identify various MH strengths and concerns, making it appealing for widescale adoption (Eklund et al., 2010). In particular, the Atypicality Scale appears to be a promising method to identify individuals who meet criteria for a SIPS-defined psychosis-spectrum diagnosis, and particular experiences (e.g., hallucinations, paranoia) appear to differentiate SIPS-Positive and SIPS-Negative individuals. Furthermore, the performance of the Atypicality Scale for identifying individuals at risk for or in the early stage of psychosis may be enhanced by dropping items that do not appear to be linked to the psychosis-spectrum symptomatology and may better capture anxiety or obsessive-compulsive traits (e.g., unable to stop self from doing bad things, does things over and over and can’t stop). Given the BASC-2’s broad scope, this tool may be particularly useful for simultaneously screening for common comorbid concerns (e.g., depression, anxiety; Thompson et al., 2015), which is an important treatment consideration for early/attenuated psychosis (Addington et al., 2017).

These results have important implications, given that many clinicians are trained to use the BASC-2 and it is a commonly used tool in clinical and non-clinical settings by various providers (Reynolds & Kamphaus, 2002). The BASC-2 is a consistent, reliable tool that is designed for and standardized in college settings within and outside of the U.S. (Ahn et al., 2014; Nowinski et al., 2008). Efforts are currently underway to implement psychosis-spectrum screening into K-12 schools (Meyer et al., 2019), and the BASC-2, in both self- and parent-report forms, may be helpful in these efforts for earlier identification (Thompson et al., 2013; 2014). Given these results and pending further research, similar identification efforts in colleges/universities may also consider the BASC-2.

Limitations

This study was limited by its small sample. Although promising, results should be interpreted with caution and are subject to validation in larger samples. Furthermore, the sample included individuals who met criteria for CHR and full-threshold disorders. A larger sample would allow for exploration of differences across subgroups. Additionally, we used an enriched sample, all of whom endorsed potential psychosis-risk symptoms at intake (before completing the BASC-2). It is unclear how the Atypicality Scale would perform in a similar sample that was not filtered through first-line screening. Despite the extensive norming of the BASC 2, the author suggested T score cutoff for risk was not as favorable in terms of prediction as our empirically defined approach. It is likely that the normative sample is not representative of our unique clinical sample, which is to be expected given our small sample and expected variability due to regional and recruitment differences.

For this study, pre-screened students were able to decline the SIPS despite clinician recommendation. It is possible that those who declined were systematically distinct on unmeasured variables. Factors such as stigma, fear, and limited insight, which are common in emerging psychosis (Yang et al., 2019), may have impacted participation. Thus, it is impossible to determine if our sample is representative of the larger population. The BASC-2 creators have more recently released the BASC-3 (Reynolds & Kamphaus, 2015). The BASC-3 has two new items (“people think I’m strange” and “I have trouble controlling my thoughts”), and one BASC-2 item has been dropped (“someone else controls my thoughts”). Given these subtle changes, it will be necessary to explore the performance of BASC-3 Atypicality in relation to psychosis-spectrum assessment. Most items were retained, however, making it likely that the BASC-3 Atypicality Scale performs similarly to that of the BASC-2.

Conclusions

Accurate identification and treatment in the early stages of psychosis are crucial steps to improving outcomes. Early intervention efforts in general MH settings may be hampered by a lack of validated screening and assessment tools. University clinics are uniquely positioned to screen large populations of help-seeking young adults for signs of psychosis and refer to appropriate services. The BASC-2 is a useful tool for broadband MH screening among help-seeking students, and it may be a promising screening tool for early psychosis specifically. The Atypicality Scale demonstrated promising rates of accuracy for identifying SIPS-Positive individuals in this sample. Overall, results suggest that screening for and assessing risk in a university clinic offers the promise of early identification to a larger population relative to specialty clinics.