Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Jun 30.
Published in final edited form as: J Abnorm Child Psychol. 2014 May;42(4):649–658. doi: 10.1007/s10802-013-9809-x

Youth-caregiver Agreement on Clinical High-risk Symptoms of Psychosis

Shana Golembo-Smith 1, Peter Bachman 2, Damla Senturk 3, Tyrone D Cannon 4,5,6, Carrie E Bearden 7,8
PMCID: PMC4074879  NIHMSID: NIHMS599970  PMID: 24092494

Abstract

Early identification of individuals who will go on to develop schizophrenia is a difficult endeavor. The variety of symptoms experienced by clinical high-risk youth make it difficult to identify who will eventually develop schizophrenia in the future. Efforts are being made, therefore, to more accurately identify at-risk individuals and factors that predict conversion to psychosis. As in most assessments of children and adolescents, however, both youth and parental report of symptomatology and resulting dysfunction are important to assess. The goals of the current study were to assess the extent of cross-informant agreement on the Structured Interview for Prodromal Symptoms (SIPS), a widely-used tool employed to determine clinical high-risk status. A total of 84 youth-caregiver pairs participated. Youth and caregiver raters displayed moderate overall agreement on SIPS-rated symptoms. Both youth and caregiver ratings of youth symptomatology contributed significantly to predicting conversion to psychosis. In addition, youth age and quality of youth-caregiver relationships appear to be related to cross-informant symptom ratings. Despite differences on individual SIPS domains, the majority of dyads agreed on youth clinical high-risk status. Results highlight the potential clinical utility of using caregiver informants to determine youth psychosis risk.

Keywords: CHR, Clinical high risk, Assessment, Psychosis, Adolescents, Cross-informant agreement

Research has highlighted numerous benefits of identifying at-risk individuals before they develop full-blown psychotic illness (e.g., McGorry and Edwards 1998; Perkins et al. 2005; Ruhrmann et al. 2003; Wyatt and Henter 2001). Currently, a widely used tool for diagnosing an at-risk mental state is the Structured Interview for Psychosis-Risk Syndromes (SIPS; McGlashan et al. 2010). The SIPS is a semi-structured, clinician-administered face-to-face interview that assesses subthreshold positive symptoms germane to clinical high-risk (CHR) status, as well as negative, disorganized, and general symptoms that are also characteristic of early psychosis and at-risk states (Yung et al. 1996).

While determination of CHR status, or a putative at-risk syndrome, via the SIPS does not require other informant or caregiver ratings, as with most assessment of youth psychopathology, gathering information from a variety of sources is likely to be advantageous (Achenbach 2005; De Los Reyes and Kazdin 2005). Collateral interviews may be particularly important when assessing subthreshold psychotic symptoms, given the difficulty youth often have with reporting symptom onset and timelines, assessing the impact of symptoms on behavior, and determining whether possible delusional and grandiose symptoms (based on youth self-report) are actually culturally normative (e.g., Ho et al. 2004; Lee and Schepp 2011).

A variety of factors appear to influence cross-informant agreement on ratings of psychopathology, both on self- and parent-report forms, and in clinical interviews. In general, participants often report more symptoms than caregiver informants (Martin et al. 2004; Sourander et al. 1999), particularly those with internalizing, rather than externalizing, concerns (Bird et al. 1992; Penney and Skilling 2012; Salbach-Andrae et al. 2009). Demographic variables, such as participant age and gender, may mediate level of agreement on symptom indices. Studies have suggested, for example, that youth-caregiver agreement rates are higher for younger children than for adolescents (Achenbach et al. 1987; Handwerk et al. 1999) and higher between girls and caregiver informants than between boys and caregiver informants (e.g., Kazdin et al. 1983; Kolko and Kazdin 1993; Safford et al. 2005). Other studies, however, have reported inconclusive findings regarding age and gender (Choudhury et al. 2003; Rey et al. 1992), or the opposite results, in which older children actually have higher rates of child-parental agreement on certain symptomatic indices than do younger children (e.g., Berg-Nielsen et al. 2003; Grills and Ollendick 2003) and male youth higher rates of agreement with caregivers than their female counterparts (Sourander et al. 1999). Severity of symptomatology has also been suggested as a moderator of cross-informant agreement rates, with higher severity/comorbidity rates associated with lower agreement (Handwerk et al. 1999; Salbach-Andrae et al. 2009). Additionally, family factors, such as family discord and parental psychopathology, have been shown to increase youth-caregiver symptom report discrepancies (Berg–Nielsen et al. 2003; Christensen et al. 1992; Grills and Ollendick 2003; Kolko and Kazdin 1993; Youngstrom et al. 1999).

In cases of youth with diagnoses involving both internalizing and externalizing features, a combination of caregiver and participant report may be necessary to provide the most accurate picture of the constellation of symptoms (Youngstrom et al. 2004). Although this question has not yet been directly investigated in youth with subthreshold psychotic symptoms, this principle may also apply, given that such symptoms can manifest in both internalized (e.g., positive symptom scale items) and externalized (e.g., disorganized symptom scale items) experiences (Keshavan et al. 2009; Lee and Schepp 2011; Neumann et al. 1995).

Despite intense interest in focusing screening efforts toward the most vulnerable individuals and in creating more targeted identification techniques (Compton 2004), early identification of youth who will go on to develop schizophrenia remains challenging, due to the heterogeneous and subjective nature of symptoms experienced during the CHR phase of the illness. Research has identified some factors that put youth at greatest risk for becoming acutely psychotic. In particular, unusual thought content, suspiciousness, social withdrawal, and substance abuse are predictive of conversion to psychosis (Cannon et al. 2008). In addition, environmental factors, such as warm, supportive family environments are predictive of reduction of symptomatology in CHR youth (O'Brien et al. 2006; 2009; Schlosser et al. 2010). No published studies to date, however, have specifically examined cross-informant agreement on CHR symptoms in a clinically referred sample.

Aims and Hypotheses

Given the importance of caregivers' insight into subthreshold psychotic symptoms, for assessment purposes and prognostic benefits, the goals of the current study were to: (1) assess the extent of participant-caregiver agreement on SIPS symptom severity ratings; (2) examine whether demographic and psychosocial factors modify cross-informant agreement; and (3) assess which youth- and caregiver-rated symptoms best predict conversion to full-blown psychosis. It was hypothesized that (1) there would be higher agreement for behavioral manifestations of more external and apparent forms of symptomatology (SIPS disorganized symptom scale, SIPS general symptom scale) relative to SIPS positive symptoms such as unusual thought content, which more closely tap internal experiences (Lee and Schepp 2011). It was further hypothesized that (2) participant and family characteristics may significantly impact agreement rates, given previous research on cross-informant ratings of psychopathology. Specifically, we predicted that higher subjective reports of familial distress (Perceived Criticism questionnaire scores) and higher symptom severity (Global Assessment Functioning scores) would be associated with lower levels of cross-informant agreement. We ran exploratory analyses to examine the precise impact of participant age and gender on youth-caregiver level of agreement, given the mixed findings regarding these variables within the literature base (e.g., Achenbach et al. 1987; Berg–Nielsen et al. 2003; Grills and Ollendick 2003; Handwerk et al. 1999; Kazdin et al. 1983; Kolko and Kazdin 1993; Safford et al. 2005; Sourander et al. 1999). Lastly, it was hypothesized that (3) caregiver and youth report of youths' unusual thought content and youths' suspiciousness would significantly predict conversion to full-blown psychosis, as these factors were shown to significantly predict conversion when drawn from youth self-reports (Cannon et al. 2008).

Method

Participants

Participants in this study were recruited through enrollment in a larger longitudinal prospective research study at the Center for the Assessment and Prevention of Prodromal States (CAPPS) at the University of California, Los Angeles (UCLA). To be eligible for CAPPS inclusion, individuals must be between the ages of 12 and 35, speak English fluently, and meet criteria for at least one of three CHR states as determined by the SIPS, namely: (1) attenuated psychotic symptoms that have begun or worsened in the past year (Attenuated Positive Symptom Syndrome, APS); (2) transient, recent-onset fully psychotic symptoms (Brief Intermittent Psychotic Syndrome, BIPS); or (3) a significant drop in social/role functioning in conjunction with schizotypal personality disorder (SPD) diagnosis or presence of psychotic disorder in first-degree relative (Genetic Risk and Deterioration, GRD). The APS and BIPS prodromal syndrome categories describe individuals with subthreshold (i.e., not fully psychotic) and fleeting, fully psychotic experiences, respectively. As both SPD diagnosis and having a first-degree relative with psychosis are believed to reflect a genetic predisposition to psychosis, the GRD group represents individuals with both trait and state risk factors for developing psychosis (Tsuang et al. 1999). These three categories were originally developed by Yung and colleagues (Yung et al. 1996) to identify individuals at heightened risk for developing psychotic disorders. Within 2–2.5 years, 16–35 % of participants meeting criteria for one of these three CHR states develop diagnosable psychotic disorders (Cannon et al. 2008; Yung et al. 2008). Included individuals could meet criteria for more than one CHR syndrome.

Individuals with significant head injuries or those who met criteria for diagnosis of drug or alcohol abuse or dependence were excluded. In addition, only participants with symptom ratings independently provided by another informant (i.e., biological or adoptive parent, or other close relative) were included in the current analyses.

A total of 84 participant-caregiver dyads were included in the study. Mean age of admission into CAPPS was 15.90 (SD=2.13), with participant ages ranging from 12 to 20. Male participants comprised 62 % of the sample, while 46 % of the participants was non-Caucasian. The other informant group was comprised of mothers (85 %), fathers (8 %), and siblings (7 %).

Procedures

This study was approved by the Institutional Review Board of the University of California, Los Angeles. Prior to study enrollment, participants and caregivers provided written informed consent. Minor participants and their legal guardians provided written assent and consents for participation, respectively. Participants and their caregiver informants were each interviewed by separate clinical assessors, who had undergone a thorough assessment training and diagnostic reliability check. Participant and caregiver interviewers were blind to each others' ratings at the time symptom evaluations were made. Interviewers were assigned to conduct interviews in a balanced manner such that they alternated between interviews of youth and caregivers, and so that each interviewer conducted approximately the same number of youth and caregiver interviews. Participants suspected to be in the CHR range, based on self- and caregiver-report, were presented to a clinical consensus team meeting who collaboratively determined if the individual qualified as a study participant. During these team meetings, participant and caregiver interviewers presented justification for their respective SIPS item scores. Participants not qualifying as CHR were referred to outside clinical services. Participant and caregiver dyads were assessed at baseline, and every 6 months thereafter, up to a maximum follow-up period of 2 years, for current symptom severity, psychosocial functioning, Axis I disorders, and conversion to full-blown psychosis, among other domains. Though not the focus of the current study, other data, including baseline and follow-up neurocognitive functioning were collected from study participants (see Niendam et al. 2007 for neurocognitive assessment procedures). Meyer and colleagues (2005) and Addington et al. (2007) describe the training, assessment, consensus, and establishment of interrater reliability procedures in greater detail.

Measures

After informed consent, participants and caregiver informants were separately administered the SIPS interview (Version 4.0, McGlashan et al. 2003) by independent trained clinical interviewers. Specific SIPS scales include: positive symptom scale (unusual thought content/delusional ideas, suspiciousness/persecutory ideas, grandiose ideas, perceptual abnormalities/hallucinations, and disorganized communication), negative symptom scale (anhedonia, avolition, ideational richness, occupational functioning, decreased expression of emotion, decreased experience of emotions and self), disorganized symptom scale (odd behavior, bizarre thinking, impairment in personal hygiene, and trouble with focus and attention), and general symptom scale (sleep difficulty, motor disturbance, dysphoric mood, and impaired tolerance to normal stress). Individual SIPS items from each scale (e.g., usual thought content/delusional ideas, anhedonia) are scored on a 6-point Likert-style scale, with higher scores on the SIPS items indicating more severe symptomatology. The five positive symptom items are used to determine CHR status, with items scored 0–2 generally indicating absence of symptoms or questionable presence of symptoms, items scored 3–5 indicate symptoms in the attenuated positive CHR range, and items scores of 6 indicating symptoms in the psychotic range. Youth were considered to have converted to overt psychotic disorder if they reported symptoms at a psychotic intensity (any SIPS positive item score) for at least one hour a day, at least four days per week during the past month, as in Cannon et al. 2008. A series of specific probes are used to determine item ratings. Example probes for SIPS positive scale items include “Have you ever been confused at times whether something you have experienced is real or imaginary?” (unusual thought content/delusional ideas), “Do you ever feel people might be intending to harm you?” (suspiciousness/persecutory ideas), “Do you ever think of yourself as a famous or particularly important person?” (grandiose ideas), “Do you ever feel that your ears are playing tricks on you?” (perceptual abnormalities/hallucinations), and “Are you aware of any ongoing difficulties getting your point across, such as finding yourself rambling or going off track when you talk?” (disorganized communication). SIPS interviews also contribute to a global assessment of functioning (GAF) score. Psychometric information regarding the SIPS, as well as diagnostic reliability and case consensus procedures have been previously published (Meyer et al. 2005).

In addition to the SIPS, participants and caregivers were administered the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) Axis I Disorders (SCID; First et al. 2002) to assess DSM-IV psychiatric diagnoses, and to ensure that putative at-risk symptoms were not better accounted for by another Axis I diagnosis. At each assessment time point, patients and participants each completed the Perceived Criticism questionnaire (PC), parent and child versions (Hooley and Teasdale 1989; Schlosser et al. 2010). The PC examines the extent to which youth perceive criticism in their caregivers (youth version) and the extent to which caregivers believe they are critical towards their children (caregiver version). Baseline PC scores have been shown to be predictive of CHR youth follow-up functioning (Schlosser et al. 2010). For the purposes of this study, parent and youth perceived criticism ratings functioned as a measure of youth-perceived and caregiver-perceived familial distress. Youth and caregivers rated items on a single scale from 1 (not at all critical) to 10 (very critical indeed), with probes such as “How critical is your mother of you?”/” How critical are you of your son or daughter?” The PC has demonstrated robust test-rest reliability and convergent validity (Chambless and Blake 2009; Chambless and Steketee 1999; Hooley and Teasdale 1989).

Statistical Analyses

Statistical analyses were performed using SPSS for Windows, release 19.0. Descriptive statistics included participant age, race/ethnicity, and relationship to other informant. The degree of agreement between participant and caregiver SIPS ratings was assessed using intraclass correlation coefficients, with coefficients >0.8 indicating almost perfect agreement; 0.6–0.8 substantial agreement, 0.4–0.6 moderate agreement, 0.2–0.4 low agreement, and <0.2 poor agreement (ICC; Martin Bland and Altman 1986). In addition to participant and caregiver SIPS interviews determining study inclusion, SIPS items scores were employed to designate specific CHR states (i.e., APS, BIPS, GRD, or a combination of CHR states). A kappa statistic was calculated to examine agreement on CHR classification (i.e., caregiver-youth agree that youth is CHR or caregiver-youth disagree that youth is CHR).

A number of caregiver-youth dyads had SIPS positive scale data, but were missing ratings from other scales. To address systematic differences in demographic variables, symptom severity, or level of agreement between those with and without missing data, we compared dyads with a complete set of data (i.e., ratings on all four SIPS scales) to those who were missing one or more domains. Chi-square analyses were used to compare those with and without missing data with regard to overall agreement on CHR classification and youth gender, and Mann–Whitney tests were employed to compare these two groups on youth age and GAF score.

A binary regression analysis was performed to examined the impact of baseline characteristics (youth age, gender, GAF, youth perceived caregiver criticism, caregiver perceived caregiver criticism) on youth-caregiver agreement (Myers 1990). Youth and caregiver reports of perceived caregiver criticism were measured using the PC scale and functioned as a proxy of familial distress and quality of youth-caregiver relationship. Youth-caregiver agreement was classified into one of two categories: youth-caregiver agreement on youth being CHR, or youth-caregiver disagreement regarding youth being CHR. A backward likelihood ratio model was employed because this analysis was primarily exploratory. This method starts with all five of the predictors. At each step, predictors that contribute the least to the model, based on maximum partial likelihood estimates, are removed. Ultimately, the analysis finds the best combination of variables to explain the largest proportion of the variance in outcome. To test for stability of these results, a second, forward stepwise regression likelihood ratio model was employed to examine which variables would be retained in both models. The forward likelihood ratio model also performs multiple iterations based on whether adding removing a variable would make a significant difference in terms of how well the model data fits the data model; predictors that significantly change the model fit are retained in the final model.

Lastly, backward and forward stepwise likelihood ratio and forward likelihood ratio method logistic regression analyses were employed to evaluate youth and caregiver positive SIPS items in predicting conversion to overt psychosis. The five positive SIPS items, as rated by both caregiver and youth, were used as predictors, with conversion to psychosis (yes or no) as the outcome variable. Again, both backward and forward methods were used to examine the consistency of results.

Results

Descriptive Statistics

As noted in the Introduction, a total of 84 participant-caregiver pairs were included in the study analyses. There were 73 participant-caregiver pairs with complete data available for all positive symptom items, with 77 pairs having complete data for the grandiosity item, 78 pairs having complete data for the disorganized communication item, 79 pairs having complete data for the unusual thought content items, 80 pairs having complete data for the perceptual anomalies item and 82 pairs having complete data for the suspiciousness item. There were 44 participant-caregiver dyads with complete data available for the general symptom scale and 43 participant-caregiver dyads had complete data available for the negative and disorganized symptom scales. In the sample, 81 % of participant-caregiver dyads reported that they were living together as of the baseline assessment (n =68), 2 % reported that they were not living together at baseline (n =3), and current household data were missing from 17 % of dyads (n =13). Highest mean number of years of education completed by participants as of baseline was 9.3 (SD=1.98). Participants had a mean estimated baseline IQ of 105.78 (SD=13.80), with scores ranging from 73 to 128.

Clinical High-Risk State Classification

Based on participant report, 60 (71 %) CHR participants were categorized as in the APS CHR state, 18 (21 %) in the BIPS CHR state, 2 (3 %) as GRD, and 4 (5 %) participants as meeting criteria for more than one CHR category. Of the youth participants, 68 (81 %) who were considered CHR based on self-report information would also be considered at risk (i.e., SIPS scores above the at-risk level of severity) based on caregiver SIPS report. Of those considered CHR by both youth and caregiver report, 48 (72 %) dyad interviews led to the same CHR state categorization. There were 19 (28 %) caregiver interviews that led to the participant being considered CHR, but not in the same CHR category as the corresponding participant interview. The Kappa statistic for participant-caregiver agreement on CHR status was 0.50 (n =84). Table 1 summarizes specific CHR state classification (APS, BIPS, GRD, or multiple states) by participant and caregiver reports.

Table 1.

Clinical high-risk (CHR) state classification by participant-based and caregiver report

Caregiver-based interview CHR state Participant-based interview CHR state
 Total
APS BIPS GRD Multiple CHR States
 APS 40 8 0 2 50
 BIPS 0 8 0 1 9
 GRD 1 0 1 1 3
 Multiple CHR States 4 1 1 0 6
 Not CHR 15 1 0 0 16
Total 60 18 2 4 84
Open in a new tab

APS Attenuated Positive Symptom Syndrome; BIPS Brief Intermittent Psychotic Syndrome; GRD Genetic Risk and Deterioration

Assessment of Selection Bias

There were 49 dyads that had complete data and 36 were considered to have an incomplete data set. There were no significant differences between these two groups in terms of agreement regarding CHR status, youth gender, youth age, or youth GAF score.

Participant-Caregiver Intraclass Correlation Coefficients

Overall intraclass correlation coefficients ranged from low to moderate, with low agreement on positive scale total, negative scale total, and disorganized scale total, as well as on suspiciousness and perceptual anomalies. There was moderate agreement on general scale total, and on unusual thought content, grandiosity, and disorganized communication. A summary of ICC statistic values are presented in Table 2.

Table 2.

Summary of ICC for each SIPS scale and SIPS positive scale individual items

Scale Positive scale item ICC (95 % CI) Description
Positive Scale Total 0.22 −0.01–0.42 Low
Unusual thought content 0.43 0.23–0.60 Moderate
Suspiciousness 0.39 0.19–0.56 Low
Grandiosity 0.44 0.24–0.60 Moderate
Perceptual anomalies 0.25 0.03–0.44 Low
Disorganized communication 0.57 0.39–0.70 Moderate
Negative scale total 0.37 0.08–0.60 Low
Disorganized scale total 0.36 0.07–0.59 Low
General scale total 0.44 0.17–0.65 Moderate
Open in a new tab

Binary Logistic Regression: Factors Influencing Agreement

A backward stepwise likelihood ratio binary logistic regression was performed in which youth age, gender, GAF, and youth-perceived parental criticism as well as caregiver-perceived caregiver criticism were entered into the model to predict caregiver/youth agreement on CHR status. Moving backward, youth age, gender, and perceived criticism scores were eliminated as predictor terms because they weakened the model. The result of the final model χ2 remained significantly different from the constant-only model at 11.53 (df =1, p < 0.01) and successfully classified 78 % of participants. Youth age (Wald χ12=3.10, p =0.08) and caregiver reports of perceived criticism (Wald χ12=3.66, p =0.06), were retained in the final model, though they were not significant.

A second, forward stepwise likelihood ratio binary logistic regression analysis was employed to examine stability of results. As in the backward stepwise regression likelihood ratio analysis, youth and caregiver reports of their own were retained in the final model. Identical model χ2 statistic, Wald χ12, odds ratios, and confidence intervals were found using the forward stepwise regression likelihood method. A summary of these analyses are Wald χ2, odds ratios, and confidence intervals of the variables that were retained in the final model area presented in Table 3.

Table 3.

Summary of binary logistic regression analysis final model predicting CHR status agreement

Wald χ2 p B (SE) Exp(B) 95 % CI)
Constant 0.47 0.49 5.04 (7.36)
Youth Age 3.10 0.08 0.72 (0.41) 2.05 (0.92–4.54)
Caregiver Perceived Criticism 3.66 0.06 −0.94 (0.49) 0.39 (0.15–1.02)
Open in a new tab

Model χ22 =11.53, p<0.01; degrees of freedom for all variables=1; Hosmer-Lemeshow χ72 =4.65, p=0.70; R2 =0.35 (Cox & Snell), 0.53 (Nagelkerke)

Binary Logistic Regression: Predicting Conversion to Psychosis

Date from 2-year follow-up assessments were available for 96 % of participants; those without 2-year follow-up data (n =3) were not entered into the analysis. Of the CHR participants, 21 (26 %) in the sample converted to psychosis within 2 years of ascertainment, whereas 60 did not. A backward stepwise likelihood ratio binary logistic regression was performed in which separate youth and caregiver ratings SIPS positive scale ratings of unusual thought content/delusional ideas, suspiciousness/persecutory ideas, grandiose ideas, perceptual abnormalities/hallucinations, and disorganized communication were used to predict binary conversion to full-blown psychosis. Moving backward, youth ratings of suspiciousness/persecutory ideas, grandiose ideas, perceptual abnormalities/hallucinations, and disorganized communication and caregiver ratings of unusual thought content/delusional ideas, grandiose ideas, and perceptual abnormalities/hallucinations were eliminated as predictor terms because they weakened the model. The result of the final model χ2 remained significantly different from the constant-only model at 18.16 (df =3, p <0.001) and successfully classified 83 % of participants. Youth ratings of unusual thought content/delusional ideas (Wald χ12=3.36, p =0.07), caregiver ratings of suspiciousness/persecutory ideas (Wald χ12=6.39, p <0.01), and caregiver ratings of youth disorganized communication (Wald χ12=3.69, p =0.06), were retained in the final model.

A second analysis, utilizing a forward stepwise regression was employed to examine the stability of the results. Employing this method, the final model χ2 remained significantly different from the constant-only model at 14.37 (df =2, p <0.01) and successfully classified 80 % of participants. Caregiver ratings of youth suspiciousness/persecutory ideas (Wald χ12=7.09, p <0.01), and caregiver ratings of youth disorganized communication (Wald χ12=4.63, p <0.05), were retained in the final model.

Wald χ2, odds ratios, and confidence intervals of the variables that were retained in the final models of both analyses are presented in Table 4.

Table 4.

Summary of binary logistic regression analysis final model predicting conversion

Method Wald χ2 p B (SE) Exp(B) (95 % CI)
Backward Constant 14.89 <0.001 −4.41 (1.15)
Youth—Unusual Thoughts 3.36 0.07 0.41 (0.22) 1.50 (0.97–2.32)
Caregiver—Suspiciousness 6.39 0.01 0.47 (0.19) 1.60 (1.11–2.31)
Caregiver—Disorganized 3.69 0.06 0.41 (0.21) 1.51 (0.99–2.30)
Forward Constant 18.69 <0.001 −2.89 (0.67)
Caregiver—Suspiciousness 7.09 0.008 0.47 (0.18) 1.59 (1.13–2.25)
Caregiver—Disorganized 4.63 0.03 0.45 (0.21) 1.56 (1.04–2.35)
Open in a new tab

Backward Model χ22 = 18.16, p <0.001; degrees of freedom for all variables=1 Hosmer-Lemeshow χ82 =5.12, p =0.75; R2 =0.23 (Cox & Snell), 0.33 (Nagelkerke)

Foward Model χ22 =14.37, p <0.001; Hosmer-Lemeshow χ82 =5.52, p =0.60; R2 =0.19 (Cox & Snell), 0.27 (Nagelkerke)

Discussion

To our knowledge, this is the first study to specifically examine youth–caregiver agreement on CHR psychotic symptoms in a clinically referred sample. It revealed several important findings. First, participants and caregivers report different symptoms, but cross-sectionally, they are in moderate agreement about CHR status. Second, youth-caregiver agreement rates on different domains may be moderated by youth age and quality of relationship with caregiver. Third, caregiver raters offer different, but significant, predictors of later conversion to psychosis.

With respect to overall caregiver-youth agreement on youth CHR status, 81 % percent of caregiver-participant dyads in this clinically-referred sample agreed that the youth in question was at CHR for developing psychosis. After correcting for chance agreement, kappa was moderate (0.5), and individual SIPS scales had ICCs ranging from low to moderate. It was hypothesized that there would be more agreement on more observable, behavioral manifestations of at-risk symptoms than of more internal experiences. This hypothesis was somewhat supported; there was relatively higher agreement on the SIPS domains of general, negative, and disorganized symptoms, than on positive symptoms. This is consistent with the notion that many subthreshold positive symptoms are primarily internal in nature (e.g., seeing shadows out of the corner of one's eyes), while other factors associated with risk status, such as anhedonia (negative scale), difficulty with hygiene (disorganized scale), or exhibiting motor disturbances (general scale), may be easier for collateral informants to accurately report (e.g., Jackson et al. 1994; Nugent et al. 2013).

Consistent with previous research on youth-caregiver reports of psychopathology, no single demographic variable significantly accounted for agreement rates. While caregiver-perceived caregiver criticism and youth age were retained in the agreement models, they were not significant predictors of agreement status. It was hypothesized that familial distress (higher Perceived Criticism questionnaire scores) and overall symptom severity (lower GAF scores) would be negatively associated with cross-informant agreement. This hypothesis, too, was supported to some extent; while caregiver-perceived caregiver criticism was associated – though not significantly – with lower levels of caregiver-youth agreement, symptom severity was not related to agreement. This is somewhat consistent with research that has demonstrated that youth reports of parent–child relationship are not related to agreement rates (Nugent et al. 2013), but that overall familial distress does contribute to discrepancies (Christensen and Margolin 1988; Christensen et al. 1992; De Los Reyes and Kazdin 2005; Grills and Ollendick 2003; Jensen et al. 1988). Odds of caregiver-youth dyads being in agreement on CHR status, however, increased 2.05 times for every 1 year age increase, suggesting an association between youth age and concordance with caregiver ratings. Other studies have found a similar potential relationship, possibly accounted for by older youth being better able to express their experiences to their parents or by older youth having more serious, and therefore more recognizable, symptoms (Berg–Nielsen et al. 2003; Edelbrock et al. 1985; Grills and Ollendick 2003; Rapee et al. 1994).

The third hypothesis was that, among positive scale symptoms, youth and caregiver ratings of youth subthreshold delusional ideation (unusual thought content/delusional ideas, suspiciousness/persecutory ideas) would significantly predict conversion to full-blown psychosis. This hypothesis, too, was partially supported. Backward binary regression analyses indicated that youth self-report of CHR delusional ideation as well caregiver report of youth suspiciousness/persecutory ideas and caregiver report of youth disorganized communication significantly contributed to predicting conversion to full-blown psychosis, however, only caregiver ratings of youth suspiciousness/persecutory ideas and youth disorganized communication were retained in both the forward stepwise and backward stepwise models. This is somewhat consistent with previous research indicating that, among other factors, youth self-reported unusual thought content and suspiciousness are predictive of conversion to psychosis. This suggests that caregivers are not only just as valuable as identified CHR youth in reporting risk status, but that caregivers may in fact be more accurate in their view of the participant's risk of conversion. Other studies have suggested that parents may be better informants than youth for both internalizing and externalizing symptoms, especially for younger children (Smith 2007). Caregivers, for example, may be more attuned to youth's increasingly disorganized communication than the youth themselves or than clinical interviewers.

There are clinical implications of these findings with regard to assessing subthreshold psychotic symptoms. The fact that most of the caregiver informants were able to describe participant symptoms that fell in the CHR range indicates that caregiver informants may be an acceptable substitute when participants are unable to report on their own experiences. Many CHR participants' communication is affected by thought disorder, such in the case of those suffering from disorganization, inattention, and poverty of speech associated with CHR status, or those with positive symptoms such as suspiciousness of the interviewer. Clinicians should be aware, however, that nearly one in five participants in our sample would not be considered CHR by caregiver report. That said, no caregivers reported subthreshold symptoms in youth who did not endorse subthreshold psychotic symptoms themselves. Youth and caregiver informants may have differential value depending on what domain is being assessed; youth may be better reports of less observable symptoms, while caregivers may be better able to report of behavioral manifestations of symptoms (Carpenter et al. 1976; Jackson et al. 1994). Ideally, a combination of caregiver and participant report will be used in clinical settings to assess psychosis risk, particularly positive symptoms, which contributed to predicting conversion both from participant and caregiver ratings.

Several limitations to this study should be noted. While our overall sample size was moderate (84 dyads), the number of youth who converted to overt psychosis was small, limiting power for this analysis. Nevertheless, our conversion rate was very comparable to other recent studies (e.g., Cannon et al. 2008; Fusar-Poli et al. 2012; McGorry et al. 2012; Ruhrmann et al. 2010). Missing data was also a limitation; however, there do not appear to be any systematic differences between dyads with and without complete datasets. Finally, some differences between caregiver and youth report may have been due to interviewer effects. While this is certainly a possibility, informant-interviewer assignments were balanced to reduce this risk.

Future research in this area will likely be enhanced by expanded follow-up assessment timeframes with a larger sample of individuals converting to psychosis. More substantial sample sizes would allow for more fine-grained analysis of predictors, including the interaction between youth age, symptom severity, and youth/caregiver agreement status on conversion rates. In addition to a more sensitive analysis of conversion predictors, a better understanding of non-converters, including other domains of later adulthood psychopathology, may stem from such research.

These initial results, though involving a modest sample size and exploratory in nature, are promising in that most caregiver informants reported symptoms that were in the same range as participant report (e.g., absent, subclinical, psychotic). This is important, given that the vast majority (93 %) of caregiver informants were parents, and parents are key to young people accessing treatment. In fact, many individuals falling within the CHR period are minors (Cannon et al. 2008). Considering the connection between earlier entry into mental health treatment and better functional outcomes, future work may benefit from targeting parental awareness of early warning signs of psychosis. Large-scale efforts to increase early recognition of subclinical psychosis among clinicians, school staff, and the public have been relatively successful in decreasing duration of untreated psychosis as well as reducing incidence rates of schizophrenia (Falloon 2000; Joa et al. 2008). Similar awareness campaigns designed specifically to target youth caregivers may be helpful to further increase early identification of youth at risk for converting to psychosis.

Acknowledgments

Sponsorship Funding for this study was provided by NARSAD Young Investigator Award (Maxine and Jack Zarrow Investigator Award to CEB), National Institute of Mental Health grants K23MH74644 (CEB), MH65079 (TDC) and NIMH P50 MH066286 (TDC and CEB), as well as donations from the International Mental Health Research Organization (IMHRO) and Staglin Music Festival for Mental Health to the UCLA Foundation.

References

  1. Achenbach TM. Advancing assessment of children and adolescents: Commentary on evidence-based assessment of child and adolescent disorders. Journal of Clinical Child and Adolescent Psychology. 2005;34:541–547. doi: 10.1207/s15374424jccp3403_9. [DOI] [PubMed] [Google Scholar]
  2. Achenbach TM, McConaughy SH, Howell CT. Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin. 1987;101:213–232. [PubMed] [Google Scholar]
  3. Addington J, Cadenhead KS, Cannon TD, Cornblatt B, McGlashan TH, Perkins DO, Heinssen R. North American prodrome longitudinal study: a Collaborative multisite approach to prodromal schizophrenia research. Schizophrenia Bulletin. 2007;33:665–672. doi: 10.1093/schbul/sbl075. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Berg-Nielsen TS, Vika A, Dahl AA. When adolescents disagree with their mothers: CBCL–YSR discrepancies related to maternal depression and adolescent self–esteem. Child: Care, Health and Development. 2003;29(3):207–213. doi: 10.1046/j.1365-2214.2003.00332.x. [DOI] [PubMed] [Google Scholar]
  5. Bird H, Gould M, Staghezza B. Aggregating data from multiple informants in child psychiatry epidemiological research. Journal of the American Academy of Child and Adolescent Psychiatry. 1992;31:78–85. doi: 10.1097/00004583-199201000-00012. [DOI] [PubMed] [Google Scholar]
  6. Cannon TD, Cadenhead K, Cornblatt B, Woods SW, Addington J, Walker E, Seidman LJ. Prediction of psychosis in youth at high clinical risk: a Multisite longitudinal study in North America. Archives of General Psychiatry. 2008;65:28–37. doi: 10.1001/archgenpsychiatry.2007.3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Carpenter WT, Sacks MH, Strauss JS, Bartko JJ, Rayner J. Evaluating signs and symptoms: Comparison of structured interview and clinical approaches. The British Journal of Psychiatry. 1976;128:397–403. doi: 10.1192/bjp.128.4.397. [DOI] [PubMed] [Google Scholar]
  8. Chambless DL, Blake KD. Construct validity of the perceived criticism measure. Behavior Therapy. 2009;40:155–163. doi: 10.1016/j.beth.2008.05.005. [DOI] [PubMed] [Google Scholar]
  9. Chambless DL, Steketee G. Expressed emotion and behavior therapy outcome: a Prospective study with obsessive–compulsive and agoraphobic outpatients. Journal of Consulting and Clinical Psychology. 1999;67:658. doi: 10.1037//0022-006x.67.5.658. [DOI] [PubMed] [Google Scholar]
  10. Choudhury MS, Pimentel SS, Kendall PC. Childhood anxiety disorders: Parent–Child (dis) agreement using a structured interview for the DSM-IV. Journal of the American Academy of Child & Adolescent Psychiatry. 2003;42:957–964. doi: 10.1097/01.CHI.0000046898.27264.A2. [DOI] [PubMed] [Google Scholar]
  11. Christensen A, Margolin G. Conflict and alliance in distressed and nondistressed families. In: Hinde RA, Stevenson-Hinde J, editors. Relationships within families: Mutual influences. Clarendon; Oxford: 1988. pp. 263–282. [Google Scholar]
  12. Christensen A, Margolin G, Sullaway M. Interparental agreement on child behavior problems. Psychological Assessment. 1992;4:419–425. [Google Scholar]
  13. Compton MT. Considering schizophrenia from a prevention perspective. American Journal of Preventive Medicine. 2004;26:178–185. doi: 10.1016/j.amepre.2003.10.003. [DOI] [PubMed] [Google Scholar]
  14. De Los Reyes A, Kazdin A. Informant discrepancies in the assessment of childhood psychopathology: a Critical review, theoretical framework, and recommendations for further study. Psychological Bulletin. 2005;131:483–509. doi: 10.1037/0033-2909.131.4.483. [DOI] [PubMed] [Google Scholar]
  15. Edelbrock C, Costello AJ, Dulcan MK, Kalas R, Conover NC. Age differences in the reliability of the psychiatric interview of the child. Child Development. 1985;56:265–275. [PubMed] [Google Scholar]
  16. Falloon IRH. General practice recruitment for people at risk for schizophrenia: the Buckingham experience. Australian and New Zealand Journal of Psychiatry. 2000;34:S131–S136. doi: 10.1080/000486700233. [DOI] [PubMed] [Google Scholar]
  17. First MB, Spitzer RL, Gibbon M, Williams JBW. Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Patient Edition. (SCID-I/P) Biometrics Research, New York State Psychiatric Institute; New York: 2002. [Google Scholar]
  18. Fusar-Poli P, Bonoldi I, Yung AR, Borgwardt S, Kempton MJ, Valmaggia L, et al. Predicting psychosis: Meta-analysis of transition outcomes in individuals at high clinical risk. Archives of General Psychiatry. 2012;69:220–229. doi: 10.1001/archgenpsychiatry.2011.1472. [DOI] [PubMed] [Google Scholar]
  19. Grills AE, Ollendick TH. Multiple informant agreement and the anxiety disorders interview schedule for parents and children. Journal of the American Academy of Child and Adolescent Psychiatry. 2003;42:30–40. doi: 10.1097/00004583-200301000-00008. [DOI] [PubMed] [Google Scholar]
  20. Handwerk M, Larzelere R, Soper S, Friman P. Parent and child discrepancies in reporting severity of problem behaviors in three out-of-home settings. Psychological Assessment. 1999;11:14–23. [Google Scholar]
  21. Ho BC, Flaum M, Hubbard W, Arndt S, Andreasen CN. Validity of symptom assessment in psychotic disorders: Information variance across different sources of history. Schizophrenia Research. 2004;68:299–307. doi: 10.1016/j.schres.2003.07.006. [DOI] [PubMed] [Google Scholar]
  22. Hooley JM, Teasdale JD. Predictors of relapse in unipolar depressives: Expressed emotion, marital distress, and perceived criticism. Journal of Abnormal Psychology. 1989;98:229–235. doi: 10.1037//0021-843x.98.3.229. [DOI] [PubMed] [Google Scholar]
  23. Jackson HJ, McGorry PD, McKenzie D. The reliability of DSM–III prodromal symptoms in first–episode psychotic patients. Acta Psychiatrica Scandinavica. 1994;90:375–378. doi: 10.1111/j.1600-0447.1994.tb01609.x. [DOI] [PubMed] [Google Scholar]
  24. Jensen PS, Xenakis SN, Davis H, DeGroot J. Child psychopathology rating scales and interrater agreement: II. Child and family characteristics. Journal of the American Academy of Child & Adolescent Psychiatry. 1988;27:451–461. doi: 10.1097/00004583-198807000-00013. [DOI] [PubMed] [Google Scholar]
  25. Joa I, Johannessen JO, Auestad B, Friis S, McGlashan T, Melle I, et al. The key to reducing duration of untreated first psychosis: Information campaigns. Schizophrenia Bulletin. 2008;34:466–472. doi: 10.1093/schbul/sbm095. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Kazdin AE, French NH, Unis AS. Child, mother, and father evaluations of depression in psychiatric inpatient children. Journal of Abnormal Child Psychology. 1983;11:167–179. doi: 10.1007/BF00912083. [DOI] [PubMed] [Google Scholar]
  27. Keshavan MS, Eack SM, Montrose DM, Abela MM, Bangalore SS, Diwadkar VA, Prasad KM. Do premorbid impairments predict emergent `prodromal' symptoms in young relatives at risk for schizophrenia? Early Intervention in Psychiatry. 2009;3:213–220. doi: 10.1111/j.1751-7893.2009.00135.x. [DOI] [PubMed] [Google Scholar]
  28. Kolko DJ, Kazdin AE. Emotional/behavioral problems in clinic and nonclinic children: Correspondence among child, parent and teacher reports. Journal of Child Psychology and Psychiatry. 1993;34:991–1006. doi: 10.1111/j.1469-7610.1993.tb01103.x. [DOI] [PubMed] [Google Scholar]
  29. Lee H, Schepp KG. Multiple informants in assessing stress and symptoms in adolescents with schizophrenia. Archives of Psychiatric Nursing. 2011;25:120–128. doi: 10.1016/j.apnu.2010.06.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Martin Bland J, Altman D. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;327:307–310. [PubMed] [Google Scholar]
  31. Martin JL, Ford CB, Dyer-Friedman J, Tang J, Huffman LC. Patterns of agreement between parent and child ratings of emotional and behavioral problems in an outpatient clinical setting: When children endorse more problems. Journal of Developmental & Behavioral Pediatrics. 2004;25:150-–155.. doi: 10.1097/00004703-200406000-00002. [DOI] [PubMed] [Google Scholar]
  32. McGlashan TH, Miller TJ, Woods SW, Rosen JL, Hoffman RE, Davidson L. Structured Interview for Prodromal Syndromes (SIPS) Version 4.0 Yale University; New Haven: 2003. [Google Scholar]
  33. McGlashan T, Walsh B, Woods S. The psychosis-risk syndrome. Handbook for diagnosis and follow-up. Oxford University Press; New York: 2010. [Google Scholar]
  34. McGorry PD, Edwards J. The feasibility and effectiveness of early intervention in psychotic disorders: the Australian experience. International Clinical Psychopharmacology. 1998;13:S47–S52. [Google Scholar]
  35. McGorry PD, Nelson B, Phillips LJ, Yuen HP, Francey SM, Thampi A, et al. Randomized controlled trial of interventions for young people at ultra-high risk of psychosis: Twelve-month outcome. The Journal of Clinical Psychiatry. 2012;74:349–356. doi: 10.4088/JCP.12m07785. [DOI] [PubMed] [Google Scholar]
  36. Meyer SE, Bearden CE, Lux SR, Gordon JL, Johnson JK, O'Brien MP, Niendham TA, Loewy RL, Ventura J, Cannon TD. The psychosis CHR in adolescent patients viewed through the lens of DSM-IV. Journal of Child and Adolescent Psychopharmacology. 2005;15:434–451.. doi: 10.1089/cap.2005.15.434. [DOI] [PubMed] [Google Scholar]
  37. Myers RH. Classical and modern regression with applications. Vol. 2. Duxbury Press; Belmont: 1990. [Google Scholar]
  38. Neumann CS, Grimes K, Walker EF, Baum K. Developmental pathways to schizophrenia: Behavioral subtypes. Journal of Abnormal Psychology. 1995;104:558–566. doi: 10.1037//0021-843x.104.4.558. [DOI] [PubMed] [Google Scholar]
  39. Niendam TA, Bearden CE, Zinberg J, Johnson JK, O'Brien M, Cannon TD. The course of neurocognition and social functioning in individuals at ultra high risk for psychosis. Schizophrenia Bulletin. 2007;33:772-–781. doi: 10.1093/schbul/sbm020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Nugent KL, Kline E, Thompson E, Reeves G, Schiffman J. Assessing psychotic–like symptoms using the BASC–2: Adolescent, parent and teacher agreement. Early intervention in psychiatry. 2013 doi: 10.1111/eip.12019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. O'Brien MP, Gordon JL, Bearden CE, Lopez SR, Kopelowicz A, Cannon TD. Positive family environment predicts improvement in symptoms and social functioning among adolescents at imminent risk for onset of psychosis. Schizophrenia Research. 2006;81:269–275. doi: 10.1016/j.schres.2005.10.005. [DOI] [PubMed] [Google Scholar]
  42. O'Brien MP, Zinberg JL, Ho L, Rudd A, Kopelowicz A, Daley M, et al. Family problem solving interactions and 6-month symptomatic and functional outcomes in youth at ultra-high risk for psychosis and with recent onset psychotic symptoms: a Longitudinal study. Schizophrenia Research. 2009;107:198–205. doi: 10.1016/j.schres.2008.10.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Penney SR, Skilling TA. Moderators of informant agreement in the assessment of adolescent psychopathology: Extension to a forensic sample. Psychological Assessment. 2012;24:386–401. doi: 10.1037/a0025693. [DOI] [PubMed] [Google Scholar]
  44. Perkins DO, Hongbin G, Boteva K, Lieberman JA. Relationship Between Duration of untreated psychosis and outcome in first-episode schizophrenia: a Critical review and meta-analysis. American Journal of Psychiatry. 2005;162:1785-–1804. doi: 10.1176/appi.ajp.162.10.1785. [DOI] [PubMed] [Google Scholar]
  45. Rapee RM, Barrett PM, Dadds MR, Evans L. Reliability of the DSM-III-R childhood anxiety disorders using structured interview: Interrater and parent–child agreement. Journal of the American Academy of Child & Adolescent Psychiatry. 1994;33:984–992. doi: 10.1097/00004583-199409000-00008. [DOI] [PubMed] [Google Scholar]
  46. Rey JM, Schrader E, Morris-Yates A. Parent–child agreement on children's behaviours reported by the Child Behaviour Checklist (CBCL) Journal of Adolescence. 1992;15:219–230. doi: 10.1016/0140-1971(92)90026-2. [DOI] [PubMed] [Google Scholar]
  47. Ruhrmann S, Schultze-Lutter F, Klosterkötter J. Early detection and intervention in the initial prodromal phase of schizophrenia. Pharmacopsychiatry. 2003;36:S162–S167. doi: 10.1055/s-2003-45125. [DOI] [PubMed] [Google Scholar]
  48. Ruhrmann S, Schultze-Lutter F, Salokangas RK, Heinimaa M, Linszen D, Dingemans P, Birchwood M, Patterson P, Juckel G, Heinz A, Morrison A, Lewis S, von Reventlow HG, Klosterk tter J. Prediction of psychosis in adolescents and young adults at high risk: Results from the prospective European Prediction of Psychosis study. Archives of General Psychiatry. 2010;67:241–251. doi: 10.1001/archgenpsychiatry.2009.206. [DOI] [PubMed] [Google Scholar]
  49. Safford SM, Kendall PC, Flannery-Schroeder E, Webb A, Sommer H. A Longitudinal look at parent–child diagnostic agreement in youth treated for anxiety disorders. Journal of Clinical Child & Adolescent Psychology. 2005;34:747–757.. doi: 10.1207/s15374424jccp3404_16. [DOI] [PubMed] [Google Scholar]
  50. Salbach-Andrae H, Lenz K, Lehmkuhl U. Patterns of agreement among parent, teacher and youth ratings in a referred sample. European Psychiatry. 2009;24:345–351.. doi: 10.1016/j.eurpsy.2008.07.008. [DOI] [PubMed] [Google Scholar]
  51. Schlosser DA, Zinberg JL, Loewy RL, Casey-Cannon S, O'Brien MP, Bearden CE, et al. Predicting the longitudinal effects of the family environment on prodromal symptoms and functioning in patients at-risk for psychosis. Schizophrenia Research. 2010;118:69–75. doi: 10.1016/j.schres.2010.01.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Smith SR. Making sense of multiple informants in child and adolescent psychopathology: a Guide for clinicians. Journal of Psychoeducational Assessment. 2007;25:139–149. [Google Scholar]
  53. Sourander A, Helstel L, Helenius H. Parent-adolescent agreement on emotional and behavioral problems. Social Psychiatry and Psychiatric Epidemiology. 1999;34:657–663. doi: 10.1007/s001270050189. [DOI] [PubMed] [Google Scholar]
  54. Tsuang MT, Stone WS, Faraone SV. Schizophrenia: a Review of genetic studies. Harvard Review of Psychiatry. 1999;7:185–207. [PubMed] [Google Scholar]
  55. Wyatt RJ, Henter I. Rationale for the study of early intervention. Schizophrenia Research. 2001;51:69-–76. doi: 10.1016/s0920-9964(01)00242-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Youngstrom E, Izard C, Ackerman B. Dysphoria-related bias in maternal ratings of children. Journal of Consulting and Clinical Psychology. 1999;67:905–916.. doi: 10.1037//0022-006x.67.6.905. [DOI] [PubMed] [Google Scholar]
  57. Youngstrom EA, Findling RL, Calabrese JR. Effects of adolescent manic symptoms on agreement between youth, parent, and teacher ratings of behavior problems. Journal of Affective Disorders. 2004;82:S5–S16. doi: 10.1016/j.jad.2004.05.016. [DOI] [PubMed] [Google Scholar]
  58. Yung AR, McGorry PD, McFarlane CA, Jackson HJ, Patton GC, Rakkar A. Monitoring and care of young people at incipient risk of psychosis. Schizophrenia Bulletin. 1996;22:283-–303. doi: 10.1093/schbul/22.2.283. [DOI] [PubMed] [Google Scholar]
  59. Yung AR, Nelson B, Stanford C, Simmons MB, Cosgrave EM, Killackey E, et al. Validation of “prodromal” criteria to detect individuals at ultra high risk of psychosis: 2 year follow-up. Schizophrenia Research. 2008;105:10–17. doi: 10.1016/j.schres.2008.07.012. [DOI] [PubMed] [Google Scholar]

RESOURCES