The development of the Content of Attenuated Positive Symptoms Codebook for those at clinical high risk of psychosis

Abstract

Examining the content of psychotic symptoms may be relevant for understanding symptom development particularly in those in the very early stages of psychosis. The Content of Attenuated Positive Symptoms codebook was developed as a means to overcome methodological issues in the current literature. Three separate samples at clinical high risk of psychosis were used to develop, modify and finalize the codebook. The inter-rater reliability was moderate to excellent. Specific content items were consistently endorsed across samples. This codebook offers a reliable way to assess content in the clinical high-risk population.

Introduction

Efforts to better understand psychosis have recently been made by studying those who appear to be at high risk for psychosis on the basis of clinical symptoms, typically attenuated positive symptoms (McGlashan, Walsh, & Woods, 2010; Yung & McGorry, 1996). Well-established criteria to identify those who may be at clinical high risk (CHR) for developing psychosis have been developed (McGlashan et al., 2010; Yung et al., 2005). Individuals who meet these criteria experience attenuated positive symptoms that are below the threshold of full-blown psychotic symptoms. Interestingly, the content of positive symptoms seems to have been deemed irrelevant as the topic is neglected in many research efforts. It is possible that examining the content of attenuated positive symptoms in CHR individuals might provide some relevant insights into the development of psychosis (Thompson et al., 2010).

Past studies that have focused on positive symptom content have approached the topic using a wide range of methodologies, making it difficult to obtain an accurate picture of the relevance of symptom content. First, there is diagnostic variation within samples, e.g.: schizophrenia samples that include both acute and relative remission states (Larkin, 1979), paranoid states, schizoaffective disorder (Lucas, Sainsbury, & Collins, 1962), psychosis in general (Close & Garety, 1998), alcoholic and medically ill patients (Mott, Small, & Anderson, 1965), and psychiatric (Andrew, Gray, & Snowden, 2008) and non-psychiatric voice hearers (Honig et al., 1998). Some studies examined the relationship of content with specific traits such as gender, socioeconomic status, marital status, and birth order (Lucas et al., 1962).

Secondly, there has been a lack of consistency in the definition of positive symptom content. Some studies used broad categories such as positive versus negative (Andrew et al., 2008; Close & Garety, 1998). Others have examined more specific content such as sexual (Gittleson & Dawson-Butterworth, 1967; Gittleson & Levine, 1966; Thompson et al., 2010), or content observed in auditory hallucinations (Honig et al., 1998). Studies often vary in the use of the same term, e.g. the definition of grandiosity differed in three separate studies (Lucas et al., 1962; Raune, Bebbington, Dunn, & Kuipers, 2006; Rudden, Sweeney, Frances, & Gilmore, 1983).

Thirdly, a wide range of methods have been used to elicit positive symptom content, for example, semi-structured interviews (Close & Garety, 1998; Honig et al., 1998; Larkin, 1979; Nayani & David, 1996), specifically developed interviews for specific information (Gittleson & Dawson-Butterworth, 1967; Gittleson & Levine, 1966; Mott et al., 1965; Thompson et al., 2010), chart reviews or staff interviews (Lucas et al., 1962; Read & Argyle, 1999) and a few included relatives (Lucas et al., 1962; Rudden et al., 1983).

Overall, there have been very few studies focusing on the positive symptom content and of those published there are methodological concerns and little replication. Recently, two studies have examined the content of attenuated positive symptoms in relation to past trauma in the CHR population (Thompson et al., 2009; Thompson et al., 2010). It has been suggested that symptom content may be understandable within the particular context of an individual’s life and potentially related to past experiences, possibly those related to trauma (Thompson et al., 2010). Thus, it may be important to examine the content of symptoms in those at CHR of developing psychosis: first, because of the current interest in the impact of past experiences on the development of psychosis (van Os, Rutten, & Poulton, 2008); and secondly, because of the unique opportunity to longitudinally observe symptom development in those at CHR. For the first time we may be able to study the content of attenuated positive symptoms in the context of past experiences to better understand symptom development. However, this requires a reliable method to assess the positive symptom content.

This paper describes the development of a codebook to assess the Content of Attenuated Positive Symptoms (CAPS). Through the development of the CAPS codebook, the aim was to develop a systematic and reliable coding scheme to explore explicit meanings in text through quantitative analysis (Neuendorf, 2002).

Methods

Design

The codebook was developed in four stages and was based on data from vignettes that were written to describe the symptom presentation of 121 subjects from three separate samples.

Development of the codebook

Stage 1 – Literature review

A systematic search of the literature was conducted by searching all peer-reviewed, English language citations on the databases: (a) Medline, (b) PubMed, and (c) Google Scholar. The key words used included “symptom content,” “content analysis,” “subjective,” “experience,” and “phenomenology” combined with “psychosis,” “schizophrenia” and “symptoms.” This search yielded 45 potentially relevant articles. A separate manual search was conducted which included a scan of the titles in the reference list of articles that appeared to be appropriate. This yielded a total of 68 articles, to give a total of 113 potentially relevant articles for review.

From the set of citations, each title was screened individually for potential relevance and relevant abstracts were reviewed individually. This process yielded 32 citations, for which full articles were obtained and reviewed. The criteria for this final review were that: (a) the content of the paper focused on themes, content and/or phenomenology of positive symptoms in schizophrenia spectrum disorders; and (b) the focus was on individuals with a chronic course of schizophrenia, first episode, or CHR populations. The 32 articles from the literature review were read for themes, content, and/or phenomenology of positive symptoms. From the literature six categories were relevant to unusual thought content, two to suspicious ideas, one to grandiose ideas, four to auditory experiences and one to visual, olfactory, and tactile experiences. These categories were used to create Version 1 of the CAPS codebook (See Table 1). Categories are defined as overarching themes of the content and items are specific descriptors of the categories. For example, a man is suspicious that an evil force might be following him at work. Here, types of suspicious ideas, characters, and locations are relevant categories. Items in those categories would be thoughts of being followed, evil spirits, and work.

Table 1.

Categories and items of content for the CAPS codebook.

Positive symptom	Category name	Number of items in each category in version 1	Number of items in each category in version 4
P1: Unusual thought content	Content of unusual thoughts	14	15
	Unusual sexual thoughts	18	7
	Unusual religious thoughts	6	3
	Unusual thoughts regarding self	5	2
	Unusual thoughts regarding world	2	NA
	Affective content in unusual thoughts	4	NA
P2: Suspicious ideas	Types of suspicious thoughts	6	10
	Character(s) in suspicious thoughts	6	7
	Location(s) in suspicious thoughts	NA	5
P3: Grandiose ideas	Content of grandiose thoughts	4	6
P4: Perceptual experiences Auditory	Type of unusual sound	NA	8
	Amount of negative content of voices	5	3
	Amount of positive content of voices	5	3
	Conversational types	17	NA
	Character(s) in conversations	8	6
Visual	Types of visual experiences	3	8
Olfactory	Types of olfactory experiences	2	9
Tactile	Types of tactile experiences	2	3

NA: not available for that version.

Description of sample and symptoms for stages 2–4 of codebook development

Sample

Three samples were used in this study. The samples came from three independent studies in Toronto and Calgary of individuals who were determined to be at CHR. Sample-1 came from the Access, Detection and Psychological Treatment study (Addington et al., 2008, 2011), Sample-2 came from the Calgary North American Prodrome Longitudinal study (Addington, 2011) and Sample-3 came from the Toronto Enhancing the Prospective Prediction of Psychosis study (Addington, Penn, Woods, Addington, & Perkins, 2007).

All samples were recruited in similar ways from a variety of sources including family physicians, and community mental health teams (Addington et al., 2008). Participants were between 12 and 35 years old and all met the Criteria of Prodromal States (COPS) using the Structured Interview for Prodromal Symptoms (SIPS) (McGlashan et al., 2010). Although the COPS includes diagnosis of three different CHR syndromes, we only included the vignettes of participants that met criteria for the Attenuated Positive Symptom Syndrome which includes the emergence or worsening of symptoms within the past year in at least one of unusual thought content, suspicious ideas, grandiosity, perceptual abnormalities, or disorganized communication. Exclusion criteria for participants in all of these studies were (a) met criteria for any current or lifetime axis I psychotic disorder, (b) had a prior history of treatment with an antipsychotic, (c) had an IQ italic> than 70 or (d) had a past or current history of a clinically significant central nervous system disorder which may confound or contribute to prodromal symptoms.

Symptoms

The symptoms under study are the attenuated positive symptoms from the Scale of Prodromal Symptoms (SOPS) (McGlashan et al., 2010), which is one of the most widely used scales. The symptoms are P1-unusual thought content, P2-suspicious ideas, P3-grandiosity, P4-perceptual abnormalities and P5-disorganized communication. We did not use P5 in this project since it is not relevant for symptom content. The descriptions of symptoms were taken from comprehensive vignettes developed prior to the baseline assessments for participants in all three samples. All raters of these participants demonstrated excellent reliability on the SOPS, which is the scale within the SIPS interview to assess symptom severity. The kappa statistic was used to compare raters’ agreement with “gold standard” ratings on the SOPS. Kappas for the different samples ranged from 0.95 to 0.98 for the SOPS positive symptoms (Addington, Cadenhead, et al., 2007).

Text selection

Vignettes were written based on the SIPS semi-structured interview and used to obtain a consensus diagnosis of a prodromal syndrome. All assessments were done by two trained clinical raters who collaborated to write the vignette. The vignette focused on relevant background information including family history of mental illness, DSM-IV diagnosis based on the SCID, the Global Assessment of Functioning ratings and each of the five attenuated positive symptoms in the SOPS. In the vignette both longstanding and recent positive symptoms were described in detail as well as the frequency, intensity and conviction of each symptom. A rating was provided for each symptom. The vignettes were written so that another rater could read the information provided and give a reliable rating for each symptom. All vignettes were presented either at a site meeting or on an international call (for multi-site studies), both of which included well-trained and reliable SIPS/SOPS raters. The purpose of the conferences was to make a consensus decision on the ratings of each vignette as well as the diagnosis. Questions about ratings or subsequent diagnosis would be raised and ultimately a consensus was reached on each symptom rating, diagnosis, and admittance into the particular study.

Units of data analysis

From Sample-1, 46 vignettes were available describing sub-threshold positive symptoms, 45 vignettes from Sample-2, and 30 vignettes from Sample-3. Each vignette was separated into four smaller units of data analysis: P1-unusual thought content, P2-suspicious ideas, P3-grandiose ideas, and P4-perceptual abnormalities. Most vignettes did not contain all the positive symptoms as typically participants did not endorse all four symptoms. There were 184 potential units of analysis from Sample-1, 180 from Sample-2, and 120 from Sample-3. The specific number of units of data analysis for each positive symptom is presented in Table 2. For all samples, each unit of data analysis was randomly selected through the use of computer-generated list and analyzed as an individual unit.

Table 2.

Inter-rater reliability for Samples 1, 2, and 3.

Sample	Number of units of data analysis	Number of items per unit of data analysis	Total number of items	Agreement (%)	Kappa
One
P1	27	56	1512	94.78	0.73*
P2	30	25	750	91.73	0.74*
P3	18	9	162	88.89	0.66*
P4	28	51	1428	98.18	0.89*
Two
P1	44	50	2200	96.45	0.78*
P2	41	30	1230	91.30	0.69*
P3	18	9	162	86.42	0.57*
P4	44	51	2244	96.08	0.79*
Three
P1	30	26	780	95.64	0.74*
P2	22	21	462	92.64	0.76*
P3	14	6	84	95.24	0.87*
P4	17	36	612	94.61	0.65*

^*p bold> 0.0001.

Stage 2 – Rater training and establishing relevancy to the CHR population

Three raters (CM, EF, MA) participated in the development of the codebook and coder training. In Stage 2, Version 1 of the CAPS codebook was tested on units of data analysis based on the vignettes from Sample-1. The first five randomized units of analysis for each positive symptom were selected for initial and joint coding by the three raters. The content in each unit of data analysis was reviewed so each item in the codebook could be rated as absent or present relevant to that unit. Discrepancies between raters were discussed and a consensus was reached. Next, the three raters independently rated the next five units of analysis for each positive symptom from Sample-1. The raters then discussed their ratings, any discrepancies and again a consensus was reached. Changes to Version 1 of the codebook were made. Changes included clarification of items to enhance agreement between raters and making items more relevant or typical for the content expressed by the CHR population. Additions included adding categories based on the presence of that category in more than one vignette and changing the wording of categories to better suit a CHR population. These changes and additions were often related to the fact that symptoms in the CHR population are attenuated and not held with conviction. A report of all changes made is available on request from JA (jmadding@ucalgary.ca). After Stage 1 and practice Stage 2, Version 2 of the CAPS codebook contained six categories for unusual thoughts, three for suspicious ideas, one for grandiosity, and eight for unusual perceptual experiences.

Stage 3 – Verification of relevancy to CHR population

Stage 3a – Independent coding of vignettes

Using Version 2 of the CAPS code-book, the remaining units of data analysis from the Sample-1 vignettes, were rated independently by two raters, who were blind to each other’s scores. This included 27 vignettes for unusual thought content, 30 for suspicious ideas, 18 for grandiose ideas and 28 for perceptual abnormalities. A third consensus rater was used when there were differences between the 2 raters so the frequency of specific items could be determined.

Stage 3b – Inter-rater reliability

Kappa was used to compare inter-rater agreement on the presence or absence of individual items in the codebook based on the vignettes from Sample-1. A total Kappa score was calculated for each positive symptom (See Table 2). Kappa scores were in the substantial to excellent range for all positive symptoms. Scores for unusual thought content (k = 0.73), suspicious ideas (k = 0.74), and grandiose ideas (k = 0.658) were in the substantial range. The score for perceptual abnormalities (k = 0.89) was in the excellent range.

Stage 3c – Frequency of items

A frequency analysis was completed to determine the presence of each item across participants for each positive symptom. Details of the frequencies of all items are available on request from JA.

Stage 3d – Codebook modification

Based on stages 3b and 3c, items in codebook Version 2 were reviewed for suitability and/or relevance and were either merged with other items, reworded or deleted based on a consensus of all raters and JA. (Decisions made available from JA). This led to Version 3 of the CAPS codebook.

Stage 4 – Additional verification to the CHR population

Stage 4a – Independent coding of vignettes with a second sample

Version 3 of the CAPS codebook was tested using the vignettes from Sample-2. All units of data analysis were randomized and then coded independently by two raters who were blind to each other’s scores. A third consensus rater was used if needed to determine the item frequency. Sample-2 included 44 vignettes for unusual thought content, 41 for suspicious ideas, 18 for grandiose ideas and 44 for perceptual abnormalities.

Stage 4b – Inter-rater reliability

For Sample-2 vignettes, a total Kappa score was calculated for each positive symptom (See Table 2). Kappa scores were in the moderate to substantial range for all positive symptoms. The score for grandiose ideas (k = 0.57) was in the moderate range. Scores for unusual thought content (k = 0.78), suspicious ideas (k = 0.69), and perceptual abnormalities (k = 0.79) were in the substantial range.

Stage 4c – Frequency of items

A frequency analysis was completed to determine the presence of each item across participants for each positive symptom. A comparison was made between the results from Sample-1 and Sample-2 regarding the frequency of specific items. If there were items within the codebook that demonstrated a low frequency in both samples, the item was again either merged with other items, reworded, or deleted from the codebook (Decisions made available from JA). This led to Version 4 of the CAPS codebook.

Stage 5 – Verification of final codebook

Stage 5a – Independent coding of vignettes with a third sample

Version 4 of the CAPS codebook was tested using the vignettes from Sample-3. All units of data analysis were randomized and coded independently by two raters, who were blind to each other’s scores. A third consensus rater was used if needed to determine the frequency of the items. For this coding we introduced two new raters who were trained to use the codebook as described for Sample-1. Sample-3 included 30 vignettes for unusual thought content, 22 for suspicious ideas, 14 for grandiose ideas, and 17 for perceptual abnormalities.

Stage 5b – Inter-rater reliability

For Sample-3 vignettes, a total Kappa score was calculated for each positive symptom (See Table 2). The inter-rater reliability analysis revealed that the Kappa scores were in the substantial to excellent range for all positive symptoms. Scores for unusual thought content (k = 0.74), suspicious ideas (k = 0.76), and perceptual abnormalities (k = 0.65) were in the substantial range and the score for grandiose ideas (k = 0.87) was in the excellent range.

Stage 5c – Frequency of items

A frequency analysis was completed to determine the presence of items in the final version of the codebook across participants for each positive symptom (Available from JA). The final version of the CAPS code-book outline is presented in Table 1. A full copy is available on request from JA. Figure 1 summarizes the development of the codebook.

Figure 1.

Development of the CAPS codebook.

Results

Reliability

The reliability for rating positive symptom content across the three samples was in the moderate to excellent range. Only one score, P3 in Sample-2, was in the moderate range. Two scores, P4 in Sample-1 and P3 in Sample-3 were in the excellent range. All other scores across the three samples were in the substantial range. Modifications to the codebook improved reliability for P2 and P3 while P1 remained consistent. There was a drop in reliability for P4 although it was still in the substantial range. This may have been a result of the significant reduction in the number of items. Overall, the reliability was very good for the final version particularly considering we added two new raters.

Frequencies

Various items were found to be more frequently endorsed across the three samples compared to other items. Under unusual thought content, being perplexed by reality, thoughts of the supernatural, loss of control of thought, special attention from others, and overvalued beliefs, such as superstitions and meaningful coincidences, were most frequently endorsed. Thoughts with sexual, religious, and superstitious content, as well as thought interference were infrequently endorsed.

Under suspicious ideas, participants most often endorsed content that reflected that others were thinking negatively of them or wanting to emotionally harm them, that they were being watched, or felt the need to be hypervigilant. In no specific order, friends, acquaintances and strangers were most often the focus of suspicious ideas, although many times actual persons were not specified. The location for suspicious ideas was typically school, home or work.

Under grandiose thoughts, content reflecting status, intelligence, and skills/talents/abilities were most often endorsed, whereas content reflecting the ability to influence/help others or having unrealistic goals/plans were endorsed less frequently.

Under perceptual abnormalities, the majority of participants who heard voices could not specify if the content of the voices was pleasant or unpleasant. Participants most often thought they themselves were the focus of the voices’ comments. Indistinct sounds like ringing/banging/hissing/clapping, hearing one’s name being called and other distinct noises, like footsteps, were also frequently endorsed auditory experiences. The most frequently endorsed content of visual perceptual abnormalities included an increased sensitivity to light, seeing spots/floaters, vague figures/shadows, and faces/people. Olfactory experiences were most often reported to be unpleasant. The most common content of tactile experiences were aches and pains, a touching sensation, numbness/tingling, and burning/coldness.

Discussion

This paper describes the development of a codebook to assess the content of attenuated positive symptoms in those at CHR for psychosis as a means to begin to study the development of early psychotic experiences. With little previous research to guide the process, vignettes from three separate samples were used to develop the CAPS codebook to produce a reliable way to examine the content that would overcome past methodological issues. The codebook offers an objective means to assess content so that it could be explored in a standardized way both within and between studies. The current results demonstrate that high levels of inter-rater reliability can be reached when rating content using a codebook.

We report for the first time the range and frequency of content that is reported by these young people at CHR. Comparison of our results with samples of individuals with an established schizophrenia illness is difficult due to the differences in attenuated positive symptoms. Sexual content was infrequently found across the current samples contrary to other CHR studies who have reported a high level of sexual content in the CHR population (Thompson et al., 2010). It is possible that such differences may be accounted for by the variation in the methods used to explore content and that Thompson and colleagues were focusing specifically on sexual content and those who had experienced sexual trauma (Thompson et al., 2009; Thompson et al., 2010).

There are several strengths in the methods used to develop the codebook. First, all of the data collected on the attenuated positive symptoms across the three samples was completed using semi-structured interviews and well-established measures (SIPS/SOPS). All participants in all three samples met the same diagnostic criteria which were conducted with excellent reliability and thus excellent consistency with respect to the samples. Prior to assessing content, items were randomized so that content and not subjects were being assessed. With each sample the codebook was modified. Finally, this is the first study to our knowledge that has reported the inter-rater reliability for a content assessment measure.

There are limitations to this study. First, the vignettes used were not originally written for the purpose of exploring symptom content. Therefore, it is possible that there are some details that were absent in the vignettes that may have contributed to the overall content. Similarly, there may have been content not asked about in the actual interviews. However, in determining whether a symptom meets the relevant criteria a rater has to elicit as much information as possible to determine the rating. Although the expectation for detail was standardized across participants, there was variation in the detail provided in the vignette which varied across both participants and samples. Generally, the later studies had more relevantly detailed vignettes.

There are limitations to the codebook itself. First, items added were based on clinical relevance for the CHR population and clinical experience, therefore there may be items that did not occur in our samples and would have relevance for other samples. Secondly, three of the five raters were part of the development group. However, raters of the codebook were not all trained to rate on the SIPS/SOPS and using a codebook to rate from vignettes does not require the same extensive training as to elicit content through a structured interview.

The clinical implication of this study is that the use of a reliable measure, like the CAPS codebook, would allow clinicians to explore content relevant to those at CHR since it is the “at risk” samples from whom we may learn the most about symptom development.

In summary, this is a first effort to develop an objective measure to assess symptom content. Clearly, the codebook needs to be tested in different settings with other populations to assess its validity and reliability. It was based on the SIPS/SOPS and could potentially be modified so that it readily captures content for symptoms in other equally well-established and used measures such as the Comprehensive Assessment for At Risk Mental States which was developed by the Melbourne group (Yung et al., 2005). Therefore although this is an important first step it requires further input from other studies and research groups.