Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Oct 1.
Published in final edited form as: J Dual Diagn. 2012 Sep 10;8(4):305–314. doi: 10.1080/15504263.2012.723311

Predictors of Self-Stigma in Schizophrenia: New Insights Using Mobile Technologies

Dror Ben-Zeev 1,2, Rochelle Frounfelker 1, Scott B Morris 3, Patrick W Corrigan 3
PMCID: PMC3584451  NIHMSID: NIHMS412404  PMID: 23459025

Abstract

Objective

Self-stigma has significant negative impact on the recovery of individuals with severe mental illness, but its varying course is not well understood. Individual levels of self-stigma may vary over time and fluctuate in response to both external/contextual (i.e., location, activity, social company) and internal (i.e., psychiatric symptoms, mood) factors. The aim of this study was to examine the relationship between self-stigmatizing beliefs and these factors, as they occur in the daily life of individuals with schizophrenia.

Methods

Mobile technologies were used to longitudinally track momentary levels of self-stigma, psychotic symptoms, negative affect, positive affect, activity, and immediate social and physical environment in twenty-four individuals with schizophrenia, multiple times daily, over a one-week period.

Results

Multilevel modeling showed that participants’ current activity was associated with changes in self-stigma (χ2= 10.53, p <0.05), but immediate location and social company were not. Time-lagged analyses found that increases in negative affect (β=0.11, p<0.01) and psychotic symptom severity (β=0.16, p<0.01) predicted increases in the intensity of self-stigmatizing beliefs. Psychotic symptoms were found to be both an antecedent and a consequence (β=0.08, p<0.01) of increased self-stigma.

Conclusions

Our findings support a framework for understanding self-stigma as an experience that changes based on alterations in internal states and external circumstances. Mobile technologies are an effective methodology to study self-stigma and have potential to be used to deliver clinical interventions.

Keywords: mobile technologies, stigma, schizophrenia, ecological momentary assessment (EMA), mHealth

Severe psychiatric illnesses can have devastating effects on the lives of individuals coping with them and their family members. In addition to the negative impact of the psychopathological symptoms and cognitive impairment that many psychiatric conditions often entail, individuals with psychiatric illnesses such as schizophrenia also contend with stigma: the public stereotypes, prejudice, and discrimination associated with being labeled mentally ill (Corrigan & Ben-Zeev, 2011; Corrigan & Kleinlein, 2005; Link & Phelan, 2006). When individuals with mental illness endorse and internalize the negative stereotypes and prejudices associated with having mental health conditions, self-stigmatizing beliefs emerge (Corrigan & Watson, 2002). Self-stigma is associated with decreased self-esteem and self-efficacy, poorer social and vocational functioning, increases in psychiatric symptom severity, and decreases in treatment adherence (Cavelti, Kvgic, Beck, Rusch & Vauth, 2011; Kleim et al., 2008; Sirey et al., 2001; Wahl, 1999; Watson, Corrigan, Larson & Sells, 2007; Yanos, Roe & Lysaker, 2010; Yanos, Roe, Markus & Lysaker, 2008). While estimates vary, recent studies in Europe and the United States found prevalence rates of self-stigmatizing beliefs among individuals with severe mental illness to be 42% and 36%, respectively (Brohan, Elgie, Satorius & Thornicroft, 2010; West, Yanos, Smith, Roe & Lysaker, 2011;).

Self-stigma may change within individuals over time (Dalky, 2011; Griffiths, Christensen, Jorm, Evans & Groves, 2004; Macinnes & Lewis, 2008; Michalak et al., 2011). Culture-specific norms and values play an important role in determining the public attitudes held towards mental illness in one’s environment (Abdullah & Brown, 2011; Lee, 2002; Pescosolido, Martin, Lang & Olafsdottir, 2008; Weiss, Jadhav, Ruguram, Vounatsou & Littlewood, 2001). Given the close link between public stigma and self-stigma, certain contexts may therefore be more harmful, while others, more protective. In addition, internal experiences such as psychiatric symptom exacerbation or changes in affect may influence one’s sense of self-efficacy and self-worth (Livingston & Boyd, 2010; Lysaker, Davis, Warman, Strasburger & Beattie, 2007; Markowitz, Angell & Greenberg, 2011). Examining the relationships between these time-varying and potentially malleable risk factors and self-stigmatizing beliefs may help inform interventions that can prevent or reduce self-stigma and mitigate its consequences.

Prior research has been methodologically limited in its ability to capture the various micro-level variables that may impact self-stigma and change constantly over the course of a typical day in the lives of people with mental illness, including immediate context (location, activity, social milieu), psychiatric symptom severity, and affect. Mobile technologies offer innovative opportunities to examine these relationships. Data collection strategies such as Ecological Momentary Assessment (Shiffman, Stone & Hufford, 2008) or Experience Sampling Methods (Csikszentmihalyi & Larson 1987; Delespaul & deVries, 1987) use mobile devices to prompt participants to complete self-report measures, multiple times a day, over multiple days or weeks. Computerized mobile research strategies that integrate prompting, assessment, and data storage in a single device have been shown to be feasible, acceptable, and valid methods of data collection in individuals with schizophrenia (Granholm, Loh & Swendsen, 2008). Several studies have demonstrated that even participants who are quite symptomatic and endorse paranoid ideation and hallucinations (Ben-Zeev, Ellington, Swendsen & Granholm, 2011; Delespaul, deVries, & van Os, 2002), substance abuse (Henquet, Murray, Linszen & van Os, 2005; Swendsen, Ben-Zeev, & Granholm, 2011), depression and anxiety (Ben-Zeev, Morris, Swendsen & Granholm, 2011), and acute daily stress (Kimhy et al., 2010) are not deterred by using mobile devices for self-monitoring in real-time and in their own environment, even for extended periods of several months (Granholm, Ben-Zeev, Link, Bradshaw & Holden, 2011). The current study is the first to use mobile technologies to examine the relationships between self-stigma and other dynamic factors (i.e., psychotic symptoms, affect, location, company, activity) as they occur in real-time and in real-world settings.

METHODS

Participants

This study was conducted in accordance with the Declaration of Helsinki and approved by the institutional review boards of the Illinois Institute of Technology and Thresholds Psychiatric Rehabilitation Centers. Twenty-four individuals participated in the study. Participants were recruited from ads placed on bulletin boards at Thresholds Psychiatric Rehabilitation Centers, in the National Alliance on Mental Illness newsletter, and on public bulletin boards in Chicago. Participants met criteria for a DSM-IV diagnosis of schizophrenia or schizoaffective disorder, had a mean age of 44.87 years (SD= 9.27), and were predominantly male (71%), never married (71%), and African-American (79%). Seventy-five percent graduated high school or had some college education. The mean length of psychiatric illness reported by these participants was 16.95 years (SD=8.50; range= 4–36 years). All participants completed the study successfully and returned the Personal Digital Assistant (PDA) units intact at the end of the week.

Procedures

There was a complete discussion of the study with potential participants and written, informed consent was obtained after this discussion. All research candidates were administered a structured diagnostic interview, the Structured Clinical Interview for DSM-IV Axis I Disorders-Patient Edition (SCID-I/P; First, Spitzer, Gibbon & Williams, 1995), in order to verify diagnosis of schizophrenia or schizoaffective disorder and to assess additional lifetime and current mental disorders. Those who met the inclusion criteria then completed a battery of self-report and interview measures that included demographic information, the Beck Depression Inventory II (BDI-II; Beck, Steer & Brown, 1996), and the Beck Anxiety Inventory(BAI; Beck & Steer, 1993). They were also rated by trained assessors with the Scales for the Assessment of Positive and Negative Symptoms (SAPS and SANS, respectively) of schizophrenia (Andreasen, 1984; Andreasen & Olsen, 1982).

Following assessment, a 30-minute training session was provided concerning the meaning of all computerized Ecological Momentary Assessment questions and response choices, as well as training on how to operate and charge a PDA programmed to run the Experience Sampling Program version 4.0 (ESP; Barrett & Barrett), a free open-source Ecological Momentary Assessment software package. Participants were given a Tungsten E2 PDA that runs on a PALM OS Garnet (V.54) operating system and asked to carry it with them over a 1-week period. The PDAs were programmed to generate a prompting auditory signal 6 times every day in intervals that were spread out equally +/− 10 minutes (determined randomly) between 9 a.m. and 10 p.m. Approximately equal intervals were used to ensure that data were collected from all parts of the day, and that the participant response burden was spread out as much as possible. Participants were instructed to complete the mobile measure that appeared on the PDA screen following the prompt. Mobile assessments took approximately 2–3 minutes to complete and all responses were automatically stored in the PDA memory. Participants had a 5-minute window to complete questionnaires following each prompt after which the PDA would go into a hibernation mode and participants could not access the questionnaire for delayed reporting. All responses were time/date stamped electronically and items that were not completed were stored as missing data. Two practice mobile assessments were completed in the laboratory under the supervision of the research staff in order to resolve any difficulties or address questions. Each PDA had a label with the lab’s contact number to call in case of technical difficulties or with any questions during the week. All messages received were responded to within 24 hours by research staff. Participants were also called by a research assistant 48 hours after commencing, and 24 hours before their scheduled second lab visit, to troubleshoot any problems and to remind them of their return appointment.

During the second visit, all participants were debriefed and asked whether any major or unexpected life events took place during the week of data collection (e.g., unexpected health concerns, moving to a new residence) that might have contributed to response patterns that were atypical for them. Data identified as anomalous in the debriefings were not included in the analyses. There were only two such instances: when a PDA accidentally prompted signaling after 10 p.m., and when a participant described an occasion when a family member accidentally responded to the prompt.

Measures

Mobile Assessment

Participants were asked to complete a self-report questionnaire that assessed multiple domains including current location, company, activity, positive affect, negative affect, psychotic symptoms, and self-stigma. The topic of focus for each item was presented in capital letters, so that after a few administrations participants could identify the question and respond quickly (see Table 1). Participants could not skip over questions and needed to enter their answers by using the PDA stylus (“pointer”) to tap on 1 of 5 response boxes before being able to continue to the next question on the following screen. Participants were instructed to select the “other” response in situations when they thought the multiple choice options provided did not capture their current activity well, or they were engaged in several activities simultaneously (e.g., watching television and eating). When a questionnaire was completed, the participant was notified, and the PDA would “hibernate” until the next signal. Affect items were selected from the Positive and Negative Affect Schedule(PANAS) (Watson, Clark & Tellegen, 1988). Psychotic symptom items were modelled after the Psychotic Symptom Rating Scales (PSYRATS) (Haddock, McCarron, Tarrier & Faragher, 1999) and mobile assessment questions that have been shown to have convergent validity with the PANSS positive symptom subscale in previous studies (Granholm et al., 2008). Self-stigma items were selected from the Self Stigma of Mental Illness Scale (SSMIS) (Corrigan, Watson & Barr, 2006).

Table 1.

Mobile Assessment Question Items and Response Options

Questions Touch-screen response options
Location

  • WHERE are you right now?

Company

  • At this moment, who is WITH you?

Activity

  • At this moment, what are you DOING?

  • [0] At my home [1] At a friend or family member’s home [2] At work or a place of business [3] At a clinic or other mental health setting [4] Outside

  • [0] I am alone [1] A family member [2] A friend or spouse [3] A co-worker or classmate [4] A doctor, case manager, or other mental health professional

  • [0] Inactive (TV, music, resting) [1] Eating, dressing, hygiene [2] Shopping, chores, cooking [3] Work, school, or active leisure [4] Other
Positive Affect (PA)

  • How ACTIVE do you feel right now?

  • How INTERESTED do you feel right now?

  • How PROUD do you feel right now?

  • How EXCITED do you feel right now?

Negative Affect (NA)

  • How JITTERY do you feel right now?

  • How SCARED do you feel right now?

  • How HOSTILE do you feel right now?

  • How UPSET do you feel right now?”

  • How GUILTY do you feel right now?”

  • [0] Slightly or not at all [1] A little [2] Moderately [3] Quite a bit [4] Extremely
Symptoms

  • Since the last questionnaire, how strongly did you believe someone was SPYING or PLOTTING against you?

  • Since the last questionnaire, how strongly did you believe you could READ others peoples THOUGHTS or they could read yours?

  • Since the last questionnaire, how strongly did you believe others could communicate with you through the TV or RADIO?

  • Since the last questionnaire, how strongly did you believe you had SPECIAL POWERS and could do things nobody else could do?

  • Since the last questionnaire, to what extent did you hear VOICES that others could not hear?

  • [0] Not at all [1] A little [2] Moderately [3] Quite a bit [4] Extremely
Self-Stigma

  • How strongly do you agree with this statement? “Because I have a mental illness, I am DIRTY and UNKEMPT”

  • How strongly do you agree with this statement?

    “Because I have a mental illness, I am unable to get or keep a REGULAR JOB”

  • How strongly do you agree with this statement?

    “Because I have a mental illness, I am UNPREDICTIBLE”

  • How strongly do you agree with this statement?

    “Because I have a mental illness, I am to BLAME for my problems”

  • [0] Not at all [1] A little [2] Moderately [3] Quite a bit [4] Extremely
Open in a new tab

Overview of Analyses

Four continuous composite scores were created for the analyses from momentary mobile assessment items: a Positive Affect score was calculated as the sum of 4 positive mood items (active, interested, proud, excited) and a Negative Affect score as the sum of 5 negative mood items (jittery, scared, hostile, upset, guilty). A Symptoms score was calculated as the sum of 5 psychotic symptom items. A Self-Stigma score was calculated as the sum of 4 self-stigma items.

Multilevel modeling of longitudinal data was conducted to explore within-person time-varying relationships between the different predictors and stigma. The multilevel approach has a number of features that make it ideal for the analysis of time-varying relationships (Hox, 2010). First, the analysis disentangles within-person and between-person variance, thereby allowing examination of relationships as they unfold within an individual over time, unconfounded by stable differences between persons. Second, the analysis accounts for the non-independence of observations within persons, and allows for flexible specification of covariance structure of within-person residuals (e.g., heterogeneous variance or autocorrelation). Third, the analysis is able to appropriately handle data where each individual provides a different number and spacing of repeated observations, a common challenge in event sampling research. Analyses were conducted using HLM 6.08 (Raudenbush, Bryk, Cheong, & Cogdon, 2004).

The model consisted of three levels: observations within day within person. Random intercepts were allowed to vary across days and persons; while all other predictors had fixed (non-varying) coefficients. In order to examine time-varying relationships, all predictors were entered at the lowest level of analysis. No predictors were included in the level 2 (day) or level 3 (person) models.

In longitudinal data, the assumption that errors are independent and identically distributed is often violated (Singer & Willet, 2003). Initial analyses were conducted to identify the best fitting error structure for the repeated measures. A model with heterogeneous within-person error variance was found to fit the data best for perceptions of stigma; while for symptoms, a first-order autoregressive error structure was selected.

The first set of analyses examined the effects of current contextual factors on changes in self-stigma. A time-lagged approach was used, predicting time t stigma from current context, while controlling for stigma at time t-1. As such, the coefficients for the contextual factors reflect the relationship between current context and change in the outcome. Prior stigma was centered around the grand mean.

Predictors (i.e., Location, Company, and Activity) were included in the analyses as categorical variables and were represented as a set of dummy codes, with the reference categories defined as being “at home”/”alone”/“inactive.” Coefficients in the regression models reflect the difference between the indicated category and the reference category. For each categorical predictor, an omnibus test was initially conducted using a multiple-parameter chi-square test to determine if there were any differences across categories. If the omnibus test was significant, all pairwise comparisons were tested using contrasts among the regression coefficients in the HLM model.

Omnibus tests were evaluated for significance at the .05 level. To minimize type I error buildup across the 10 pairwise comparisons within each categorical predictor, a Bonferroni correction was applied. To minimize power loss, the Bonferroni adjustment was applied to a liberal alpha level of .10, resulting in an adjusted significance level of .10/10 = .01 for the pairwise comparisons.

The second set of analyses examined psychiatric symptoms as both predictors and outcomes of Self-Stigma. Given the important role affect plays in both psychotic symptoms (Ben-Zeev et al., 2011; Freeman & Garety, 2003; Smith et al., 2006) and beliefs about one’s self (Ben-Zeev, Granholm, & Cafri, 2009; Thewissen et al., 2011), Positive Affect and Negative Affect were also included as predictors in these analyses. A time-lagged approach was used, predicting time t outcomes from predictors at both time t-1 and time t, controlling for the outcome at time t-1. Because the current time predictors were examined controlling for their previous values, the coefficients for current time predictors represent the extent to which change in predictors is associated with change in the outcome. Measurement occasions with missing data on the relevant variables at either time t or time t-1 were excluded from each analysis. Time-lagged predictors were included only within the same day (i.e., overnight time lags were excluded), reducing the number of possible observations to 5 per day. The number of usable observations for each participant ranged from six to thirty-five (M=31), with a total of 738 observations. All predictors were centered around the grand mean.

RESULTS

Descriptive characteristics

At baseline, the clinical sample experienced on average mild positive symptom (M=23.58, SD=18.86, range= 0–74) and negative symptom (M=10.54, SD=10.95, range=0–40) severity, mild to moderate symptoms of anxiety (M= 15.83, SD= 8.44, range=2–33), and moderate depression symptom severity (M=21.20, SD=12.54, range= 0–51). Participants received an average of 41.4 prompting signals during the week and complete data were available for all question items on 88% of the observations. Based on their responses to the repeated mobile assessments, participants spent most of their time at home (63%). They were often alone (60%) or with family (20%), and spent much of their time inactive (39%), eating (21%) or engaged in “other” activities (20%). The majority of participants (83%) reported experiencing symptoms of psychosis at some stage of the data collection week.

Means, standard deviations and correlations among the study variables are reported in Table 2. In order to fully represent the multi-level structure of the data (i.e., within-person variation over time and individual differences between persons), descriptive statistics are provided representing both within-person and between-person components. Within-person correlations reflect the extent to which momentary assessments of different variables are related across time, whereas between-person correlations reflect the extent to which an individual’s average score is related to the average on other variables.

Table 2.

Descriptive Statistics

Ma SD Within SD Between Correlation
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
1. Location: Home 0.63 0.45 0.18 −.58 −.62 −.25 −.33 .28 .17 −.43 −.36 −.20 .44 −.16 .04 −.52 −.14 .01 .37 .14 .32 - - - -
2. Location: Friend’s Home 0.08 0.25 0.10 −.34 .01 .08 −.03 −.42 .19 .23 .32 .13 −.07 .02 −.09 .40 −.22 −.01 −.31 −.02 −.18 - - - -
3. Location: Work 0.11 0.28 0.15 −.41 −.13 −.23 −.17 −.08 −.11 .30 .15 −.26 −.31 .24 .00 .16 .13 .01 −.11 −.09 −.10 - - - -
4. Location: Clinic 0.05 0.20 0.04 −.30 −.08 −.05 .21 −.11 .14 −.24 −.01 .67 −.12 −.01 .00 .05 .19 −.02 −.39 −.08 −.24 - - - -
5. Location: Outside 0.14 0.34 0.05 −.55 −.13 −.14 −.11 .12 −.56 .36 .17 .32 −.31 −.06 .03 .38 .27 −.01 −.01 −.07 −.16 - - - -
6: Company: Alone 0.60 0.45 0.19 .27 −.25 −.11 −.21 .04 −.67 −.53 −.44 .06 −.39 .36 .29 −.04 .15 .18 .37 .08 .35 - - - -
7. Company: Family 0.20 0.37 0.15 −.04 .33 −.11 −.10 −.04 −.60 −.18 −.04 −.14 .59 −.36 −.06 −.38 −.24 −.02 −.35 −.20 −.07 - - - -
8. Company: Friend 0.12 0.30 0.13 −.05 .05 .04 .08 −.05 −.43 −.18 .33 −.20 .07 −.15 −.26 .25 −.04 −.15 .02 .13 −.35 - - - -
9. Company: Coworker 0.04 0.18 0.05 −.22 −.03 .28 .18 −.03 −.20 −.11 −.14 −.08 −.16 .00 −.30 .52 −.04 −.23 −.27 .10 −.23 - - - -
10. Company: Doctor 0.05 0.21 0.03 −.27 −.08 .14 .37 .09 −.31 −.11 −.07 −.04 −.45 .17 .02 .23 .39 .02 −.08 −.12 −.04 - - - -
11. Activity: Inactive 0.39 0.44 0.22 .42 .05 −.25 −.15 −.30 .17 −.04 −.03 −.16 −.13 −.65 −.26 −.61 −.32 −.15 .01 .11 −.01 - - - -
12. Activity: Eating 0.21 0.38 0.15 .14 .05 −.08 −.11 −.10 −.03 .12 .01 −.03 −.14 −.36 −.15 .34 −.25 .05 −.03 −.13 −.05 - - - -
13. Activity: Chores 0.08 0.27 0.05 −.10 .04 .08 −.02 .05 −.04 .07 .02 −.01 −.07 −.26 −.15 −.01 .12 .26 .13 .10 .14 - - - -
14. Activity: Working 0.12 0.31 0.11 −.28 −.05 .29 .16 .08 −.17 −.05 .05 .34 .10 −.24 −.27 −.11 −.22 .19 −.16 .05 .01 - - - -
15. Activity: Other 0.20 0.38 0.12 −.32 −.09 .07 .17 .33 −.01 −.09 −.03 −.06 .26 −.41 −.25 −.18 −.18 −.10 .09 −.14 −.02 - - - -
16. Stigma (current) 2.28 1.35 2.07 −.04 .02 .03 −.06 .05 .02 −.03 .00 −.05 .05 −.06 −.06 .04 .01 .09 .23 −.14 .55 - - - -
17. Symptoms (current) 3.57 1.72 4.09 .01 −.04 .06 −.01 −.03 −.06 .01 .06 −.04 .06 −.01 −.05 .14 −.06 .00 .25 .15 .53 - - - -
18. Positive Affect (current) 5.59 2.52 3.75 −.18 −.03 .12 .12 .09 −.15 .04 .00 .16 .09 −.19 −.08 .01 .26 .08 .04 −.01 .15 - - - -
19. Negative Affect (current) 2.19 1.94 2.15 .01 −.04 −.04 .02 .04 −.05 .05 .00 −.04 .04 −.06 .00 .06 −.02 .04 .24 .28 .07 - - - -
20. Stigma (time t-1) 2.28 1.36 2.02 −.06 .06 −.01 −.04 .06 .00 −.03 −.02 .01 .09 .04 −.08 −.02 .05 .00 .23 .09 .00 .06 - - -
21. Symptoms (time t-1) 3.57 1.82 4.04 −.07 .02 .06 .01 .02 −.03 −.05 .03 .04 .09 .01 −.05 .01 .00 .02 .20 .39 .02 .12 .16 - -
22. Positive Affect (time t-1) 5.59 2.70 3.65 .00 −.05 .03 .02 .00 .01 .02 −.08 .06 .02 .00 .01 −.03 .03 −.01 −.05 −.03 .20 .00 −.02 −.01 -
23. Negative Affect (time t-1) 2.18 1.95 2.10 −.02 .03 −.03 −.01 .03 .01 −.07 .05 .04 .00 .03 −.01 .03 −.01 −.04 .14 .05 −.04 .07 .21 .23 .06
Open in a new tab

Note. The lower triangle reports within-person correlations; the upper triangle reports between-person correlations. Because time varied only within-persons, lagged variables are not reported for between-subject correlations.

a

Means for categorical indicators represent the proportion of participants who indicated each category.

In order to verify that self-stigma and symptoms showed meaningful fluctuations over time, the variance components associated with each level of analysis were obtained from a 3-level HLM model (occasions nested within day nested within person) with no predictors. Self-Stigma and Symptom ratings varied both between (74% for Self-Stigma, 84% for Symptoms) and within individuals (18% within day and 8% across days for Self-Stigma, 9% within day and 7% across days for Symptoms), supporting the analysis of within-person relationships.

Immediate Contextual Factors

The first set of within-person analyses examined context variables as concurrent predictors of change in Self-Stigma. Separate analyses were conducted for each type of context variable (i.e., Location, Company, and Activity). Table 3 summarizes the results for HLM of contextual factors and Self-Stigma. Current location and company were not significantly associated with change in Self-Stigma. Activity was a significant predictor of change in Self-Stigma. Self-Stigma level increased on average 0.30 points (SD =0.12) when doing unspecified ‘other’ activities, and this change was significantly greater than the slight decrease of 0.11 points (SD = 0.04) in Self-Stigma when eating (p=.005). No other differences were statistically significant. However, the difference between ‘other’ activity and inactivity (where there was no change from previous Self-Stigma) approached significance (p = 0.02). Although the mean Self-Stigma when doing chores showed a similar increase (0.23 points, SD = 0.10) to the ‘other’ category, it was not significantly different from when inactive (p = 0.16) or eating (p = 0.05) (Table 3).

Table 3.

Current Location, Company, and Activity predicting change in Self-Stigma

Predictors Self-Stigma
Coeff SE t df p
Model 1
Intercept 2.34 0.39
Self-Stigma(t-1) 0.11 0.05 2.20 23 0.04
Location (1.64) 4 0.80
 Homea -- --
 Friend’s home −0.04 0.19
 Work 0.06 0.17
 Clinic −0.21 0.22
 Outside 0.08 0.13
Model 2
Intercept 2.41 0.37
Self-Stigma (t-1) 0.14 0.06 2.49 23 0.02
Company (4.11) 4 0.29
 Alonea -- --
 Family member −0.10 0.14
 Friend −0.15 0.17
 Coworker −0.43 0.27
 Doctor 0.17 0.23
Model 3
Intercept 2.28 0.39
Self-Stigma(t-1) 0.11 0.05 2.08 23 0.05
Activity (10.53) 4 0.03
 Inactivea -- --
 Eating −0.11 0.13
 Chores 0.23 0.16
 Working 0.11 0.16
 Other 0.30 0.13
Open in a new tab

Note. Values in parentheses are multiple-parameter χ2 statistics representing the overall significance of the categorical predictor. Coefficients represent the contrast between the indicated category and reference category (At home; Alone; Inactive). Coeff=Coefficient; SE= Standard Error; df= degrees of freedom

a

Reference Category.

Psychotic Symptoms and Affect

Table 4 summarizes the results of a second set of analyses examining Symptoms, Positive Affect, and Negative Affect as a predictors of Self-Stigma. A time-lagged approach was used, predicting time t outcomes from predictors at both time t-1 and time t, controlling for the outcome at time t-1. Both previous Negative Affect and change in Negative Affect were significantly related to increases in subsequent Self-Stigma. A positive relationship was also found between increases in Symptom severity and Self-Stigma (Table 4).

Table 4.

Previous and current Symptoms, Positive Affect, and Negative Affect predicting change in Self-Stigma

Predictors Self-Stigma
Coeff SE β t df p
Intercept 2.43 0.39
Self-Stigma (t-1) 0.10 0.04 0.10 2.30 23 0.03
PA (t-1) −0.02 0.02 −0.04 −1.46 730 0.14
NA (t-1) 0.05 0.02 0.10 2.17 730 0.03
Symptoms (t-1) 0.04 0.03 0.08 1.63 730 0.10
PA 0.01 0.02 0.03 0.83 730 0.41
NA 0.10 0.02 0.11 3.92 730 <0.01
Symptoms 0.09 0.03 0.16 3.23 730 <0.01
Open in a new tab

Note. PA = Positive Affect; NA = Negative Affect; Coeff=Coefficient; SE= Standard Error; β = beta; df= degrees of freedom.

To examine whether the relationship between Symptoms and Self-Stigma is bidirectional, a third analysis was conducted with Symptoms as the outcome, and Stigma, Positive Affect, and Negative Affect as the predictors. Table 5 summarizes the findings and shows that no previous (time t-1) predictors were significantly related to change in symptoms. However, changes in Negative Affect and Self-Stigma were both predictive of changes in Symptoms (Table 5).

Table 5.

Previous and current Self-Stigma, Positive Affect, and Negative Affect predicting change in Symptoms.

Predictors Symptoms
Coeff SE β t df p
Intercept 3.62 0.78
Symptoms(t-1) 0.01 0.03 0.01 0.40 730 0.69
PA (t-1) 0.01 0.02 0.01 0.72 730 0.47
NA (t-1) −0.02 0.03 −0.01 −0.69 730 0.49
Self-Stigma (t-1) 0.02 0.04 0.01 0.60 730 0.55
PA 0.01 0.02 0.01 0.26 730 0.80
NA 0.20 0.03 0.13 6.88 730 <0.01
Self-Stigma 0.15 0.04 0.08 3.40 730 <0.01
Open in a new tab

Note. PA = Positive Affect; NA = Negative Affect; Coeff=Coefficient; SE= Standard Error; β = beta; df= degrees of freedom.

DISCUSSION

The current study is the first to examine the relationships between self-stigma and varying contextual and internal factors in the course of the daily lives of individuals with schizophrenia. A number of factors proved to be viable predictors of the intensity of self-reported self-stigmatizing beliefs. The activity individuals were engaged in was associated with changes in self-stigma, but immediate physical location and social company were not. Increases in negative affect predicted increases in self-stigma, but changes in positive affect did not. Finally, psychotic symptom severity proved to be both a predictor and an outcome of intensification in self-stigmatizing beliefs.

Our findings support a framework for understanding self-stigma as an experience that ebbs-and-flows in conjunction with changing internal states and circumstances. The methodology used enabled us to determine that fluctuations in self-stigma can occur within relatively short timeframes. While a high proportion of the variance in stigma was between persons, suggesting stable individual differences, there was also substantial variation in the experience of stigma within persons, and these within-person fluctuations were related to current activities, mood and symptoms.

The bidirectional associations we found between symptoms of psychosis and self-stigma seem especially detrimental, suggesting a reciprocal relationship. For example, it is possible that when an individual believes they are incapable of getting a job due to their illness, they may subsequently experience auditory hallucinations infused with self-deprecating or humiliating content. When these hallucinations intensify and are experienced as uncontrollable, concerns about pursuing employment goals may seem even more justified and valid, contributing to greater self-judgment and worsening of symptoms, thus perpetuating a vicious cycle. While exacerbation of one factor may lead to worsening in the other, our findings also suggest that interventions that can effectively reduce self-stigmatizing beliefs may also have a positive effect on symptom severity, and vice versa.

The methodology deployed in this study enabled us to gain insights about the dynamic interaction between self-stigma and time-varying factors. Previous studies on the course and risk factors of self-stigma in mental illness were limited by their use of retrospective or long interval (e.g., months, years) prospective designs that were incapable of capturing frequent fluctuations and short term temporal relationships between variables. Additional strengths of utilizing mobile technologies for data collection include improved ecological validity, and the ability to identify relationships that individuals would likely not consciously report (Myin-Germeys, Birchwood & Kwapil, 2011).

Mobile technologies may prove to be instrumental in helping combat self-stigma in the years ahead. Existing psychosocial stigma reduction strategies often consist of individual or group interventions aimed at minimizing susceptibility to the negative impact of self-stigma and enhancing resilience (Heijnders & van der Meij, 2006; Weiss, Ramakrishna & Somma, 2006). Cognitive therapy in particular has been found to be both feasible and useful in helping people change negative cognitive schemata and combat self-stigma (Corrigan & Calabrese, 2005). Mobile platforms such as cellular phones and smartphones could be used as a medium for delivering such clinical interventions in real-time, in response to dynamic conditions (Kelly et al., 2012; Luxton, McCann, Bush, Mishkind & Reger, 2011). Researchers have developed and conducted preliminary tests on mobile interventions that can be used to prompt medication adherence and the use of coping strategies to increase socialization and decrease distress associated with symptoms among individuals with schizophrenia (Granholm et al., 2011). Similar strategies can be developed to monitor changes that predict self-stigma in real-time and decrease the negative impact of these factors as well. If an individual completing self-report measures generated by a smartphone indicates that he or she is experiencing salient psychotic symptoms (a temporal predictor of increased severity of self-stigmatizing beliefs), it could prompt the delivery of content (e.g., text, audio, video) designed to challenge dysfunctional beliefs about the relationship between illness and incompetence or dangerousness: “Hearing voices doesn’t mean you can’t have a job. Symptoms come and go, but you have the power to decide how much they get in the way of doing what you want.”

This study has several limitations. Most notably, the small sample size limits the generalizability of our findings. In addition, our analyses might have been incapable of detecting relationships between variables due to insufficient statistical power. Future Ecological Momentary Assessment studies on the interaction between self-stigma, symptoms, affect, and context should include more participants from diverse clinical and demographic populations. The mobile assessment measure was limited in range of questions and response options used to assess context, affect and symptoms. We were unable to identify the specific activities in the activities subcategory of “other.” There are many more variations in contextual factors that may predict self-stigma that were not captured by our multiple choice response options. Thus further research that explores these factors in more detail is warranted.

Self-stigma has a tremendous negative impact on individuals with severe mental illness and is related to decreased self-efficacy, increased psychiatric symptom severity, and lower social and vocational functioning. Longitudinal research with mobile technologies is particularly promising, as it allows researchers to track predictors and outcomes of self-stigma in real-time and real-world settings. Mobile platforms are capable of not only facilitating the monitoring of changes in self-stigma, but also prompting the delivery of cognitive interventions that may help individuals mitigate the negative impact of self-stigma on one’s ability to function in daily life roles (e.g., parent, spouse, co-worker) as it occurs.

Figure 1.

Figure 1

Open in a new tab

Change in Self-Stigma from previous time period as a function of type of activity.

Acknowledgments

This study was supported by funding provided by the National Center on Adherence and Self Determination (National Institute of Mental Health DCISR MH08598-01).

Footnotes

DISCLOSURES

The authors report no financial relationships with commercial interests.

References

  1. Abdullah T, Brown TL. Mental illness stigma and ethnocultural beliefs, values and norms: An integrative review. Clinical Psychology Review. 2011;31:934–948. doi: 10.1016/j.cpr.2011.05.003. [DOI] [PubMed] [Google Scholar]
  2. Andreasen NC. The scale for the assessment of positive symptoms (SAPS) The Iowa City, IA: University of Iowa; 1984. [Google Scholar]
  3. Andreasen NC, Olsen S. Negative versus positive schizophrenia: Definition and validation. Archives of General Psychiatry. 1982;39:789–794. doi: 10.1001/archpsyc.1982.04290070025006. [DOI] [PubMed] [Google Scholar]
  4. Barrett DJ, Barrett LF. The experience sampling program (ESP) (Version 4.0) 2005 Retrieved from http://www.experience-sampling.org/esp.pdf.
  5. Beck AT, Steer RA. Beck Anxiety Inventory manual. San Antonio, TX: Psychological Corporation; 1993. [Google Scholar]
  6. Beck AT, Steer RA, Brown GK. Manual for Beck Depression Inventory II (BDI-II) San Antonio, TX: Psychology Corporation; 1996. [Google Scholar]
  7. Ben-Zeev D, Drake RE, Corrigan βW, Rotondi AJ, Nilsen W, Depp C. Using contemporary technologies in assessment and treatment of serious mental illness. American Journal of Psychiatric Rehabilitation (in press) [Google Scholar]
  8. Ben-Zeev D, Ellington K, Swendsen J, Granholm E. Examining a cognitive model of persecutory ideation in the daily life of people with schizophrenia: A computerized experience sampling study. Schizophrenia Bulletin. 2011;37:1248–1256. doi: 10.1093/schbul/sbq041. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Ben-Zeev D, Granholm E, Cafri G. Symptoms of depression and anxiety mediate the relationship between self-esteem and paranoia. Schizophrenia Research. 2009;115:377–378. doi: 10.1016/j.schres.2009.06.009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Ben-Zeev D, Morris S, Swendsen J, Granholm E. Predicting the occurrence, conviction, distress, and disruption of different delusional experiences in the daily life of people with schizophrenia. Schizophrenia Bulletin. 2011;38:826–837. doi: 10.1093/schbul/sbq167. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Brohan E, Elgie R, Satorius N, Thornicroft G. Self-stigma, empowerment and perceived discrimination among people with schizophrenia in 14 European countries: The GAMIAN-Europe study. Schizophrenia Research. 2010;122:232–238. doi: 10.1016/j.schres.2010.02.1065.. [DOI] [PubMed] [Google Scholar]
  12. Cavelti M, Kvgic S, Beck E, Rusch N, Vauth R. Self-stigma and its relationship with insight, demoralization, and clinical outcome among people with schizophrenia spectrum disorders. Comprehensive Psychiatry. 2011;53:468–479. doi: 10.1016/j.comppsych.2011.08.001.. [DOI] [PubMed] [Google Scholar]
  13. Corrigan PW, Ben-Zeev D. The particular role of stigma. In: Cohen NL, Galea S, editors. Population mental health: Evidence, policy, and public health practice. New York, NY: Routledge; 2011. pp. 92–116. [Google Scholar]
  14. Corrigan PW, Calabrese JD. Strategies for assessing and diminishing self-stigma. In: Corrigan PW, editor. On the stigma of mental illness: Practical strategies for research and social change. Washington, D.C: American Psychological Association; 2005. pp. 239–256. [Google Scholar]
  15. Corrigan PW, Kleinlein P. The impact of mental illness stigma. In: Corrigan PW, editor. On the stigma of mental illness: Practical strategies for research and social change. Washington, D.C: American Psychological Association; 2005. pp. 11–44. [Google Scholar]
  16. Corrigan PW, Watson AC, Barr L. The self–stigma of mental illness: Implications for self–esteem and self–efficacy. Journal of Social and Clinical Psychology. 2006;25:875–884. doi: 10.1521/jscp.2006.25.8.875. [DOI] [Google Scholar]
  17. Corrigan PW, Watson AC. The paradox of self-stigma and mental illness. Clinical Psychology: Science and Practice. 2002;9:35–53. doi: 10.1093/clipsy.9.1.35. [DOI] [Google Scholar]
  18. Csikszentmihalyi M, Larson R. Validity and reliability of the experience-sampling method. Journal of Nervous and Mental Disorders. 1987;175:526–536. doi: 10.1097/00005053-198709000-00004. [DOI] [PubMed] [Google Scholar]
  19. Dalky HF. Mental illness stigma reduction interventions: Review of intervention trials. Western Journal of Nursing Research. 2011;34:520–547. doi: 10.1177/0193945911400638. [DOI] [PubMed] [Google Scholar]
  20. Delespaul P, deVries M. The daily life of ambulatory chronic mental patients. Journal of Nervous and Mental Disorders. 1987;175:537–544. doi: 10.1097/00005053-198709000-00005. [DOI] [PubMed] [Google Scholar]
  21. Delespaul P, deVries M, van Os J. Determinants of occurrence and recovery from hallucinations in daily life. Social Psychiatry and Psychiatric Epidemiology. 2002;37:97–104. doi: 10.1007/s001270200000. [DOI] [PubMed] [Google Scholar]
  22. First MB, Spitzer RL, Gibbon M, Williams JBW. Structured Clinical Interview for DSM-IV Axis I Disorders-Patient Edition (SCID-I/P), Version 2.0. New York State Psychiatric Institute: Biometrics Research; 1995. [Google Scholar]
  23. Freeman D, Garety PA. Connecting neurosis and psychosis: The direct influence of emotion on delusions and hallucinations. Behaviour Research and Therapy. 2003;41:923–947. doi: 10.1016/S0005-7967(02)00104-3. [DOI] [PubMed] [Google Scholar]
  24. Granholm E, Loh C, Swendsen J. Feasibility and validity of computerized ecological momentary assessment in schizophrenia. Schizophrenia Bulletin. 2008;34:507–514. doi: 10.1093/schbul/sbm113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Granholm E, Ben-Zeev D, Link PC, Bradshaw K, Holden J. Mobile Assessment and Treatment for Schizophrenia (MATS): A pilot trial of an interactive text- messaging intervention for medication adherence, socialization, and auditory hallucinations. Schizophrenia Bulletin. 2012;38:414–425. doi: 10.1093/schbul.sbr155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Griffiths KM, Christensen H, Jorm AF, Evans K, Groves C. Effect of web-based depression literacy and cognitive-behavioural therapy interventions on stigmatizing attitudes to depression. British Journal of Psychiatry. 2004;185:342–349. doi: 10.1192/bjp.185.4.342. [DOI] [PubMed] [Google Scholar]
  27. Haddock G, McCarron J, Tarrier N, Faragher EB. Scales to measure dimensions of hallucinations and delusions: The psychotic symptom rating scales (PSYRATS) Psychological Medicine. 1999;29:879–889. doi: 10.1017/S0033291799008661. [DOI] [PubMed] [Google Scholar]
  28. Heijnders M, van der Meij S. The fight against stigma: An overview of stigma-reduction strategies and interventions. Psychology, Health and Medicine. 2006;11:353–363. doi: 10.1080/13548500600595327. [DOI] [PubMed] [Google Scholar]
  29. Henquet C, Murray R, Linszen D, van Os J. The environment and schizophrenia: The role of cannabis use. Schizophrenia Bulletin. 2005;31:608–612. doi: 10.1093/schbul/sbi027. [DOI] [PubMed] [Google Scholar]
  30. Hox JJ. Multilevel analysis: Techniques and applications. 2. New York, NY: Routledge; 2010. [Google Scholar]
  31. Kelly J, Gooding P, Pratt D, Ainsworth J, Welford M, Tarrier N. Intelligent real-time therapy: Harnessing the power of machine learning to optimize the delivery of momentary cognitive-behavioural interventions. Journal of Mental Health. 2012 doi: 10.3109/09638237.2011.638001. Advance online publication. [DOI] [PubMed] [Google Scholar]
  32. Kimhy D, Delespaul P, Ahn H, Cai S, Shikhman M, Lieberman JA, Sloan RP. Concurrent measurement of “real-world” stress and arousal in individuals with psychosis: Assessing the feasibility and validity of a novel methodology. Schizophrenia Bulletin. 2010;36:1131–1139. doi: 10.1093/schbul/sbp028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Kleim B, Vauth R, Adam G, Stieglitz R, Hayward P, Corrigan P. Perceived stigma predicts low self-efficacy and poor coping in schizophrenia. Journal of Mental Health. 2008;17:482–491. doi: 10.1080/09638230701506283. [DOI] [Google Scholar]
  34. Lee S. The stigma of schizophrenia: A transcultural problem. Current Opinion in Psychiatry. 2002;15:37–41. doi: 10.1097/00001504-200201000-00007. [DOI] [Google Scholar]
  35. Link BG, Phelan JC. Stigma and its public health implications. Lancet. 2006;267:528–529. doi: 10.1016/S0140-6736(06)68184-1. [DOI] [PubMed] [Google Scholar]
  36. Livingston JD, Boyd JE. Correlates and consequences of internalized stigma for people living with mental illness: A systematic review and meta-analysis. Social Science & Medicine. 2010;71:2150–2161. doi: 10.1016/j.socscimed.2010.09.030. [DOI] [PubMed] [Google Scholar]
  37. Luxton DD, McCann RA, Bush NE, Mishkind MC, Reger GM. mHealth for mental health: Integrating smartphone technology in behavioral healthcare. Professional Psychology: Research and Practice. 2011;42:505–512. doi: 10.1037/a0024485. [DOI] [Google Scholar]
  38. Lysaker PH, Davis LW, Warman DM, Strasburger A, Beattie N. Stigma, social function and symptoms in schizophrenia and schizoaffective disorder: Associations across 6 months. Psychiatry Research. 2007;149:89–95. doi: 10.1016/j.psychres.2006.03.007. [DOI] [PubMed] [Google Scholar]
  39. Macinnes DL, Lewis M. The evaluation of a short group programme to reduce self-stigma in people with serious and enduring mental health problems. Journal of Psychiatric and Mental Health Nursing. 2008;15:59–65. doi: 10.1111/j.1365-2850.2007.01212.x. [DOI] [PubMed] [Google Scholar]
  40. Markowitz FE, Angell B, Greenberg JS. Stigma, reflected appraisals, and recovery outcomes in mental illness. Social Psychology Quarterly. 2011;74:144–165. doi: 10.1177/0190272511407620. [DOI] [Google Scholar]
  41. Michalak E, Livingston JD, Hole R, Suto M, Hale S, Haddock C. ‘It’s something that I manage but it is not who I am’: Reflections on internalized stigma in individuals with bipolar disorder. Chronic Illness. 2011;7:209–224. doi: 10.1177/1742395310395959. [DOI] [PubMed] [Google Scholar]
  42. Myin-Germeys I, Birchwood M, Kwapil T. From environment to therapy in psychosis: A real-world momentary assessment approach. Schizophrenia Bulletin. 2011;37:244–247. doi: 10.1093/schbul/sbq164. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Prescosolido BA, Martin JK, Lang A, Olafsdottir S. Rethinking theoretical approaches to stigma: A framework integrating normative influences on stigma (FINIS) Social Science and Medicine. 2008;67:431–440. doi: 10.1016/j.socscimed.2008.03.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Raudenbush SW, Bryk AS, Cheong YF, Cogdon RT. HLM 6.08 Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International, Inc; 2004. [Google Scholar]
  45. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annual Review of Clinical Psychology. 2008;4:1–32. doi: 10.1146/annurev.clinpsy.3.022806.091415. [DOI] [PubMed] [Google Scholar]
  46. Sirey JA, Bruce ML, Alexopoulos GS, Perlick DA, Friedman SJ, Meyers BS. Stigma as a barrier to recovery: Perceived stigma and patient-rated severity of illness as predictors of antidepressant drug adherence. Psychiatric Services. 2001;52:1615–1620. doi: 10.1176/appi.ps.52.12.1615. [DOI] [PubMed] [Google Scholar]
  47. Smith B, Fowler D, Freeman D, Bebbington P, Bashforth H, Garety P. Emotion and psychosis: Links between depression, self-esteem, negative schematic beliefs and delusions and hallucinations. Schizophrenia Research. 2006;86:181–188. doi: 10.1016/j.schres.2006.06.018. [DOI] [PubMed] [Google Scholar]
  48. Swendsen J, Ben-Zeev D, Granholm E. Real-time electronic ambulatory monitoring of substance use and symptom expression in schizophrenia. American Journal of Psychiatry. 2011;168:202–209. doi: 10.1176/appi.ajp.2010.10030463. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Thewissen V, Bentall RP, Oorschot M, á Campo J, van Lierop T, van Os J, Myin-Germeys I. Emotions, self-esteem, and paranoid episodes: An experience sampling study. British Journal of Clinical Psychology. 2011;50:178–195. doi: 10.1348/014466510X508677.. [DOI] [PubMed] [Google Scholar]
  50. Wahl OF. Mental health consumers’ experience of stigma. Schizophrenia Bulletin. 1999;25:467–478. doi: 10.1093/oxfordjournals.schbul.a033394. [DOI] [PubMed] [Google Scholar]
  51. Watson AC, Corrigan P, Larson JE, Sells M. Self-stigma in people with mental illness. Schizophrenia Bulletin. 2007;33:1312–1318. doi: 10.1093/schbul/sbl076. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Watson D, Clark LA, Tellegen A. Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personal and Social Psychology. 1988;54:1063–1070. doi: 10.1037/0022-3514.54.6.1063. [DOI] [PubMed] [Google Scholar]
  53. Weiss MG, Jadhav S, Raguram R, Vounatsou P, Littlewood R. Psychiatric stigma across cultures: Local validation in Bangalore and London. Anthropology and Medicine. 2001;8:71–87. doi: 10.1080/13648470120063906. [DOI] [Google Scholar]
  54. Weiss MG, Ramakrishna J, Somma D. Health-related stigma: Rethinking concepts and interventions. Psychology, Health and Medicine. 2006;11:277–287. doi: 10.1080/13548500600595053. [DOI] [PubMed] [Google Scholar]
  55. West ML, Yanos PT, Smith SM, Roe D, Lysaker PH. Prevalence of internalized stigma among persons with severe mental illness. Stigma Research and Action. 2011;1:3–10. doi: 10.5463/sra.v1i1.9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Yanos PT, Roe D, Markus K, Lysaker PH. Pathways between internalized stigma and outcomes related to recovery in schizophrenia spectrum disorders. Psychiatric Services. 2008;59:1437–1442. doi: 10.1176/appi.ps.59.12.1437. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Yanos PT, Roe D, Lysaker PH. The impact of illness identity on recovery from severe mental illness. American Journal of Psychiatric Rehabilitation. 2010;13:73–93. doi: 10.1080/15487761003756860. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES