Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Aug 1.
Published in final edited form as: Stigma Health. 2017 Apr 6;3(3):195–203. doi: 10.1037/sah0000089

Development and Preliminary Psychometric Evaluation of a Patient-Reported Outcome Measure for Lung Cancer Stigma: The Lung Cancer Stigma Inventory (LCSI)

Heidi A Hamann 1,2,3,4, Megan J Shen 5, Anna J Thomas 2, Simon J Craddock Lee 3,4, Jamie S Ostroff 6
PMCID: PMC6208151  NIHMSID: NIHMS864802  PMID: 30393760

Abstract

Introduction

Among patients with lung cancer, stigma is associated with negative psychosocial and behavioral outcomes. There is a need to develop psychometrically robust patient-reported outcome (PRO) measures for stigma that incorporate perspectives of patients diagnosed with lung cancer. As part of our multi-phase process of measure development and validation, we report on scale formation and preliminary psychometric evaluation of the Lung Cancer Stigma Inventory (LCSI).

Method

Building on previously reported concept elicitation (Phase I) work, Phase II of LCSI development involved item generation and refinement, informed by literature review, provider input, and patient (N=20) feedback. Phase III focused on initial psychometric scale evaluation in a unique sample of 231 lung cancer patients.

Results

Based on provider input and patient cognitive interviews, 49 items were included in a preliminary measure. In an exploratory factor analysis (EFA) of the 37 retained items, three factors emerged: Perceived Stigma, Internalized Stigma, and Constrained Disclosure. Internal consistency of the final, 25-item LCSI scale was high (Cronbach’s alpha= 0.89) and the three subscales demonstrated good internal consistency. The test-retest correlation was high (r = 0.91), suggesting strong stability of measurement over time. There was good convergent validity between the LCSI and an existing measure of lung cancer stigma, the Cataldo Lung Cancer Stigma Scale (CLCSS; r= 0.58, p< 0.001).

Discussion

In a multi-phase process, we have developed a reliable, multi-dimensional measure of lung cancer stigma, the Lung Cancer Stigma Inventory (LCSI). Subsequent work will be conducted to establish further evidence of validity and clinically meaningful change.

Keywords: stigma, lung cancer, patient-reported outcome, psychometric, measurement

Lung cancer is the leading cancer killer for both men and women in the United States, and it is associated with more annual deaths than breast, prostate, and colon cancer combined (American Cancer Society [ACS], 2016). Despite recent advances in early detection (e.g., Aberle et al., 2011; Alberg, Brock, Ford, Samet, & Spivack, 2013) and treatment (e.g., Rosell et al., 2012), most lung cancers are still diagnosed at late stages, have poor treatment outcomes, and are associated with significant supportive care needs. Although there are other known environmental and genetic risk factors for lung cancer, smoking is responsible for almost 90 percent of lung cancer cases (ACS, 2016; Alberg et al., 2013).

The robust causal connection between smoking and lung cancer underscores the relevance of tobacco control interventions to reduce lung cancer morbidity and mortality, but it may also generate highly salient stigma (i.e., negative appraisal and devaluation) toward lung cancer patients (Bayer & Stuber, 2006). Lung cancer stigma can be conceptualized as a form of health-related stigma in which a person perceives and potentially internalizes an experience of rejection, blame, or devaluation directly linked to the belief that their behavior (i.e., smoking) has caused their current health condition (i.e., lung cancer; Cataldo, Jahan, & Pongquan, 2012; Van Brakel, 2006). Emerging evidence suggests pervasive societal stigma toward lung cancer. For example, in an assessment of societal attitudes toward lung versus breast cancer, Sriram and colleagues (2015) reported both explicit and implicit bias toward lung cancer among cancer patients, caregivers, health care providers, and the general public. Other investigations have identified blaming responses and other biased perceptions of clinicians toward their lung cancer patients (Hamann et al., 2013; Morse, Edwardsen, & Gordon, 2008). In our previous work, 95% of lung cancer patients described experiencing at least one element of perceived stigma (Hamann et al., 2014).

Social psychological theory points to factors that are most often tied to the internalization of stigma, including: (1) guilt and shame (Mak et al., 2007); (2) self-blame (Major, Kaiser, & McCoy, 2003); (3) anticipated stigma (belief that one will be socially stigmatized if they reveal their identity; Quinn & Chaudoir, 2009); and (4) negative affect (e.g., ashamed, worthless) from perceiving and experiencing stigma within encounters (Major et al., 2003). Investigations from other health-related domains have focused on adverse effects of stigma, including psychosocial impairment, limited disclosure (Clark, Lindner, Armistead, & Austin, 2003), social isolation (Turan et al., 2016), and reduced treatment adherence (Katz et al., 2016; Sweeney & Vanable, 2016). When examining these factors and consequences, it becomes clear that stigma is commonly perceived as well as internalized within the context of social interactions. In the case of lung cancer, internalized stigma (including self-blame, guilt, and regret) has been associated with negative psychosocial and interpersonal outcomes for lung cancer patients (Chambers et al., 2012; Chapple, Ziebland & McPherson, 2004; Else-Quest, LoConte, Schiller, & Hyde, 2009; Holland, Kelly, & Weinberger, 2010; LoConte, Else-Quest, Eickhoff, Hyde, & Schiller, 2008; Weiss, Weinberger, Schwerd, & Holland, 2012). For example, in recent studies, patient-reported lung cancer stigma has been associated with increased depressive symptoms (Cataldo & Brodsky, 2013; Gonzalez & Jacobson, 2012), lower quality of life (Cataldo et al., 2012), delays in medical help-seeking (Carter-Harris, Hermann, Schreiber, Weaver, & Rawl, 2014), and reduced satisfaction with provider communication (Shen, Thomas, & Ostroff, 2015). In this vein, it has also been argued that lung cancer stigma may be a barrier to treatment adherence, engagement in clinical trials, and other cancer care outcomes (Hamann et al., 2014; Sriram et al., 2015).

Increased recognition of lung cancer stigma and its potential consequences calls attention to the need for identifying valid and reliable measurement tools to assess lung cancer stigma. To date, most investigators (e.g., Cataldo, Slaughter, Jahan, Pongquan, & Hwang, 2011; Gonzalez & Jacobsen, 2012) interested in measuring stigma experienced by patients with lung cancer have adapted stigma scales developed for use with HIV (e.g., Berger, Ferrans, & Lashley, 2001) and other potentially stigmatized patient populations. While expedient, the potential pitfall of repurposing scales developed for use with other patient populations is that item content most germane to the targeted patient population may be overlooked. In embarking on this multi-phase, scale development and validation project, our goal was to develop a psychometrically sound measurement tool for the assessment of stigma experienced by patients with lung cancer. The LCSI item pool was guided by themes expressed during a qualitative concept elicitation phase (Phase 1) conducted with patients diagnosed with lung cancer. Inclusion of this qualitative phase of concept elicitation provides investigators with greater confidence that the LCSI captures relevant aspects of the lung cancer stigma experienced by patients with lung cancer. Our approach is consistent with best practices for item and scale development that call for incorporating patient perspectives to identify relevant, population-appropriate constructs and to validate patient-reported outcome (PRO) instruments (Rothrock, Kaiser & Cella, 2011; US Department of Health and Human Services [DHHS], 2009). Current recommendations for developing PROs emphasize the importance of patient and expert input, including, but not limited to: (I) conceptual model development; (II) item generation and improvement; and (III) psychometric analysis.

Accordingly, we have conducted a multi-phase process for development and evaluation of a new measure of stigma among lung cancer patients, the Lung Cancer Stigma Inventory (LCSI). In Phase I, we aimed to establish content validity by developing a conceptual blueprint or framework for item development. To achieve this goal, we conducted qualitative interviews with lung cancer patients and elicited relevant clinical themes (Hamann et al., 2014). Our qualitative analyses revealed that lung cancer stigma appeared to be a bi-dimensional construct representing perceived and internalized stigma as well as maladaptive consequences such as social isolation. In the current report, we describe the next phases of measure development for the Lung Cancer Stigma Inventory (LCSI), including scale formation (Phase II) and preliminary psychometric evaluation (Phase III). This rigorous, conceptual, and patient-engaged measure development and validation process provides a strong foundation to continue our understanding and amelioration of lung cancer stigma.

Method

Overview

Building on our conceptual model (Hamann et al., 2014 and Figure 1 shown here) and recommended psychometric methods for the development of PRO measurement tools (e.g., Basch et al., 2015; Rothrock et al., 2011), Phase II of LCSI development involved item generation and refinement, informed by a review of existing stigma measurement tools, along with provider and patient input. Phase III focused on initial psychometric scale evaluation, with particular emphasis on assessing reliability and convergent validity. The study was approved by appropriate institutional review boards.

Figure 1.

Figure 1

Open in a new tab

Conceptual model of lung cancer stigma. Reprinted from “Stigma among patients with lung cancer: A patient-reported measurement model,” by H. A. Hamann, J. S. Ostroff, E. G. Marks, D. E. Gerber, J. H. Schiller, and S. J. C. Lee, 2014, Psycho-Oncology, 23, pp. 81–92. Copyright 2014 by the John Wiley and Sons. Reprinted with permission. See the online article for the color version of this figure.

Phase II: Item Generation and Refinement

Item generation involved: (1) adapting the wording of items from existing stigma scales for use with lung cancer patients, and (2) generating additional items based on the content domains elicited during qualitative interviews with lung cancer patients. We conducted a thorough literature review of existing stigma measurement tools with a focus on lung cancer and other potentially stigmatized health conditions (e.g., HIV, obesity, mental illness, other cancers; see Table 1 for a list). In addition, the co-investigators (HAH, JSO, SJCL) constructed items that specifically reflected domains (perceived and internalized stigma) and subdomains of the conceptual model (Hamann et al., 2014). Items were refined through both provider and patient input. Each item was reviewed by one of three multi-disciplinary, lung cancer expert teams (each including a thoracic oncology clinician and a psychosocial clinician) to rate (on a 1–5 scale) for relevance and clarity (Lynn, 1986; Patrick et al., 2011).

Table 1.

Measures Reviewed during the Initial Item Development Phase

Name of Measure Author (Year) Population Number of Items
Cataldo Lung Cancer Stigma Scale Cataldo et al. (2011) Cancer (Lung) 31
Chronic Illness Anticipated Stigma Scale Earnshaw, Quinn, Kalaichman, and Park (2013) Chronic Illness 12
Depression Self Stigma Scale Kanter et al. (2008) Depression 32
Generalized Anxiety Stigma Scale Griffiths et al. (2011) Anxiety 20
HIV Internalized Stigma Measure Sayles et al. (2008) HIV 28
HIV Internalized Stigma Scale Visser et al. (2008) HIV 12
HIV Stigma Scale Berger et al. (2001) HIV 40
Perceptions of Stigma Sowell et al. (1997) HIV 13
HIV/AIDS and TB-related Stigma Scale Van Rie et al. (2008) HIV and TB 23
HIV/AIDS Stigma Instrument-PLWA Holzemer et al. (2007) HIV/AIDS 33
Internalized AIDS-Related Stigma Scale Kalichman et al. (2007) HIV/AIDS 6
Internalized Stigma of Mental Illness Scale Ritsher et al. (2003) Mental Illness 29
Perceived Cancer-Related Stigma Scale Loconte et al. (2008) Cancer 6
Psychosocial Discomfort Scale Tsuchiya et al. (2012) Cancer (Breast) 25
Revised Stigma Scale Wright et al. (2007) HIV 10
Self-Stigma of Depression Scale Barney et al. (2010) Depression 16
Shame and Stigma Scale Kissance et al. (2013) Cancer (Head & Neck) 20
Social Impact Scale Fife & Wright (2000) Cancer, HIV 24
Social Stigma Scale MacDonald & Anderson (1984) Cancer (Rectal) 5
Stigma Scale for Chronic Illness Rao et al. (2009) Neurological Disorders 24
The Stigma Scale King et al. (2007) Mental Illness 28
Open in a new tab

Highly relevant items (scores averaging 4.0 or higher) were then administered and evaluated using cognitive interview techniques (Willis, Royston, & Bercini, 1991; Willis, 2005) by 20 lung cancer patients treated at outpatient oncology clinics associated with an NCI-designated cancer center in the Southern region of the United States. A primary goal of cognitive interviewing, an evidence-based, qualitative practice of investigating respondents’ understanding of survey items (Willis, 2005), is to evaluate whether respondents interpret items in the manner intended by researchers. For our cognitive interviewing process, we used a hybrid approach combining “think-aloud” in which participants were asked to comment on each item’s wording, and suggest, if needed, alternate wording to enhance relevance and clarity of items, followed by “post-test” debriefing for overall acceptability, comprehension, any additional domains, parent study purpose, and referrals to additional resources. Interviewees were also asked to comment on the scale instruction and the Likert-scale response options. Items were evaluated for readability by the Flesch-Kinkaid Grade Level Index readability measure (Kincaid, Fishburne, Rogers, & Chissom, 1975), with a desired readability of 6th grade or lower level.

Phase III: Psychometric Evaluation

The pool of refined items emerging from Phase II was then field-tested in a multi-site sample of 231 lung cancer patients, recruited through outpatient oncology clinics associated with NCI-designated cancer centers in the Southern and Eastern regions of the United States. Eligible patients were adults (18 years and older) with a lung cancer diagnosis and/or treatment in the last 12 months, who were able to read and comprehend English, and had capacity to comprehend study information (see Figure 2 for information on recruitment).

Figure 2.

Figure 2

Open in a new tab

Consort diagram of patient recruitment and enrollment (Reprinted with permission from Supportive Care in Cancer)

Participants completed the items, as part of a larger survey, on either a web-based data platform (Research Electronic Data capture; REDCap; Harris et al., 2009) or in paper form. The measures for the current analysis include a demographic/clinical questionnaire, the refined LCSI items, and an existing stigma measure to assess convergent validity, the Cataldo Lung Cancer Stigma Scale (CLCSS; Cataldo et al., 2011). The CLCSS is a 31-item instrument, adapted from the HIV Stigma Scale (Berger et al., 2001) composed of four subscales: stigma and shame, social isolation, discrimination, and smoking. High internal consistency for the total stigma scale has been demonstrated in multiple lung cancer samples (Cronbach’s α = 0.97 and 0.96; Cataldo et al., 2011; Cataldo & Brodsky, 2013). Each item is measured on a 4-point Likert-type scale (strongly disagree, disagree, agree and strongly agree), with higher values indicating greater agreement with the item. Surveys were counterbalanced so that half the sample completed the LCSI before the CLCSS, while the other half completed the CLCSS before the LCSI. After initial survey completion, a subset of 88 participants were provided a second administration of only the LCSI items to assess test-retest reliability; 55 participants (63%) returned this retest within 60 days following completion of the initial survey (retest return averaged 20.9 days [SD= 10.6 days] after completion of the initial survey).

Phase III: Statistical Analyses

The psychometric analysis plan followed procedures outlined in Nunnally and Bernstein (1994). The first round of analyses yielded item means, standard deviations, and item-total correlations; items with standard deviations <1.0 and/or poor item-total correlations (< 0.25) were removed from further analysis unless determined to be conceptually relevant by the investigators and expert consultants. In the second round of analyses, an exploratory factor analysis (EFA) with oblique (Oblimin) rotation was conducted to determine the underlying structure of the measure. We chose to conduct an EFA instead of a Confirmatory Factor Analysis (CFA); although we had developed a multi-dimensional model of stigma, we did not have a strong hypothesis regarding the number and types of factors that would emerge. Moreover, this approach is commonly taken in the initial phases/development of a new scale that lacks a previously identified factor structure (Child, 1990). Items that loaded strongly on a single factor (> 0.40) and that had a unique contribution to this factor (> 0.10 difference between factor values) were maintained. Cronbach’s alpha was calculated to assess internal reliability of the total scale and subscales, while convergent validity was assessed by correlating LCSI scores with scores on the CLCSS.

Results

Phase II

The first round of item development identified 240 unique items that were nominated by the investigators for capturing potentially relevant stigma themes. Expert input on item relevance and clarity reduced the item pool to 113. Items were then evaluated using two rounds of cognitive interviews with a total of 20 lung cancer patients (see Table 2 for demographic and clinical characteristics of this sample). After the first round of cognitive interviews (n=12), the investigators removed 46 items due to the following reported issues: redundancy (16 items [24%]), lack of clarity (13 items [19%]), difficult/confusing wording (7 items [10%]) or disliked by respondents (10 items [15%]). Removal of these items resulted in 67 remaining items that were then presented to the second round of cognitive interviewee patients (n=8). Based on this additional round of patient feedback, minor modifications to item wording were made, but the items were generally judged as relevant, appropriate, and understandable. Feedback about response options indicated a preference for a 5-point Likert-type scale with a response stem that focused on magnitude of endorsement inclusive of time since the initial diagnosis (as opposed to real time [now] or a fixed time interval). Final investigator evaluation of conceptual clarity reduced the item pool to 49 items. In general, this reduction eliminated items judged to be consequences of stigma (see Figure 1) rather than actual manifestations of stigma. The final 49 items had a readability estimate of a 5.7 grade level as determined by the Flesch-Kinkaid Grade Level Index readability measure (Kincaid et al., 1975). The final instructional set and response stem states, “To what degree has each of the following happened to you since your lung cancer diagnosis?” with 5-point response options ranging from: “not at all”, “slightly”, “somewhat”, “moderately”, or “extremely”.

Table 2.

Demographics and Clinical Characteristics of Phase II and Phase III Patient Participants

Characteristic Phase II (N=20) Phase III (N=231)
Number Percentage Number Percentage
Age (in years) M = 57.05 SD = 6.11 M = 62.80 SD = 10.96
Gender
 Female 12 60 147 63.6
 Male 8 40 84 36.4
Race*
 White 14 70 182 78.8
 Black/African-American 3 15 33 14.3
 Asian/Pacific-Islander 2 10 8 3.5
 AI or Alaska Native 1 5 0 0
 Other 0 0 6 2.6
Ethnicity*
 Hispanic 1 5 7 3.0
 Non-Hispanic 19 95 220 95.2
Education*
 Less than college degree 13 65 117 50.6
 College degree or higher 6 30 113 48.9
Marital Status*
 Married/Partnered 14 70 145 62.8
 Single, Never Married 1 5 21 9.1
 Divorced 4 20 26 11.3
 Widowed 0 0 27 11.7
Type of Lung Cancer
 Non-small cell lung cancer (NSCLC) 15 75 183 79.2
 Small cell lung cancer (SCLC) 4 20 22 9.5
 Information not available 1 5 26 11.3
Disease Stage
 Stage I 0 0 27 11.7
 Stage II 4 20 19 8.2
 Stage III 4 20 43 18.6
 Stage IV 11 55 128 55.4
 Information not available 1 5 14 6.1
Smoking Status*
 Never smoker (<100 cigarettes in lifetime) 5 25 60 26.0
 Former smoker (>100 cigarettes in lifetime; not smoking at time of study) 11 55 149 64.5
 Current smoker (>100 cigarettes in lifetime; smoking at time of study) 3 15 18 7.8
Open in a new tab
*

Note. Not All Percentages = 100% Due To Missing Data.

Phase III

Demographic and clinical characteristics of the 231 participating Phase III patients are presented in Table 2. The average age of participants was 63 years, 64% were female, and almost half were college graduates. Results from an analysis of means, standard deviations, and item-total correlations indicated that 27 items did not have sufficient response variability (standard deviation <1) and/or strong scale associations (defined by r < 0.25). Among these 27 items, 15 were judged as conceptually relevant by investigators and expert consultants and were therefore retained in the analysis. The remaining 12 items were removed from the scale, resulting in 37 items being included for the next analytic step.

Participants with complete data for all 37 items (N=195; 85%) were included in an open exploratory factor analysis (EFA), used to extract latent factors from the data. Ten factors had eigenvalues >1.0, with a clear delineation between three strong factors (accounting for 43% of the explained variance) and the other weaker factors (accounting for 24% of explained variance). Based on the presence of these three strong factors, a second EFA was run with a forced three-factor solution for these 37 items (see Table 3). An examination of the rotated factors (Oblimin) indicated that 27 items fulfilled factor loading criteria (defined as loading >.40 and >.10 difference between factors). Another two items did not meet criteria, but were maintained in Factor 2 because of strong conceptual relevance. Of the total 29 items retained, 11 items were included in Factor 1 (Internalized Stigma), 12 items in Factor 2 (Perceived Stigma), and 6 items in Factor 3 (Constrained Disclosure). Two items on Factor 1 and two items on Factor 2 were removed based on their content redundancy (see red items in Table 3), resulting in a final total of 25 items (9, 10, and 6 items remained in Factors 1, 2, 3, respectively). The three factors were correlated but somewhat distinct, with a correlation of r= 0.48 between Factors 1 and 2, r= 0.31 correlation between Factors 2 and 3, and r= 0.30 between Factors 1 and 3 (all p values < .001).

Table 3.

Factor Loadings of EFA with a Forced 3-Factor Solution (Step II, Study Population N=231)

Item Factor 1 (Internalized Stigma) Factor 2 (Perceived Stigma) Factor 3 (Constrained Disclosure)
1. I have been angry at tobacco companies for their role in causing lung cancer.   .280 −.072   .267
2. I have felt ashamed about getting lung cancer.   .413   .222   .343
3. People have assumed that lung cancer is always caused by smoking.   .229   .178   .139
4. I have blamed myself for having lung cancer.   .888   .091 −.166
5. I have tended to blame myself for my lung cancer.   .855   .098 −.100
6. I have felt bad about the burden that lung cancer puts on my family.   .446 −.090   .309
7. Because of my lung cancer, I have worried about being a burden to others.   .451   .028   .200
8. I have been careful who I’ve told about my lung cancer. −.118 −.113   .795
9. I have felt I did something to cause my lung cancer.   .797   .130 −.125
10. Thinking about lung cancer has made me wish I had lived my life differently.   .818 −.095 −.066
11. I have been concerned that doctors did not catch my lung cancer as early as they should have.   .196 −.048   .327
12. I have felt guilty about my lung cancer.   .710   .174   .161
13. It has been hard to tell people that I have lung cancer.   .060 −.125   .804
14. I have kept information about my lung cancer to myself. −.037   .101   .676
15. Having lung cancer has made me feel like I’ve made mistakes.   .841   .059 −.016
16. I have noticed more fundraising efforts for other cancers compared to lung cancer.   .295   .003   .105
17. I have thought that past behavior contributed to my lung cancer.   .748   .157 −.153
18. I have wondered if I could have prevented lung cancer by changing certain behaviors.   .770 −.044 −.004
19. I have felt regret about my lung cancer.   .619 −.097   .086
20. I have regretted telling certain people about my lung cancer. −.071   .258   .555
21. I have had an urge to keep my lung cancer a secret.   .032   .018   .765
22. I have stopped spending time with some people since my lung cancer diagnosis.   .018   .226   .457
23. People have judged me negatively for having lung cancer.   .076   .482   .248
24. Family or friends have told me I was to blame for getting lung cancer. −.132   .682 −.074
25. Because of my lung cancer, I have been treated poorly by others. −.126   .495   .198
26. People have said that those with lung cancer get what they deserve.   .119   .440   .024
27. Medical providers have told me that I caused my lung cancer.   .101   .383 −.058
28. People who don’t know me well have blamed me for getting lung cancer.   .107   .538   .137
29. I have wondered whether I would have gotten better medical care if I had another type of cancer.   .085   .181   .284
30. People close to me have thought I was to blame for my lung cancer.   .253   .643   .004
31. I have not gotten as much support from family or friends compared to people with other types of cancer. −.001   .225   .168
32. My family or friends have blamed me for having lung cancer.   .106   .783 −.136
33. When people who don’t know me well have learned about my lung cancer, they have asked if I smoked.   .168   .224 −.079
34. Friends or family have considered me responsible for getting lung cancer.   .186   .596   .022
35. People have told me I was to blame for getting lung cancer. −.100   .716 −.139
36. Medical providers have judged me negatively because I have lung cancer.   .063   .372   .054
37. People have treated me poorly because of my lung cancer. −.160   .725   .083
Open in a new tab

Note. Gray indicates items that were retained in factors for final LCSI. Red indicates items removed due to content redundancy.

Reliability

Internal consistency of the final 25-item LCSI scale was high (Cronbach’s alpha= 0.89) and the subscales also demonstrated good internal consistency (Factor 1 alpha = 0.90; Factor 2 alpha = 0.74; and Factor 3 alpha = 0.82). The test-retest correlation was also high (r = 0.91 when including prorated data from the 54 participants who completed at least 17 of 25 LCSI items at test and retest intervals; r = 0.91 when including the 37 participants who completed all LCSI items at both time points), suggesting strong stability of measurement over time.

Convergent validity

Table 4 details both full scale- and factor-level (subscale) comparisons between the LCSI and the CLCSS. The Pearson correlation between the full scale LCSI and the full scale CLCSS was r= 0.58 (p< 0.001), demonstrating good convergent validity. Factor-specific correlations indicated that the Internalized Stigma subscale of the LCSI demonstrated modest convergent validity with the total CLCSS scale (r=.39, p <.01) whereas the Perceived Stigma (r=.49, p <.01) and the Constrained Disclosure (r=.54, p <.01) subscales had good convergent validity with the total CLCSS scale. These correlations indicate that the new LCSI subscales are convergent with the subscales of the CLCSS, while still showing unique assessment of lung cancer stigma concepts.

Table 4.

Pearson Correlations between LCSI and CLCSS (Total and Subscales)

Demographic/attitudes 1 2 3 4 5 6 7 8 9
1. LCSI total —
2. LCSI Internalized Stigma .89** —
3. LCSI Perceived Stigma .73** .48** —
4. LCSI Constrained Disclosure .60** .30** .31** —
5. CLCSS total .58** .39** .49** .54** —
6. CLCSS stigma .60** .45** .41** .59** .87** —
7. CLCSS smoking .60** .50** .50** .31** .72** .48** —
8. CLCSS social isolation .33** .20** .31** .36** .88** .69** .47** —
9. CLCSS discrimination .45** .28** .43** .47** .86** .65** .58** .70** —
Open in a new tab

Note. LCSI = Lung Cancer Stigma Inventory; CLCSS = Cataldo Lung Cancer Stigma Survey

**

p < .001

Discussion

Based on current psychometric guidelines for PRO measure development and our conceptual model (Hamann et al., 2014), we developed a reliable, new measure of lung cancer stigma, the Lung Cancer Stigma Inventory (LCSI) (scale can be accessed through the National Cancer Institute Grid-Enabled Measures Database [GEM] https://www.gem-measures.org/Public/MeasureDetail.aspx?mid=2603&cat=2). In this paper, we report preliminary psychometric analyses demonstrating scale reliability and validity. Three unique, internally consistent, and conceptually relevant factors emerged: Perceived Stigma; Internalized Stigma; Constrained Disclosure. Two factors reflected our Phase I concept elicitation work identifying patient-reported themes of negative appraisal and devaluation from others (Perceived Stigma), as well as self-blame, guilt, and regret (Internalized Stigma). Although our initial conceptual model acknowledged social avoidance and disengagement, we were surprised by the strength of the third unique factor (Constrained Disclosure). The emergence of this third factor of Constrained Disclosure as a distinct, important element of lung cancer stigma sets the stage for the subsequent development and testing of potential interpersonal intervention pathways (i.e., empathic provider communication skills training, social support) to mitigate the negative effects of lung cancer stigma (e.g., depression, avoidance, inhibited symptom reporting, diminished health related quality of life).

Convergent validity was demonstrated by good association between the new LCSI and the existing CLCSS. Despite this association, the two measures are not redundant and include several points of divergence between the various subscales. For example, the LCSI Internalized Stigma factor was only moderately correlated with the total CLCSS. Additionally, the LCSI Constrained Disclosure factor was analytically distinct from a conceptually related factor in the CLCSS (Social Isolation), suggesting a unique construct reflecting a nuanced interpersonal aspect of stigma. Patient input from the Phase I qualitative interviews revealed aspects of stigma (e.g., social avoidance, disclosure discomfort) that were incorporated into the LCSI item pool. This three dimensional measurement model of the LCSI both enhances the lung stigma measurement literature and provides intra- and interpersonal targets for the subsequent development and testing of psychosocial interventions.

There are some limitations that should be noted and addressed in subsequent research. First, certain content areas, such as perceived stigma attributable to medical providers, did not yield items with strong psychometric properties and therefore were omitted from the final LCSI scale. While conceptually relevant, these items may have been worded in such a way as to capture only overt rather than more subtle experiences of perceived stigma encountered during typical interactions with health care providers. We are now conducting observational studies capturing actual patient-provider consultations to gain a better understanding of the nature of perceived stigma attributable to medical care providers. Despite multi-site recruitment, our sample had a limited representation of Latina/o, Asian American, and Native American participants. Further investigation with more inclusive samples, including assessments of measurement invariance across groups, will be important to establish the LCSI as an appropriate lung cancer stigma measurement tool for diverse populations. The LCSI is now only available in English and subsequent work is needed to translate this measure into other languages. Our sample size did not allow examination of stability of the results beyond the test-retest correlation analyses. Additional work will be needed to bolster knowledge about response stability. In addition, further work is needed to establish whether this new measurement tool can be used to examine clinically meaningful change in lung cancer stigma for use as a primary outcome measurement tool for planned trials of patient- and provider-focused interventions to mitigate stigma among lung cancer patients, and possible influences on treatment adherence, trial participation, and other outcomes.

Overall, strengths of this study include a rigorous psychometric protocol for scale development and validation characterized by iterative phases involving patient input and data collection. We made a concerted effort to conduct this study with a heterogeneous, clinic-based, multi-site sample of patients diagnosed with lung cancer, including never smokers. In addition to following a thorough process of patient-engaged item and scale development, the LCSI was compared to an existing lung cancer stigma survey providing initial evidence for convergent validity. The new 25-item LCSI was found to be stable and internally consistent, with three robust subscales representing relevant constructs identified in patient interviews. Guided both by social psychological theory and other investigations of health-related stigma, subsequent psychometric work will be conducted to establish further evidence of validity. For example, we will examine key associations of stigma with relevant patient-reported (e.g., depressive symptoms), behavioral (e.g., treatment adherence), and other cancer care outcomes. Future effort will also focus on the measure’s utility in identifying clinically significant levels of stigma and detecting clinically meaningful change associated with interventions.

Acknowledgments

We would like to thank David Gerber, Joan Schiller, Lisa Carter-Hams, Dinah Foster, Rachel Funk, Sharon Woodruff, and Jeff Kendall for their expert consultation on study items. Emily Marks, Cassidy Cisneros, Maria Funes, Robin Higashi, Allen Liao, Kenleigh Roden-Foreman, Adam Loewen, Joanne Sanders, Lisa Peterson, and Sarah Borderud all assisted with participant recruitment and data management. Janine Cataldo, Tom Atkinson, Ira Bernstein, and Sarah Price provided valuable feedback on previous drafts of this manuscript. This work was supported by grant awards from the National Cancer Institute (R03CA154016, Hamann; R03193986, Shen; T32CA009461, Ostroff), the National Lung Cancer Partnership and its North Carolina affiliate (now Free to Breathe; Hamann), the Lung Cancer Research Foundation (Ostroff), as well as a National Institutes of Health Support Grant (P30CA008748), which provides partial support for the MSKCC Behavioral Research Methods Core Facility used in conducting this investigation. The National Institutes of Health Support Grant (P30 CA142543-06) provided partial support for the Biostatistics Core at the Simmons Cancer Center. The UT Southwestern Center for Translational Medicine (NIH/NCATS UL1TR001105) facilitated our use of the REDCap system (NIH/NCATS UL1-RR024982).

References

  1. Aberle DR, Adams AM, Berg CD, Black WC, Clapp JD, Fagerstrom RM, Sicks JD. Reduced lung-cancer mortality with low-dose computed tomographic screening. New England Journal of Medicine. 2011;365(5):395–409. doi: 10.1056/NEJMoa1102873. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Alberg AJ, Brock MV, Ford JG, Samet JM, Spivack SD. Epidemiology of lung cancer: Diagnosis and management of lung cancer, 3rd ed.: American College of Chest Physicians evidence-based clinical practice guidelines. CHEST Journal. 2013;143(Suppl. 5):e1S–29S. doi: 10.1378/chest.12-2345. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. American Cancer Society. Cancer Facts & Figures 2016. 2016 Retrieved from http://www.cancer.org/acs/groups/content/@research/documents/document/acspc-047079.pdf.
  4. Barney LJ, Griffiths KM, Christensen H, Jorm AF. The Self-Stigma of Depression Scale (SSDS): Development and psychometric evaluation of a new instrument. International Journal of Methods in Psychiatric Research. 2010;19(4):243–254. doi: 10.1002/mpr.325. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Basch E, Spertus J, Dudley RA, Wu A, Chuahan C, Cohen P, Goertz C. Methods for developing patient-reported outcome-based performance measures (PRO-PMs) Value Health. 2015;18(4):493–504. doi: 10.1016/j.val.2015.02.018. [DOI] [PubMed] [Google Scholar]
  6. Bayer R, Stuber J. Tobacco control, stigma, and public health: Rethinking the relations. American Journal of Public Health. 2006;96(1):47–50. doi: 10.2105/AJPH.2005.071886. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Berger BE, Ferrans CE, Lashley FR. Measuring stigma in people with HIV: Psychometric assessment of the HIV stigma scale. Research in Nursing & Health. 2001;24(6):518–529. doi: 10.1002/nur.10011. [DOI] [PubMed] [Google Scholar]
  8. Carter-Harris L, Hermann CP, Schreiber J, Weaver MT, Rawl SM. Lung cancer stigma predicts timing of medical help-seeking behavior. Oncology Nursing Forum. 2014;41(3):E203–210. doi: 10.1188/14.ONF.E203-E210. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Cataldo JK, Brodsky JL. Lung cancer stigma, anxiety, depression and symptom severity. Oncology. 2013;85(1):33–40. doi: 10.1159/000350834. [DOI] [PubMed] [Google Scholar]
  10. Cataldo JK, Slaughter R, Jahan TM, Pongquan VL, Hwang WJ. Measuring stigma in people with lung cancer: Psychometric testing of the Cataldo Lung Cancer Stigma Scale. Oncology Nursing Forum. 2011;38(1):46–54. doi: 10.1188/11.ONF.E46-E54. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Cataldo JK, Jahan TM, Pongquan VL. Lung cancer stigma, depression, and quality of life among ever and never smokers. European Journal of Oncology Nursing. 2012;16(3):264–269. doi: 10.1016/j.ejon.2011.06.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Chambers S, Dunn J, Occhipinti S, Hughes S, Baade P, Sinclair S, O’Connel DL. A systematic review of the impact of stigma and nihilism on lung cancer outcomes. BMC Cancer. 2012;12(1):184. doi: 10.1186/1471-2407-12-184. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Chapple A, Ziebland S, McPherson A. Stigma, shame, and blame experienced by patients with lung cancer: A qualitative study. BMJ. 2004;325:1470. doi: 10.1136/bmj.38111.639734.7C. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Child D. The essentials of factor analysis. 2nd. London: Cassel Educational Limited; 1990. [Google Scholar]
  15. Clark HJ, Lindner G, Armistead L, Austin BJ. Stigma, disclosure, and psychological functioning among HIV-infected and non-infected African-American women. Journal of Women’s Health. 2003;38(4):57–71. doi: 10.1300/j013v38n04_04. [DOI] [PubMed] [Google Scholar]
  16. Durso LE, Latner JD. Understanding self-directed stigma: Development of the weight bias internalization scale. Obesity. 2008;16(Suppl. 2):S80–S86. doi: 10.1038/oby.2008.448. [DOI] [PubMed] [Google Scholar]
  17. Earnshaw VA, Quinn DM, Kalichman SC, Park CL. Development and psychometric evaluation of the chronic illness anticipated stigma scale. Journal of Behavioral Medicine. 2013;36(3):270–282. doi: 10.1007/s10865-012-9422-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Else-Quest NM, LoConte NK, Schiller JH, Hyde JS. Perceived stigma, self-blame, and adjustment among lung, breast and prostate cancer patients. Psychology & Health. 2009;24(8):949–964. doi: 10.1080/08870440802074664. [DOI] [PubMed] [Google Scholar]
  19. Fife BL, Wright ER. The dimensionality of stigma: A comparison of its impact on the self of persons with HIV/AIDS and cancer. Journal of Health and Social Behavior. 2000;41(1):50–67. [PubMed] [Google Scholar]
  20. Gonzalez BD, Jacobsen PB. Depression in lung cancer patients: The role of perceived stigma. Psycho-Oncology. 2012;21(3):239–246. doi: 10.1002/pon.1882. [DOI] [PubMed] [Google Scholar]
  21. Griffiths KM, Batterham PJ, Barney L, Parsons A. The Generalised Anxiety Stigma Scale (GASS): Psychometric properties in a community sample. BMC Psychiatry. 2011;11(1):184. doi: 10.1186/1471-244X-11-184. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hamann HA, Ostroff JS, Marks EG, Gerber DE, Schiller JH, Lee SJC. Stigma among patients with lung cancer: A patient-reported measurement model. Psycho-Oncology. 2014;23(1):81–92. doi: 10.1002/pon.3371. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Hamann HA, Lee JW, Schiller JH, Horn L, Wagner LI, Chang VT, Fisch MJ. Clinician perceptions of care difficulty, quality of life, and symptom reports for lung cancer patients: an analysis from the Symptom Outcomes and Practice patterns (SOAPP) study. Journal of Thoracic Oncology. 2013;8(12):1474–83. doi: 10.1097/01.JTO.0000437501.83763.5d. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics. 2009;42(2):377–81. doi: 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Holland JC, Kelly BJ, Weinberger MI. Why psychosocial care is difficult to integrate into routine cancer care: Stigma is the elephant in the room. Journal of the National Comprehensive Cancer Network. 2010;8(4):362–366. doi: 10.6004/jnccn.2010.0028. [DOI] [PubMed] [Google Scholar]
  26. Holzemer WL, Uys LR, Chirwa ML, Greeff M, Makoae LN, Kohi TW, Wantland D. Validation of the HIV/AIDS Stigma Instrument—PLWA (HASI-P) AIDS Care. 2007;19(8):1002–1012. doi: 10.1080/09540120701245999. [DOI] [PubMed] [Google Scholar]
  27. Kalichman SC, Simbayi LC, Cloete A, Mthembu PP, Mkhonta RN, Ginindza T. Measuring AIDS stigmas in people living with HIV/AIDS: The Internalized AIDS-Related Stigma Scale. AIDS Care. 2009;27(1):87–93. doi: 10.1080/09540120802032627. [DOI] [PubMed] [Google Scholar]
  28. Kanter JW, Rusch LC, Brondino MJ. Depression self-stigma: A new measure and preliminary findings. The Journal of Nervous & Mental Disease. 2008;196(9):663–670. doi: 10.1097/NMD.0b013e318183f8af. [DOI] [PubMed] [Google Scholar]
  29. Katz IT, Ryu AE, Onuegbu AG, Psaros C, Weiser SD, Bangsberg DR, Tsai AC. Impact of HIV-related stigma on treatment adherence: systematic review and meta-synthesis. Journal of the International AIDS Society. 2013;16(3 Suppl 2):18640. doi: 10.7448/IAS.16.3.18640. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Kincaid JP, Fishburne RP, Rogers RL, Chissom BS. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Memphis, Tennessee: Institute for Simulation and Training, University of Central Florida; 1975. [Google Scholar]
  31. King M, Dinos S, Shaw J, Watson R, Stevens S, Passetti F, Serfaty M. The Stigma Scale: Development of a standardised measure of the stigma of mental illness. The British Journal of Psychiatry. 2007;190(3):248–254. doi: 10.1192/bjp.bp.106.024638. [DOI] [PubMed] [Google Scholar]
  32. Kissane DW, Patel SG, Baser RE, Bell R, Farberov M, Ostroff JS, Shah JP. Preliminary evaluation of the reliability and validity of the Shame and Stigma Scale in head and neck cancer. Head & Neck. 2013;35(2):172–183. doi: 10.1002/hed.22943. [DOI] [PubMed] [Google Scholar]
  33. Lillis J, Luoma JB, Levin ME, Hayes SC. Measuring weight self-stigma: The Weight Self-Stigma Questionnaire. Obesity. 2010;18(5):971–976. doi: 10.1038/oby.2009.353. [DOI] [PubMed] [Google Scholar]
  34. LoConte NK, Else-Quest NM, Eickhoff J, Hyde J, Schiller JH. Assessment of guilt and shame in patients with non–small-cell lung cancer compared with patients with breast and prostate cancer. Clinical Lung Cancer. 2008;9(3):171–178. doi: 10.3816/CLC.2008.n.026. [DOI] [PubMed] [Google Scholar]
  35. Lynn MR. Determination and quantification of content validity. Nursing Research. 1986;35(6):382–386. [PubMed] [Google Scholar]
  36. MacDonald LD, Anderson HR. Stigma in patients with rectal cancer: A community study. Journal of Epidemiology and Community Health. 1984;38(4):284–290. doi: 10.1136/jech.38.4.284. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Major B, Kaiser CR, McCoy SK. It’s not my fault: When and why attributions to prejudice protect self-esteem. Personality and Social Psychology Bulletin. 2003;29(6):772–781. doi: 10.1177/0146167203029006009. [DOI] [PubMed] [Google Scholar]
  38. Mak WW, Cheung RY, Law RW, Woo J, Li PC, Chung RW. Examining attribution model of self-stigma on social support and psychological well-being among people with HIV+/AIDS. Social Science & Medicine. 2007;64(8):1549–1559. doi: 10.1016/j.socscimed.2006.12.003. [DOI] [PubMed] [Google Scholar]
  39. Morse DS, Edwardsen EA, Gordon HS. Missed opportunities for interval empathy in lung cancer communication. Archives of Internal Medicine. 2008;168(17):1853–8. doi: 10.1001/archinte.168.17.1853. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Nunnally JC, Bernstein IH. Psychometric Theory. 3rd. New York: McGraw-Hill; 1994. [Google Scholar]
  41. Patrick DL, Burke LB, Gwaltney CJ, Leidy NK, Martin ML, Molsen E, Ring L. Content validity-Establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: Part 2-Assessing respondent understanding. Value in Health. 2011;14:978–988. doi: 10.1016/j.jval.2011.06.013. [DOI] [PubMed] [Google Scholar]
  42. Quinn DM, Chaudoir SR. Living with a concealable stigmatized identity: the impact of anticipated stigma, centrality, salience, and cultural stigma on psychological distress and health. Journal of personality and social psychology. 2009;97(4):634–651. doi: 10.1037/a0015815. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Rao D, Choi SW, Victorson D, Bode R, Peterman A, Heinemann A, Cella D. Measuring stigma across neurological conditions: The development of the stigma scale for chronic illness (SSCI) Quality of Life Research. 2009;18(5):585–595. doi: 10.1007/s11136-009-9475-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Ritsher JB, Otilingam PG, Grajales M. Internalized stigma of mental illness: Psychometric properties of a new measure. Psychiatry Research. 2003;121(1):31–49. doi: 10.1016/j.psychres.2003.08.008. [DOI] [PubMed] [Google Scholar]
  45. Rosell R, Carcereny E, Gervais R, Vergnenegre A, Massuti B, Felip E, Paz-Ares L. Erlotinib versus standard chemotherapy as first-line treatment for European patients with advanced EGFR mutation-positive non-small-cell lung cancer (EURTAC): A multicentre, open-label, randomised phase 3 trial. The Lancet Oncology. 2012;13(3):239–246. doi: 10.1016/S1470-2045(11)70393-X. [DOI] [PubMed] [Google Scholar]
  46. Rothrock NE, Kaiser KA, Cella D. Developing a valid patient-reported outcome measure. Clinical Pharmacology & Therapeutics. 2011;90(5):737–742. doi: 10.1038/clpt.2011.195. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Sayles JN, Hays RD, Sarkisian CA, Mahajan AP, Spritzer KL, Cunningham WE. Development and psychometric assessment of a multidimensional measure of internalized HIV stigma in a sample of HIV-positive adults. AIDS and Behavior. 2008;12(5):748–758. doi: 10.1007/s10461-008-9375-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Shen MJ, H A, Thomas AJ, Ostroff JS. Association between patient-provider communication and lung cancer stigma. Supportive Care in Cancer, Advance online publication. 2015 doi: 10.1007/s00520-015-3014-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Sriram N, Mills J, Lang E, Dickson HK, Hamann HA, Nosek BA, Schiller JH. Attitudes and Stereotypes in Lung Cancer versus Breast Cancer. PLoS One. 2015;10(2):e0145715. doi: 10.1371/journal.pone.0145715. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Sowell RL, Lowenstein A, Moneyham L, Demi A, Mizuno Y, Seals BF. Resources, stigma, and patterns of disclosure in rural women with HIV infection. Public Health Nursing. 1997;14:302–312. doi: 10.1111/j.1525-1446.1997.tb00379.x. [DOI] [PubMed] [Google Scholar]
  51. Sweeney SM, Vanable PA. The Association of HIV-Related Stigma to HIV Medication Adherence: A Systematic Review and Synthesis of the Literature. AIDS and Behavior. 2016;20(1):29–50. doi: 10.1007/s10461-015-1164-1. [DOI] [PubMed] [Google Scholar]
  52. Tsuchiya M, Horn S, Ingham R. Development of the Psycho-social Discomfort Scale (PsDS): Investigation of psychometric properties among Japanese breast cancer survivors. Psycho-Oncology. 2012;21(2):161–167. doi: 10.1002/pon.1880. [DOI] [PubMed] [Google Scholar]
  53. Turan B, Smith W, Cohen MH, Wilson TE, Adimora AA, Merenstein D, Turan JM. Mechanisms for the Negative Effects of Internalized HIV-Related Stigma on Antiretroviral Therapy Adherence in Women: The Mediating Roles of Social Isolation and Depression. Journal of Acquired Immune Deficiency Syndromes. 2016;72(2):198–205. doi: 10.1097/QAI.0000000000000948. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. U.S. Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, & Center for Devices and Radiological Health. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. 2009 Retrieved from http://www.fda.gov/downloads/Drugs/…/Guidances/UCM193282.pdf.
  55. Van Brakel WH. Measuring health-related stigma- a literature review. Psychology, Health, and Medicine. 2006;11(3):307–314. doi: 10.1080/13548500600595160. [DOI] [PubMed] [Google Scholar]
  56. Van Rie A, Sengupta S, Pungrassami P, Balthip Q, Choonuan S, Kasetjaroen, Chongsuvivatwong V. Measuring stigma associated with tuberculosis and HIV/AIDS in southern Thailand: Exploratory and confirmatory factor analyses of two new scales. Tropical Medicine & International Health. 2008;13(1):21–30. doi: 10.1111/j.1365-3156.2007.01971.x. [DOI] [PubMed] [Google Scholar]
  57. Visser MJ, Kershaw T, Makin JD, Forsyth BW. Development of parallel scales to measure HIV-related stigma. AIDS and Behavior. 2008;12(5):759–771. doi: 10.1007/s10461-008-9363-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Weiss T, Weinberger M, Schwerd AM, Holland J. A 30-year perspective on psychosocial issues in lung cancer: How lung cancer “Came Out of the Closet”. Thoracic Surgery Clinics. 2012;22(4):449–56. doi: 10.1016/j.thorsurg.2012.07.008. [DOI] [PubMed] [Google Scholar]
  59. Willis GB, Royston P, Bercini D. The use of verbal report methods in the development and testing of survey questionnaires. Applied Cognitive Psychology. 1991;5(3):251–267. doi: 10.1002/acp.2350050307. [DOI] [Google Scholar]
  60. Willis GB. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: SAGE Publications, Inc.; 2005. [Google Scholar]
  61. Wright K, Naar-King S, Lam P, Templin T, Frey M. Stigma scale revised: Reliability and validity of a brief measure of stigma for HIV-positive youth. Journal of Adolescent Health. 2007;40(1):96–98. doi: 10.1016/j.jadohealth.2006.08.001. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES