Psychometric evaluation of the HIV symptom distress scale

Abstract

The objective of this paper is to psychometrically validate the HIV Symptom Distress Scale (SDS), an instrument that can be used to measure overall HIV symptom distress or clinically relevant groups of HIV symptoms. A secondary data analysis was conducted using the Collaborations in HIV Outcomes Research U.S. Cohort (CHORUS). Inclusion criteria required study participants (N=5,521) to have a valid baseline measure of the AIDS Clinical Trial Group Symptom Distress Module, with an SF-12 or SF-36 completed on the same day. Psychometric testing assessed unidimensionality, internal consistency and factor structure using exploratory and confirmatory factor analysis, and structural equation modeling (SEM). Construct validity examined whether the new measure discriminates across clinical significance (CD4 and HIV viral load). Findings show that the SDS has high reliability (α=0.92), and SEM supports a correlated second-order factor model (physical and mental distress) with acceptable fit (GFI=0.88, AGFI=0.85, NFI=0.99, NNFI=0.99; RMSEA=0.06, [90% CI 0.06 – 0.06]; Satorra Bentler Scaled, C2 =3274.20; p=0.0). Construct validity shows significant differences across categories for HIV-1 viral load (p< 0.001) and CD4 (p< 0.001). Differences in mean SDS scores exist across gender (p< 0.001), race/ethnicity (p< 0.05) and educational attainment (p < 0.001). Hence, the HIV Symptom Distress Scale is a reliable and valid instrument, which measures overall HIV symptoms or clinically relevant groups of symptoms.

Introduction

Over the past twenty years self-report instruments have been used for the assessment of quality-of-life in patients diagnosed with HIV/AIDS (Cleary, Fowler et al. 1993; Bozzette, Hays et al. 1994; Nokes, Wheeler et al. 1994; Whalen, Antani et al. 1994; Lenderking, Testa et al. 1997; Gifford, Shively et al. 1998). Early in the HIV/AIDS epidemic several researchers adopted the modular approach to quality-of-life assessments, treating each domain as a separate module (Lenderking, Testa et al. 1997). Generic measures had the advantage of enabling comparisons across different diseases, while specific measures had the sensitivity to detect smaller but clinically meaningful changes (Hays and Shapiro 1992; Cleary, Fowler et al. 1993). Specific to the domain symptom distress an instrument was designed and validated to identify perceived individual symptoms reported by HIV infected patients (Justice, Holmes et al. 2001), known as the HIV Symptom Distress Module (SDM). The construct distress, in an HIV population, is characterized by phenomena such as fatigue/low energy, fevers, dizziness, tingling/hand/foot pain, memory loss, nausea/vomiting, diarrhea, depression, anxiety, sleep problems, skin problems, cough/shortness of breath, headache, appetite loss, bloating/gas, muscle/joint pain, sexual problems, weight gain weight loss, and hair loss. Presently, this 20-item SDM is widely incorporated as part of the battery of self-report tools used in clinical studies, with the intended purpose to improve the measurement of symptoms for clinical management, patient-oriented research and adverse drug reactions (Justice, Chang et al. 2001).

Due to interest amongst clinical trial researchers to report on overall HIV symptom phenomena or clinically relevant groups of HIV symptoms, a summative rating scale for the 20-item SDM has been developed called the HIV Symptom Distress Scale (SDS). Its utility will most likely benefit HIV clinician/researchers measuring patient-reported outcomes in comparative HIV trials. Thus the primary aim of this article is to describe the validation of the SDS and report on its factor structure and psychometric properties examined across sociodemographic groups.

Methods

Data Source and Data Collection

This is a secondary data analysis of a cross-sectional subset of the Collaborations HIV Outcomes Research US Cohort (CHORUS), which is a population-based observational study designed to follow the clinical and epidemiological outcomes of adults with HIV infection in the US (Becker, Raffanti et al. 2001). Through the use of an electronic medical record (EMR), the project received real time data collected during routine medical visits. Beginning in August 1997 through 2005, participating physicians offered the opportunity to participate in the project to all of their HIV positive patients who were 18 years of age and older. Information captured at every clinic visit are medical history, physical assessment and plan, medications, laboratories, procedures, and demographics, which has been described in detail elsewhere (Becker, Raffanti et al. 2001; Marc, Zerden et al. 2011). Quality of life (SF-12 and SF-36 (Ware and Sherbourne 1992; Gandek, Ware et al. 1998) and the 20-item HIV SDM (Justice, Holmes et al. 2001)) were captured through patient-completed questionnaires every six months. A proportion of the study participants are foreign born and were administered a Haitian-Creole and Spanish language versions of these instruments.

Study Inclusion Criteria

As of September 2005, a total of 9,197 patients consented to participate in CHORUS. Amongst this population, a total of 5,521 participants met the criteria for inclusion into a study focused on HIV quality of life, which created this subset of CHORUS (Marc, Wang et al. 2007). Inclusion criteria required that subjects have: (a) a valid baseline SDM completed, with (b) an SF-12 or SF-36 completed on the same day. A valid SDM was defined as one with all the items endorsed, or an instrument with only positive answers (response = 1 to 4) indicated. A valid SF-12 was defined as one with at least half of the questions answered to be scored. For patients with an SF-36, only the subset of SF-12 questions were used for the analysis and were scored according to the algorithm for SF-12 (Ware and Sherbourne 1992; Gandek, Ware et al. 1998).

Study Population

Participants in this subset of CHORUS (N=5,521) are predominately male (88.6%) with primary mode of HIV transmission reported as homosexual/bisexual exposure (75.5%) (Table 1). The median age is 43 years (SD ± 8.7) and median values for baseline immune parameters are 429 CD4 cells/μl and 2.6 for HIV log₁₀ RNA copies/ml. This population is ethnically diverse with representation among non-Hispanic White (70.7%), African American (17.6%) and Hispanic (7.8%) participants.

Table 1.

Baseline Population Characteristics

Number of Subjects (N)	5,521
Demographics
Mean Age in Years (±SD)	43 (±8.7)
Range in Years	18–91
Male	4,892 (88.6%)
Female	629 (11.4%)
Immune Parameters
CD4 Count (Median ± SD)	429 (± 275.6)
Log10 RNA (Median ± SD)	2.6 (± 1.2)
Race/Ethnicity
White	3,905 (70.7%)
Black	971 (17.6%)
Hispanic	430 (7.8%)
Other	215 (3.9%)
Mode of Transmission
Homosexual/Bisexual	4,168 (75.5%)
Heterosexual	866 (15.7%)
Occupational	41 (0.7%)
Injection Drug Use	232 (4.2%)
Blood Products	70 (1.3%)
Unknown	144 (2.6%)

Ethics

The CHORUS Project received Independent Review Board approval at Vanderbilt University (Nashville, TN), the Mount Sinai School of Medicine (New York, NY), and Copernicus Group IRB (Cary, NC), described elsewhere (Becker, Raffanti et al. 2001). This secondary analysis was approved by the Institutional Review Board of the Weill Medical College of Cornell University.

Measures

HIV Symptom Distress Module

The 20-item SDM (Justice, Holmes et al. 2001) uses a rating scale from 0 to 4 points. Study participants were asked to rate the degree of bother they experienced for each symptom during the prior two weeks (‘0=do not have symptom’, ‘1=have symptom, but no bother’, ‘2=have symptom, little bother’, ‘3=have symptom, bother’, ‘4= have symptom, bothers me a lot’) (a higher score indicating greater symptom distress). Based upon the inclusion criteria and coding algorithm for this dataset, all 20-items have a response endorsed. Imputations were not performed on missing SDM data. If patients did not respond this was interpreted as “never experiencing” the symptom and coded as ‘0’.

HIV Symptom Distress Scale

Responses for the 20-item SDM were transformed using a range of 0 to 100, maintaining the order of the categories. The ordinal variables were multiplied by a constant of 25 to create the new summative rating scale (Spector 1992). Consistent with the name of the instrument a higher score indicates greater symptom distress (Table 2).

Table 2.

Characteristics of the 20-Item Symptom Index (Original and Rescaled)

		Rescaled†
Variables	Number of Observations	Mean	Std. Deviation
energy	5521	47.70	35.93
fevers	5521	20.43	29.81
dizziness	5521	23.20	30.69
tingling	5521	30.80	35.49
remembering	5521	30.95	33.79
nausea	5521	16.99	28.15
diarrhea	5521	32.09	34.48
depression	5521	41.50	34.61
anxiety	5521	38.77	34.92
insomnia	5521	41.20	37.60
itching	5521	31.25	35.10
coughing	5521	20.77	30.39
headaches	5521	25.37	32.13
appetite loss	5521	19.96	30.93
bloating	5521	30.53	34.55
muscle ache	5521	33.99	36.15
sexual problems	5521	35.87	37.74
weight gain	5521	33.34	37.57
weight loss	5521	20.01	32.41
hair loss	5521	16.17	28.32

^†Original scaling was recoded as follows:

0=”I do not have symptoms” to “0”(Best Health, No Symptom Distress);

1= “Have symptom, it doesn’t bother me” to “25”;

2= “Have symptom, it bothers me a little” to “50”;

3= “Have symptom, it bothers me” to “75”, and

4= “Have symptom, it bothers me a lot” to “100” (Worst Health, Greatest Symptom Distress)

SF-12

The construction of the SF-12 was based on a pool of items developed as part of the Medical Outcomes Study (Wu, Hays et al. 1997), and there is substantial evidence for its’ reliability, construct and predictive validity and responsiveness (Delate and Coons ; Ware and Sherbourne 1992; Gandek, Ware et al. 1998). The scale is scored on a 0–100 scale (a higher score indicates better health), with physical and mental health subscales (Ware, Kosinshi et al. 1995).

Sociodemographic Variables

Educational level was categorized as a five-level ordinal variable: less than high school (1); high school (2); some college (3), college graduate (4), or graduate (5). Race/ethnicity was categorized as a four-level nominal categorical variable: White (1), Black (2), Hispanic (3) and Other (4).

Statistical Analysis

Descriptive Statistics

Univariate statistics were generated, and ceiling/floor effects were considered substantial if ≥ 20% of subjects scored at the upper-most or bottom-most end of the scale (Testa and Nackley 1994). Internal consistency reliability was assessed using Cronbach’s alpha (DeVellis 2003). The method of known-groups comparisons (Spector 1992) was used to evaluate construct validity to determine the extent to which the SDS score was able to discriminate between patients differing clinically according to their immune parameters (HIV disease severity measured with baseline CD4 and HIV viral load). A test for trend was used to examine construct validity across the ordered groups for HIV viral load and CD4 counts. Between group comparisons were performed across sociodemographics using two tailed t-tests for continuous variables and chi-square for tests of proportions. The alpha-level used for determining significance was 0.05. Software programs used to manage and analyze the data included STATA (CCISD-CECI 2005), SPSS (2001), and EXCEL (Microsoft Corporation. 2000).

Factor Analysis and Model Testing

Exploratory and confirmatory factor analysis were performed to evaluate unidimensionality and the factor structure of the CHORUS symptom distress data. Exploratory factor analysis accounts for the correlation structure among the observed variables. The method of principal component analysis was used to identify the unobservable ‘latent’ factors that underlie or explain the set of observed variables (Nunnally 1978; Streiner and Norman 1989; Coste, Bouee et al. 2005). A scree plot of the positive eigenvalues of the adjusted correlation matrix was generated to determine if one or two factors accounted for most of the common variances among the 20-items.

Confirmatory factor analysis (CFA) was used to determine the number and nature of the different constructs measured by the instrument. The analysis involved validating certain a priori hypotheses and its anticipated relationships between the observed and latent variables (Whalen, Antani et al. 1994; Lenderking, Testa et al. 1997). The hypotheses purport that the symptoms represent a single domain of health for HIV-infected patients – specifically the underlying construct of symptom distress.

Since the original response choices for the SDM are ordinal and polychotomous in nature ranging from (0 to 4), PRELIS software was used to calculate polychoric correlations accounting for the ordinality of items (Byrne 1998). LISREL analysis was then conducted using the polychoric correlations obtained using PRELIS (Jöreskog and Sörbom 1998).

The factor analysis included all 20 items on the observations with full data. For the one-factor model the structure was specified with one latent variable (factor) according to the corresponding observed variables (items). The two-factor model specified two latent variables. Model fit was evaluated primarily with descriptive fit statistics, incremental fits, and errors of approximation (Bentler and Bonnet 1980; Browne and Cudeck 1993; Kline 2004).

Results

Scale Statistics and Reliability

Univariate statistics show a mean score (x̄=29.5) on a 0 to 100 point scale (higher score indicating greater symptom distress). There were no ceiling or floor effects observed, with 4.1% (N=224) of the study population having a minimum score of 0, and 0.1% with a maximum score of 100; with α=0.92 for the summary measure, confirming a high level of internal consistency. The reliability coefficient suggests that a large proportion of the scale’s total variance is attributed to a common source. A stratified analysis across select sociodemographic groups (i.e., age, educational attainment, gender and race/ethnicity) was performed to assess representative reliability of the new summary measure, showing internal consistency ranging from α=0.85 to α=0.93 across these groups.

Construct Validity

Using a test for trend results suggest a statistically significant monotonic relationship between the SDS score, HIV viral load and CD4 counts. As the scale score increases suggesting greater HIV symptom distress, HIV-1 viral load significantly increases (Z= 8.54, p <0.001), even after adjusting for age and baseline measures of CD4 cells/μl, duration of antiretroviral therapy (ART), exposure to ART and AIDS defining event (Table 3). Conversely, a statistically significant inverse monotonic relationship is observed with CD4 counts (Z= −13.15, p <0.001), controlling for age and baseline measures of HIV log₁₀ RNA, duration of ART, exposure to ART and AIDS defining event.

Table 3.

Construct Validity Across Categories of Baseline HIV Viral Load and CD4

HIV RNA	Obs	Mean SDS	Std. Dev.	Min	Max	p-value∫
0–499	1,944 (35.2%)	25.9	19.4	0	100	< 0.001
500–4,999	620 (11.2%)	30.2	21.3	0	87.5
5,000–49,999	681 (12.3%)	30.9	22.0	0	100
50,000–499,999	421 (7.6%)	34.3	22.4	0	100
500,001+	42 (0.8%)	41.3	23.5	2.5	87.5
Missing	1,813 (32.8%)	-	-	-	-

CD4 cells/μl	Obs	Mean SDS	Std. Dev.	Min	Max	p-value∫
<100	263 (4.8%)	41.6	22.7	0	95	< 0.001
100–199	368 (6.7%)	34.7	21.6	0	87.5
200–299	583 (10.6%)	31.0	20.8	0	93.75
300–500	1,334 (24.2%)	29.1	20.8	0	100
>=501+	1,675 (30.3%)	25.1	19.5	0	100
Missing	1,298 (23.5%)

^∫A test for trend across the ordered groups.

Sociodemographic Characteristics

Gender

Examining the crude data, male study participants reported significantly lower symptom distress scores (x̄=28.8) compared to female participants (x̄=35.5). This remained statistically significant (p <0.001) even after adjusting for age and baseline measures of HIV log₁₀ RNA, CD4 cells/μl, duration of ART, exposure to ART and AIDS defining event (Table 4).

Table 4.

Mean SDS Score Across Sociodemographic Groups

SDS Summary Measure	Male	Female
Observations (N)	4892	629
Mean	28.8	35.5
SD	20.7	24.2
Min	0	0
Max	100	100
p-value†		M:F <0.001†
p-value‡ (Adjusted)		M:F <0.001‡

SDS Summary Measure	White	Black	Hispanic	Other
Observations (N)	3905	971	430	215
Mean†	29.8	28.5	29.2	30.4
SD	20.8	22.6	21.5	22.3
Min	0	0	0	0
Max	100	100	88.8	85
p-value†		B:W <0.001†	B:H <0.001†
p-value‡ (Adjusted)		B:W <0.001‡	B:H =0.003‡

SDS Summary Measure	<HS	HS	Some College	College Grad	Post- Graduate
Observations (N)	303	1055	1521	1561	776
Mean†	37.9	32.7	30.5	25.3	26.3
SD	24.2	23.3	21.1	18.9	19.3
Min	0	0	0	0	0
Max	100	95	95	100	100
p-value†	<0.001	<0.001	<0.001	ns	Reference
p-value‡ (Adjusted)	N=3267, β= −2.19; F =42.30 (7,3259); p < 0.001

^†Student’s T-Test

^‡Adjusting for age and baseline measures.

Definitions: HIV-1 RNA viral load results were closest to baseline, defined as 0 – 90 days prior to baseline log transformed, continuous. CD4 results were closest to baseline, defined as 0 – 90 days prior to baseline (cells/μl), continuous. Duration of ART prior to baseline (months), (continuous, naïve patients have zero months). Baseline exposure to antiretroviral therapy is defined as naïve to ART, ART-experienced and currently on therapy, or ART-experienced and not currently on therapy. AIDS-defining event (clinical diagnosis) at baseline, dichotomous, 0 = no, 1 = yes)

Race/Ethnicity

Black study participants were less likely to endorse items indicative of bothersome symptoms, compared to White participants, even when the severity of their HIV disease was greater. The mean symptom distress score for Black participants(x̄=28.5) was statistically significantly lower when compared to White (x̄=29.8, p=0.002) and Hispanic study participants (x̄=29.2; p=0.02). However, this finding was unexpected because Black participants had greater severity of HIV disease at baseline when compared to White participants. The mean baseline viral load for Blacks compared to Whites was statistically significantly higher (B_x̄=3.27 copies/ml; W_x̄=2.91 copies/ml; p=<0.001), and mean CD4 counts were statistically significantly lower (B_x̄=410/μl; W_x̄=488 cells/μl; p=<0.001). These observed differences persisted even after adjusting for age, baseline measures of HIV-1 viral load, CD4, duration of ART, exposure to ART and AIDS defining event.

Educational Attainment

In a crude model, individuals with lower levels of education reported greater bothersome symptoms. Using a multivariate linear regression model, this association remained statistically significant (N=3267, β= −2.19; F =42.30 (7,3259); p < 0.001) even after adjusting for age, gender, race and baseline measures of HIV log10 RNA, CD4 count, duration of ART, exposure to ART and AIDS defining event (Table 4).

Exploratory and Confirmatory Factor Analysis

Results from the initial estimates of the principal component analysis suggest that the first common factor is associated with an eigenvalue of 7.52, which accounts for 37.6% of the proportion of the variance in the total communality. Treating the variables as ordinal data, two out of four fit indexes were greater than 0.90 [goodness-of-fit (GFI)=0.83, adjusted goodness-of-fit (AGFI)=0.80, normed fit index (NFI)=0.98 and non-normed fit index (NNFI)=0.98] for the one-factor model. The root mean square error of approximation suggests an acceptable fit (RMSEA=0.07, [90% CI 0.07 – 0.07]); with the Chi-square Corrected for Non-normality (χ2 =2672.69; p=0.0).

When the scree plot of eigenvalues was generated results showed a clear drop from the first factor to the second, which supports the hypothesis that most questions are related to a single common factor symptom distress. A two-factor confirmatory analysis was then performed using maximum likelihood estimation procedures. Results suggest that two factors are associated with 41.4% of the variance accounted for in the total communality.

Comparing the RMSEA between the one factor and two-factor CFA model using polychoric correlations, we found the two-factor model a better fit, with two out of four fit indexes greater than 0.90 (i.e., GFI=0.88, AGFI=0.85, NFI=0.99 and NNFI=0.99). Here, the root mean square error of approximation suggests an acceptable fit with the variables treated as ordered categories (RMSEA=0.06, [90% CI 0.06 – 0.06]), with the Satorra Bentler Scaled Chi-square (χ2 =3274.20; p=0.0) and Chi-square Corrected for Non-normality (χ2 =2271.52; p=0.0). From this analysis two primary factors emerged, referred to as mental and physical symptom distress, reflecting clinically relevant groups of symptoms. The items for depression, anxiety and insomnia loaded on the “mental” factor, and all other items loaded on the “physical” factor.

Statistical Properties of the Symptom Distress Sub-Scales

Results from univariate statistics show the median score equal to 41.67 (SD± 30.4) for the mental subscale, and 23.5 (SD± 21.0) for the physical subscale. There are no ceiling or floor effects for either, and α=0.81 and α=0.90, respectively. Pearson correlations show a statistically significant relationship between the two (r=0.70; p<0.001).

Comparison of SF-12 and Symptom Distress Subscales

A comparison of the SDS to the SF-12 confirms that the subscales for both instruments have a statistically significant inverse association (r_mental = −0.7331, p <0.001; (r_physical =−0.6550, p <0.001), respectively.

Limitations

There are several limitations to this study, specifically three types of missing data in this subset of CHORUS: (1) Unendorsed items on the SDM were set to “0” with the assumption that respondents who failed to endorse did not experience a symptom; (2) there is educational information coded as unknown or missing; and (3) although non-English instruments were administered in Haitian-Creole and Spanish a variable for ‘language version’ is not available in CHORUS.

Unanswered Questions on the Symptom Distress Module

There is no way to distinguish whether the unanswered questions resulted from the patient failing to endorse the item, or that the patient did not experience the symptom. As such, descriptive and exploratory factor analyses were re-run on the data recoding all unanswered questions that were set to “0” (“I do not have this symptom”) reset as “missing”. This was done to determine if the factor structure of the data would change.

First, the pattern of missing data for each of the 20- items was evaluated to examine whether unendorsed items were systematic in the order of appearance in the SDM. Results confirm that the proportion of patients not answering the question did not increase for items appearing at the lower positions of the instrument.

A polychoric exploratory factor analysis was performed using M-Plus software (Muthén and Muthén 2007) assuming missing at random (MAR) for unanswered questions. MAR means that the missing data are not related to the value of the variables in the SDM, though they may be related to the underlying factors of the SDS. Results show practically the same factor structure as that obtained when unanswered questions were coded “0” (do not have the symptom). Both sets of analyses support a correlated second-order factor model with subscales related to mental and physical HIV symptom distress.

Missing Educational Data

A sensitivity analysis was performed to determine whether subjects with unknown or missing educational information (N=305 combined) differed in any way from those who reported educational milestones, across age, race, gender or immune parameters. Ad hoc analyses showed that participants with this type of missing data did not differ significantly from their counterparts across sociodemographic characteristics. However, participants with missing educational information had greater severity of disease reporting a statistically higher mean baseline log HIV viral load (x̄ =3.23±1.17; N=157; df=3706; t=0.004) than their counterparts reporting educational milestones (x̄ =2.94±1.24; N=3551).

Missing Language Version

A limitation of the reliability analyses is that the “language version” was not collected as part of the original CHORUS dataset, and information on the translation method was not provided with this copy of the dataset. Therefore all versions of the instrument have been evaluated as part of the psychometric analyses. Due to the fact that information on country of birth was available for 69.1% (N=3744) of the study participants a sensitivity analysis excluded all foreign-born (N=308) to assess Cronbach’s alpha. Findings showed that the reliability of the summary measure remained unchanged at α = 0.92 for the American-born study participants (N=3,436).

Discussion

The greatest utility of the HIV Symptom Distress Scale (SDS) is to clinician/researchers involved in comparative clinical trials measuring improvement, or decline of HIV symptom distress or clinically relevant groups of symptoms in an HIV-infected population. In order for a clinical measure to be useful it must be sensible, reliable and valid (2006).

First, the SDS is sensible because it is clinically applicable, easily understood and easy to score. Second, the SDS is reliable as demonstrated by the high internal consistency of the overall measure, indicating that the symptoms are related to a single underlying construct. Structural equation modeling supports a correlated second-order factor model with subscales related to mental and physical HIV symptom distress. Third, study findings establish the validity of the SDS, which significantly discriminates clinical differences across baseline measures of HIV-1 viral load and CD4. Fourth the SDS detects statistical differences between self-reports from diverse sociodemographic groups, which is consistent with other literature showing that women with HIV/AIDS report substantially poorer health than men in several HRQOL domains (Wu, Revicki et al. 1997; Mrus, Williams et al. 2005); persons with lower educational levels report poorer HIV quality-of-life than there more educated counterparts (Worthington and Krentz 2005); and Black participants have greater severity of clinical disease than Whites (Raczynski, Taylor et al. 1994; Sarma, Wei et al. 2003). Finally, future comparative HIV trials should anchor the SDS to clinical results and/or compare the SDS to scores of other health status instruments that measure distress. This strategy will help to interpret the meaning of the SDS and help to establish cut-off scores for HIV clinical distress across studies.

Figure 1.

Path Diagram for the One-Factor Model (Polychoric Correlation)

Figure 2.

Two-Factor Model (Polychoric Correlation)