Reliability of health-related quality-of-life indicators in cancer survivors from a population-based sample, 2005, BRFSS

Summary

Objective

The current emphasis in cancer survivorship research, which includes health-related quality of life (HRQoL), drives the need to monitor the nation’s cancer burden. Routine, ongoing public health surveillance tools, such as the Behavioral Risk Factor Surveillance System (BRFSS), may be relevant for this purpose.

Study design

A subsample of the 2005 Missouri BRFSS was used to estimate test–retest reliability of HRQoL questions among persons who did and did not report a personal cancer history.

Methods

Retest interviews were conducted by telephone 14–21 days after the initial data collection (n=540, 67% response rate). Reliability was estimated overall and by cancer history using intraclass correlation coefficients (ICCs) and kappa statistics.

Results

The majority of retest respondents were White, female and married, with 13% reporting a history of cancer. Overall, point estimates of the reliability coefficients ranged from moderate to excellent (κ=0.57–0.75). There were no statistically significant differences in test–retest reliability between persons with and without a history of cancer, except for self-reported pain (ICC=0.59 and ICC=0.78, respectively).

Conclusions

In general, BRFSS questions appear to have adequate reliability for monitoring HRQoL in this community-dwelling population, regardless of cancer history.

Introduction

One in four deaths in the USA is attributed to cancer.¹ Cancer is the second leading cause of death in the USA and, as such, is a leading public health concern.¹ Cancer is distinguishable from other diseases by its related symptoms, such as high levels of pain, debilitating treatment sequelae² and its long-term impact on function and well-being.³ Given advances in early detection and treatment, cancer survivorship research now focuses on late effects of treatment, quality of life, follow-up care and post-treatment surveillance.⁴

The National Cancer Institute’s Cancer Surveillance Research Implementation Plan⁵ has outlined population monitoring as an avenue for measuring the nation’s cancer burden. This includes monitoring outcomes beyond incidence, survival and mortality, such as health-related quality of life (HRQoL). Comparing HRQoL in cancer populations with the population at large requires a comparison with normative groups using a generic measure.^2,3 Further, when the objective is to monitor the nation’s cancer burden for policy guidance, a HRQoL metric used in multiple diseases and the general population may be warranted.³ Few national data sources currently exist that can be used to monitor HRQoL among cancer survivors⁶ in comparison with the general population.

One data source for monitoring HRQoL among cancer survivors nationally as well as locally may be the Behavioral Risk Factor Surveillance System (BRFSS). A review of more than 60 studies from the Centers for Disease Control and Prevention (CDC) websites^7,8 and of literature from a search for studies that assessed data quality, validity and reliability of indicators using the BRFSS found few reports on the reliability of HRQoL. A comprehensive review of BRFSS reliability from 2001⁹ reported moderate reliability for the general health question (κ=0.42–0.47). However, this general health measure resulted from a study of 35–63-year-old adults in Finland, with a 1-year post test.¹⁰ The review also reported that the general health scale from the Medical Outcomes Study 36-Item Short Form had an intraclass correlation coefficient (ICC) of 0.87.¹¹ No reliability studies had been undertaken regarding general health or quality of life (poor physical or mental health days in the past month) from the BRFSS at the time of the review.⁹ Only one known BRFSS study has since examined the reliability of HRQoL indicators, using a population-based sample in Missouri.¹²

The population of cancer survivors in the USA is increasing steadily, reaching almost 10 million by 2001, or 3.5% of the population.¹³ Due to treatment side-effects, late complications and secondary cancers, the reliability of HRQoL measures may differ between cancer survivors and the general population. To the authors’ knowledge, no population-based study has compared the reliability of HRQoL questions from the BRFSS across cancer history. This paper describes the results of a test–retest study of HRQoL indicators across cancer history using a subsample of the Missouri BRFSS participants.

Methods

Sample

The BRFSS is an ongoing state-based, computer-assisted telephone survey system used to track health conditions and risk behaviours. It is implemented annually by the 50 state health departments, the District of Columbia, Puerto Rico, Guam and the US Virgin Islands in collaboration with the CDC. The BRFSS is a cross-sectional survey that uses random-digit-dial techniques to survey non-institutionalized adults aged ≥18 years in the USA (http://www.cdc.gov/brfss). During 2005, Missouri completed 5164 BRFSS surveys^14,15 for a median Council of American Survey Research Organizations (CASRO) response rate of 57.8% and cooperation rate of 78.5%.¹⁶ Of these participants, 867 Missourians were interviewed in April or May (CASRO rate: 55.2% and 55.1%; cooperation rate: 76.0% and 75.7%, respectively).¹⁷ Comparatively, the median CASRO rate for 2005 for all states was 51.1% (range 34.6–67.4%) and the median cooperation rate was 75.1% (range 58.7–85.3%).¹⁶ Of the 867 respondents potentially eligible for the retest interview, 56 were excluded from being re-interviewed for not completing or only partially completing (through the demographics section)¹⁸ the initial survey. The remaining 811 participants were invited to participate in a second (retest) interview 14–21 days after completing their initial BRFSS survey. All calls were made by trained interviewers at the University of Missouri-Columbia according to CDC’s standard BRFSS protocol.¹⁸

Instrument and definitions

Cancer history was determined by a ‘yes’ or ‘no’ response to ‘Have you ever been told by a doctor, nurse or other health professional that you had cancer?’

Several HRQoL indicators were collected, including self-rated health, social and emotional support, life satisfaction, physical and mental health status, pain and activity limitations. Response options to the general health question, ‘Would you say that, in general, your health is’ were: ‘poor’, ‘fair’, ‘good’, ‘very good’ or ‘excellent’. Response options to ‘How often do you get the social and emotional support you need?’ were ‘always’, ‘usually’, ‘sometimes’, ‘rarely’ or ‘never.’ Life satisfaction response options (‘In general, how satisfied are you with your life?’) ranged from ‘very satisfied’ to ‘very dissatisfied’. The following were treated as continuous variables: ‘Now thinking about your physical health, which includes physical illness and injury, for how many days during the past 30 days was your physical health not good?’ (physical distress); ‘Now thinking about your mental health, which includes stress, depression and problems with emotions, for how many days during the past 30 days was your mental health not good?’ (mental distress); ‘During the past 30 days, for about how many days did poor physical or mental health keep you from doing your usual activities, such as self-care, work or recreation?’; and ‘During the past 30 days, for about how many days did pain make it hard for you to do your usual activities such as self-care, work or recreation?’. ‘Yes’ or ‘no’ responses to activity limitation, asked as ‘Are you limited in any way in any activities because of physical, mental or emotional problems?’, were examined. Frequent mental distress was dichotomized at ≥14 days of poor mental health.¹⁹ Frequent physical distress, frequent poor physical or mental health, and frequent pain were also calculated.²⁰

Several sociodemographic factors characterized the sample, including marital status, race, employment, age, education, income and gender. Marital status was categorized as married compared with those divorced, widowed, separated, never married or a member of an unmarried couple. Black and other races were combined due to small cell sizes. Employment status was categorized as employed or self-employed compared with out of work, homemaker, student, retired or unable to work.

Statistical analysis

Comparisons of descriptive characteristics across cancer history used Chi-squared or Fisher’s exact tests. The authors examined reliability for the HRQoL questions overall and stratified by cancer history. As the vast majority of cancers occur among persons aged ≥55 years²¹ and to reduce the difference in age between persons with and without cancer, reliability across cancer history was also examined among respondents aged ≥55 years. Cohen’s simple kappa statistic was used for dichotomous variables, weighted kappa was used for ordinal variables, and ICC from a one-way random effects model²² was used for continuous variables. The following rule of thumb was used to evaluate reliability: values >0.75 generally reflect excellent agreement, values <0.4 reflect poor agreement, and intermediate values reflect moderate agreement.^23,24 Ninety-five percent confidence intervals (CI) accompany kappa statistics and ICCs. Significance tests were used to further examine statistically significant differences in reliability among persons who reported a cancer history compared with those who did not.

Only prevalence estimates from the entire Missouri BRFSS sample are weighted to the Missouri population. Retest data are not treated as a probability sample and therefore the BRFSS’ complex sampling design is not considered.^25,26 Proportions are calculated for retest participants; therefore, percentages may not sum to 100 due to rounding or missing values. For demographic information alone, if data were missing from the initial interview, the retest data were used. For cancer history, responses at the initial interview were used for stratified analyses. All ‘don’t know/not sure’ and ‘refused’ responses were set to missing. Data were analysed using SAS/Stat Version 9.1 (Cary, NC, USA).²⁷

Results

Out of 811 eligible subjects, 540 completed the retest survey (67% response rate). Compared with non-respondents, respondents were more likely to be White, aged ≥55 years and not employed (P<0.05). No differences were found for Hispanic ethnicity, education, marital status, gender or cancer history (results not shown). Table 1 describes the overall 2005 Missouri BRFSS interviews, as well as the respondents invited to complete and who completed the retest. Almost half of the retest respondents were aged ≥55 years, 10% reported Black or other race, over half (55%) were married, nearly 57% had at least some college education, half were employed or self-employed, the majority were female (64%), half reported an annual household income <$35,000 (49%), and 13% had been told by a health professional that they had cancer. The median number of days between interviews was 12 (range 7–35 days).

Table 1.

Descriptive characteristics^a of Missouri 2005 Behavioral Risk Factor Surveillance System (BRFSS) and retest participants, overall and by cancer history.

	BRFSS participants, 2005		Invited for retest, April/May 2005		Retest participants, April/May 2005
					Overall		Cancer		No cancer		P- value
	n=5164	%b	n=811	%c	n=540	%c	n=69	%c	n=470	%c
Have you ever been told by a doctor, nurse or other health professional that you had cancer?											–
Yes	536	7.7	97	12.0	69	12.8	–	–	–	–
No	4343	92.3	713	87.9	470	87.0	–	–	–	–
Cancer type
Breast	110	18.9	25	3.1	20	3.7	–	–	–	–
Skin	81	14.7	14	1.7	11	2.0	–	–	–	–	–
Prostate	48	10.1	9	1.1	6	1.1	–	–	–	–
Other	287	56.4	47	5.8	30	5.6	–	–	–	–
Age group (years)
18–54	2788	68.2	453	55.9	281	52.0	17	24.6	263	56.0	<0.05
≥55	2349	31.9	358	44.1	259	48.0	52	75.4	207	44.0
Race
Black/other	590	12.8	99	12.2	54	10.0	6	8.7	48	10.3	0.83
White	4548	87.2	708	87.3	484	89.6	63	91.3	420	89.7
Marital status
Married	2860	61.9	439	54.1	295	54.6	34	49.3	261	55.5	0.37
Divorced/widowed/separated/never married/unmarried couple	2299	38.1	371	45.7	245	45.4	35	50.7	209	44.5
Education
Less than high school graduate	658	11.4	95	11.7	56	10.4	2	2.9	54	11.5
High school graduate	1924	33.7	275	33.9	177	32.8	22	31.9	154	32.8	0.09
Some college	1298	27.9	226	27.9	158	29.3	20	29.0	138	29.4
College graduate	1279	27.1	215	26.5	149	27.6	25	36.2	124	26.4
Employment
Employed or self-employed	2739	61.1	435	53.6	272	50.4	14	20.3	257	54.7	<0.05
Out of work/homemaker/student/retired/unable to work	2419	38.9	375	46.2	268	49.6	55	79.7	213	45.3
Gender
Female	3160	52.0	516	63.6	345	63.9	48	69.6	296	63.0	0.35
Male	2004	48.0	295	36.4	195	36.1	21	30.4	174	37.0
Annual household income ($)
<15,000	705	10.9	111	13.7	80	14.8	12	19.1	68	15.6
15,000–<25,000	895	16.1	158	19.5	104	19.3	12	19.1	91	20.9	0.88
25,000–<35,000	695	15.4	116	14.3	81	15.0	9	14.3	72	16.5
≥35,000	2207	57.7	347	42.8	235	43.5	30	47.6	205	47.0

^aBased on original interview.

^bWeighted.

^cNumbers may not sum to 100 due to rounding or missing values.

Table 1 also compares respondents in the retest interview across cancer history. Those with a cancer diagnosis were significantly more likely to be aged ≥55 years (P<0.05) and not employed (P<0.05). Cancer history was not significantly different across other demographics.

Table 2 describes the HRQoL indicators at the initial interview and retest. Summarizing both the initial and retest interviews, the majority of the sample (75.2–79.4%) reported that their general health was good or better. Out of the last 30 days, respondents at the initial interview and retest reported respective means of 5.2 and 4.9 days of physical distress, 4.0 and 3.1 days of mental distress, 6.3 and 5.5 days of poor physical or mental health prohibiting usual activities, and 5.5 and 5.2 days of pain making it difficult to perform usual activities. Approximately 29% of the sample reported activity limitation. Respondents reported that they ‘sometimes’, ‘rarely’ or ‘never’ received the social and emotional support needed 22.6 and 18.9% of the time (respective to the initial and retest interviews). Approximately 8% of the sample was generally dissatisfied or very dissatisfied with their life. Frequent mental distress was less commonly reported than frequent physical distress or frequent pain.

Table 2.

Test–retest reliability of health-related quality-of-life questions from the Missouri Behavioral Risk Factor Surveillance System retest sample, overall and by cancer history (n=540), 2005.

	Overall					≥55 years of age
	Initial interview, n (%)a	Retest interview, n (%)a	Overall, reliability (95% CI)	Cancer, reliability (95% CI)	No cancer, reliability (95% CI)	Cancer, reliability (95% CI)	No cancer, reliability (95% CI)
General health
Excellent	76 (14.1)	59 (10.9)
Very good	177 (32.8)	169 (31.3)
Good	153 (28.3)	201 (37.2)
Fair	87 (16.1)	74 (13.7)
Poor	46 (8.5)	37 (6.9)	0.65 (0.60–0.69)	0.67 (0.56–0.79)	0.63 (0.58–0.69)	0.63 (0.49–0.76)	0.63 (0.56–0.70)
Physical distressb	536	540
n, median, mean (SD)	0.0, 5.2 (9.5)	0.0, 4.9 (8.9)	0.72 (0.68–0.76)	0.79 (0.69–0.87)	0.69 (0.64–0.73)	0.79 (0.65–0.87)	0.66 (0.58–0.73)
Mental distressb	534	540
n, median, mean (SD)	0.0, 4.0 (8.1)	0.0, 3.1 (7.0)	0.71 (0.66–0.75)	0.74 (0.61–0.83)	0.70 (0.65–0.75)	0.51 (0.28–0.69)	0.59 (0.50–0.68)
Poor physical or mental healthb ,c	298	287
n, median, mean (SD)	0.0, 6.3 (9.9)	0.0, 5.5 (9.4)	0.63 (0.55–0.70)	0.55 (0.27–0.74)	0.64 (0.55–0.72)	0.50 (0.13–0.75)	0.52 (0.33–0.67)
Activity limitation
Yes	160 (29.6)	154 (28.5)
No	380 (70.4)	382 (70.7)	0.75 (0.69–0.81)	0.80 (0.65–0.94)	0.73 (0.66–0.80)	0.77 (0.59–0.94)	0.64 (0.53–0.75)
Social and emotional support
Always	243 (45.0)	235 (43.5)
Usually	170 (31.5)	190 (35.2)
Sometimes	65 (12.0)	60 (11.1)
Rarely	42 (7.8)	30 (5.6)
Never	15 (2.8)	12 (2.2)	0.57 (0.51–0.62)	0.53 (0.36–0.71)	0.57 (0.51–0.64)	0.52 (0.32–0.71)	0.46 (0.36–0.57)
Life satisfaction
Very satisfied	202 (37.4)	207 (38.3)
Satisfied	290 (53.7)	271 (50.2)
Dissatisfied	34 (6.3)	40 (7.4)
Very dissatisfied	8 (1.5)	6 (1.1)	0.61 (0.55–0.67)	0.46 (0.28–0.64)	0.64 (0.57–0.70)	0.46 (0.26–0.66)	0.59 (0.49–0.69)
Painb	533	530
n, median, mean (SD)	0.0, 5.5 (10.4)	0.0, 5.2 (9.9)	0.75 (0.71–0.78)	0.59 (0.41–0.73)d	0.78 (0.74–0.81)d	0.51 (0.28–0.69)d	0.73 (0.65–0.79)d
Frequent physical distress
Yes	91 (16.9)	90 (16.7)
No	445 (82.4)	450 (83.3)	0.68 (0.60–0.76)	0.67 (0.48–0.86)	0.67 (0.57–0.77)	0.66 (0.44–0.87)	0.64 (0.50–0.78)
Frequent mental distress
Yes	70 (13.0)	50 (9.3)
No	464 (85.9)	490 (90.7)	0.61 (0.51–0.72)	0.62 (0.34–0.90)	0.61 (0.50–0.73)	0.46 (0.02–0.90)	0.44 (0.24–0.65)
Frequent poor physical or mental healthc
Yes	69 (23.2)	52 (18.1)
No	229 (76.8)	235 (81.9)	0.64 (0.52–0.75)	0.50 (0.24–0.76)	0.66 (0.53–0.79)	0.43 (0.11–0.76)	0.51 (0.29–0.73)
Frequent pain
Yes	97 (18.0)	88 (16.3)
No	436 (80.7)	442 (81.9)	0.69 (0.61–0.78)	0.51 (0.29–0.72)	0.73 (0.64–0.82)	0.43 (0.19–0.68)	0.64 (0.50–0.78)

CI, confidence interval; SD, standard deviation.

^aUnless otherwise noted.

^bRepresents intraclass correlation coefficient; all other reliability results are Cohen’s kappa.

^cOnly asked of those reporting at least 1 day of physical or mental distress.

^dComparisons across cancer history, P<0.05

A personal cancer history demonstrated excellent reliability, although it was not perfect (κ=0.91, 95% CI 0.86–0.96). Overall results for activity limitation and pain suggest excellent reliability, while the other indicators demonstrated moderate reliability (Table 2). None of the measures indicated poor reliability. Across cancer history, reliability estimates did not differ statistically except for the pain measure. Results suggest significantly higher reliability for pain across cancer history, both overall (cancer history: ICC=0.59, 95% CI 0.41–0.73; no cancer history: ICC=0.78, 95% CI 0.74–0.81) and when restricting the sample to those aged ≥55 years (cancer history: ICC=0.51, 95% CI 0.28–0.69; no cancer history: ICC=0.73, 95% CI 0.65–0.79). A significant difference was not found for frequent pain.

Discussion

This is the first population-based study to compare reliability estimates of HRQoL indicators between cancer survivors and the general population using the BRFSS. Overall, reliability was moderate to excellent. The overall findings are consistent with those of another study examining HRQoL from the Missouri BRFSS.¹² Additionally, the present finding that cancer diagnosis demonstrated excellent reliability (κ=0.91) is consistent with findings demonstrated for similarly worded BRFSS questions, such as hypertension²⁸ and diabetes²⁸ (κ=0.82 and 0.86, respectively). This study further adds that reliability for HRQoL measures across cancer history was generally moderate to excellent in this population, both overall and among respondents aged ≥55 years.

Using the BRFSS for cancer surveillance, while cross-sectional, may provide additional information for the National Cancer Institute’s Cancer Surveillance Research Implementation Plan.^2,5 Such population-based monitoring could, for example, take the form of capturing ongoing, up-to-date HRQoL information on a large scale.² Changes in health status over time, and variation by geography or population subgroups, including racial and ethnic disparities, could help to identify targeted areas for further investigation or intervention.²

This study has several limitations to consider. First, as these retest results using Missouri BRFSS data were not treated as a probability sample, statistical inferences are not made beyond this study. However, these results do provide an important look at how the HRQoL measures perform across cancer history in this population. Second, the sample size for those reporting a cancer history was modest in the retest sample. Confidence intervals were reported to assist with interpreting the precision of the estimates, especially for those with cell sizes <50. It would be important to confirm these findings in a larger sample and further examine the reliability of these HRQoL indicators based on cancer type, cancer stage and length of time since diagnosis. Third, the time interval between the BRFSS interview and retest ranged from 7 to 35 days. While this interval is consistent with other studies,^12,25,28 longer intervals may be associated with greater variation and therefore lower reliability. These data cannot explain whether the lower reliability for the pain measure among those diagnosed with cancer reflects a change in the respondent’s treatment; however, the wording of questions regarding pain in the last 30 days may minimize any impact of the time interval. Despite these limitations, this study has several strengths including a strong response rate, consistency of results, trained interviewers who regularly call for the BRFSS, and some information to address a gap in the literature.

In summary, this study provides an initial look at estimates of the test–retest reliability of HRQoL questions from the BRFSS across cancer history. Measurement issues surrounding cancer survivorship are challenging and need to consider late and long-term medical effects as well as comorbid conditions.⁴ These data suggest that it is possible to monitor HRQoL in cancer survivors with similar reliability as in the general population using BRFSS data, with the exception of the pain measure. Replication studies are needed in other states to inform consideration of the BRFSS for population surveillance of HRQoL in cancer survivors.