Validating the PROMIS Cognitive Function Short Form in Cancer Survivors

Abstract

Purpose:

The Patient-Reported Outcome Measurement Information System Cognitive Function Short Form 8A (PROMIS Cog) could provide a shorter, useful alternative to the often used Functional Assessment of Cancer Therapy – Cognition (FACT-Cog) in research. This study aimed to determine the convergent validity and internal reliability of the PROMIS Cog in 3 separate samples of breast cancer survivors and to explore clinical cut points.

Methods:

Data from three samples of breast cancer survivors (BCS) were used for this secondary analysis. Convergent validity was determined by evaluating correlation strength among the derived PROMIS Cog and measures of depression, anxiety, stress, fatigue, sleep, loneliness, the FACT Cognitive Function. Clinical cut-points for the PROMIS Cog were determined by plotting the receiver operating characteristic curves.

Results:

3 samples of breast cancer survivors (N=471, N=132, N=90) were included. Absolute values of correlations demonstrating convergent validity ranged from 0.21 to 0.82, p’s <.001, and were comparable to correlations with the full FACT Cognitive Function 18 item perceived cognitive impairments scale. ROC curve plots indicated a clinical cut off <34 for the combined sample.

Conclusion:

The 8-item PROMIS Cog demonstrated good convergent validity and internal reliability in breast cancer survivors, comparable to the 18-item FACT Cog PCI. The PROMIS Cog 8a is a brief self-report measure that can be easily incorporated into CRCI research designs or used in clinical settings.

Introduction

Cancer-related cognitive impairment (CRCI) in memory, attention, and other abilities is distressing and burdensome, and can negatively impact cancer survivors’ quality of life, social functioning and occupational functioning [1, 2]. It is generally accepted that approximately 30% of newly diagnosed cancer patients, 75% of patients undergoing adjuvant therapies, and 35% of cancer survivors experience CRCI [3].

Patient reported outcomes (PROs) assessing cognition are being recognized for their contribution in evaluation of CRCI [4], without a consensus on choice of PRO instrument. Our working group, The Cancer Neuroscience Initiative, recently suggested the use of the Patient-Reported Outcome Measurement Information System (PROMIS) Cognitive Function Short Form 8A (PROMIS Cog) as a minimum measure in all CRCI research including cognitive PROs [5], in order to facilitate cross study comparisons and meta analyses.

Presently, many researchers use the 18-item Functional Assessment of Cancer Treatment-Cognitive Function Perceived Cognitive Impairment (PCI) scale. The brief 8-item PROMIS Cog includes most of the same items as the longer PCI, since they were developed from similar item banks and common developers[6]. The PROMIS Cog may therefore function as a useful, briefer assessment tool for CRCI, but there is currently limited psychometric evidence for this instrument in oncology populations. In this study, we determined the convergent validity and internal reliability of the PROMIS Cognitive Function Short Form 8a in 3 samples of breast cancer survivors and explored clinical cut points.

Methods

Data from three samples of breast cancer survivors were used for this secondary post-hoc analysis: 1) the Thinking and Living with Cancer (“TLC”) study [7], 2) the Mind Body Study (“MBS”) [8]; and 3) the Cognition After Breast Cancer (“CABC”) study [9]. All study procedures and protocols were in accordance with the Declaration of Helsinki and were approved by local Institutional Review Boards. All participants provided written informed consent prior to data collection. Data for perceived cognitive functioning (measured with the FACT Cog) and other measures were included in these analyses. Since subjective and objective measures of CRCI are commonly unrelated [4], we focused on other cognitive PRO’s and measures for mood, distress, and sleep to measure convergent validity as done in other studies since they are contemporaneously conceptualized as symptom clusters [10].

Thinking and Living with Cancer (“TLC”) Study

The multi-site prospective TLC study cohort has been previously described in detail [7]. In brief, study criteria included females with a new diagnosis of stage 0-III breast cancer age 60 or older. We selected the 12-month assessment timepoint for analyses (following adjuvant treatment and/or initiation of hormone therapy). Demographic and clinical information were obtained via medical record. Other measures included the Pittsburgh Sleep Quality Index (PSQI), Functional Assessment of Chronic Illness Therapy – Fatigue (FACT-F), State Trait Anxiety Inventory – State (STAI-S), and the Center for Epidemiologic Studies Depression Scale (CESD).

Mind Body Study (“MBS”)

The MBS study was a prospective longitudinal study examining the effects of endocrine therapy for treating breast cancer in females, and has been extensively described elsewhere [8]. Briefly, the study included participants aged 21-65 years, who completed primary cancer treatment in the last three months for breast cancer (stage 0-III). Data from the final follow-up assessment, 3-6 years after study enrollment, was used in this study. Demographic and clinical information was obtained via self-report, and self-report measures included the FACT-Cog, Beck Depression Inventory-2^nd edition (BDI-2), Multidimensional Fatigue Symptom Inventory (MFSI), the PSQI, and STAI-S.

Cognition After Breast Cancer (“CABC”) Study

All CABC study procedures have also been previously described [9]. Females between the aged 21 to 65 years with a history of breast cancer (stage 0-III) treated with chemotherapy within the past 10 years were enrolled. Data included demographic and clinical characteristics, the FACT-Cog, the PROMIS version 1.0 Emotional Distress-Anxiety short form 8a (PROMIS Anxiety) and Depression- short form 8a (PROMIS Depressive), Fatigue-short form 8a (PROMIS Fatigue), the Perceived Stress Scale (PSS), the PSQI, and the UCLA Loneliness Scale revised version 3 (UCLA Loneliness).

PROMIS Cog Scores

For this analysis, the PROMIS Cog scores were derived from the FACT-Cog, which includes seven of the same items as the PROMIS Cog (CogA1; CogA3, CogC7, CogF23, CogC31, CogC32, CogMT2). See Table 1. We recoded the seven items to be congruent with the PROMIS Cog scale and have meaningful interpretation of the derived clinical cut points. Since 1 item of the PROMIS Cog is not included in the FACT-Cog, we created one additional item using an average score across the 7 FACT Cog items to re-create the 8-item total score for the PROMS Cog. The 8 items were summed, and total raw scores used in the analyses. This method of mean substitution was used based on the instructions for scoring FACT instruments[11] and is common practice to handle missing data in clinical measures[12]. If participants were missing more than two of the seven items, they were excluded from analyses.

Table 1.

Comparison of PROMIS Short Form 8a to FACT-Cog

PROMIS Short Form 8a Items	Approximate FACT-Cog Items
1. I have had trouble forming thoughts.	CogA1
2. My thinking has been slow.	CogA3
3. I have had trouble concentrating.	CogC7
4. I have had to work really hard to pay attention or I would make a mistake.	CogF23
5. I have had to work harder than usual to keep track of what I was doing.	CogC31
6. It has seemed like my brain was not working as well as usual (maps to “My thinking has been slower than usual.” From the FACT-Cog PCI, however not exact match).	CogC32
7. I have had trouble shifting back and forth between different activities that require thinking.	CogMT2
8. I have had trouble adding or subtracting numbers in my head.	No PCI item, mean substitution used to create item

Data Analyses

Frequencies, means, and standard deviations were used to describe basic characteristics for the three samples. Comparative convergent validity was determined by evaluating correlations size (using a reference of 0.1 to 0.3 as small/weak correlation, 0.4 to 0.6 as a moderate correlation, and .7 or higher as a strong correlation)[13] between the derived PROMIS Cog and FACT-Cog PCI with measures others have used to demonstrate the validity of the FACT-Cog PCI [10] including depression, anxiety, stress, fatigue, sleep quality, loneliness and the FACT-Cog Cognitive Abilities, Quality of Life, and Cognitive Comments from Others subscales. Internal reliability was determined using the Cronbach alpha for total scales and with each item deleted. A threshold of 0.8 or greater was used for Cronbach alpha [14]. In order to identify comparable clinical cut-points for the PROMIS Cog, we plotted the receiver operating characteristic (ROC) curves of the PROMIS Cog using the published [15] and recently validated [16] FACT-Cog PCI 18 cut-point score of <54.

Results

The TLC sample included 471 breast cancer survivors (average 1.3 years from diagnosis), the MBS sample included 132 breast cancer survivors (average 5 years from diagnosis), and the CABC study included 90 breast cancer survivors (on average 3.5 years since diagnosis) See Table 2.

Table 2.

Demographic and Clinical Characteristics of the Samples (All Female Participants)

	Thinking and Living with Cancer Study (N=471)	Mind Body Study (N=132)	Cognition After Breast Cancer Study (N=90)
Age in years Mean (SD) Range	69.6 (5.7) 60-92	56.4 (8.0) 36-69	48.7 (9.0) 24-65
Years of Education	<16: 199 (42%) 1 16: 112 (24%) >16: 159 (34%)	<16 26 (20%) 16 42 (32%) >16 63 (48%)	<16 18 (20%) 16 34 (38%) >16 35 (39%)
Identified as Minority/non-white	83 (18%)	27 (21%)	15 (17%)
Employed At Time of Study	143 (30%)1	85 (64%)	78 (87%)
Months Since Diagnosis Mean (SD) Range	15.7 (2.67) 6.0-33.0	58.0 (8.1) 41.8-82.6	42.0 (32.4) 6.8 to 120
Breast Cancer Stage 0 I II III	79 (17%) 277 (60%) 88 (19%) 21 (5%)	19 (15%) 58 (44%) 42 (32%) 12 (9%)	19 (21%) 51 (57%) 19 (21%)
Frequency of Chemotherapy	117 (25%)	71 (54%)	90 (100%)
Frequency of Radiation	287 (61%)	98 (75%)	59 (66%)
Frequency of Hormone Therapy	371 (80%)	96 (73%)	71 (79%)

¹data available for n=470

Abbreviations: SD= standard deviation

The Cronbach’s alphas for the PROMIS Cog ranged from 0.89 (TLC) to 0.91 (MBS) to 0.97 (CABC). Internal consistencies were also calculated for the PROMIS Cog with each item removed and ranged from 0.963 to 0.972 for the CABC study, 0.939 to.849 for the MBS study, and 0.913 to 0.889 for the TLC study (see Supplementary Materials Table 1). The Absolute values for the correlations among PROMIS Cog and convergent validity measures ranged from 0.21 to 0.82 (p’f < .001, see Table 3) across studies. ROC curve plots indicated a clinical cut-off <34 for the PROMIS Cog and area under the curves (AUC’s) ranged from 0.96—0.98, indicating outstanding discrimination[17]. Sensitivity ranged from 0.89—0.98, and specificity ranged from 0.79—0.89, see Table 4. ROC curve plots were also calculated for the FACT Cog PCI 20, as some researchers use the 20 item subscale rather than the 18 item subscale (see Supplementary Materials Table 2).

Table 3.

Convergent validity analyses comparing correlations between FACT-Cog Perceived Cognitive Impairments 18 item subscale and PROMIS Cog 8a

	Thinking and Living with Cancer Study (N=471)		Mind Body Study (N=132)		Cognition After Breast Cancer Study (N=90)
Construct	FACT-Cog PCI- 18	PROMIS Cog 8a	FACT-Cog PCI-18	PROMIS Cog 8a	FACT-Cog PCI-18	PROMIS Cog 8a
Depression r	−0.338**	−0.378**	−.646**	−.643**	−.55**	−.56**
(specific measure)	(CES-D)		(BDI-II)		(PROMIS-Emotional Distress- Depressive Short Form 8a)
Anxiety r	−0.260**	−0.290**	−.455**	−.467**	−.52**	−.51**
(specific measure)	(STAI State-Anxiety subscale)		(STAI State-Anxiety subscale)		(PROMIS-Emotional Distress- Anxiety Short Form 8a)
Stress r	-	-	−.534**	−.516**	−.66**	−.67**
(specific measure)	-		(PSS)		(PSS)
Fatigue r	0.342**	0.366**	.542**	.559**	−.62**	−.62**
(specific measure)	(FACIT-F)		(MFSI-Vigor subscale)		(PROMIS-Fatigue Short Form 8a)
Sleep r	−0.205**	−0.215**	−.322**	−.315**	−.51**	−.48**
(specific measure)	(PSQI)		(PSQI)		(PSQI)
Loneliness	-	-	-	-	−.51**	−.51**
(specific measure)	-		-		(UCLA Loneliness v 3)
Cognitive Abilities	0.619**	0.596**	.746**	.690**	.82**	.82**
(specific measure)	(FACT-Cog PCA subscale)		(FACT-Cog PCA subscale)		(FACT-Cog PCA subscale)
Quality of Life	0.554**	0.569**	.731**	.749**	.78**	.77**
(specific measure)	(FACT-Cog QOL subscale)		(FACT-Cog QOL subscale)		(FACT-Cog QOL subscale)
Comments from Others about Cognition	0.340**	0.362**	.617**	.649**	.62**	.63**
(specific measure)	(FACT-Cog CO subscale)		(FACT-Cog CO subscale)		(FACT-Cog CO subscale)
Cognitive Complaints: Squire Memory Questionnaire			.659**	.562**
Cognitive Complaints: Patient’s Assessment of Own Functioning Inventory Total			.710**	.665**

Abbreviations: BDI-II: Beck Depression Inventory-II ; CES-D: Center for Epidemiologic Studies Depression Scale; CO: Comments from Others; FACIT-F: Functional Assessment of Chronic Illness Therapy Fatigue; FACT Cog: Functional Assessment of Cancer Therapy - Cognitive Function; MFSI: Multidimensional Fatigue Symptom Inventory; PCA: Perceived Cognitive Abilities; PCI: Perceived Cognitive Impairments; PROMIS: Patient-Reported Outcomes Measurement Information System; PSS: Perceived Stress Scale; PSQI: Pittsburgh Sleep Quality Index; STAI: State-Trait Anxiety Inventory; QOL: Quality of Life

^**p < .001

Table 4.

Descriptive statistics, area under the curve for the receiver operating characteristics for the PROMIS Cog based on the FACT Cog PCI 18 (<54) clinical cut offs for 3 samples

	Thinking and Living with Cancer Study (N=471)	Mind Body Study (N=132)	Cognition After Breast Cancer Study (N=90)	Combined Samples
Raw total mean (SD), range	34.94 (5.77) 13.33-40	34.03 (6.13) 16-40	26.85 (9.96) 8-40	33.72 (7.04) 8-40
T-score mean (SD), range ^	52.85 (8.36) 31.18-63.48	51.14 (8.03) 33.92-63.48	43.82 (11.10) 22.41-63.48	51.35 (9.18) 22.41-63.48
PCI-18 Cut Off	<34	<34	<34	<34
PCI-18 AUC	0.977	0.966	0.972	0.963
PCI-18 Sensitivity	0.928	0.921	0.982	.925
PCI-18 Specificity	0.892	0.844	0.794	.851

Abbreviations: AUC: area under the curve; CABC: Cognition After Breast Cancer; MBS: Mind Body Study; PCI: Perceived Cognitive Impairments; ROC: receiver operating characteristic; TLC: Thinking and Living with Cancer

^{^}Truncated raw scores converted to T scores using PROMIS conversion table for the PROMIS Cog 8a

Discussion

This study provides evidence for strong internal consistency of the 8-item PROMIS Cog, similar comparative convergent validity as the FACT-Cog PCI with measures of depression, anxiety, stress, fatigue, sleep, loneliness, and a preliminary clinical cut point (<34).

Correlations between PROMIS Cog and other cognitive PRO’s as well as measures of depression, anxiety, fatigue, stress, and sleep quality were consistent with previous reports in cancer survivors [4] and consistent with recent validity analyses for the FACT-Cog [10]. We found stronger correlations among these variables in the CABC and MBS samples than the TLC study which may be attributed differences in age (CABC and MBS were younger), differences in employment status (more were employed in the CABC and MBS samples), and/or differences in treatment histories—more than half of the samples in CABC and MBS were treated with chemotherapy compared to a quarter of the TLC study. Chemotherapy history, and employment status have been linked to poorer self-reported CRCI [4]. Younger age and employment status are also risk factors for greater psychosocial distress, and fatigue in BCS [18-20].

AUCs, sensitivity and specificity scores indicated using a cut-off of 34 for the PROMIS Cog. We used another cognitive PRO, the FACT-Cog PCI, to determine the clinical cut-off for the PROMIS Cog, which is common practice[15, 21, 22], however cut-points may be different if standardized cognitive tests or other functional measures are used, and should be further studied for clinical utility.

Study limitations should be considered. The PROMIS Cog scores were derived from seven overlapping items of the FACT-Cog PCI subscale and 1 mean substitution item, not the PROMIS Cog short form 8a itself, resulting in raw summed scores being used in the analyses. Results should be replicated with the full instrument and converted to T scores with standard errors (as recommended by the PROMIS scoring manual). The three samples used in these analyses only included breast cancer survivors after treatment, who were highly educated, limiting generalizability. Replication and validation of the published PROMIS Cog 8a are needed in other socio-demographically diverse cancer populations, including earlier and/or later in the cancer treatment trajectory age groups, and primary cancer diagnoses.

Conclusion

These results provide preliminary psychometric evidence for using the PROMIS Cog and support recommendations to use the PROMIS Cog in CRCI research [5]. The PROMIS Cog 8a is a brief self-report measure that can be easily incorporated into CRCI research and clinical settings. These findings also offer a starting point for additional psychometric evaluation (e.g., sensitivity to change over time, test-retest reliability) and for determining classification methods for cognitive PROs in CRCI research. Using the short PROMIS Cog should enhance the ease of routine collection of cognitive data in clinical trials, geriatric assessments, and observational studies of broad populations of cancer survivors.

List of Abbreviations

AUC

area under the curves

BDI-2

Beck Depression Inventory-2^nd edition

CABC

Cognition After Breast Cancer

CESD

Center for Epidemiologic Studies Depression Scale

CRCI

Cancer-related cognitive impairment

FACT-Cog

Functional Assessment of Cancer Therapy – Cognition

FACT-F

Functional Assessment of Chronic Illness Therapy – Fatigue

MFSI

Multidimensional Fatigue Symptom Inventory

PCI

Perceived Cognitive Impairment

PSQI

Pittsburgh Sleep Quality Index

PROs

Patient reported outcomes

PROMIS Anxiety

PROMIS version 1.0 Emotional Distress- Anxiety short form 8a

PROMIS Cog

Patient-Reported Outcome Measurement Information System Cognitive Function Short Form 8a

PROMIS Depressive

PROMIS version 1.0 Emotional Distress- Depressive short form 8a

PROMIS Fatigue

PROMIS version 1.0 Fatigue short form 8a

ROC

receiver operating characteristic

STAI-S

State Trait Anxiety Inventory – State

TLC

Thinking and Living with Cancer

Data Availability:

The datasets analyzed in the current study are available from the corresponding author on reasonable request.