Using consideRATE to Evaluate Patient Experience in a Cancer Center: Psychometric and Healthcare Assessments

Abstract

Background:

consideRATE is a patient- and care-partner-reported measure of care experience during serious illness. We used consideRATE with patients and care partners at the Dartmouth Cancer Center to assess patient experience, evaluate psychometric properties, and explore scoring approaches.

Methods:

Patients and care partners who are aged 18+ and English proficient participated in a cross-sectional survey. Participants completed consideRATE (8 items), CANHELP Lite (21 items), and demographic questions. Reliability was assessed using Cronbach’s α, and validity was evaluated with Pearson’s correlations. Continuous and top-box scoring approaches were used. Psychometric properties were analyzed for patients, care partners, and subgroups with lower educational attainment or income.

Results:

244 participants (114 patients, 128 care-partners, 2 unspecified) completed the survey. consideRATE has internal reliability (α = 0.86); the correlation (r) between consideRATE continuous scoring and CANHELP Lite scores for all participants was 0.5; p < 0.001, for patients 0.5; p < 0.001, for care-partners 0.5; p < 0.001, for patients (n = 71) with lower educational attainment 0.5, p < 0.001, and for patients (n = 50) with lower income 0.7, p < 0.001. We also found correlations between consideRATE top-box scoring and CANHELP Lite scores for all participants (rpb = 0.4, p < 0.001), with stronger associations among the patient (rpb = 0.5, p < 0.001) and lower income (rpb = 0.6, p = 0.005) subgroups. We found discriminant validity between consideRATE and Single-item Health Literacy (SIL) measures for continuous scoring (r = −0.05 to 0.09, p > 0.05) and top-box scoring (r = −0.02 to 0.09, p > 0.05). We found no significant difference in overall experience between patients with solid and hematologic malignancy cancer categories.

Conclusion:

We demonstrated in this sample of patients attending a cancer center that consideRATE has good internal reliability and is well correlated with CANHELP Lite.

1. Background

Patients with cancer have a constellation of needs that are often insufficiently addressed [1, 2]. Measurement of patient-reported outcomes in oncology is a valuable tool for quality assessment and improvement that can be deployed at both the individual and population levels [3]. The National Comprehensive Cancer Network (NCCN) Quality and Outcomes Committee developed a list of priority concepts, quality metrics, and measurement strategies for assessing cancer care delivery by recognizing the importance of measurement and quality benchmarking in cancer care [4]. They noted the importance and gap of high-quality “end-of-life” measures and a “significant gap” in measuring what matters most to patients, including patient experiences [4]. The council also noted persistent issues in implementing and documenting such measures in routine care [3].

The consideRATE questions, a patient-reported measure of serious illness care experience, may address these measurement needs and provide an opportunity to improve patients’ and care partners’ experiences [5]. This unique measure was created for individuals facing serious illnesses, including those nearing the end of life. It focuses on addressing elements of care that have been identified as meaningful to patients with serious illnesses [5]. Unlike other measures [6], the completion time for consideRATE is short, average ranging from an average of 3 min to slightly over 6 min, and this measure complements other self-reported quality measures, such as The “Heard & Understood” item [5, 7, 8]. The consideRATE questions are flexible and can be used across settings and diagnoses [5, 9, 10]. Also, the consideRATE questions may enhance routine care measurement for cancer patients, provide valuable feedback to care teams, and serve as an outcome measure in research and quality improvement [5, 7].

Preliminary psychometric work on the consideRATE questions proved promising, but the tool had not yet been validated in real-world clinical settings or among patients with cancer and their care partners [7]. For example, we validated the consideRATE questions with older adults (≥ 50 years old) in an online randomized survey using simulated patient experiences [7]. In this controlled environment, the consideRATE questions were reliable and valid. However, the real-world psychometric properties of the consideRATE questions were not yet known. Importantly, validation with care partners had not yet occurred. Without real-world validation, it remained uncertain whether the consideRATE questions accurately capture the serious illness care experience for cancer patients and their care partners.

To address these gaps, we aimed to describe psychometric properties of the consideRATE questions when administered to patients and their care partners at a National Cancer Institute (NCI) designated comprehensive cancer center in rural New England.

2. Methods

2.1. Design

We conducted a cross-sectional survey study with convenience sampling. We obtained informed consent and provided surveys on iPads or on paper in-person. We reported results using the Checklist for Reporting of Survey Studies (CROSS) (Appendix 1) [11]. The Dartmouth Health Institutional Review Board (IRB) approved this study in June 2021 (DH IRB STUDY02000560).

2.2. Participants

We recruited patients with cancer and care partners (family members, caregivers, or friends who accompanied patients) in a National Cancer Institute (NCI) Designated Comprehensive Cancer Center, The Dartmouth Cancer Center (DCC), in Lebanon, New Hampshire [12]. NCI-Designated Cancer Centers have high survival rates and healthcare quality ratings. This makes them well-suited for validation to minimize confounding variables (e.g., providing suboptimal care delivery, lack of medical resources), and maximize clarity in the first real-world assessment of the psychometric properties of the consideRATE questions. Validation in more variable quality settings is an important area for future research [13, 14]. Also, DCC was selected as the study site because it is our host institution and provided a feasible setting to test and validate the measure. As outlined in our study protocol, we determined the sample size in consultation with a statistician based on an alpha level of 0.05 and a power of 99% to detect a 0.2-point change on the consideRATE questions. We recruited participants who checked in at the main DCC reception area, where most but not all cancer patients check-in. While there were instances in which both patients and care partners agreed to participate in the study and complete the survey on two different iPads, there were also a few instances when patients or care partners declined to participate (e.g., patients said “want to rest before their appointment”, “not feeling well to complete it today”, “maybe next time”) but their care partners or the patient were eager to participate and provide feedback. In these cases, only the care partner or the patient completed the survey.

To be eligible, participants needed to be a patient or care partner of a patient at the DCC; able to consent and read English; and 18 years of age or older. Parents or guardians of children with cancer were eligible to participate as care partners. We excluded those who could not provide informed consent, those under the age of 18, and persons who were incarcerated.

2.3. Survey Validation

We created an online survey using the Qualtrics survey platform [15]. We designed the survey and analytic plan based on previously conducted online validation studies [7, 16, 17]. The 10-min survey (Appendix 2) consisted of 43 questions and included basic demographics, a single item assessing health literacy [18], and healthcare experience-related questions, including the consideRATE questions, and the CANHELP Lite Patient questionnaire [5, 6].

2.4. Survey Elements

2.4.1. Participant Characteristics

Participants provided demographics, including age, gender, race or ethnicity, family income, cancer diagnosis, highest level of education, and health literacy.

2.4.2. The ConsideRATE Questions

We previously published the development and online validation of consideRATE [7]. Briefly, consideRATE is an 8-item patient-reported experience measure (PREM), supported by descriptive icons, that evaluates serious illness care experiences. It inquires about key elements of patient experience including attention, communication and respect for physical symptoms, emotional symptoms, environment of care, plan of care, prognosis and personal preferences and values [5, 7]. The consideRATE questions can be administered to people in diverse settings, including outpatient, inpatient, and home care. The first seven items are scored on a four-point Likert-like scale ranging from 1 (very bad) to 4 (very good) with a “doesn’t apply” option. The eighth item is an open-ended question that asks, “Are there any other things you want to share?” We did not include the optional ninth question about desired anonymity as it was not relevant for research purposes.

2.4.3. CANHELP Lite

CANHELP Lite (Canadian Health Care Evaluation Project) is a 21-item patient-facing and validated serious illness experience measure [6]. CANHELP Lite consists of items concerning the management of physical and emotional symptoms and the environment of care. All items are scored on a five-point Likert-like scale ranging from 1 (not at all) to 5 (completely) with a “doesn’t apply” option. We selected CANHELP Lite to validate consideRATE because it is a widely used and psychometrically validated measure designed for evaluating the subjective care experience in serious illness contexts. Other measures like the Consumer Assessment of Healthcare Providers and Systems (CAHPS) and the Supplemental Items for the CAHPS Cancer Care Survey largely assess whether customer-service type activities occurred, not the patient-reported interpretation of quality. This makes CANHELP Lite a superior comparison.

2.5. Procedures

We collected data between May 2022 and May 2023. Patients and care partners were approached by research assistants in the waiting room before their scheduled visits, and invited to participate in the study, reflecting on their experiences with serious illness care received at the cancer center “in the last few weeks.” They were instructed to base their responses on any previously-received care visits at the hospital and the clinic, rather than their upcoming visit. After informed consent and before their visit, participants completed surveys (iPad, Qualtrics QR code, paper).

2.6. Statistical Analyses

Prior to analysis, we removed anyone who did not progress past the first phase of the survey, which included demographics (n = 14). Participants did not need to complete all questions on consideRATE and CANHELP Lite measures to be included in analyses. We used chi-squared analyses to test for mean differences between patients and care partners for selected covariates. In the second analysis phase, we performed psychometric tests of the reliability, discriminant validity, and convergent validity, using a Cronbach’s alpha coefficient for internal consistency and Pearson correlation test, of the consideRATE questions across scoring methods. Given the conceptual differences between consideRATE and CANHELP Lite and the real-world nature of this study, we interpreted convergent validity statistics, recognizing that perfect alignment was unlikely due to their differing characteristics and the variability in real-world data based on population of interest [19]. We defined weak correlation as less than r = 0.3, moderate between r = 0.3 to r = 0.7, and strong correlation as at least r = 0.7 [20]. We used SPSS Statistics Version 28.0 [21].

2.6.1. Overall Instrument Scoring

We tested the convergent validity of the consideRATE questions across two scoring approaches: continuous scoring and top-box scoring [16, 17]. For continuous scoring, we used the original consideRATE four-point and CANHELP five-point scales. We calculated mean consideRATE question scores across all seven items and mean CANHELP Lite scores across all 21 items [6]. We used mean scores to test for convergent and discriminant validity tests (using Pearson’s correlation). For participants who did not score all questions on either measure, the total number of questions used to calculate the mean was adjusted to equal the total number of answered questions.

For top-box scoring, we dichotomized overall consideRATE scores by allocating a “1” to participants who responded “very good” to all measure items they completed, and “0” to participants who did not. We used this top-box approach to reflect the overall perceived experience (e.g., “Was the participant’s overall experience very good?”), an approach with more sensitivity to small variations across items given that consideRATE consists of only 7 items. We again used the continuous CANHELP scoring, because this is the original measure scoring approach that has been validated. We compared the two measures and tested convergent validity using point-biserial correlation tests. Furthermore, given the brevity and simplicity of consideRATE, we have elected to calculate top-box scores at the patient level across the entire measure. This is consistent with the methodology used in many other brief and flexible PREMs, like collaboRATE, a measure of shared-decision making [16].

In addition to continuous and top-box scoring, we initially planned to report on a third method of scoring consideRATE, a high-low dichotomization scoring (overall mean scores below the midpoint of 2.5 were “low” and scores 2.5 or higher were “high”). We did not have sufficient participants in the “low” group to reliably report the results of these analyses.

2.6.2. Item-by-Item Scoring: Convergent Validity

To perform item-by-item convergent validity tests of the consideRATE questions, two independent reviewers (JN and CHS) matched the seven consideRATE questions with seven out of 21 CANHELP Lite questions based on concept. The reviewers resolved discrepancies after review. For continuous scoring, we compared individual items using the consideRATE 4-point scale to matched items on the CANHELP Lite five-point scale, using Pearson’s correlation tests. For top-box scoring, we compared individual items of consideRATE scored using a top-box approach (“very good” vs. all else) to match items on the CANHELP Lite five-point scale, using point-biserial correlation tests.

2.6.3. Discriminant Validity

To test for discriminant validity, we identified a measure that is conceptually distinct from the serious illness experience measured in consideRATE, a validated single-item health literacy (SIL) measure [18, 22, 23]. This single-question assesses self-reported confidence in handling medical forms, which serves as a proxy for health literacy. The question asks “How confident are you in filling out medical forms by yourself?” and offers the following response options: “Extremely”, “Quite a bit”, “Somewhat”, “A little bit”, “Not at all.” We selected this item because it does not directly address consideRATE item or and does not overlap conceptually with any of the seven consideRATE questions.

2.7. Subgroup Analyses: Psychometric Assessment

Overall and item-level convergent validity analyses were conducted across all participants, as well as patient-only and care partner-only subgroups. We also conducted subgroup analyses (to assess convergent validity using Pearson’s correlation tests) for participants across demographic variables (e.g., low income and participants with low educational attainment) to examine the generalizability of the consideRATE questions across socio-economic status. This provides insight into whether the measure performs consistently across diverse patient populations and identifies whether known disparities in healthcare experiences are reflected in the scores, supporting both the measure’s validity and its potential utility for quality improvement efforts [24]. For cancer diagnoses, we selected the most common cancer types represented in our patient population for subgroup analyses because patients may interact with different clinicians based on their diagnosis, which may influence how they experience their care. These analyses were based on the overall instrument means. We defined participants with low income as those who earn less than $35,000 annually, for a family of four; this is less than the 138% Federal Poverty Level qualification for Medicaid [25]. We defined participants with low educational attainment as those who have a high school diploma or equivalent.

2.8. Evaluating Outpatient Cancer Patient Experiences

2.8.1. Subgroup Analyses: Cancer Categories

We evaluated patient experience using the consideRATE questions continuous scoring approach. We conducted subgroup analyses across cancer categories to assess whether care experience varies across cancer types due to differences in disease trajectory, treatment intensity, and patient-provider interactions [26]. First, we selected the two most common reason for visit in our dataset: “blood or bone marrow” and “breast cancer” appointments. We conducted an independent t-test to assess whether there was a significant difference in patient experience based on overall consideRATE mean score between these two subgroups. Second, we dichotomized all 12 cancer diagnoses associated with reason for visit into “solid” cancer category (e.g., breast cancer, sarcoma) and “hematologic malignancy” cancer category (e.g., leukemia, lymphoma). We conducted an independent t-test to determine whether there was a significant difference in patient experience between these two categories.

2.8.2. ConsideRATE Open-Ended Question

Two raters (JN and CHS) reviewed open-text responses to consideRATE question eight, “Are there any other things you want to share? Write them here.” We scored responses as positive (1), negative (2), or unclear (3). When comments were mixed, we marked them as unclear. We subsequently compared overall consideRATE scores with binary favorability ratings (positive vs. negative) by calculating a Phi coefficient for top-box and Pearson coefficient for continuous scores. JN and CHS resolved conflicts through discussion. Agreement between the two reviewers was 88% (Weighted kappa = 0.972).

3. Results

3.1. Participant Characteristics

A total of 258 patients and care partners participated in the survey, representing a response rate of approximately 83% (258 out of 311 approached). However, after screening out incomplete surveys, we had a final sample of 244 participants. Approximately half of our participants were patients (n = 114, 47%) and the other half were family members or care partners (n = 128, 53%). Two participants did not specify patient or care partner roles. Most participants were female (n = 170, 70%); one was non-binary. Most participants identified as white or caucasian (n = 232, 95%) and non-Hispanic (n = 238, 98%). The most common reason for visiting the cancer center was related to blood or marrow transplant (20%) (Table 1). Approximately 15% of participants (n = 36) chose “doesn’t apply” for one of the seven consideRATE questions and 7% of participants (n = 16) chose “doesn’t apply” for at least half of the consideRATE questions. Three percent of participants (n = 8) chose to skip one of the consideRATE questions and no participant chose to skip more than two consideRATE questions. Six percent of participants (n = 14) chose to skip at least half of the CANHELP questions.

Table 1.

Baseline characteristics among patients with cancer and care partner or family member

	Care partner 128, No. (%)	Patient 114, No. (%)	p-valueb
Age
18–34	10 (7)	7 (6.1)	< 0.001
35–54	33 (26)	32 (28)
55–74	69 (54)	62 (54)
75+	16 (13)	13 (11)
Palliative care
No	81 (63)	64 (56)	0.19
Yes	30 (23)	25 (22)
Cancer diagnosesc
Blood or bone marrow	25 (20)	24 (21)	0.76
Breast	15 (12)	20 (18)
Endocrine	2 (2)	6 (5)
Gastroenterology	14 (11)	6 (5)
Genitourinary	2 (2)	2 (2)
Gynecology	6 (5)	5 (4)
Head or neck	4 (3)	3 (3)
Melanoma	4 (3)	5 (4)
Neuro	1 (1)	5 (4)
Pediatric	1 (1)	None
Sarcoma	2 (2)	2 (2)
Thoracic	5 (4)	6 (5)
Unknown	36 (28)	25 (23)
Health literacyd
Confident	115 (90)	102 (89)	< 0.001f
Not confident	13 (10)	12 (11)
Educational attainment
Some high school or high school/GED degree	61 (48)	57 (50)	< 0.001
2-year degree	16 (13)	6 (5)
4-year degree	37 (29)	23 (20)
Graduate degreee	14 (11)	28 (25)
Income
$75,000+	51 (40)	44 (39)	0.075
$50,000–$74,999	28 (22)	17 (15)
$25,000–$49,999	29 (23)	35 (31)
Less than $25,000	10 (8)	13 (11)
Insurance
No	2 (2)	1 (1)	< 0.001
Yes	126 (98)	113 (99)
Type of insurance
Employer	51 (40)	41 (36)	< 0.001
Medicaid	10 (8)	14 (12)
Medicare	24 (19)	14 (12)
Medicare plus supplemental	27 (21)	32 (28)
Purchased	8 (6)	6 (5)
Other	7 (6)	7 (6)

3.2. Overall Measure

Overall scores using the consideRATE questions continuous scoring approach skewed high, with a mean of 3.72 (possible range 1–4) and standard deviation of 0.38 (Figure 1). Using a top-box approach, 45% of participants received a top-box score of 1 (indicating the highest possible experience of care rating) and 55% received a top-box score of 0 (less than highest possible rating).

Figure 1.

Histogram of consideRATE overall mean score per participant (n=233)¹

¹ 11 participants who only provided “doesn’t apply” responses to The consideRATE Questions were excluded from this figure and other analyses

3.3. Psychometric Assessment

3.3.1. Reliability

The consideRATE questions demonstrated reliability with internal consistency of 0.86 (Table 2). Cronbach’s alpha coefficient for the 7-item measure did not significantly change when an independent item was deleted (Table 2).

Table 2.

Psychometric tests using overall instrument scoring

Convergent validity	All participants r (p-value)	Patients only r (p-value)	Care partners only r (p-value)	Low-income r (p-value)	Low educational attainment r (p-value)
Mean consideRATE questions scores versus mean CANHELP lite score	0.5 (< 0.001)***	0.5 (< 0.001)***	0.5 (< 0.001)***	0.7 (< 0.001)***	0.5 (< 0.001)***
Top-box dichotomous consideRATE questions scores versus mean CANHELP lite score	0.4 (< 0.001)***	0.5 (< 0.001)***	0.3 (0.003)**	0.6 (< 0.001)***	0.4 (0.005)**
Reliability	Mean (SD)	Scale variance if item deleted	Corrected item-Total correlation	Squared multiple correlation	Cronbach’s alpha if item deleted
Overall cronbach alpha = 0.864
Physical item	3.79 (0.48)	4.327	0.584	0.393	0.852
Surrounding item	3.70 (0.49)	4.346	0.558	0.329	0.856
Feelings item	3.80 (0.45)	4.289	0.664	0.527	0.841
What matters most item	3.84 (0.39)	4.399	0.718	0.569	0.837
Plan item	3.78 (0.45)	4.277	0.667	0.472	0.840
Affairs item	3.66 (0.50)	4.174	0.630	0.476	0.846
Expect item	3.71 (0.47)	4.234	0.655	0.452	0.842

^aTo compute correlation coefficients we used Pearson correlations for continuous scoring and point-biserial correlation tests for top-box scoring.

3.3.2. Convergent Validity: Overall Instrument Scoring

Among all participants, we found a statistically significant moderate correlation (r = 0.5; p < 0.001) between the overall continuous scores of the consideRATE questions and CANHELP Lite indicating convergent validity. For top-box scoring among all participants, we found a statistically significant weak-moderate correlation (rpb = 0.4; p < 0.001) between the overall top-box scoring of the consideRATE questions and the CANHELP Lite continuous scores, indicating convergent validity.

For continuous scoring participant subgroup analyses, we found significant moderate correlations between consideRATE and CANHELP Lite: patients only (r = 0.5, p < 0.001); care partners only (r = 0.5, p < 0.001); participants with low income (r = 0.7; p < 0.001); and participants with low educational attainment (r = 0.5; p < 0.001) (Table 2). For top-box scoring subgroup analyses, we found significant correlations ranging from weak to moderate: patients only (rpb = 0.5, p < 0.001); care partners only (rpb = 0.3, p = 0.003); participants with low income (rpb = 0.6, p < 0.001); and participants with low educational attainment (rpb = 0.4, p = 0.005).

3.3.3. Convergent Validity: Item-by-Item Scoring

For continuous scoring among all participants, we found statistically significant moderate correlations (ranging from r = 0.4 (p < 0.001) to 0.6 (p < 0.001)) across all individual matched consideRATE continuous scoring and CANHELP Lite questions, indicating convergent validity (Table 3). For top-box scoring among all participants, we found statistically significant weak to moderate correlations (ranging from rpb = 0.3 (p < 0.001) to rpb = 0.5 (p < 0.001)) across all individual questions, indicating convergent validity for most items (Table 3).

Table 3.

Matching consideRATE questions with CANHELP lite questions for convergent and discriminant validity

Convergent validity			ConsideRATEd
Construct summary Care of	ConsideRATE questions	CANHELP lite questions	Continuous r (p-value)	Top-box r (p-value)
Physical problems	Q1: “How would you rate our attention to your physical problems? Things like pain, dry mouth or trouble breathing”	Q1: “How satisfied are you that physical symptoms you had (for example: Pain, shortness of breath, nausea) were adequately assessed and controlled?”	0.5 (< 0.001)***	0.4 (< 0.001)***
Emotional problems	Q2: “How would you rate our attention to your feelings? Things like feeling sad, worried, or like a burden”	Q2: “How satisfied are you that emotional problems you had (for example: Depression, anxiety) were adequately assessed and controlled?”	0.5 (< 0.001)***	0.4 (< 0.001)***
Surrounding problems	Q3: “How would you rate our attention to your surroundings? Things like noise, light, or warmth”	Q3: “How satisfied are you with the environment or the surroundings in which you were cared for?”	0.6 (< 0.001)***	0.5 (< 0.001)***
What matters most	Q4: “How would you rate our respect for what matters to you? Things like values, preferences about care, or important activities”	Q4: “How satisfied are you that you received consistent information about your condition from all doctors and nurses looking after you?”	0.6 (< 0.001)***	0.4 (< 0.001)***
Plans	Q5: “How would you rate our communication about your plans? Things like medicines, procedures or place of care”	Q5: “How satisfied are you with discussions with your doctor(s) about where you would be cared for (in hospital, at home, or elsewhere) if your condition worsened?”	0.5 (< 0.001)***	0.4 (< 0.001)***
Affairs	Q6: “How would you rate our attention to your affairs? Things like a will, finances, or advance directives for care”	Q6: “How satisfied are you that you were able to manage the financial costs associated with your illness?”	0.4 (< 0.001)***	0.3 (< 0.001)***
What to expect	Q7: “How would you rate our communication about what you can expect? Things like illness getting worse or time left to live”	Q7: “How satisfied are you with discussions with your doctor(s) about the use of life-sustaining technologies (for example: CPR or cardiopulmonary resuscitation, breathing machines, dialysis)?”	0.6 (< 0.001)***	0.5 (< 0.001)***

^aTo compute correlation coefficients we used Pearson correlation tests for continuous scoring and point-biserial correlation tests for top-box scoring.

^bAll r coefficients were rounded to the nearest tenths.

^cThe single-item health literacy measure asks “How confident are you in filling out medical forms by yourself?” and offers the following response options: “Extremely”, “Quite a bit”, “Somewhat”, “A little bit”, “Not at all.”

^dWe rounded all Pearson’s correlation coefficients to the nearest tenth.

^*p < 0.05.

^**p < 0.01.

^***p < 0.001.

3.3.4. Discriminant Validity

We found no significant correlations for consideRATE and SIL measure, indicating discriminant validity (Table 3). We found no significant correlations across most mismatched items, indicating discriminant validity (Table 3).

3.3.5. Convergent Validity: Item-Level Subgroup Analyses

For continuous scoring, item-level analyses for participant subgroups were mostly statistically significant, with weak to high correlations (Appendix 3). The items and participant subgroups in which we did not find significant correlations between matched items were consideRATE Q1 “physical problems” for the patients only subgroup, Q2 “emotional problems” for participants with low educational attainment, and Q4 “what matters most” for participants with low educational attainment (Appendix 3). For top-box scoring, item-level analyses for participant subgroups were mostly statistically significant, with weak to moderate correlations (Appendix 3). For participants with low educational attainment, we did not find significant correlations between matched items for consideRATE Q2 “emotional problems”, Q3 “surrounding problems”, Q4 “what matters most”, and Q6 “affairs” (Appendix 3).

3.4. Evaluating Patient Experience

3.4.1. consideRATE Overall Scoring

First, participants who reported being present for a “blood or bone marrow” appointment (n = 49) received an overall consideRATE mean score of 3.68 and those reporting being present for a “breast cancer” (n = 35) appointment received an overall consideRATE mean score of 3.72. We found no significant difference between these two overall mean scores (p = 0.617). Second, we found no significant difference between participants who reported a “solid” cancer category for reason for visit (n = 116, mean score = 3.71) and those who reported a “hematologic malignancy” cancer category (n = 49, mean score = 3.68) (p = 0.62).

3.4.2. consideRATE Open-Ended Question

A total of 63 (26%) participants provided free-text responses. We received 36 positive comments, 17 negative comments, and 10 unclear comments. The correlation coefficient between free-text comments (positive, negative) and overall mean consideRATE scores were not statistically significant for continuous (r = 0.2, p = 0.236) or top-box (phi = 0.1, p = 0.328) scoring.

In cases where the consideRATE score was positive and the comment was negative, participants often offered specific constructive criticism in areas beyond the scope of the questionnaire (Appendix 4). The most negative consideRATE scores did not come with associated open-text critiques as we received no comments from participants who had an overall consideRATE mean score less than 3 (Appendix 4).

4. Discussion

In the first real-world clinical and psychometric test of the consideRATE questions, a brief, plain language measure rooted in constructs of importance to patients and care partners, we found evidence of both validity and reliability in a routine outpatient cancer care setting.

We conducted three psychometric assessments: convergent validity, discriminant validity, and internal reliability. (1) For convergent validity assessment among all participants and all other subgroups, continuously-scored consideRATE demonstrated stronger convergent validity compared to top-box consideRATE scores (sometimes known as proportion of top scores or top score analysis). However, top-box scores had acceptable validity and may be easier for clinical teams to interpret and use in routine care. While continuous consideRATE scores among care partners-only subgroups revealed moderate convergent validity consistent with continuously scored patients-only subgroups, top-box consideRATE scores among care partners-only subgroups revealed slight decrease in correlation coefficient for convergent validity inconsistent with top-box scored patient-only subgroups. Although we found care partner scores to be valid broadly, it is interesting that we found a lower convergent correlation between consideRATE top-box and CANHELP Lite scores specifically among care partners-only. This may suggest a slightly reduced degree of construct alignment for one or both measures in this subgroup, which could be due to several factors. For example, CANHELP Lite was originally developed with patients in mind, and its adaptation for care partners may capture overlapping but not identical constructs. Further investigation will be needed to determine whether this finding reflects meaningful differences in how care partners assess the care experience. (2) For discriminant validity assessment, we found no correlation between our health measure consideRATE and single-item health literacy measure. (3) For internal reliability assessment, we found high internal consistency in the consideRATE questions. This is expected because consideRATE is a healthcare experience measure and each health-related item is likely to be positively correlated with the overall health construct being evaluated [27].

These results provide patients and care partners’ direct assessment of their cancer care experiences, an essential step in instrument validation [28]. The results of this study are consistent with previous consideRATE simulated online validation work and support the continued use of consideRATE in routine care and research [7, 9]. The association between consideRATE and CANHELP Lite was less strong than that seen in our previous simulated online validation work, which likely was due to the previous study’s highly controlled nature [7]. Given the imperfect comparison between the two measures, along with the expected variation within this real-world sample of patients and care partners, these scores may provide sufficient evidence to support convergent validity in these populations [6, 7]. Results of the current study not only contribute a pragmatic assessment of consideRATE, but suggest that an alternative top-box method of scoring is also acceptable, particularly when used with patients. Given ceiling effects in continuous consideRATE scores, top-box scoring may provide a more useful and meaningful metric for routine assessment of serious illness care experience [16, 28–30]. Notably, we have found significant variation in top-box scores across a variety of serious illness settings. For instance, we found the highest scores (70% top box “very good”) in inpatient palliative and hospice care, where the work of difficult decision-making is complete. And lower scores in outpatient serious illness environments, like Nephrology Clinic (35% top box) and Cancer Center (45% top box) environments, where care is complex and ongoing.

Our previous work generated evidence of the consideRATE questions’ psychometric validity in an online study [7]. Importantly, the current study marks our first use of the consideRATE questions in a clinical setting, specifically at an NCI-Designated cancer center. This study also demonstrates the use of the consideRATE questions with a wide age range, from young adults (18 years and older) to older adults (75 years and older).

4.1. Implications (Clinical and Research)

In recent years, a number of measures assessing serious illness care experiences have been developed, validated across multiple sites, and endorsed by the Centers for Medicare & Medicaid Services’ consensus-based entity, the Partnership for Quality Measurement (PQM) [31–33]. The “Measuring What Matters” project (MWM), a joint endeavor from the American Academy of Hospice and Palliative Medicine and the Hospice and Palliative Nurses Association, identified a need for measures of patient/family care experience in serious illness [34]. Like the CANHELP and CANHELP Lite, consideRATE measures the subjective experiences of care quality but does so with a particular focus on accessibility for those with serious illnesses and potentially associated cognitive impairments, by providing visual icons next to their relevant texts [5, 7]. The consideRATE questions may complement the instrument developed by Gramling and colleagues, “Feeling Heard and Understood” [8]. Concerning cancer care, a move towards “Measuring What Matters,” may be a welcome additive to discrete outcome and experience measures, as well as quality of life measures. Furthermore, the consideRATE questions contribute to the mission of the Cancer Outcomes Measurement Working Group (COMWG) and its call to identify future research priorities concerning patient needs and satisfaction [35]. Given consideRATE measures discrete outcomes that matter most to patients and care partners, but also allows them to offer their subjective interpretation, this measure may be consistent with that call. The consideRATE questions fill an important gap in the literature for psychosocial care providers.

The consideRATE questions are being studied and implemented in diverse clinical settings relevant to serious illness populations, including those with an inpatient focus, a specialty palliative care focus, and potentially mixed diagnosis [36–38]. This increased activity is generating opportunities for future benchmarking and comparisons across various clinical contexts. In addition to feasibility and acceptability studies, consideRATE is being used as an outcome in studies of serious illness [7]. Beyond research applications, and perhaps particularly important for psychosocial care in oncology, consideRATE has proven to be both feasible and useful as a routine care assessment and service recovery mechanism in routine care by addressing patient concerns and satisfaction, thereby helping to improve the overall care experience [9].

Our work across multiple contexts with consideRATE has created opportunities for us to reflect on PREMs and their place in the universe of patient-reported measures. Patient-reported experience measures (PREMs) are opportunities to describe objective patient experiences and patient satisfaction. We acknowledge that there is a distinction between patient experience and satisfaction, as well described by Bull and colleagues [39–41]. The utility of consideRATE may be at least in part because, unlike many similar measures, it is not strictly a measure of patient experience or satisfaction, but a marriage of the two, centering on the psychological and social needs of patients (Figure 2) [38, 42–44]. Although these types of blended experience and satisfaction measures may exist, there is little work to describe these measures as conceptually different from satisfaction or experience measures. We believe that to rigorously measure and improve the complex care of patients, particularly those with serious illness, there must be a multiplicity of measures and outcomes to allow researchers and clinicians to effectively measure and improve what matters most across that continuum. While we recognize that satisfaction measures can be influenced by factors outside of healthcare teams’ control (e.g., long wait times, cultural or language barriers), we believe that capturing whether patients and care partners feel that care met their needs provides important insight beyond the process measures alone that exclude subjective interpretation [31]. Also, by combining experience and satisfaction constructs, consideRATE supports a more holistic evaluation of care that can inform patient-centered improvements without unfairly personalizing individual clinicians for differences in scores for factors beyond their control.

Figure 2.

The experience and satisfaction measure spectrum

4.2. Limitations

Recruiting participants in-person may have led to a social desirability bias among participants and unconscious selection bias by the recruitment team. COVID-19 may have influenced recruitment due to universal masking and increased telehealth visits. Additionally, administering the CANHELP Lite, a much longer instrument, in addition to consideRATE may have been burdensome to patients and led to rushed completion or incomplete surveys [5, 6]. To validate the individual items of consideRATE within the meta-construct of serious illness care experience, we matched them with individual CANHELP Lite scores [7]. This is no standard use of the CANHELP Lite measure; therefore, the item-level validity should be understood within that context. Additionally, we did not exclude consideRATE responses for participants that answered at least one item, potentially influencing the consideRATE overall individual score for these particular participants by overestimating or underestimating associations depending on the patterns of missing responses. The CANHELP Lite developers recommend excluding missing data in cases where more than half of responses are missing, but for congruity we did not adopt this approach in our study. Some participants chose “doesn’t apply” for all or some items. We cannot know for sure why all participants selected this option, but many participants shared with our research team verbally that they did so because it was their first visit to the clinic and they felt it was premature to assess serious illness experience in these domains. Several expressed a preference for completing the survey after their initial visit, when they would be more familiar with the care team and setting. Additionally, a few patients were not feeling well prior to their appointments and asked family members to complete the survey on their behalf, which may have contributed to fewer patients completing the survey compared to care partners. In future research, we see an opportunity to assess serious illness care experience longitudinally so that we can see how scores and “doesn’t apply” responses change across the serious illness experience journey. We note however that the ability of consideRATE to have care partners complete surveys as proxies helped us collect useful information even when patients could not participate. We were unable to determine how similar patient and care partners dyad scores were on consideRATE due to the anonymity of the survey. Participants completed the survey about their previous experience prior to their routine scheduled appointment and their responses could have been influenced by recall bias. The generalizability of our results may be limited by the inclusion and exclusion criteria used in this study and the fact that data were collected from a single site. Future studies in multiple types of settings with diverse populations will be needed, as well as with larger sample sizes and further qualitative exportation of drivers of negative scores. Also, future studies may definitely determine whether care partners might be able to function as proxy measure completers in some situations.

5. Conclusion

We demonstrated the first real-world use of consideRATE, and we think there is mounting evidence to support the use of consideRATE by both psychosocial researchers and clinicians working in the area of serious illness, including oncology. In this context, consideRATE could be used to measure the effectiveness of research and quality improvement interventions, as well as to monitor care quality across clinical contexts and serve as a mechanism for service recovery.

BOX 1. The consideRATE questions⁵.

	How would you rate our attention to your physical problems? Things like pain, dry mouth or trouble breathing
	How would you rate our attention to your feelings? Things like feeling sad, worried or like a burden
	How would you rate our attention to your surroundings? Things like noise, light or warmth
	How would you rate our respect for what matters to you? Things like values, preferences about care or important activities
	How would you rate our communication about your plans? Things like medicines, procedures or place of care
	How would you rate our attention to your affairs? Things like a will, finances or advance directives for care
	How would you rate our communication about what you can expect?Things like illness getting worse or time left to live
	Are there any other things you want to share? Write them here