User-Centered Design of the consideRATE Questions, a Measure of People’s Experiences When They Are Seriously Ill

Abstract

Context.

No brief patient-reported experience measure focuses on the most significant concerns of seriously ill individuals.

Objectives.

The objective of the study was to develop the consideRATE questions.

Methods.

This user-centered design study had three phases. We reviewed the literature and consulted stakeholders, including caregivers, clinicians, and researchers, to identify the elements of care most important to patients (Phase 1). We refined items based on cognitive interviews with patients, families, and clinicians (Phase 2). We piloted the measure with patients and families (Phase 3).

Results.

Phase 1 resulted in seven questions addressing the following elements: 1) care team attention to patients’ physical symptoms, 2) emotional symptoms, 3) environment of care, 4) respect for patients’ priorities, 5) communication about future plans, 6) communication about financial and similar affairs, and 7) communication about illness trajectory. Phase 2 participants included eight patients, eight family members, and seven clinicians. We added an open-text comment option. We did not identify any other issues that were important enough to participants to include. Response choices ranged from one (very bad) to four (very good), with a not applicable option (does not apply). Phase 3 involved 15 patients and 16 family members and demonstrated the acceptability of the consideRATE questions. Most reported that the questions were not distressing, disruptive, or confusing. Completion time averaged 2.4 minutes (range 1e5).

Conclusion.

Our brief patient-reported serious illness experience measure is based on what matters most to patients, families, and clinicians. It was acceptable to patients and families in a regional sample. It has promise for use in clinical settings.

Introduction

Care of people who are seriously ill needs improvement, as patients have persistent unmet needs.¹ Seriously ill individuals with cancer, for example, value and report lack of support concerning their daily life, feelings, and identifies as individuals.² And they have symptoms, like breathlessness and anxiety, that their care teams are often unaware of or do not address.³

In response, researchers and clinicians are trying to measure care during serious illness, including identifying and defining the population with serious illness, measuring symptom management, and health care expenditures before death.^1,4,5 There are many different ways to measure the quality of care, ranging from indirect structural measures like the number of beds in a facility, to process measures like the number of inpatients seen by the palliative care service.⁶ Outcome measures assess patients’ actual health status, like lung function.⁶ But there is another category of measurement, independent of outcomes: patient experience and satisfaction measures. Experience measures capture care elements that are important to patients, and satisfaction measures capture their subjective assessment of health care quality. A well-crafted patient experience or satisfaction measuredindependent of outcome measuresdfor seriously ill individuals could facilitate assessment and improvement of care.⁷

There are many measures concerning serious illness experience or quality of care before death.^8,9 Unfortunately, many of these, like The Palliative Outcome Scale or QUAL-E, mix outcomes, such as pain, and experiences, such as whether the health care team addressed concerns, making it challenging to isolate patients’ experience.⁸ Those measures that do measure experience, like the Canadian Health Care Evaluation Project Questionnaire are long, requiring time and concentration that may be burdensome for people suffering from cognitive difficulties associated with serious illness.^10e12 Or cannot flexibly move between patient and family respondents with the same measure, like the FAMCARE suite of tools. We were unable to identify a brief patient-reported experience measure for people with serious illnessda gap in the literature.

We, therefore, aimed to develop a patient-reported experience measure that was

• Brief enough to be used in busy clinical settings

• Flexible enough that patients or their caregivers could complete in any setting where people with serious illnesses receive care

• Focused on addressable aspects of care the care team could improve

• Based on seriously ill people’s care priorities

Methods

Our multistep, user-centered design, and community engagement study had three phases (Fig. 1).

Fig. 1.

Iterative the consideRATE questions development.

• Phase 1: Development

• Phase 2: Refinement with seriously ill individuals, families, and clinicians across a variety of care settings

• Phase 3: Testing with seriously ill individuals and families across a variety of care settings

A multidisciplinary stakeholder group, including families of seriously ill people, palliative care physicians, and researchers, guided this study.

Phase 1: Development

Literature Review

Elements of the Serious Illness Experience.

We searched Ovid (MEDLINE) to identify articles about the aspects of care experience most crucial to seriously ill individuals. We intended to identify a recent high-quality systematic review or another review (Appendix I).

Stakeholder Consultation

Elements of the Palliative Interview.

We consulted our two palliative care physician stakeholders about the typical palliative care clinical interview flow and the aspects of serious illness experience appropriate for a measure. We also discussed desired measure characteristics.

Draft Measure.

We developed a draft measure, hewing to the desired measure characteristics detailed by the stakeholder team.

Phase 2: Refinement

We refined the measure by interviewing seriously ill patients, their family members, and clinicians using cognitive interview techniques.¹³ Participants shared their understanding of each item and identified confusing language.¹⁴ We iteratively modified the measure during two rounds of interviews. See Appendix II for COnsolidated criteria for REporting Qualitative research checklist.¹⁵

Screening and Eligibility Criteria.

We identified eligible patients, family members, and clinicians by attending the palliative care team’s daily team meetings and through personal contacts with clinicians at Dartmouth-Hitchcock Medical Center (DHMC). For eligible patients, we collected their room numbers (inpatient) or appointment times (outpatient) and approached them independently. We purposively sought diversity of socioeconomic status and age.

Seriously ill individuals able to provide consent, aged 18 or older, and able to speak English, as well as their caregivers were eligible for inclusion. We operationalized serious illness using the surprise question, a tool for identifying patients with a high risk of death.^16,17 Using colleague referrals, we consented and conducted cognitive interviews with clinicians of seriously ill patients.

Data Collection

Interviewer.

C. H. S. conducted the interviews. At the time, she was a female health services researcher with an MPH degree and extensive qualitative research experience with vulnerable individuals.

Recruiting Participants.

We used recruitment and consent techniques approved by Dartmouth’s Committee for the Protection for Human Subjects. We provided a study information sheet with information about our motivations for doing the study. If prospective participants declined, we politely disengaged. We did not establish relationships with patient and family participants before beginning the study.

We scheduled meeting times in private rooms with clinicians, with whom we had pre-existing relationships.

Additional Data Collection.

We administered a brief demographic questionnaire to each participant and assigned unique ID numbers. We also took field notes, which we kept in study lockers.

Interviews.

We allowed time for participants to review the draft measure in its entirety. Then, we asked them to read the items one by one. We followed up with questions and probes focused on comprehension, including whether there were any points of confusion, lack of clarity, or unfamiliar words.¹³ We asked participants what the questions and terms meant to them in their own words. We also asked them what they thought the items were intended to measure. We incorporated the think-aloud technique and offered participants alternative versions of measure elements for consideration.¹⁴ We also used the interviews to assess the acceptability and ease of use of different response choices. Occasionally, family members or hospital staff were present for interviews.

The cognitive interview guide is provided in Appendix III. We did not pilot it but designed it based on similar measure development projects.^18,19 We did not offer repeat interview opportunities or return transcripts to participants for review.

We stopped conducting interviews when we reached content saturation so that we could incorporate stake-holder suggestions before beginning another round.^20,21 We confirmed saturation through conversations with coauthors.

Analysis.

We recorded and transcribed each interview using a Health Insurance Portability and Accountability Act compliant service. C. H. S. and P. S. coded the data and conducted thematic analysis in between data collection rounds.²¹ We independently read a sample of transcripts, developing descriptive codes and memos using ATLAS.ti (Scientific Software Development GmbH). We reviewed the codes and agreed on a codebook, which we used for the remaining transcripts. The code hierarchy included overall measure feedback, feedback on specific items, and feedback on response choices. Subcodes explored more specific phenomena under each category: suggestions for measure revision, intended constructs, ideas about measure revisions, and overall positive or negative impressions.

By examining memos, codes, and field notes, we developed themes to summarize our findings, confirming with team members. We did not undertake member checking.

Phase 3: Testing

Data Collection.

We screened and recruited new participants for this phase. C. H. S. and A. J. conducted pilot testing.

Conducting Pilot Test.

After gaining informed consent from participants, we administered the consideRATE questions, along with a demographic and acceptability questionnaire, which included an open-text response option. We timed measure completion.

Analysis.

We calculated the mean, median, minimum, and maximum time to complete the measure. We also calculated the mean, median, minimum, and maximum scores across all items, excluding does not apply responses.

Results

Phase 1: Development

Literature Review

Elements of the Serious Illness Experience.

We identified a systematic review by Virdun et al. in 2015, which quantitatively represented the most important elements of inpatient serious illness care. These elements guided our construct selection process (Fig. 2).

Fig. 2.

Most important elements of serious illness care comprising the consideRATE suite constructs. Note: Based on the systematic review by Virdun et al.²² Numbers correspond to ranked elements of serious illness experience, according to patient priorities. Adequate environment for care and minimizing burden tied for fifth most important elements of care.

Measure Characteristics.

In partnership with our eight- member stakeholder group, we determined a need for a new measure with the following criteria:

• Brevity: less than five minutes to complete

• Flexibility: useable by individuals with a variety of serious illnesses in inpatient, outpatient, or home care; and by patients or their family members as proxies

• Addressability: experiences that care teams could realistically improve

• Meaningfulness: essential topics for seriously ill individuals.

Draft Items.

We drafted an initial measure, which addressed seven constructs of high relevance to patients, including the one provided here.

• Attention to physical symptoms

• Attention to emotional symptoms

• Attention to the environment of care

• Respect for patient priorities and values

• Communication about plans

• Attention to financial or other affairs

• Communication about illness trajectory (prognosis)

The items encompass the elements of care rated as most important by seriously ill people (Fig. 2). The initial first draft of the consideRATE questions included seven items. It had Likert-like response choices on a four-point scale, ranging from one representing poor serious illness experience to four representing excellent serious illness experience, in an effort to capture a range of experiences.

Phase 2: Refinement

Participants.

Twenty-three people participated in cognitive interviews: eight people with serious illness, seven family members, and eight clinicians.

Patients and Caregivers.

Patient and family participants were mostly white, older, and with average income for Northern New England.²³ We conducted 10 inpatient interviews, five outpatients, and one in the community (Table 1). Three patients and three family members declined, and two participants with- drew because of fatigue or mental health concerns. In addition, many potential participants did not expressly decline but were unable to participate because of logistical concerns.

Table 1.

Participant Characteristics

Characteristics	Cognitive Interviews		Pilot Test (n ¼ 33)
	Round 1 (n ¼ 13)	Round 2 (n ¼ 10)
Patient and family participants
Race/ethnicity
American Indian or Alaska Native	0	0	1
Asian, Black or African	0	0	0
American, Native
Hawaiian, or Other Pacific
Islander
White or Caucasian	13	7	31
Spanish/Latina	0	0	0
Education
Postgraduate	2	0	9
Bachelors	4	5	5
Associates	0	0	3
Some college	1	0	10
High school	1	2	10
Health literacya (Chew’s single-item health literacy screener)
High	3	5	8
Low	1	1	4
Median income	$28,800	$100,000	$75,000
	n ¼ 3	n ¼ 4	n ¼ 7
Clinician participants
Years of practice (mean)	29	16
Type of clinician
Physician		1
Nurse	1	0
Advance practice nurse	2	0
Social worker	0	1
Medical student	0	0

^aHigh health literacy defined as a response of excellent; low health literacy defined as a response of quite a bit or below.

Interviews ranged from 45 minutes to two hours, with an average of one hour. Inpatient interviews were lengthy because of interruptions and distractions in the hospital setting.

Clinicians.

We interviewed three physicians, two advanced practice nurses, one nurse, and one social worker. Specialties included palliative care, oncology, and general surgery (Table 1).

Findings and Iterative Changes.

Significant changes to the consideRATE questions are detailed in Table 2. Our codebook reflected the structure of Table 2, with codes representing overall feedback and feed- back on the individual elements of the measure. We used many subcodes.

Table 2.

Changes in the ConsideRATE Question Instructions and Items

	Issue	List of Changes Made to Measure	Participant Feedback on Issue
Feedback on measure overall
Instructions	Instructions unclear	Instructions separated into who?, what?, where? and when? sections	So most of these questions reflect how the care team is doing? [in reference to instructions]] That’s kind of hard to read into this …—patient, Round 1 (P_DH_3)
Items	Constructs unclear	Constructs named explicitly. Items rephrased into questions	What is it [the question] asking? When I think I might die.—patient, Round 1 (P_DH_2)
		Your added ahead of each construct	How would you score attention to feelings? How would you score attention … ? I want to put your in there. That’s my sense about it—patient, Round 2 (CL_DH_14)
Responses	Questions not always relevant or modifiable	Not applicable option added (no opinion: Round 1; does not apply: Round 2)	If they have not brought this up, then there should be something that is not discussed or not discussed yet—clinician, Round 1 (CL_DH_6)
Feedback on individual items
Attention to symptoms	Construct unclear	References to bowel movements and urination removed	My first reaction to that was needing assistance to pee—family member, Round 1 (F_DH_12)
Attention to surroundings	Question not always relevant or modifiable	Not applicable option added (no opinion Round 1; does not apply, Round 2)	Those are not necessarily things that I could modify as a clinician—clinician, Round 1 (CL_DH_7)
Respect for patient priorities	Construct unclear	Examples modified. Added the word, respect in Round 2	It’s not specific enough. […] It could be food, whether you were cold, whether you want to go to the next place—family member, Round 1 (F_DH_13)
Communication about plans	Construct unclear	Examples changed to things like medicines, procedures, or place of care	I guess I’d want more clarification on place of care a little bit. I think sometimes people assume—and I just had this conversation with our doctors yesterday about, Can he stay here? Can he go to the new palliative care unit here?—family member, Round 1 (F_DH_12)
Communication about illness trajectory (prognosis)	Construct unclear	Item moved to end of measure. Example changed to things like time left to live	I guess the thing is that everyone’s going to die, so “possibility” is not even a word because it’s like none of us are getting out of here alive—clinician, Round 1 (CL_DH_7) I think, as a patient, that [laughter] the first thing is a little bit overwhelming and that’s the one that’s like, Wait? What do you know that they have not told me yet? I would put the simpler things, the less threatening things first, for one thing—clinician, Round 1 (CL_DH_6)

Overall Observations

Appreciation.

In Round 1, participants appreciated the first draft. One family member (F_DH_11) said “glad you are working on this” because “this will help people” and “it’s an excellent idea.”

Difficulty With Comprehension.

Our most salient finding was the difficulty with which participants understood our initial draft measuredeven those with high levels of education. One patient (P_DH_2) interpreted the item about the quality of prognostic communicationdwhat to expectdincorrectly. He thought the item was a question about how much longer he would live, rather than a question about whether his care team had adequately communicated his expected survival.

Family members we interviewed, even those who worked in the medical field, also struggled to comprehend simple concepts, likely because of fatigue and emotional exhaustion. When probed about a question, one family member (F_DH_12), a health professional herself noted, “I’m too tired to think about it _ “ and trailed off.

To accommodate this particular challenge associated with the serious illness population, we revised our measure to ensure all language was as plain as possible. We changed the word measure to questions. We also put all instructional headers in the form of questions, per the Federal Plain Language Guidelines. We used active language and short sentences. Our final version is written at a second-grade level. We learned that the word rate is preferable to score because participants were familiar with it from consumer contexts.

In Round 2, participants suggested that more specific language would help them understand how the consideRATE question items relate to them and their experience. So, we added your ahead of each construct and our in front of each action.

To further improve understandability, we added icons representing each construct during Round 2, which participants said made them easier to conceptualize (Fig. 3).

Fig. 3.

Changes in the consideRATE question instructions and items. Note: See Appendix IV for complete measure items.

Opportunities for In-Depth Feedback.

We also found that patients and families wanted the chance to share more information. So, we added an open-text question about any other concerns.

Minor Themes.

Some participants wanted their care teams to know their responses to the consideRATE questions, whereas others preferred to stay anonymous. In response, we added an optional item where participants could share their names.

We also found that participants wanted facilities to customize the image on the front and back covers.

Items.

Participants requested more guidance about the patient priorities item, asking us to, name a few that would fall into the category that you’re asking about, which would let me know more about answering it to make it clearer (P_DH_3).

Similarly, participants requested more guidance about the plans item. Concerning the symptoms item, participants asked for further clarification about whether the measure referred to physical symptoms or function. In addition, participants struggled to understand how they would answer the surroundings item if they were outpatient, so we added a not applicable response option. Many participants also pointed out that the language about prognosis was unclear and that everyone has a possibility of death, so we revised the phrasing to focus on time left to live.

Participants also suggested reordering the items in the measure. In our initial draft, the prognosis item appeared first. Participants expressed concern about the item they perceived as most difficult coming first. We rearranged the items to map the flow of a clinical visit: physical and emotional symptoms first, charged topics like prognosis and priorities second.

For Round 2, participants still struggled with the item about respect for patient priorities. Based on discussions with stakeholders during interviews, we added the word respect ahead of the construct, which helped them understand more consistently. The question now reads, How would you rate our respect for what matters to you?

Beyond the addition of a free-text item for any other things that may be important for seriously ill individuals or their families to share, we did not identify any missing or redundant items in our measure.

Phase 3: Testing

Participants.

We pilot tested the questions with 15 patients and 16 family members, 20 from the inpatient setting, including both inpatient wards and intensive care units, and 11 from the outpatient setting. Twenty-five were patients or family members of patients cared for by the palliative care team, four by the general surgery team, and two by the oncology team. Demographics were consistent with those for northern New England (Table 1).²³ We did not collect diagnoses.

Completion Time.

Mean measure completion time was 2.4 minutes, and median completion was 2.5 minutes. Completion time ranged from one to five minutes.

Responses.

The overall mean response, on a scale from 1 (very bad) to 4 (very good), was 3.6 (SD ¼4), in between good and very good. The minimum score was 2.7, and the maximum score was 4.

Reactions.

Participants expressed appreciation for the measure during our pilot test. One participant noted that the measure had very good information (P_DH_7). Two family members of a person hospitalized with serious illness asked why this sort of thing had not been done earlier in the process (F_DH_5; F_DH_6).

Thirty respondents said completing the questions was not at all distressing. One respondent said it was somewhat distressing, noting it was hard to answer for someone else (F_DH_14). Twenty-nine respondents said that completing the questions was not at all disruptive, although one respondent said it was somewhat disruptive. We cannot determine if the measure was disruptive itself or if study-related forms and questionnaires contributed.

Five respondents said the questions were somewhat confusing. These participants were typically family members or friends, noting that it was difficult to understand how to fill it out, whether it was on behalf of the seriously ill person or for themselves.

Changes.

In response to this confusion, we made one change to the measure after the pilot test, further clarifying the role of family members and friends, adding, Who? People who are ill. Fill out for yourself if you are ill. Or based on the ill person’s experience, if you care for them (Table 2). The final consideRATE questions are provided in Appendix V.

Final Measure.

The final measure has nine items concerning care team’s attention to physical problems, feelings and affairs, respect for what matters to patients, communication about plans and what to expect, as well as an open text response option for any other things. In addition, users have the option to share their name with the care team or remain anonymous. The consideRATE questions have a four- point Likert-like response scale, including very bad (1 point), bad (2 points), good (3 points), and very good (4 points), with no neutral option. There is a does not apply choice, which does not confer points. Scores convey information via both overall means and by item means.

Discussion

Main Findings

We developed a brief measure of serious illness experience, with eight items that most patients or family members can complete in less than three minutes. No other similar measure exists. The measure is based on the priorities of people who are seriously ill.²²

With user-centered design and community engagement, we were able to develop questions that patients, family members, and clinicians found acceptable, through dramatically simplifying the phrasing of our questions and instructions. Piloting revealed the measure is easy to use, suggesting hospitals and health systems could use it in both inpatient and outpatient care environments.

Strengths and Limitations

Use of established user-centered design techniques, along with community engagement, for measure development is a strength of this study. In addition, we included a diverse sample of participants, including individuals across socioeconomic strata, with varying levels of education and income. This sampling approach allowed us to ensure that our measure was simple enough to be understandable and acceptable to people from all socioeconomic groups.

One limitation of this study was the lack of racial, ethnic, cultural, and geographic diversity. DHMC is in an area of rural New England, which is predominantly white and where the residents speak English.²³ Future studies of the consideRATE questions should include a more racially and ethnically diverse population to determine if the measure is understandable and acceptable to a more diverse population. In addition, future studies should examine the consideRATE question scores across multiple diseases and conditions and explicitly considering index condition. A high proportion of our participants came from the palliative care setting, where scores might tend to be higher because of palliative care’s expertise and focus on the patient and family experience.

Results in Context

Although other patient-reported measures focus on serious illness, they have limitations. First, most, like the Canadian Health Care Evaluation Project Lite, are too long to reasonably be completed by most people with serious illness during routine care, which can be cognitively burdensome.^11,24 Others, like the FAM- CARE Patient, are narrowly focused on one disease, condition or place of care, making generic serious illness experience measure difficult.²⁵ And still others, like the QUAL-E, mix outcomes, like symptoms, with patient experience or satisfaction, making it difficult to isolate patient experience and satisfaction independent of other indicators.²⁶ Our measure allows measurement throughout the illness journey, whether in the intensive care unit or in the primary care clinic. It also allows the flexibility to transition measure completion from patients to their family members as proxies, which is essential for tracking serious illness over time, as patients often lose cognitive capacity to complete measures as they become increasingly ill.²⁷

Because our measure is based on the elements of care that seriously ill individuals think are most important, it is a care experience measure.²⁸ Patient experience is loosely defined, but a synthesis across 14 years of literature suggests it may include patients’ interactions across the continuum of care relative to their expectations.²⁸ Although some groups define patient experience more narrowly, as a set of activities that can be objectively measured, we posit that because serious illness is so personal and culturally informed, a more inclusive definition that encompasses patients’ expectations is necessary.²⁹ So, instead of a simple binary assessment of whether these aspects of care occurred, our measure also has a patient-reported quality assessment. It is not a simple satisfaction measure.³⁰ We note in addition, that in user testing, patients appreciated the opportunity to rate the quality of their care experiences.

The consideRATE questions are a part of a portfolio of quick and easy point-of-care tools designed to be easily integrated into regular care to improve it.^18,19

Our findings echo previous research that details the extent to which the burden of living with serious illness affected both patient and family member participants.³¹ In our cognitive interviews, we found that both patients and families, regardless of educational background, and sometimes to their surprise, had difficulties processing complex concepts. Fatigue and emotional distress influence the ability of individuals to process complex information, so it makes sense patients and family members had difficulty with cognition.^32,33 Furthermore, the effects of multiple medications, hospitalization, and chronic pain on patients themselves may compound cognitive difficulties.^34,35

This relationship between serious illness and cognition has implications for designing and implementing interventions in clinical practice and for future research with this population. Had we not used user-centered design and community engagement to develop the consideRATE questions, we likely would have failed to consider the impact of serious illness on cognitive processing and might have created a measure that was too complex to be used in this setting.

Future Research and Next Steps

An additional finding with implications for future research is the challenge of doing in-depth qualitative research in the inpatient setting. Unpredictable visits by consulting physicians and other allied health professionals, or the need for procedures or other care, often delayed or interrupted data collection.

Next steps in the development of the consideRATE questions included an online validation of the tool and will include a real-world validation in a variety of clinical settings.

Conclusions

The experiences of people who are seriously ill are particularly relevant as the population ages.^1,36

The consideRATE questions are simple, brief, and capture the elements of serious illness care most important to people who are ill. If validated, they may be a useful way to measure serious illness experience. Because optimizing illness experience, as opposed to changing outcomes of illness, is a common goal of serious illness care, a measure such as the consideRATE questions could support practice improvement. Community clinics and academic health systems could easily use the consideRATE questions to routinely measure the experience of serious illness in their patients and use the results to improve their care.

Key Message.

In this article, we report on the development of a new tool for measuring patient experience in serious illness visits, the consideRATE questions. The questions are brief, generic, and have the potential to improve the serious illness experience.