Abstract
Background:
Despite known benefits of planning for end of life, no digital tool exists to help patients with advanced cancer and their loved ones plan for death comprehensively. To address this unmet need, we developed a preliminary version of an innovative website to help patients with advanced cancer prepare for end of life.
Methods:
Guided by the ORBIT model for behavioral intervention development, in this ‘Phase 1b’ study, patients with advanced cancer (n=10) and their caregivers (n=10)) participated in a “Think Aloud” exercise and usability protocols to optimize the end-of-life planning website. The website was iteratively refined in collaboration with the partnering company throughout the study. Participants also completed the System Usability Scale and Acceptability E-Scale, with a priori benchmarks established for usability and acceptability.
Results:
Patients (N=10) and caregivers (N=10) completed usability testing. Patients were majority female (80%), White (100%), and a mean age of 58 years. Caregivers (N=10) were majority male (60%), spouse/partner (90%), White (90%), and a mean age of 59 years. For patients, a priori hypotheses were met for both acceptability and usability. For caregivers, acceptability was just below the cutoff and usability exceeded the cutoff. Overall, patients and caregivers reported high levels of satisfaction and found the website helpful, with specific suggestions for changes (e.g., add more information about information security, improve text legibility).
Conclusions:
The findings of this study will inform modifications to optimize an innovative website to support patients with advanced cancer prepare holistically for the end of life.
Keywords: cancer, patients, caregivers, death, user-computer interface
Introduction
Addressing end-of-life concerns for patients with advanced cancer and their loved ones is critically important to improving care for this high need population. There have been widespread attempts to improve rates of advance care planning (ACP) engagement, including shifting national care guidelines of the National Comprehensive Cancer Network (NCCN),1 American Society of Clinical Oncology (ASCO),2 and Centers for Medicare and Medicaid Services (CMS)’s Oncology Care Model.3 While much research has focused on improving completion of advance directives, with some promising results,4,5 little research has focused on how to equip patients with advanced illness to plan for their end of life more holistically. Despite the absence of tools for this, most patients with advanced cancer view their own end-of-life planning more broadly than merely planning for their end-of-life care. In addition to planning for their treatment preferences, patients indicate that resolving emotional, relational, and spiritual issues; not being a burden to one’s family and having their financial affairs in order as being critically important to them.7,8 These non-medical issues have been neglected in prior interventions despite their critical importance to patients with advanced illness and their families. With increased demand for advance directive documentation since the onset of the COVID-19 pandemic,6 this is an especially critical area of needed research.
To address these unmet needs, we partnered with a tech company developing an innovative website to help patients prepare for the end of life across multiple domains, including medical (advance care planning (ACP); financial planning; burial arrangements; and emotional, relational, and spiritual care. The key cornerstones of the website include the following: (1) taking a comprehensive approach to planning for end of life; (2) creating an interactive checklist with easy-to-follow instructions as well as guidance and resources to make planning each step simple and easy to follow; and (3) embedding features that allow for easy transfer of tasks from patients to caregivers if and when desired by the patient facing an advanced illness. The utilization of digital health was selected as the most effective, innovative, tailored, and easy to disseminate approach to improving patients’ engagement in comprehensive end-of-life planning. Across multiple studies, digital health applications have demonstrated acceptability and feasibility and are well-received by users for behavioral change interventions.9
The aim of this study was to obtain feedback from key advisors (advanced cancer patient-caregiver dyads) using “Think Aloud” exercises, usability testing protocols, and an iterative design approach to inform modifications to the beta version of the website. We hypothesized that patient and caregiver ratings of the web-based program would meet a priori and standardized benchmarks for acceptability (scores of 24 or higher on the Acceptability E-Scale)10 and usability (scores of 68 or higher on the System Usability Scale).11
Materials and Methods
Theoretical Framework
The comprehensive end-of-life planning website, developed in collaboration with our tech partner, is grounded in the Goal-Setting Theory of Motivation, which indicates that the best completion of a goal occurs when they are: (1) specific and challenging; (2) linked back to feedback on results; and (3) create commitment and acceptance.12 Self-efficacy and deadlines further improve the effectiveness of goals. Using this framework, the tech company designed the website to provide clear guidelines for completing tasks related to end-of-life planning. Each goal within this website is: (1) specific (task listed clearly), (2) linked back to feedback (live update on progress and completion of goals); and (3) creates commitment and acceptance (clearly ties each goal to the overall goal of planning for one’s future). Additionally, the website includes ordered steps that individuals can follow to track their progress in meeting each goal. This website is unique in that it comprehensively links end-of-life planning tasks, allowing patients to complete all needed tasks in one, central location. It allows patients to store their information securely and share it only after a life event, such as hospitalization or death, with a trusted contact. Furthermore, the website provides guided assistance with each task to ensure patients know how to complete each step, which is lacking from many prior tools developed.
Participants and Procedures
Participants were recruited from an NCI-designated comprehensive cancer center. Eligibility criteria included: (1) diagnosis of poor prognosis advanced cancer defined as locally advanced or metastatic cancer (e.g., pancreaticobiliary, esophagogastric, hepatocellular carcinoma, lung, or gynecological cancer) and/or disease progression following at least first line chemotherapy; (2) access to a computer or mobile device; and (3) the ability to provide informed consent. Exclusion criteria included: (1) not fluent in English; (2) severely cognitively impaired (as measured by Short Portable Mental Status Questionnaire scores of < 6)13; (3) too ill or weak to complete the interviews (as judged by the interviewer); and (4) children and young adults under age 18 years. Caregiver eligibility criteria included: (1) the person (family member or friend) whom the patient indicates being an informal caregiver; (2) English speaking; (3) 18 years of age or older; and (4) able to provide informed consent.
Local IRB approval was granted in accordance with the US Federal Policy for the Protection of Human Subjects. To minimize participant burden and in response to COVID precautions, all recruitment and consenting was conducted remotely. To recruit eligible patients, research staff reviewed the electronic medical records (EMR) and worked with the clinic staff to review the daily clinic schedule for eligible patients. After confirming eligibility with the patient’s treating oncologist, the research assistant called study participants, confirmed eligibility, and explained the study to the patient. If a patient expressed interest, they were emailed an introductory letter, a study information sheet, and electronically consented to the study via REDCap. Once consented, the interviews were conducted over the telephone and the study questionnaires were completed via REDCap.
Measures
Patient and caregiver demographics
Patient and caregiver demographics were measured via self-report assessing age, gender, race, ethnicity, marital and parental status, employment status, education, income, religious affiliation, and insurance coverage.
Patient clinical variables
Patient clinical variables were assessed using records from medical charts and included the following: cancer diagnosis and stage, treatment received, co-morbid conditions (i.e., Charlson Comorbidity Index, CCI), and Karnofsky performance status.
Acceptability
Acceptability of the intervention was assessed using the Acceptability E-Scale. The Acceptability E-Scale is a 6-item scale, scored on a 5-point Likert scale (1=difficult to understand; 5=easy to understand), with previously demonstrated good internal consistency (Cronbach’s alpha=.76).
Usability
Usability of the intervention was assessed using the System Usability Scale (SUS). The SUS is a 10-item scale, scored on a 5-point Likert scale (1=strongly disagree; 5=strongly agree), with previously reported excellent internal reliability (Cronbach’s α = 0.91).
Satisfaction
Satisfaction was measured with a single 10-point Likert scale item in which patients rated how satisfied they were with the website (1 = not at all satisfied, 10 = extremely satisfied). Helpfulness was rated on a single 5-point Likert scale item in which patients rated how helpful the website was (1 = not at all helpful, 5 = very helpful).
Usability Testing
The content of the website includes detailed instructions, checklists, and assistance in completion of multiple end-of-life planning tasks across the following domains: (1) Assign a deputy, (2) Personal information, (3) Critical legal and health information, (4) Funeral wishes, (5) Online accounts and devices, (6) Veterans benefits, (7) Dependents, (8) Financial assets and debts, (9) Recurring expenses, and (10) Emotional support. Participants viewed and interacted with all checklists and features of the website while engaging in “Think Aloud” exercises. The “Think Aloud” captured the process participants used when interacting with the intervention. During the think aloud exercise, participants were encouraged to provide their feedback, in real time, while engaging with the website. After completing the field test (~35 minutes), participants completed the usability protocol which assessed perceived usability and acceptability of the website as well as user-centered design. This brief interview (~15 minutes) included recording errors and difficulties as well as assessing participants’ likes, dislikes, and recommended changes for the website. Each participant received a $35 gift card as compensation. Following each session, study staff summarized and communicated participant feedback to the developers to make revisions to the website in real-time. Figure 1 outlines the details on the iterative design process of real time development with our tech partner.
Figure 1. Iterative design process.
Working closely with Peacefully developers, the website was refined after each interview and upon completion of usability testing.
Analyses
Qualitative Analyses
Qualitative Description (QD) was used to analyze the data.14–16 QD is appropriate for the proposed study because it focuses on providing a rich description of one’s experience of an event, which in the proposed study is participants’ experience of using the intervention. Audio recordings of the semi-structured interviews were professionally transcribed and coded by study staff for the following categories based on the semi-structured interview guide: (1) positive feedback, (2) negative feedback, and (3) suggestions for changes. All transcripts were coded separately by two individual coders for manifest (e.g., frequencies) and latent (e.g., underlying themes) variables. Upon completion, two independent researchers extracted major themes from coded transcripts and met together to resolve discrepancies and create a comprehensive list of feedback to share with the developers. The comprehensive listing was further summarized according to site navigation, content, and design and included positive feedback, negative feedback, and suggestions for changes from patients, caregivers, and overall/combined sample of patients and caregivers. Recommended changes which will be included in the next version of the website prototype.
Quantitative Analyses
To analyze acceptability of the intervention, scores on the Acceptability E-Scale were summed to produce an overall score ranging from 6 to 30. According to prior research, a score of 80% or higher (total score of 24 or higher) is considered acceptable for use10 and thus was utilized as the cutoff in the proposed study. To analyze usability, a contribution score for each item was summed from the SUS scale. Then, the total score was multiplied by 2.5 to produce an overall score ranging from 0 to 100. Prior research indicates that a SUS score greater than 68 is considered above average usability17–20 and ready for widespread use and thus was utilized as the cutoff in the proposed study.
Results
Patients (N=10) and caregivers (N=10) completed usability testing (see Table 1 for sociodemographic characteristics). Patients were majority female (80%), White (100%), married/partnered (90%), and a mean age of 57.7 years. The majority of caregivers (N=10) were a spouse/partner of the patient (90%). Caregivers were majority male (60%), White (90%), married/partnered (90%), and a mean age of 58.3 years. Patients had advanced disease and diagnosis of lung cancer (60%), head and neck (10%), and “other” cancer (30%) (see Table 2 for clinical characteristics).
Table 1:
Sociodemographic characteristics of patients (N=10) and caregivers (N=10).
| Patients (N = 10) | Caregivers (N = 10) | Overall (N=20) | |
|---|---|---|---|
| Mean (SD) | |||
| Age (years) | 57.7 (7.48) | 58.8 (11.47) | 58.3 (9.44) |
| N (%) | |||
| Sex Assigned at Birth | |||
| Male | 2 (20.0%) | 6 (60.0%) | 8 (40.0%) |
| Female | 8 (80.0%) | 4 (40.0%) | 12 (60.0%) |
| Hispanic/Latino | |||
| Yes | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| No | 10 (100.0%) | 10 (100.0%) | 20 (100.0%) |
| Missing | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Race | |||
| White or Caucasian | 10 (100.0%) | 9 (90.0%) | 19 (95.0%) |
| Black or African American | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Asian | 0 (0.0%) | 1 (10.0%) | 1 (5.0%) |
| American Indian or Alaskan Native | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Native Hawaiian or Other Pacific Islander | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Multi-Racial | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Other (Hispanic or Latino) | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Other | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Relationship Status | |||
| Married/partnered | 9 (90.0%) | 9 (90.0%) | 18 (90.0%) |
| Not Married/partnered | 1 (10.0%) | 1 (10.0%) | 2 (10.0%) |
| Religion | |||
| Protestant | 3 (30.0%) | 1 (10.0%) | 4 (20.0%) |
| Agnostic | 2 (20.0%) | 2 (20.0%) | 4 (20.0%) |
| Atheist | 1 (10.0%) | 2 (20.0%) | 3 (15.0%) |
| Catholic | 1 (10.0%) | 1 (10.0%) | 2 (10.0%) |
| Other/missing | 3 (30.0%) | 4 (40.0%) | 7 (35.0%) |
| Employment Status | |||
| Employed, full time | 0 (0.0%) | 4 (40.0%) | 4 (20.0%) |
| Employed, part time | 3 (30.0%) | 2 (20.0%) | 5 (25.0%) |
| Not employed | 7 (70.0%) | 4 (40.0%) | 11 (55.0%) |
| Highest Education Level Completed | |||
| Graduated high school or equivalent | 1 (10.0%) | 1 (10.0%) | 2 (10.0%) |
| Some college | 2 (20.0%) | 3 (30.0%) | 5 (25.0%) |
| Graduated college | 3 (30.0%) | 4 (40.0%) | 7 (35.0%) |
| Completed post-graduate/professional degree | 4 (40.0%) | 2 (20.0%) | 6 (30.0%) |
| Total Household Income (Annual) | |||
| Less than $21,000 | 1 (10.0%) | 0 (0.0%) | 1 (5.0%) |
| $21,000 - $39,999 | 0 (0.0%) | 1 (10.0%) | 1 (5.0%) |
| $40,000 - $65,999 | 1 (10.0%) | 2 (20.0%) | 3 (15.0%) |
| $66,000 - $105,999 | 2 (20.0%) | 0 (0.0%) | 2 (10.0%) |
| $106,000 or more | 6 (60.0%) | 7 (70.0%) | 13 (65.0%) |
| Refused/ don’t know | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Insurance Status (Loved One’s Insurance Status) | |||
| Insured | 10 (100.0%) | 10 (100.0%) | 20 (100.0%) |
| Not insured | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Primary Informal Caregiver Relationship | |||
| Spouse/Partner | 9 (90.0%) | - | - |
| Son or Daughter | 1 (10.0%) | ||
Table 2:
Clinical characteristics of patients (N=10).
| Cancer type/site | N (%) |
|---|---|
| Lung | 6 (60.0%) |
| Head and neck | 1 (10.0%) |
| Other | 3 (30.0%) |
| Current cancer stage | N (%) |
| Stage I | 0 (0.0%) |
| Stage II | 0 (0.0%) |
| Stage III | 0 (0.0%) |
| Stage IV | 8 (80.0%) |
| Not in medical record | 2 (20.0%) |
| Current chemotherapy | N (%) |
| Yes | 9 (90.0%) |
| No | 1 (10.0%) |
| Chemotherapy lines completed | N (%) |
| 1 line | 1 (10.0%) |
| 2 lines | 1 (10.0%) |
| 3+ lines | 7 (70.0%) |
| Don’t know/missing | 1 (10.0%) |
| Currently taking narcotic pain medications | N (%) |
| Yes | 0 (0.0%) |
| No | 10 (10.0%) |
| Current radiation therapy | N (%) |
| Yes | 0 (0.0%) |
| No | 10 (10.0%) |
| Current immunotherapy | N (%) |
| Yes | 3 (30.0%) |
| No | 7 (70.0%) |
| Current targeted therapy | N (%) |
| Yes | 2 (20.0%) |
| No | 8 (80.0%) |
| Comorbidity Conditions** | |
| Diabetes | 1 (10.0%) |
| Diabetes with end organ damage | 1 (10.0%) |
| Any tumor | 10 (100.0%) |
| Metastatic solid tumor | 10 (100.0%) |
| Metastasis | N (%) |
| Yes, metastasis indicated | 10 (100.0%) |
| No, metastasis not indicated | 0 (0.0%) |
| Mean (SD) | |
| Karnofsky Score * | 8.3 (0.58) |
| ECOG Score | 0.9 (0.88) |
Only N=3 patients had Karnofsky score in their medical chart, so this value represents only those individuals.
Because individuals could have more than one comorbidity, column percentages sum to >100%.
Quantitative Findings
Acceptability and usability ratings included patients and caregivers (see Table 3), with comparison between patients and caregivers (see Figure 2).
Table 3.
Acceptability and usability data from patients (N=10) and caregivers (N=10).
| Patients (N=10) | Caregivers (N=10) | Overall (N=20) | |
|---|---|---|---|
| Acceptability E-Scale [Total possible score range: 6–30] | |||
| Range | 17–30 | 17–29 | 17–30 |
| Mean (SD) | 24.7 (4.35) | 22.9 (4.07) | 23.8 (4.20) |
| SUS Scale [Total possible score range: 0–100] | |||
| Range | 65.0–87.5 | 57.5–82.5 | 57.5–87.5 |
| Mean (SD) | 73.8 (6.15) | 70.0 (8.42) | 71.9 (7.43) |
| Other Acceptability Questions | |||
| Not helpful at all (1) – Very helpful (5) | |||
| Range | 3–5 | 3–5 | 3–5 |
| Mean (SD) | 4.0 (0.94) | 4.1 (0.74) | 4.1 (0.83) |
| Not difficult at all (1) – Very difficult (5) (A score >1 prompted the following 4 sub-questions) | |||
| Range | 1–3 | 1–3 | 1–3 |
| Mean (SD) | 1.4 (0.70) | 1.3 (0.67) | 1.4 (0.67) |
| Individual scores >1 | 3 (30.0%) | 2 (20.0%) | 5 (25.0%) |
| Was the website difficult because the wording was too complicated? | |||
| No | 3 (100.0%) | 2 (100.0%) | 5 (100.0%) |
| Was the website difficult because the ideas were too complicated? | |||
| No | 3 (100.0%) | 1 (50.0%) | 4 (80.0%) |
| Was the website difficult because there was too much information? | |||
| No | 3 (100.0%) | 1 (50.0%) | 4 (80.0%) |
| Was the website difficult because there was not enough information? | |||
| No | 0 (0.0%) | 1 (50.0%) | 1 (20.0%) |
| Overall, the website has 9 sections to complete. Do you think that is: | |||
| An acceptable number of sections/tasks | 10 (100.0%) | 10 (100.0%) | 20 (100.0%) |
| Regarding the amount of information on the website, do you think there is: | |||
| The right amount of information | 8 (80.0%) | 5 (50.0%) | 13 (65.0%) |
| Not at all satisfied (1) – Extremely satisfied (10) | |||
| Range | 5–9 | 6–9 | 5–9 |
| Mean (SD) | 7.8 (1.75) | 7.3 (1.16) | 7.6 (1.47) |
Figure 2. Acceptability and Usability Ratings from Patients, Caregivers, and Overall.
For the Acceptability E-Scale, a total score of ≥24 is considered acceptable for use and is used as the cutoff in this study. For the System Usability Scale, a score ≥68 is considered above average usability and is used as the cutoff in this study.
Patients.
A priori hypotheses were met for both acceptability and usability, with a mean score of 24.7 (SD=4.35) on the Acceptability E-Scale and a mean score of 73.8 (SD=6.15) on the System Usability Scale (see Table 3 and Figure 2). Overall, patients found the website very helpful (mean score of 4.0 out of 5) and reported high levels of satisfaction (mean score of 7.8 out of 10).
Caregivers.
Acceptability was just below the cutoff at a mean of 22.9 (SD=4.07) and usability exceeded the cutoff with a mean score of 70.0 (SD=8.42) on the System Usability Scale (see Table 3 and Figure 2). Overall, caregivers found the website very helpful (mean score of 4.1 out of 5) and reported high levels of satisfaction (mean score of 7.3 out of 10).
Qualitative Findings
Patients and caregivers provided their input about positive and negative aspects of the website and suggestions for changes (see Table 4). Key themes were organized by ‘site navigation’, ‘content’, and ‘design’, as reported by both patients and caregivers, patients, and caregivers.
Table 4.
Feedback from Patients and Caregivers
| Patients | Caregivers | Overall/combined | |
|---|---|---|---|
| Positive |
Site navigation
|
Site navigation
|
Site navigation
|
| Negative |
Site navigation
|
Site navigation
|
Site navigation
|
| Suggestions for changes |
Site navigation
|
Site navigation
|
Site navigation
|
All participants reviewed the Assign a Deputy and Personal Information sections of the website. Each of the remaining sections were reviewed by two patient-caregiver dyads.
Overall/combined (shared feedback from patients and caregivers).
For site navigation, they reported that the website was intuitive, and the organization of the dashboard was user friendly. They identified a bug, noting that some of the branching logic was not working properly. For content, they reported there being helpful information that people should be aware of regarding end of life planning. They liked that it included links to resources, information, and embedded tools on the webpage, and they noted that it was helpful to have all personal and financial information in one place. Additionally, they noted that the website was helpful for organizing and storing information. However, they reported privacy concerns (e.g. providing social security numbers), and indicated more information could be provided on why sensitive information is being collected, how it will be used, if it is optional, and how it will be protected. They also shared some confusion about how some sections of the website worked. They recommended providing a list of documents upfront to guide their process through the website better. For design, they described it as calming and peaceful (e.g. colors, fonts, and pictures are not harsh or loud). Suggestions included improving text legibility including increasing the font size and making it consistent across the website. They also noted that more color or visual interest could be added to the website. Suggestions for changes to site navigation included improving the menu (e.g., the ability to click into specific sections of the website, add back button).
Patients.
For site navigation, some reported that it was difficult to navigate with multiple scroll bars. Additionally, patients reported difficulty with signing into the website. For content, patients reported confusion about terms used (e.g. difference between will and trust). Patients recommended including prompts to check periodically that information is up-to-date and more instructions about which documents should be completed. For design, they reported that they did not like the wording of the front page. Patients noted that the site did not feel personal.
Caregivers.
For site navigation, caregivers said that it was not clear that you could skip over sections and that some sections did not show up as expected. For content, caregivers indicated certain website features may not be applicable to everyone. They liked the embedded tools and resources, such as the ‘find a therapist’ tool. They suggested assuring the patient that they do not need to fill out all sections of the website in one sitting. For design, caregivers reported that the text boxes were too small. Suggestions for changes to site navigation included reordering some sections and using branching logic to cut down on questions. They also suggested providing an ‘up next’ description.
Discussion
The findings of this study will inform modifications to optimize an innovative website to support patients with advanced cancer prepare holistically for the end of life. Patient ratings of the web-based platform met a priori hypotheses for acceptability and usability. For caregivers, usability exceeded the cutoff and acceptability of the website was just below the cutoff. Both patients and caregivers noted privacy concerns with sharing sensitive information (e.g. social security number) via the website and wanted more information about information security and protection in a next iteration.
Our findings highlight the nuanced nature of intervention development in the context of patients with advanced illness. There remains a gap in digital health interventions targeting patients with advanced cancer.21 There were differing opinions among caregivers about the amount of information provided on the website. The variability in preferences, such as for mode of delivery and frequency of use, advocate for building in flexibility to intervention design and delivery such as options to skip sections and to complete individually or with a caregiver. There is emerging research studying patient-facing ACP approaches, including a mixed-methods comparative effectiveness trial currently underway of in-person ACP discussions with trained facilitators versus web-based ACP using interactive videos.22
Clinically, this tool may help to reduce burden on patients and caregivers in preparing holistically for the end of life. When a loved one dies, it takes upwards of 500 hours to settle their affairs.23,24 This does not account for the countless hours spent actually grieving a loss and the reality that grief in fact never fully dissipates – with a peak in grief at six months after death.25 Engagement in advance care planning may prompt the beginning of the grieving process in patients and may help to promote grief resolution in caregivers.26 Inclusion of embedded resources for emotional support is essential, such as a ‘find a therapist’ tool and flexibility to engage in end of life planning tasks on the website at your own pace. Further work is needed to address the role of grief in end-of-life preparation among patients with advanced illness and their loved ones.
Scalable implementation of digital health programs requires cross-sector partnerships and multidisciplinary collaborations.27 Our partnership with a tech company represents a strategic alliance to accelerate intervention development for patients with advanced cancer. By providing patient and caregiver feedback after each interview, the tech developers were able to revise the website in real-time.
While delivery of patient-centered palliative care is recommended, the evidence for technology-mediated, patient-centered approaches to ACP remains limited.28–30 This study contributes new knowledge about end-of-life planning for patients with advanced cancer and cross-sector partnerships to accelerate digital health interventions in this understudied population. In future research and next steps, we will evaluate the preliminary efficacy of the finalized version of the website with patients with advanced cancer to improve engagement in end-of-life planning. Limitations of this study include a small sample size that is primarily White, female, and insured. As such, further research is needed with underserved populations.
Conclusion
Our comprehensive end-of-life planning website is feasible and acceptable to patients and caregivers with advanced cancer. This study contributes to the evidence-base for intervention development and testing with this understudied population.
Acknowledgments
We sincerely appreciate the patients and caregivers who participated in this study. This research was supported by grants from the National Cancer Institute to Megan J Shen, PhD, P30 CA015704, K07207580, and R44 NR020341. REDCap at ITHS is supported by the National Center For Advancing Translational Sciences of the National Institutes of Health under Award Number UL1 TR002319.
Funding statement:
This research was supported by grants from the National Cancer Institute to Megan J Shen, PhD, K07CA207580, P30 CA015704 and R44 NR020341. REDCap at ITHS is supported by the National Center For Advancing Translational Sciences of the National Institutes of Health under Award Number UL1 TR002319.
Footnotes
Ethics approval: This study was approved by the Fred Hutchinson Cancer Center Institutional Review Board.
Patient consent: Informed written consent was obtained from all study participants.
Conflict of interest statement: JG has a conflict of interest as the CEO/Founder of the tech company developing this website.
Data availability:
Data is available upon reasonable request to the principal investigator.
References
- 1.Network NCC. NCCN Guidelines. April 28, 2023. https://www.nccn.org/guidelines/category_1
- 2.Smith TJ, Temin S, Alesi ER, et al. American Society of Clinical Oncology provisional clinical opinion: the integration of palliative care into standard oncology care. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. Mar 10 2012;30(8):880–7. doi: 10.1200/jco.2011.38.5161 [DOI] [PubMed] [Google Scholar]
- 3.Services CfMaM. Oncology Care Model (OCM) Second Annual Evaluation Report and next steps on Oncology Innovation. 2023. https://innovation.cms.gov/data-and-reports/2020/cms-perspective-ocm
- 4.Schubart JR, Levi BH, Bain MM, Farace E, Green MJ. Advance Care Planning Among Patients With Advanced Cancer. J Oncol Pract. Jan 2019;15(1):e65–e73. doi: 10.1200/jop.18.00044 [DOI] [PubMed] [Google Scholar]
- 5.Sudore RL, Schillinger D, Katen MT, et al. Engaging Diverse English- and Spanish-Speaking Older Adults in Advance Care Planning: The PREPARE Randomized Clinical Trial. JAMA Intern Med. Dec 1 2018;178(12):1616–1625. doi: 10.1001/jamainternmed.2018.4657 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Auriemma CL, Halpern SD, Asch JM, Van Der Tuyn M, Asch DA. Completion of Advance Directives and Documented Care Preferences During the Coronavirus Disease 2019 (COVID-19) Pandemic. JAMA Network Open. 2020;3(7):e2015762–e2015762. doi: 10.1001/jamanetworkopen.2020.15762 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Heyland DK, Dodek P, Rocker G, et al. What matters most in end-of-life care: perceptions of seriously ill patients and their family members. Cmaj. Feb 28 2006;174(5):627–33. doi: 10.1503/cmaj.050626 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Steinhauser KE, Christakis NA, Clipp EC, et al. Preparing for the end of life: preferences of patients, families, physicians, and other care providers. J Pain Symptom Manage. Sep 2001;22(3):727–37. doi: 10.1016/s0885-3924(01)00334-7 [DOI] [PubMed] [Google Scholar]
- 9.Payne HE, Lister C, West JH, Bernhardt JM. Behavioral Functionality of Mobile Apps in Health Interventions: A Systematic Review of the Literature. JMIR mHealth and uHealth. 2015/02/26 2015;3(1):e20. doi: 10.2196/mhealth.3335 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Tariman JD, Berry DL, Halpenny B, Wolpin S, Schepp K. Validation and testing of the Acceptability E-scale for web-based patient-reported outcomes in cancer care. Applied nursing research : ANR. Feb 2011;24(1):53–8. doi: 10.1016/j.apnr.2009.04.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Usability.gov. System Usability Scale. https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
- 12.Lunenburg F Goal-setting theory of motivation. International Journal of Management, Business, and Administration. 2011;15(1):1–6. [Google Scholar]
- 13.Pfeiffer E A short portable mental status questionnaire for the assessment of organic brain deficit in elderly patients. J Am Geriatr Soc. Oct 1975;23(10):433–41. doi: 10.1111/j.1532-5415.1975.tb00927.x [DOI] [PubMed] [Google Scholar]
- 14.Sandelowski M What’s in a name? Qualitative description revisited. Research in Nursing & Health. 2009:n/a–n/a. doi: 10.1002/nur.20362 [DOI] [PubMed] [Google Scholar]
- 15.Sandelowski M Whatever happened to qualitative description? Research in nursing & health. Aug 2000;23(4):334–40. [DOI] [PubMed] [Google Scholar]
- 16.Sullivan-Bolyai S, Bova C, Harper D. Developing and refining interventions in persons with health disparities: the use of qualitative description. Nursing outlook. May-Jun 2005;53(3):127–33. doi: 10.1016/j.outlook.2005.03.005 [DOI] [PubMed] [Google Scholar]
- 17.Usability.gov. System Usability Scale (SUS). Accessed 21 March, 2018. https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
- 18.Bangor A, Kortum P, Miller J. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of Usability Studies. May 2009. 2009;4(3):114–123. [Google Scholar]
- 19.Brooke J SUS: A Retrospective. Journal of Usability Studies. February 2013. 2013;8(2):29–40. [Google Scholar]
- 20.Sauro J Measuring Usability with the System Usability Scale. 22 March, 2018. Accessed 22 March, 2018. https://measuringu.com/sus/
- 21.Lee K, Kim S, Kim SH, et al. Digital Health Interventions for Adult Patients With Cancer Evaluated in Randomized Controlled Trials: Scoping Review. J Med Internet Res. Jan 6 2023;25:e38333. doi: 10.2196/38333 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Resick JM, Arnold RM, Sudore RL, et al. Patient-centered and efficacious advance care planning in cancer: Protocol and key design considerations for the PEACe-compare trial. Contemporary clinical trials. Sep 2020;96:106071. doi: 10.1016/j.cct.2020.106071 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Shen M How the science of grief can inform better bereavement leave in the U.S. Medium. https://medium.com/@meganjohnsonshen/how-the-science-of-grief-can-inform-better-bereavement-leave-in-the-u-s-3d7a1876ec83
- 24.Cuadra D Why bereavement leave isn’t enough: What employees need after losing a loved one. ebn. https://www.benefitnews.com/news/employer-bereavement-policies-may-need-to-change
- 25.Barry E How Long Should It Take to Grieve? Psychiatry Has Come Up With an Answer. https://www.nytimes.com/2022/03/18/health/prolonged-grief-disorder.html
- 26.Falzarano F, Prigerson HG, Maciejewski PK. The Role of Advance Care Planning in Cancer Patient and Caregiver Grief Resolution: Helpful or Harmful? Cancers (Basel). Apr 20 2021;13(8)doi: 10.3390/cancers13081977 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Sucala M, Cole-Lewis H, Arigo D, et al. Behavior science in the evolving world of digital health: considerations on anticipated opportunities and challenges. Transl Behav Med. Mar 16 2021;11(2):495–503. doi: 10.1093/tbm/ibaa034 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Allsop MJ, Chumbley K, Birtwistle J, Bennett MI, Pocock L. Building on sand: digital technologies for care coordination and advance care planning. BMJ Support Palliat Care. Jun 2022;12(2):194–197. doi: 10.1136/bmjspcare-2021-003304 [DOI] [PubMed] [Google Scholar]
- 29.Birtwistle J, Millares-Martin P, Evans CJ, et al. Mapping and characterising electronic palliative care coordination systems and their intended impact: A national survey of end-of-life care commissioners. PLoS One. 2022;17(10):e0275991. doi: 10.1371/journal.pone.0275991 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Temel JS, Petrillo LA, Greer JA. Patient-Centered Palliative Care for Patients With Advanced Lung Cancer. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. Feb 20 2022;40(6):626–634. doi: 10.1200/jco.21.01710 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Data is available upon reasonable request to the principal investigator.