App-based assessment of patient-reported outcomes in the Molecular Tumor Board in the Center for Personalized Medicine—(TRACE)

Abstract

Background

Biomarker-based therapies are increasingly used in cancer patients outside clinical trials. Systematic assessment of patient-reported outcomes (PRO) is warranted to take patients’ perspectives during biomarker-based therapies into consideration. We assessed the feasibility of an electronic PRO assessment via a smartphone application.

Methods

An interdisciplinary expert panel developed a smartphone application based on symptom burden and health-related quality of life (HRQoL) metrics reported in a retrospective analysis of 292 neuro-oncological patients. The app included validated assessments of health-related quality of life (HRQoL), the burden of symptoms, and psychological stress. Feasibility and usability were tested in a pilot study. Semi-structured interviews with patients and health care professionals (HCP) were conducted, transcribed, and analyzed according to Mayring´s qualitative content analysis. Furthermore, we assessed compliance and descriptive data of ePROs.

Results

A total of 14 patients have been enrolled, (9 female, 5 male). A total of 4 HCPs, 9 patients, and 1 caregiver were interviewed regarding usability/feasibility. The main advantages were the possibility to complete questionnaires at home and comfortable implementation in daily life. Compliance was high, for example, 82% of the weekly distributed NCCN distress thermometer questionnaires were answered on time, however, with interindividual variability. We observed a median distress score of 5 (range 0–10, 197 results, n = 12, weekly assessed) and a median Global health score of 58.3 according to the EORTC QLQ-C30 instrument (range 16.7–100, 77 results, n = 12, monthly assessed).

Conclusions

This pilot study proved the feasibility and acceptance of the app. We will therefore expand its application during biomarker-guided therapies to enable systematic PRO assessments.

Biomarker-based therapies for cancer patients are increasingly used outside of clinical trials. However, the outcome evaluation is challenging. The most frequently applied method is a progression-free survival (PFS) ratio, where, in each patient, PFS on biomarker-based therapy (PFS2) is compared to PFS on the immediate prior systemic therapy (PFS1). A PFS2/PFS1 ratio of ≥1.3 indicates a favorable response to the biomarker-based therapy.¹ We previously reported a PFS2/PFS1 ratio >1.3 in 31.3% of neuro-oncology patients treated at the Center for Personalized Medicine Tübingen.²

Besides PFS as a clinical outcome measure, it is of high importance to gain insight into the subjective experiences of patients treated with biomarker-based therapy.³ So far, assessment of the patient’s perception regarding symptoms, side effects, and psychosocial burden has not been incorporated in the evaluation of biomarker-based therapy. Physicians might underestimate the severity of symptoms and side effects, however. A better involvement of the patient could enhance the precision and reliability of this subjective data and therefore would optimize the efficacy of clinical care.⁴ Collecting information directly from patients might improve the accuracy of the data.⁵ Considering the heterogeneity of the patients in molecular tumor boards, routine systematic assessment of patient-reported outcomes (PRO) is challenging due to variations in disease and/or treatment-associated symptoms, side effects, and impact on HRQol in different tumor types and due to the variety of prescribed drugs. At the same time, these patients often have advanced disease, are highly burdened by symptoms and frequent QoL assessment might be urgently needed.⁶ There is increasing interest in the use of electronic patient-reported outcome (ePRO) measures as a tool in the care of oncologic patients.⁷ ePROs can collect real-time data about the patients’ general condition using digital surveys and, therefore, can enable fast and personalized monitoring strategy for various domains, such as symptoms, side effects of therapies, every-day functioning, and/or health-related quality of life (HRQoL) and they can be answered in a comfortable environment, for example, at home.

Digital tools such as electronic health applications enable the collection of information under real world conditions.^8,9 Therefore, we aimed to develop an application for patients in biomarker-based therapy and included patients’ perspectives in the development. Although prior testing of eHealth systems is recommended,^10,11 usability evaluation has been an under-represented topic in publications. At the same time, it is important to involve patients and healthcare professionals (HCP) in such projects to consider the users’ perspective.¹²

In the present study, we evaluated a smartphone application (app) called “TRACE” (App-based assessment of patient-reported outcomes in the Molecular Tumor Board in the Center for Personalized Medicine) for the systematic assessment of PROs and physical symptoms in patients treated with biomarker-based therapies at the Center for Personalized Medicine.

Materials and Methods

Study Design

The study aimed to explore the feasibility and usability of a smartphone app to collect ePROs from patients on biomarker-based therapy. The main areas of interest in this pilot study were the patients’ experience with the app and the study team members’ experience regarding feasibility as well as usability of the TRACE app as they introduced the patients to the app and supported them subsequently.

This was a single-arm center pilot study conducted at our Center for Personalized Medicine and the referring departments from 12/2021 to 12/2022.

The main outcome was feasibility and usability evaluated by an interview. Furthermore, feasibility was defined to be proven if 70% of the participating patients used the app for  ≥ 6 months (end of study).

All procedures performed in studies involving human participants were in accordance by the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The study design was approved by the ethics committee (Ethics Committee of the University Hospital Tübingen, No. 1016/2020BO1) and was conducted within the prospective observational molecular tumor board study (MTB@ZPM, NCT03503149) (2).

Retrospective Data Analysis and Development of the App

Based on retrospective assessments of symptom burden in 292 patients with a neuro-oncological diagnosis (01/2018–12/2019) enrolled in MTB@ZPM, we developed the TRACE app (Cancer patients under targeted therapy: App-based assessment of patient-reported outcomes) in an interdisciplinary expert panel.

According to the observed symptom profile, the panel evaluated validated instruments to measure health-related quality of life (HRQoL), symptom burden, and psychological distress for their utility as part of the app. The development process included several expert discussions and the implementation of an electronic version of the developed and validated questionnaires regarding HRQoL, distress, activity, and fatigue (section “Study Instruments and Implemented ePROs”) in the TRACE app.

Course of the Prospective Study

Patients were invited to participate in this study at the time of initiation of the respective biomarker-based therapy recommended for their disease. They were invited to participate in person within their routine consultation and were informed about the study procedures including a semi-structured interview. Written and informed consent was obtained from all participants. They were instructed to download the TRACE app on their smartphone (details are provided in Supplementary Figure 1). The ePRO system then automatically generated an ID, which served as a pseudonym in the study and was communicated to the study team so that no identifying personal data was included in the ePRO system.

All Data Were Stored in a Secured Server Space, Accessible Only for Study Personnel

The ePRO system assigned questionnaires to the IDs combined with push notifications on the patients’ smartphones. Questionnaires were available until the next scheduled assessment and patients received notifications as soon as a new questionnaire was due. Furthermore, study team members were informed of patients who had not completed questionnaires within the scheduled time period, so that they could remind patients by phone or e-mail to answer their assigned questionnaires and/or resolve any technical problems.

Study visits at our center were scheduled at baseline (V1) and after 3 (V2), and 6 (V3) months. At V1, the patients’ history, socio-demographic and clinical data were assessed. At V2, we conducted a semi-structured interview with the patients but also with the HCPs, if they volunteered, to explore additional potentially relevant content that was not incorporated in the current version as well as satisfaction with using the app. At all study visits (V1–V3), clinical status, patients’ and caregivers’ satisfaction were assessed exploratively. The end of the study was reached after app usage over 6 months.

Participants of the Prospective Study Testing the App

Adult (age >18 years) cancer patients who were started on a new biomarker-based therapy according to the MTB’s recommendation were eligible to participate in this study. They were approached in the outpatient department and provided written and informed consent. Participants were required to have access to the internet via a smartphone and they had to be able to handle a smartphone and/or willing to be supported by their caregiver. In 1 patient, the interview was conducted with the caregiver instead of the patient. The caregiver was approached in the outpatient clinic and gave written informed consent.

Furthermore, HCPs, who were members of the study team, were invited to participate in the semi-structured interview. They introduced the app to the patients and provided support in case of any questions. Before they were interviewed, they tested the app by themselves.

Study Instruments and Implemented ePROs

At baseline, data on medical history and socio-demographics were collected. The following PRO questionnaires were included in the app

• HRQoL: European Organisation of Research and Treatment of Cancer (EORTC) questionnaires¹³ with an assessment of physical, role, cognitive, emotional, and social functioning as well as symptom scales of fatigue, pain, nausea/vomiting, dyspnea, insomnia, appetite loss, constipation, diarrhea, and financial difficulties (query interval: 1 month).

• EORTC disease specific modules according to the diagnosis of each patient, in order to apply the app in all patients of the Center for Personalized Medicine, eg, patients with a neuro-oncological diagnosis additionally completed the brain module (BN20)¹⁴ (query interval: 1 month).

• Psychosocial burden: NCCN Distress Thermometer^15–17 accompanied by the problem list (query interval: 1 week).

• Fatigue: Brief Fatigue Inventory (BFI)¹⁸ (query interval: quarterly).

• Activity: European Quality of Life 5 Dimensions (EQ-5D)¹⁹ and Description of Work Productivity and Activity Impairment (WPAI) Questionnaire: General Health (WPAI:GH)²⁰ (query interval: quarterly).

Semi-Structured Interview

The interview guide was developed by the study team consisting of (neuro)oncologists, study nurses, informaticians/computer scientists, health scientists, and the app developer in a consensus process using a deductive and inductive approach. Feasibility was defined as the ability to handle the app (with or without support from caregivers) with regards to the patient-specific overall situation and usability was defined as how user-friendly the app was for patients, caregivers, and study team members, for example, how often any user problems occurred and if any technical or design issues were detected.^21,22

Questions were developed to capture the patients’ and HCPs’ perspectives as freely as possible. The aim was to identify any additional challenges regarding the electronic assessment, handling, and feasibility but also regarding content-related issues from the users’ perspective. The interviews were conducted by SV and LD, who were specifically trained by MH and MR, using a self-developed, semi-structured interview guide. The interviewers were not involved in the clinical care of the participants and therefore were introduced to the participants before the first encounter. The interviews were conducted in a separate room and in absence of any other participants. Study subjects were interviewed face-to-face and in German language. The interview was recorded. Subsequently, the interviews were transcribed and analyzed by M.H., S.V., and M.R. The interview questions for the patients are presented in Table 1 (additional questions developed for caregivers and HCPs are provided in Supplementary Material).

Table 1.

Semi-Structured Interview Guide

0. Opening question
0.1 What is your opinion on completing questionnaires regarding your well-being on your smartphone instead of on paper?
1. Usability
1.1 How user-friendly was the app for you?
1.2 What aspects were particularly cumbersome/problematic? What was easy and what aspects did you like about the app?
1.3 How time-consuming was using the app for you? How much time did you require every day to complete the questionnaire through the app?
1.4 How did you experience navigating through the app?
1.5 What changes/adaptations would you propose to make the app more user-friendly?
2. Suitability for daily use
2.1 Which difficulties did you experience? What did you do in case of difficulties? If support was needed: How did that influence your responses using the app?
2.2 How was it for you to respond to the questions in the presence of other people? (Why were you uncomfortable using the app in the presence of others?)
2.3 Did you think the time needed to use the app/complete the questions was too long, too short, or just right? How did it correlate with your expectations when you enrolled in this study?
2.4 How easy was it to integrate the use of the app into your daily life? And why? Please rate on a scale of 0–10 (0 is not at all easy, 10 is very easy).
3. Personal perception
3.1 Has your mood changed before/during/after using the app? If so, how?
3.2 Did the use of the app affect your mental or physical symptoms? If so, how did this manifest?
3.3 Do you think that the use of the app influenced your relationship with your caregivers? If so, how?
3.4 Did the use of the app make you feel more assured/ more secure/more confident in regards to tumor-directed therapies?
3.5 On a scale of 0–10, to which extent was the daily use of the app emotionally burdening to you? (0 not at all burdening, 10 very burdening)
4. Conclusion
4.1 What additional suggestions do you have for us?

Data Analysis

For the retrospective part of this study, data regarding neurological symptoms and HRQoL were obtained from the available medical records and analyzed using frequency counts. The notes from the transcribed interviews were sorted into usability and feasibility domains and main and subordinate contents were defined according to Mayring.^23–25 The results were summarized using frequency counts. The qualitative data are reported in accordance by the Consolidated criteria for Reporting Qualitative research (COREQ) guideline.²⁶ Quantitative data regarding compliance with the assessment as well as patient-reported outcome results were analyzed descriptively. Descriptive statistics, such as median, minimum, maximum, absolute, and relative frequencies were used to analyze quantitative data using SPSS Version 28.0.0.0 (190).

Results

Symptom Burden and HRQoL Profiles in the Retrospective Study Cohort

In the retrospective analysis, we included 290 patients with a neuro-oncological diagnosis, who underwent comprehensive molecular profiling in the Center of Personalized Medicine. A total of 149 (51.4%) were male, the median age was 51 years (18–83), n = 41 lived alone (14.1%) and the majority had a glioblastoma (72/290, 24.8%). Details are provided in Figure 1 and Supplementary Table 1.

Figure 1.

Tumor types in the retrospective and prospective patient cohort. The retrospective cohort included solely neuro-oncology patients and the prospective cohort other cancer diagnoses.

Neurological symptoms and deficits were observed in 219/290 patients (75.5%). Reported general symptoms included declining general condition (73/290, 25.2%), headache, or other pain (104/290, 35.9%).

The psychosocial burden reported comprised of anxiety (67/290, 23.1%), depressive symptoms (45/292, 15.5%), and sleep disturbances (29/290, 10%). We detected also burden of caregivers (40/290, 13.8%).

According to the obtained symptom profile, validated instruments focusing on cancer patients’ symptoms, fatigue, daily activity, unmet needs and distress were selected and included in the app.

Participants of the Prospective Pilot Study

A total of 14 patients were enrolled between 10/2021 and 10/2022, 5 were male and nine were female. Twelve patients started using the app, data cut off was on 31-DEC-2022, 2 patients did not start using the app, 1 due to disease progression and 1 patient was no longer interested in the study. The majority of enrolled patients (8/12) had a neuro-oncology tumor diagnosis. Details are provided in Figure 1 and Table 2.

Table 2.

Patients Enrolled in the Feasibility Study

Pat No.	Gender	Age (years)	Diagnosis	Time since initial diagnosis (months)	Molecular target	Targeted therapy	Time period targeted therapy (months)	Number of prior targeted therapies (months)	Initial ECOG	Initial KPS	Months TRACE app used	End of study reached (usage app ≥ 6 months)
01	Female	57	Gallbladder carcinoma	26.1	FGFR2— KIIAA1598	FGFR inhibitor (Pemigatinib)	2	0	0	90	7.2	Yes
02	Female	34	Spinal ependymoma, CNS WHO grade 3	9.6	NF2	mTOR inhibitor (Everolimus)	ONG	0	2	70	5.1**	Yes
03	Female	61	Melanoma	72.5	CDKN2A, CDK4, CDK6, CCND2	CDK4/6 Inhibitor (Palbociclib)	1	1 (4)	1	80	11.9	Yes
04	Male	33	Diffuse midline glioma, CNS WHO grade 4	30.1	NF1	MEK inhibitor (Trametinib)	ONG	1 (7.1)	1	90	7.0	Yes
05	Female	48	High-grade astrocytoma with piloid features	16.7	NF1	MEK inhibitor (Trametinib, Selumetinib)	ONG	1 (6.5)	2	60	8.1	Yes
06	Male	33	Glioblastoma, IDH wildtype, CNS WHO grade 4	90.6	EGFR	Tyrosine kinase inhibitor (Afatinib)	1	0	2	60	1.1	No, due to disease progression
07	Female	61	Gallbladder carcinoma	30.6	FGFR2—BICC1	FGFR inhibitor (Pemigatinib, Lenvatinib)	ONG	1 (16.7)	1	80	13.3	Yes
08	Female	34	Pineal parenchymal tumor of intermediate differentiation, CNS WHO grade 2	273.3	DICER1	Nortiptyllin/ Gemcitabine	2	0	1	70	1.1	No, due to disease pro-gression
09	Male	28	Astrocytoma, IDH-mutant, CNS WHO grade 2	73	IDH1	IDH inhibitor (Ivosidenib)	4.6	0	1	80	5.3**	Yes
10	Female	70	Gallbladder carcinoma	42.4	FGFR2—RBFOX2	FGFR inhibitor (Pemigatinib)	ONG	0	1	80	7.2	Yes
11	Male	55	Meningioma, CNS WHO grade 2	209	NF2	mTOR (Everolimus)	ONG	0	1	70	2.6	No, due to clinical deterioration1
12	Female	32	Papillary tumor of the pineal region WHO grade 2	48.4	NF2, PTEN	mTOR inhibitor (Everolimus)	ONG	0	0	90	6.3	Yes
13	Female	67	Oligodendroglioma, CNS WHO grade 2	130.8	IDH1	IDH inhibitor (Ivosidenib)	n/a	0	2	60	0	Disease pro-gression
14	Male	59	Meningioma, CNS WHO grade 2	92.2	NF2	mTOR inhibitor (Everolimus)	n/a	0	1	80	0	No longer interested

¹The caregiver of the patient was interviewed.

Gray-shaded fields indicate patients (2) who gave informed consent to the study but did not start using the app.

Abbreviations: KPS = Karnofsky Performance Status, ONG = ongoing.

^**The patient started the app in July 2022, due to data cut-off in 12/2022, the 6 months were not reached.

In total, 9 patients and 1 caregiver were interviewed regarding usability and feasibility, 9 patients completed the end of study visit (V3) after 6 months. Drop-outs occurred due to disease progression or clinical deterioration (n = 3, in one of these patients the caregiver was interviewed instead of the patient). The mean duration of the interview with patients and the caregiver was 12 min (range 8–21 min). Four HCPs as members of the study team were interviewed (n = 2 employed as clinical research coordinators, n = 2 medical students), 2 males and 2 females, all of them used at least 1 mobile application and were involved in the introduction of the app to patients and support in case of any problems.

Feasibility and Usability

The development of the app involved 14 meetings of the interdisciplinary expert panel, and several rounds of pre-testing by the experts. We involved clinicians (oncologists, gynecologists, dermatologists, surgeons, radiation oncologists, and neurologists), software engineers and programmers, medicine students as well as nurses.

At V2 (after 3 months of usage) the patients were interviewed regarding usability and feasibility. According to the inductive/deductive approach, the main topics of the interview and at the same time main categories for analysis were “feasibility” and “usability” as defined above. The subcategories subsequently derived from patients’ answers were:

• handling

• design

• difficulties

• content of PRO questionnaires

• answering questions in the presence of others

• integration into daily life

• effects (eg, on mood) of answering the questions

• recommendations

An additional subcategory of the interview with the HCPs was “data collection.” Both, patients, and health care professionals evaluated the app as user-friendly, and patients felt no burden in filling out the questionnaires via the app. The main advantages of the app-based assessment were the option to complete questionnaires at home and the easy implementation in daily life, details of the subcategories and the quotations are provided in Table 3. One patient reported technical problems at the beginning. Furthermore, interviewees also reported that the questions were challenging at times, time-consuming, and questionnaire requests became too frequent for some patients.

Table 3:

Main Categories, Subcategories, and Translated Quotations of the Semi-Structured Interviews

Patients/caregiver
Usability	Main statement	Example quotations
Handling	The app is user-friendly.	[Usability of the app is good] „ because its use is self-explanatory. Because it is easy to move from one question to the next question” (Pat. No. 01, line 28-30)
Design	Clear structured, simple design	„The design is simple and straight-forward, easy to use for everyone.” (Pat. No. 03, line 30-31)
Content	Questions sometimes are challenging/more time-consuming	“Some questions are more difficult to answer, one needs to think longer about them. For example questions that require a “yes” or “no” response but which you cannot answer by a clear/definite “yes” or “no.”” (Pat. No. 01, lines 83–85)
Difficulties	Mainly no difficulties	“Well, sometimes it took some time for the app to load. The questions were not displayed correctly, one could only see the “TRACE” symbol. Only when trying it again after 2–3 days, did it work again. Maybe it’s due to my mobile phone.” (Pat. No. 09, lines 91–95)
Feasibility	Main statement	Example quotations
Answering in the presence of others	Rare problems with completing the questionnaires in the presence of others	“No big deal. Because I think this should be talked about more in the public, anyways. It’s still a taboo issue. Because the topic [cancer] is frequently taboo, still.” (Pat. No. 01, lines 68–70)
App vs. paper	Questionnaires via app are preferable, more modern, and timesaving	“It is easier. You don’t need direct contact [with the clinic] and you don’t have to go into the clinic to fill out the questionnaires.” (Pat. No. 12, lines 23–24)”
Integration into everyday life	Good integration into the daily routine	“You can just pick up your phone and operate the app. It wasn’t a big effort, and it was quick.” (Pat. No 01, lines 57–59)
Effects of answering the questions	No burden due to filling out the questionnaires	“Yes, I had a better overview of my health status and quality of life.” (Pat. No. 03, lines 74)
Health care professionals
Usability	Main statement	Example quotations
Handling	Simple layout of the app and self-explanatory questionnaires	“I experienced it as very self-explanatory. But that could also be due to my generation because such apps are easier and more intuitive for ‘digital natives’.” (HCP No. 1, lines 38–39)
Design	Mainly satisfied with the design of the app. Easy handling for the patients	“Well, the questionnaires are easy to find. But I think it would be good for patients to know the progress status (when which questionnaire has to be answered).” (HCP No. 3, lines 32–33)
Satisfaction	Strong satisfaction, as the app has made a lot of things easier for the staff. But less patient contact.	“I think the app fulfills its purpose. It also worked well on both operating systems, Android and IOS.” (HCP No. 1, lines 43–45)
Feasibility	Main statement	Example quotations
App vs. paper	Innovation in the field of digitalization of medicine. Easier for patients, questionnaires can be completed at any time.	“Most people have a smartphone. We save paper and the patients can do it from home and don’t have to come to the clinic.” (HCP No. 4, lines 25–30)
Implementation	Introducing the app into everyday life can occur without any problems.	“Without any problems. The only difficulty probably was the sometimes-instable internet connection.” (HCP No. 1, lines 29–30)
Data collection	Easy data collection, by use of mandatory fields, all items are completed and there is fewer missing data.	[the app is] “Feasible not only in oncology. Dependent on compliance and willingness of the patient. Dependent on the cooperation of other departments and disciplines.” (HCP No. 2, lines 57–59)
Integration into everyday life	Less effort for patients, the same amount of effort for staff.	”Another advantage is that you can do it at home while drinking coffee or reading the newspaper. I think it can be integrated well.” (HCP No. 4 lines 76–94)

HCP = health care professional.

The electronic data collection by use of mandatory fields was useful for health care professionals in order to avoid missing data in PROs.

Compliance

Compliance with questionnaires during study participation (V1–V3/V1-drop-out) was high, for example, 82% of weekly distributed NCCN Distress Thermometers were answered on time. However, we observed a high interindividual variance, evident by a wide time range of when assessments were completed (range 24–100%, assessment duration 4–56 weeks). We observed a total of 197 completed DT questionnaires during the study (V1–V3/V1-drop-out). Six patients wished to use the app beyond V3 because they were motivated to continue using the app. Therefore, a total of 297 answers were reported at the data cut off on 31-DEC-2022, however, only the data until V3 of each patient are considered in our analysis.

Descriptive ePRO Results

We observed a median Distress score of 5 (range 0–10, n = 12) in 197 completed questionnaires included in the analysis, assessed weekly. The median Global health score was 58.3 according to the EORTC QLQ-C30 instrument (range 16.7–100, 77 results, n = 12, monthly assessed, questions 29 and 30).

Patients reported various physical and emotional problems regardless of the median distress score and tumor entity. The most common physical problems were fatigue, constipation, appetite loss, mucosa irritation, and dry skin. All but 1 patient reported several emotional problems. The main reported practical problem was mobility. Patients reported problems with social contact. Details are provided in Table 4. As an example, the disease trajectory after initiation of the biomarker-based therapy is displayed in Figure 2.

Table 4.

Median Distress Score and Indicated Problems per Patient (According to NCCN Distress Thermometer)

Patient ID	DT score median (range)	Main problems* indicated on the problem list (frequency of indicated problem/all DT assessments per patient)
01	1 (0–7)	Appetite (7/15)
02	8 (3–10)	Childcare (9/15), financial problems (13/15), depression (9/15), anxiety (7/15), worries (14/15), loss of interest (12/15), appearance (6/15), constipation (13/15), fatigue (15/15), abilities of daily life (14/15), constipation (9/15), concentration (7/15), dry nose (14/15), pain (7/15), sexuality (12/15), sleeplessness (14/15)
03	8 (5–8)	Housing situation (8/25), mobility (11/25), depression (20/25), anxiety (14/25), sadness (14/25), worries (11/15), appearance (20/25), dyspnea (25/25); bladder control (16/25), appetite (17/25), fatigue (25/25), abilities of daily life (16/25), constipation (12/25), concentration (25/25), nausea (9/25), pain (11/25), skin (23/25), sleeplessness (23/25), tingling in hands and feet (13/25)
04	5 (4–8)	Mobility (14/22), anxiety (11/22), sadness (11/22), worries (21/22), constipation (9/22), diarrhea (13/22), fatigue (20/22); nausea (11/22), skin (21/22)
05	2 (1–3)	Mobility (19/22), work (7/22), fatigue (16/22), abilities of daily life (14/22)
06	3 (3–3)	Mobility (2/2), nervousness (1/2), loss of interest (1/2) body care (1/2), dyspnea (1/2) constipation (2/2), fatigue (1/2), feeling swollen (1/2) abilities of daily life (1/2), constipation (2/2), concentration (2/2), mucosa irritation (1/2), nausea (1/2), dry nose (2/2)
07	3 (0–9)	Loss of interest (16/21), constipation (10/21), fatigue (7/21), concentration (20/21), mucosa irritation (13/21), skin (10/21)
08	8 (6–8)	Childcare (3/5), mobility (3/5), work (3/5) social contacts (3/5), appearance (4/5), bladder control (2/5), concentration (3/5), mucosa irritation (3/5), dry nose (3/5), pain (4/5), sexuality (4/5)
09	7 (5–8)	Financial problems (23/24), anxiety (12/24), nervousness (9/24), worries (19/24), appearance (17/24), fatigue (24/24), abilities of daily life (8/24); mucosa irritation (11/24); pain (14/24), sleeplessness (23/24)
10	3 (2–6)	Mobility (11/24), sadness (10/24), worries (18/24), loss of interest (8/24), body care (10/24), constipation (14/24), fatigue (23/24), constipation (20/24), concentration (17/24), mucosa irritation (21/24), dry nose (22/24), pain (10/24)
11	4 (4–4)	Financial problems (3/6), social contact (5/6), depression (2/6), nervousness (4/6), loss of interest (3/6), body care (5/6), fatigue, (3/6) abilities of daily life (2/6), concentration (4/6), pain (2/6)
12	4 (1–8)	Anxiety (6/17), nervousness (13/17), worries (17/17), pain (7/17), dry skin (11/17)

Median distress score is provided per patient (second column) assessed by the NCCN distress thermometer and the frequency of reported problems on the accompanying problem list (third column), in brackets: frequency of indicated problem/all DT assessments per patient.

^*Indicated are problems reported in at least 30% of the individual screenings.

Figure 2:

Clinical course and patient-reported outcomes of patient No. 04 about the clinical condition and the radiologic results of a 33-year-old patient diagnosed with a midline glioma, CNS WHO grade 4. We detected an NF1 stop_gained mutation as an clinically actionable molecular target. The patient was treated with trametinib according to the MTB recommendation over 6 months until disease progression. Figure 1A. The lowest horizontal bar shows the clinical course in weeks since treatment started with indicated MRI scans for clinical and radiological response monitoring, and the second row shows the course of the NCCN distress score. Created with BioRender. The courses of KPS, EQ-5D, WPAI, and BFI are shown in the upper section. Figure 1B Selected subscales and symptoms of HRQoL measures by EORTC QLQ-C30 questionnaire.

Abbreviations: BFI = brief fatigue inventory, DT = distress score, EQ-5D = European Quality of Life 5 Dimensions, GHS = global health scale, KPS = Karnofsky performance score, WPAI = Work Productivity and Activity Impairment (WPAI) Questionnaire: General Health.

Discussion

The purpose of the pilot study was to develop the electronic health application TRACE and to assess its usability and feasibility from a user perspective in patients on biomarker-based therapy at the Center of Personalized Medicine in Tübingen. The app was rated as being clear and easy to use by patients, 1 caregiver, and HCPs.

In the retrospective analysis, we assessed patients with benign and malignant neuro-oncological diagnoses who underwent comprehensive molecular profiling. Based on these results modular approach to the app development was followed, and patient-reported outcome measures were chosen to cover a variety of cancer diagnoses. On the one hand, as the assessment panel was developed based on a neuro-oncological cohort, it might have been optimal to evaluate the app in a more homogeneous cohort including only patients with neuro-oncological diseases. On the other hand, by our approach, we were able to develop the app including instruments which can be applied also to patients with diagnoses other than brain tumors.

In the prospective study, we therefore included patients diagnosed with different tumor types and showed that the app was applicable regardless of the underlying diagnosis.

Usage of the app was only limited by technical difficulties. For example, a stable internet connection had to be ensured and the surveys had to be adapted to the operating systems. Through push notifications, HCPs were able to detect when questionnaires were not completed and could remind the users.

Furthermore, the interviews revealed, that issues using the app, including technical problems, could be resolved promptly. Although we did not plan an evaluation of the content included in the application, we are reporting it now because of the patients’ feedback. Regarding extent of the ePRO assessments, most users criticized the subjectively perceived high frequency of the assessments and the structure of questions (see also Table 3). This effect has also been reported by others.¹² By reducing the frequency of assessments, the patients would be less burdened, and their compliance and motivation to complete questionnaires might increase. However, frequent assessments—especially in off-label therapies—seem to be helpful as all patients reported considerable issues and problems, displayed in Table 4: The patients indicated problems in a considerable frequency until V3 (6 months) and we observed associations of ePRO results with the clinical course (Figure 1). In a subsequent study, we would adapt the frequency and timing of the questionnaires accordingly taking into consideration the results derived from the interview but also the symptom burden reported by the patients. Furthermore, it would be important for the app to register automatically when or if assessments were submitted by the patients themselves or by a caregiver.

Despite the small sample size, we observed that the results of PROs can be used to optimize health care in the future, eg, by referring burdened patients to a psycho-oncologist, social counseling, or physiotherapy.

An important limitation of the TRACE study is the potential risk of attrition bias. Only patients in a stable clinical condition might be able to handle the app. We saw this in 3 patients who dropped out from the study, because of clinical disease progression. Thus, critical factors regarding compliance not only include length and complexity of the PROM, relevant topics to patients and tumor type but also patients’ experience using an app and their clinical condition.²⁷

The patients treated with biomarker-based therapies frequently have advanced cancers, suffer from various symptoms, and often face an unfavorable prognosis. Therefore, caregiver involvement and an adaptation of the app to observer-reported outcomes might be an option to assess PROs in patients with declining clinical conditions. Although differences in patients’ and caregivers’ perspectives are frequently observed, including the caregivers in PRO assessments might be helpful, when patients have difficulties completing the questionnaires on the app—especially when they have relevant neurocognitive and physical symptoms.²⁸

Directions for future research would be systematic measures of the ePROs by using our app in a larger population and involving the caregivers and including physical parameters (eg, heart rate). Furthermore, it might be helpful to build a web-based tool where patients can sign up and participate in ePROS in which results might be submitted directly to the clinical care providers who can decide on the immediate next steps necessary.

Conclusions

Developing an application for cancer patients in the MTB demands special attention to user-feedback. The evaluation helped improve the usability and feasibility of the app. Using the content-related analysis by an inductive deductive approach enabled us to achieve more broadly defined statements and facilitated an iterative development process. This pilot study proved the feasibility and acceptance of an app enabling PRO assessment in patients with biomarker-guided therapy. We identified relevant aspects that should be considered when using apps in these vulnerable groups and stronger involvement of caregivers in the assessment would be helpful.

Conflict of Interest

G.T. has served on advisory boards (Bayer, Boehringer Ingelheim, CureVac, Novocure), as a consultant (Bayer, Boehringer Ingelheim, CureVac), as a steering committee member in non-interventional trials (Bayer, Novocure) and financial compensation for all these activities was provided as institutional funding to the University Hospital Tübingen. C.G. is a consultant for Need Inc. A.F. reports honoraria for presentations for BMS, MSD, Novartis, Pierre-Fabre; Travel support and congress participation support from BMS, Pierre-Fabre, Novartis; Advisory Boards from MSD, BMS, Novartis, Pierre-Fabre, Immunocore and institutional funding from BMS Stiftung Immunonkologie, outside the submitted work. The other authors declare no conflict of interest.

Funding

This study was funded by the Baden-Wuerttemberg (Germany) Ministry of Science, Research and the Arts, under reference number 42-04HV.MED (19)/15/1 as part of the project ZIV (Zentrum fuer Innovative Versorgung).

Authorship statement

Conception and design of the study: M.R., C.G., Gh.T. Expert discussions for app development: M.R, M.H., S.V., Ö.Ö., K.R., Y.M., M.W., N.E., P.E., N.P., A.F., A.B., D.Z., M.B., N.M., C.G., Gh.T. Retrospective analysis: PB, MR. Development of semi-structured interview guide: MR, L.D., S.V., M.H., L.W., L.S., C.A. Semi-structured interviews, transcription, analysis, categorization: M.R., L.D., F.S., M.H., L.G., H.H., S.C., S.V. Translation of interviews, reports, and language edits: S.K. Statistics: M.H., L.D., F.S.; Figures: H.B. Manuscript writing: M.R. and M.H. wrote the first draft. Manuscript review, editing, and final approval before submission: all authors.