A PRO-cision Medicine Intervention to Personalize Cancer Care Using Patient-Reported Outcomes: Intervention Development and Feasibility-Testing

Abstract

Purpose:

PRO-cision medicine refers to personalizing care using patient-reported outcomes (PROs). We developed and feasibility-tested a PRO-cision Medicine remote PRO monitoring intervention designed to identify symptoms and reduce the frequency of routine in-person visits.

Methods:

We conducted focus groups and one-on-one interviews with metastatic breast (n=15) and prostate (n=15) cancer patients and clinicians (n=10) to elicit their perspectives on a PRO-cision Medicine intervention’s design, value, and concerns. We then feasibility-tested the intervention in 24 patients with metastatic breast cancer over 6-months. We obtained feedback via end-of-study surveys (patients) and interviews (clinicians).

Results:

Focus group and interview participants reported that remote PRO symptom reporting could alert clinicians to issues and avoid unneeded/unwanted visits. However, some patients did not perceive avoiding visits as beneficial. Clinicians were concerned about workflow. In the feasibility-test, 24/236 screened patients (10%) enrolled. Many patients were already being seen less frequently than monthly (n=97) or clinicians did not feel comfortable seeing them less frequently than monthly (n=31). Over the 6month study, there were 75 total alerts from 392 PRO symptom assessments (average 0.19 alert/assessment). Patients had an average of 4 in-person visits (vs. expected 6.5 without the intervention). Patients (n=19/24) reported high support on the end-of-study survey, with more than 80% agreeing with positive statements about the intervention. Clinician end-of-study interviews (n=11/14) suggested that PRO symptom monitoring be added to clinic visits, rather than replacing them, and noted the increasing role of telemedicine.

Conclusions:

Future research should explore combining remote PRO symptom monitoring with telemedicine and in-person visits.

PLAIN ENGLISH SUMMARY

Recently, there has been interest in routinely monitoring patients’ symptom reports to identify and address concerns. In this study, we aimed to develop and test an intervention where patients’ symptom reports could be used for monitoring and to avoid monthly routine in-person visits for patients who did not need or want them. We interviewed 30 patients with cancer and 10 clinicians to get their input on why they thought the intervention might be helpful, and any concerns they had about it. The input from the interviews and focus groups informed the intervention’s design, which we then feasibility-tested in 24 patients. In the interviews/focus groups, patients generally supported the idea of remotely reporting their symptoms and avoiding unnecessary visits; clinicians wanted to identify how the symptom reports would fit in their workflow. In the feasibility-test, only 10% of patients enrolled; in many cases, they were already having clinic visits less frequently or their clinicians did not feel comfortable seeing them less frequently. Among the 24 patient participants, the number of visits decreased (about 4 visits/patient vs. expected 6.5 visits/patient without the intervention). Patients generally supported the intervention. Clinicians felt that the symptom reports should be added to visits, rather than replacing them. Towards the end of the feasibility-test, the COVID-19 pandemic hit and led to an overall increase in telemedicine. Future research should explore how remote symptom reporting, telemedicine, and in-person visits can be combined to ensure that the right patients get the right care at the right time.

INTRODUCTION

The term PRO-cision Medicine, coined by Snyder, refers to personalizing care using patient-reported outcomes (PROs) [1]. Interest in routine PRO monitoring in advanced cancer patients has increased based on recent randomized controlled trial (RCT) findings [2–4]. For example, Basch et al. found that monitoring symptoms as an adjunct to routine visits improved health-related quality-of-life (HRQOL), increased ability to stay on chemotherapy, decreased emergency department admissions, and improved survival [2–3]. These benefits were even greater in computer-inexperienced patients. Notably, the intervention tested by Basch et al. monitored symptoms, but was not used to tailor visit frequency. Systematic symptom reporting may be even more valuable when used to determine which patients need a visit when [1]. The RCT by Denis et al. used this approach in patients with advanced lung cancer, comparing follow-up based on weekly patient symptom reports to routine follow-up with CT scans every 3–6 months [4]. Their intervention produced improved survival, associated with better performance status at relapse, which enabled optimal treatment. Intervention patients were also more likely to have stable/improved HRQOL versus controls. Intervention patients had less imaging but more unscheduled visits versus control patients. Other studies also support the value of PRO symptom reporting [5–14].

Because of the US’ historically fee-for-service financing, medical care has generally been organized around visits. If there was no visit, there was no reimbursement. As the US transitions towards value-based reimbursement [15–16], there are greater opportunities to use symptom monitoring not only as an adjunct to visits, but to monitor patients remotely to determine which patients need to be seen and which patients can avoid unneeded/unwanted visits. The surge in telemedicine associated with the COVID-19 pandemic has further expanded opportunities to tailor care delivery to patient’s needs [17].

We sought to develop a “PRO-cision Medicine” intervention in which remote symptom monitoring is used to tailor visit scheduling for metastatic breast and prostate cancer patients based on need. We selected the metastatic setting because of the long disease course during which patients undergo a series of treatments associated with a range of side effects, which may require additional supportive care [18–19]. To ensure the PRO-cision Medicine intervention met the needs of patients and providers, we first conducted a qualitative study to obtain their input on the design of the intervention, which we then tested in a feasibility study.

METHODS

The qualitative study and feasibility-test were approved by the Johns Hopkins School of Medicine Institutional Review Board (IRB).

Qualitative Study

Study Design:

To inform the design of the PRO-cision Medicine intervention, we conducted cross-sectional, exploratory focus groups and one-on-one interviews with clinicians and patients. Because the intervention of using PRO symptom monitoring to tailor patients’ visits affects both patients and clinicians, it was important to get both groups’ perspectives at the design stage.

Participants:

We recruited patients with metastatic breast and prostate cancer receiving care at three Johns Hopkins cancer center clinical sites, and the medical oncologists/nurse practitioners (NPs) involved in their care. Patients who were at least 21 years old, expected to live at least 6 months, and able to communicate in English were eligible. Medical oncologists/NPs/fellows who manage patients with metastatic breast or prostate cancer at a participating site were eligible. All participants underwent an IRB-approved oral consent process.

Focus Group/Interview Conduct:

Patient and clinician participants could attend one of several pre-scheduled, in-person focus groups or could schedule a one-on-one, in-person or phone interview. The target total sample size was 40: 15 metastatic breast cancer patients, 15 metastatic prostate cancer patients, and 10 medical oncologists/NPs. To obtain input from vulnerable populations, we purposively sampled patients such that at least 5 of the 15 in each tumor group were either a racial minority, did not graduate college, or reported “rare” or “never” computer use. The sample sizes were shaped by project resources and were intended to be exploratory and allow for adequate and diverse input from the three stakeholder groups.

Between November 2018-March 2019 a facilitator with a long history of conducting qualitative interviews and focus groups with cancer patients and clinicians (SH) conducted the focus groups and interviews using a semi-structured guide appropriate for both focus groups and interviews, and for both patients and clinicians. Interview/focus group topics addressed key intervention characteristics: proposed symptom list, reporting frequency, patients’ comfort-level with technology, visit scheduling, and expectations regarding how symptom reports would be acted on (Table 1-Column 1). The focus groups/interviews were audio recorded and professionally transcribed.

Table 1:

List of topics in qualitative interviews and summary of results

TOPIC	SUMMARY OF THEMES AND EXAMPLE QUOTATIONS
General Themes
Why they think it would be valuable	Alerting to issues: • I just think sometimes people let some of the symptoms go and wait for their appointment. And if you have an opportunity to reach out prior then the doctor can decide whether iťs serious enough to come in. (P18) • … many times patients donť think about sharing some of these things and this kind of helps give symptoms a more concrete measure. (Clinician FB) Remembering symptoms: • I think that when you’re actually there in a visit you tend to forget things that are going on that you might want to mention. (P14) Avoid unnecessary visits: • Well I mean like you said, if I’m feeling absolutely fine and I haven’t had any even minor symptoms, why bother traveling an hour to basically be asked the exact same questions, and then drive an hour back. (P3) • Well I think for someone who is busy …I mean I don’t work outside of home, but I do have five kids and I home school, so for someone who either has a life like mine, or who works, not having to come in to the office when things are going smoothly I think for me is preferable. So if it could be a phone call, a questionnaire, something that I could take a few minutes at home to do and submit, that is much easier than having to come in to the office. (P2) • … there are only a number of slots in a day to see a physician and there are some who probably need to be seen in those slots and others that don’t. So I think the utility can maybe open up someone’s time. (Clinician FG)
Concerns that would need to be addressed	Still need/want visits: • I think it is appropriate to try, but just because if somebody says, I don’t see anything or don’t feel anything, that doesn’t mean you shouldn’t have a visit. (P6) • My whole history is based on communicating with people. And I do like to have a personal communications relationship with my physician. (P15) Clinician workflow issues • … that was my concern is who’s looking … this is great that they fill this out, but who actually looks at it to respond to it? And is it replacing something that we’re not doing already? Because you don’t want to make work for everybody without some benefit to the patient. So if this is once a week, that seems like a lot in terms of somebody’s got to look at all these once a week, and who is that person? And is it not me, hopefully. Because tha’ts a lot. And is it going to be like a requirement to check off that I did look at it? (Clinician FG) • I think for us as very busy physicians, the idea of having to go through these questionnaires, it just adds immediate panic. Who’s going to look at it? When? So I like that the perspective, which is something that I thought about, of actually these being used to triage patients that are doing well enough, not need to be seen so frequently, but not so much patients who are going through acute side effects. (Clinician FG)
Themes Related to Intervention Design
Types of patients	Generally agreed that it would be appropriate for patients like them Some idea that it could be made available for a subset of patients • And maybe it’s not something that will be available to all patients all the time, but can be assigned to patients on specific treatment or a discretion of the team. (Clinician FG)
Symptoms to report	General agreement that the key symptoms were covered, even if not currently being experienced • You’re covering a lot of territory on these. I think, depending upon the individual, at any given moment in time, some may be valid and others may not be. (P22) • … they covered a variety of things, not only physically, but like some mental. (P14) • …a lot of these symptoms that they have here, there’s very few that I have. And so, yeah, there’s just very few that I have. When I was looking over it, it was either rare or never. (P19)
Frequency of symptom reporting	Frequency based on whether symptoms are new or bothersome: • I think it could be helpful if patients are having symptoms. For myself personally, if I were symptom-free, I don’t know that I would want to take the time to fill one out. (P2) • …if a symptom came on, would I have access that I could report that without waiting two months to report it? If I had something that came on very suddenly, would I be able to, with this app, be able to report this, like, bing, a little alert here, I’m having this problem? (P5) • I guess it depends on whether a symptom is new or not. So there’s certain symptoms that I have all the time. Like I said, neuropathy, I have that all the time. And maybe i’ts something that you report on every other week or once a month or there should be a way to say, hey, I have this new thing going on. (P20)
Mode of administration (e.g., app, MyChart, etc.)	General support for electronic but no one particular mode: • …my cell phone would be the easiest probably, the quickest I could do it. (P6) • Everybody has a Smartphone these days, so like having an app is really good. (P16) • I like my computer, but I can see how it could be beneficial on a cell phone. But I like my computer. I personally like the computer.(P11) • I am personally not very phone savvy, so I don’t really fool with apps or anything like that. But I am very much like an email, computer kind of girl. (P14) Not appropriate for everyone • … there’s a certain portion of the population that can’t use computers, don’t pay their bills on computers, don’t do any of that stuff. So I imagine they wouldn’t be interested in participating in a program like this. (P5)
Whether they still want scheduled appointments/how frequently	Definitely want to have at least some clinician visits scheduled and be seen quickly if needed • So you need to be able to make an appointment quickly … if you need it. (Patient FG)
Option to request visit	General support to allow for a visit request
Feasibility of adding patient visits	Can add visits when medically needed • …for an existing patient we usually get people in pretty quickly, or at least a phone call if nothing else, to see what’s really going on. (C9)
How/where/when results are presented to clinicians and expectations for responses	If a report had no concerning scores, patients did not express consistent opinion on whether the clinician should see the results, and clinicians wanted to minimize data overload • I wouldn’t need to hear back if I didn’t have any symptoms. (P10) • I’d rather not have more data in front of me that doesn’t require … action. (C9) For reports that had concerning scores, suggestion that response time should be related to severity • … where the patient put the checkmark would dictate how quickly they get some kind of response. Obviously, the further to the left on this form that that checkmark goes, I think the system should have built into it a timeframe for the clinician. (P22) Patients and clinicians also referred to the standard 3-day response time for MyChart messages • I know when I have contacted any of my providers through MyChart they always say, allow three days, so I would say within several days I would expect that someone would reach out to me and at least get more information on side effects or … I would say three days at the max. (P2) • So I would encourage us to use … the same kind of parameters we use, we tell patients, say, if you’re going to send a MyChart message it’s not something we’re going to necessarily respond to right away. It may take up to three days. If it’s something acute and severe you need to call a number. (Clinician FG)
Comfort with management via technology	While patients were generally comfortable with technology, recognition that some patients might not be: • I think so. I think it would also depend upon the patient. I don’t know that it would be appropriate for a 70-year-old who barely can operate a Smartphone to send a text to expect them to use an app to report their symptoms. (P8)
Comfort with telemedicine visits	Recognition that it could be a good option for some patients (e.g., rural, mobility issues) but not necessarily for everyone Visual is valued (i.e., not just audio) Patients noted they still had to come in for medication administrations, blood draws, etc., anyway; could not take the place of physical exams

P=from a patient one-on-one interview; C=from a clinician one-on-one interview; FG=from a focus group participant

Analysis:

Transcripts were analyzed thematically to inform the PRO-cision Medicine intervention’s design [20]. An initial codebook was developed using the interview topics (Table 1-Column 1) to identify deductive codes. Using MAXQDA [21], SH initially coded five interviews, which were reviewed by CS; discrepancies in coding approach were discussed and reconciled. SH then deductively coded the remaining transcripts, with CS reviewing the coded transcripts to confirm coding agreement. Any remaining discrepancies in code application were resolved by discussion and consensus. After all interviews were coded, detailed reports were prepared for each parent code, which SH and CS reviewed to identify key themes for each topic. The two authors compared themes and developed a joint summary of the findings.

Feasibility-Test

Study Design:

Informed by the qualitative interviews/focus groups, we conducted a single-arm, unblinded, feasibility-test of the PRO-cision Medicine intervention in patients with metastatic breast cancer. The study aimed to assess the feasibility of the PRO-cision Medicine intervention in practice and to obtain patients’ and clinicians’ perspectives regarding it. Given the study objectives and the funding available, it was considered preferable to test the intervention in as many patients as possible, rather than include a control arm.

Participants were recruited from the same clinical sites as the qualitative study and followed for 6-months. We tracked recruiting to achieve a purposive sample with at least 25% of patients representing racial minorities or having lower education (because electronic reporting was required, we did not purposively sample based on computer use).

Study Participants:

Recruitment occurred October-December 2019. Adult patients diagnosed with metastatic breast cancer with a life expectancy of at least 6 months who were undergoing treatment and who would normally have oncologist/NP visits at least monthly were eligible. While patients with prostate cancer were included in the qualitative study, we did not include them in the feasibility-test because they are commonly seen in clinic less frequently than monthly.

Patients had to be able and willing to report on symptoms via Epic® MyChart®. Patients enrolled in another clinical trial that influenced their visit schedule and/or health service use were not eligible. There was no target sample size; one of the outcome measures was the number of patients who enrolled during the pre-specified recruitment period, which was determined based on logistical considerations (i.e., study funding). Patients provided written informed consent.

The medical oncologists/NPs were asked to provide their perspectives on the intervention via an end-of-study interview. Because the clinicians were co-investigators, an IRB-approved waiver of consent document was included with the emailed interview invitation, and the interview script specified that completion of the interview represented consent.

Intervention:

Figure 1 depicts the tested PRO-cision Medicine intervention versus usual care. We asked patients’ clinicians to estimate how many visits they would have during the 6-month study period under usual care and then scheduled them for half as many visits (e.g., patients who would have been seen monthly were scheduled for visits every other month). The frequency of laboratory tests and imaging studies were unchanged. Patients were asked to report on 17 symptoms: 14 items from the revised Edmonton Symptom Assessment System (ESAS-r) [22–24], plus three symptoms relevant to this patient population (hot flashes/flushes, vomiting, cough) and an option to add “other.” All symptoms were rated using the ESAS-r 11-point numeric rating scale from 0=none to 10=worst possible. This version of the ESAS-r has been used by the Michigan Oncology Quality Consortium to monitor patients taking oral oncolytics and was found to be feasible in both community and academic practices [22]. The specific symptoms were confirmed to be relevant for the feasibility-test during the qualitative study.

Figure 1: Depiction of feasibility-tested PRO-cision Medicine intervention vs. usual care.

Legend: At the top, depiction of care delivery under usual care (if the patient had not been enrolled in the study). This example illustrates a patient who would have had in-person visits monthly. Between visits, if the patient had issues, the patient could have contacted the provider by phone call or MyChart message. There is no symptom monitoring. At the bottom, depiction of care delivery under the tested PRO-cision Medicine intervention. The patient is only pre-scheduled for half as many visits (every other month), is asked to complete symptom reports monthly, with the option to report symptoms in the intervening weeks. Clinicians’ offices can follow up on concerning scores by phone, MyChart message, or add a visit. Throughout, the patient may contact the provider by phone or MyChart message. There is no change to the schedule of laboratory testing and imaging studies (not depicted).

Clinicians ordered the questionnaires through the Epic electronic health record (EHR) questionnaire system, and patients completed the questionnaires via the MyChart patient portal. Patients were asked to complete the symptom reports monthly, with the option to complete the symptom reports during the intervening weeks. For the optional symptom reports, patients were first asked whether they had any new or bothersome symptoms that require attention and whether they would like to request an appointment. This was used because patients in the qualitative study reported that they did not want to respond to the same questions every week if nothing had changed. It also aimed to address the clinicians’ concerns regarding information overload. Finally, it ensured that patients who wanted to be seen inperson could be. If a patient requested an appointment, a member of the clinical team contacted the patient to schedule a visit. If a patient did not complete a weekly or monthly symptom report for 35 days, the clinical team called the patient.

Patients’ symptom questionnaire results were available to their clinician in Epic via the Synopsis View, InBasket, and the “Other Orders” folder in Chart Review. Alerts were generated for symptoms rated ≥4, representing moderate to severe symptoms [22–24]. Clinicians were asked to respond to alerts within three business days, consistent with other MyChart responses, and report to the research team how they responded (e.g., MyChart message, phone call, scheduled visit). Patients were directed to contact the clinic directly for urgent issues. Clinicians could use the information from the symptom reports (with or without alerts), along with laboratory results and imaging studies, to determine whether a patient should be seen for a visit sooner than the next scheduled visit, whether a phone consult or electronic message was needed, or whether the patient was stable.

Study Conduct:

At enrollment, we collected basic sociodemographic and clinical information from the patient’s clinician and medical record. At the end of the study, patients completed a survey [6–7] and clinicians completed a semi-structured interview to ascertain what aspects of the intervention worked well or required refinement.

Outcomes and Measures:

Similar to previous studies [7], the primary outcomes assessed the feasibility of the PRO-cision Medicine intervention, including the percentage of eligible patients who enrolled, who completed the monthly symptom reports, who completed the optional weekly symptom reports, and frequency of alerts. From the clinician side, we evaluated timeliness and actions taken in response to alerts. Secondary outcomes included the number of medical oncologist/NP visits overall, and compared to the number of visits the clinicians reported each patient would have had in usual care. We also describe other health service use (e.g., phone calls, hospitalizations, urgent care visits).

RESULTS

Qualitative Study

Participant Characteristics (Table 2):

Table 2:

Patient characteristics for qualitative study and feasibility test

Characteristic	Qualitative Study N=30	Feasibility-Test N=24
Cancer Type n(%)
Breast	15 (50.0)	24 (100.0)
Prostate	15 (50.0)	-
Age
Mean (SD)	63.8 (14.08)	62.67 (9.73)
Median (range)	64.5 (35–94)	64.5 (36–79)
Gender n(%)
Male	15 (50.0)	2 (8.3)
Female	15 (50.0)	22 (91.7)
Race n(%)
Black	4 (13.0)	5 (20.8)
White	24 (80.0)	18 (75.0)
Asian	0 (0)	1 (4.2)
Multiple	2 (7.0)	-
Education n(%)
High school graduate or less	6 (20.0)	3 (12.5)
Some college or college graduate	12 (40.0)	4 (16.6)
Post-graduate work	12 (40.0)	17 (70.8)
Computer Use n(%)
Regular	24 (80.0)	-
Occasional	5 (17.0)	-
Rare	1 (3.0)	-
Never	0 (0)	-
Line of therapy n(%)
First	-	8 (33.3)
Second	-	5 (20.8)
Third	-	2 (8.3)
Fourth or higher	-	9 (37.5)
Site of metastases n(%)
Bone only	-	3 (12.5)
Not bone only	-	20 (83.3)
Unknown	-	1 (4.2)
Hormonal status n(%)
ER-/PR-	-	4 (16.7)
ER+/PR-	-	6 (25.0)
ER+/PR+	-	14 (58.3)
HER2 Status n(%)
Positive	-	7 (29.2)
Negative	-	17 (70.8)
Distance from clinic (miles)
Mean (SD)	-	26.39 (26.14)
Median (range)	-	11.5 (2–80)
Time from metastatic disease diagnosis to enrollment n(%)
0–6 months	-	4 (16.7)
7–12 months	-	5 (20.8)
13–36 months	-	5 (20.8)
37–60 months	-	4 (16.7)
More than 60 months	-	5 (25.0)
Time spent on current therapy at enrollment n(%)
0– 6 months	-	15 (62.5)
7–12 months	-	3 (12.5)
13–36 months	-	6 (25.0)

SD=standard deviation; ER=estrogen receptor; PR=progesterone receptor

-Not available/Not applicable

We recruited the planned 15 breast and 15 prostate cancer patients (mean age 63.8 years, 80% White) and 10 clinicians. There was one clinician focus group (n=8) and one patient focus group (n=3); one-on-one interviews were conducted with the remaining participants. Twelve patients were a racial minority, had less than a college degree, and/or self-reported as computer inexperienced.

Key Findings (Table 1-Column 2):

Participants noted the value of symptom reporting in alerting clinicians to emergent or bothersome issues, helping patients remember (and clinicians know) symptoms occurring between visits, and avoiding unneeded/unwanted visits. Some patients, however, did not see the benefits of avoiding visits, and clinicians were concerned about the workflow and time implications of reviewing the symptom reports and addressing issues.

Participants generally supported the appropriateness of the PRO-cision Medicine intervention for metastatic breast and prostate cancer patients. Respondents reported that the proposed list of symptoms for the assessment were appropriate and appreciated inclusion of both physical and mental aspects; however, many patients noted that they were not currently experiencing those issues. No symptoms were consistently identified as missing from the list; respondents supported the idea of including an “other (specify)” category. Recommendations for reporting frequency ranged from daily to quarterly, with several suggestions to report when symptoms are new or bothersome. While there was a general preference for completing the symptom reports electronically, there was no strong preference regarding mode (e.g., smartphone, computer). Respondents also recognized that some patients might be less comfortable with electronic reporting than others. While patients supported using remote symptom reporting to decrease in-person visit frequency, they did not want to forego all in-person visits. They preferred less frequent pre-scheduled appointments, with the option to add visits if needed. Clinicians noted that add-ons to their schedule are not uncommon, and that they could also discuss issues by phone.

Having clinicians address symptom reports with concerning issues within three days was considered consistent with other information reported through the MyChart patient portal. For urgent issues, patients should be instructed to contact the clinic by phone. Neither patients nor clinicians felt it was imperative for clinicians to review reports with no concerning issues. Participating patients felt comfortable being managed using technology, but noted other patients may not be. Clinicians reported concerns related to workflow, responsibility for reviewing/triaging the symptom reports, and the (nonbillable) time required for the intervention. Because the focus groups and interviews occurred before the COVID-19 pandemic, there was little familiarity with telemedicine, but participants acknowledged that it could be a good option for some patients and that having video would be valuable, compared to audio-only phone consults.

Feasibility-Study

Patient Enrollment and Characteristics (Table 2):

Of 236 patients screened, 24 (10%) enrolled. Of the 212 not enrolled, the most common reason was that the patients were already being seen less frequently than monthly (n=97, 46%) or clinicians did not feel comfortable seeing the patients less frequently than monthly (n=31; 15%). Other reasons for non-enrollment included participation in a conflicting clinical trial (n=27), life expectancy less than 6 months (n=9), and declined (n=13). Recruitment of 35 patients was pending when recruitment closed.

Among the 24 enrolled patients, the mean age was 63 years, 75% were White, and 71% had completed post-graduate work. Clinically, 25% had been diagnosed with metastatic disease more than 5 years prior, and 54% were on their first or second line of therapy.

Two patients did not complete the 6-month follow-up (1 died, 1 transitioned to hospice). When COVID-19 restrictions to in-person care were instituted in March 2020, participants had, on average, 2 months left in the study (range 1.5–3 months).

Patterns of Symptom Reporting and Clinician Responses (Table 3):

Table 3:

Patterns of symptom reporting, alerts, and clinician responses from feasibility-test

OPTIONAL WEEKLY SYMPTOM REPORTS-Patterns of Reporting and Alerts
Number (%) of patients completing optional weekly symptom reports	23/24 (95.8)
Total number (mean[SD]/patient) optional weekly symptom reports completed	335 (13.96 [8.23])
Min-Max	0–26
Total number of assessments with a visit request	3 unique patients
Did visit request lead to a scheduled visit? n(%)
-Yes	2 (66.7)
-No	1 (33.3)
Total number (mean[SD]/patient) assessments reporting new or bothersome symptoms	43 (1.79 [1.96])
Min-Max	0–6
Total number (mean[SD]/patient) alerts triggered for moderate or severe symptom	40 (1.67 [1.97])
Min-Max	0–6
aNumber (% of total number of symptom alerts) for each symptom	Moderate rated 4–6	Severe rated >7
-Pain	8 (20.0)	3 (7.5)
-Tiredness	23 (57.5)	5(12.5)
-Drowsiness	9 (22.5)	3 (7.5)
-Nausea	1(2.5)	1(2.5)
-Lack of appetite	8 (20.0)	8 (20.0)
-Shortness of breath	4(10.0)	6(15.0)
-Depression	5(12.5)	1(2.5)
-Anxiety	5(12.5)	2(5.0)
-Well-being	23 (57.5)	3(7.5)
-Constipation	4(10.0)	1 (2.5)
-Diarrhea	6(15.0)	4(10.0)
-Tingling/numbness	9(22.5)	6(15.0)
-Mouth sores	5(12.5)	0(0.0)
-Rash	2(5.0)	2(5.0)
-Hot flashes/flushes	4(10.0)	1(2.5)
-Vomiting	1(2.5)	1(2.5)
-Cough	4(10.0)	0(0.0)
-Other	12(30.0)	6(15.0)
MONTHLY SYMPTOM REPORTS-Patterns of Reporting and Alerts
Number (%) of patients completing monthly symptom reports	17/24 (71.0)
Total number (mean[SD]/patient) monthly symptom reports completed	57 (2.38 [2.30])
Min-Max	0–6
Total number with a visit request	1
Did visit request lead to a scheduled visit? n(%)
-Yes	1(100)
-No	0 (0)
Total number of alerts for exceeding 35 days without reporting (neither weekly nor monthly assessment)	6
Total number (mean[SD]/patient) alerts triggered for moderate or severe symptom	26 (1.08 [1.74])
Min-Max	0–6
bNumber (% of total number of symptom alerts) for each symptom	Moderate rated 4–6	Severe rated >7
-Pain	5 (19.2)	3(11.5)
-Tiredness	9 (34.6	2(7.7)
-Drowsiness	4(15.4)	0 (0.0)
-Nausea	1 (3.8)	0 (0.0)
-Lack of appetite	5(19.2)	2 (7.7)
-Shortness of breath	9 (34.6)	2 (7.7)
-Depression	1 (3.8)	0 (0.0)
-Anxiety	1(3.8)	1(3.8)
-Well-being	12 (46.2)	2 (7.7)
-Constipation	3(11.5)	0 (0.0)
-Diarrhea	2 (7.7)	2 (7.7)
-Tingling/numbness	4(15.4)	4(15.4)
-Mouth sores	0 (0.0)	0 (0.0)
-Rash	0 (0.0)	0 (0.0)
-Hot flashes/flushes	4 (15.4)	2 (7.7)
-Vomiting	0 (0.0)	0 (0.0)
-Cough	0 (0.0)	1(3.8)
-Other	6 (23.1)	3(11.5)
Patterns of Clinician Alert Responses
Number of total alerts	75
Did clinicians address in the 3-business day window? n(%)
-Yes	64 (85.3)
-No	7 (9.3)
-Missing	4 (5.3)
Did clinicians address alerts without prompting by research team? n(%)
-Yes	66 (88.0)
-No	9 (12.0)
cNumber (% of total alerts) addressed by: (not mutually exclusive)
-MyChart message	8 (10.7)
-Phone call	41 (54.7)
-Adding a visit	17 (22.7)
-During already scheduled visit	1 (1.3)
-Other	7 (9.3)
-Not addressed	4 (5.3)

SD=standard deviation

^aDenominator=40 total alerts, but there could have been more than one moderate or severe symptom reported for each alert

^bDenominator=26 total alerts, but there could have been more than one moderate or severe symptom reported for each alert

^cDenominator=75 total alerts; 68 alerts were addressed with 1 method, and 3 alerts were addressed with 2 methods

A total of 335 optional weekly assessments were completed by 23 of the 24 enrolled patients (96%). Of 3 patients who requested an additional visit on their symptom assessment, 2 had a visit scheduled. Patients reported new or bothersome symptoms in 43 assessments (average 1.79/patient), 40 of which included moderate or severe symptoms (average 1.67/patient). The most commonly reported moderate or severe symptoms were tiredness (70% of alerts) and well-being (65%).

A total of 57 monthly symptom reports were completed by 17 of the 24 enrolled patients (71%), 2.38 assessments per enrolled patient, on average. One patient requested an additional visit, which was scheduled. There were 26 assessments that triggered alerts for moderate or severe symptoms (average 1.08/patient). The most commonly reported moderate or severe symptoms were well-being (53.9% of alerts), other (44.6%), tiredness (42.3%), and shortness of breath (42.3%).

There were 6 alerts for patients who did not complete either a weekly or monthly symptom assessment for longer than 35 days.

Patterns of Health Service Use (Table 4):

Table 4:

Patterns of health service use from feasibility test

Patterns of Care
Mean (SD) number of expected medical oncologist/NP visits per patient without the intervention	6.5 (0.88)
Total number of clinic visits	114
Total number of medical oncologist/NP clinic visits	95
-Mean (SD) number of medical oncologist/NP clinic visits per patient	3.96 (1.90)
-Number (%) of clinic visits with medical oncologist/NP prompted by an alert	2/95 (2.1)
Total number of phone consults	206
Total number of phone consults with medical oncologist/NP	73
-Mean (SD) number of phone consults with medical oncologist/NP per patient	3.04 (2.82)
-Number (%) of phone consults with medical oncologist/NP prompted by an alert	36/73 (49.3)
Total number of MyChart messages	134
Total number of MyChart messages with medical oncologist/NP	100
-Mean (SD) number of MyChart messages with medical oncologist/NP per patient	4.17 (4.86)
-N (%) of MyChart messages with medical oncologist/NP prompted by an alert	4/100 (4.0)
Total number of video visits	17
Total number of video visits with medical oncologist/NP	14
-Mean (SD) number of video visits with medical oncologist/NP per patient	0.58 (0.72)
-N (%) of video visits with medical oncologist/NP prompted by an alert	0 (0.0)
Mean (SD) number of treatment/infusion visits per patient	6.58 (4.88)
N (%) of patients with an urgent care visit	2 (8.3)
aN (%) of patients with an emergency department visit	4 (16.7)
bN (%) of patients hospitalized	6 (25.0)

SD=standard deviation; NP=nurse practitioner

^a3 patients had one emergency department visit; 1 patient had 3 emergency department visits

^b5 patients were hospitalized once; 1 patient was hospitalized twice

Without the intervention the average expected number of visits to a medical oncologist/NP was 6.5; the average number of visits among the enrolled patients was about 4. The difference was statistically significant based on a post hoc paired t-test (mean difference=2.54; 95%CI=1.79–3.29; p<.0001). To account for COVID-19’s impact, we repeated the test and included the 14 video visits with the medical oncologist/NP: mean difference=1.96; 95%CI=1.252.67, p<.0001. There were 206 phone calls, 73 with the medical oncologist/NP, including 36 in response to an alert. There were 134 MyChart messages, 100 with the medical oncologist/NP, including 4 in response to an alert. Two patients had urgent care visits, 4 patients had emergency department visits (1 patient 3 times), and 6 patients were hospitalized (1 patient twice). Two patients were sent to the emergency department based on symptom report alerts.

Patient End-of-Study Survey (Table 5):

Table 5:

Patient end-of-study survey responses

	n(%)
FEEDBACK FORM ITEM (n=19)	Strongly Agree	Agree Disagree/Strongly	Disagree	Not Applicable
It was easy to report on my symptoms using Epic MyChart	14 (74)	4 (21)	1(5)	-
I liked being able to report on my symptoms using Epic MyChart	13 (68)	5 (26)	1 (5)	-
Reporting my symptoms using Epic MyChart was helpful for deciding whether I needed an oncology/nurse practitioner visit	9 (47)	7 (37)	3 (16)	-
Reporting on my symptoms monthly, with the option to report on my symptoms weekly, was helpful	11 (58)	6 (32)	2 (11)
I was scheduled to see my oncologist/nurse practitioner as often as I felt I needed	16 (84)	3 (16)	0 (0)	-
I feel comfortable having my care managed, at least to some extent, by telephone calls and MyChart messages	14 (74)	2 (11)	3 (16)	-
I liked being able to request an appointment as part of the Epic MyChart symptom questionnaire	12 (63)	4 (21)	3 (16)	-
I was able to see my oncologist/nurse practitioner as soon as I needed to for problems with my symptoms	11 (58)	4 (21)	0 (0)	4 (21)
I heard from my doctor’s office by MyChart message or telephone call if I reported a symptom that I felt required attention	15 (79)	1 (5)	0 (0)	3 (16)
The amount of time it took for my doctor’s office to contact me when I reported a symptom that I felt required attention was appropriate	13 (68)	3 (16)	0 (0)	3 (16)
The symptom questionnaire included the symptoms I was experiencing	10 (59)	5 (29)	2 (11)	-
It is appropriate to alert my doctor’s office for answers of 4 or higher	13 (72)	5 (28)	0 (0)	-

- indicates that it was not a response option for that item

Of the 24 enrolled patients, 19 completed the end-of-study survey. Support was high, with more than 80% agreeing with positive statements about the intervention. In open-ended comments, patients reported appreciating the opportunity to stay connected with their clinical team, though others reported having no symptoms or found completing the questionnaire a chore. In general, three days for the clinic to respond to concerning scores was considered appropriate for non-urgent issues; many patients heard back in one day while others reported delayed follow-up. One patient reported that the symptom survey identified significant issues she had thought were trivial. In terms of frequency for reporting, suggestions ranged from once monthly to twice monthly to weekly, with one participant suggesting that issues be reported when they happened. Almost all participants felt that they were seeing their oncologist/NP often enough, but one reported that the “survey did not take the place of an in-person visit” for someone who needs “to talk and ask questions.” Some patients liked using MyChart for communication, but others preferred visits or phone calls. Some patients experienced hassles trying to schedule appointments when needed. The list of symptoms was considered appropriate, with no other symptom consistently being raised in the comments.

Clinician End-of-Study Interviews (Table 6):

Table 6:

Feedback from clinician exit interviews

Eligibility criteria:

• Generally considered appropriate

• Requirement that patients be currently seen at least monthly limited the eligible pool

• Concern that some patients would not routinely report their symptoms via MyChart, or that the symptom questionnaire would not be sensitive enough to identify problems

• Other factors noted to have affected recruitment included:

º Patients’ willingness to be seen less frequently

º Patients’ enrollment on clinical trials that influenced visit frequency

º Short enrollment period that occurred around the holidays

Experience using Epic EHR survey functionality:

• Clinicians expressed a desire for:

º Easier ways to order the questionnaires

º Better access to the results

º Enhanced visualization of the data

º Improved navigation of the alerts

Symptom questionnaire:

• Supported approach for the optional weekly reports and monthly assessment

• Generally endorsed the content of the symptom questionnaire

• Suggested that patients be able to request a phone call or a visit, rather than just a visit

• Found the alerts helped for patient management and saw minimal value in reviewing the data for patients with no alerts

Alerts:

• Recommended that alerts should not trigger for chronic issues but focus on incident symptoms

• Interested in tailoring the alert levels to the symptom

• Endorsed 3 business days as sufficient time to respond to alerts

• Noted role of alerts in identifying concerning issues that they were able to address and providing valuable information on how patients were doing between visits

• Agreed they could manage the symptom alerts effectively remotely

• Able to accommodate in the few cases when visits needed to be added

Of the 14 clinician co-investigators, 11 participated in an end-of-study interview (8 MDs, 3 NPs; 5/11 had directly recruited patients to the study, some MDs and NPs co-managed patients not directly recruited by themselves). The interviews occurred in July 2020, in the midst of the COVID-19 pandemic.

The clinicians thought the eligibility criteria were generally appropriate. However, in many cases, where clinicians felt comfortable seeing a patient less often than monthly, they were already doing so, thus limiting the eligible pool. Clinicians noted that they felt less comfortable decreasing visit frequency for patients whose disease is unstable, have high disease burden, are on active chemotherapy, are perceived as being anxious, or have a history of missed appointments. There were also concerns related to relying on symptom reporting to monitor patients (e.g., patients would not complete the questionnaire, questionnaire would not be sensitive enough). When asked which patients would be possible candidates for this intervention, beyond those with metastatic disease, clinicians mentioned other stable populations such as survivors of early-stage disease who are undergoing surveillance. However, many clinicians felt that the symptom monitoring should be added to clinic visits, rather than trying to replace them. The role of telemedicine in decreasing the need for in-person visits was raised, possibly in conjunction with symptom monitoring.

Clinicians found the questionnaire ordering process through the Epic EHR non-intuitive, and several reported that they either needed the research team to help them do it, or to do it for them. Clinicians also noted that the results, in the “Other Orders” folder in Chart Review or in Synopsis View, were not easy to find, and they preferred for the results to be directed to their InBasket. While in this study, the alerts were directed to the research team, who forwarded them to the appropriate clinicians, in the future, clinicians wanted the alerts directed to them or to clinic staff.

The clinicians generally endorsed the symptom questionnaire, including the content and the optional weekly reports and monthly assessment. One clinician suggested having the ability to tailor the questionnaire frequency depending on the patient. Recommended modifications included the ability to request a phone call or a visit (rather than just a visit), and alerts for incident symptoms (not chronic issues). There was also some interest in prioritizing alerts based on the symptom (e.g., “shortness of breath” could indicate an urgent issue requiring immediate notification and follow-up).

The clinicians found the alerts helped identify concerning issues and felt comfortable managing these issues remotely. They saw minimal value in reviewing the data for patients with no alerts. One clinician expressed concern that patients reported urgent issues that required immediate attention. There was some preference for phone management to avoid extensive back-and-forth via MyChart messages and because hearing the patient’s voice can be helpful. There were few instances when visits needed to be added, and these unscheduled visits could generally be accommodated without too much burden.

DISCUSSION

PROs can play an important role as health care transitions to emphasize patient-centeredness and value rather than volume. We developed a PRO-cision Medicine remote symptom monitoring intervention to tailor visit frequency based on PROs, and feasibility-tested the intervention in metastatic breast cancer patients. We selected metastatic cancer given the success of previous studies in this population [2–4] and the long care trajectory with multiple treatments and side effects that can require supportive care.

While using remote symptom monitoring to decrease the frequency of in-person visits was largely endorsed by patients and clinicians in our qualitative study, the feasibility-test found that many clinicians were already decreasing visits for sufficiently stable patients such that 61% of approached patients were ineligible (46% already seen less than monthly; 15% clinicians uncomfortable seeing less often). Even though the PRO-cision Medicine intervention was effective in decreasing visits among the small fraction (10%) of patients who were enrolled (from an expected average of 6.5 visits to about 4 visits over the 6-month study period), decreasing visit frequency might not be the optimal endpoint for evaluating the benefits of the intervention.

Two patients were sent to the emergency department based on their symptom reports. This scenario is positive because it alerted clinicians to emergent issues, but also a concern because the issues required urgent attention, which is not the intention of the symptom monitoring system. While it is unlikely that a routinely scheduled visit would have caught the urgent symptoms, the patients may have been more likely to call the clinic in the absence of the symptom reporting. Notably, in multiple places the symptom reporting system urged patients not to report urgent issues via MyChart. It is possible that patients did not realize the urgency of their symptoms. Based on the experience of these two patients, we sent an email to all participating patients reminding them to contact their doctor’s office directly for issues requiring immediate attention and not to report urgent issues via the symptom report.

Several limitations of the feasibility-test are worth noting. Some patterns-of-care towards the end of the study may have been affected by the COVID-19 pandemic; the repeated post hoc t-test incorporating video visits may not fully account for the COVID-19 care delivery impacts. Further, we have no basis for comparing rates of phone calls, MyChart messages, hospitalizations, emergency department encounters, and urgent care visits. As described above, we used the limited study funds to test the intervention in as many patients as possible. While our sampling ensured representation from patients who identify as a minority race and/or with lower education in the qualitative study and feasibility-test, participation in the feasibility-test required willingness to report symptoms electronically, and over 70% of feasibility-test patients reported post-graduate education. From a health-system resource standpoint, it is helpful to achieve efficiencies in a subset of patients. However, it is important to continue to pursue opportunities to incorporate all interested patients in such interventions, particularly given evidence that symptom monitoring has even greater benefits in patients with less computer inexperience, lower education, and minority race [2,6].

The disruptions in care delivery caused by the COVID-19 pandemic further emphasize the importance of delivering the right care to the right patient at the right time, not only for this study, but more generally. It has resulted in a substantial shift to telemedicine [17]. Given the low rates of eligibility for our feasibility-test, and the transformations in care related to COVID-19, further refinement and testing of the PRO-cision Medicine intervention will focus on in-person visits, telemedicine, and remote symptom monitoring, combined in a patient-centered way to ensure that the right patients get the right care at the right time.