Feasibility of Large-Scale Implementation of an Electronic Patient-Reported Outcome Remote Monitoring System for Patients on Active Treatment at a Community Cancer Center

PURPOSE:

The use of digital symptom monitoring with patient-reported outcomes (PROs) has been shown to improve patient outcomes. The evidence of benefit has been largely derived from research studies. The feasibility of adopting this technology in the real-world setting is unknown.

METHODS:

We report on the clinical implementation of a proprietary electronic patient-reported outcome (ePRO)–based digital symptom monitoring platform at the Highlands Oncology Group practice, a large community oncology practice. We present here our experience with patient enrollment, engagement, and retention; reasons for discontinued use; proportion of reports generating alerts and containing severe symptoms; and the responses to alerts including nursing telephone consultations and urgent office visits.

RESULTS:

Over an approximately 17-month period, 923 patients were successfully enrolled. Patients enrolled from June 20, 2020, through November 30, 2021, with follow-up through February 28, 2022. Retention rates at 3, 6, 9, and 12 months were 94%, 88%, 73%, and 67%, respectively, with greater retention at 12 months in patients age 65 years or older. Few patients discontinued use for reasons related to the platform (n = 47; 5%). Of the 25,311 ePRO reports submitted, 49% (n = 12,334) exceeded the predefined alert thresholds and 8% (n = 1,920) included severe symptoms. The nursing team responded within 24 hours by telephone to 31.2% (n = 3,910) of all reports with alerts. Of reports with severe symptoms, 72.7% (n = 1,395) received a call. Only 6.4% (n = 249) of phone calls required an office evaluation within 72 hours of the report.

CONCLUSION:

This single-center experience indicates that an ePRO-based digital symptom monitoring platform can be effectively implemented at a large scale with a high level of long-term patient engagement. Most reports could be effectively resolved by nurses, and physician intervention was infrequently required.

INTRODUCTION

Quality care for patients with cancer requires open lines of communication between patients and their care teams. Real-time patient reporting of issues and concerns and timely clinician responses are necessary to improve patient outcomes. Delivering this real-time patient engagement presents a major organizational challenge, and lends itself to a technology-based solution.

The use of digital symptom monitoring with patient-reported outcomes (PROs), often referred to as electronic PROs (ePROs), has been shown to improve patient satisfaction and quality of life, increase time on treatment, reduce ER/hospital utilization, and prolong overall survival.^1-8

Despite these many advantages, and multiple platforms that have been developed and modeled,^9-16 the evidence of benefit has been largely derived from research studies that recruited only a small number of patients^3,5,8 or larger studies with only short follow-up.¹⁵

The seminal publication in this area by Basch et al³ randomly assigned 766 patients at Memorial Sloan Kettering to symptom monitoring intervention or usual care. These patients were enrolled over 4 years. Patient diagnoses were limited to genitourinary (32%), gynecologic (22%), breast (20%), and lung (25%) cancers. The extent of patient reporting was not described, although median time on study was only 3.7 months. Although the primary end points of this landmark study were met, clearly establishing the value of remote monitoring, the ability to scale this approach more widely and rapidly cannot be assessed by this publication.

Basch has subsequently launched a multicenter trial of a digital monitoring system in the community setting, the PRO-TECT trial.^5,6 The trial was opened in 2017 in 50 sites with an accrual goal over 4 years of 1,100 patients. The trial randomized 52 community practices by site to active patient engagement or usual care with each single site limited to 50 patients. As reported in an ASCO plenary abstract, 1,191 patients were accrued in these 50 sites, and 593 patients were randomly assigned to the active patient management arm. The study reported patient and clinician satisfaction, physical function, symptom burden, and health-related quality of life, all of which were improved by the active intervention. Patient engagement remained high during the study period. These encouraging preliminary results confirm the findings Basch reported previously. However, the generalizability of this experience is diminished by the extremely modest accrual per site over the time period of the study.

The largest experience in the community has been reported by Patt et al¹⁵ with Texas Oncology. In the report, 73% (1,841) of enrolled patients (there were 4,375 patients approached and 2,522 patients enrolled) submitted an ePRO report over a 6-month period. Although patient engagement was initially high, it fell from 72% to 54%, with the overall follow-up time very limited; at the 3-month point, only 10% of enrolled patients had been on the platform long enough to be assessed for continued engagement. The clinical implications of these patient reports were not described. Although this report does establish the feasibility of a large-scale launch in the community, the extremely short follow-up and the lack of details regarding the impact of the data reporting on patient outcomes make it unclear whether the deployment was in fact successful.

The adoption of digital monitoring systems into routine clinical practice remains limited.^5,15,16 The feasibility of more widespread adaptation of this approach must start with high patient enrollment, patient engagement in submitting scheduled and as-needed reports, and a high level of patient retention over time. We hypothesized that with workflow and electronic medical record integration, appropriate training, and a user-friendly patient interface, a digital symptom monitoring ePRO platform could be effectively executed at a large scale in the community oncology care setting. We report here the preliminary experience of Highlands Oncology Group, a large community oncology practice with a proprietary ePRO-based digital symptom monitoring platform.

METHODS

Design

This study reports the preliminary experience of implementing a proprietary ePRO-based digital symptom monitoring platform in a large community practice. The primary aim was to evaluate feasibility using criteria of patient enrollment, engagement, and retention, and the reasons for discontinued use. The secondary aim was to evaluate the proportion of reports generating alerts and the subgroup of alerts including reports of severe symptoms; and the responses to these reports including nursing telephone consultations and urgent office visits.

Setting

Highlands Oncology Group is a community oncology practice in the state of Arkansas consisting of 22 physicians and 28 advanced practice providers. The practice has four sites of service. They participated in the Oncology Care Model and use a centralized nurse triage model.

Patient Population

All adult patients with cancer in each participating treatment site starting parenteral therapy were invited to use the ePRO system at the time of their pretreatment education program.

Intervention

The Canopy Care ePRO-based digital symptom monitoring platform is a proprietary modular, cloud-based communication system that incorporates integrated interfaces for patients and clinical staff.

Patients can submit scheduled and unscheduled reports of their distress and specific symptoms using either a symptom reporting application, which can be used on a smartphone or tablet, or an interactive voice response (IVR) interface (telephone report). The symptom monitoring application incorporates a 10-point linear analog patient distress scale and a comprehensive problem list that includes physical symptoms as well as psychosocial issues reported using a verbal rating scale. The symptoms list in the application was modeled after several validated symptom evaluation scales including the National Comprehensive Cancer Network distress thermometer, the Memorial Symptom Assessment Scale, and the Edmonton Symptom Assessment Score. Patients rated each problem on a four-point rating scale (mild, moderate, severe, and worst possible). The IVR interface facilitated reporting of overall patient well-being and a more focused list of relevant symptoms.

At the time of enrollment, a reporting schedule was defined for each individual patient on the basis of the diagnosis, treatment, and comorbidities. Most frequently, the schedule was weekly. In addition patients were instructed to submit unscheduled reports on an as-needed basis in the event of worsening symptoms or deterioration in general well-being.

A clinician-facing web-based interface incorporates a continuously updated work queue of patients' reports generated by the digital symptom monitoring. Symptom-specific thresholds were defined by the practice for each individual patient on the basis of diagnosis, treatment, and comorbidities. In addition, practices further classified these alerts as severe or standard on the basis of clinical judgment. Patient reports that exceeded this severity threshold were given priority, and severe alerts were given highest priority. When opened, the patient's ticket includes personal, disease-related, and treatment-related information, a color-coded tabulation of distress, and symptom reports.

Computer-guided triage pathways for structured telephone evaluation by oncology nurses were developed to triage patients who can be managed with a telephone consultation alone, and those who require expedited or urgent in-person evaluation. On the basis of individual patient circumstances and nursing judgment, not all alerts exceeding the threshold resulted in a telephone consultation or an urgent visit.

The practice rollout was initiated in June 2020. Rollout occurred sequentially across practice sites. Patients in this study were enrolled any time before December 1, 2021, and follow-up was through February 28, 2022.

Data Collection

Demographic and clinical variables were obtained from structured electronic health record (EHR) data. ePRO-specific data (eg, date of enrollment, date and reason for report, alerts/symptoms, and reason for opt-out) were collected through the ePRO platform and linked to patient-level EHR data.

Demographic variables included age at the time of enrollment (analyzed as a continuous variable, as 10-year categories, and as a binary variable [< 65 v ≥ 65 years]), sex (male v female), and race/ethnicity (non-Hispanic White, non-Hispanic Black, Hispanic or Latino, and other). Patients were categorized by primary tumor type using the International Classification of Disease, Ninth and Tenth Revisions, Clinical Modification (ICD-9-CM and ICD-10-CM) codes.^17,18 Tumor types were placed into the following broad categories: GI, thoracic, genitourinary, breast, hematologic, melanoma, head and neck, brain, other, and unknown. A patient was identified as having metastatic disease if there was a record of any ICD code containing secondary and neoplasm that occurred on or before the first relevant structured administration.

ePRO enrollment date was collected, along with all dates corresponding to subsequent reports. All symptoms associated with each report were collected. If a patient opted out after enrollment, the opt-out reason was collected. Opt-out reasons were categorized as related to the platform (bothersome, not interested, too complicated, causing too much stress, and does not own a phone) or not related to the platform (deceased, discharged to hospice, no longer a patient at Highlands, finishing treatment/treatment complete/not actively treating). Office evaluations/visits were identified via structured EHR data.

Data Evaluation and Statistics

In patients enrolled on the ePRO, descriptive statistics (absolute and relative frequencies) were reported for categorical demographic variables overall and by sex. Crude reporting frequency was calculated for each patient as the total number of reports per time (in months) from enrollment to last report; and was categorized as < 1, 1 to < 2, 2 to < 3, 3 to < 4, or > 4 reports per month. The distribution of reporting frequency was reported overall and by age (< 65 v ≥ 65 years).

Time to last report was calculated as the time from date of enrollment to date of last report. A Kaplan-Meier approach was used to calculate retention. Analyses were indexed to the date of enrollment. An event was defined as the date of last report in patients who did not opt out of Canopy and who had ≥ 45 days between their last report and last structured visit date; or who had an opt-out reason that was related to the platform. Patients were censored at the last report date if they had not opted out but had < 45 days between the last report and last structured visit date; or if they opted out for reasons not related to the Canopy platform. Patients with follow-up after the study period end date were administratively censored on February 28, 2022. Kaplan-Meier curves were generated for all patients enrolled on Canopy, overall and stratified by age (< 65 v ≥ 65 years). The log-rank test was used to compare the retention distribution by age. Since the median retention was not reached, retention probabilities were calculated at 3, 6, 9, and 12 months after enrollment.

The number and proportion of reports that generated any alert (standard or severe) or an alert because of at least one severe symptom was calculated. Reports that resulted in at least one telephone evaluation within 24 hours of the report date/time were identified; and among those reports that generated an alert and resulted in a telephone evaluation, the distribution of alert-generating symptoms was reported. Additionally, the number and proportion of reports followed by at least one office evaluation within 72 hours of the report date was calculated.

Analyses were conducted using R statistical computing software, version 4.0.2.¹⁹

RESULTS

Patient Population and Enrollment

Between June 22, 2020, and November 30, 2021, 3,072 patients were treated with parenteral anticancer therapies at Highlands. Since the program was rolled out sequentially across the different practice sites, not all treated patients were eligible to enroll at the start of the measurement period. Overall, 1,173 eligible patients were invited to enroll into the digital symptom monitoring system and 923 (79%) were successfully enrolled with multiple reports. The most common reason patients chose not to enroll was lack of interest (60%).

Of these 923 patients, 296 discontinued at some point after enrollment. The most common reasons for discontinuation were unrelated to the ePRO platform: death and discharge to hospice (47% and 18% of opt-out reasons, respectively).

Very few patients (approximately 5%) discontinued use of the monitoring system for reasons related to the platform: not interested (n = 31), bothersome (n = 10), too complicated (n = 4), too stressful (n = 1), and does not own a phone (n = 1).

Enrolled patients varied across tumor type (solid and hematologic), age, and metastatic status (Table 1), and included patients on active treatment at the time of implementation as well as new treatment starts over the approximately 17-month period. Most patients elected to use the application (794; 86%), whereas 71 (7.7%) elected to use the IVR interface and for 58 (6.3%), the preferred interface was not documented.

TABLE 1.

Patient Characteristics

Reporting and Retention

The median duration of follow-up was 9 months. The average number of reports submitted by patients per month is presented in Table 2. Most patients (64.7%) submitted two or more reports per month. The patient retention at 3, 6, 9, and 12 months was 94%, 88%, 73%, and 67%, respectively (Fig 1). We observed a statistically significant difference in retention between patients age older or younger than 65 years, with patients age 65 years or older more likely to be continuously reporting at 12 months (73% v 62%, P = .0011).

TABLE 2.

Reports per Month Among Successfully Enrolled Patients

FIG 1.

Kaplan-Meier estimates of time to last Canopy report: (A) retention probability of all patients and (B) retention probability stratified by age < 65 years and ≥ 65 years. Two patients had a missing value for age and were not included in the analysis stratified by age.

Management of ePRO Reports

During the study period, 25,311 ePRO reports were submitted. Of the 25,311 ePRO reports submitted, 49% (n = 12,334) exceeded the predefined alert thresholds and 8% (n = 1,920) included severe symptoms. The nursing team responded within 24 hours by telephone to 31.2% (n = 3,910) of all reports with alerts. Of reports with severe symptoms, 72.7% (n = 1,395) received a call.

Overall, 15.5% of all reports generated a phone call. In reports that triggered nursing team/staff telephone calls within 24 hours, the most common alert-generating symptoms were high levels of distress, pain, nausea, weakness, diarrhea, dyspnea, and fatigue (Table 3).

TABLE 3.

Distribution of Alert-Generating Symptom Reports That Triggered a Nurse Specialist Telephone Consultation (N = 3,910)

3,661 (93.6%) of symptom reports leading to a phone call were managed over the telephone by the nurse. The remaining 249 (6.4%) required an office evaluation within 72 hours of the report.

DISCUSSION

Despite the established evidence base that digital symptom monitoring with PROs can improve patient outcomes^1-8 and the multiple platforms that have been developed and modeled,^9-16 adoption of this approach in community oncology practices has been limited.

Barriers to the more widespread diffusion of this intervention have been described and can be classified as organizational, clinician, patient, and implementation-related barriers^20-22 (Table 4). The design of the specific proprietary ePRO digital symptom monitoring platform used in this study has been informed by these considerations, with emphasis on ease on the patient use experience, and the development of a clinician interface that is easily integrated into practice workflows and the electronic medical record. This descriptive report does not purport to describe an optimal implementation strategy.

TABLE 4.

Organizational, Clinician, Patient, and Implementation-Related Barriers to ePRO Uptake

The success reported here compares favorably with published experience. The long-term patient retention in this real-world study was comparable with that reported in the recently presented US national cluster randomized trial.^5,6 The generalizability of this experience is supported by the observation that the enrolled population of patients had similar demographics in relation to age, sex, and cancer site distribution to patients seen in routine clinical practice.²³

Age and lack of familiarity with technology have been reported as potential barriers to ePRO implementation.^20-22 In our experience, advanced age was not a barrier to retention. Indeed, we observed a statistically significant difference in patient reporting frequency and retention by age, with patients age 65 years and older more likely to be continuously reporting at 12 months, contrary to expectations. In Basch's MSKCC trial,³ one third of patients were computer-inexperienced, and this group benefited at least as much from the symptom monitoring intervention. In our cohort, fewer than 10% chose interactive voice response. Of those using the app-based response, very few patients discontinued the use of the system because of difficulty or burden.

Patients submitted a significant number of reports each month: close to half exceeded a predetermined symptom threshold and 10 percent were classified by the clinical team as severe symptoms. As expected, this resulted in a higher rate of telephone calls for patients with severe alerts (72.7%) than those with standard alerts (30%). Most issues could be effectively resolved by the telephone consultation and only approximately 5% of patients required an urgent office visit. The primary symptoms responsible for these interventions mirror the most common complaints in this treated population,²⁴ and also mirror the most common reasons for emergency room visits in patients with cancer.^25,26 Not all alerts generated a telephone call. Clinical judgment was exercised by nurses during the evaluation of patient reports including all reports defined as severe.

Our study has strengths and limitations. The cohort described is large and was enrolled over a short period of time. There was a very low rate of patient drop-out. The patients enrolled are similar in age, sex, and cancer type to those seen in routine clinical care. In addition, the symptoms they report are also qualitatively similar to those seen in routine clinical practice. The experience in a single site allowed for standardization of processes; however, it diminishes the generalizability of these results, and this warrants reproduction of the experience in other centers with different ethnicity and case distributions. We acknowledge the potential for selection bias between users and nonusers of the ePRO system. Finally, it is difficult to evaluate the impact of the COVID-19 pandemic. Like many community oncology practices, Highlands Oncology Group successfully continued treating patients during the pandemic. It is difficult to speculate how the pandemic influenced enrollment, engagement, and management of patient reports.

On the basis of these preliminary data, we are pursuing several projects. Further work is needed to determine impact on patient outcomes, and the generalizability of this experience across different treatment settings and different disease and treatment groups.

In conclusion, this single-center experience supports the hypothesis that implementing an ePRO-based digital symptom monitoring platform is feasible in routine clinical practice at a large scale. We provide evidence that patients can remain engaged over a prolonged period of time with regular reporting and low attrition. These reports result in telephonic interventions in a sizable percentage of patients, with only a small proportion requiring office evaluation.

AUTHOR CONTRIBUTIONS

Conception and design: Nathan I. Cherny, Lavi Kwiatkowsky, Jeff Hunnicutt, Eric Schaefer, Tracy Thurow, Michael Kolodziej

Administrative support: Jeff Hunnicutt, Thaddeus Beck, Eric Schaefer, Tracy Thurow

Provision of study materials or patients: Thaddeus Beck, Eric Schaefer

Collection and assembly of data: Nathan I. Cherny, Christina M. Parrinello, Lavi Kwiatkowsky, Thaddeus Beck, Eric Schaefer, Michael Kolodziej

Data analysis and interpretation: Nathan I. Cherny, Christina M. Parrinello, Lavi Kwiatkowsky, Thaddeus Beck, Eric Schaefer, Michael Kolodziej

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Feasibility of Large-Scale Implementation of an Electronic Patient-Reported Outcome Remote Monitoring System for Patients on Active Treatment at a Community Cancer Center

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/op/authors/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).