Implementation and impact of an electronic patient reported outcomes system in a phase II multi-site adaptive platform clinical trial for early-stage breast cancer

Abstract

Objectives

We describe the development and implementation of a system for monitoring patient-reported adverse events and quality of life using electronic Patient Reported Outcome (ePRO) instruments in the I-SPY2 Trial, a phase II clinical trial for locally advanced breast cancer. We describe the administration of technological, workflow, and behavior change interventions and their associated impact on questionnaire completion.

Materials and Methods

Using the OpenClinica electronic data capture system, we developed rules-based logic to build automated ePRO surveys, customized to the I-SPY2 treatment schedule. We piloted ePROs at the University of California, San Francisco (UCSF) to optimize workflow in the context of trial treatment scenarios and staggered rollout of the ePRO system to 26 sites to ensure effective implementation of the technology.

Results

Increasing ePRO completion requires workflow solutions and research staff engagement. Over two years, we increased baseline survey completion from 25% to 80%. The majority of patients completed between 30% and 75% of the questionnaires they received, with no statistically significant variation in survey completion by age, race or ethnicity. Patients who completed the screening timepoint questionnaire were significantly more likely to complete more of the surveys they received at later timepoints (mean completion of 74.1% vs 35.5%, P < .0001). Baseline PROMIS social functioning and grade 2 or more PRO-CTCAE interference of Abdominal Pain, Decreased Appetite, Dizziness and Shortness of Breath was associated with lower survey completion rates.

Discussion and Conclusion

By implementing ePROs, we have the potential to increase efficiency and accuracy of patient-reported clinical trial data collection, while improving quality of care, patient safety, and health outcomes. Our method is accessible across demographics and facilitates an ease of data collection and sharing across nationwide sites. We identify predictors of decreased completion that can optimize resource allocation by better targeting efforts such as in-person outreach, staff engagement, a robust technical workflow, and increased monitoring to improve overall completion rates.

Trial Registration

https://clinicaltrials.gov/study/NCT01042379.

Introduction

As the United States healthcare landscape continues to evolve, increasing importance has been assigned to the concept of precision (or personalized) medicine. Within this paradigm, it is vital for physicians to develop a multifaceted understanding of each patient’s health before, during, and after treatment. For decades, clinician documentation through visit notes has been the primary source for the prospective collection of adverse events (AE) data. While physician-reported AEs represent an important body of information, a number of research studies demonstrate that drug-related symptom severity is often underestimated or missing entirely from clinician notes, suggesting that this type of documentation alone is inadequate for capturing the full spectrum of severity and duration of patient side effects from drug treatment.^1,2

Clinical trials provide a unique and effective platform through which to explore the patient journey through treatment and beyond, allowing for more cohesive and longitudinal collection of drug impact and toxicities. In the setting of clinical research, the ability to reliably collect and take action on patient-specific data is essential.³ Patient-Reported Outcomes (PRO) represent an important data stream through which patients can self-report on symptoms, side effects, and quality of life (QOL) during active treatment and follow-up. PRO supplements traditional clinician AE reporting by enabling better documentation and identification of patient symptoms.¹ In particular, PRO may have a major benefit in the follow-up phase, when the frequency of clinic visits markedly decreases for patients. The purpose of our effort is to evaluate the contribution of standardized and consistent patient reported outcomes (ePROs) in the context of a study that has rigorous AE physician reporting.

Here, we describe the implementation of ePRO as a series of patient and staff-oriented engagement activities designed to address some of the major issues that act as a barrier to adherence. We also describe the ePRO platform within the framework of an agile, responsive multi-site clinical trial framework including staff engagement and technology interventions.

Methods

Clinical study environment

I-SPY2 is a phase II multi-site adaptive platform clinical trial evaluating the effect of novel neoadjuvant therapies for locally advanced (stage 2 and 3) breast cancer, which evaluates patients through therapy and for 10 years after surgery. In general, as part of the I-SPY2 protocol, in the active treatment phase, patients receive 12 weeks of an experimental regimen that may include a taxane alone or in combination with immunotherapies, or novel agents, followed by 4 cycles of doxorubicin and cyclophosphamide. Patients receive standard adjuvant therapy as appropriate (endocrine therapy or radiation or biologics) for up to a year. Toxicities are monitored during active treatment and for up to 6 months after neoadjuvant (active) treatment ends. Adjuvant treatments are collected along with vital status at 6-month intervals for 2 years and annually thereafter.

Alongside a multidisciplinary team of clinicians, investigators, patient advocates, and statisticians, we developed an ePRO system to collect critical AE and QOL data from I-SPY2 participants. Our ePRO platform was designed to integrate with the I-SPY2 electronic data capture system (EDC) and therefore impose minimal burden on clinical research staff, which is commonly cited as a barrier to successful long-term implementation of ePRO technologies.⁴ Unlike the majority of existing ePRO platforms which are used solely by one academic institution,⁴ our platform acted as a singular central database for 27 I-SPY2 Trial sites across the United States (Figure S1), many of which use different electronic health record systems.

To implement this large-scale change in the I-SPY2 Trial across all 27 sites, we assembled a working group consisting of physicians, patient advocates, clinical research experts, a programming team, and central ePRO coordinators.

Selection of ePRO items

Collectively, this group designed a series of PRO questionnaires using validated questions from the Health Measures Patient Reported Outcomes Measurement Information System (PROMIS) and the National Cancer Institute’s Patient Reported Outcomes—Common Terminology Criteria for Adverse Events (PRO-CTCAE) instruments. We use these tools to standardize reporting within the oncology research space, focusing on a growing need to capture patients’ self-reported symptoms and toxicities. As shown in Figure S2, the content and length of questionnaires differed by study timepoint. The longest questionnaires, Screening and Follow-up 1-4, contained a maximum of 126 branching-logic questions.

Technology solution for implementation

Using the OpenClinica EDC, the programming team developed rules-based logic to automate the scheduling of questionnaires based on the I-SPY2 treatment schedule, with the goal to increase the efficiency and accuracy of data collection while simultaneously reducing the workload on Clinical Research Coordinators (CRC). Data was collected directly in OpenClinica to eliminate the need for data translation between unique systems, as was historically required when PRO data was collected via paper-based questionnaires and manually transferred into a free-standing electronic database.

Beginning in September of 2020, in collaboration with the CRCs, the central ePRO coordinators began piloting the use of the ePRO platform at the University of California, San Francisco (UCSF). Over the course of seven months, CRCs at UCSF accrued I-SPY2 patients to the ePRO sub-study, and provided feedback on both technological and workflow issues related to the new electronic platform. Each piece of feedback was reviewed by the ePRO working group, and was used to correct any system errors and develop in-depth training materials for CRCs.

The UCSF pilot also coincided with a trial-wide shift to OpenClinica as the primary EDC for I-SPY2, which allowed integration of the ePRO technology and workflow seamlessly into the larger I-SPY2 data collection pipeline. After OpenClinica was implemented across I-SPY2 sites, we expanded the ePRO sub-study to the remaining 26 sites, concluding the pilot phase in June of 2021.

Given the quantity and diversity of I-SPY2 sites, it was important that the training materials were malleable to the unique workflows at each site. By providing a workflow “road map” to CRCs (Figure 1B), we aimed to ease the transition from paper- to electronic-based data capture in the PRO setting. In addition to each site’s initial training session, the central ePRO coordinators provided CRCs with monthly progress reports and offered additional training sessions for sites demonstrating poor compliance with the new platform.

Figure 1.

(A) I-SPY2 adaptively randomizes patients to 12 weeks of a drug treatment regimen, followed by 4-biweekly cycles of Adriamycin Cyclophosphamide (AC) and surgery. I-SPY2 screens multiple experimental regimens in addition to standard neoadjuvant chemotherapy. Magnetic resonance imaging (MRI) and residual cancer burden (RCB) are used to determine response to chemotherapy. (B) All sites were provided with a workflow “road map” to show how and when users should engage with the ePRO platform. Key dates from the PRO Scheduler are highlighted in yellow in the CRC column of the figure. Diamonds represent decision points. Rectangles represent tasks or automated steps in the ePRO workflow. Circles and dotted lines indicate future-state interventions.

Operational implementation

In the OpenClinica EDC, CRCs entered key clinical dates into the PRO Scheduler case report form, as shown in Figure S3. Based on these dates, the system automatically scheduled ePRO questionnaires according to the timeline shown in Figure S2. Patients received a maximum of 27 ePRO questionnaires during the screening, treatment, and follow-up phases of I-SPY2. The questionnaires sent during the drug treatment phases primarily consisted of PRO-CTCAE questions, FACIT-GP5, distress thermometer, and fear of recurrence, while benchmark timepoints (ie, Screening, inter-regimen, pre-surgery, and follow-up) also included PROMIS questions. Each time a questionnaire was scheduled within the EDC, an email containing a unique link to the questionnaire was automatically sent to the patient by OpenClinica.

Based on the volume of patient accrual, each I-SPY2 site was given one or two tablet computers with internet connectivity as a means to facilitate questionnaire completion in the clinic environment. This tool also enabled patients without access to internet-capable devices an opportunity to participate in the ePRO sub-study. In addition, our programming team developed a series of data visualization reports in Metabase that utilized real-time data from the EDC. The central ePRO coordinators and CRCs were each granted access to site-specific reports, allowing for global tracking of individual patients. Each site’s Metabase report pulled data directly from OpenClinica and coded questionnaire status into five categories: Completed, Skipped, Stopped, Scheduled, and Data Entry Started (Figure S4). The Metabase report enabled CRCs to monitor patients’ questionnaire status and history in real-time, allowing them to quickly determine which patients should be provided with a tablet in the clinic.

Focus groups

Throughout the Pilot and the all-sites ePRO implementation, we worked with patient advocates in a focused group format to garner advice on critical patient communications. Direct patient feedback through the pilot also allowed us to generate standardized patient-facing materials in preparation for the all-site rollout. Future studies will include using PROs to perform active symptom monitoring within the trial.

Statistical analysis of factors associated with completion

Descriptive statistics were used to visualize the completion over time. Subgroup analyses were performed to analyze the impact of age, race and ethnicity on completion. For this, a per patient completion percentage was computed by looking at the number of completed surveys as a percentage of the total number of surveys administered to each patient. Age, race and ethnicity were binarized into two categories: ≤45 years old vs >45 years old, white vs non-white and not-Hispanic vs Hispanic, and Independent-samples t-test were conducted to compare the completion rates between each subgroup. Baseline raw PROMIS data was scored using the HealthMeasures PROMIS Scoring Service. The calculated summary T-scores generated for each PROMIS bank use response pattern scoring to generate a standardized score where 50 is the mean the reference population and 10 is the standard deviation (SD) of that population. Spearman’s correlation was calculated between each item bank’s summary T-score with the patient level completion rate. PRO-CTCAE data across the first 6 weeks of treatment was summarized using the mean grade independently across each symptoms and factor (frequency, severity, and interference). For each of these, patients were dichotomized into two groups, where average grade >2 was considered as case group vs ≤2. Independent-samples t-test were conducted to compare the completion rates between the two groups for each symptom and factor. Significant variables were visualized using boxplots.

All statistical analyses and visualizations were performed using R version 4.2.2.

Ethics approval

The procedures used in this study adhere to the tenets of the Declaration of Helsinki and were performed under individual site Institutional Review Board approval, and all patients provided written informed consent prior to enrollment for study participation and publication of de-identified data.

Availability of data and materials

Subject-level data for this study is available to approved investigators completing a request form available at: https://www.ispytrials.org/collaborate/proposal-submissions.

Results

Internal pilot study

Beginning in September of 2020, in collaboration with the CRCs, the central ePRO coordinators began piloting the use of the ePRO platform at UCSF.

The initial completion rate in the first quarter was high with a screening survey completion rate (green line) of 87.5% and overall completion rate (blue line) of 56.6%. However, the screening completion rates dipped in the second quarter to 33.3% (Figure 2). When we began our pilot at UCSF, OpenClinica was exclusively being used for ePRO, but not yet as the primary EDC for I-SPY2. This notable dip in screening survey completion from 87.5% in the first quarter to 33.3% in the second quarter corresponds precisely to the migration to OpenClinica as the primary EDC. While the overall questionnaire completion rate at UCSF declined marginally with increasing accrual over the first nine months of the pilot, the screening questionnaire completion rate improved upon the introduction of tablets into the clinical workflow at UCSF in January of 2021 (Figure 2). With increasing familiarization with survey protocols, systems, staff engagement, and a rigorous training plan, the overall questionnaire completion rate at UCSF began to increase from June 2021, reaching nearly 70% by the fall of 2021. In the same time period, the screening questionnaire completion rate reached 100% and stayed at that level through March of 2022.

Figure 2.

ePRO questionnaire completion rates at UCSF; Quarterly. Starting in August of 2020, questionnaires were split into six, three-month segments, based on the date each questionnaire was sent to a patient. Both the Screening questionnaire completion rate (green line) and the overall questionnaire completion rate (blue line) are shown. Various interventions are indicated by red arrows. The table below the graph indicates the raw number of questionnaires and patients corresponding to each segment.

All site rollout

In June of 2021, the ePRO platform was launched at 26 additional I-SPY2 sites across the United States. As of March 1st, 2022, a total of 173 patients were enrolled on the ePRO platform (mean age, 47.1, age range, 20-78 years). 115 patients (66.5%) were I-SPY2 participants from sites other than UCSF, while 58 patients were enrolled at UCSF (Figure 3). This cohort was demographically diverse with 122 white patients (70.5%) and 34 non-white patients (19.6%), as well as 26 Hispanic patients (15%) (Table 1). Across all sites, CRCs failed to register a total of 13 patients (6.9% of all eligible patients) on the ePRO sub-study during the first eight months after the ePRO platform was activated.

Figure 3.

Cohort breakdown of the173 I-SPY2 patients that were included in this analysis based on their ePRO enrollment and treatment enrollment status. A total of 59 patients were included in the UCSF Pilot.

Table 1.

Demographic information including age, race, and ethnicity for 173 I-SPY2 patients who received ePRO questionnaires.

	Overall (N = 173)
Age
Mean (SD)	47.1 (12.3)
Median [Min, Max]	46.0 [20.0, 78.0]
Race
Asian	13 (7.5%)
Black	20 (11.6%)
White	122 (70.5%)
Other	1 (0.6%)
Not reported	16 (9.2%)
Unknown	1 (0.6%)
Ethnicity
Hispanic	26 (15.0%)
Not Hispanic	139 (80.4%)
Not reported	8 (4.6%)

Improvement in questionnaire completion over time

As noted in the pilot, increasing familiarization with the survey protocol, systems, tablet integration, and heavy staff engagement, all within an agile, responsive framework, led to improved completion rates. In an analysis on the first eight weeks after a site’s initial ePRO training separately from the remaining time each was active on ePROs, 60% of the sites had a greater than 10% increase in overall ePRO completion between the early and late time periods (Figure S5).

Impact of monitoring site progress

Utilizing data from the Metabase tracker (Figure S4), we implemented a hierarchical categorization system in order to monitor site progress and offer targeted support to CRCs, as shown in Figure S6a. Categorization was determined by Screening questionnaire completion rates, overall questionnaire completion rates, frequency of individual patient engagement with the platform, and percentage of eligible patients enrolled on ePRO.

As seen in Figure S6b, the completion rates for sites in the red category, especially across the first 12 weeks of therapy, were notably lower than those in the yellow and green categories. With an overall completion rate of 70.9%, sites in the green category consistently interacted with the ePRO platform and frequently engaged with patients (Figure S6b). The majority of patients belonged to sites that were categorized as yellow, and hence were monitored closely for completion, but no intervention was necessitated. Patients from red sites, who had direct engagement to improve survey completion, showed a significant improvement in the follow up time periods in survey completion.

Demographic factors associated with higher completion rates

Of the patients who completed more than one questionnaire, 38.1% completed 75%-100% of the questionnaires they received. The majority of patients completed between 30% and 75% of the questionnaires they received, with only 4.1% of patients completing none of the questionnaires sent to them. Patients who had only been sent one questionnaire (screen-fail patients), were excluded from this analysis.

Subgroup analyses were performed to analyze impact of patient demographics on completion rates, that is the total number of received surveys that a patient completed. There were no statistically significant variations in patients’ completion rates by age, race or ethnicity (Table 2).

Table 2.

Subgroup analyses by patient demographics on completion rates.

Item	Subgroup		P-value
Age
	≤ 45 years old	> 45 years old
N	83	85	.11
Mean completion rate (variance)	56.8% (11.0%)	64.9% (9.4%)
Race
	White	Non-White
N	120	35	.78
Mean completion rate (variance)	61.3% (10.1%)	63.1% (10.4%)
Ethnicity
	Not Hispanic	Hispanic
N	139	25	.07
Mean completion rate (variance)	48.7% (14.2%)	63.4% (9.2%)

Then, we were interested in looking at if early engagement with the ePRO questionnaires was related to higher overall engagement. Indeed, patients who completed the ePRO questionnaire at the screening timepoint were significantly more likely to complete more of the surveys they received at later timepoints (mean completion of total surveys of 74.1% vs 35.5%, P < .0001). In visualizing completion based on engagement with first survey received, the difference in overall engagement with the surveys was most apparent during the follow-up timepoints, where within the subgroup of patients who completed the first survey they received, the completion rate varied between 50% and 100%, as opposed to within the subgroup of patients who did not complete the first survey they received, where the completion rate was almost half, varying between 0% and 30% (Figure 4A).

Figure 4.

Factors affecting completion rates. (A) Completion rates based on engagement with the first questionnaire received. (B) Association of baseline QOL with patient survey completion. (C) Association of early patient symptoms with patient survey completion. Asterisks above the plot highlight significance level (P < .05[*], <.01[**], <.001[***]).

We further we analyzed the difference in completion between study arms. There was no notable difference in overall questionnaire completion between the control arms (55.7% completion) and the investigational arms (56.2% completion). The control group had a slightly higher, though statistically insignificant, screening completion rate than the investigational group, at 85.7% and 74.5%, respectively.

Symptoms and QOL factors associated with higher completion rates

Next, we were interested in looking at baseline patient symptomology and QOL triggers that might impact later patient engagement with the surveys. We first looked at the relation between baseline PROMIS QOL measures and percentage of later administered surveys completed. Baseline PROMIS Social Functioning T score was significantly correlated with completion rates, with patients with higher baseline social functioning having higher overall survey completion rates over the course of treatment and follow up (ρ = 0.223, P < .05). Other baseline PROMIS items were not significantly correlated with overall survey completion rates (Figure 4B).

Then, we looked at early treatment symptoms (ie, symptoms over the first 6 weeks of treatment) using PRO-CTCAE to determine symptoms that might associate with a decline in survey completion later in treatment. Patients that reported higher than a level 2 (moderate or more) mean Abdominal Pain, Blurry Vision, Decreased Appetite, Dry Eyes, Dry Mouth, Urinary Frequency, Headache, Hot Flashes, Insomnia, Joint or Muscle Pain, Nausea, Palpitations or Shortness of Breath had lower later completion percentages of the surveys received (Figure 4C).

Discussion

The ePRO implementation was an agile roll-out that included tablet integration, staff re-engagement, training, and technical integration with the clinical trial management system. While the iterative process makes it difficult to understand the individual impact of each intervention, the various interventions as a whole improved patient engagement and survey completion. Our work highlights the benefits of an agile and iterative implementation process that began with a pilot at a few sites. We listened and responded to the CRCs at each site, patients that are active in the trial, and our staff, which helped us troubleshoot more rapidly than a traditional linear implementation.

A central pillar of our ePRO implementation protocol in I-SPY2 was collaboration between all stakeholders throughout evolution and rollout of the system. During the development of the ePRO system, as showcased by the PRO working group’s collaboration and feedback cycle, detailed planning and testing allowed for an automated system to function within a complex workflow (Figure 1B). Once the system was active in the pilot and across sites, the Central ePRO Coordinators were pivotal to initiating and implementing feedback within the PRO working group, acting as liaisons between CRCs and other stakeholders including patient advocates, clinical research experts, and principal investigators. This paper provides analysis on the UCSF pilot study, followed by an analysis of data from all additional I-SPY2 sites, using questionnaire completion rates as a proxy for successful implementation and workflow adaptation.

In medicine, new technologies are effective only to the extent that they are integrated into clinical workflows. Although we designed our ePRO platform, in part, with the goal to improve data collection for CRCs, it became evident during the pilot phase that technology alone would not achieve this aim: it also required a strategy for implementation that enabled integration within the trial and clinic setting. According to Jensen et al the most effective existing ePRO systems share user-friendly features such as at-home assessments and integration into larger electronic data capture systems.⁴ Our implementation mirrors this approach to better integrate with patient lives, by increasing assessment flexibility.

The internal pilot study at UCSF was a pivotal phase for determining the best way to implement the ePRO workflow at other sites. Through direct feedback from the UCSF CRCs, we learned that the shift in modality of QOL data collection was viewed as a large-scale culture shift within the trial; after ten years of optional paper-based QOL data collection, the transition to mandatory electronic PROs was challenging for CRCs and had a considerable impact on workflow. Unsurprisingly, this was the primary reason that CRCs requested additional support from the ePRO team.

During the pilot, we observed that the screening period was an especially work-intensive time of the trial for CRCs and medical providers, which made it challenging to ensure that the ePRO Screening questionnaire was completed by the patient prior to the initiation of study treatment. While it was important for future data analysis to collect screening information from each patient, a patient’s completion of the screening questionnaire also emerged as a significant predictor of long-term engagement through treatment and follow-up, and thus, was foundational to the creation of our training workshops and materials. In addition, to ensure the critical completion of the Screening questionnaire, we integrated tablets into our workflow, which allowed CRCs to provide patients with real-time access to their questionnaires in the clinic. Of note, this intervention was introduced during the COVID-19 pandemic, which resulted in varying rates of utility depending COVID-19 regulations on in-person work for CRCs. The vital importance of the screening survey.

Given the complex nature of this platform trial and the diverse clinical structures at each of the 27 sites across the United States, any change in trial procedures, and in this case, QOL data collection, require a culture shift. When implementing ePRO at non-UCSF sites, we considered the unique needs of each site when adapting their workflows to adjust for both the new EDC and the new PRO platform.

After our initial training session for all I-SPY2 sites, we found that each site required more personalized engagement. The hierarchical color system allowed us to work with sites on an individual basis and triage the sites who needed more support, while simultaneously providing incentive for sites to integrate ePRO into their existing workflow. CRC bandwidth and staff turnover were factors that reduced success, and thus we could anticipate and work with sites to overcome these adverse factors. Given that consistent collaboration between CRCs and the ePRO team was key to the success of the pilot, we had to establish an accessible communication pathway between non-UCSF sites and the ePRO team to ensure optimal support for struggling sites. Given the limited resources of CRCs, we found that “screen fail” data was not indicative of a site’s ability to use the ePRO platform. Therefore, we elected not to demote sites in color status when patients who did not continue past Screening (“screen-fail” patients) failed to complete the Screening questionnaire.

Despite the progress made in our study, it had some limitations. Since UCSF investigators, staff, and leadership are quite invested in the ePRO efforts, they are expected to adhere better to the interventions we used to increase completion rates among patients. In addition, throughout the implementation process, we observed that increasing familiarization with the survey protocol, systems, tablet integration, and heavy staff engagement led to improved completion. There were sites, however, that had less staff engagement, were newer to I-SPY and its protocols, and thus their completion rates were less linear compared to others. To understand and overcome some of these barriers, we used a quality improvement cycle, by creating metrics, tracking progress and developing interventions to improve, and provide feedback in a continuous manner.^5,6 Site-by-site analysis and overall questionnaire completion rates informed us about site compliance and adoption of the ePRO platform. The 18-month internal pilot analysis taken alongside the all-sites early and late completion analysis show the importance of allowing sites an adjustment period to acclimate to the ePRO platform and adapt their workflows accordingly. However, it is important to note that changes in early and late completion rate discussed above cannot exclusively be attributed to the various interventions, as comfort with OpenClinca also played a large role in CRC ability to consistently engage with the ePRO platform.

Contrary to our hypothesis that younger cohorts would be more comfortable using technology and therefore better able to complete electronic-based questionnaires, ePRO completion percentages were not significantly different between younger and older patients. While the median age of our patient population in I-SPY2 is lower than the median age of patients with breast cancer, patients that enroll in clinical trials are usually significantly younger than the overall patient population.⁷ It is often assumed that older women are less literate with technology but increasingly, people of all ages are now reliant on and facile with using technology if the software and tools are simple. Furthermore, race and ethnicity did not significantly impact survey completion rates either. While earlier reports have specifically found age and other demographics as co-variates impacting survey completion, our implementation and survey design was accessible across patient demographics.^8,9

Previous studies have described the power of PRO in clinical care, providing a more holistic picture of a patient’s health.^10–13 In addition, electronic PRO has many invaluable benefits, including real time data availability, automatic triggers to supportive care, more comprehensive and holistic evaluation of novel therapeutics in clinical trials reduced patient as well as CRC burden, and fewer missing data compared to paper-based reports.^4,14–18 However, as reported in this paper, ePROs can also help in real time prediction and identification of patients. In this paper, we reported metrics based on early patient reports from survey, including lower baseline social functioning and early presence of symptoms such as abdominal pain, blurry vision, decreased appetite, dry eyes, dry mouth, urinary frequency, headache, hot flashes, insomnia, joint or muscle pain, nausea, palpitations or shortness of breath that might interfere with a patient’s engagement with the ePRO platform. The early indicators in this study can help identify patients that might benefit from increased outreach without waiting for patients to miss surveys before personalized reminders can be sent. To address this, we started a “Central Calling” pilot at UCSF in March of 2022, the goal of which is to call patients and offer supportive reminders to complete their scheduled ePRO questionnaires. In addition, the decline in engagement as a patient gets further out from the treatment into follow-up highlights that especially during the post-treatment and -surgery follow-up period, it is understandable that patients may also benefit from additional reminders to complete their questionnaires. The results of this pilot project are forthcoming yet promising in regard to the feasibility of offering this service at additional I-SPY2 sites. In future studies, we will continue to iterate and utilize on what we have learned on the administration of technological, workflow, and behavior change interventions and their associated impact on questionnaire completion rates.

Author contributions

Amrita Basu, Dawn L. Hershman, and Laura Esserman conceptualized the study. Anna Northrop, Anika Christofferson, Saumya Umashankar, Adam Asare, and Amrita Basu performed the data curation. Michelle Melisko, Thelma Brown, Diane Heditsian, Susie Brain, Carol Simmons, Paolo Castillo, Tina Hieken, Kathryn J. Ruddy, Candace Mainor, Anosheh Afghahi, Sarah Tevis, Anne Blaes, Irene Kang, Adam Asare, Laura Esserman, Dawn L. Hershman, and Amrita Basu developed the methodology. Dawn L. Hershman, Amrita Basu, Laura Esserman, and Adam Asare provided project supervision and program administration. Laura Esserman, Amrita Basu, and Adam Asare acquired funding. Adam Asare developed the software for the study. Adam Asare, Laura Esserman, and Amrita Basu provided the resources. Anna Northrop, Anika Christofferson, and Saumya Umashankar performed the analyses. Anna Northrop, Anika Christofferson, and Saumya Umashankar developed the visualizations. Anna Northrop, Anika Christofferson, Saumya Umashankar, and Michelle Melisko wrote the paper. Saumya Umashankar, Michelle Melisko, Dawn L. Hershman, Amrita Basu, and Laura Esserman edited the manuscript. All authors reviewed and edited the manuscript.

Supplementary data

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

Research reported in this manuscript was supported by the National Cancer Institute of the National Institutes of Health under award number P01CA210961. The authors wish to acknowledge the generous support of the study sponsors, Quantum Leap Healthcare Collaborative (QLHC). The authors sincerely appreciate the ongoing support for the I-SPY2 Trial from the Safeway Foundation, the William K. Bowes, Jr. Foundation, Give Breast Cancer the Boot, QLHC, and the Breast Cancer Research Foundation.

Conflicts of interest

Co-author T.H. reports research support from Genentech and SkylineDX BV. A.B. reports grant support from the University of Minnesota Cancer Center (P30CA077598) and by National Institutes of Health grants R01CA267977, 1R01CA277714-01, R21AG080503, and R13CA278261. I.K. was a consultant for Gilead, Caris Life Sciences, and Daiichi Sankyo. A.A. was an employee of Quantum Leap Healthcare Collaborative. L.E. was an uncompensated board member for Quantum Leap Healthcare Collaborative; payment from UpToDate; member of Blue Cross Blue Shield medical advisory panel and travel reimbursement from Blue Cross Blue Shield. All other authors have no potential conflicts to report.

Data availability

Subject-level data for this study is available to approved investigators completing a request form available at: https://www.ispytrials.org/collaborate/proposal-submissions.