Online tool for monitoring adverse events in patients with cancer during treatment (eRAPID): field testing in a clinical setting

Abstract

Objectives

Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice (eRAPID) is an online system developed to support patient care during cancer treatment by improving the detection and management of treatment-related symptoms. Patients can complete symptom reports from home and receive severity-based self-management advice, including notifications to contact the hospital for severe symptoms. Patient data are available in electronic records for staff to review. Prior to the commencement of a randomised controlled trial (RCT), field testing of the intervention was undertaken to troubleshoot practical issues with intervention integration in clinical practice.

Design

Observational clinical field testing.

Setting

Medical oncology breast service in a UK cancer centre.

Participants

12 patients receiving chemotherapy for early breast cancer and 10 health professionals (oncologists and specialist nurses).

Intervention

Patients were asked to use the eRAPID intervention and complete weekly online symptom reports during four cycles of chemotherapy. Clinical staff were invited to access and use patient data in clinical assessments.

Analysis

Descriptive data on the frequency of online symptom report completion and severe symptom notifications were collated. Verbal and written feedback was collected from patients and staff and semistructured interviews were conducted to explore patient experiences. Interviews were transcribed and analysed thematically.

Results

The testing ran from January 2014 to March 2014. Feedback from patients and staff was largely positive. Patients described eRAPID as ‘reassuring’ and ‘comforting’ and valued the tailored management advice. Several changes were made to refine eRAPID. In particular, improvement of the clinical notification, patient reminder systems and changes to patient and staff training.

Conclusions

The field testing generated valuable results used to guide refinement of eRAPID prior to formal intervention evaluation. Feedback indicated that eRAPID has the potential to improve patients’ self-efficacy, knowledge and confidence with managing symptoms during treatment. A large-scale RCT is underway with data collection due to finish in October 2018.

Strengths and limitations of this study.

• The strengths of this field usability testing include the mixed methodological approach (combining qualitative and observational data) and the involvement of both patient and staff representatives.

• This type of embedded clinical testing is vital for determining how new interventions work in practice.

• The results are limited by the focus on one cancer group.

• An ongoing randomised controlled trial will assess the impact of the Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice online intervention on patients with cancer, oncology staff and health services.

Introduction

Systemic cancer treatment is associated with a range of side effects which can negatively impact patients’ quality of life (QOL) and become life threatening.¹ As patients typically receive chemotherapy in outpatient settings, they are largely required to self-monitor symptoms at home. Patients can lack confidence in making decisions between obtaining clinical support or self-managing² and can delay seeking medical advice,^{3 4} heightening the risk of symptom escalation and hospital admissions.⁵ Conversely subgroups of patients may routinely contact the hospital for reassurance in relation to mild side effects.²

There is growing evidence that the utilisation of patient-reported outcome measures during cancer treatment can aid the timely identification of physical and psychosocial needs, facilitate patient–doctor communication and assist decision-making.^6–10 There has been a drive to develop electronic systems to allow remote real-time patient monitoring during cancer.^11–13 Positive patient benefit (including QOL and survival) was recently reported in a US trial of an online system for metastatic cancer treatment.¹⁴

We developed an online intervention for supporting patient care during and after cancer treatment. Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice (eRAPID)¹⁵ is web based and accessible from home or mobile device, for patients to complete symptom reports and receive severity-based advice. Recommendations for self-management are provided for milder symptoms and advice for when to contact the hospital for severe issues. A graphing feature allows patients to review personal symptom data over time. The system includes a facility to notify healthcare teams via email when a severe symptom is reported. Patient-reported data are transferred in real time to be accessed by health professionals through the hospital electronic patient records (EPRs) for use in routine consultations and assessments.

Following recommended usability principles of agile development,¹⁶ we involved staff and patient representatives throughout the eRAPID developmental processes, using an iterative design approach.¹⁷ Results from the technical usability testing are reported elsewhere.^{15 18} Here, we describe the field usability testing phase where staff and patients used eRAPID in a real-life clinical setting to troubleshoot practical issues not identified by standard usability testing^{19 20} and allow streamlining of eRAPID clinical integration prior to commencing a formal randomised controlled trial (RCT).²¹

Aims

For patients, the aims were to ensure:

• System training was sufficient and feasible.

• Routine (weekly) online symptom report completions were acceptable.

• Self-management advice and severe symptom notifications were useful and appropriate.

For clinical staff, the aims were to ensure:

• System training provided was sufficient and feasible.

• Symptom report data were easy to access in the EPR and comprehensible.

• Severe symptom notifications were correctly activated.

In addition, we wanted to assess the overall reliability of the Information Technology (IT) underpinning eRAPID.

Methods

Patient and public involvement

The eRAPID research programme has involved patient representation and collaboration at each stage from conception, funding application (involvement of a patient and public involvement coapplicant) and research delivery (membership on project management and steering committees). The eRAPID intervention was designed with substantial input from patient representatives from our dedicated Research Advisory Group (RAG). RAG members contributed to initial usability testing of the patient facing aspects of the IT system and advised on the content of the self-management advice.²² This paper describes the subsequent phase of usability work, where the intervention was taken for the first time for field testing in a clinical context with patients receiving chemotherapy and their associated care teams.

Clinical setting

The field usability testing was conducted in the breast medical oncology service at St James’s University Hospital, Leeds, UK. The Leeds Teaching Hospitals NHS Trust Research & Innovation Department approved the exercise as a service evaluation. Procedures were undertaken in line with Data Protection²³ and Good Clinical Practice guidelines.²⁴

Patient eligibility and identification

Patient eligibility criteria were (1) early breast cancer diagnosis, (2) starting at least four planned cycles of adjuvant/neoadjuvant systemic treatment, (3) internet access at home and (4) proficient English (to understand symptom assessments and self-management advice). Patients were identified by clinical staff and eRAPID was introduced by an oncologist or clinical nurse specialist (CNS). Interested patients were given an information sheet and passed to the eRAPID team for further information.

eRAPID demonstration and training

The eRAPID system is described in detail elsewhere.¹⁵ See figure 1 for a system overview.

Figure 1.

Overview of eRAPID system illustrating the flow of data from inside and outside the hospital. EPR, electronic patient record; eRAPID, Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice; NHS, National Health Service.

Patient symptom report and training

Participating patients received written information to take home and the researcher arranged to meet them at their first chemotherapy visit, where unique eRAPID login details were provided alongside a user manual. Patients were asked to complete the online eRAPID symptom report weekly (or more frequently if preferred) throughout four chemotherapy cycles (approximately 12 weeks). The online symptom report contained items from the locally devised patient-reported adverse event¹⁸ item bank based on the Common Terminology Criteria for Adverse Events grading system.²⁵ Working in collaboration with the breast oncology staff, 12 core items were chosen for the main report (including pain, fatigue, physical activity, bowel function, sleep, temperature, chill, sore mouth and appetite). There was also the option for patients to add details on additional issues via a drop-down list of further symptoms and a free text option.

Clinical staff training

The breast CNSs were shown how to access eRAPID symptom report data in the EPR and given a one-page instruction sheet. They were also added to a mailing list to receive severe symptom notifications and encouraged to contact patients where feasible. Oncologists were trained on a needs-driven basis, receiving a demonstration and a one-page instruction sheet immediately prior to an eRAPID patient consultation.

Evaluation methods

Patient evaluation

Information on system acceptability and general feedback were collected through a number of methods:

• The number of full symptom report completions and adherence to the weekly completion schedule.

• Patients were provided with email and telephone contact details for the research team and a researcher met with the patient at their routine hospital appointments to check progress. The content of these communications was documented and collated.

• Patients were asked to complete brief feedback in the user manual covering ease of system use and general comments or recommendations.

• Patients were interviewed at the end of the testing. Semistructured interviews explored views on the technical practicalities, relevance/impact of the self-management advice and staff use of the reports and impact on medical management.

Clinical staff evaluation

Testing of the eRAPID system from the professionals’ perspective involved:

• Completion of brief written feedback forms to record use of the eRAPID data and any impact on the consultation.

• Direct observation of a subset of consultations where eRAPID information was available for staff. The researcher sat in the room and took field notes to describe how staff used eRAPID data.

• Details of any severe symptom notifications sent to staff during the 12-week assessment were documented along with any action taken.

• Ad hoc verbal feedback from staff was also documented by researchers throughout the 12-week assessment.

Evaluating the reliability of IT processes

Any IT issues reported by researchers, patients or staff during the assessment period were logged along with the action taken.

Iterative refinement of eRAPID

Throughout testing the research team collated feedback and identified issues were regularly fed back to the eRAPID project management team (consisting of oncologists, nurses, health informatics experts, patient representatives and researchers). The team decided how issues should be resolved and where eRAPID could be improved for the future RCT. A full report was prepared at the end of the testing, documenting all identified issues and actions taken.

Analysis

Patient and clinical staff evaluation

Descriptive accounts of the number of completed eRAPID symptoms reports were created along with the frequency of severe symptom notifications. Verbal feedback, comments from the user manuals, written feedback and notes taken during clinic observations were assimilated and categorised into themes.

Interviews were audio recorded, transcribed verbatim and managed in NVivo V.9. Initially a pragmatic approach was employed where any important issues raised in interviews were taken to the project management team to guide any immediate action. Interview data were subsequently fully coded and analysed thematically.²⁶

Results

Participants

Patients

Testing took place between January and March 2014 with 12 patients (mean age=47.5 years, SD=10.3, range 33–73 years) (see figure 2).

Figure 2.

Overview of patients approached and participating in the eRAPID field usability testing network. eRAPID, Electronic patient self-Reporting of Adverse-events: Patient Information and aDvice.

Clinical staff

Ten members of the breast care team participated including two adjuvant breast CNSs and eight doctors (four senior oncology consultants, four oncology trainees). Patients typically saw a CNS routinely throughout the 12 weeks and had one appointment with an oncologist before their third chemotherapy cycle.

Evaluation of eRAPID

Figure 3 shows a summary of data collected from staff and patients.

Figure 3.

Summary of evaluation data collected from patients and staff. CNSs, clinical nurse specialists.

Compliance with weekly symptom reports

Over the testing period 104 full symptom reports were completed, 42% (5/12) of patients completed the report 11–13 times, 33% (4/12) completed 7–9 times and 25% (3/12) completed 4–6 times. Average adherence to weekly completion (ie, actual/expected completions per patient) was 63% (range 33%–92%).

Interviews revealed the most common reason for non-completion of the symptom report was simply forgetting. Most patients were in favour of a text or email reminder (n=8). Patients reported not completing the report when very unwell, stating it was not a priority. Others were unsure how often they should be completing. Feedback from the user manuals and interviews demonstrated that all the patients found eRAPID easy to use and did not report problems locating, logging in or using the system. A couple of minor suggestions were made for improvement, which were subsequently addressed (see table 1).

Table 1.

Summary of identified issues and actions taken to refine the intervention following field usability testing

Theme/area	Issue identified	Action taken
Procedures for remote access and symptom report completion
Graphs	Patients have the option to report ‘other’ free text symptoms at the end of the symptom report. However, the graphs displaying these symptoms looked odd when symptoms were not reported regularly.	Decision made to remove these graphs as they did not add much value and were confusing for patients.
	The headings on the graphs (symptom names) did not always correspond with those on the questionnaire	The research team to ensure labelling kept consistent.
System usability	Patients have access to a link at the end of their symptom report (‘email your feedback’) to email their self-management advice to themselves. However, patients expected this link to enable them to provide the research team with feedback on the eRAPID system.	Wording changed this from ‘email your feedback’ to ‘send this information to your email address’.
Symptom report	Patients wanted to provide additional information about symptoms, such as when they experienced them or the type of pain they had.	Two changes were implemented: If a patient reported ‘severe’ symptoms, they were then asked a branching question to determine if it was a current problem or a problem that had now resolved. A free text box was added to the pain question so that patients could provide information about the site of pain.
	One patient felt there were too many questions and that they were not all relevant. She suggested that we should add an option for patients to say ‘I feel fine’ or ‘My symptoms haven’t changed’.	After discussion with the research and wider project management teams, we decided against implementing this, as it would not be as useful for clinical practice.
Practicalities of completion	The most common reasons for not completing were forgetting, feeling too unwell, not experiencing symptoms or not realising they should complete weekly.	Implementation of an automated reminder system to send patients weekly reminders via text or email. In patient training, researchers will emphasise the importance of completing weekly, even if they are not experiencing symptoms.
Self-management advice	One patient queried what to do if symptoms are not improving when you are following the advice and suggested we encourage patients to talk to their clinical team if this is the case.	This advice was added to the self-management feedback
	Suggestion to add some additional links to well used external websites to make it a more complete resource.	After discussion, it was decided not to add links for external websites, as we would not be able to ensure that they were always up to date, and patients are directed to these sites by the clinical team.
	Add specific advice on achy veins and hot flushes.	Self-management advice was added for these issues.
Notifications	Severe symptom notifications were being triggered for patients reporting retrospective problems (due to item framing asking patients to report symptom experience within the last 7 days).	A branching question was added to ask patients ‘Is this a current problem?’ if a severe symptom was reported. A notification would then be sent only if patients answered yes to this question.
	Several notifications were triggered for physical activity when patients felt it was not warranted.	Following discussion with clinical staff the threshold was increased for this item. In addition to the branching question regarding whether the symptom is current, a second branching question was added for this symptom to ask if patients had help at home.
Staff notifications
Notifications	Clinical staff suggested that it may be helpful to have the facility to comment on a notification in the EPR to let other staff know it had been actioned (eg, by phoning the patient).	This facility was added so staff could mark a notification as ‘responded’ and make an annotation.
Accessibility and interpretability of eRAPID symptom report data for staff
User interface	Several clinical staff members commented that it would be very useful for them to be able to see chemotherapy cycles on patients’ symptom report graphs.	Red triangle added on the graphs to denote date of chemotherapy cycle delivery.
	Where a patient score was 0, it looked like the item had not been completed.	This was only an issue for the patients’ first completion (which showed as a bar graph, rather than a line graph), these were amended so that it was clearer when symptoms were scored as 0.
	The line graphs depicting patient symptom reports had a red line to show where symptoms became severe and a notification would be triggered. This was confusing for staff.	Red line showing severity levels was removed.
	Staff found the symptom reports less useful when patients were not completing regularly. Patients were not always aware if staff were using their symptom reports or not.	In future training staff were asked to encourage patients to complete regularly and explicitly refer to and use the results in consultations.

Severe symptom notifications

Eight severe symptom notifications were activated, seven were not appropriate due to the framing of the symptom items which asked patients to report symptom experience within the last 7 days. This led to occasions where severe symptoms were being retrospectively reported with notifications activated for resolved problems. One patient found this experience alarming and stopped using eRAPID as a consequence.

It brought the alert up, and the hospital rang and thought I might possibly need an admission, I must admit that scared me a little bit…I said well no actually, these symptoms were a few days ago and now I’m absolutely fine… it were fantastic that they rang so quickly and I think it’s a great system for that, but I just thought oh no, I don’t want to go to hospital (Female, 44)

In addition, patients sometimes felt that notifications were not warranted for the symptom severity experienced, in particular, those for low physical activity:

It more or less panicked you a bit and said contact the hospital immediately… Because I still think I’d classed it correctly. It didn’t warrant an ambulance at the door or anything like that. (Female, 47)

However, the notifications worked well for the one patient who experienced an injection site reaction for several days. On completing the symptom report, she followed the advice to contact the hospital:

I wasn’t sure whether I should, so that advice was good. It will be good for people like that who are borderline. (Female, 73)

Following testing, the symptom report was refined to accommodate the notification issues identified (see table 1).

Thematic analysis of patient interviews

The following themes were identified:

Increasing knowledge and confidence

The self-management advice empowered patients by providing information and support to personally manage symptoms. Patients felt confident doing this, in the knowledge that the system provided a ‘safety net’:

It’s like a life line when you feel isolated when you’re at home and feeling poorly…you can have a lot of questions or problems regarding your illness and with one click they can be answered and absorbed within minutes. (Female, 49).

I would recommend it to anyone. It’s like a safety net for you and gives you the help to keep on going on through your treatment (Female, 73)

I think it does make you feel a bit happier—if you read something on there that says well, you will feel like this but you can do this, this and this… you’re happy with that (Female, 50)

Supporting decision-making

Patients felt that using eRAPID helped reduce their worry by aiding decision-making helping them feel more knowledgeable about when to self-manage and when to contact the hospital:

I’m a bit of a worrier and I think ‘shall, shan’t I, am I over-reacting?’ But I think that then would confirm to people that yeah, you should really ring the hospital so… it’s just like a little bit of extra home support isn’t it really? (Female, 33)

Coping strategy

Some patients found the symptom graphing feature useful for understanding patterns and this could be both reassuring and motivating.

For me personally, I just think I can’t do this anymore, I don’t like this, this is awful, and I find that I’m very disheartened and I can’t see an end to it……But then when you look at the graphs, you can think, but I did get better…and my mouth has got better, and my diarrhoea has stopped and… you can see that there is a pattern and that it will get better. It makes me feel better. (Female, 60)

Staff evaluation

Notifications for severe symptoms

Clinical staff agreed that notifications for retrospectively reported severe symptoms were not relevant and that some of the notifications sent for low physical activity were unwarranted. When this issue was identified, email notifications were redirected to the eRAPID research nurse for the remainder of the field testing, who liaised with patients and clinical staff where needed.

eRAPID patient symptom report data

The staff feedback forms indicated that symptom report data were easy to access and interpret and useful for identifying issues/problems for discussion (n=6), confirming knowledge of the patient’s problems, (n=5), providing additional information (n=5) and contributing to management (n=3). Several staff commented that it would assist interpretation of symptom reports if the EPR graphical displays included dates of chemotherapy delivery. In addition, staff commented that symptom information was most useful where patients had routinely provided reports throughout treatment.

The consultation observations confirmed that staff could easily access the symptom reports in the EPR but there were variations in utilisation. Some staff viewed the data but did not explicitly mention this to patients whereas others used it as a point of reference to guide the consultation and made this clear to patients.

Reliability of IT processes

The IT processes were largely stable. The notification system was reliable, with the patient and staff severe symptom notifications activated as expected. The patient symptom reports became temporarily unavailable to staff at one point. The problem was reported to the team and was resolved by the IT manager that day.

Refinement of eRAPID intervention and processes for integration

Following feedback from staff and patients, several improvements were made to streamline the integration of eRAPID into the clinical setting (table 1).

Discussion

The aim of this field usability testing was to observe end users (staff and patients) use of eRAPID in a real-life clinical setting in order to troubleshoot practical issues which may not be identified through standard usability testing.^{19 20} Feedback received from both patients and staff was positive and demonstrated the system was well received but also led to important modifications and improvements. The process allowed streamlining of intervention integration into clinical practice prior to formal evaluation.²¹

The majority of the notifications triggered for severe symptoms were for resolved symptoms patients reported retrospectively. This led to two key adjustments to the system prior to the RCT to avoid ‘false’ notifications and limit unnecessary patient worry and clinical burden. First, we added a branching question to allow patients reporting symptoms to provide further clarification on whether the symptom was ongoing or had been resolved. Second, the physical activity severity threshold was raised as both patients and clinical staff felt the original setting was too low. This was off-putting to patients and encouraging unwarranted hospital contact. For additional safeguarding, a further branching question was also added to this item to determine if patients reporting problems had help/support at home and assist with the identification of more vulnerable individuals.

Patients found eRAPID easy to use but many forgot to routinely complete the weekly report. As a consequence, the proposed text message/email reminder system was subsequently established. Patients reported valuing the self-management advice, particularly specific advice about when to contact the hospital and several patients described the system as ‘reassuring’ and a ‘safety net’. The testing highlighted the reciprocal relationship between patient and staff engagement in the system. Although staff felt the symptom reports were most valuable when routinely completed by patients, they did not always explicitly mention using the data in consultations. Moving forward in the RCT, we conducted a series of one-to-one and group training sessions with relevant staff involved in chemotherapy delivery and assessments (oncologists, CNSs, preassessment nursing teams). The sessions have included didactic elements (describing the eRAPID developmental work and the evidence supporting the use of using patient-reported data in clinical practice) and practical demonstrations of how to access patient-reported data. More recently an online training package was also created (accessible via a hyperlink in the EPR) which allows staff to view information as required. This online resource includes practical refreshers on accessing patient symptom reports along with interactive case studies that demonstrate how the data can be interpreted and used. There is emphasis within all the training formats on the importance of making overt reference to symptom reports with patients in clinical encounters to endorse the value of patient-reported data and encourage ongoing completions.

The interviews revealed the intervention could have the potential to increase patient self-efficacy and engagement with the management of their care. A recent systematic review has demonstrated the importance of self-efficacy in managing pain, symptoms and function in patients with cancer.²⁷ In addition, high levels of patient activation (how engaged a patient is in their own healthcare) are associated with an array of improved health behaviours and health outcomes^28–30 while lower levels of activation are associated with higher use of hospital resources.³¹

In the large-scale eRAPID RCT, the main outcomes focus on patient QOL and clinical process data (contacts with the hospital, emergency admissions) but we will also explore psychological variables that may help us more fully understand how patients can benefit from eRAPID.²¹ Specifically, we will explore the relationship between patients’ self-efficacy, patient activation³² and utilisation of the eRAPID system, self-management advice and symptom graphs. The WebChoice system in Norway has demonstrated that enabling patients to self-manage can be more beneficial for patients than symptom reporting alone.³³

This field usability testing had some limitations. We only had the capacity (in terms of both staffing resources and time restrictions) to evaluate the system with patients with early breast cancer. These patient groups are relatively young compared with other adult cancer groups and are more likely to be digitally agile. However, internet access continues to increase³⁴ and previous work has indicated that electronic systems are acceptable in other cancer groups.11 15 35–37 The RCT evaluates eRAPID in a broader population, specifically patients receiving chemotherapy for breast, gynaecology or colorectal cancer have been recruited from both adjuvant and metastatic treatment pathways.²¹

In summary, the field testing helped endorse the practical potential of eRAPID for supporting patient care but importantly uncovered issues which would not have been identified with standard usability testing alone. This was an invaluable exercise prior to the commencement of the ongoing RCT (data collection due to be completed in October 2018) which will evaluate the potential benefits of eRAPID for patients, staff and the National Health Service.