Table 1.

Summary of identified issues and actions taken to refine the intervention following field usability testing

Theme/area	Issue identified	Action taken
Procedures for remote access and symptom report completion
Graphs	Patients have the option to report ‘other’ free text symptoms at the end of the symptom report. However, the graphs displaying these symptoms looked odd when symptoms were not reported regularly.	Decision made to remove these graphs as they did not add much value and were confusing for patients.
	The headings on the graphs (symptom names) did not always correspond with those on the questionnaire	The research team to ensure labelling kept consistent.
System usability	Patients have access to a link at the end of their symptom report (‘email your feedback’) to email their self-management advice to themselves. However, patients expected this link to enable them to provide the research team with feedback on the eRAPID system.	Wording changed this from ‘email your feedback’ to ‘send this information to your email address’.
Symptom report	Patients wanted to provide additional information about symptoms, such as when they experienced them or the type of pain they had.	Two changes were implemented: If a patient reported ‘severe’ symptoms, they were then asked a branching question to determine if it was a current problem or a problem that had now resolved. A free text box was added to the pain question so that patients could provide information about the site of pain.
	One patient felt there were too many questions and that they were not all relevant. She suggested that we should add an option for patients to say ‘I feel fine’ or ‘My symptoms haven’t changed’.	After discussion with the research and wider project management teams, we decided against implementing this, as it would not be as useful for clinical practice.
Practicalities of completion	The most common reasons for not completing were forgetting, feeling too unwell, not experiencing symptoms or not realising they should complete weekly.	Implementation of an automated reminder system to send patients weekly reminders via text or email. In patient training, researchers will emphasise the importance of completing weekly, even if they are not experiencing symptoms.
Self-management advice	One patient queried what to do if symptoms are not improving when you are following the advice and suggested we encourage patients to talk to their clinical team if this is the case.	This advice was added to the self-management feedback
	Suggestion to add some additional links to well used external websites to make it a more complete resource.	After discussion, it was decided not to add links for external websites, as we would not be able to ensure that they were always up to date, and patients are directed to these sites by the clinical team.
	Add specific advice on achy veins and hot flushes.	Self-management advice was added for these issues.
Notifications	Severe symptom notifications were being triggered for patients reporting retrospective problems (due to item framing asking patients to report symptom experience within the last 7 days).	A branching question was added to ask patients ‘Is this a current problem?’ if a severe symptom was reported. A notification would then be sent only if patients answered yes to this question.
	Several notifications were triggered for physical activity when patients felt it was not warranted.	Following discussion with clinical staff the threshold was increased for this item. In addition to the branching question regarding whether the symptom is current, a second branching question was added for this symptom to ask if patients had help at home.
Staff notifications
Notifications	Clinical staff suggested that it may be helpful to have the facility to comment on a notification in the EPR to let other staff know it had been actioned (eg, by phoning the patient).	This facility was added so staff could mark a notification as ‘responded’ and make an annotation.
Accessibility and interpretability of eRAPID symptom report data for staff
User interface	Several clinical staff members commented that it would be very useful for them to be able to see chemotherapy cycles on patients’ symptom report graphs.	Red triangle added on the graphs to denote date of chemotherapy cycle delivery.
	Where a patient score was 0, it looked like the item had not been completed.	This was only an issue for the patients’ first completion (which showed as a bar graph, rather than a line graph), these were amended so that it was clearer when symptoms were scored as 0.
	The line graphs depicting patient symptom reports had a red line to show where symptoms became severe and a notification would be triggered. This was confusing for staff.	Red line showing severity levels was removed.
	Staff found the symptom reports less useful when patients were not completing regularly. Patients were not always aware if staff were using their symptom reports or not.	In future training staff were asked to encourage patients to complete regularly and explicitly refer to and use the results in consultations.