Abstract
Background
Significant adverse events (AE) during cancer therapy disrupt treatment and escalate to emergency admissions. Approaches to improve the timeliness and accuracy of AE reporting may improve safety and reduce health service costs. Reporting AE via patient reported outcomes (PROs), can improve clinician–patient communication and making data available to clinicians in ‘real-time’ using electronic PROs (ePROs) could potentially transform clinical practice by providing easily accessible records to guide treatment decisions. This manuscript describes the development of eRAPID (electronic patient self-Reporting of Adverse-events: Patient Information and aDvice) is a National Institute for Health Research-funded programme, a system for patients to self-report and manage AE online during and after cancer treatment.
Materials and methods
A multidisciplinary team of IT experts, staff and patients developed using agile principles a secure web application interface (QStore) between an existing online questionnaire builder (QTool) displaying real-time ePRO data to clinicians in the electronic patient record at Leeds Teaching Hospitals NHS Trust. Hierarchical algorithms were developed corresponding to Common Terminology Criteria for Adverse Events grading using the QTool question dependency function. Patient advocates (N = 9), patients (N = 13), and staff (N = 19) usability tested the system reporting combinations of AE.
Results
The eRAPID system allows patients to report AE from home on PC, tablet or any web enabled device securely during treatment. The system generates immediate self-management advice for low or moderate AE and for severe AE advice to contact the hospital immediately. Clinicians can view patient AE data in the electronic patient record and receive email notifications when patients report severe AE.
Conclusions
Evaluation of the system in a randomised controlled trial in breast, gynaecological and colorectal cancer patients undergoing systemic therapy is currently underway. To adapt eRAPID for different treatment groups, pilot studies are being undertaken with patients receiving pelvic radiotherapy and upper gastrointestinal surgery.
ISRCTN88520246.
Keywords: adverse events, CTCAE, electronic patient-reported outcomes, electronic patient records, integration, patient self-management
Key Message
This article describes the secure real-time integration of electronic patient-reported adverse event (AE) data into the electronic patient record system (EPR) in a leading UK Cancer Centre. eRAPID (electronic patient self-Reporting of Adverse events Patient Information and aDvice) enables AE reporting from home, gives self-management advice for low or moderate AE and for severe AE immediate advice to contact clinicians who view AE data in EPR and receive an email alert.
Introduction
Significant adverse events (AE) during cancer therapy can disrupt treatment and impair quality of life (QOL) [1, 2]. In the UK, The National Confidential Enquiry Report into Patient Outcome and Death highlighted increased emergency department admissions and limited support for patients experiencing chemotherapy-related AE [1]. An audit of the acute oncology service in Leeds Hospitals NHS Trust (LTHT) found patients often delay reporting even severe symptoms [3]. Approaches to improve care and avoid preventable emergency admissions may contribute to improving cancer patients’ safety and QOL and reduce health costs.
Reporting symptoms and side-effects via patient-reported outcomes (PROs), can improve clinician–patient communication and the accuracy of clinician-reported AE [4–6]. The last 20 years has seen an increase in electronic PRO systems [7] providing timely and effective solutions to capture and utilise PRO data in a range of clinical contexts [7, 8].
Patients can report QOL and chemotherapy AE via PROs in clinical practice using home internet [9] or mobile phones (ASyMS) [10]. Successful implementation of electronic PRO systems include: PatientViewpoint, [11], the Computer-based Health Evaluation Software [12], and the Symptom Tracking and Reporting System for PRO AE reporting during chemotherapy [13]. To support online reporting patient education is also recommended [14] along with an ethical responsibility to provide patients with clear guidance on managing severe AE and alerting their clinical team [15].
Making data available to clinicians in ‘real time’ using electronic PRO (ePRO) could have the most transformative effect on clinical practice by providing easily accessible records to guide treatment decisions [16]. The increased use of EPRs and patient access to health records has created opportunities for the integration of PRO systems with clinical data.
A challenge of linkage to an internet based ePRO system is to maintain the security of sensitive patient data in the EPR. In the UK NHS, the single secure N3 network connects all NHS organisations [17]. Approved third parties (e.g. prescribing systems) access to the NHS is subject to strict governance.
Previously, in Leeds Cancer Centre a highly scalable electronic online questionnaire management software QTool was developed by X-lab (a private software company) for ePRO data collection. QTool facilitates building and scheduling of flexible questionnaires to match clinical needs. When complete, the PRO data is instantly accessible and downloadable for analysis. QTool was used in a study of 600 cancer survivors [18] successfully linking group PRO data to the cancer registry. However, PRO responses were not integrated into the EPR.
The Leeds Cancer Centre EPR (PPM-Patient Pathway Manager) is used in NHS trusts across the Yorkshire region [19] to coordinate the care of over 2 million patients. PPM has an integrated clinical trials module enabling allocation of patients to research studies, management of trial documents and correlation of demographic and clinical information with trial data.
Building on these platforms we initiated the National Institute for Health Research (NIHR) funded eRAPID programme (electronic patient self-Reporting of Adverse-events: Patient Information and aDvice) [20]. eRAPID aimed to design and evaluate a system for patients to self-report and manage AE during and after cancer treatment. To support the programme we developed an innovative, secure interface between the QTool system and the existing Leeds Cancer Centre EPR known as patient pathway manager (PPM).
The aims of this project were to develop:
Secure real-time integration of the online questionnaire system (QTool) with the EPR in Leeds Cancer Centre (within N3 network restrictions) including a user-friendly and easily accessible display of PRO data for clinicians.
Clinically based algorithms providing patients with immediate, automated tailored advice on managing AE, and notification via email for severe AE to clinicians.
Improved usability and functionality of the QTool patient interface to support patients to login securely and easily report and manage AE from web enabled devices.
Methods
Theoretical foundations and rationale for approach of the eRAPID system
Following recommended Criteria for Reporting the Development and Evaluation of Complex Interventions in Healthcare [21], we have described the theoretical underpinnings of eRAPID and the aims, essential functions and rationale for selection of the individual components (supplementary Table S1, available at Annals of Oncology online).
Methodological approach and procedures
A multi-disciplinary project team led the development that included researchers, lead oncology clinicians, health informatics professionals (X-Lab), a patient advocate group and lay members from the National Cancer Research Institute patient and public involvement group. Key roles included a technical lead working within a disciplined agile framework [22] to enhance the software to fit stakeholder specifications. A researcher in a liaison role translated the research and clinical needs to the IT team and led system usability testing.
A wider stakeholder group of clinical staff (N = 19), Patient advocates (N = 9) and patients (N = 13) administrators and researchers was continuously consulted throughout to (i) elicit initially the requirements of the system and (ii) validate the content and design by reviewing each iteration of the eRAPID platform. This was achieved either via direct interviews, meetings participation or, telephone contacts. Responses were charted thematically [23] and if required changes made to the next iteration.
An overview of the work in chronological order is provided in Figure 1. Below, we report the methods and results for each aim.
Secure real-time integration of the online questionnaire system (QTool) with the EPR including a user-friendly and easily accessible display of PRO data for clinicians.
Figure 1.
Flow chart of timelines of the development of the eRAPID system.
Method
Stakeholder requirements were elicited via interviews and summarised to inform the technical specifications (supplementary Table S2, available at Annals of Oncology online). A key challenge when integrating PRO data into the EPR was to develop a scalable way to display PRO results to clinicians whilst maintaining the security of patient data. QStore, a web application, was developed (hosted within the NHS network) to access the QTool secure anonymised interface and retrieve and store PRO data utilising the existing clinical trials module in PPM. QStore was developed using ASP.NET MVC and SQL Server. Using Task Scheduler, a .NET windows application retrieves and stores PRO data from QTool every 5 min.
Results
The resulting data flow through the system is illustrated in Figure 2 with descriptions of how the system meets the needs of the key stakeholders. Identifiable and sensitive patient data is held securely in the EPR database behind the NHS firewall. QStore links each patient EPR record to QTool and allocates a unique QTool login name.
Figure 2.
eRAPID system diagram describing the data flow from public internet to hospital network.
Patients are given a postcard with their login details which allows access to eRAPID via the website portal (Figure 3) from home, hospital kiosks or any web-enabled device. On completion of the AE items patients can view their own data (over time) in tabular or graphical formats, and may receive severity-dependent advice (see aim 2). They have the facility to print or email the results. eRAPID generates weekly text message or email reminders to encourage completion. Adherence is monitored though system tracking (website visits and questionnaire completion) and evaluated by the number of appropriate contacts with the hospital, alerts and admissions. QStore pulls PRO data every few minutes and displays it to clinicians in tabular or graphical format with level 3 symptoms highlighted in red (Figure 3).
2. Clinically based algorithms providing patients with immediate, automated tailored advice on managing AE, and notification via email for severe AE to clinicians.
Figure 3.
Screen shots of the patient view: (A) eRAPID website portal screen, (B) QTool patient welcome page with links to questionnaires, previous responses and feedback, (C) example of the eRAPID symptom report, (D) AE Self-management advice generated from QTool when patient reports mild/moderate symptoms, (E) if patients report low level AE they are directed to the eRAPID website for self-management advice, (F) if patients report a severe AE and if it is a current problem advice (in red) to telephone the hospital is generated, (G) patient view of the tabular summary of AE reported, (H) Patient view of the graphical summary of their responses. Clinician view: (I) graphical display of a one-time completion of PRO results, (J) display of PRO reported results in the EPR in tabular form with severe symptoms indicated in red, (K) graphical display of completion over time with red triangles indicating chemotherapy cycles.
Methods
The existing question scoring dependency function in QTool was utilised to construct hierarchical algorithms triggering immediate severity dependent advice based on AE grading dependent on how the patient answers a question or combination of questions. The eRAPID symptom report questionnaires are patient reported AE adaptations (PRAE) [24] of the gold standard clinician reporting system (CTCAE version 4.0) [25]. The responses for each question are allocated to a score from 0 to 3 corresponding to the CTCAE severity grades and the United Kingdom Oncology Nursing Society (UKONS) triage forms [26] (see Table 1).
Table 1.
CTCAE (version 4.0) vomiting item, corresponding patient reported version and CTCAE and UKONS severity grading (0–3)
| CTCAE (version 4.0) Vomiting item | PRAE item Have you been sick (vomited)? | CTCAE Grade | CTCAE Version 4.0 General description of severity grading | UKONS |
|---|---|---|---|---|
| N/A | N/A | 0 | N/A | No problem reported and no advice needed. |
| 1–2 episodes (separated by 5 min) in 24 h | I have vomited 1–2 times in a 24-h period | 1 | Mild; asymptomatic or mild symptoms; clinical or diagnostic observations only; intervention not indicated | Mild problem not requiring medical attention, self-management is appropriate |
| 3–5 episodes (separated by 5 min) in 24 h | I have vomited 3–5 times in a 24-h period | 2 | Moderate; minimal, local or non-invasive intervention indicated; limiting age-appropriate instrumental ADL | Potentially serious problem, may require medical attention |
| ≥6 episodes (separated by 5 min) in 24 h; tube feeding, TPN or hospitalization indicated | I have vomited six or more times in a 24-h period | 3 | Severe or medically significant but not immediately life-threatening; hospitalization or prolongation of hospitalization indicated; disabling; limiting self- care ADL | Potential medical emergency, requires urgent medical review |
ADL, activities of daily living; instrumental ADL (preparing meals, shopping for groceries or clothes, using the telephone, managing money, etc.), self-care ADL (bathing, dressing and undressing, feeding self, using the toilet, taking medications and not bedridden).
Five hierarchical algorithms were developed corresponding to the UKONS severity levels 0–3 ranging from A1 (most severe) to D (least severe) (supplementary Table S3, available at Annals of Oncology online). For severe symptoms triggering A1, an email notification is automatically sent to one (or more) clinicians advising them to view the PRO report in the EPR. Corresponding AE advice was sourced from local and national guidelines accessible via QTool or a customised website [27]. Both the algorithms and patient advice were developed though iterative review with the stakeholder group: 19 clinicians, 9 patient advocates and 13 patients via face-to face or telephone interview, and though discussion at project management meetings, see supplementary Table S1, available at Annals of Oncology online, for demographic detail. Additionally, we asked the clinicians to complete the eRAPID symptom report questionnaire (from the patient perspective) following two chemotherapy-related AE scenarios (supplementary Table S5, available at Annals of Oncology online) [23]. The scenarios were designed to test the algorithms to ensure relevant severity-dependent advice was triggered.
Results
The clinical algorithms, implemented in QTool via the scoring system and dependencies, allow patients to receive immediate targeted advice for low or moderate AE to help self-management with links to relevant webpages for more detailed information. For severe AE they are advised to contact the hospital immediately (see Figure 3).
Clinicians receive an email notification for severe symptoms) detailing the patient’s QTool username and the symptom(s) reported. A report corresponding to the notification was created in the EPR identifying the patient with the QTool username, allowing the clinician to open the medical records and respond to the notification. The scenario-based testing of the algorithms did not reveal any issues.
3. Improved usability and functionality of the QTool patient interface.
Methods
To test the eRAPID symptom reports and algorithms we initially engaged 19 clinicians and 9 patient advocates who reported combinations of symptoms and severities. Comments were collated on: logging in, accessing the system, navigation through the questionnaire; accessing their results and self-management advice and printing or emailing results. Staff and patients responded verbally via semi-structured audio-recorded interview and written comments. Data were charted and analysed thematically [23]. Changes were made and subsequently the full eRAPID system was tested in a convenience sample of 13 patients receiving chemotherapy using interviews and written comments for feedback for demographic details, see supplementary Table S4, available at Annals of Oncology online.
Results
Key points from staff and patient feedback included: patient safety, accessing the 24-h hotline number within the QTool management advice, accessing and navigating through the system and patients viewing and accessing results. Significant improvements were made to the system see (supplementary Table S6, available at Annals of Oncology online).
Discussion
We have successfully developed an online system for PRO reporting of AE with an existing EPR in real-time whilst maintaining patient confidentiality and security and meeting the key stakeholder specifications.
Patients can now report AE from home on PC, tablet or smart phone securely during treatment and receive appropriate management advice via the severity dependent algorithmic questionnaire scoring. Clinicians can view patient AE data in the EPR and receive email notifications when patients report severe symptoms. The immediate severity-related guidance on how to manage AE is a unique feature of our system compared with other ePRO web-portals [11–13, 16].
Key security concerns were addressed though development of QStore allowing non-identifiable questionnaire response data to be transferred through NHS firewalls. We have achieved this with the full support of, and in close collaboration with, the local EPR development team supported by LTHT thus ensuring the smooth integration of the system.
The content and design of the system was developed with the programmer, research liaison, clinicians and patient advocates on the project management team and validated through usability testing. We developed an accessible intuitive staff, researcher and patient interface by identifying and troubleshooting system errors and usability issues before introduction to patients/clinicians.
In future, we will work with staff to determine the level of support needed to integrate the system into the care pathways of different treatment groups and provide extensive staff training to ensure smooth adoption of the system.
The system is functioning within the local EPR but we have adapted eRAPID to meet the needs of different patient groups in UK NHS settings. Implementation work is currently underway in Manchester and Bristol for pelvic radiotherapy and upper gastrointestinal surgery respectively.
Collection of large-scale patient reported AE data poses challenges for data capture, storage, security and integration into patient care pathways. We do not underestimate the challenges of interfacing QTool with other NHS EPR as success is dependent on local system limitations. QTool was developed for web browsers of computers or tablets. Usability could be improved by developing smartphone apps but such approach would reduce the flexibility of changing patient questions and increase the development cost of apps for different smartphone platforms.
Conclusion and future challenges
We have successfully developed a secure interface between an online integrated electronic system for PRO data collection and an EPR in a single cancer centre. A remaining challenge is the implementation in busy hospitals considering the administrative procedures and resource requirements. To assess the staff and patient training needs and the acceptability of the system, we plan to test the full eRAPID intervention in a sample of breast cancer patients on adjuvant chemotherapy. This will provide an opportunity to refine the system before its full evaluation in a randomised controlled trial in breast, gynaecological and colorectal cancer patients undergoing systemic therapy.
Supplementary Material
Acknowledgements
This work was completed by the Patient Reported Outcomes Group: Leeds Institute of Cancer Studies and Pathology, Bexley Wing, St James's Hospital, University of Leeds, Beckett Street, Leeds LS9 7TF, UK. We would like to thank Colin Johnston, Robert Carter, Joe Shaw and the wider Patient Reported Outcomes Group for their help in developing and testing the system. Clinical staff Oncologists: Geoff Hall; Daniel Swinson; Alison Young; Clive Mulatero and Virgil Sivoglo; Nurses: Sue Bramham-Exley; Sue Goulding; Evelyn Tatham; Alison Craven; Louise Taylor; Lead Cancer Nurse Krystina Koslowska; lead Chemotherapy Nurse Anne-Marie Kenny. Dietician Bryony Clarke; Physiotherapists: Clare Parsons and Allison Gullvag; Clinical Psychologist Sarah Catesby and Macmillan Information Specialist Sadie Smith. Patient advocates: We would like to thank the POG research advisory group particularly Tony Betts and Sara El-Hassani, for their contribution to usability testing. Sara El-Hassani was a passionate and dedicated patient advocate who sadly passed away in 2015. Project management team: Professors Jenny Hewison; Peter Selby; Julia Brown; Clare Hulme Jane Blazeby Norah Kearney and Rick Jones who sadly passed away in 2014; Doctors Lucy Ziegler; Clare Harley; Geoff Hall; Kevin Franks; Others: Martin Waugh and David Fox; Krystina Koslowska and Mrs Sue Kier who sadly passed away in 2012.
Funding
This article presents independent research funded by the National Institute for Health Research under its Program Grants for Applied Research Program (Reference Number RP-DG-1209-10031).The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
Disclosure
The authors have declared no conflicts of interest.
References
- 1. Mort D, Lansdown M, Smith N. et al. For Better, for Worse? A Review of the Care of Patients Who Died Within 30 Days of Receiving Systemic Anti-Cancer Therapy. London: National Confidential Enquiry into Patient Outcome and Death; 2008. [Google Scholar]
- 2. De Luigi A. Analysis of reasons for admission to the emergency department for cancer patients. Ann Oncol 2002; 13: 112.12078891 [Google Scholar]
- 3. Warrington L, Holch P, Kenyon L. et al. An audit of acute oncology services: Patient experiences of admission procedures and staff utilisation of a new telephone triage system. Support Care Cancer 24(12): 5041–5048. [DOI] [PubMed] [Google Scholar]
- 4. Velikova G, Booth L, Smith AB. et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol 2004; 22: 714–724. [DOI] [PubMed] [Google Scholar]
- 5. Basch E, Bennett A, Pietanza MC.. Use of patient-reported outcomes to improve the predictive accuracy of clinician-reported adverse events. J Natl Cancer Inst 2011; 103: 1808–1810. [DOI] [PubMed] [Google Scholar]
- 6. Black N. Patient reported outcome measures could help transform healthcare. Bmj 2013; 346: 167.. [DOI] [PubMed] [Google Scholar]
- 7. Johansen MA, Henriksen E, Horsch A. et al. Electronic symptom reporting between patient and provider for improved health care service quality: a systematic review of randomized controlled trials. part 1: state of the art. J Med Internet Res 2012; 14: e118.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Jensen RE, Snyder CF.. PRO-cision Medicine: Personalizing Patient Care Using Patient-Reported Outcomes. J Clin Oncol 2016; 34: 527–529. [DOI] [PubMed] [Google Scholar]
- 9. Basch E, Artz D, Dulko D. et al. Patient Online Self-Reporting of Toxicity Symptoms During Chemotherapy. J Clin Oncol 2005; 23: 3552–3561. [DOI] [PubMed] [Google Scholar]
- 10. McCann L, Maguire R, Miller M, Kearney N.. Patients' perceptions and experiences of using a mobile phone-based advanced symptom management system (ASyMS) to monitor and manage chemotherapy related toxicity. Eur J Cancer Care 2009; 18: 156–164. [DOI] [PubMed] [Google Scholar]
- 11. Snyder CF, Jensen R, Courtin SO. et al. PatientViewpoint: a website for patient-reported outcomes assessment. Qual Life Res 2009; 18: 793–800. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Holzner B, Giesinger JM, Pinggera J. et al. The Computer-based Health Evaluation Software (CHES): a software for electronic patient-reported outcome monitoring. BMC Med Inform Decis Mak 2012; 12: 126.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Basch E, Deal AM, Kris MG. et al. symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol 2016; 34: 557–565. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Berry DL, Hong F, Halpenny B. et al. l. The electronic self report assessment and intervention for cancer: promoting patient verbal reporting of symptom and quality of life issues in a randomized controlled trial. BMC Cancer 2014; 14: 513.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Kyte D, Draper H, Calvert M.. Patient-reported outcome alerts: ethical and logistical considerations in clinical trials. Jama 2013; 310: 1229–1230. [DOI] [PubMed] [Google Scholar]
- 16. Holzner B, Giesinger JM, Zabernigg A. et al. Patients' preferences regarding the setting of electronic patient-reported outcome assessments. Val Health 2015; 18 (7): A473. [Google Scholar]
- 17.Read I, N3 Network User Guide v1.3. 2010. http://studyres.com/doc/8041860/n3-network-user-guide (November 2016, date last accessed).
- 18. Ashley L, Jones H, Thomas J. et al. Integrating patient reported outcomes with clinical cancer registry data: a feasibility study of the electronic Patient-Reported Outcomes From Cancer Survivors (ePOCS) system. J Med Internet Res 2013; 15: e230.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Newsham AC, Johnston C, Hall G. et al. Development of an advanced database for clinical trials integrated with an electronic patient record system. Comput Biol Med 2011; 41: 575–586. [DOI] [PubMed] [Google Scholar]
- 20. Absolom K, Holch P, Warrington L. et al. Electronic patient self-Reporting of adverse-events: Patient information and advice (eRAPID): A randomised controlled trial in systemic cancer treatment. BMC Cancer 2017; 17: 318. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Möhler R, Köpke S, Meyer G.. Criteria for Reporting the Development and Evaluation of Complex Interventions in healthcare: revised guideline (CReDECI 2). Trials 2015; 16: 204. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Ambler SW, Lines M, Disciplined agile delivery: a practitioner's guide to agile software delivery in the enterprise. Pearson Education, Indiana US: IBM Press; 2012. [Google Scholar]
- 23. Rogers MLPE, Chapman R. et al. Usability Testing and the Relation of Clinical Information Systems to Patient Safety In Henriksen K, Battles JB, Marks ES (eds), Advances in Patient Safety: From Research to Implementation (Vol. 2: Concepts and Methodology). Rockville, MD: Agency For Heathcare Research and Quality (US; ) 2005. [PubMed] [Google Scholar]
- 24. Holch P, Warrington L, Potrata B. et al. Asking the right questions to get the right answers: using cognitive interviews to review the acceptability, comprehension and clinical meaningfulness of patient self-report adverse event items in oncology patients. Acta Oncol 2016; 9–10: 1220–1226. [DOI] [PubMed] [Google Scholar]
- 25.National Institutes of Health and National Cancer Institute: Common Terminology Criteria for Adverse Events (CTCAE) Version 4.0 2009; U.S.Department of Health and Human Sciences; NIH & NCI
- 26.Acute Oncology Initial Management Guidelines; http://ukons.org/contentimages/FINAL_GUIDELINE_V_1.0_11.pdf (September 2011 date last accessed).
- 27. Hector C, Holch P, Warrington L. et al. Development of online patient-advice for the self-management of low-level chemotherapy related toxicities (eRAPID): involvement of patients and staff. Psycho-Oncology. 2013; 22: 11. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.