Creation of an electronic patient-reported outcome measure platform Voxe: a mixed methods study protocol in paediatric solid organ transplantation

Abstract

Introduction

Patient-reported outcome measures (PROMs) provide an opportunity for meaningful patient engagement and shared decision-making. The objective of this research programme is to improve health outcomes for paediatric solid organ transplant patients by implementing PROMs into clinical care. The current study aims to create Voxe, a paediatric user-centred electronic PROM platform, by engaging patients and healthcare providers throughout the design and development process.

Methods and analysis

The creation of Voxe will occur over two phases that build on previous research. The user interface design phase employs a ‘user-centric’ approach to identify end-users’ needs and iteratively refine the look and layout of Voxe to meet these needs. Transplant recipients, aged 10–17, and healthcare providers will participate in three rounds of testing (24 participants total). Participants will: (1) complete task-based activities (outcomes—effectiveness and efficiency), (2) complete questionnaires (outcome—satisfaction) and (3) participate in a semi-structured interview. The following phase involves software development and Voxe usability testing. Transplant recipients, aged 8–17, and healthcare providers will participate in four rounds of iterative testing (24–40 participants total). The think-aloud technique will be employed, and participants will describe their thoughts and feelings while interacting with a Voxe prototype. Participants will: (1) log into Voxe and complete tasks (outcomes—time on task, successful task completion, frequency of critical and non-critical errors and error-free rate), (2) complete questionnaires (outcome—satisfaction) and (3) participate in a semi-structured interview. Findings will result in the creation and launch of a user-centred electronic PROM platform.

Ethics and dissemination

Research ethics board approval has been provided by The Hospital for Sick Children. This research is critical to answering methodological and operational questions to inform Voxe implementation in paediatric clinical settings and facilitate PROM data collection. Future investigations will include an implementation-effectiveness evaluation.

Strengths and limitations of this study.

• By engaging paediatric patients and healthcare providers throughout the design and development process, this study will facilitate the creation and launch of an evidence-based paediatric user-centred electronic patient-reported outcome measure platform prototype called Voxe.

• A ‘user-centric’ approach will consider the needs of paediatric patients and healthcare providers at each design phase and allow for iterative modification of wireframes to best meet their identified needs.

• The think-aloud technique will facilitate understanding of the end-user’s experience with Voxe by enabling participants to verbalise their thoughts and feelings while interacting with Voxe to complete specific tasks.

• As the ability to speak and read English is a requirement for study participation, the perspectives of those who are not able to speak and read English will be missed in these phases of the study.

• This study builds the foundation for future phases of research which will include healthcare provider Voxe orientation and competency training and an implementation-effectiveness evaluation.

Background

For children with end-stage organ failure, transplantation is a life-saving therapy.^{1 2} However, evaluating the success of solid organ transplantation based solely on objective clinical outcomes is insufficient. The patient’s subjective assessment is a crucial component in evaluating the burden of disease and can be captured via patient-reported outcome measures (PROMs).^{1 2} PROMs are defined as: ‘any report of the patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else’ (pg. 2).³ PROMs can capture a patient’s self-assessment of functional status, symptoms, treatment adherence and multiple domains of well-being and quality of life.^{4 5} In doing so, PROMs give patients a voice in their healthcare and provide an opportunity for meaningful engagement.⁶ Research indicates that the systematic collection of PROM data enhances patient–clinician communication and shared decision-making, thereby improving health outcomes.4 5 7 8 The inclusion of PROMs in clinical care assists in identifying valuable information about the impact of transplantation on patients’ symptoms as well as their functional and emotional status. This in turn may help healthcare providers to detect under-recognised and unrecognised problems (eg, depression, anxiety), resulting in more effective patient care (eg, initiation of clinical interventions) and an efficient healthcare system.

Innovative opportunities to integrate PROMs into clinical practice have been buoyed by recent advances in eHealth.^{9 10} In particular, the development and implementation of electronic PROMs (ePROMs) can help identify important, patient-valued concerns at the point of care, supporting the delivery of appropriate and timely interventions. Moreover, current platforms that support the use of ePROMs are underdeveloped and require better implementation with clinical care.^{2 6 7} Further, research on the implementation effectiveness of ePROMs in paediatric clinical settings is limited,^{2 11} giving rise to concerns that ePROMs may languish, unused and fail to realise meaningful outcomes for patients. Implementation of evidence-based interventions is important to ensure meaningful patient outcomes.¹² Achieving optimal clinical and health system outcomes for ePROMs will require more intentional and explicit study of how they might best be implemented prior to widespread implementation.^{1 2 5}

The overarching objective of this research programme is to improve health outcomes and transform the delivery of care for paediatric transplant patients in Canada by integrating ePROMs into standard clinical practice. This programme of research uses a phased approach to target the methodological and practical decisions (eg, determining which standardised PROMs to use, identifying goals for collecting PROMs, selecting patients, setting and timing of assessment) needed to guide systematic and effective implementation of ePROMs into ‘real-world’ paediatric patient care settings.¹³ Recently completed foundational research to explore these questions within paediatric solid organ transplantation consisted of the three phases of work outlined below.

Phase 1: Systematic review

A systematic review was conducted to identify PROMs used in paediatric solid organ transplantation.¹⁴ A total of 4305 studies were identified, of which 62 describing 47 PROMs were selected for analysis and were appraised for adherence to internationally recommended guidelines for item generation, item reduction and psychometric properties.¹⁵ Findings revealed six standardised PROMs that had undergone psychometric evaluation in a paediatric solid organ transplant population. This phase of work identified standardised PROMs to consider for implementation into clinical care.

Phase 2: Interviews with key stakeholders

Interviews with key stakeholders across Canada, including: (a) paediatric solid organ transplant recipients, (b) parent(s)/caregiver(s) and (c) healthcare providers, were conducted to explore perspectives regarding ePROMs implementation into clinical practice.¹⁶ Sixty-three participants across five Canadian paediatric transplant centres were interviewed, among whom nearly all (60/63; 95%) were supportive of implementing an ePROM system into clinical practice with the primary goals of: (1) integrating the transplant patient’s overall well-being into the clinical care conversation, (2) capturing the patient’s voice and increasing patient engagement and (3) informing paediatric transplant clinical care. Insights for effective PROM implementation included the remote completion of ePROMs in advance of clinical appointments for patients 8–10 years of age or older.

Phase 3: Consensus workshop

A 2-day consensus workshop was hosted in December 2018 in Toronto to further explore how ePROMs could best be implemented into paediatric transplant clinical practice. Workshop proceedings were informed by the results of Phases 1 and 2. The workshop was attended by 25 leading experts in the fields of paediatric solid organ transplantation, PROMs, implementation science and computational medicine, as well as patients, caregivers, healthcare providers, researchers and administrators from across Canada. Workshop outcomes included: (1) consensus on key methodological and operational decisions for implementing ePROMs into practice (eg, which standardised PROMS to use, the setting and timing of assessment, as well as the mode for administering ePROMs), (2) a research plan to design, develop and evaluate the usability and implementation of an ePROM platform and (3) a knowledge translation strategy to disseminate research findings to key knowledge users (eg, newsletter, peer-reviewed publications, website posting, national and international presentations). The consensus workshop captured attendees’ perspectives on practice and systems-based facilitators and barriers to implementing ePROMs and was instrumental in ensuring that future research would be relevant and meaningful to stakeholders.

Study objectives

Results from Phases 1 to 3 inform the current study, Phases 4 and 5.¹⁷ The overarching aim of the proposed study, which will be conducted within the Transplant and Regenerative Medicine Centre at The Hospital for Sick Children (SickKids), is to create an ePROM platform called Voxe, that will capture and implement patient-reported outcomes into the clinical care workflow for paediatric organ transplant patients. Specifically, Phase 4 aims to design the user interfaces of the Voxe platform, and Phase 5 aims to develop the Voxe software and conduct usability testing of Voxe in preparation for a future implementation-effectiveness trial (Phase 6). A graphical representation of the different research phases is displayed in figure 1.

Figure 1.

Measuring What Matters: Implementing Patient-Reported Outcome Measures into Clinical Practice—A Research Programme.

Methods

Phase 4: User interface design of Voxe ePROM platform

eHealth technologies designed and developed based on assumptions of end-user motivations, goals or needs, are often less effective than those that engage end-users throughout the process.^{10 18} Thus, a ‘user-centric’ approach in which end-users (ie, patients and healthcare providers) are central to the design process will guide the design and development of Voxe. This evidence-based approach will consider the needs of Voxe users at each design phase and will allow for iterative modification of wireframes, which are the static, two-dimensional visual representation or layout of Voxe, to best meet their identified needs.^19–21 A ‘user-centric’ approach is paramount for user engagement with the platform, ultimately contributing to the effectiveness of the platform itself.¹⁰

Study participants and inclusion criteria

Purposive sampling will be used to recruit 12 patient participants across age, organ type, sex, gender and ethnicity from the Transplant and Regenerative Medicine Centre at SickKids to obtain maximum variation.²² Twelve members of the patients’ interdisciplinary healthcare teams at SickKids will also be recruited purposively across professional disciplines, years of practice, sex, gender and ethnicity. This sample size is consistent with testing methods for clinical information systems.²³

Patients eligible to participate include those who are: (a) 10–17 years of age, (b) able to speak and read English and (c) heart, kidney, liver or lung transplant recipients who are a minimum of 3 months post transplant. Patients with significant cognitive impairments, as determined by a healthcare team member, will not be invited to participate. Eligible healthcare providers include any member of the interdisciplinary healthcare team within the Transplant and Regenerative Medicine Centre at SickKids.

Procedures and outcomes

The design of preliminary Voxe wireframes will be guided by: (1) stakeholder input gleaned from previous phases of research,^{14 16} and (2) design workshop processes, including identification of Voxe users (ie, patients and healthcare providers) and the tasks they will complete on their respective platforms (ie, persona and task inventory development). Following the design of preliminary wireframes, a rapid and iterative testing methodology will be used to evaluate, learn and improve Voxe prior to development (ie, coding and launch).¹⁹ Three rounds of testing sessions will be scheduled with patient and healthcare provider participants to elicit feedback on Voxe design features. Written consent and assent, as well as demographic information, will be obtained prior to study participation.

Each testing session will be conducted virtually via the Personal Health Information Protection Act-compliant version of Zoom or Microsoft Teams. During each session, International Organization for Standardization key performance indicators, consensus-base standards for technology, will be benchmarked and tracked to validate each iteration for success.^{24 25} In particular, objective and subjective standards common in user experience design testing,²⁶ will be collected to measure: (1) effectiveness—accuracy and completeness with which users achieve specific goals, displayed as a percentage of tasks successfully completed (average task completion rate is 78%; above average is considered successful task completion^{27 28}) and (2) efficiency—resources used in relation to results achieved, represented by the time it takes users to successfully complete the task.²⁹ Prior to the scheduled testing session, the URL for the testing website will be emailed to the participant. During the session, participants will complete task-based activities using incremental segments (ie, wireframes) of the Voxe platform.

Following the task-based activities, healthcare provider participants will complete the System Usability Scale, a 10-item Likert scale questionnaire to assess the key performance indicator satisfaction.^{30 31} The System Usability Scale is considered a reliable way to evaluate electronic platforms and a score of 68 is considered above average.^{30 31} Patient participant’s overall impression and experience with the Voxe platform will be evaluated using the Microsoft Desirability Toolkit.³² Patient participants will select five words from a list of product reaction words to describe their attitude towards the Voxe platform. Product reaction words, such as ‘fun’ and ‘calm’ describe intangible emotional response towards the interface.³³

Lastly, participants will share their likes and dislikes of the Voxe platform design and comment on the platform’s ease of use and elements of functionality during a virtual semi-structured qualitative interview. Semi-structured interviews foster reciprocity between the participant and interviewer, allow the interviewer to ask pertinent follow-up questions to elicit rich data and enable the participant to express themselves using their own words.^34–36 The interview guide will be developed by the study team and will be informed by clinical knowledge and experience. Interviews will be conducted by study team members trained in qualitative methods. Sessions will be audio-recorded, transcribed verbatim and de-identified to protect participant confidentiality. Recruitment for Phase 4 began in May 2020.

Data analysis

Data collected during the testing sessions, including objective and subjective International Organization for Standardization key performance indicators, will be used to refine Voxe. The research team will use content and thematic analysis to categorise the data collected during qualitative interviews.^37–39 Two members of the study team experienced in qualitative methods, will code the data independently, and categories will be reviewed and refined until consensus is reached for emerging themes. Trustworthiness will be achieved by facilitating member checking and soliciting rich description during interviews, as well as hosting frequent team meetings to support in-depth, iterative analysis with reflexive discussion among team members. Analysis will be complete once the research team agrees that thematic saturation is attained.³⁹ NVivo V.12 will be used to manage qualitative data.⁴⁰ Quantitative data collected during the testing sessions will be triangulated with qualitative themes to provide a richer understanding of end-users’ experience with Voxe. Refinements will be made to the Voxe platform design based on the triangulated data. Three rounds of iterative feedback testing will be conducted with each participant population (ie, four participants per round) until Voxe is considered acceptable to participating end-users with no further refinements identified.^{41 42} Following the third round of patient and healthcare provider iterative testing, the design team will share the final Voxe patient and healthcare provider annotated wireframes with the development team.

Phase 5: Development of ePROM platform Voxe and usability testing

Usability is defined as the ‘extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use’.²⁴ To test Voxe usability, the think-aloud technique will be employed in which participants will verbalise their thoughts and feelings while interacting with Voxe to complete structured tasks.^{43 44} The think-aloud technique is integral to understanding the end-user experience with Voxe and will highlight potential barriers to Voxe adoption that can inform its subsequent implementation.^{43 44} Semi-structured interviews and data analytics, described below, will also be conducted.⁴⁵

Study participants and inclusion criteria

Purposive sampling will be used to recruit 12–20 patient participants across age, organ type, sex, gender and ethnicity from the Transplant and Regenerative Medicine Centre at SickKids. Patients’ interdisciplinary healthcare team of 12–20 members will also be recruited purposively across professional disciplines, years of practice, sex, gender and ethnicity.²³

Patients eligible to participate will include those who are: (a) 8–17 years of age with capacity to assent/consent, (b) those able to speak and read English, (c) heart, kidney, liver or lung transplant recipients and (d) who are a minimum of 3 months post transplant. Informed consent will be obtained from the parents/legal guardians of participants who provide assent. Eligible healthcare providers include any member of the interdisciplinary healthcare team within the Transplant and Regenerative Medicine Centre at SickKids who have worked within their position for a minimum of 6 months.

Procedures and outcomes

Following Phase 4, the development team will use the Voxe patient and healthcare provider annotated wireframes to develop the respective interfaces of the Voxe ePROM platform, using an agile, scrum framework.¹⁹ A scrum framework is a project management process within a hybrid software development model that applies a flexible development process and places the needs of system end- users at the forefront to ensure that Voxe is both useful and usable.^{46 47} The principles of iterative feedback, incremental development and continual stakeholder involvement are central to this dynamic approach.^{46 47} Feature development will be phased and will include authentication, user dashboards, account settings, privacy/security controls and survey submission/review functionality.

Four rounds of iterative testing will be completed with three to five patients and three to five healthcare providers per round, as is consistent with usability testing methods for clinical information systems.²³ The first two rounds will be conducted in-person or virtual with a member of the study team, using smartphones, tablets and/or computers. Both patient and healthcare provider participants will be asked to complete a core set of tasks on Voxe which will be presented to them in the form of scenarios that they may encounter while interacting with Voxe. For example, patients will be provided with an anonymous username and password, invited to successfully log into Voxe and navigate Voxe to complete available ePROMs (ie, PedsQL Generic Core Scales⁴⁸). Healthcare providers will be invited to navigate Voxe to view and interpret sample ePROM results. Employing think-aloud methodology, participants will be encouraged to voice out loud what they are looking at, thinking, doing and feeling as they navigate the platform.^{43 45}

The last two testing rounds will be conducted to simulate ‘real-world’ settings. An automated text message or email with an embedded hyperlink will be sent to patients asking them to access Voxe remotely on a smartphone, tablet or computer. Patients will independently log into Voxe using an anonymous username and password and navigate the platform to complete available ePROMs. Healthcare providers will be asked to access Voxe on a computer and independently navigate the Voxe platform to view and interpret ePROM data entered by patients.

Objective measures to be collected include: (1) time on task—the time it takes to complete each task, (2) successful task completion (fidelity)—when the end-user achieves the end goal of the task successfully (above the average task completion rate of 78% is considered successful^{27 28}); (3) frequency of critical errors—a high severity error that could prevent an end-user from being able to complete a task, (4) frequency of non-critical errors—a low severity error that could decrease the efficiency with which an end-user completes a task and (5) error-free rate—the percentage of task completions that occurred without any errors.^{49 50} After the completion of task-based activities, participant satisfaction will be evaluated. Healthcare provider participants will complete the System Usability Scale. A score on the System Usability Scale that is greater than 68 is considered above average.^{30 31} Patient participants will complete the Microsoft Desirability Toolkit by selecting words from a list of product reaction words to describe their attitude towards the Voxe platform.^{32 33} Following each testing round, in-person or virtual semi-structured interviews will be conducted to ascertain what participants liked or disliked and why, ease of use, elements of functionality in the context of typical practice workflow, as well as suggestions for improvements. Iterative usability testing will also be conducted until Voxe is considered acceptable to participating end-users with no further refinements identified.^{41 42} Interviews will be audio-recorded, transcribed verbatim and de-identified.

Data analysis

Quantitative data including time on task, successful task completion, frequency of critical and non-critical errors and error-free rate will be analysed and descriptive statistics will be produced. Similar to Phase 4, qualitative interviews will be subject to content and thematic analysis to identify emerging themes.^{37 38} Themes will be coded and categorised using NVivo V.12 according to type and frequency of occurrence.⁴⁰ Quantitative and qualitative data will be triangulated to inform changes made to the Voxe platform.

Patient and public involvement

Phases 4 and 5 are informed by the invaluable feedback provided by stakeholders, including paediatric patients, their caregivers and healthcare providers, from previous phases of research.¹⁶ Patients’ and healthcare providers’ thoughts, feelings and perspectives about Voxe captured through research processes described in this protocol will continue to guide this research programme. Stakeholder involvement will ensure the implementation of evidence-based interventions integral to achieving meaningful patient outcomes.

Ethics and dissemination

Ethics approval and consent to participate

Institutional research ethics board (REB) approval has been provided by SickKids (REB number: 1000057043 (Phase 4); REB number: 1000067700 (Phase 5)). All participants will provide informed consent or assent prior to their involvement in the study. For participants who provide informed assent, informed consent will be obtained from the parents/legal guardians prior to study participation.

Information security

All interviews will be audio-recorded and transcribed verbatim. Transcription will be completed by a member of the study team. All transcriptions will be de-identified to protect participant confidentiality. All identifying information, both paper copy and electronic information, will be kept confidential. Use of data over the course of the study and dissemination of results will follow standard practice guidelines as determined by the SickKids REB.

Discussion and dissemination

The collection of PROMs provides the opportunity to incorporate patient-centred perspectives into paediatric clinical practice.2 6 11 51–54 The creation of the Voxe ePROM platform will reform the practice of paediatric medicine by enhancing the capacity of patient–provider partnerships to identify and address issues that are most meaningful to patients.⁵⁵ The design and development of Voxe outlined in this protocol are critical to answering important methodological and operational questions that will inform the implementation of ePROMs in paediatric clinical settings.

Engaging patients and healthcare providers throughout Voxe design and development will result in the creation and launch of a user-centred ePROM platform. For patients, Voxe will facilitate ePROM data collection in a child friendly and patient-centred manner. For healthcare providers, Voxe will facilitate convenient and timely review of patient ePROM data, collaboration within healthcare teams and shared decision-making discussions between healthcare providers and patients during clinical encounters.

During Phase 4, patients and healthcare providers will share their likes and dislikes of the design and comment on Voxe functionality and ease of use. It is critical that Voxe design iterations integrate this feedback, as the interface needs to be designed in a way that is logical, intuitive and user friendly for end-users. This process will ensure that there is an evidence-based iterative design in place before usability testing. In Phase 5, patients and healthcare providers will comment on what they liked or disliked about Voxe and why, Voxe ease of use and elements of functionality in the context of typical practice workflow, as well as suggestions to improve Voxe. Voxe will be further refined according to this feedback until a final version is produced. The final product will enhance the experience of end-users with systematically tested functionality and design. Following the procedures outlined above, creating an evidence-based ePROM platform will enable these outcomes. Findings from this study will be widely disseminated through infographics, posts on the research team’s website, peer-reviewed journal publications and presentations at patient and family educational events as well as scientific and academic conferences. Data collection is expected to be completed by the end of 2021 with publication of results in early 2022.

This research lays the groundwork for future investigations that will include Voxe healthcare provider orientation and competency training as part of a more comprehensive implementation plan. Additionally, an implementation-effectiveness evaluation (Phase 6) of the Voxe ePROM platform will be conducted to explore how Voxe can be effectively implemented in a manner that impacts paediatric transplantation patient’s health outcomes and clinical care.

Ethics statements

Patient consent for publication

Not applicable.