Flexible approaches to eCOA administration in clinical trials: The site perspective

Abstract

The Critical Path Institute convened the Support Flexible Approaches to PRO Data Collection project as part of the eCOA: Getting Better Together Initiative which was instigated to identify and address common challenges and drive positive change with eCOA implementation in clinical trials.

The project aimed to identify clinical trial stakeholders’ concerns related to electronic PRO (ePRO) implementation and propose areas of improvement via simplification and flexibility. One workstream focused on patient-/site-centric approaches for simplification and surveyed representatives of clinical sites and site monitors for their perspectives. A semi-structured questionnaire was developed and distributed via snowball sampling to site professionals and clinical research associates (CRAs) that had ePRO experience who had been identified via representative groups or sponsor-led site networks. Responses were received from various site roles across a range of global regions; the largest contribution was from the United States. Topics raised included helpdesk capabilities, technical concerns, device types, and user interfaces among others and are discussed further in this paper. The feedback derived from the questionnaire provided the basis for concrete ideas that sponsors should consider incorporating into protocol design for participant visits, technology use, devices, and methods of back-up data collection.

1. Introduction

1.1. Overview of eCOA: Getting Better Together Initiative

The eCOA: Getting Better Together Initiative (GBTI) is a pre-competitive collaboration between Critical Path Institute, clinical trial sponsors from the Patient-Reported Outcome (PRO) Consortium, providers of electronic data collection technologies and services from the Electronic Clinical Outcome Assessment (eCOA) Consortium, regulators, and other stakeholders. The initiative was launched in 2019 to identify and address challenges and drive positive change with eCOA implementation in clinical trials.

1.2. Project overview

The GBTI Support Flexible Approaches to PRO Data Collection Project aimed to identify clinical trial stakeholders’ concerns related to electronic PRO (ePRO) implementation and propose areas of simplification. The objective of one component of the project was to survey eCOA-experienced trial sites to understand their challenges and gather suggestions for simplification that may result in a more flexible, participant-/site-centric approach to ePRO implementation.

1.3. Background

Electronic collection of COA data in clinical trials has increased significantly since the first implementations in the late 1990's [1]. Extensive evidence demonstrates how electronic capture of COA data improves data quality [[2], [3], [4], [5]]. However, sites report anecdotally that eCOA can be disruptive and can impact participant engagement and compliance. Further, sites report that eCOA demands skills and resources not required for a “paper” trial and may challenge established site workflow [6]. Partnering with experienced eCOA providers may assuage such concerns, but all stakeholders are advised to ensure that systems and processes are minimally disruptive.

This project solicited site staff and clinical research associates (CRAs) perspectives on the challenges with eCOA and collected suggestions for simplifying and creating flexibility.

2. Methods

The group created an English only web-based questionnaire using Microsoft Forms that incorporated items on eCOA use, structured into seven main categories: demographics, study set-up, study oversight, solution-related issues, helpdesk experiences, eCOA back-up options, study operating models, and recommendations for improvement. The full set of questions is listed in Appendix A.

We collectively identified known partner organizations for independent questionnaire dissemination, including sponsors with established site networks, and approached professional networks and pan-industry professional groups via professional social media channels such as LinkedIn. No direct contact with respondents were made, therefore a response rate could not be derived.

The group created a statistical analysis plan, incorporating univariate and bivariate summaries to explore respondents’ experience with eCOA. Data were analyzed using Microsoft Power BI to explore trends by role and country and only those results that revealed discernible patterns are reported in the Results section.

3. Results

3.1. eCOA experience

Seventy-seven individuals responded to the Site/CRA questionnaire. Ten respondents were excluded because their response to the qualifying question indicated they did not have eCOA experience. Data from the remaining 67 respondents constituted the analysis dataset (see Table 1).

Table 1.

Description of survey respondent (n = 67) background including their role at study site, their location by continent and the extent of their experience with eCOA, (n) and (%).

	n	%
Role
Study/Site coordinators (SC)	47	70.1
CRAs/Monitors (CRAs)	5	7.5
Principal Investigators (PI)	5	7.5
Study Nurses (SN)	4	6
Study/Site Administrators (Admin)	4	6
Study Clinician/Physicians (ClinPhys)	2	2.9
Continenta
North America	39	58.2
Europe	12	17.9
Asia	6	8.9
South America	5	7.5
Oceania (All Australia)	3	4.5
Africa	2	2.9
eCOA Experienceb,c
Training participants to capture eCOA data on a provisioned device	55	82.1
Entering eCOA data using a provisioned device	53	79.1
Entering eCOA data via a web-based system	45	67.2
Training participants to use an eCOA App on their own device (BYOD)	40	60.0

Notes.

^aTwenty countries in total. The full list is available in Appendix B.

^bMore than one response selectable.

^cRespondents' therapeutic-area experience is listed in Table 2 in Appendix B.

3.2. eCOA site feasibility assessment experience

Sites were asked whether Sponsors assessed their eCOA experience prior to engaging them as a site. Thirty-one (46.3 %) respondents reported sponsors’ assessed site eCOA experience as part of site feasibility, 27 (40.3 %) indicated eCOA experience was assessed, but not robustly, and 9 (13.4 %) reported that eCOA experience was not assessed before studies were awarded.

3.3. eCOA training, training manuals and site readiness for eCOA studies

When asked about eCOA training preferences, twenty-five (37.3 %) reported the most effective to be workshop/hands-on training during an Investigator Meeting (IM). Twenty-four (35.8 %) preferred a pre-recorded online video, and 10 (14.9 %) opted for live, online training. Seven (10.5 %) opted for training during site initiation. When asked what additional training might be helpful right before study start, the majority said “preparing eCOA equipment before a participant visit” (31; [46.3 %]) or “creating and registering the participant” (22; [32.8 %]).

When asked whether sites were provided the option to practice using eCOA before having to register participants, 37 (55.2 %) responded rarely or sometimes having the chance to practice and that it was not sufficient. Seven (10.5 %) said they never had the opportunity to practice, and seven (10.5 %) reported they practiced all the time.

Sites reported wanting to resolve issues without the help desk. Fifty-seven (85.1 %) reported eCOA manuals are useful. Forty-one (61.2 %) reported the level of information in manuals is about right and seven (10.5 %) reported they contained too much information. Nineteen (28.4 %) indicated they included too little information, and the three most important topics on which they wanted more information were “assistance with troubleshooting” (10; [52.6 %]), “how to get started with the device” and “instructions about device management” (7; [36.8 %] each), and “how to order a replacement device” (2; [10.5 %]). See Tables 3, 4 and 5 in Appendix B for details.

The seven respondents who said there was too much information in eCOA site manuals were asked what three most important topics needed to be included. They reported, “how to create a participant in the system” and “how to get started with the device” (5; [25 %] each), and “instructions about device management” (3; [15 %]). Respondents provided free-text suggestions for site readiness for eCOA and commented on the benefits of eCOA manuals with clear, step-by-step instructions including screenshots -particularly of the participant's view of the system, to understand what the participant would see and to help with troubleshooting. Other common themes in the free-text suggestions were requests for more guidance on troubleshooting, being able to create test participants in the system to have hands-on practice as well as having access to refresher training before the first study visit. Also enabling participant setup in the system before participants arrive at site could help to identify and resolve technical issues in advance.

3.4. Participant perception of eCOA

Respondents were asked if patients react positively to completing questionnaires electronically. Across the sample, 41 (61.2 %) responded “Most of the Time,” 19 (28.4 %) reported “Sometimes,” and 5 (7.5 %) reported “All of the Time,” while only 2 (3.0 %) indicated “Almost never.” To evaluate a potential relationship between sites' attitudes toward eCOA and their own impression of the participant reaction to eCOA, we examined responses to this same item by responses to each point on the 0–10 numeric rating scale (NRS) item “What is your opinion when you find out that eCOA technology will be included in a trial?”, for which 0 was “I would prefer for eCOA not to be included in a trial” and 10 was “I would welcome eCOA in a trial.” The two respondents who indicated study participants “Almost never” react positively to eCOA responded “1” and “5’” on the NRS. One of the 5 respondents who noted participants react positively to eCOA “All of the Time” selected “5” on the NRS, while the other 4 selected “10”.

3.5. Participant eCOA compliance

Respondents report it is easy to tell why a participant has been non-compliant most of the time (44; [65.7 %]), with the main reasons being: participants are not familiar enough with the technology (19; [28.3 %]), participants experience device issues (16; [23.9 %]), too time-consuming (15; [22.4 %]), and participants lose interest (11; [16.4 %]).

3.6. Issue management/Technical helpdesk

Technical support should be provided to sites and participants, but it is more efficient if sites are able to resolve device issues themselves. Most respondents confirmed being able to resolve issues without contacting the helpdesk “Most of the Time” (44; [65.7 %]) or “Sometimes” (20; [29.9 %]), but most (62; [92.5 %]) would like to resolve more issues themselves.

Respondents selected up to three ways that eCOA helpdesk support could be simplified/improved, the most common being “Live chat” (49; [28.3 %]), “Resolve the issue on first contact” (33; [19.1 %°), and “24/7 availability” (31; [17.9 %]). Other responses were “Resolve the issue in 15 min” (27; [15.6 %]), “Always provide both email and telephone contact options” (25; [14.5 %]), and “Always available during working hours” (18; [10.4 %]).

3.7. Bring your own device (BYOD)

BYOD approaches were supported in our survey. Of note, BYOD has a higher preference rate versus a provisioned device (PD), with 25 respondents (37.3 %) preferring BYOD, 21 preferring PD, and 21 (31.3 %) reported no preference. Of the 40 who reported having trained participants to use BYOD, 23 (57.5 %) reported a preference for BYOD.

Respondents preferring BYOD cited flexibility and convenience for participants as reasons for their preference, with quotes using terms like “easier,” “convenience” and” more used to their own phones.” Other common reasons related to logistics (6 mentions, e.g., as “BYOD avoids the challenges that may come with shipping devices […]) and higher compliance (cited twice, e.g., “in my experience, […] there is much higher compliance all over the study period”).

Reasons for PD preference included incompatibility of participants’ personal phones, accounting for 13 citations (e.g., not all subjects have optimal devices”). Of the remaining reasons, data privacy risks (cited 3 times, e.g., “some participants fear data privacy and access to their personal devices”) and simpler issue management with one provisioned device type (cited 3 times, e.g., “The variability introduced with BYOD makes it more difficult to troubleshoot problems”).

3.8. eCOA data collection backup approaches

eCOA backup strategies vary across sponsors, but all respondents reported an electronic backup option as necessary. When estimating how often an electronic backup solution is available, eight (11.9 %) selected “Always,” 10 (14.9 %) “most of the time,” and 29 (43.3 %) “sometimes,” while 18 (26.9 %) said “rarely.”

When asked how many electronic backup options an eCOA study should have, 35 (52.2 %) respondents reported one, and 32 (47.8 %) respondents answered two.

Respondents were also asked when they train participants to use an electronic backup option. Thirty-four (50.7 %) responded “only when needed,” while 23 (34.3 %) indicated “at first patient visit.” Ten (14.9 %) chose “Not applicable,” which might indicate either the respondents’ studies did not include a backup option, the available backup solution was never needed, or the site asked the participant to use a backup without training.

3.9. Virtual site visits

Thirty-five (52 %) respondents reported it is easier to conduct a virtual visit compared to an in-clinic visit. Across roles, respondents in the US were more likely to prefer a virtual visit (26; [70.3 %] of US respondents) than respondents from the other countries (9; [30.0 %]). Those who indicated a virtual visit would be easier indicated “simpler to organize “(13 [37.1 %]), “cancellation is less likely” (7 [20.0 %]), “a virtual visit takes less time” and “the patient is more relaxed outside of the clinic” (both 5 [14.3 %]) as justification.

3.10. Sites’ general reaction to eCOA

The respondents indicated a general preference for eCOA. Using an 11-point Numeric Rating Scale (NRS) for which 0 was “I would prefer for eCOA not to be included in a trial,” and 10 was “I would welcome eCOA in a trial,” 40 (59.7 %) responded “8” or above and 22 (32.8 %) selected “6” or “7”.

All Phy/Clins and PIs (2; [100.0 %] and 5; [100.0 %], respectively), the majority of SCs (45; [95.7 %]), CRAs (4; [80.0 %]), and Admins/SNs (3 each; [75.0 %]) scored “5” or above. Among US respondents, for all roles except SN (0) and SC (12 of 26 SCs; [46.2 %]), the majority scored “8” or above.

3.11. General opinions on improving eCOA systems

Respondents were asked to identify one thing eCOA providers should improve. Sixty-three free-text responses were offered, and almost a third pertained to simplifying device interactions, including easy, brief, user-friendly questionnaires and interfaces (19; [30.2 %]), e.g., “Make the user interface as easy as possible, large font and buttons, easy to navigate,” “The duration of the questionnaires. I would do them very simple and with few steps” “Only capture necessary data,” “Be cautious about the number and frequency of the questionnaires.” The second most prevalent response pertained to improving technical help desk and troubleshooting support (7; [11.1 %]), e.g., “Having a helpdesk which is easy to contact and has provision to answer queries promptly,” “Better troubleshooting options,” “Live chat support for real-time issues.” The full list of concepts and the associated respondent quotes can be found in Table 6 in Appendix B.

4. Analysis/discussion

Most respondents had experience entering eCOA data on a PD and training trial participants to do so. Fewer, but still a substantial number, had experience using web-based eCOA and training participants to use BYOD. Further, respondents have used eCOA in trials across a wide variety of therapeutic areas (TAs). Therefore, we believe the sample was sufficiently versed in eCOA procedures to contribute meaningful data to this survey.

Generally, the respondents reported feeling positive when learning that eCOA would be used in a trial. There was an overwhelmingly positive response to the related NRS item from most respondents. Further, when asked their impression of participants’ attitudes toward eCOA, most reported a positive response at least some of the time. These results suggest that sites have an appreciation for the importance of eCOA data collection as well as confidence in their ability to learn and use eCOA. Only two respondents who indicated that patients “Almost never” react positively to eCOA responded with a low score on the NRS evaluating their own perception of eCOA, while most of the respondents who noted patients react positively to eCOA selected a “10”. These results suggest that site personnel who believe participants like eCOA report that they themselves are comfortable with it. Conversely, sites may report not wanting to use eCOA if they think their participants dislike it. It could be hypothesized that sites could base their perception of eCOA upon that of their participants, but further investigation is necessary to draw firm conclusions.

The results unequivocally indicate a need to better prepare sites to use eCOA pre-study. Sites generally did not feel well prepared to use eCOA. Insufficient assessment of sites’ eCOA experience was described, as well as a lack of opportunities to practice. Assessing site staff experience is key to providing adaptive training programs that suit site staff needs.

Hands-on training at IMs may be logistically challenging but is considered the most effective. If time and resources are prohibitive, a recorded training developed with the eCOA provider is recommended, requiring completion and understanding before enrollment begins. Some respondents reported “training at the site initiation visit” as the best approach, which suggests some may prefer to be trained closer to the site's first patient visit. A recorded training video would also satisfy these sites' desire for just-in-time training. Respondents commented that they would benefit from practicing prior to using the eCOA system with their first patient; results indicated that setting up devices for their first study participants is often challenging. This process must be made simple, clearly described and intensively trained with opportunities for practice.

eCOA site manuals are important for ensuring sites’ success with eCOA. Respondents considered manuals to be useful, especially for supporting device preparation and management. Other feedback suggested frustrations with sub-par device functionality, battery life, and charging issues, and that internet connectivity is still a problem for some participants. Training and user manuals must address these issues, especially Wi-Fi capability and procedures to enable it, as well as provide contact information for the technical help desk.

While some sites report usually resolving device issues without the helpdesk, they would like to address more issues independently which could be achieved with adequate training and troubleshooting tips within the manual. When sites do need to contact the help desk, quicker resolution could be achieved via a 24/7 live helpdesk chat, which was the most popular suggestion for helpdesk improvement. This functionality has become available, yet it is notable that in the experience of the co-authors, sponsors are not choosing to deploy it in their studies. This may be due to perceived cost implications, or perhaps a lack of awareness of its existence.

Some technical issues may lead to participant noncompliance. Site manuals should include the top reasons for participant non-compliance and recommendations for resolving them. The principal reasons for non-compliance reported in this study were participants' unfamiliarity with eCOA and device-related issues like connectivity and forgetting devices. Sites should train participants on the importance of and rationale for PRO measures and their responsibility to record data per protocol. Monitoring compliance on the eCOA provider's web portal shortly after each participant begins collecting data may help identify non-compliance as well as data upload or connectivity issues.

Sites noted that written participant training materials are helpful but rarely provided, and adequate time to train participants must be allowed for. Again, this feedback speaks to the need for sufficiently detailed participant training materials, which may be only as extensive as a quick reference guide if sites have sufficient time to train participants to proficiency.

Issues with connectivity and participants not charging or carrying their devices may be overcome with a BYOD approach. Overall, the results showed a slight preference for BYOD and showed that a third of the sample reported no preference for either BYOD or PD; but note that most of the respondents preferring PD justified their preference by stating not all participants own or are willing to use their own devices. As such, their rationale is less based on personal preference but more on the fact that BYOD is not an option for some patients, which leaves the site with no choice but to assign a provisioned device. As such, the number of respondents who reported a preference for PD based on actual choice was small. However, further research with larger samples will be necessary to confirm these findings.

A qualitative study that compared participants' experience using a PD versus BYOD demonstrated that participants’ experience completing PRO measures was consistent across both PD and BYOD, with preference evenly split [7]. In light of this, our findings might suggest that study sites would be satisfied with the selected method of data collection as long as their study participants are.

Respondents experienced in BYOD preferred BYOD over PD. This highlights that sites may be accepting BYOD once they use it, and proposing a BYOD approach in a clinical trial might not only meet site or participant preferences but offer choice and flexibility. Reasons for preferring BYOD pertained to convenience: participants are familiar with their own device, attend to it more than a PD, and are more likely to have it with them to complete ePROs. Sites also report inventory management and likelihood of needing device replacement are reduced with BYOD, which reflects the sites perception of BYOD as a way to reduce logistics burden and increase convenience.

Preference for PDs was generally to accommodate participants who do not own a suitable device or choose not to use their smartphone. Considering rates of smartphone saturation are considerably lower in countries outside North America and Europe (except South Korea) [8], sites in these regions may have concerns about BYOD; PDs may be easier to set up and troubleshoot compared to the variety of BYODs.

The results illustrate that BYOD has been used across a broad range of TAs and most respondents who had BYOD experience either preferred it or expressed no preference. While we cannot draw firm conclusions based on such small sample sizes from non-US countries, the results generally suggest BYOD would be accepted globally by BYOD-experienced sites.

Irrespective of device approach, backup solutions may be implemented and electronic is the recommended solution [9]. While all respondents agreed a backup should be provided for PDs, a third reported sponsors rarely or never provide backups and almost half reported backups are only “sometimes” used in their trials. Electronic backups are becoming more common and are often provided routinely as a complement to the primary eCOA mode.; and so, if this survey were to be replicated, we would expect to see higher rates of electronic backup use.

As decentralized clinical trials become more prevalent, we expect an increase in the number and acceptance of virtual clinic visits. Respondents were divided regarding whether virtual would be easier than in-person visits. Those who noted virtual would be easier cited convenience and fewer cancellations, while those who preferred in-clinic noted challenges with assessing health status or symptoms and connectivity with virtual. While adoption increased during the pandemic, the experience of virtual visits remains recent and limited, and we could also expect uses to evolve in the near future.

When asked to suggest one thing eCOA providers could improve, simplifying device interactions was dominant.

Long, complicated COA measures, diary questions, and general data recording requirements are burdensome and may result in noncompliance. We recommend, as suggested by respondents, to “simplify […] questionnaire schedule to increase compliance”, “only capture necessary data” and” be cautious about the number and frequency of questionnaires.”

5. Study limitations

The site/CRA questionnaire was dispatched to several groups of respondents with multinational memberships. However, as participation was voluntary, only 67 useable responses were obtained within the timeline that was allocated to the project, despite repeated reminders, and as such may the significance of these results. Most respondents were from the US and had a site/study coordinator role; additional research in a larger population specifically including other geographies and alternative roles would be beneficial to help draw firm conclusions on the questions related to trends by country or role. Another objective of the broader project is to survey ePRO experienced trial participants, to understand from their perspectives the value and the challenges they experience when completing ePROs, and gather suggestions for simplification that would result in more flexible and participant-centric approach to ePRO implementation. Such work is in early planning within the C-Path GBTI Support Flexible Approaches to PRO Data Collection Project team.

6. Summary and recommendations

Despite the study limitations, this research sheds light on ways to improve the usability of eCOA solutions and, as such, contributes to support more flexible approaches to eCOA data collection. We propose recommendations to simplify the user's experience; following these recommendations based on feedback from sites/CRAs should support increasing flexibility and usability of eCOA systems.

• •Thoroughly assess sites' eCOA experience and confidence during site feasibility.
• •Provide sites with on-time and flexible training, including hands-on training during IMs.
• •Dedicate more time to training and refreshers before participant enrollment.
• •Explain the value of eCOA to sites prior to study start.
• •Develop effective manuals that include detailed instructions on start-up steps such as device set-up, adding a new participant, and troubleshooting.
• •Design eCOA to be user-friendly, collect only necessary data and use short COA measures and simple reporting schedules.
• •Develop more efficient technical-help desk support such as a live-chat feature.
• •Consider using a hybrid PD/BYOD approach with an electronic backup option.

In addition, we recommend the performance of rigorous User Acceptance Testing (UAT) to confirm functionality and demonstrate usability of the eCOA system.

Acknowledgment of federal support

Critical Path Institute is supported by the Food and Drug Administration (FDA) of the U.S. Department of Health and Human Services (HHS) and is 55 % funded by the FDA/HHS, totaling $17,612,250, and 45 % funded by non-government source(s), totaling $14,203,111. The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement by, FDA/HHS or the U.S. Government.

Additional support for the eCOA Consortium comes from membership fees paid by members of the eCOA Consortium (https://c-path.org/programs/ecoac/).

Additional support for the PRO Consortium comes from membership fees paid by members of the PRO Consortium (https://c-path.org/programs/proc/).

Authors’ contributions

Estelle Haenel led the team of contributors. All authors contributed to the design and acquisition of the data for this research, drafted the work or revised it for critically important intellectual content, approved the final version to be published, and agreed to be accountable for all aspects of the work.

CRediT authorship contribution statement

Estelle Haenel: Conceptualization, Formal analysis, Methodology, Project administration, Resources, Supervision, Validation, Writing – original draft, Writing – review & editing. Celeste A. Elash: Conceptualization, Formal analysis, Project administration, Writing – original draft, Writing – review & editing. Katie Garner: Formal analysis, Methodology, Writing – original draft, Writing – review & editing. Megan Turner: Conceptualization, Methodology, Validation, Writing – review & editing. Scottie Kern: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Validation, Writing – original draft, Writing – review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article: