Abstract
Introduction
Digital health technologies (DHTs) provide opportunities for real-time data collection and assessment of patient function. However, use of DHT-derived endpoints in clinical trials to support medical product labelling claims is limited.
Methods
From November 2020 through March 2021, the Clinical Trials Transformation Initiative (CTTI) conducted a qualitative descriptive study using semi-structured interviews with sponsors of clinical trials that used DHT-derived endpoints. We aimed to learn about their experiences, including their interactions with regulators and the challenges they encountered. Using applied thematic analysis, we identified barriers to and recommendations for using DHT-derived endpoints in pivotal trials.
Results
Sponsors identified five key challenges to incorporating DHT-derived endpoints in clinical trials. These included (1) a need for additional regulatory clarity specific to DHT-derived endpoints, (2) the official clinical outcome assessment qualification process being impractical for the biopharmaceutical industry, (3) a lack of comparator clinical endpoints, (4) a lack of validated DHTs and algorithms for concepts of interest, and (5) a lack of operational support from DHT vendors.
Discussion/Conclusion
CTTI shared the interview findings with the US Food and Drug Administration (FDA) and the European Medicines Agency (EMA) and during a multi-stakeholder expert meeting. Based on these discussions, we provide several new and revised tools to aid sponsors in using DHT-derived endpoints in pivotal trials to support labelling claims.
Keywords: Digital health technology, Regulatory trial, Endpoint, Clinical outcome assessment
Introduction
Use of digital health technologies (DHTs) [1] continues to increase as more health care providers, clinical researchers, and organizations funding clinical research to support the development of new therapeutics (i.e., “sponsors”) turn to digital tools to provide a comprehensive view into individuals’ health and wellness [2–4]. Additionally, patients have expressed interest in using more DHTs in clinical trials [5]. DHTs, such as mobile applications, smart devices, sensors, and wearables, may provide opportunities for richer data collection and better tracking and management of health conditions [2, 6, 7]. Because sensor-generated data are often collected via passive monitoring in a patient’s living environment, DHTs have the potential to capture more realistic assessments of patient function. As a result, using DHT-derived clinical outcome assessments (COAs) [1] to measure change in disease progression or therapeutic effectiveness in clinical trials is of growing interest to trial stakeholders [5, 8, 9].
Similar to any new COA, novel DHT-derived COAs must be well-defined and reliable when measuring a clinically relevant concept of interest1 within a particular context of use2[10]. Multi-stakeholder partnerships like the Clinical Trials Transformation Initiative (CTTI) and the Digital Medicine Society (DiMe) provide tools and frameworks outlining the development of DHT-derived COAs to promote their use in regulatory clinical trials [11, 12]. The DiMe V3 framework describes at least three distinct phases for developing DHT-derived clinical trial endpoints: (1) verification of the DHT, (2) analytical validation of the outcome, and (3) clinical validation of the endpoint [11]. CTTI’s hub of digital health trial recommendations and tools [12] also provides a framework for developing DHT-derived endpoints and includes suggestions for selecting and testing DHTs, managing remote DHT data capture, and supporting decentralized trial design and conduct. Additionally, Walton et al. [13] described important elements to include in an evidence dossier for regulators that support use of DHT-derived COAs in clinical trials.
Yet, use of DHT-derived COAs as a primary or secondary clinical trial endpoint [1] (i.e., “DHT-derived endpoints”) to support the labelling claim of a therapeutic product is uncommon. In 2019, the European Medicines Agency (EMA) issued the first qualification opinion on a DHT-derived endpoint, stride velocity 95th centile (SV95C), which measured ambulation in Duchenne muscular dystrophy via a wearable device. The EMA approved SV95C for use as a secondary endpoint in a phase III clinical trial [14, 15]. However, to our knowledge, at the time of our investigation, no new medical product had been approved based on primary evidence from a DHT-derived endpoint. A 2022 review of 402 “main studies” in European Public Assessment Reports reported that no study included a digital mobility endpoint for use in marketing authorization [16]. Despite recognized benefits and opportunities to share the use of DHT-derived endpoints in trials, such as in DiMe’s open-sourced Library of Digital Endpoints [4], challenges remain in evaluating the readiness of DHT-derived COAs to serve as clinical trial endpoints [17, 18].
Sponsors designing and conducting pivotal trials for medical product approval must perceive that the benefits of using a DHT-derived endpoint outweigh the possible risks [19]. Building on previous work [20], CTTI initiated a project to identify barriers that trial sponsors face in readily using DHT-derived COAs as primary and secondary endpoints for informing label claims and to identify potential solutions for overcoming these barriers. Here, we present findings from interviews with trial sponsors and offer recommendations to support further development and use of DHT-derived endpoints in clinical trials.
Methods
We conducted a qualitative descriptive study using semi-structured interviews with representatives of sponsors of clinical trials that used DHT-derived endpoints. We aimed to learn about their experiences, including their interactions with regulators and the challenges they encountered and overcame.
Trials were identified through suggestions from CTTI’s multi-stakeholder project team as well as searching the following online resources: DiME’s Library of Digital Endpoints [4], US Food and Drug Administration’s (FDA’s) COA Qualification Program submissions for a DHT COA [21], and ClinicalTrials.gov[22] for interventional studies within the last 10 years filtered for industry funder type and the search terms “digital,” “wearable,” “sensor,” and “accelerometer.” We then purposively selected [23] trials based on whether the DHT-derived endpoint used in the study (1) supported (or could support) a labelling claim, (2) objectively measured a functional clinical outcome, and (3) was a primary or secondary endpoint. We aimed to include trials representing a variety of therapeutic areas and DHT types. We excluded trials that (1) solely relied on electronic patient-reported outcomes, due to our focus on objective measures, or (2) only used digital biomarkers as these do not measure patient function, per definition [1]. Overall, we aimed to focus our inquiry on the use of DHTs to measure performance outcome measures [1], which, at the time, comprised the majority of objectively measured digital COAs in clinical research [8]. Members of the CTTI interdisciplinary project team – representatives from patient groups, the biopharma industry, DHT manufacturers, regulatory agencies, and academia – used their professional networks to identify sponsor representatives of the selected studies. We invited these sponsors to participate in an interview if they had first-hand knowledge of the trial’s design and discussion around including a DHT-derived endpoint. Selected sponsor representatives were asked to invite colleagues to participate in the interview if their input could inform the discussion.
We conducted interviews from November 2020 through March 2021 via online videoconference. We asked sponsors to describe their decision-making process for selecting a DHT-derived endpoint, including the specific DHT used in the trial, and their process for verifying the DHT, validating the endpoint, and ascribing clinical meaning to change measured by the DHT-derived COA. All interviews were audio recorded, excluding one at the sponsor’s request in which detailed notes were taken and responses documented in a debriefing form. Audio recordings were transcribed verbatim following a transcription protocol [24].
Using applied thematic analysis [25], we organized and coded the transcripts and interviewer notes with NVivo [26]. Two analysts first deductively identified and applied a priori codes to capture all information shared on the interview topics, grouping together similar conversations across all the interviews (e.g., lessons learned). After conducting inter-coder reliability assessments to ensure consistent coding, analysts reviewed the coding reports for each a priori topic and inductively identified content-based codes that summarized sponsor statements about each topic (e.g., specific lessons learned). After grouping these content-based codes thematically in a codebook, analysts then applied these codes to the data, conducting additional inter-coder reliability assessments to ensure consistent application. When inductive coding was complete, analysts created frequency tables and matrices to visualize the relative salience of each code. Using these tools, analysts identified themes that emerged across multiple interviews and in different trial contexts. Thorough summaries of emergent themes were described in analytical reports and linked to key illustrative quotes from participants.
Results
Demographics
We conducted 11 interviews with 20 sponsors representing 11 trials and 10 different biopharmaceutical organizations, five of which were group interviews with 2–4 participants. Overall, 17 trials were initially identified; however, one sponsor declined to participate in an interview because of a lack of availability; another declined because they were not ready to share insights about an ongoing trial; and 4 sponsors were unresponsive to our invitation for an interview.
All but one of the sponsor organizations was a large-sized company with an estimated market cap of over $10 billion; the other organization had a reported market cap of less than $300 million. Sponsor representatives had been involved in protocol development and strategy and included principal research scientists, directors, chief medical officers, regulatory leads, and senior statisticians (Table 1). Various phases of trials were represented (Table 2). Four trials were initiated in 2020; the remaining trials were initiated in 2014–2018 (Table 2). We interviewed sponsors of two observational studies validating DHT-derived endpoints, which were conducted in parallel with early-phase trials of medical products with plans to incorporate the endpoints into later-phase trials. DHTs included commonly used technologies, such as commercial wearables and smartphones, and technologies intended for clinical trial purposes only. Trials focused on developing medical products for a variety of therapeutic areas, including neurology (n = 5), pulmonology (n = 2), rare diseases (n = 2), orthopedics (n = 1), and cardiology (n = 1). At the time of the interviews, findings from one trial had been submitted to European regulators to add to the product label. Several other trials were still ongoing.
Table 1.
Participant demographics
| Current role, n (%) | |
| Senior research scientist, senior research scientist-statistician | 5 (25) |
| Global program leader, DHT leader | 5 (25) |
| Director (digital health/strategies, health economics, and outcomes research) | 4 (20) |
| Chief medical officer, VP clinical operations, head of medical affairs | 3 (15) |
| Regulatory lead | 3 (15) |
| Years in role, n (%) | |
| Less than 1 year | 5 (25) |
| 1-2 years | 5 (25) |
| 3-4 years | 4 (20) |
| 5–10 years | 6 (30) |
| More than 10 years | 0 (0) |
| Total years engaged in development or use of DHT-derived endpoints in clinical research, n (%) | |
| Less than 1 year | 0 (0) |
| 1-2 years | 2 (10) |
| 3-4 years | 9 (45) |
| 5–10 years | 8 (40) |
| More than 10 years | 1 (5) |
Table 2.
Type of endpoint by study type and use in a regulator submission
| Endpoint type by study type | n (%) | Trial start dates | DHT-derived endpoint used in a regulatory submission | ||
|---|---|---|---|---|---|
| yes | no | ||||
| trial closed early due to futility of investigational product | trial ongoing1 | ||||
| Observational2 | 2 (18) | 2014 and 2018 | 0 | 1 | 1 |
| Phase 2 | 3 (27) | 0 | 0 | 3 | |
| Exploratory | 1 (9) | 2017 | 0 | 0 | 1 |
| Primary or secondary | 2 (18) | 2016 and 2020 | 0 | 0 | 2 |
| Phase 3 | 5 (45) | 0 | 1 | 4 | |
| Exploratory | 1 (9) | 2018 | 0 | 0 | 1 |
| Primary or secondary | 4 (36) | 2017 and 20203 | 0 | 1 | 3 |
| Phase 4 | 1 (9) | 1 | 0 | 0 | |
| Primary or secondary | 1 (9) | 2015 | 1 | 0 | 0 |
DHT, digital health technology.
1These trials were ongoing at the time of the interview.
2These observational studies focused on validating DHT-derived clinical outcome assessments for use in later-stage trials of investigational products already in development.
3Three of the four trials were initiated in 2020.
Challenges Experienced
Sponsors identified five key challenges to incorporating DHT-derived endpoints (Fig. 1).
Fig. 1.
Five key challenges to incorporating DHT-derived endpoints in clinical trials as identified by sponsors. COA, clinical outcome assessment; DHT, digital health technology.
Need for Additional Regulatory Clarity Specific to DHT-Derived Endpoints
Several sponsors stressed the difficulty in including DHT-derived endpoints in pivotal trials when there was no official regulatory guidance specific to the use of DHT-derived endpoints available when planning their trial. In response, some sponsors met with key opinion leaders and clinicians to identify available DHTs and suitable DHT-derived COAs. Other sponsors reviewed the FDA’s 2009 guidance for PRO development [27] stating that it offered insight into the evidence regulators would need to determine whether a new endpoint was acceptable for use in a pivotal trial. One sponsor added that it consulted other publicly available recommendations, such as CTTI’s recommendations on novel endpoints and DHT selection [28]. Some sponsors noted that without specific guidance from regulators, it was unclear how much evidence was needed for regulatory approval, and this made it difficult to plan the trial’s endpoints.
Some sponsors added that the current rate of DHT innovations and endpoint development outpaces the speed of regulatory review and may impact regulators’ ability to provide up-to-date guidance. One sponsor said:
One of the problems we face is that the technologies for the digital endpoints evolve so rapidly that usually with the time it takes to qualify them, often the technology is already outdated […] and the regulators are always a couple of steps behind (observational study sponsor).
Official COA Qualification Is Not Practical for the Biopharma Industry
Several sponsors indicated that the FDA COA qualification process is too tedious and time consuming for industry investment. Although sponsors acknowledged that qualification was not required to use the COA as an endpoint in a trial, they often stated that a qualified endpoint was preferred because it implied regulatory acceptance of the endpoint for a particular concept of interest. They noted, however, that the role of biopharma is to develop new therapeutics and not new COAs and trial endpoints. These sponsors preferred to spend their resources generating evidence to support the acceptance of a DHT-derived endpoint for a specific clinical trial in their product development pathway rather than qualifying a new COA for general use. Sponsors noted that if the investigational product does not proceed to later-phase trials, the DHT-derived endpoint may not progress in development. One sponsor said:
One of the challenges is the actual procedure for qualification of the endpoints. At the moment, the COA qualification process is so long and requires so many resources that companies don’t see the value. So, it’s usually academic consortia that go through this and not the companies involved (phase 3 trial sponsor).
Sponsors understood that multiple studies may be needed to evaluate a new COA to demonstrate accuracy and relevance given unique trial contexts. Sponsors felt that more details from regulators are needed to determine how DHT-derived COAs that are qualified for a particular concept can be used as endpoints in different therapeutic areas and patient groups without having to re-evaluate the COA for slight variations in their context of use. This would be particularly beneficial in rare disease trials, where the ability to conduct separate formative studies on the DHT-derived COA may be limited. Sponsors said that rare disease trials would have a difficult time gathering the body of evidence required if separate clinical studies were necessary to demonstrate the readiness of a DHT-derived endpoint for every unique context of use. One individual representing a rare disease trial said:
The ability to generate a package of evidence to put to the regulators for qualification for a particular endpoint is quite limited because there are so few patients. And many of the patients are already enrolled in trials, so there are not that many patients available to do this type of work (phase 3 trial sponsor).
Sponsors also said that organizational restrictions on sharing proprietary information limit dissemination of lessons learned when developing new COAs. This often results in trial sponsors working independently to develop new COAs and trial endpoints.
Lack of Comparator Clinical Endpoints
Sponsors recalled having to compare the DHT-derived endpoint with an established clinical endpoint to demonstrate that their novel assessment was capable of measuring change in disease progression. Some expressed concern with this process because DHT-derived endpoints are often capable of measuring a more patient-centric and objective assessment of the concept of interest than existing established endpoints. Sponsors explained that DHT-derived endpoints and the established endpoint often measure different aspects of a disease or concept of interest, making a direct comparison impossible. If the DHT-derived endpoint and established endpoints were at least evaluating the same concept, sponsors explained that there would then be uncertainty about how similar the changes between the two measures should be. One sponsor described the challenge of correlating novel and established endpoints:
There’s this fascinating philosophy, a science question that no one has an answer to, which is, if digital health tech shows perfect convergence with the established clinical endpoint, in principle, that sounds great. In practice, that’s really not ideal, because you’re not trying to create a duplicate version of what we already used in easily scalable fashion in clinical trials. What you’re really trying to do is create something that can supplement or add value to, or become the next generation of measurements in clinical trials. And so, what you ideally want to see is some conversion, some correlation […] And it’s a really weird philosophy. What kind of correlation is good correlation? We don't have a good answer (observational study sponsor).
Lack of Validated DHTs and Algorithms for Concepts of Interest
Sponsors often noted that very few DHTs were available given their trial context when they were designing their trial. One sponsor from a phase 2 trial using a commercial fitness/activity monitor said:
It’s very hard to find a validated device that has been studied in that population. We didn’t want to select a DHT in the trial that hasn’t at least been used in somewhat similar context, or in this disease space. So, that limits the options (phase 2 trial sponsor).
They described that the limited availability of verified and validated measurement tools affected what they could measure. A sponsor recalled that during trial planning, they discovered that some of their desired measures were not feasible with the commercial DHT they selected. They reluctantly adjusted their trial design to fit the capability of the available DHT.
Lack of Operational Support from Technology Vendors
Sponsors said that few DHT vendors have clinical trial experience. They explained they often must rely on technology vendors, or other third-party contract research organizations, to provide operational and infrastructural support, particularly when patients are directly interacting with the data collection tools. Sponsors further explained that vendors must follow specific processes to meet or exceed regulations for conducting regulatory trials, but few have the experience to provide this level of support. Sponsors require DHT vendors to provide the necessary evidence and a description of the procedures used to analytically validate the DHT and algorithm for the concept of interest, security protocols that protect patient privacy, and proven ability to produce quality data that can stand up to regulatory inspection. One sponsor voiced that while this expectation is applied to all vendors, not just DHT vendors, DHT manufacturers may be “less mature, and so, they may not have all of the regulations ready to go,” thereby narrowing the pool of vendors with whom they could form a partnership.
Recommendations from Sponsors
Acknowledging the challenges, sponsors offered recommendations to support the use of DHT-derived endpoints in clinical trials (Fig. 1). The recommendations included (1) engaging key stakeholders early, (2) incorporating DHT-derived endpoints in early-phase trials and observational studies, (3) investing in COA development initiatives, and (4) engaging technology manufacturers early in the development process.
Discussion/Conclusion
The barriers and solutions addressed in this study can enable the adoption of DHT-derived endpoints in future pivotal trials. Our findings mirror many of the development challenges recently described for these endpoints [29–31]. Collectively, these findings suggest that sponsors must demonstrate that a DHT-derived endpoint (1) assesses a meaningful aspect of patient health, (2) addresses an unmet measurement need, (3) is capable of and accurate for measuring the relevant health outcome, and (4) can detect change in disease progression, or therapeutic effectiveness, that is clinically meaningful to stakeholders [29–33]. Despite collaborative efforts to share DHT-derived endpoint use [4, 8], our findings indicate that sponsors need additional guidance to adequately and appropriately address these requirements.
After data collection and analysis was completed, CTTI shared the interview findings at meetings with clinical trial experts, a Critical Path Innovation Meeting (CPIM) with the FDA, and an Innovation Task Force (ITF) Briefing with the EMA. The CTTI project team updated existing CTTI recommendations on developing and using novel endpoints [34, 35] based on these discussions and created new resources to enhance use of DHT-derived endpoints in pivotal trials.
In addition to CTTI, other multi-stakeholder consortia, such as DiMe [4, 11] and TransCelerate [36], are actively working to develop tools and resources to advance the use of digitally derived endpoints in trials. Members of the CTTI’s project team who were also involved with the DiMe and TransCelerate initiatives helped to ensure that CTTI’s new recommendations were aligned with existing resources. Taken together, these resources provide a robust toolkit from which to advance the field. The CTTI recommendations stress the importance of initially identifying outcomes that are clinically relevant and meaningful to patients [37]. They also include (1) engaging stakeholders in the early stages of product development, (2) selecting technology after identifying the intended outcome, (3) meeting with regulators early and often to discuss the endpoint development process, (4) including DHT-derived endpoints in early-phase trials and observational cohort studies to assess fit-for-purpose and to optimally position the endpoints for interventional trials, and (5) sharing knowledge on developing DHT-derived endpoints to advance their use in other contexts. CTTI anticipates that the exchange of information may provide the scientific basis for developing or using a DHT in a trial, allow investigators and technology manufacturers to invest time and money with an assurance that the results will be universally useful, and increase the end user’s confidence in the output of the technology.
Supporting the recommendations, CTTI created new resources to promote DHT-derived endpoint adoption in future pivotal trials, including (1) a process map for sponsors, academics, and clinical operational partners (including DHT vendors) on preparing a DHT-derived endpoint for use in pivotal trials [38] that outlines key questions to consider and illustrates how to address questions that arise during medical product development; (2) a revised guide on sponsor engagement with FDA and EMA regulators during DHT-derived endpoint development [39]; and (3) sample questions for trial sponsors [40] to use when engaging stakeholders (specifically patients and caregivers, clinicians, and insurers) to identify meaningful outcome measures and whether established or novel endpoints should be used. Questions that sponsors should ask themselves when deciding whether to use a DHT-derived endpoint are also included [40]. Additionally, CTTI provides a revised flowchart that sponsors can use when developing a DHT-derived endpoint, highlighting the separate yet parallel processes of validating (1) the measurement and (2) the digital tool to capture the measurement [34]. These resources provide a robust toolkit for clinical trial stakeholders to advance DHT use in trials.
Clinical trials continue to innovate and the development and use of DHT-derived endpoints will likely continue to be a key part of trial innovations. The number of decentralized trials has rapidly increased in recent years [41] – particularly in response to COVID-19 pandemic-related restrictions – and regulators have issued new guidance to keep pace [42]. The use of DHTs for mobile data capture and outcome assessments will play a key role in future decentralized trials. Additionally, several sponsors have initiated the qualification process of new DHT-derived endpoints [43–45]; however, at the time of our interviews, few trials were using DHT-derived COAs as primary or secondary endpoints in trials to support labelling claims. The rapid development and use of DHTs and DHT-derived COAs in trials poses the main limitation to our findings. The challenges identified in our interviews reflect the state-of-the-field at the time of development and implementation of the trials discussed, which may not reflect current challenges experienced by sponsors using DHT-derived endpoints. It is possible that some of the challenges identified in our interviews are being actively addressed by various stakeholders interested in advancing the use of DHT-derived endpoints in clinical trials. Nonetheless, our findings provide insight into trial sponsors’ experiences using DHT-derived endpoints in pivotal trials that are likely still beneficial for guiding recommendations in the future.
In conclusion, regulators continue to develop guidance for additional clarity. The EMA published a Q&A document on DHT use in clinical trials [46] and the FDA recently released guidance on collecting patient data remotely using DHTs and incorporating patient input into the drug development process that may help elucidate the development process [32, 47]. The FDA also established the Digital Health Center of Excellence and opportunities, such as the CPIM, to facilitate early regulatory engagement across divisions. Similarly, EMA’s ITF Briefing meetings [48] provide an opportunity to seek regulatory advice early on. Future studies that focus on understanding the various roles of key stakeholders to inform and advance the development of novel DHT-derived endpoints may further clear the path for regulatory acceptance of these endpoints.
Acknowledgments
The authors wish to thank the sponsor representatives who participated in our interviews for sharing their candid experience and perspectives. The authors also acknowledge the contributions of the CTTI Novel Endpoint Acceptance project team and thank Brooke Walker for editorial assistance.
Statement of Ethics
The Duke University Health System Institutional Review Board (IRB) determined the study to be exempt from further ethics review (reference number Pro00106989-INIT-1.0). The IRB granted the study a waiver of informed consent. However, all participants received an information sheet prior to participation. The information sheet described the purpose of the interviews, risks and benefits to participation, and acknowledgment that participation was voluntary. It also contained contact information for the principal investigator, study coordinator, and reviewing IRB.
Conflict of Interest Statement
The authors have no conflicts of interest to declare.
Funding Sources
Funding for this work was made possible, in part, by the FDA through a cooperative agreement (U18FD005292) and grant (R18FD005292). Views expressed in this publication do not necessarily reflect the official policies of the Department of Health and Human Services, nor does any mention of trade names, commercial practices, or organization imply endorsement by the US government. Partial funding was also provided by pooled membership fees from the Clinical Trials Transformation Initiative member organizations.
Author Contributions
Brian Perry: conceptualization, data collection and curation, methodology, formal analysis, project administration, supervision, software, visualization, and preparation of the manuscript. Lindsay Kehoe: project administration, recruitment, data interpretation, and review and editing of the manuscript. Teresa Swezey: data collection, formal analysis, and review and editing of the manuscript. Quentin Le Masne: review and editing of the manuscript. Jörg Goldhahn and Alicia Staley: conceptualization and review and editing of the manuscript. Amy Corneli: conceptualization, investigation, methodology, supervision, validation, and review and editing of the manuscript.
Funding Statement
Funding for this work was made possible, in part, by the FDA through a cooperative agreement (U18FD005292) and grant (R18FD005292). Views expressed in this publication do not necessarily reflect the official policies of the Department of Health and Human Services, nor does any mention of trade names, commercial practices, or organization imply endorsement by the US government. Partial funding was also provided by pooled membership fees from the Clinical Trials Transformation Initiative member organizations.
Footnotes
Concept of interest is the specific aspect of health that the trial is directly trying to assess, such as walking ability, scratching, fall risk, etc.
Context of use is the way in which the DHT is used and its purpose of use in the trial.
Data Availability Statement
The data that support the findings of this study are not publicly available because interview transcripts contain information that could compromise the privacy of research participants. Further inquiries about the data can be directed to the corresponding author.
References
- 1. FDA-NIH Biomarker Working Group . BEST (biomarkers, EndpointS, and other tools) resource. Food and Drug Administration (US); 2021. https://www.ncbi.nlm.nih.gov/books/NBK338448/ (accessed December 21, 2022). [PubMed] [Google Scholar]
- 2. Center for Devices and Radiological Health . What is digital health? US Food and Drug Administration. Published online September 22, 2020. https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health (accessed June, 17 2022). [Google Scholar]
- 3. Marra C, Chen JL, Coravos A, Stern AD. Quantifying the use of connected digital products in clinical research. NPJ Digit Med. 2020;3(1):50–5. 10.1038/s41746-020-0259-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Library of digital endpoints. Digital medicine society (DiMe). https://www.dimesociety.org/communication-education/library-of-digital-endpoints/ (accessed June 17, 2022).
- 5. Perry B, Geoghegan C, Lin L, McGuire FH, Nido V, Grabert B, et al. Patient preferences for using mobile technologies in clinical trials. Contemp Clin Trials Commun. 2019;15:100399. 10.1016/j.conctc.2019.100399. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Hassan U, Anwar MS. Reducing noise by repetition: introduction to signal averaging. Eur J Phys. 2010;31(3):453–65. 10.1088/0143-0807/31/3/003. [DOI] [Google Scholar]
- 7. Dodge HH, Zhu J, Mattek NC, Austin D, Kornfeld J, Kaye JA. Use of high-frequency in-home monitoring data may reduce sample sizes needed in clinical trials. PLoS One. 2015;10(9):e0138095. 10.1371/journal.pone.0138095. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Perry B, Herrington W, Goldsack JC, Grandinetti CA, Vasisht KP, Landray MJ, et al. Use of mobile devices to measure outcomes in clinical research, 2010–2016: a systematic literature review. Digit Biomark. 2018;2(1):11–30. 10.1159/000486347. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. McKenna KC, Geoghegan C, Swezey T, Perry B, Wood WA, Nido V, et al. Investigator experiences using mobile technologies in clinical research: qualitative descriptive study. JMIR Mhealth Uhealth. 2021;9(2):e19242. 10.2196/19242. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Center for Drug Evaluation and Research . Qualification process for drug development tools guidance for industry and FDA staff. US Food and Drug Administration; 2020. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/qualification-process-drug-development-tools-guidance-industry-and-fda-staff (accessed May, 31 2022). [Google Scholar]
- 11. Goldsack JC, Coravos A, Bakker JP, Bent B, Dowling AV, Fitzer-Attas C, et al. Verification, analytical validation, and clinical validation (V3): the foundation of determining fit-for-purpose for Biometric Monitoring Technologies (BioMeTs). NPJ Digit Med. 2020;3(1):55–15. 10.1038/s41746-020-0260-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Clinical Trials Transformation Initiative . Digital health trials. CTTI. https://ctti-clinicaltrials.org/our-work/digital-health-trials/ accessed June 17, 2022). [Google Scholar]
- 13. Walton MK, Cappelleri JC, Byrom B, Goldsack JC, Eremenco S, Harris D, et al. Considerations for development of an evidence dossier to support the use of mobile sensor technology for clinical outcome assessments in clinical trials. Contemp Clin Trials. 2020;91:105962. 10.1016/j.cct.2020.105962. [DOI] [PubMed] [Google Scholar]
- 14. Haberkamp M, Moseley J, Athanasiou D, de Andres-Trelles F, Elferink A, Rosa MM, et al. European regulators’ views on a wearable-derived performance measurement of ambulation for Duchenne muscular dystrophy regulatory trials. Neuromuscul Disord. 2019;29(7):514–6. 10.1016/j.nmd.2019.06.003. [DOI] [PubMed] [Google Scholar]
- 15. Committee for Medicinal Products for Human Use . Qualification opinion on stride velocity 95th centile as a secondary endpoint in Duchenne muscular dystrophy measured by a valid and suitable wearable device. European Medicines Agency; 2019:77. https://www.ema.europa.eu/en/documents/scientific-guideline/qualification-opinion-stride-velocity-95th-centile-secondary-endpoint-duchenne-muscular-dystrophy_en.pdf (accessed March 21, 2022). [Google Scholar]
- 16. Jaeger SU, Wohlrab M, Schoene D, Tremmel R, Chambers M, Leocani L, et al. Mobility endpoints in marketing authorisation of drugs: what gets the European medicines agency moving? Age Ageing. 2022;51(1):afab242. 10.1093/ageing/afab242. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Polhemus AM, Kadhim H, Barnes S, Zebrowski SE, Simmonds A, Masand SN, et al. Accelerating adoption of patient-facing technologies in clinical trials: a pharmaceutical industry perspective on opportunities and challenges. Ther Innov Regul Sci. 2019;53(1):8–24. 10.1177/2168479018801566. [DOI] [PubMed] [Google Scholar]
- 18. Mantua V, Arango C, Balabanov P, Butlen-Ducuing F. Digital health technologies in clinical trials for central nervous system drugs: an EU regulatory perspective. Nat Rev Drug Discov. 2021;20(2):83–4. 10.1038/d41573-020-00168-z. [DOI] [PubMed] [Google Scholar]
- 19. Landers M, Dorsey R, Saria S. Digital endpoints: definition, benefits, and current barriers in accelerating development and adoption. Digit Biomark. 2021;5(3):216–23. 10.1159/000517885. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Clinical Trials Transformation Initiative . Developing novel endpoints. CTTI; 2022. https://ctti-clinicaltrials.org/our-work/digital-health-trials/developing-novel-endpoints/ (accessed March 7, 2022). [Google Scholar]
- 21. Center for Drug Evaluation and Research . Clinical outcome assessments (COA) qualification Program submissions. FDA. Published online 2021. https://www.fda.gov/drugs/clinical-outcome-assessment-coa-qualification-program/clinical-outcome-assessments-coa-qualification-program-submissions (accessed January 6, 2023). [Google Scholar]
- 22.Home - ClinicalTrials.gov. https://www.clinicaltrials.gov/ (accessed June 17, 2022).
- 23. Patton MQ. Qualitative research and evaluation methods. Sage Publications; 2002. [Google Scholar]
- 24. McLellan E, MacQueen KM, Neidig JL. Beyond the qualitative interview: data preparation and transcription. Field Methods. 2003;15(1):63–84. 10.1177/1525822X02239573. [DOI] [Google Scholar]
- 25. Guest G, MacQueen KM, Namey EE. Applied thematic analysis. Sage Publications.; 2012. [Google Scholar]
- 26. QSR International Pvt Ltd . NVivo 12 for windows.
- 27. United States Department of Health and Human Services . Food and drug administration, center for drug evaluation and research (CDER), center for biologics evaluation and research (CBER), center for devices and radiological health (CDRH). Guidance for industry patient-reported outcome measures: Use in medical product Development to support labeling claims. US Food and Drug Administration; 2009. https://www.fda.gov/media/77832/download (accessed March 7, 2022). [Google Scholar]
- 28. Clinical Trials Transformation Initiative . Original digital health trials projects. CTTI; 2017. https://ctti-clinicaltrials.org/our-work/digital-health-trials/original-digital-health-trials-projects/ (accessed June 17, 2022). [Google Scholar]
- 29. Goldsack JC, Dowling AV, Samuelson D, Patrick-Lake B, Clay I. Evaluation, acceptance, and qualification of digital measures: from proof of concept to endpoint. Digit Biomark. 2021;5(1):53–64. 10.1159/000514730. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Crouthamel M, Mather RJ, Ramachandran S, Bode K, Chatterjee G, Garcia-Gancedo L, et al. Developing a novel measurement of sleep in rheumatoid arthritis: study proposal for approach and considerations. Digit Biomark. 2021;5(3):191–205. 10.1159/000518024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Kruizinga MD, Stuurman FE, Exadaktylos V, Doll RJ, Stephenson DT, Groeneveld GJ, et al. Development of novel, value-based, digital endpoints for clinical trials: a structured approach toward fit-for-purpose validation. Pharmacol Rev. 2020;72(4):899–909. 10.1124/pr.120.000028. [DOI] [PubMed] [Google Scholar]
- 32. United States Food and Drug Administration . Digital health technologies for remote data acquisition in clinical investigations: guidance for industry, investigators, and other stakeholders (DRAFT GUIDANCE). FDA; 2021. [Google Scholar]
- 33. European Medicines Agency . Qualification of novel methodologies for drug development: guidance to applicants. EMA; 2020. https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/qualification-novel-methodologies-drug-development-guidance-applicants_en.pdf. [Google Scholar]
- 34. Clinical Trials Transformation Initiative . Flowchart of steps for novel endpoint development. CTTI. https://ctti-clinicaltrials.org/wp-content/uploads/2021/06/CTTI_Novel_Endpoints_Flowchart.pdf. [Google Scholar]
- 35. Clinical Trials Transformation Initiative . Steps for novel endpoint development, with suggested approaches and considerations. https://ctti-clinicaltrials.org/wp-content/uploads/2021/06/CTTI_Novel_Endpoints_Detailed_Steps.pdf. [Google Scholar]
- 36. TransCelerate Biopharma . Patient technology solutions. TransCelerate. https://www.transceleratebiopharmainc.com/assets/patient-technology-implementation-framework/patient-technology-assets/ (accessed January 10, 2023). [Google Scholar]
- 37. Clinical Trials Transformation Initiative . Developing novel endpoints generated by digital health technology for use in clinical trials. 2022. https://ctti-clinicaltrials.org/wp-content/uploads/2022/03/CTTI-Digital-Health-Trials-Novel-Endpoint-Acceptance-Recommendations.pdf. [Google Scholar]
- 38. Clinical Trials Transformation Initiative . Preparing a digitally-derived endpoint for key endpoint use. 2022. https://ctti-clinicaltrials.org/wp-content/uploads/2022/03/CTTI-Digital-Health-Trials-Novel-Endpoints-Evidentiary-Considerations-Process-Map.pdf. [Google Scholar]
- 39. Clinical Trials Transformation Initiative . Regulatory engagement opportunities when developing digitally derived endpoints. 2022. https://ctti-clinicaltrials.org/wp-content/uploads/2021/06/CTTI_Novel_Endpoints_FDA_Quick_Reference_Sheet.pdf. [Google Scholar]
- 40. Clinical Trials Transformation Initiative . Question Bank for identifying meaningful outcome measures; 2022. https://ctti-clinicaltrials.org/wp-content/uploads/2022/03/CTTI-Novel-Endpoint-Acceptance-Question-Bank-Outcome-Measures-FINAL.pdf. [Google Scholar]
- 41. GlobalData . Clinical. Trials – the movement towards decentralized clinical trials. 2022. https://www.globaldata.com/store/report/decentralized-clinical-trials-insight-analysis/#:∼:text=What%20was%20the%20size%20of,1%2C291%20(trials)%20in%202021. [Google Scholar]
- 42. United States Food and Drug Administration . Conduct of clinical trials of medical products during the COVID-19 public health emergency: guidance for industry, investigators, and institutional review boards. FDA; 2021. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/fda-guidance-conduct-clinical-trials-medical-products-during-covid-19-public-health-emergency (accessed December 21, 2022). [Google Scholar]
- 43. Center for Drug Evaluation and Research . DDT COA #000102. Physical activity accelerometry assessment for analgesic clinical trials (PAACT). FDA; Published online 2021. https://www.fda.gov/drugs/clinical-outcome-assessment-coa-qualification-program/ddt-coa-000102-physical-activity-accelerometry-assessment-analgesic-clinical-trials-paact (accessed December 21, 2022). [Google Scholar]
- 44. Center for Drug Evaluation and Research . DDT COA #000114. Chronic heart failure- activity monitor-based endpoint measure. FDA; Published online December 15, 2021. https://www.fda.gov/drugs/clinical-outcome-assessment-coa-qualification-program/ddt-coa-000114-chronic-heart-failure-activity-monitor-based-endpoint-measure (accessed December 21, 2022). [Google Scholar]
- 45. European Medicines Agency .Letter of support for mobilise-D digital mobility outcomes as monitoring biomarkers. 2021. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/fda-guidance-conduct-clinical-trials-medical-products-during-covid-19-public-health-emergency (accessed December 21, 2022). [Google Scholar]
- 46. European Medicines Agency . Questions and answers: qualification of digital technology-based methodologies to support approval of medicinal products. 2020. https://www.ema.europa.eu/en/documents/other/questions-answers-qualification-digital-technology-based-methodologies-support-approval-medicinal_en.pdf (accessed January 3, 2023). [Google Scholar]
- 47. Center for Drug Evaluation and Research . FDA patient-focused drug development guidance series for enhancing the incorporation of the patient’s voice in medical product development and regulatory decision making. FDA. Published online June 28, 2022. https://www.fda.gov/drugs/development-approval-process-drugs/fda-patient-focused-drug-development-guidance-series-enhancing-incorporation-patients-voice-medical (accessed September 16, 2022). [Google Scholar]
- 48. European Medicines Agency . Innovation in medicines. 2021. https://www.ema.europa.eu/en/human-regulatory/research-development/innovation-medicines (accessed January 6, 2023).
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The data that support the findings of this study are not publicly available because interview transcripts contain information that could compromise the privacy of research participants. Further inquiries about the data can be directed to the corresponding author.