Clinical Evaluation of Digital Therapeutics: Present and Future

Abstract

Objectives

Digital therapeutics (DTx) are software-based therapeutic interventions based on clinical evidence. Randomized clinical trials (RCTs) are often the source of clinical evidence, similar to conventional drugs or medical devices. However, novel approaches such as the use of real-world data or digital biomarkers are also utilized. This article aimed to review how DTx products have been clinically evaluated.

Methods

DTx products approved by the US Food and Drug Administration as of 2020 were reviewed and products with sufficient published information were selected. Pivotal clinical trials were analyzed according to the elements of the Consolidated Standards of Reporting Trials (CONSORT) guideline. Case reviews were presented for other clinical evaluation strategies, considering the small number of publications.

Results

Most approved DTx products used RCTs for clinical evaluations. Similar to conventional RCTs, parallel-group designs with statistical hypothesis testing were adopted. However, DTx trials were often not blinded due to practical issues and involved various comparator groups. In addition, DTx products could be readily evaluated in home-based settings and delivered through the internet. Other evaluation approaches included retrospective analyses using insurance claims data or usage data, which enabled long-term evaluations of effectiveness. Digital biomarkers obtained from real-time and continuous log data were also used to improve the objectiveness of endpoints.

Conclusions

RCTs accounted for the majority of DTx evaluations. The designs of DTx trials were comparable to those of drug or device trials, but blinding and comparator elements were often different. Furthermore, the use of real-world data and digital biomarkers are also being tried.

I. Introduction

The Digital Therapeutic Alliance defines digital therapeutics (DTx) as “evidence-based therapeutic interventions driven by high-quality software programs to prevent, manage, or treat a medical disorder or diseases” [1,2]. We summarized the three major characteristics of DTx based on its definition [2]. First, DTx are based on “software.” They can be considered as examples of software as a medical device (SaMD), which is a classification used by the US Food and Drug Administration (FDA) [3]. This means that the “software itself,” rather than the hardware upon which it is deployed, is classified as a medical device. Thus, DTx can be freely implemented on a regular smartphone or a tablet instead of being specifically installed on approved medical devices.

Second, DTx are “therapeutic” interventions. DTx are similar to drugs and traditional medical devices in that they should have “therapeutic” effects. This is an important point that distinguishes DTx from general health care applications.

Third, DTx should be “evidence-based.” The characteristic of being “evidence-based” means that appropriate medical evidence is required based on the risk level of DTx. Therefore, it is often required that clinical trial results should be published in peer-reviewed journals and/or reviewed by regulatory agencies, and that real-world evidence and device performance data should be obtained and analyzed [4].

The Digital Therapeutic Alliance classifies DTx into the following three categories according to the purpose: treat a disease, manage a disease, improve a health function [5]. It is recommended to conduct appropriate validation processes for each of these categories. Early DTx were focused on preventing/monitoring medical diseases or disorders or optimizing medication [4]. However, in recent years, increasingly many DTx applications have also been approved as independent therapeutics, mainly in the field of psychiatry [4].

The clinical evaluation of DTx could be compared to the evaluation of efficacy and safety for drug products, as DTx and drugs share similar properties. In general, the effectiveness of drugs has been confirmed through randomized, double-blind, placebo- or active-controlled clinical trials [6]. The clinical phase should be preceded by preclinical toxicological evaluations. DTx are similarly evaluated through clinical trials. However, due to the nature of DTx as software, DTx are exempt from preclinical evaluations that are mandatory for drugs [6]. In addition, blinding and assigning comparators, which are key elements in clinical trials of drugs, are often difficult to conduct due to the inherent properties of DTx as medical devices [6]. Hence, the clinical evaluation of DTx products requires an integrated approach that reflects the characteristics of DTx as both therapeutics and medical devices [6].

In this sense, the evaluation process for DTx could refer to the clinical evaluation processes of drugs; however, key features of medical devices should also be considered. Faris and Shuren [7] listed the following characteristics that distinguish clinical trials of medical devices from those of drugs, which can be equally applied to DTx: (1) device trials tend to enroll fewer participants than drug trials, (2) many device trials assess iterative improvements of previous-generation devices, (3) the device design or procedure may be modified during the trial, (4) device trials are less likely to be blinded or randomized than drug trials, (5) adaptive designs are increasingly common, and (6) existing data can partially or fully substitute for prospective trial data.

However, the use of existing or retrospective data to substitute for prospective trial data suggests that real-world data could be utilized in the clinical evaluation process. Although the definition of real-world data varies among authors, this term usually refers to data collected from real clinical settings rather than randomized clinical trials (RCTs) [8]. The scope of real-world data incorporates claims data, patient registry, post-marketing surveillance data, and pragmatic clinical trials [8].

From this standpoint, the evaluation of DTx through real-world data can be considered as an optional approach, supplementing RCTs [7]. In particular, DTx can be validated with a historical control group based on existing registry data, which is a strategy that has been used in clinical trials of medical devices [7,9]. In addition, since DTx are based on software, real-time log data are collected. Therefore, clinical evaluation methods using machine learning or artificial intelligence are being actively discussed [10].

In this review, we discuss the evaluation strategies of DTx from two different perspectives: clinical trials and real-world data. New approaches such as digital biomarkers are also discussed. Based on these perspectives, requirements for the global approval of DTx are suggested.

II. Clinical Evaluation of DTx through Clinical Trials

1. Similarities to Clinical Trials of Drug Products

Confirmatory RCTs are the gold standard for the clinical evaluation of drugs. RCTs usually involve statistical tests to prove superiority, equivalence, or non-inferiority by comparing a treatment group and a placebo or active control group under a controlled environment. The essentials of RCTs are described in the Consolidated Standards of Reporting Trials (CONSORT) guideline [11], by which the quality of the results is evaluated. Table 1 shows the key elements of RCTs included in the CONSORT guideline [12].

Table 1.

Items related to the methodology of clinical evaluation used in the CONSORT statement

Section	Item#	Checklist item
Trial design	3a	Description of trial design (such as parallel, cross-over, split-mouth) including allocation ratio.
	3b	Important changes to methods after trial commencement (such as eligibility criteria), with reasons.
Participants	4a	Eligibility criteria for participants.
	4b	Settings and locations where the data were collected.
Interventions	5	The interventions for each group with sufficient details to allow replication, including how and when they were actually administered.
Outcomes	6a	Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed.
	6b	Any changes to trial outcomes after the trial commenced, with reasons.
Sample size	7a	How the sample size was determined.
	7b	When applicable, explanation of any interim analyses and stopping guidelines.
Randomization
Sequence generation	8a	Method used to generate the random allocation sequence.
	8b	Type of randomization; details of any restriction (such as blocking and block size).
Allocation concealment mechanism	9	Mechanism used to implement the random allocation sequence (such as sequentially numbered opaque envelopes), describing any steps taken to conceal the sequence until interventions were assigned.
Implementation	10	Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions.
Blinding	11a	If done, who was blinded after assignment to interventions (such as participants, treatment providers, those assessing outcomes) and how.
	11b	If relevant, description of the similarity of interventions.
Statistical methods	12a	Statistical methods used to compare groups for primary and secondary outcomes.
	12b	Methods for additional analyses, such as subgroup analyses and adjusted analyses.

Adapted from Moher et al. [11].

Currently FDA-approved DTx products have been clinically evaluated through RCTs, similarly to the process for drug products. The clinical trial designs for the selected FDA-approved DTx products on the market that have sufficient clinical trial information as of 2020 [1,2] are summarized in Table 2 [13–22]. All the clinical trials analyzed herein adopted a randomized, parallel group design. In addition, clinical endpoints validated in conventional drug clinical trials were evaluated using predefined statistical tests. These aspects suggest that the methodology of conventional RCTs can be consistently applied for the evaluation of DTx. The methodology was implemented irrespective of whether DTx were developed as independent therapeutics or adjunctive therapies for other approved drugs [6].

Table 2.

Summary of the clinical evaluation of digital therapeutics through clinical trials

Digital therapeutics	Manufacturer	Patients	Trial design	Randomization	Blinding	Interventions	Total number of participants	Primary endpoint
reSET [13,14]	Pear Therapeutics	Patients with drug use disorder	Parallel design	Stratified randomization (1:1)	Open	Twice per week for 12 weeks Conventional treatment + reSET Conventional treatment group	507	Reduction rates in drug use (via both self-reporting and urine drug test) and treatment retention rates
reSET-O [15]	Pear Therapeutics	Patients with opium use disorder	Parallel design	Stratified randomization	Open	12 weeks Local community drug abuse treatment + reSET-O Local community drug abuse treatment + therapist consultation	206	Maximum abstinence period of opium and cocaine intake
Somryst [16]	Pear Therapeutics	Patients with chronic insomnia (excluding major depression symptoms)	Parallel design	Stratified randomization (1:1)	Sham control group	6 weeks SHUTi: Insomnia treatment program HealthWatch: sham control group	1,149	Depression symptoms lasting for 6 months (PHQ-9)
PEAR-004 [17]	Pear Therapeutics	Schizophrenia patients	Parallel design	Random allocation (1:1)	Single blinding of evaluator, sham control group	16 weeks Medication + PEAR-004 Medication + sham control group	112	1. Change in Positive and Negative Syndrome Scale (PANSS) with reference to the base scale2. Dropout rate
Bluestar [18]	WellDoc	Patients with diabetes	Parallel design	Cluster random allocation	Single (researcher)	1 year Typical therapy (control group) Feedback via smartphone Feedback via smartphone + primary care physician checking raw data Feedback via smartphone + primary care physician checking the analyzed data	213	HbA1c level after 1 year
Endeavor [19]	Akili	Pediatric attention deficit hyperactivity disorder (ADHD) patients	Parallel design	Stratified randomization (1:1)	Quadruple blinding (patients, physicians, researchers, evaluators) sham control group	5 days per week for 4 weeks AKL-T01: DTx AKL-T09: Similar to AKL-T01 but not ADHD related program	348	Change in Test of Variables of Attention-Attention Performance Index (TOVA API) with reference to the base index
NightWare [20]	NightWare	Patients with post-traumatic stress disorder (PTSD)	Parallel design	Random allocation (1:1)	Quadruple blinding (patients, physicians, researchers, evaluators), sham control group	60 days NightWare Therapeutic System: wearable devices and vibration stimulation provided Sham NightWare: wearable devices without vibration stimulation provided	270	Change in Pittsburgh Sleep Quality Index (PSQI) with reference to the base index
Propeller [21]	Propeller Health	Patients with asthma or chronic obstructive pulmonary disease	Parallel design	Stratified randomization (1:1)	Open	1 year Propeller Health System: inhaler DTx linked to smartphone Control: Sensor only with no feedback	495	Change in short-acting beta 2 agonist use
ProAir Digihaler [22]	Teva	Patients with asthma or chronic obstructive pulmonary disease	Parallel design	Random allocation (1:1)	Open	12 weeks ProAir Digihaler: inhaler linked to a digital system Control: general inhaler	333	Change of Asthma Control Test (ACT) scores with reference to the base score

2. Distinctive Characteristics of DTx Clinical Trials

DTx clinical trials had the following distinctive characteristics from conventional RCTs. The characteristics of DTx trials are summarized in Table 3 [13,15,16,18–21,23].

Table 3.

Characteristics of clinical trials of digital therapeutics

Components	Characteristics	Cases
Trial design	Most trials are conducted with parallel designs similar to conventional RCTs, but a crossover design may also be attempted depending on the indication.	- RCT divided into general therapy + DTx and general therapy [15] - A 2×2 crossover design of voice-bot DTx and a control group in patients with ADHD [23]
Randomization	Stratified randomization is most often performed, but in the case of facility-based processing, cluster randomization is also performed by facility.	- Randomization of treatment groups stratified by major variables such as treatment facility, main abuse component, and drug ban period [13] - A clinical study conducted with random allocation of cluster groups by primary care facilities in the state of Maryland [18]
Blinding	In case of evaluating in addition to the existing therapy, the evaluation is conducted in an open trial design, but a sham control group may be created, and blinding may be applied.	- A study comparing typical therapy + DTx and typical therapy groups in an open way [15] - A study using quadruple blinding by creating a sham control group with a placebo app excluding the function for ADHD treatment [19]
Participants	If the purpose is to prevent disease, manage disease, or optimize medication, many participants are patients with chronic diseases. If the DTx are used for therapeutic purposes, participants are often patients with psychiatric diseases. For relatively mild diseases, there have also been attempts to recruit participants from web communities.	- A study providing feedback on the use of an asthma medication inhaler linked to a smartphone [21] - A study recruiting and screening patients with insomnia, excluding depression, through Facebook [16]
Control Group	Control groups vary depending on the device associated with the DTx, and in many cases, a sham control group is created by removing several functions. When the functions are complex, a control group may be created by subdividing the functions.	- A program without insomnia treatment function [16] - Wearable devices without vibration function [20] - A digital inhaler application without feedback [21] - A study comparing diabetes management effects by subdividing control groups according to the use of simple feedback, raw log data, and analyzed data [19]
Number of participants	The number of participants vary from small pilot studies to large-scale research with over 1,000 participants.	- A study in which 1,149 patients with insomnia were registered through patient recruitment, promotion, and screening on a web portal [16]
Adverse events	Few adverse events related to DTx were reported.	- No adverse events related to DTx were reported among 1,149 patients who participated in a clinical trial for insomnia [16]

DTx: digital therapeutics, RCT: randomized controlled trials, ADHD: attention deficit hyperactivity disorder.

Blinding for DTx is often difficult. In conventional drug clinical trials, a placebo drug with an identical shape to that of the active treatment drug is often implemented. However, as DTx products are software with various forms, a “placebo” is often not feasible or possible. Instead, a “sham control,” which is occasionally used in evaluations of medical devices, is implemented. However, since it is often difficult to establish blinding of the sham control, clinical trials are instead conducted with an open-label design regarding the treatments. In particular, when the two treatment groups are conventional therapy versus conventional therapy plus DTx, blinding cannot be applied unless a separate sham control is prepared. This feature is shown in Table 2, where most trials had an open rather than a blinded design.

In addition, DTx have various forms of comparators. As mentioned above, DTx trials often involve a sham control. The sham control for DTx trials can be quite variable, unlike the placebos used in drug trials. Sham controls can be constructed in a way that excludes or transforms the main features of the intervention and leaves only the auxiliary function. For example, Pear Therapeutics’ Somryst used a separate software that does not include a therapeutic effect for insomnia, named HealthWatch, as a sham control. The software was designed to include only minor elements of the software being tested (e.g., the interactive interface) [16]. Similar cases can also be found for Akili’s Endeavor [19] or the wearable NightWare [20] with the vibration function removed.

When DTx are not independent therapeutics, each step of using DTx could be analyzed as a comparator. A representative case is WellDoc’s Bluestar, which is used for glycemic control in patients with diabetes [18]. In this study, three-subdivided comparators were implemented: feedback through a smartphone, having the attending physician check only the data log, and having the attending physician check the final analyzed data [18].

Finally, DTx are available in decentralized environments. DTx as independent therapeutics have primarily focused on psychiatric conditions [16]. These DTx products had various indications ranging from drug/opioid use disorders that require facility-level treatment (such as reSET and reSETO [12,14]) to insomnia, which can be treated at home [16]. The diversity of the clinical settings where DTx products are provided indicates that DTx products could be utilized in decentralized environments.

Another important aspect of DTx trials is digital delivery. Since DTx are software, they could be delivered through the internet. In addition, patient recruitment from internet communities or social media could be more readily performed. In a trial using Pear Therapeutics’ Somryst, participants were recruited through Facebook. Patient screening was also conducted through a web portal [16]. This concept is closely related to that of the decentralized clinical trial [24] and has the advantage of enabling recruitment of a large number of participants with little cost.

III. Evaluation Using Real-World Data

Clinical evaluations using claims data can be used to evaluate long-term clinical effects that are difficult to assess in clinical trials. Several cases that utilized retrospective data for the evaluation of DTx have been found. In the case of Pear Therapeutics’ reSET and reSET-O, cost-effectiveness was analyzed in 351 opioid use disorder patients based on post-market insurance claims data [25,26]. Similarly, retrospective cohort data were used to analyze the therapeutic effects [27] or conduct an economic analysis [28] of DTx for patients with type 2 diabetes and hypertension.

Usage data from DTx for managing chronic diseases or optimizing medication compliance are also utilized to evaluate long-term clinical effects. NaturalCycle has obtained market approval for the purpose of contraception (menstrual cycle management); its menstrual cycle predictions were validated through an analysis of 18,548 person-years of menstrual cycle data obtained from 22,785 women [29]. Perx Health’s mobile application to improve medication adherence also confirmed improvements in medication compliance based on data obtained from the application users [30]. As these types of DTx could readily collect large amounts of usage data, clinical evaluations using usage data are expected to be increasingly common.

IV. Concepts of Digital Biomarkers and Exposure-Response of DTx

The therapeutic effects of DTx have been evaluated through validated endpoints in conventional clinical trials. However, recent attempts to use digital endpoints are also gaining attention [6]. Clinical evaluations of therapeutics for central nervous system–related disorders often face difficulties due to subjective and highly variable self-reported endpoints [6]. These characteristics necessitate a larger number of participants, higher study costs, and longer study periods [6].

Digital endpoints can be a favorable alternative as they could be collected in real time with little cost. The development of digital endpoints is aligned with DTx, where considerable amounts of log data are obtained in real time. Accordingly, if clinical feasibility is confirmed, digital endpoints can make a major contribution to improving evaluation efficiency.

As the number and scope of DTx continue to expand, attempts to develop biomarkers specifically for DTx as clinical indicators are also gaining attention. An example is an attempt to distinguish between amyloid-positive and amyloid-negative patients by analyzing gait changes in the early phase of Alzheimer disease [6,31]. Since gait change data were difficult to obtain in real time in the past, analyses were limited. However, due to the spread of digital devices, it has become possible to collect larger amounts of data, accelerating the development and validation of gait changes as a digital biomarker.

Under these circumstances, the importance of artificial intelligence is being emphasized [10,32,33]. As the data obtained from DTx are continuous data obtained in real time through sensors (e.g., gait, inhaler usage, blood sugar changes, and sleep patterns), artificial intelligence can be used to recognize patterns from these data [32]. This should be coupled with real-time monitoring functions, which are automated, efficient, expandable, and easy to operate [32].

As the scope of DTx has recently expanded to virtual reality, more biomarkers have become available for DTx [34]. A variety of information, such as behavior and facial expressions, can be collected from patients wearing virtual reality equipment, and the data can be used to develop biomarkers for treatment or prognoses through artificial intelligence [34]. A previous study evaluated executive dysfunction by analyzing patterns of brain waves and eye movement data obtained from 360° virtual reality equipment through a machine learning algorithm [35]. Considering that virtual reality-based DTx are being actively applied for pain management [36], digital biomarkers could be potentially developed as objective evaluation tools for pain assessment.

DTx could also be interpreted from the perspective of an exposure-response relationship [4]. Exposure to a drug corresponds to the dose, administration interval, and concentration, and the corresponding concepts for DTx may be time, frequency, and duration of DTx use [4]. The response to DTx can be assessed using conventional biomarkers, but also can be evaluated using novel digital endpoints. Further studies on the exposure-response relationship of DTx are necessary since a standard analytical method has not been established [4].

V. Discussion

We found that most examples of the clinical evaluation of DTx used clinical trials, which are the standard for the evaluation of drugs and medical devices. However, novel approaches that use real-world data and digital biomarkers are also on the way. An analogy between the elements of DTx and components of chemical drugs suggests the future direction of DTx [37]. The “active ingredient” of DTx corresponds to the component that shows a therapeutic effect, and it is necessary to validate its efficacy using the aforementioned clinical evaluation methodologies. The “excipient” of DTx is the user interface that maximizes the efficacy of the active ingredient [37]. As DTx products require more active engagement from patients than drugs, the importance of the “excipient,” or the user interface, cannot be overlooked. In other words, the socio-cultural background of the patients who will use DTx should be taken into account to achieve the desired therapeutic effects. In order for DTx to receive global approval, global standards for the clinical evaluation of the effectiveness of the “active ingredients” should be established, while local considerations for “excipients” should simultaneously be taken into account.