Conducting Psychological Intervention Research in the Information Age: Reconsidering the “State of the Field”

Abstract

Enthusiasm for technology in mental health has evolved as a function of its promise to increase the reach and impact of services, particularly for traditionally at-risk and underserved groups. Preliminary findings suggest that technology-enhanced interventions indeed hold promise for increasing engagement in and outcomes of evidence-based treatment approaches. The time- and resourceintensive nature of traditional randomized control trials, however, may be even more of a challenge for further advancement in this area, given the rapid innovation of consumer driven new product development. Accordingly, this review aims to summarize how a broader range of scientific designs and analyses may be necessary in order to further advance and optimize the reach and impact of technology-enhanced psychological practice. Examples of various approaches are provided and recommendations are provided for future work in this area.

Enthusiasm for technology in evidence-based psychological practice has evolved as a function of its promise to increase the reach and impact of services beyond traditional community-based treatment settings, particularly for those groups who are considered at-risk yet underserved, such as those who are low income and/or geographically isolated (see Aguilera & Muench, 2012; Enock & McNally, 2013; Jones et al., 2013 for reviews). Findings to date suggest the potential for increased client engagement in and adherence to technology-enhanced treatments (e.g., Gros et al., 2013; Lindhiem, Bennett, Rosen, & Silk, 2015; Musiat & Tarrier, 2014). Further advancement of science and practice in this area, however, may depend on revisiting what has come to define state of the field or gold standard for treatment outcome research.

Specifically, the rapidly evolving nature of technology far outpaces the slow and meticulous pace (i.e., 5 to 7 years from funding to publication) of our traditional standard of intervention research, the randomized controlled trial (RCT) (Ioannidis, 1998; Riley, Glasgow, Etheredge, & Abernethy, 2013). For example, as cleverly conveyed by Riley and colleagues (2013), a study funded in 2006 to test the intervention potential of the Nintendo Wii game console would be published in 2012, a 6-year lag during which the functionality of the Wii would most likely be replaced by software applications that offer similar functionality using more portable and convenient devices such as the Apple iPhone and/or iPad, as well as Google’s Android. Building upon this example, this review considers how greater flexibility in our approach to research design and analyses may be necessary to further advance and optimize the actual reach and impact of technology-enhanced intervention science (Riley et al., 2011; Rothwell, 2005; Wu et al., 2006).

Importantly, this review is not meant to be exhaustive. Our goal, instead, was to integrate, summarize, and extend various literatures on design and design considerations relevant to technology-enhanced intervention science in particular. To this end, we highlight the use of both within- and between-subjects designs in technology-enhanced intervention research, as well as more recent adaptations and approaches. As such, we drew our examples from research on a range of populations (e.g., adults, children), disorders (e.g., anxiety, behavior disorders, depression), and evidence-based treatment approaches (e.g., behavioral activation, behavioral parent training) with the goal of highlighting the broad relevance and generalizability of the points that are raised for consideration.

Within-Subjects Designs

A range of within-subject designs that served as a hallmark of early behavioral theory and research are making a relative come-back in psychological intervention research as well (Kazdin, 2010; Nock, Michel, & Photos, 2007), including research on technology-enhanced interventions. In general, the hallmarks of within-subject designs are two-fold: A) Repeated measures of the behavior in one or a relatively small number of participants and/or B). Every participant is exposed to all values or levels of the independent variable (see Shadish, Cook, & Campbell, 2001 for a review). For a number of reasons, within-subject designs are particularly relevant for the study of technology-enhanced interventions (e.g., Clough & Casey, 2015a; Dallery et al., 2013; Riley et al., 2013).

First, technology often makes it easier for researchers to maintain frequent contact with participants and collect large quantities of data with minimal effort for both the participants and researchers (Dallery et al., 2013; Riley et al., 2013). Additionally, isolating the aspects of the intervention that are driving clinical change is of particular interest and can inform the iterative development process of technology-enhanced interventions. Therefore, being able to systematically introduce and remove aspects of the technology intervention and measure the subsequent behavioral changes not only has the potential for researchers to maximize the efficacy of treatment, but also begins to elucidate the underlying processes of change. Third, and perhaps, most importantly, these methods decrease the time and, likely also, cost of evaluating the intervention (Clough & Casey, 2015a; Riley et al., 2013). Given the rapid evolution of technology, the need for on-going updating and refinement, and the cost of developing technology, it may be beneficial to test technology enhancements with a smaller number of participants to allow for faster and less costly evaluation of these interventions. In fact, a number of researchers have begun to use these methods to establish initial efficacy and help refine existing technologies. Two such examples will be presented here.

In the first example, Anton and colleagues (2016) used what can be considered an exploratory or pilot case series analysis to examine patterns within and across low-income parents of clinically-referred young children with behavior disorders to determine whether use and/or frequency of use of technology, as well as specific components of technology-enhanced treatment, were associated with variability in treatment outcomes. Children from low-income families are more likely to develop early onset disruptive behavior disorders compared to their higher income counterparts. Low-income families of children with early onset behavior disorders, however, are less likely to engage in the standard of care treatment, Behavioral Parent Training (BPT), than families from other sociodemographic groups. Prior research demonstrated that relative to families randomized to one standard BPT program, Helping the Noncompliant Child (HNC; McMahon & Forehand, 2003), families randomized to Technology-Enhanced HNC (TE-HNC) were more likely to come each week for session, to complete their home practice, and to participate in their mid-week phone check-in (Jones et al., 2014).

In order to better understand trends within the TE-HNC group, Anton and colleagues (2016) examined correspondence between variability in caregiver use of and attitudes toward technology-enhanced services with treatment outcome. It is perhaps interesting to note that there was relatively little variability in technology use between families (i.e., average use of specific components of the technology ranged from 52 to 92% across all families), and even less variability in treatment outcomes across cases (i.e., majority of families were full treatment responders). That said, patterns of findings suggested that those families who used the technology more and more consistently evidenced better outcomes pre-to-post-treatment than families who used the technology relatively less or less consistently.

In the second example, Clough and Casey (2015b) used a multiple baseline single-case design to re-evaluate an adjunctive smartphone application, PsychAssist, for the treatment of adult anxiety. After the initial design and evaluation of PsychAssist, which includes CBT informed homework exercises, handouts, and activities to enhance traditional face-to-face treatment for anxiety, the smartphone application was adapted based on end-user feedback and, therefore, needed to be re-evaluated. Due to the time and cost-intensive nature of traditional RCTs, the researchers sought out a single-case design to allow them to efficiently and rigorously assess the efficacy of the newer version of the application. Four adults with anxiety disorders were recruited to participate in a multiple baseline study. Each participant was randomized to engage in a baseline observation period that varied in length (i.e., 3-6 weeks), in order to increase internal validity. After the baseline period, all participants engaged in the 9-week therapy program. For the duration of the study, participants completed self-report measures of their anxiety symptoms weekly and change in anxiety symptomatology was assessed. Results suggested that the introduction of the PsychAssist technology was associated with a decrease in anxiety symptomatology. This study demonstrates the relative flexibility of these approaches, and the use of single-case design to aid in the ongoing adaptation and evaluation of technology-enhanced interventions.

Case studies are a preliminary step, however, and not without limitation, primarily being the lack of generalizability. That is, a case study or series design allows investigators to describe and report trends and patterns within an individual or small group of clients, which can be a critical starting point for identifying research questions, generating hypotheses, and the development of study design and methods. Yet, such steps must be taken with caution given that case study findings may or may not be representative or typical of the broader population the treatment aims to serve. Investigators interested in the feasibility and impact of technology-enhanced mental health services may, in turn, use trends obtained in a case study or series to subsequent between-group designs; however, we argue that depending on the study question and hypotheses that within-group designs may provide adequate data for subsequent decision making in their own right as well.

Between-Subjects Designs

Transitioning from within-subject designs, we turn to a range of between-subjects designs that have been used in research on technology-enhanced interventions. These include designs that aim to test the efficacy or effectiveness of a technology-enhanced interventions, as well as adaptations and extensions of these designs that are more recently being considered.

Efficacy Trials.

It has been nearly two decades since the American Psychological Association (1995) highlighted the need for empirically-support treatment manuals. Embedded within such a recommendation was a focus on the “randomized control trials and their logical equivalents (efficacy research) as the standard for drawing causal inference about the effects of an intervention” (APA, 1995, p. 274). Indeed, the RCT remains the gold standard of evidence-based practice in psychology, given that attributes such as random assignment of participants to the experimental or control group increase confidence that between-group group differences are a result of the intervention rather than chance (e.g., Bauer, Okon, Meermann, & Kordy, 2012; Enock & McNally, 2014; Gustafson et al., 2014). There are also well-documented scientific and practical challenges of the RCT that seem to be magnified in the study of technology-enhanced interventions in particular (see Deaton & Cartright, 2017; Spieth et al., 2016 for reviews).

For example, research using RCTs in general rarely (if ever) can actually randomly sample individuals from the population in addition to then randomly assigning individuals to treatment group (see Shean, 2014 for a review). Instead, by using fairly standard research and recruitment practices, which aim to increase the scientific rigor, as well as feasibility, of the study (e.g., eligibility criteria), researchers instead rely on non-probability sampling in order to select those participants who are the greatest “fit” for the study questions, hypotheses, and design. The standard practice of limiting RCTs to individuals with a single diagnosis, rather than the comorbidity that is more typical of the population or no diagnosis at all but still experiencing clinically significant distress, in turn, substantially limits the generalizability of our study designs and results (Shean, 2014). Further limiting the clinical utility of the RCT is the protracted timeline relative to other research designs. For example, the completion of an RCTs can take up to 5 to 7 years to move from initial funding of the project to the publication of the data. Such a protracted timeline is becoming increasingly unwieldy in a wide range of fields as the types of questions we are asking become more specific and the cost of failing to reject the null hypothesis even more costly. Given that the rapid evolution of technology is far outpacing our rigorous standard of science, technology-enhanced intervention research is a prime example of the proliferation of challenges inherent in continuing to think about the RCT as the gold standard of evidence-based psychology science (see Jones, 2014; Kumar et al., 2013; Mohr et al., 2015 for reviews). Related to the timeline of RCTs, replication or lack of replication is a final example of a challenge inherent in our gold standard of evidence-based psychological science. While replication is one of the criteria necessary for a psychological intervention achieving the highest designation of APA’s “well-established”, studies that solely aim to replicate another investigative team’s results are not necessarily rewarded in academic research settings that favor independence and innovation of ideas. Replication of RCTS, in turn, is likely even less incentivized when doing so takes yet another 5 to 7 years for study completion and publication of results. On top of timeline, however, replication of RCTs on technology-enhanced interventions bare the additional burden of technology expense and the probability that the replication would be occurring during or shortly after advances in technology making the findings of the research even more obsolete.

With the aim of responding to such challenges while also attempting to maintain a premium on internal validity, various adaptations of the RCT have been proposed to test efficacy, including non-inferior or equivalence designs (Gros et al., 2013). Whereas pure RCTs generally seek to compare two treatments and determine whether one is significantly different or better than the original treatment, equivalence designs compare two interventions to determine whether one is generally equivalent in clinical value to the more established intervention (Wellek, 2003). Similarly, non-inferior designs are used to determine if a new (or enhanced) intervention or delivery modality does not significantly differ from the standard of intervention in community-based settings or treatment as usual (Greene, Morland, Durkalski & Frueh, 2008; Wiens, 2001). In the case the technology-enhanced interventions as an example, the goal is not whether the technology necessarily improves outcomes, but rather does it offer relatively the same clinical value or no less than the previously established margin of inferiority with the added benefit of cost-effectiveness, accessibility, and/or efficiency.

This point is highlighted by Aciemo and colleagues (2016) who used a non-inferiority design to compare in-person versus telehealth delivery of an exposure-based treatment, Behavioral Activation and Therapeutic Exposure (BA-TE), involving imaginal exposure, situational exposure and behavioral activation in a sample of veteran participants suffering from PTSD or major depressive disorder (MDD) (N = 201). This study sought to establish a solution to current barriers to care associated with traditional in-person modes of psychological treatment such as stigma, travel time and cost. Masters level therapists provided eight 1.5 hour sessions of BA-TE either in person (IP) at a clinic or using home-based telehealth (HBT) with the participant at their home. The majority of the HBT group participants were able to use their own personal electronics to install and use an encrypted internet-based video service comparable to “Skype” or “Facetime.” Symptom improvement for both MDD and PTSD was comparable at post treatment and at 3- and 6-month follow up assessments between the traditional in-person BA-TE group and the experimental HBT BA-TE group. These findings provide evidence that PTSD and MDD treatment can be effectively and safely administered using home-based telehealth. This provides a possible solution to barriers to treatment associated with some clinical populations without sacrificing clinical value.

Another strategy for investigating the efficacy of an adjustment or enhancement to an established evidence-based psychological intervention is to utilize benchmarks derived from extant RCTs in the literature (Wade et al. 1998; Weersing & Hamilton, 2005). Essentially, samples in prior studies function as the control group and, in turn, allow nvestigators to make inferences about the generalizability of the standard of care treatment (see Weersing & Wiesz, 2002 for an example). Such outcomes of comparison are not limited to reduction in symptoms, but could also be applied to a number of research questions associated with treatment-outcomes such as treatment engagement, completion, or long-term recovery rates, to name a few (see Hunsley & Lee, 2007 for a review).

Building upon this approach, some have argued that our consideration of successful outcomes with technology-enhanced treatments should be broadened to include higher levels of engagement and, in turn, a greater potential for benefit compared to no treatment at all (Danaher & Seely, 2009). The rationale is that the vast majority of those who could benefit from mental health care fail to seek out and/or have access to mental health services (Roll, Kennedy, Tran, & Howell, 2013). A myriad of challenges have been cited including stigma, lack of access, and barriers to treatment when services are available (Comer, 2015; Kazdin & Blasé, 2011; Khanna, Kerns, & Carper, 2014). Some have proposed that a more appropriate research question is whether technology enhances the likelihood that those who may not otherwise engage in services are able to do so effectively and/or whether outcomes are better than no treatment at all (Danaher & Seely, 2009).

Effectiveness trials.

As with the broader field of intervention research, the RCTs described above tell us about efficacy (i.e., does the intervention produce the desired treatment outcome in the context of a rigidly controlled design focused in internal validity), but not effectiveness (i.e., does the treatment yield the same outcomes in real-world community-based settings in which threats to internal validity are high and the focus must be on external validity). While efficacy studies bring us closer to understanding if and how we can capitalize on innovations in technology to improve health outcomes, there seems to be a lag in moving beyond efficacy research into the study of effectiveness. This remains the case despite the fact that the foundational work identifying the challenges of implementing such treatments into real world settings has been established (e.g. Glasgow, Phillips, & Sanchez, 2014; Ramsey, Lord, Torrey, Marsch, & Lardiere, 2016; Schwamm, 2014), as well as a growing body of research examining practicing clinicians’ interest and hesitations towards integrating technology-based approaches into their work (Kuhn et al., 2014; Schueller, Washburn, & Price, 2016; see Anton & Jones, 2017 for a review).

Furthermore, theoretical approaches to the implementation of technology-enhanced or technology-based interventions have been proposed (Quanbeck et al., 2014). For example, Quanbeck and colleagues (2014) proposed a model of evaluating implementaion efforts using RE-AIM, a framework for evaluating the implementation of public health interventions by assessing: reach, efficacy, adoption, implementation, and maintenance (Glasgow, Vogt, & Boles, 1999). As such, much of the foundational work to prepare for implementing technology-enhanced or technology-based interventions has been accomplished; however, few studies have actually moved the field toward the latter phases of adoption, implementation, and maintenance. We present two notable exceptions here.

Glasgow and colleagues (2014), the creators of the RE-AIM framework, used this model to both build and implement a technology-based weight loss and hypertension intervention deployed in urban community health centers aimed at reducing health disparities. The authors utilized a “randomized, pragmatic design” with input from stakeholders at every point in design in an attempt to streamline the implementation process into the creation of the program itself. The program tested, Be Fit Be Well, was delivered via the internet or an interactive voice response system over the phone, print materials, interventionist coaching calls, and monthly group sessions. It should be noted patients were allowed to choose whether they would prefer to access the prograrm via the internet or phone in order to maximize reach to those without internet access and increase generalizabilty to real world situations where clients will only engage with technologies they feel comfortable using. The intervention lasted 24 months and eligible patients in three community health centers were randomized to care as usual or the intervention arm. All participants were compensated for their assessments. The study used the REAIM framework to evalute the program’s effectiveness and efficacy. Reach examined enrollment rates of eligible participants and examined what types of patient characteristics predicted participation. Effectiveness measured traditional effectiveness outcomes, in this case, group differences in weight loss, blood pressure, medication adherence, and health related quality of life. Adoption examined the rates of adoption by primary clinics invited to participate as well as rates of participation of clinicians within these clinics. Implementation studied costs of the program, delivery rates by interventionsts, and participation rates by each aspect of the intervention among participants. Furthermore, each of these areas was evaluated for variation by participant, clinician, and clinic differences. Lastly, maintenance was investigated on both an individual and setting level. Overall, this study is a strong example of an effectiveness-implementation hybrid design (Bernet, Widens, & Bauer, 2013) aimed at producing a teachnology-based intervention that can more quickly and efficiently produce effective results in a real world setting.

The second example we will present piloted an interactive phone application in 22 patients with borderline personality disorder and a substance use disorder in traditional outpatient dialectical behavioral therapy (DBT) settings (Rizvi, Dimeff, Skutch, Carroll, & Linehan, 2011). In this example, researchers approached standard outpatient DBT clinics about the study, explained it further to those clinics intersted in participating, and had therapists provide flyers to all eligible clients. Interested clients then contacted the research team to set up screening at their respective outpatient clinic. Participants were given phones for use in the study and were compensated for their participation. The study assessed clients use of the mobile application based on rates of completion of daily assessments and frequency of use of the applications coaching features. Client satisfaction with and perceived usability of the application was assessed with a questionnaire at post-assessment as well as client ratings of the helpfulness of the application’s coaching after each coaching request via a less than one-minute questionnaire embedded within the application. The efficacy of the application was assessed by examining treatment effects on self and therapist-ratings of skill use, as well as self reports of depression, number of general symptoms, and urges to use substances. As such, this study exemplifies how prelimary pilot studies of new technology-enhanced treatments should and can incorporate measures of both efficacy and effectiveness.

Hybrid effectiveness-implementation trials.

Building upon the distinction between efficacy and effectiveness trials reviewed above, Curran and colleagues (2012) proposed the hybrid design paradigm or blended designs that include components, in varying degrees, of: effectiveness (i.e., focus on external validity, heterogeneous samples, “real-world” settings, range of clinical and related outcomes); dissemination (i.e., efforts and strategies to communicate information regarding evidence-based treatment with the goal of clinician engagement and use); and implementation (i.e., explicit efforts to promote routine and sustained uptake and adoption of treatments) science. Importantly, hybrid designs were not developed to address technology-enhanced psychological intervention research in particular; however, we believe that the nuances of this design approach may be particularly fitting nonetheless.

Complementary to a hybrid model approach, Anton and Jones (2017) proposed a five-stage model to address organizational readiness for and clinician acceptance of technology-enhancements to evidence-based treatments, as well as the relevance of current adoption strategies for technology-enhanced services. Importantly, this framework was not intended to replace other models, but rather to enhance them by focusing on the specific nuances and factors inherent in the science and translation of technology-enhanced mental health services. Consistent with this approach, the true potential of technology-enhanced mental health treatments remains unknown until strategies to successfully integrate technology-enhanced services into practice are developed and tested in large-scale implementation studies.

Adaptive or dynamic intervention designs.

Adaptive or dynamic designs are not orthogonal to hybrid designs, but the literatures have indeed evolved relatively separately. Generally, adaptive designs are characterized by four key components: 1) Sequence of decisions (e.g., If a client is non-responsive to treatment A, then what is the next step?); 2) A set of pre-determined treatment options (e.g., medication, behavioral, and/or cognitive-behavioral); 3) A set of tailoring variables which are cue to determine change in some aspect of treatment (e.g., lack of response to treatment and/or unintended side-effects), and 4) A sequence of decision-rules (i.e., how, when, and why to adjust treatment intensity, type, etc.) (see Lei, Nahum-Shani, Lynch, Oslin, & Murphy, 2012 for a review).

One example of an adaptive design is the sequential multiple assignment randomized trial (SMART) (see Collins, Murphy, & Bierman, 2004; Lei et al., 2012 for reviews). Essentially, SMART designs functionally allow investigators to operationalize a series of data-driven decision-rules in order to optimally tailor how and when the intensity, type, and/or sequence of treatment components are adapted based a particular client’s presenting needs and changing patterns of symptoms. As such, SMART designs offer an ideal platform of sorts for testing iterative design and development options in technology-enhanced services research.

Although we were not able to find an example of a SMART design in technology-enhanced services research, Clough and Casey (2015) provide an informative example that essentially describes a design in which clients are initially randomized to 3 versus 6 sessions with a supportive smartphone application, which they call “PsychAssist.” Clients in each group are then defined as 3 and 6 session responders, who continue with the smartphone application alone, versus 3 and 6 session non-responders, who are then subsequently randomized to further support via email or telephone. Building upon this hypothetical example, SMART designs allow investigators to answer questions regarding the most robust technology-enhanced intervention, as well as how best to tailor the intervention depending on the individual client.

Microtrials.

One design that has been proposed elsewhere, although not yet used in technology-enhanced intervention research maps well onto this approach – microtrials (see Leijten et al., 2015 for a review). As defined elsewhere (Howe et al., 2010), microtrials test the effects of relatively brief, discrete experimental manipulations hypothesized to impact more proximal processes (i.e., mechanisms) rather than the more distal outcomes that are the traditional focus of intervention research. Using the rationale proposed by Leijten and colleagues in the area of parenting interventions then, microtrials would afford the opportunity to test not if a technology-enhanced intervention is efficacious in terms of the ultimate targeted outcome, but rather how they are working to effect more proximal endpoints or surrogates. As such, microtrials could be a starting point, providing clues for example about which types of technology and/or functionality of technology yields the most substantive impact on a desired process. For example, if microtrial testing of several technology-enhanced components of the intervention (e.g., text message reinforcers, skills modeling videos, surveys of symptoms) reveal the most improvement in a proximal target after the text messages in particular, then the investment in more sophisticated (and expensive) functionality (e.g., surveys, videos etc) may not be necessary.

Analytic Reconsiderations in Technology-Enhanced Psychological Science

Given the discussions regarding alternative design approaches in technology-enhanced psychological science, it follows that our traditional analytic approaches may need to be reconsidered as well. In part, such reconsiderations are necessary given that a move away from the gold standard RCT would also make it more difficult to determine if studies are sufficiently powered to detect difference(s) between groups due to intervention if that difference is indeed present. To this point, there has been much discussion of new or modified metrics of treatment outcome. For example, Lee and colleague’s (2014) recommendations regarding pilot trials more generally provide a reasonable set of guidelines for studies of technology-enhanced interventions as well, particularly in the early phases of development and testing. For example, Lee et al. recommend moving away from significance levels (i.e., p < .05) to the use of less conservative significance thresholds, such as 85% or 75% confidence intervals. Then, the confidence interval (CI) can be evaluated for minimally clinically important difference (e.g., mean treatment difference is above zero and the CI includes or is above the minimally clinically important difference).

There are other ways of examining clinically meaningful change or practical significance as well. For example, the reliable change index (RCI; Jacobson & Truax, 1991) indicates change attributable to treatment is most likely not due to chance (i.e., RCI 1.96; also see Abramowitz, 1998 for modifications for use with small samples). Normative comparisons (Kendall & Grove, 1988), which are often used in combination with the RCI, assess if scores at post-treatment are distinguishable from individuals in the normative range.

Finally, meaningful clinical change may also be assessed by examining treatment response in technology-enhanced psychological intervention research. For example, in the aforementioned study by Anton and colleagues (2016) treatment response was characterized by comparing the pre- and post-treatment Eyberg Child Behavior Inventory (ECBI; Eyberg & Pincus, 1999) change score to the aggregated pre-post ECBI change scores published in prior BPT research (see Self-Brown et al., 2012; Ware et al., 2008 for meta-analyses). Treatment response status was defined by full, partial, or minimal responders based on if they achieved clinically significant change and, if so, on which ECBI subscales.

Regardless of whether investigators are using measures of significance thresholds, clinical significance, and/or treatment response, we agree with Lee and colleagues’ (2014) recommendation that a surrogate or indirect measurement of the effect should also be considered in technology-enhanced psychological science. As they described, a reasonable surrogate endpoint is one that has an established prognostic value for the clinical outcome; however, because it is more proximal to the treatment it should require fewer participants, less time, and, in turn, fewer costs. Interestingly, this approach is also quite consistent with the National Institute of Mental Health’s experimental therapeutics approach that focuses on the target or underlying mechanism accounting for a change in symptomatology as a result of treatment. So, for example, if an investigator posits that technology-enhancements will improve engagement and, in turn, clinical outcomes in a psychological intervention, then a reasonable surrogate endpoint in a pilot trial would be measurement of treatment engagement (i.e., presuming there is data to suggest that higher levels of engagement indeed result in better treatment outcomes).

Summary & Conclusions

As highlighted elsewhere, technology is a “new frontier in mental health,” which by definition includes potential advances in the reach and impact of services, as well as challenges inherent in a new model of care and the science that informs that model (NIMH, 2017). Consistent with this notion, we propose substantively broadening our standards of design and analysis in the study of technology-enhanced psychological intervention in order to advance the mission of both science and practice and to do so much more efficiently and effectively. Inherent within this suggestion, however, is the acknowledgement that such a shift will impact broader standards in the study of evidence-based psychological science as well. For example, standards of peer review at both the funding and publication levels of review will have to change to accommodate greater acceptance of a broader range of designs that may afford a combination of both scientific rigor and cost-effectiveness (Sanson-Fisher, Bonevski, Green, & D’Este, 2007). Although there has been discussion among funding leaders regarding the importance of maximizing the knowledge generated by relatively costly research in and development of technology-enhanced services (Riley et al., 2013; Rothwell, 2005; Wu et al., 2006), RCTs are still much more likely to be published in top-tier, peer-review journals than more basic or simply alternative designs (Sanson-Fisher, Bonevski, Green, & D’Este, 2007).