Future directions for the examination of mediators of treatment outcomes in youth

Abstract

How do psychological therapies work? How can we enhance treatment to improve outcomes? Questions of mediation lie at the heart of these inquiries. However, within the child and adolescent treatment literature, studies of mediation often rely on methodological and statistical approaches that limit the inferences that can be drawn from study findings. This future directions review delineates some of these issues and suggests improvements through two interrelated paths. We propose that mediation studies in the youth treatment literature will be enhanced (a) by adopting best practices in nomothetic (group-based) methodologies for assessing putative mediating variables and conducting appropriate statistical analyses, and (b) by increasing the use of idiographic (individual-focused) approaches to youth outcome research through mediation studies that use innovative designs, data collection techniques, and analytic methods. We discuss the applicability of findings using these approaches to the treatment of youth in particular.

Mediation helps us understand possible mechanisms through which changes in one variable bring about changes in an outcome (Little, 2013). Studies of mediation have the potential to provide valuable information regarding how treatments work and to identify the mechanisms of change within a given treatment. By using innovative and tailored study designs and analytic approaches, future studies may be able to answer important questions about treatments that can be used to improve their efficiency and efficacy.

When discussing mediation, it is essential to define terminology clearly to ensure that statistical findings are interpreted accurately (Holmbeck, 1997). Mediation refers to the process by which an independent variable is able to influence a dependent variable (Baron & Kenny, 1986). In the treatment outcome literature, mediation typically refers to the process by which an intervention (independent variable) influences treatment outcomes (dependent variables). To maximize inferential value, mediating variables should be assessed after randomizing participants to treatment to demonstrate that treatment condition had an effect on the mediating variables (i.e., that changes in the mediating variable are specific to a given treatment). Mediating variables should also be assessed prior to the posttreatment assessment to demonstrate that changes in the mediator occur prior to change in the outcome (termed “temporal precedence;” Gollob & Reichardt, 1991; Kazdin, 2007).

In the context of treatment outcomes, it is important to distinguish between mediators of treatment outcome and true mechanisms of change, as not all mediators of treatment outcome qualify as mechanisms of change. To declare a mediational process a mechanism of change, a causal relationship between the mediating variable and treatment outcomes must be established (Kraemer, Wilson, Fairburn, & Agras, 2002). For example, once a putative mediator has been identified, mechanisms of change can be examined by conducting a randomized controlled trial (RCT) in which a treatment that is enhanced with components associated with the putative mediators is shown to be more efficacious than the original treatment (Kraemer et al., 2002). Moreover, identifying putative mediators that are redundant or are actually not change mechanisms can help remove unnecessary components of intervention to lower cost and risks to patients (Kraemer et al., 2002).

In the following pages, we argue that mediation studies in the youth literature can be enhanced (a) by adopting best practices in nomothetic (i.e., group-based) studies for assessing putative mediators and conducting appropriate statistical analyses, and (b) by increasing the use of an idiographic (i.e., individual-focused) approach through mediation studies that use innovative designs, data collection techniques, and analytic methods. The goal of the current paper is to highlight issues and challenges common in past mediation studies, to identify limitations inherent in the nomothetic (group-based) approach, and to describe future directions for evaluating mediators of treatment outcome in child and adolescent mental health. In our discussion of future directions for mediation analyses, we focus particularly on idiographic (individual-based) analytical techniques, assessment methods, and study designs that yield valuable information about mediators on an individual level. We discuss the applicability of such findings to treatment of youth in particular.

Methodological and Analytic Challenges in the Mediation Literature

Past literature reviews have highlighted the methodological and analytic limitations in the existing mediation literature, the majority of which is nomothetic in design (e.g., Maric, Wiers, & Prins, 2012; Kazdin, 2009; Kraemer et al., 2002). Although we highlight the most prevalent limitations in this section, Maric and colleagues (2012) provide comprehensive guidelines for improving mediation analyses from multiple perspectives, which we recommend as a foundation for best practices of nomothetic mediation studies (see Table 1).

Table 1.

Our restatement of the Maric, Wiers, and Prins (2012) 10 rules for mediation studies in the youth treatment outcome.

1. Theory provides the basis for the selection of mediators. Answer the question “What is expected to bring about change” in order to test whether treatment is impacting the key processes.

2. Mediation analyses may be improved by investigating potential non-mediators. A non-mediating variable can be one that is expected not to be affected by the treatment being tested, or a variable that is expected to be changed by another treatment.

3. Use adequate measures of the mediating variable when examining statistical mediation. Multiple informants and methods of assessment should be used when possible.

4. Include at least three assessment time points to establish temporal precedence of change in the mediator. Measures of both mediators and treatment outcomes should be taken at all assessment points to test for the reciprocity of mediating effects by comparing the timing of the changes in variables

5. Include at least two treatment conditions. A control group or an alternative active treatment can be used to compare changes in mediating variables, though use of another active treatment condition allows for ruling out nonspecific therapeutic factors that may mediate outcome.

6. After a study meets the requirement of temporal precedence, it is possible to establish the causal relationship between the mediator and the outcome variables. This can be tested using an experimental manipulation of the mediating variable. By comparing two treatment conditions where the only difference is a manipulation in the mediating process and participants are randomized to conditions, it is possible to draw more definite causal conclusions.

7. Future studies should include both moderators and mediators of treatments in the same model, to detect potential moderation of a mediated effect, or mediation of a moderated effect. Inherent in this point is the idea that moderators and mediators must be clearly differentiated and defined.

8. Address the possibility of different orders of causal relations among the mediator and dependent variables: examine the temporal relationship between mediator and outcome. For instance, relying only on statistical significance of mediation models could lead researchers to incorrectly conclude that a particular variable mediates treatment outcomes when change in the outcome variable precedes change in the mediating variable.

9. Single-case experimental designs may be used for initial considerations of mediators.

10. Statistical challenges can pose an obstacle to the study of mediation in youth treatment outcome research (Kraemer et al., 2002; Kraemer, Shrout, & Rubio-Stipec, 2007; Weersing, & Weisz, 2002) but can be addressed by using recently developed advanced approaches.

A major issue in the mediation literature is the selection of mediators. It is crucial to consider the theoretical underpinning of the intervention when identifying variables that are expected to bring about change and testing whether treatment is influencing the processes believed to cause or maintain the disorder. Kazdin (2009) noted the importance of not only identifying a conceptually relevant variable but also understanding how it operates as a means of producing symptom change. The identification of theoretically relevant mediators can also be established by including variables not expected to be mediators and comparing their changes during treatment. A non-mediating variable can be one that is not expected to mediate the treatment being tested or a variable that is hypothesized to be a mediator of another treatment (Maric et al., 2012). Last, it can be useful to include and test multiple measures of each variable to extract latent variables not biased by measurement error (Cole & Maxwell, 2003).

In addition to projects that include measures of mediators and non-mediators, studies are most informative when they include an active intervention condition that is not expected to change putative mediators. Use of an alternate active treatment as a comparison allows one to rule out nonspecific factors (e.g., having sought treatment, positive regard, or time spent in session) that may mediate outcomes across a variety of treatments. To assess the mediating influence of individual treatment components that are typically nested within a treatment program, studies using a dismantling design can untangle the influence of components by separating them into treatment conditions.

A common shortcoming in mediation studies of treatments in the child and adolescent literature is the failure to establish the temporal relationship between the mediating and outcome variables (Gaynor, 2017; Kendall, Olino, Carper & Makover, 2017), or the inaccurate interpretation of the temporal relationship of variables in cross-sectional studies (De Los Reyes, 2017). For example, when mediation processes are examined in cross-sectional data, it typically generates biased estimates of longitudinal mediation parameters (Maxwell & Cole, 2007). Studies must include at least three assessment time points in order to establish temporal precedence of change in the mediator relative to change in outcome variables. However, even when a longitudinal design is incorporated, studies do not always take full advantage of the longitudinal nature of the data (Maxwell & Cole, 2007). To fully examine the trajectories of mediators and outcome measures over the course of treatment, it is best for both mediators and treatment outcomes to be measured at all assessment points–to test for the reciprocity of mediating effects by examining the timing of the changes in both potentially mediating and outcome variables. Additionally, the timing of assessments should reflect theory regarding when the greatest mediation is expected to occur. Measurements taken too early prevent showing that treatment has an effect on the mediating variable whereas measurements taken too late prevent showing that changes in the mediator occur prior to changes in the outcome (Maric et al., 2012).

Establishing a causal relationship between two variables is particularly difficult; as a result, very few studies are able to establish a fully causal relationship between a mediator and outcome variables, even when temporal precedence is established. That said, causal relationships can be tested using an experimental manipulation of the mediating variable. MacKinnon (2008) outlined the (a) blockage and (b) enhancement designs as approaches to manipulation, whereby the mediating variable is either removed from one treatment condition or enhanced in one treatment condition. By comparing two treatments where the only difference is a manipulation of the mediating variable (and participants are randomized), it is possible to draw more definite causal conclusions. Ideally, a study testing a proposed mediator would include low, high, and medium levels of a proposed mediator (Kazdin, 2009) to examine how outcome varies as a function of levels of the manipulated dose. It should be noted, however, that even when a causal relationship is established it may not have “stationarity,” meaning that the causal relationships may not remain stable over time (Cole & Maxwell, 2003).

Finally, statistical limitations pose an obstacle to the applicability of mediation findings in youth treatment outcome research (Kraemer et al., 2002; Kraemer, Shrout, & Rubio-Stipec, 2007; Weersing, & Weisz, 2002). Most past mediation studies have relied on the Barron and Kenny (1986) causal steps approach (Figure 1) paired with the Sobel test (1982). The Baron and Kenny (1986) causal steps approach and the Sobel Test (Sobel, 1982), though leading edge in their day, have more recently been enhanced to optimize the accuracy of findings (Hayes, 2009). For example, the causal steps approach has relatively low power for detecting the indirect effect of the independent variable on the dependent variable through the mediating variable (Fritz & MacKinnon, 2007; MacKinnon, Lockwood, Hoffman, West, & Sheets, 2002) and it depends on inferences from the outcome of a set of hypothesis tests rather than a direct quantification of the intervening effect (Maric et al., 2012). The Sobel Test has also been criticized for the assumption that the sampling distribution of the indirect effect is normal when in actuality the product of coefficients tends to be asymmetric with nonzero skewness and kurtosis (Bollen & Stine, 1990; Stone & Sobel, 1990; Hayes, 2009). Further, the Sobel Test does not account for the possibility that a mediated effect may be present whether or not the treatment effect (path c’ in Figure 1) is non-significant after adjustment for the mediator. Other strategies have been developed to test the significance of indirect effects, such as bootstrapping, which tends to have higher power and excellent control of Type I error. Bootstrapping generates an empirical representation of the sampling distribution of the indirect effect by treating the obtained sample as a population, and repeatedly resampling during analysis as a means of mimicking the original sampling process (Hayes, 2009).

Figure 1.

Baron and Kenny (1986) mediation diagram.

Other statistical techniques have been developed to specifically examine mediation questions. For example, path analysis within the structural equation modeling (SEM) framework is a multivariate technique that can provide an appropriate inference framework for mediation analyses (Gunzler, Chen, Wu, & Zhang, 2013). SEM can be used in a variety of complex situations, including when the mediation model includes multiple independent variables, mediators, or outcomes. SEM allows for extension to longitudinal data within a single framework, and concurrently tests the mediating variable as both a cause for the outcome and an effect of the intervention (Gunzler et al., 2013). Bootstrapping can be used within the SEM framework to provide more accurate estimates of the indirect effect.

Limitations inherent in the nomothetic approach

Nomothetic studies, when appropriately designed and analyzed, yield important information about relationships between variables at the group level. However, there remain limitations to the applicability of these group-based findings for individuals in treatment. A nomothetic approach assumes that patients change over time similarly to one another and all change because of the same mechanisms of treatment (Kazdin, 2009). However, not all patients change identically and, as Kazdin (2009) pointed out, even if all patients change because of an identical mechanism, the timing and patterns of change may vary. Most research aimed at the study of therapeutic change in youth has taken a nomothetic approach, aggregating samples to strive to identify what predicts, moderates, and mediates outcomes for the “average” youth (Nock, Janis, & Wedig, 2008). Deviations from the “average” youth, or intra-individual variability, have typically been viewed as error in common analytic techniques such as linear regression and other ANOVA/ANCOVA-based approaches (Collins & Sayer, 2000; Molenaar, 2004). However, it can be argued that only a small number of youth present for treatment similarly to the “average” youth, which can limit the transportability of findings from research studies to clinical settings.

The notion that most participants are not “typical” was demonstrated in a study by Borkenau and Ostendorf (1998) where they sought to determine whether what holds true on the group level can generalize to the individual level. Consider the Big Five personality traits, which have been shown to be stable over time, and consistently map onto a five-factor structure when examined in factor analytic studies at the group level. Borkenau and Ostendorf (1998) had participants (N = 22) complete a measure of the Big Five traits each night for 90 days. When the authors used factor analytic approaches that examine inter-individual variation, the five-factor model was supported. However, when the authors used factor analytic approaches that examined intra-individual variation (i.e., one person’s variation across the 90 administrations of the measure) the factor structure was not consistent across participants and did not map onto the five-factor model (Borkenau & Ostendorf, 1998). The intra-individual models differed among participants in both the number of factors extracted and in how the factors related to the individual items.

In the context of the study of change in youth receiving psychological treatments, the Borkenau and Ostendorf (1998) study illustrates that we cannot assume that results from analyses that aggregate individuals and analyze their inter-individual variability will generalize to individual participants. How can we use information from these studies to inform treatment with individual patients when the analytic techniques used do not generalize back to individual patients? This complaint about research is common among practitioners (Youngstrom & Van Meter, 2016) and warrants attention. Increased use of idiographic approaches that complement nomothetic studies may be one way to proceed, a call that has been echoed in recent years (Barlow & Nock, 2009; Kazdin, 2000; Kazdin & Kendall, 1998; Kazdin & Nock, 2003; Maric, Wiers, & Prins, 2012; Molenaar, 2004; Nesselroade, Gerstorf, Hardy, & Ram, 2007).

Another frequent assumption in many analytical approaches to mediation is that change is linear or quadratic, which is rare in practice (Hayes & Feldman, 2007; Hayes, Laurenceau, Feldman, Strauss, & Cardaciotto, 2007; Laurenceau, Hayes, & Feldman, 2007). To successfully model nonlinear change, multiple assessments from the same individual must be collected prior to, during, and after treatment. Novel approaches to data collection and analysis offer promise to improve the ability to identify mediators of treatment outcome, and eventually, mechanisms of therapeutic change.

Despite some improvements in mediation studies, we have not yet sufficiently examined fundamental questions regarding how youth psychological treatments work (Kendall et al., 2009; Prins & Ollendick, 2003; Weersing & Weisz, 2002). This situation is particularly true for studies of youth (Hinshaw, 2002; Kraemer et al., 2002). Weersing and Weisz (2002) reviewed 67 studies of youth interventions and found that although over half of the studies reviewed measured potential mediators, only six included a formal mediation test. The authors noted that mediational inferences in these six studies were diminished because in all studies reviewed, the mediating variable was assessed at posttreatment (i.e., concurrent with outcome), which does not provide temporal precedence. Given existing limitations, what are the next steps for examining mediation of treatment outcomes in youth?

An exciting consequence of the ubiquity of powerful personal computers is the proliferation of statistical techniques for examining mediation in a more nuanced way. Additionally, advanced theoretical models from other disciplines are now being applied in psychology to model mediational processes. Though a review of all possible techniques for examining mediation is beyond the present scope, we consider several of the many exciting advances in the study of treatment mediation.

Dynamical Systems Approaches

One approach to the study of therapeutic change over time in an idiographic manner is dynamical systems. Dynamical systems approaches have been used in a variety of disciplines to model the nonlinear behavior of a variety of systems. Dynamical systems approaches have been increasingly used in mental health generally (Vallacher, Coleman, Nowak, & Bui-Wrzosinska, 2010; Vallacher & Nowak, 1997), as well as specifically to study mechanisms of change in CBT (Fisher, Newman, & Molenaar, 2011; Hayes & Feldman, 2007; Hayes et al., 2007; Hayes & Strauss, 1998; Hayes, Yasinski, Ready, Laurenceau, & Chen, under review) and other models of psychological treatment (Pascual-Leone, 2009). A dynamical system is one that changes over time in response to input from the environment and from itself at an earlier point in time. Dynamical systems are often studied through the use of differential equations to model individual trajectories (Boker, 2001; Boker & Laurenceau, 2006; Salvatore & Tschacher, 2012; Thelen & Smith, 2006) or other techniques that use individual time-series data to analyze change on a person-by-person basis and allow for variation in the shape of intra-individual change such as directed acyclic graphs (DAGs; for review, see Foster, 2010), hidden Markov modeling for categorical data, and other analytical techniques (for review, see Gates & Liu, 2016).

The dynamical systems approach is concerned with how variables relate to each other and change over time. The approach emphasizes the study of stable patterns, termed “attractors,” and how these patterns destabilize and change over time to be replaced by new attractors, termed “phase changes.” These patterns serve as a better unit of analysis than individual variables because they concurrently emphasize the relationships both within and between variables (Kelso, 1995; Thelen & Smith, 2006). Given that many therapies are designed to dislodge stable and maladaptive patterns of cognition and behavior that maintain a disorder and replace these patterns with more adaptive ways of functioning, dynamical systems approaches can provide a useful framework for quantifying the processes active prior to, during, and after receiving treatment (Hayes & Yasinski, 2015). The use of a dynamical systems approach to study mechanisms of change in CBT, for example, allows for the integration of information to inform novel experimental paradigms designed to provide information about how CBT achieves its beneficial effects. Through a better understanding of how CBT achieves its beneficial effects and what factors differentiate treatment responders from nonresponders, interventions can be personalized to target treatment-refractory youth.

A small number of studies have used dynamical systems analyses to examine mediation and the mechanisms of therapeutic change. Though not the traditional way mechanisms of change are evaluated (i.e., through an RCT; see Kraemer et al., 2002), the dynamical systems approach is able to test potential mechanisms of change because of its use of measurement methods and analytical techniques that examine moment-to-moment changes at the individual level. Given the lack of studies using this approach with youth, we focus on some of the methods as used with adults. The approach can, and should, be extended to youth populations.

Spectral analyses and dynamic factor models

Fisher and colleagues (2011) applied dynamical systems analyses to model data from 33 adult participants who each completed multiple diary entries daily while receiving either CBT or applied relaxation for Generalized Anxiety Disorder (GAD). Using spectral analyses and the residual variance from dynamic factor models (i.e., vector autoregressive models with a one-day lag), the authors found that spectral power interacted with time to predict treatment response over a 1-year follow-up, such that lesser spectral power (less intense and frequent variability in anxiety symptoms) predicted increases in reliable change over one-year follow-up. Also, greater spectral power (more intense and frequent variability in anxiety symptoms) predicted decreases in reliable change over the follow-up. Additionally, the residual variance from the dynamic factor models significantly moderated the slope of reliable change over the follow-up period, such that greater order in dynamical systems (i.e., less residual variance from the dynamic factor models) predicted increases in reliable change over the follow-up and less order (i.e., greater residual variance from the dynamic factor models) predicted decreases in reliable change over the follow-up (Fisher et al., 2011).

Newman and Fisher (2013) examined whether the duration of GAD moderated response to CBT versus its components (cognitive therapy [CT] and self-control desensitization [SCD]), and whether increases in dynamic flexibility of anxious systems during therapy was a mediator of the moderated effect of duration of GAD on outcome (i.e., mediated moderation). Consistent with their prior work (i.e., Fisher et al., 2011), the authors quantified dynamic flexibility as the inverse of spectral power from daily to intradaily oscillations in daily diary data. Results revealed that duration of GAD moderated outcome such that those with longer duration showed greater reliable change from component treatments compared to CBT alone. Increases in flexibility over the course of therapy fully mediated the moderating effect of GAD duration on condition. The authors concluded that individuals who have had GAD for a longer duration may respond better to more focused treatment, and that the mechanism by which this moderation occurs is through the establishment of flexible responding during treatment (Newman & Fisher, 2013).

State space grids and the GridWare program

The computer program GridWare (Lamey, Hollenstein, Lewis, & Granic, 2004) allows for the construction of state space grids, which are two-dimensional planes formed by the intersection of two axes (Hollenstein, 2013). The two axes represent two distinct variables, such as positive affect and negative affect, and users can plot the activation (i.e., score) of each variable that each individual endorses over time. In this manner, GridWare captures activation of one variable (i.e., a single attractor), activation of multiple variables (i.e., multiple attractors), and the movement from the activation of one variable to the activation of another (i.e., phase changes). The program uses multiple methods to quantify attractors, phase changes, and other variables relevant to dynamical systems (see, DiDonato, England, Martin, & Amazeen, 2013; Granic, Hollenstein, Dishion, & Patterson, 2003; Granic & Lamey, 2002; Hayes et al., under review). A sample state space grid from the GridWare program is presented in Figure 2.

Figure 2. Sample state space grid from GridWare program.

Note: Values on variable 1 and variable 2 are plotted on the X- and Y- axis, respectively. Empty circles represent the starting point. Lines with arrows represent movement over time. Larger circles indicate that a participant stayed in that area of the grid for a longer period of time.

Hayes and Yasinski (2015) used GridWare to re-analyze data from 27 participants in an open trial of cognitive therapy for personality disorders. The authors coded treatment sessions for positive network activation (adaptive functioning) and negative network activation (maladaptive functioning). Using hierarchical regression, the authors found that more dispersion (more instability) in the second phase of therapy predicted more improvement in personality disorder symptoms and positive network strength at posttreatment. The authors concluded that the GridWare program represents a relatively easy to use quantitative method for studying the process of change among patients receiving psychological treatment (Hayes & Yasinski, 2015).

Latent Difference Scores

Latent difference scores and bivariate latent difference scores (LDS and BLDS, respectively; McArdle, 2009; McArdle & Hamagami, 2001; McArdle et al., 2004; McArdle & Nesselroade, 1994) are appropriate for studying temporal relationships between putative mediators and treatment outcomes. The BLDS approach allows researchers to concurrently examine three types of change over time: additive change over time (α), multiplicative change over time (β), and the effect that changes in one variable have on later changes in another variable over time (γ). The use of these three change components allows for a variety of nonlinear trajectories to emerge. Additionally, the use of the structural equation modeling (SEM) framework allows for the inclusion of participants with incomplete data.

Figure 3 presents a BLDS model for data collected at four time points. As shown, the α parameter represents a coefficient similar to a latent growth curve model, where change over time is consistent. The β parameter represents a coefficient explaining change from one session to the next. Finally, when examining the temporal relationship between two variables over time, the γ parameter represents the effect of one variable at time T on another variable at time T + 1. The use of BLDS models allows researchers to model both uni-directional and bi-directional (i.e., reciprocal) effects to determine the temporal relationships between two variables. Given that one of the primary methodological and statistical concerns with research examining change is establishing temporal precedence of a mediating variable, BLDS represent a timely addition to the statistical repertoire of treatment outcome researchers.

Figure 3. Prototypical bivariate latent difference score model with four time points.

Note: Traditionally, labeled parameters are constrained to be equal. Unlabeled parameters are fixed to 1.

The BLDS approach was used, for example, in a study examining the relationship between the therapeutic alliance and symptom improvement over the course of CBT for child anxiety. Using data from a large RCT, Marker, Comer, Abramova, and Kendall (2013) found that the relationship between the therapeutic alliance and symptom change varied depending on who was reporting the alliance and symptoms. Changes in mother- and therapist-reported therapeutic alliance significantly predicted later changes in child anxiety symptoms. Changes in child-reported anxiety also predicted later change in father- and therapist-rated alliance. Finally, it was also reported that the relationship between child-reported anxiety and child-reported alliance was not significant. The findings suggested that mothers develop an alliance early and this is predictive of symptom change, whereas fathers needed to see symptom change before developing an alliance. In addition, the relationship between therapist-reported therapeutic alliance and symptom change was bidirectional, with changes in alliance predicting later changes in anxiety, and changes in anxiety predicting later changes in alliance (Marker et al., 2013). These findings underscore the importance of not relying on a single assessment source and/or time point when considering putative mediators.

Promising Study Designs for Examining Mediators

Use of these analytic approaches does require individual participants to frequently and consistently provide information over time. Therefore, it can be costly to conduct idiographic research and some of the assessment methods discussed next can be burdensome to participants, particularly youth. Thus, it may not always be feasible for researchers to take the idiographic approach to examining mediators of treatment outcome. Several novel study designs have been developed that balance the practical concerns of conducting idiographic research without limiting researchers’ ability to make useful inferences regarding meditational findings. These include adaptive intervention trials such as the Sequential Multiple Assignment Randomized Trial (SMART; Almirall, Nahum-Shani, Sherwood, & Murphy, 2014; Nahum-Shani, et al., 2012), as well as single case designs, among others.

A SMART is a randomized trial study design whereby participants are initially randomized to a study condition and frequently reassessed to determine if the initial intervention is having an effect. Based on decision rules that are set up a priori, a participant can be re-randomized at given times dependent on their treatment response, other factors that the researchers decide, or in the case of unrestricted SMARTs, based on time alone. SMART studies are well-suited to answer a variety of research questions related to how best to personalize interventions to individual patient presentations and what variables may mediate or moderate treatment response (see, Almirall & Chronis-Tuscano, 2016). The Methodology Center at the Pennsylvania State University has a website that offers a variety of resources for researchers interested in learning more about SMARTs (http://methodology.psu.edu).

Although novel study designs such as SMART are being developed, more traditional treatment outcome studies can also be used to study dynamical systems. The following section outlines the necessary assessment methodology for modeling dynamical systems during intervention studies.

Assessment Methods for Idiographic Research

Dynamical systems approaches offer analytical techniques to understand mediators and mechanisms of change. However, research taking a dynamical systems perspective requires intensive longitudinal data, and the use of multiple assessment points throughout treatment has been advocated (Collins & Sayer, 2000, 2001; Kraemer et al., 2002). Although the data necessary for a dynamical systems approach requires frequent, nuanced methods of data collection and sophisticated analyses, the ultimate goal of this approach is to reduce complexity in clinical practice and increase the efficiency of treatments. Therefore, the financial and time investment needed to employ dynamical systems methods ideally will allow clinicians to distill interventions and make targeted clinical decisions guided by actuarial data (Fisher & Boswell, 2016).

One of the common assessment techniques used in idiographic research is ecological momentary assessment (EMA). EMA methods collect real-time data in participants’ natural environments, with the opportunity to assess changes over time and across situations (Shiffman, Stone, & Hufford, 2008). EMA methods reduce the retrospective recall bias associated with self-report questionnaires (Piasecki, Hufford, Solhan, & Trull, 2007; Solhan, Trull, & Wood, 2009). With measurement precision, studies using EMA methods invite inquiry into previously challenging research questions.

Whereas EMA began with beepers, daily diaries, and automated phone calls, the rapid rate at which technology is advancing has changed the face of EMA data collection. Smartphones and tablets are common, with 73% of youth aged 13–17 owning a smartphone, and 58% of youth owning a tablet (Pew Research Center, 2015). Given the ubiquity of smartphones, scientists have teamed with application (“app”) developers to develop smartphone apps capable of improving the scientific process, including the collection of EMA data (Estrin & Sim, 2010; Kendall, Carper, Khanna, & Harris, 2015; Luxton, McCann, Bush, Mishkind, & Reger, 2011). Some researchers have developed ecological momentary intervention (EMI) apps that provide targeted and individualized interventions in real time (Pramana, Parmanto, Kendall, & Silk, 2014). Data gathered using EMA methods (repeated assessments of the same individual over time) allow for the statistical modeling of both inter- and intra-individual change and provide opportunities for numerous secondary analyses. In the context of mediation and the search for mechanisms of change, EMA methods gather intensive longitudinal data in the participant’s everyday lives, which allows researchers to create precise mediation models and to examine temporal relationships between variables across time.

It would not be appropriate to discuss assessment methods without including a brief discussion of evidence-based assessment. Numerous researchers have advocated for inclusion of multiple assessments of the same construct and the inclusion of information from multiple sources given that youth self-report can be limited by a variety of developmental factors (see De Los Reyes et al., 2015). Methods such as the diagnostic likelihood ratio (DLR; for application to child and adolescent psychology, see, Youngstrom & Van Meter, 2016) have been developed to integrate information from a variety of sources and informants to create a comprehensive picture of an individual’s difficulties. Evidence-based assessment should serve three broad functions: prediction, prescription, and process (see, Youngstrom, 2013; Youngstrom & Van Meter, 2016). The most relevant function to the present discussion is the process whereby assessment focuses on monitoring treatment progress, assessing potential mediating variables, and adjusting interventions accordingly. It is our hope that future researchers will design studies that serve these three functions of evidence-based assessment, which will ultimately further our understanding of mediators and moderators of treatment response.

Conclusions

Despite calls to examine mediators of youth treatment outcomes, the availability of data to test mediation hypotheses (Weersing & Weisz, 2002), and the publication of best practice guidelines for examining mediation (Holmbeck, 1997; Kraemer et al., 2002; Maric et al., 2012), mediation remains largely understudied. Moreover, the literature faces methodological and statistical shortcomings that limit our ability to use research outcomes to make changes in clinical practice. We believe that mediation is an incredibly important area of treatment research that merits following best practice guidelines and using novel analytical and data collection methods.

Despite Holmbeck’s (1997) early efforts, there continues to be an inconsistent use of terminology regarding mediation. For example, when describing significant mediational findings, authors may conclude that a variable mediates treatment outcome when there was an absence of an alternative treatment or comparison condition; in this case, the variable mediates the relationship between the independent and dependent variable, but does not mediate treatment outcome. This inconsistency is also problematic when the measure of the putative mediating variable did not have temporal precedence; authors sometimes note the lack of temporal precedence as a study limitation, but abstracts often present the findings as providing support for mediation without including this important caveat. Consistent with Holmbeck (1997), we encourage authors and reviewers to revisit the recommended best practices (e.g., Holmbeck, 1997; Kraemer et al., 2002; Maric et al., 2012) for the design and reporting of mediation. Consistency in terminology will facilitate progress.

Another reason that the field has struggled to identify mediators of treatment response may the reliance on nomothetic analytical techniques. By aggregating participants into groups and performing analyses based on group means, there is a limited ability to generalize back down to the individual participant (Molenaar, 2004; Molenaar & Campbell, 2009). Providing psychological treatment has idiographic features and the use of analytical techniques that capture both inter- and intra-individual differences offer promise.

Many of the concepts discussed in this article dovetail quite nicely with the National Institute of Mental Health’s (NIMH) focus on the Research Domain Criteria (RDoC). The RDoC is a classification system for research on mental illness that focuses on identifying mechanisms underlying psychopathology (Insel et al., 2010). The analyses, assessment methods, and study designs discussed herein focus on moving beyond symptom-level measures and toward integrating multiple measures and sources of information with the goal of illuminating patterns of cognition, affect, and behavior underlying child psychopathology. Indeed, the idiographic approaches are designed to uncover relationships between variables across time, which are not restricted to a single diagnostic category. For example, a researcher could create a state space grid of daily ratings of positive and negative affect across the course of treatment. Using the GridWare program to calculate the dispersion of this grid across specific sessions of treatment would represent a measure of affect regulation that could be used in a latent growth curve mediation model whereby the slope of the latent growth curve mediates treatment outcome. There are a multitude of possibilities for designing studies consistent with RDoC.

Finally, the dynamical systems approach to data analysis may further our efforts to understand the mediators and mechanisms through which psychological treatments achieve their beneficial effects. Four dynamical systems analytic techniques were reviewed: spectral analyses, dynamic factor models, state space grids and the GridWare program, and latent difference score models. These analyses, though more complex than ANOVA and linear regression analyses, remain available in standard statistical software packages. For example, the R packages “lomb” (https://cran.r-project.org/package=lomb) and “cts” (https://cran.r-project.org/package=cts) allow users to conduct spectral analyses with unevenly-spaced data (as would typically be the case for self-reported EMA data or when missing data is present). Dynamic factor models and latent difference score models can be carried out in standard SEM software packages such as MPlus (https://www.statmodel.com/), LISREL (http://www.ssicentral.com/lisrel/), and the R package “Lavaan” (https://cran.r-project.org/package=lavaan). Through analysis at the individual level, these approaches allow an idiographic understanding of therapeutic change. One of our hopes is that child and adolescent psychological treatment researchers will incorporate idiographic techniques into the study of mediation so that we can continue to enhance the efficiency, specificity, and efficacy of our interventions.