Abstract
There has been slow progress in the development of interventions that prevent and/or reduce mental health morbidity and mortality. The National Institute of Mental Health (NIMH) launched an experimental therapeutics initiative with the goal of accelerating the development of effective interventions. The emphasis is on interventions designed to engage a target mechanism. A target mechanism is a process (e.g., behavioral, neurobiological) proposed to underlie change in a defined clinical endpoint and through change in which an intervention exerts its effect. This article is based on discussions from an NIMH workshop conducted in February 2020 and subsequent conversations among researchers employing this approach. We discuss the components of an experimental therapeutics approach such as clinical outcome selection, target definition and measurement, intervention design and selection, and implementation of a team science strategy. We emphasize the important contributions of different constituencies (e.g., patients, caregivers, providers) in deriving hypotheses about novel target mechanisms. We highlight strategies for target mechanism identification using published and hypothetical examples. We consider the decision-making dilemmas that arise with different patterns of results in purported mechanisms and clinical outcomes. We end with considerations of the practical challenges of this approach and the implications for future directions of this initiative.
What is the most efficient and effective manner to develop and test innovative interventions to prevent and treat psychopathology? In 2015, the National Institute of Mental Health (NIMH) included the Experimental Therapeutics Initiative in the NIMH Strategic Plan (National Institute of Mental Health, 2015). The goal was to “translate the growing understanding of the factors that cause and sustain mental illness into new or improved approaches to prevention and treatment” (NIMH Director’s Message, Gordon, 2017). Specifically, interventions were to be designed to engage a target mechanism. A target mechanism is a process (e.g., affective, cognitive, behavioral, neurobiological) defined and measured at any level of analysis (e.g., neural circuit function, behavioral observation) that is proposed to underlie change in a defined clinical endpoint and through change in which an intervention exerts its effect.
In February 2020, leaders of the NIMH convened a workshop with teams of investigators who had been examining psychosocial interventions within the experimental therapeutics framework. Investigators discussed the opportunities afforded by this initiative and the dilemmas experienced. Since then, there have been numerous conversations further attempting to codify the mandates of this initiative into effective therapeutic responses. Our intention in writing this paper is to serve as a thought piece for future researchers to consider. We summarize what we have learned in responding to this initiative and describe challenges that may arise when defining and testing novel mechanistic targets. This article begins with a history of the experimental therapeutics approach within NIMH and then considers phases of the experimental therapeutics framework including clinical outcome selection, target identification and measurement, intervention design, and team selection and the challenges inherent in each phase. We consider different patterns of results and possible decisions resulting from these patterns. We end with some considerations for future directions of this initiative.
History of the Experimental Therapeutics Initiative within NIMH
In a posting dated June 12, 2012, then NIMH Director, Dr. Thomas Insel, reflected on the disappointing progress regarding the development of novel pharmacological agents that meaningfully affect the morbidity and mortality of mental illness (Insel, 2012). Paul et al. (2010) had earlier offered a systematic analysis of the barriers to research innovation and productivity in the pharmaceutical industry. These authors concluded that attrition in the later phases (Phase II and Phase III trials) of the development pipeline was a huge detriment to productivity. In contrast, strategic target selection in the early phases of development was a critical determinant to maximize the probability of technical success (e.g., regulatory approval) for a given drug candidate (Paul et al., 2010).
The Experimental Therapeutics Initiative was introduced into the NIMH Strategic Plan in 2015 (National Institute of Mental Health, 2015). It was initially proposed as a strategy to increase the efficiency and effectiveness of selecting and testing novel drug candidates by optimizing target selection (Grabb et al., 2016). In the context of psychiatric disorders, targets for intervention could be selected based on an understanding of central nervous system function (CNS) purported to contribute to a specified clinical symptom or disorder. Following the identification of a sensitive CNS target, an appropriate investigational drug/strategy would be identified as an intervention tool that could affect that CNS target. Key elements of this approach were that CNS target engagement could be reliably measured and that engagement of that CNS target mechanism had the potential to impact clinical symptoms. Proof-of-concept studies would then be designed to demonstrate whether a drug evidences target engagement. Ideally, such studies would be designed using a patient population that is relatively homogenous for the presence/level of the CNS target and/or clinical feature purported to be influenced by that target (Grabb et al., 2016). These strategies were proposed to be essential features in maximizing benefit and minimizing risk further along the development pipeline (Insel, 2012, 2014; Paul et al., 2010). This proposal was coined a “fast-fail” initiative, one designed to assign more resources early in the development cycle by focusing on strategies for optimizing target selection that could fail quickly if sufficient engagement was not achieved. Sufficient mechanistic target engagement would then lead to the next phase of development – demonstrating that change in the mechanistic target contributed to change in the clinical outcome. Correspondingly, a failure to demonstrate sufficient mechanistic target engagement would result in the “fast-fail” of that trial. Thus, improving target definition was proposed as a strategic priority for intervention research and development efficiency (Grabb et al., 2016; Grabb et al., 2020) and became central to the experimental therapeutics initiative of the NIMH.
The fast-fail approach as applied to pharmaceutical development for psychiatric symptoms has shown some preliminary success (Grabb et al., 2020). For example, Krystal et al. (2020) targeted the transdiagnostic symptom of anhedonia and provided evidence of target engagement (e.g., reward circuit activation via κ-opioid receptor antagonism to address anhedonia, Krystal et al., 2018; Krystal et al., 2020) and then mechanism-driven improvement in clinical symptoms (Krystal et al., 2020; Pizzagalli et al., 2020). See Grabb et al. (2020) for other examples.
Related Initiatives at the National Institutes of Health (NIH)
Yet target mechanism definition, evidence of target mechanism engagement, and the measurement challenges therein require unique assumptions and evidence within the context of psychosocial interventions. Addressing these challenges has been explored in depth in the context of other initiatives within the NIH and by other authors. For example, such discussions may benefit from incorporating the conceptual and methodological advances devoted to understanding the mechanisms of health behavior change, the mandate of the NIH Science of Behavior Change (SOBC) Research Network. Shared terminologies and strategies across initiatives stand to greatly synergize advances in how we define and explore mechanisms of change more broadly, a focus of the SOBC Research Network (see Nielsen et al. (2018) for a summary of the SOBC initiative). For some related commentaries about the potential value and challenges of an experimental therapeutics framework, see Holmes et al. (2014) and Singh et al. (2020b). Sheeran et al. (2017) wrote a thoughtful and thorough review that defines and operationalizes the experimental medicine approach to health behavior change. We attempt to incorporate the terminology employed in their review throughout this paper.
Goals of the Experimental Therapeutics Approach
In essence, the experimental therapeutics approach has three core goals: 1) learn more about the mechanisms that increase risk, maintain psychiatric symptom(s), or sustain the impairment related to symptom(s); 2) develop effective strategies to intervene upon the identified target mechanism; and 3) by doing so, improve the risk profile or trajectory of mental disorders. Given that these core assumptions are met (an identified target mechanism, engagement of that target mechanism, and validation of the targeted mechanism via corresponding clinical change), new information may become available to potentiate intervention strategies (Sheeran et al., 2017). Establishing a mechanism-intervention-outcome relationship may afford researchers greater precision in personalizing intervention strategies. For example, interventions may be more effective for individuals with elevations on a proposed mechanism or in titrating treatment dose: a greater dose of the intervention may result in greater change in the targeted mechanism and subsequent greater symptom change (or, alternatively, elucidate step-function or threshold effects), thereby further personalizing intervention strategies. This process may guide hypotheses about treatment non-response: individuals who fail to respond may not have sufficiently engaged the target, or the engaged target was not valid (it was not strongly related to the clinical outcome), as examples see Sheeran et al. (2017).
One proposed benefit of this treatment development initiative is that clinical trials are to be designed so that negative results help advance our understanding of relevant target mechanisms, thus ruling out proposed mechanisms that may not be amenable to change or not related to the target outcome. With this increased emphasis on informative “fast-fail” innovation, there is also an increase in financial expenditures for this early phase of intervention development. Finally, this initiative was intended to lead to improved treatments. For example, some intervention strategies may have a multitude of different skills and associated mechanisms that putatively underlie each of these skills. Imagine that this treatment results in an effective treatment response for a proportion of individuals, but also that a proportion of individuals fail to respond—what Sheeran et al. (2017) refer to as treatment response heterogeneity. At the end of a trial with these constituents, the lives of a proportion of people have been improved; however, we might not truly understand why improvement occurred. We also may not have immediate hypotheses about why certain individuals failed to respond. Were there differences in target engagement across individuals? Were there differences in the validity of the target such that it was related to clinical outcomes in some but not others? Thus, the overriding intention of this initiative is to efficiently design clinical trials to optimize what can be learned in terms of how the treatments work, for whom, at what dose, and what can be done for non-responders. In the sections that follow, we discuss considerations for working in the experimental therapeutics framework for psychosocial interventions for mental health symptoms or disorders.
Choosing a Clinical Outcome
Change in a mechanistic target may not contribute to change in all symptoms within a psychiatric syndrome. Given that psychiatric diagnoses comprise a constellation of diverse symptoms—any combination of which can result in the same diagnosis with much phenotypic diversity—it stands to reason that any mechanistic treatment development may need to be targeted to a precise symptom or symptoms proposed to share a common mechanism, rather than complex diagnostic entities (Grabb et al., 2020). Although not impossible, the challenge of the researcher would be to establish that the same mechanism underlies the expression of all symptoms of a psychiatric syndrome and that thereby reducing that mechanism results in broad sweeping change. Since phenotypic heterogeneity and different pathways to similar outcomes (equifinality) make this goal unlikely, a focus on mechanisms underlying specific symptoms (either within or across disorders) may be beneficial. For example, Pizzagalli, et al. (2020) targeted the clinical outcome of anhedonia. Elevated scores on a self-report measure of anhedonia served as an initial screen for eligibility, and impairing levels of anhedonia were confirmed by clinical interview (Pizzagalli et al., 2020). Notably, the psychiatric diagnoses of the sample included major depressive disorder, bipolar disorder, social anxiety disorder, panic disorder, and post-traumatic stress disorder. Targeting a clinical symptom rather than a syndrome may increase the efficiency of conducting clinical trials by allowing for precision in screening for eligibility—a strategy that may maximize the probability of success when conducting a targeted mechanistic trial.
However, there are important caveats to consider with this approach. In his influential article on modern validity theory, Kane emphasizes that validity is not a property of a test (i.e., a screening measure for anhedonia is not “valid”) but rather, validity is about the appropriateness of interpretations of a score, decisions that are based on these interpretations, and the actions taken as a result of those interpretations and decisions (Kane, 2013; Messick, 1989; Sawatzky et al., 2017). Consider the example of screening for treatment study eligibility. The score on a screening measure is interpreted as a degree of a certain symptom or feature. The decision made regarding this interpretation is whether an individual can advance to the next stage of a clinical trial. Given the important decisions based on the interpretation of this score, issues of justice are paramount: differential item functioning between groups may also lead to unfair exclusion of potentially eligible participants (Sawatzky et al., 2017). Differential item functioning occurs when items perform differently for different groups for reasons not intended by the measurement developers. Higher-stakes decisions require stronger validity arguments to justify the use of scores to inform those decisions.
Patient voices are also paramount to consider in clinical outcome selection. Yet, domains that patients value may not be an intervention focus or assessed as an outcome in studies designed by research teams. For example, Chevance et al. (2020) surveyed patients with current or prior depression or bipolar disorder (N = 1912), informal caregivers (N = 464), and healthcare professionals (N = 627) from 52 countries to create an exhaustive list of clinical outcome domains that are prioritized by patients (i.e., “most difficult aspect of depression to live with”). Findings revealed that many of the domains most frequently mentioned by patient and caregiver groups were not routinely assessed in clinical trials (Chevance et al., 2020). Researchers such as Weinfurt (2019) and Anvari & Lakens (2021) go one step further and advise using patient perceptions of change that are meaningful (i.e., noticeable and valuable) to guide the delineation of benchmarks for clinical endpoints. To the degree that items on a patient-reported outcome measure reflect an outcome noticeable and valuable to the patient (e.g., I am able to get out of bed every morning), the choice of measurement tools may be more straightforward. Other methods, however, such as anchoring, can also be used to define the smallest unit of change patients perceive as meaningful (Anvari & Lakens, 2021; Weinfurt, 2019). Thus, if researchers focus on clinical symptoms rather than syndromes in defining clinical outcomes, a priority in this decision-making process is choosing a feature that meaningfully impacts the experience of the patient and one that patients would prioritize as needing change.
Defining a Target Mechanism
Considerations for clinical outcome selection are, by necessity, integrated with the assumptions that guide target mechanism definition and selection. As noted earlier, a target mechanism is a process (e.g., affective, cognitive, behavioral, neurobiological) defined and measured at any level of analysis (e.g., neural circuit function, behavioral observation) that is proposed to underlie change in a defined clinical endpoint and through change in which an intervention exerts its effect. Thus, the core components of any target mechanism, irrespective of level of analysis, are that it can be precisely operationalized, reliably measured and modified, and that the time course of change in the mechanism can be linked causally to a change in a clinical outcome. From the funding opportunity announcement: (National Institute of Mental Health, 2020)
“An appropriate target is an intervening variable that has either been demonstrated to be associated with risk for a mental disorder, with a clinical symptom or functional deficit, or is hypothesized (based on empirical evidence) to impact the biological or psychological pathway through which a clinical or functional benefit would be expected to occur. Thus it is hypothesized that change in the target will mediate the intervention’s clinical or functional impact.”
Providing a Rationale for Target Selection
Given these assumptions, what is the body of evidence needed to establish that a novel mechanism is worth targeting? We discuss considerations when defining a target mechanism. To this end, the framework of argument-based validity may be helpful in outlining a process of evidence gathering for target definition, measurement, and decision-making (Kane, 2013). As typically employed in the context of psychological and educational testing, argument-based validity is a process of establishing the validity of the interpretation of test scores for a particular use (Kane, 2013; Weinfurt, 2021). Proponents of modern validity theory emphasize that tests are not valid: rather, the acceptable interpretation of a score from a test and the use and decisions based on that score become valid based on the body of evidence collected in support of the Interpretation/Use Argument proposed by the researchers (Hawkins et al., 2021; Kane, 2013; Messick, 1989; Sawatzky et al., 2017). As stated by Kane (2013),
“Scores are used to support claims. These claims are not generally self-evident and merit evaluation. To validate an interpretation or use of (test) scores is to evaluate the plausibility of the claims based on the test scores.”
We consider several domains in which evidence in support of a target mechanism is essential: evidence that a target mechanism is associated with a clinical outcome, evidence that a certain magnitude of change in the target mechanism would be likely to produce change in the clinical outcome, and evidence to advance understanding of the target mechanism. We discuss each of these in turn.
The target mechanism must plausibly impact the clinical outcome. One critical component in target definition is establishing the logical argument and evidence as to why a given mechanism would be associated with a defined clinical endpoint. Thus, evidence that variations in the target mechanism predict later clinical outcomes and the related empirical and theoretical support that changing this target mechanism could noticeably and valuably impact the designated clinical outcome is the essence of one component of a validity argument (Anvari & Lakens, 2021; Weinfurt, 2019). A key question to ask at this stage is whether targeting this proposed mechanism is sufficient: i.e., if this mechanism were modified, would the designated clinical outcome show a corresponding improvement/change? Evidence to justify this stage can be theoretical and/or based on existing empirical evidence; however, the logic of an association between the proposed mechanism and the clinical outcome would need to be established.
There must also be evidence that a proposed magnitude of change in the target mechanism would conceivably produce change in the clinical outcome. Evidence is needed that scores of a certain measure/task/etc. are stable in conditions when change is not occurring and reliably reflect change when change is occurring. Existing evidence about the target mechanism and its measurement may not have been collected to demonstrate sensitivity to change. Thus, issues such as the reliable and precise measurement of this target mechanism, issues of carry-over effects with repeated measurements, and the magnitude of target change needed to produce noticeable and valuable clinical outcome change are all considerations that researchers may need to incorporate in this argument.
Even more challenging circumstances apply if the researcher has defined complex mechanistic pathways. There can be multiple pathways concurrently impacting a clinical outcome (that perhaps interact with/synergize one another) and/or there may be sequential mediation (multiple mechanistic steps, or mechanisms for the mechanism). The change trajectory of the target mechanism is important to measure and may inform the schedule of assessments (e.g., ensuring the timing of assessments adequately captures nonlinear targets). Aspects of the target mechanism may change over time necessitating measurement strategies that can accommodate and capture such shifts. Response shift is a term used to capture the change in meaning in a patient-reported outcome over time (Sawatzky et al., 2017); similar considerations may apply to target measurement. Multiple measures may be needed to assess these diverse target mechanisms, which raises psychometric issues that need to be considered when comparing effects on multiple mechanisms (e.g., differences in reliability, validity, error variance, differential performance/difficulty at levels of pathology). These considerations all impact the type of evidence needed to support the validity of interpretations from target measurements.
Evidence to Advance Understanding of the Target Mechanism
An additional component of the validity argument is operationalizing the assumptions about how this proposed mechanism functions. Table 1 lists considerations for evaluating mechanistic change. What are the conditions via which this mechanism is proposed to impact this clinical outcome? The answer(s) to this question have implications for how (and when) the process is measured and perhaps, the strategy for intervention.
Table 1.
Considerations for Target Measurement
What is the evidence and argument that the proposed target could impact the clinical outcome? |
What degree of change in the target mechanism would plausibly change the clinical outcome? |
— What is the stability/typical variation of the mechanism when not being targeted for change? |
— What is the schedule of assessments of the target mechanism that would not produce carry-over effects? |
— What is the schedule of assessments of the target mechanism that would not produce undue burden? |
What is the rationale and evidence for the degree of change in the target mechanism needed? |
What are the boundary conditions of the target mechanism? |
— What are the relevant contexts for the measurement of this mechanism? |
○ Within the classroom? |
○ Randomly throughout one’s daily routine? |
○ Triggered by an event or experience? |
— What are the conditions under which the dysfunction in this mechanism is likely to emerge (i.e., what should trigger a measurement)? |
○ Environmental cues (e.g., smoking cues)? |
○ Events (e.g., evaluative tasks, meal times, time)? |
○ Experiences (e.g., states of fear)? |
— Who or what are the appropriate data sources? |
○ Self-report? |
○ Known observer? |
○ Trained observer? |
○ Biosensors? |
○ Global Positioning System? |
Note. Questions to consider when defining a novel target mechanism
For example, is this a feature that could be measured in “cold” conditions or at rest? Is this something that an individual can reliably and accurately report on, or is this a biomarker that can be measured via sensors? Would an observer further add to the validity of what is observed, or would this alter the nature of what is being measured? Is this a process that emerges in certain circumstances (e.g., interpersonal situations but not academic performance situations)? How would one prioritize the measures if there are several, and how does one determine which is the primary target measure? While certain levels of measurement would seem to be immune to some of these considerations, probing the conditions in which a mechanism exerts its effect may add greater precision and potency to the process being measured and perhaps the resulting treatment effect.
Consider this hypothetical example. Imagine that a researcher had robust evidence, both from prior literature and from work performed in their own laboratory, that inhibitory control as measured via a computerized stop-signal task was impaired in individuals with bulimia nervosa (Wu et al., 2013). The researchers had further evidence that this deficit in motor inhibition as indexed by the stop-signal task was associated with the frequency and intensity (i.e., number of calories consumed) of binge eating. They proposed that improvement in motor inhibition had the potential to improve binge eating. The researcher strengthened this empirical evidence with a thoughtful theoretical argument of cue reactivity theory in the context of food (Jansen, 1998) and how weakened inhibitory control may make one vulnerable to succumb to food in response to powerful food cues, even when not in an energy deficit state. The researcher then planned to use their findings and this body of prior research as the basis for establishing a mechanistic intervention proposed to engage the target of inhibitory control with the ultimate intention of reducing binge eating frequency. They proposed using change in performance on the stop-signal task as an index of target engagement. With the conceptual and empirical association between these two features established, some additional considerations that could inform the measurement strategy are: What is the variability in this capacity/target mechanism when an individual is in an unchanging state (i.e., stability of performance on the stop-signal task prior to intervention when binge eating is not changing)? What is the justification for the degree of change in inhibitory control that is proposed to impact binge eating? Are there further opportunities to better understand this mechanism? For example, would measuring this capacity under different conditions strengthen our understanding of the mechanism and perhaps the sensitivity of our ability to measure change (e.g., inhibitory control in emotional contexts, food contexts)? How frequently can one measure this capacity without carryover (learning) effects (Langenecker et al., 2007)? Armed with this information, the researcher could then make a logical case for the degree of change that would indeed be valid change, with the amount of change resulting in clinically meaningful change (i.e., noticeable and valuable change in the clinical outcome) being the focus of the next stage of the investigation (Table 1).
There are also opportunities for novel measurement development/deployment that may further substantiate the researcher’s argument. Continuing with the same example, imagine the researcher learns that in states of elevated negative affect, inhibitory control becomes even more impaired as measured using an ambulatory go/no-go task employed within an ecological momentary assessment protocol (Smith et al., 2020). The researchers question whether the evidence base is robust enough to make that ambulatory task the primary outcome. Collecting those data during the course of the intervention may further fuel hypotheses about how the mechanism operates while using the context of the trial to further develop a valid measurement strategy. Thus, much like pilot items are embedded into standardized tests of achievement that are not used in the calculation of a student’s score, incorporating additional measurement strategies of a target may be an opportunity to probe the limits of and understand more about a mechanism.
Processes of Target Selection Example 1: Working “Backward” from A Clinical Endpoint
In this next section, we provide some hypothetical processes for target definition. There are many potential starting points to aid in the design of interventions intended to engage a target mechanism. As one example, we start with the clinical outcome for which researchers are interested in developing an effective intervention strategy. They are trying to work through putative mechanisms of symptom expression to define a meaningful target mechanism and to help in the design of the intervention to address that target (Figure 1). Figure 1 illustrates one hypothetical pipeline using a clinical endpoint. Table 2 presents additional examples and thought exercises to define mechanistic targets given a clinical endpoint.
Figure 1. An Example Pipeline of Target Mechanism Identification and Links to Clinical Symptom Improvement.
Note. Colored gears symbolize potential target mechanisms. The symbol and color of a target mechanism (gear) correspond with the clinical symptom that is proposed to improve by targeting that mechanism. So the green gear with the lightning bolts is proposed to improve the clinical symptom of green lightning bolts. The orange gear (target mechanism) with a plus sign is proposed to improve the clinical symptom of orange plus signs. When these are the same symbol and color (green lightning bolt gear, green lightning bolt symptom) the target mechanism is valid (i.e., it impacts the clinical symptom). When there are different colors and symbols between target and symptom, a different target mechanism is proposed to contribute to that clinical symptom. In that case, there is the need for further investigation to probe the contributions to treatment response heterogeneity (Sheeran et al., 2017). While these steps are depicted in a linear fashion for ease of understanding, the process is likely iterative as knowledge gained about the mechanism informs assessment and treatment strategies.
1,2. From among a variety of potential mechanisms, the researchers select a target mechanism hypothesized to result in meaningful clinical change in symptoms or syndromes if engaged.
3. The assessment strategy for the mechanism is derived from the assumptions about how the mechanism operates.
4–6. Individuals for whom the mechanism was targeted (wrench/treatment engaging gear/target mechanism) and valid for a given clinical symptom –(orange plus gear/orange plus symptom) improve. Symptoms associated with the non-targeted mechanism (green gear) do not improve. Individuals whose clinical symptom was driven by a different mechanism (green lightning gear/orange plus symptom) also do not improve.
An interesting exploration is if symptom relief from the targeted symptom (orange plus sign) is sufficient to impact functioning and quality of life, thereby designating an intervention successful for the syndrome. However, for those with only the non-targeted mechanism/symptom (green lightning), a novel intervention targeting the non-targeted mechanism/symptom (green lightning) is needed. In another case, the individual with a lack of correspondence between mechanism and symptom (green mechanism/orange symptom) has a different mechanism that contributes to the symptom: the targeted (orange) mechanism lacks validity for that individual and they are treatment non-responders.
Table 2.
Questions that may guide mechanistic hypotheses
What is the function of the symptom? | Considerations |
---|---|
Are there discernible reinforcement contingencies that reliably predict an increase in the frequency of a target clinical symptom? For example, is binge eating (a symptom) reliably predicted by negative affect of any form? What consequences follow performance of a behavioral symptom? Does the uncertainty about whether or not to eat the next bite of a “forbidden” food during a binge get translated as an error signal with related changes in dopamine neurotransmission? |
A target consideration and resulting treatment consideration could be addressing the function of that symptom with the assumption that a symptom would decrease if its function is being addressed by more adaptive and effective means. |
What capacities need to be strengthened? | Considerations |
A symptom may be hypothesized to result from a relative weakness in certain affective/cognitive/behavioral capacities. For example, impulsivity may result from weakened abilities to tolerate distress. Thus, strengthening distress tolerance will reduce impulsive behavior. |
Multiple clinical symptom targets may be justified if they share a common proposed mechanism (e.g., behaviors that are justified to be impulsive). |
How can a skill asset be repurposed to be more adaptive? | Considerations |
Rather than a relative weakness, some symptoms reflect an excess of a feature, a behavior that may have adaptive elements (e.g., the drive in anorexia nervosa, perseverative interests in autism) Manipulating this feature may result in a reduction in other features. For example, perseverative interests may be utilized as a foundation for reciprocal social interactions in individuals with autism. |
The symptoms that constitute a diagnostic syndrome may have diverse developmental courses, with different symptoms influencing the expression and/or maintenance of the others. |
Viewing a clinical symptom from a behavioral perspective, one could consider the reinforcement contingencies that increase the emergence of that symptom and the consequences that ensue when that symptom manifests. This strategy of target identification and definition may work well for behavioral symptoms of psychiatric disorders but less well for psychiatric symptoms that cannot be described as concrete events (e.g., anhedonia).
For example, imagine a researcher is interested in targeting social avoidance (the clinical outcome) in individuals with social anxiety disorder. They have convincing data that individuals who are non-responders to the current standard of care have exaggerated vasovagal responses to a variety of cue elicitors, both social and non-social (Kara & Doğan, 2021). The researchers hypothesize two pathways to social avoidance—the fear of negative evaluation resulting from this vasovagal response and/or the threat of harm from vasovagal syncope (Owens et al., 2018). Thus, avoidance is maintained by minimizing threat either due to fear of the negative consequences of fainting and/or the negative evaluations of others. Imagine the researchers have an orthostatic intervention shown to improve sympathetic outflow, a change in blood pressure recovery that improves the vasovagal response (Tao et al., 2019). Thus, a comparison of relevant mechanisms for social avoidance could be to determine whether improving cardiovascular function is sufficient to decrease social avoidance versus an intervention that focuses on a cognitive process and employs cognitive restructuring of perceived negative evaluations. In this example, the researcher may wish to fully articulate the cognitive mechanism so that the comparison of both arms results in new information about the clinical outcome and cognitive mechanism. In terms of screening for participation, the researchers could consider screening for elevations on the clinical outcome of social avoidance, the target mechanism of elevated vasovagal syncope, and the comparative target of fear of negative evaluation.
An alternative pathway may be to consider strengthening a capacity that would result in symptom improvement. For example, imagine an individual with schizophrenia who has prominent avolition. This individual may have a core cognitive mechanism of “defeatist performance beliefs” driving their limited goal-directed activity (e.g., beliefs such as, “Why should I bother to try? I will just fail”) (Grant & Beck, 2009). However, this mechanism may not be one that can be impacted without first getting the individual behaviorally activated and interacting with the environment to facilitate opportunities for countering those beliefs. Indeed, cognitive restructuring strategies to address defeatist beliefs may not be functionally meaningful unless they occur in the context of behavioral activation (Grant & Beck, 2009). The intervention may therefore need to target behavioral activation and evaluate changes in both behavior and defeatist performance beliefs as mechanisms underlying improvement in avolition. Table 3 highlights an in-depth example of how these considerations were adjudicated using the clinical outcome of intrusive traumatic memories.
Table 3.
Extended Example Using Reduction of Intrusive Traumatic Memories (Iyadurai et al., 2018;James et al., 2015;Kanstrup, Singh, et al., 2021)
Clinical Outcome: Reduce Frequency of Intrusive Traumatic Memories (Singh et al., 2020a) |
|
Mechanistic Target: Limit memory consolidation or re-consolidation (James et al., 2015) |
|
Intervention Strategy: Tax working memory with imagery after cued recall of intrusive traumatic image |
|
Theory and Evidence Supporting the Target • Intrusive memories are proposed to result, in part, from facilitated memory encoding, particularly of the affective and sensory features of prior stressful events, via potentiated amygdala activation after a stressful/traumatic event combined with weakened hippocampal activity down-modulating contextual binding (Bisby & Burgess, 2017; Bisby et al., 2020; Sierk et al., 2019) : thus salience is enhanced while context is suppressed. • According to Baddeley’s working memory theory, there is limited processing capacity for information of the same type (e.g., visual imagery) at the same time (Baddeley, 2003). • There is evidence that intrusive memories after trauma take the form of mental images (Holmes et al., 2005; Holmes & Mathews, 2005). • Following a stressful life event, consolidation of that memory occurs within hours of the event, providing an important, but time-limited opportunity to influence initial consolidation (McGaugh, 2000). • Reconsolidation refers to the process of reactivating a previously consolidated memory, a process that renders the memory malleable with re-stabilization needed for memory persistence (Nader et al., 2000). • There is evidence that tasks designed to interfere with consolidation or re-consolidation of an intrusive memory can impact its (re)occurrence, i.e., the number of times the memory intrudes to mind (intrusive recall). Thus, these tasks may reduce the frequency of intrusive traumatic memories (Lau-Zhu et al., 2019). An example of a domain-specific imagery task that may interfere with consolidation of an intrusive memory is playing the video game Tetris using mental rotation instructions (Iyadurai et al., 2018; James et al., 2015; Kanstrup, Singh, et al., 2021; Lau-Zhu et al., 2019). |
|
Measurement Strategy and Assumptions of Target: Limit Memory (Re)consolidation Following
• Traumatic Event (Proof of Concept)(James et al., 2015) • The traumatic event is remembered • Intrusive memories of the traumatic event are experienced as visual images • Individuals are aware of these memories as they occur and can record them in a daily diary (primary outcome, so an assumption of the clinical assessment strategy). • Prior hotspot moments in traumatic memory can be evoked via a cue. ○ For recent memory, that cue allows orientation to hotspots in memory. ○ For older memories, there is a necessary time-window following cueing of a traumatic memory to allow time for reconsolidation processes to be initiated. • A taxing domain-congruent task needs to follow the cue for a traumatic memory hotspot. Engagement in cognitive tasks needs to occur following this time window for reconsolidation processes to be initiated. • To isolate these assumptions, the following control conditions were designed: ○ No Task Control ○ Reactivation of Traumatic Memory Only ○ Visually Taxing Working Memory Only (Tetris game) ○ Reactivation plus Visually Taxing Working Memory • Results indicated, as predicted, that only the combination of visual working memory plus cued traumatic memory significantly reduced the number of subsequent intrusive memories over time. |
|
Logic Underlying Intervention
Taxing working memory with a domain-congruent task prevents reconsolidation of a traumatic memory in terms of its sensori-emotional aspects |
|
Initial Pilot Testing (Iyadurai et al., 2018) This intervention was translated to an emergency department setting and provided to individuals within 6 hours of an automobile accident who were lucid and had memory of the event. A similar pattern of findings was found in a separate emergency room application (Kanstrup, Singh, et al., 2021). Detailed single case studies work further support target engagement via this intervention strategy (Iyadurai et al., 2020; Kanstrup, Kontio, et al., 2021; Kessler et al., 2018) |
Another consideration when starting target exploration from the outcome part of the equation is focusing on the impact of a symptom rather than the symptom itself. For example, in the context of complex psychosomatic disorders for which a well-defined and treatable disease process has not been identified (e.g., functional abdominal pain), and given evidence that individuals with similar diagnostic severity evidence varying degrees of functional impairment, a researcher can begin to formulate mechanistic hypotheses about affective, cognitive, or behavioral factors that could be leveraged to improve functional impairment rather than facilitate symptom reduction.
Processes of Target Selection Example 2: Starting with a Mechanistic Process/Basic Clinical Finding
Researchers may start with a mechanistic target that may be relevant to a specific symptom/syndrome and then create or identify an intervention designed to impact that target mechanism. For example, imagine a schizophrenia researcher who believes that anhedonia results from an impairment in anticipatory pleasure but not consummatory pleasure. Anticipatory pleasure relates to the predictions one makes about future rewards and the enjoyment one derives from that anticipation whereas consummatory pleasure is in-the-moment enjoyment of a reward (Wu et al., 2017). Through neuroimaging evidence that is replicated, they document that impaired anticipatory pleasure is associated with reduced activation of the ventral striatum, and demonstrate intact responses in this region during reward receipt (i.e., consummatory pleasure), thereby showing a dissociation in a neurophysiological mechanism hypothesized to be selectively related to the symptom of interest. This scientist may not have a background in intervention research and may thus need to either identify an intervention in the literature that impacts this target mechanism or develop one in collaboration with an interventionist. The scientist would then determine whether the intervention impacted the mechanism (ventral striatum activation when anticipating rewards) and if improvements in anhedonia were mediated by the magnitude of change in the ventral striatum from pre- to post-treatment.
Alternatively, a mechanism might hold transdiagnostic relevance for several psychiatric syndromes. For example, emotion regulation abnormalities are core to many psychiatric disorders and are associated with distress and impaired functional outcomes (e.g., vocational, social) (Fernandez et al., 2016). Many potential mechanisms could be viable targets for emotion regulation, depending on the stage of emotion regulation focused on (Gross, 1998; Gross, 2015) and how the construct is defined. For example, over-identification of the need to regulate could be associated with reduced autonomic activation, or dysfunctional emotion-attention interactions could lead to heightened emotional awareness; impaired implementation of explicit strategies to down-regulate arousal may result from diminished effort or reduced prefrontal inhibitory control over the amygdala.
How would one go about deciding how to operationalize the construct of emotion regulation (e.g., which stage to focus on; implicit versus explicit) (Gyurak et al., 2011)? Once operationalized, which mechanisms would be the most relevant to focus on for enhancing a specific clinical outcome (e.g., social functioning)? These decisions may be driven by the strength of evidence tying an abnormality (e.g., decreased prefrontal cortical-subcortical functional connectivity, inefficient executive control network functioning) in one aspect of the construct (e.g., the implementation stage of emotion regulation) to the clinical outcome of interest in comparison to other potential mechanisms (e.g., effort allocated during implementation). The magnitude and replicability of linkage between the clinical outcome and mechanism should be critical in this decision-making process, as well as psychometric considerations (e.g., reliability of measurement). After a mechanism has been identified (e.g., decreased prefrontal cortical-subcortical functional connectivity, inefficient executive control network functioning), the scientist might then identify an intervention that already exists within the literature that has been shown to augment this mechanism (e.g., an emotional working memory cognitive training program that increases prefrontal activation, executive control network reconfiguration) (Schweizer et al., 2013) that could hold transdiagnostic relevance for enhancing social functioning in numerous disorders (e.g., Stevens et al., 2016).
Process of Target Selection Example 3: Integrating Patient, Caregiver, and Provider Observations and Insights
“For we found, to our great surprise at first, that each individual hysterical symptom immediately and permanently disappeared when we had succeeded in bringing clearly to light the memory of the event by which it was provoked and in arousing accompanying affect, and when the patient had described the event in the greatest possible detail and had put the affect into words. Recollection without affect almost invariably produces no result.”
Careful observations of patient experiences have been an influential source of novel hypothesis generation throughout the history of psychological study. Patient and caregiver observations and insights are increasingly valued as important resources for improving the efficacy of therapeutic strategies. Much of this emphasis has been on targeting clinical endpoints that are meaningful to patients; yet, descriptions and observations of how patients attempt to manage symptoms or daily life challenges may aid in novel target identification. For example, individuals who have remitted from a psychiatric diagnosis without formal treatment may be a vital information source (Whiteford et al., 2013). Unpacking the sources of information individuals draw from to discern their experience of psychiatric symptoms may be quite revealing, particularly when there is a discrepancy between sources (e.g., patient and provider). The processes and information sources individuals draw from to discern symptom experience may spur novel insights into potential target definitions. Observations from informal caregivers (e.g., “what do you consider most important to address in a depressed person?”, Chevance et al., 2020) are often overlooked hypothesis-generating sources. Although many mechanisms may not be consciously knowable, discussions of why or how a specific clinical symptom persists can help focus conversations with different constituents leading to potentially valuable information for target exploration.
Choosing, Modifying, or Designing an Intervention
The next challenge is finding, modifying, or designing an intervention that has been shown to engage the target mechanism or could plausibly engage the target based on existing evidence. As noted earlier, the design of target-focused intervention strategies is intended to result in more parsimonious treatment design. Starting with the mechanism to discover the treatment may be the most straightforward pathway. However, an alternative approach is to focus on treatment non-response within a clinical group to guide intervention development. Consider a group of patients diagnosed with schizophrenia who fail to respond to social skills training designed to improve poor social functioning. One might hypothesize that they fail to respond because the target was not valid for that group. In other words, while the non-responders demonstrated poor social skills at screening and demonstrated improvement in social skills (i.e., they demonstrated target engagement) they did not demonstrate improvement in social functioning—improved social skills did not drive a change in social functioning.
In terms of decision-making, the researchers could entertain several hypotheses. For example, they could conclude that this group needed stronger target engagement—a longer dose of treatment to further enhance social skills. However, they ruled out this hypothesis as they had benchmarked their target against functional endpoints that could be observed and coded (e.g., starting and maintaining conversations). Alternatively, they could hypothesize that the target mechanism was invalid because their poor social functioning was not driven by poor social skills, it was driven by another mechanism— “asocial beliefs” that limited motivation to perform social skills, something that was learned through exploration of treatment non-response (Grant & Beck, 2010). It could be that social skills improvement is a separate pathway to social engagement, that an alternative parallel pathway (e.g., asocial beliefs) is needed to strengthen clinical improvement, or that these mechanisms would function synergistically to lead to greater benefits in social functioning than social skills training alone. An intervention that combines the two could be used to strengthen the effects of either treatment alone, as done in Eric Granholm’s Cognitive Behavioral Social Skills Training (CBSST; Granholm et al., 2014). One key consideration would be understanding how (or if) these mechanisms relate to each other and to the designated outcome. For example, one could define mechanisms related to each component of the intervention (e.g., facial affect perception vs. asocial beliefs) and track whether some individuals demonstrate clinical improvement due to changes in the mechanism underlying one of the intervention components or if changes in both are required for gains to occur.
Another challenging strategy would be trying to discern the mechanism of a multi-component intervention that has been effective for a subset of individuals with a given psychiatric diagnosis. Given the previous discussion, much preliminary work may be needed to parse the intervention, isolate the mechanism(s) for certain components, and propose linkages to certain symptoms within the psychiatric syndrome. Still, treatments that were designed with a purported mechanism offer an interesting potential opportunity. Take, for example, Eye Movement Desensitization and Reprocessing (EMDR), a treatment that has shown efficacy in reducing symptoms of posttraumatic stress disorder across numerous randomized controlled trials (for review, see Chen et al., 2014). Patients respond to bilateral sensory stimulation (e.g., visual, auditory, tactile) while holding in mind aspects of a targeted memory and the somatic and emotional experiences that accompany that memory. Several neurobiological theories have been proposed as mechanisms of action. Examples include that EMDR taxes working memory, particularly the visuospatial sketchpad, and allows for re-consolidation of less affectively vivid and more cognitively integrated memories (Lilley et al., 2009; van Veen et al., 2016); or that EMDR enhances inter-hemispheric communication, (Samara et al., 2011); or that EMDR evokes an orienting response with accompanying de-arousal—hypotheses that all have some empirical or theoretical evidence as being potential therapeutic mechanisms.
However, as noted by Calancie et al. (2018), the essence of the intervention, a.k.a. the nature and temporal dynamics of eye movements (e.g., smooth pursuit movements, reactive motor movements), have not been systematically studied during treatment. In fact, there is evidence that the type and/or speed of oculomotor dynamics recruit distinct brain networks or structures that may differentially facilitate or impede therapeutic effects (Calancie et al., 2018). Thus, an experimental therapeutics approach might focus on understanding, manipulating, and measuring the acute and sustained effects of different oculomotor dynamics elicited during treatment. More systematic study of a parameter of the intervention that can be manipulated, oculomotor characteristics, may potentiate the intervention or increase efficacy for non-responders while helping to adjudicate or integrate other neurobiological hypotheses regarding the underlying mechanisms that lead EMDR to have efficacy. If the mechanism of this treatment were to be (further) elucidated, the aspect of the intervention that drove that change in mechanism defined, and the change in mechanism linked to a clinical outcome, then the intervention strategy would be strengthened for individuals with elevations on that mechanism and clinical outcome.
Existing interventions offer a multitude of opportunities for novel intervention development. Careful coding of moment-to-moment process variables within a therapy session or within a therapeutic conversation offers other fruitful data sources to explore change processes. Dismantling analyses or Multiphase Optimization Strategy (MOST) designs (Collins, 2018)—techniques that probe the incremental effects of various intervention components—may be more suitable for matching intervention strategies with a given target mechanism (e.g., Ashford et al., 2010). Tailoring or adapting effective therapeutic techniques for certain developmental phases (e.g., for young children) (Zucker et al., 2017) or environmental contexts provide other rich opportunities.
Some researchers have chosen a task that probes a target and added training elements to use as an intervention strategy (Haller et al., 2021). These interventions (e.g., cognitive training) may target a specific mechanism (e.g., working memory and prefrontal activation) that can be measured via the same outcome delivered in the intervention (e.g., the same working memory task). Using a near and far transfer design (Barnett & Ceci, 2002) may allow researchers to determine whether a generalized benefit has occurred. In addition to successful intervention delivery of a target working memory task, the researchers would administer a second, previously untrained working memory test to see if intervention effects generalize to this related, but novel task.
The choice of control condition may offer another interesting opportunity. Although potentially not feasible in the initial intervention development phase of research, the control condition may provide additional opportunities for hypothesis testing the viability and strength of both a proposed mechanism and a putative alternative. A trial is designed as a proof-of-concept for moving a proposed mechanism, whereas a control condition could provide further support for that mechanism by providing a counterpoint or an opportunity to test a competing proposed mechanism. A control group can also provide a further test of the validity of a chosen target. For example, natural/non-experimental change in a valid target should be associated with change in a clinical endpoint, while natural variation of target change may help to explain outcome variability.
This section offered diverse thought exercises to guide mechanistic intervention development. The focus on a target mechanism may streamline such development, a process that may offer some creative opportunities for intervention developers as they are freed from the challenge of figuring out what to target and can focus instead on how to effectively intervene on a chosen mechanism.
Forming Your Team
The implementation of interventions from the diverse pathways described above will likely require many types of methodological and theoretical expertise that would be uncommon to exist in a single investigator. By its very nature, the experimental therapeutics approach is one that calls for significant collaboration. This is particularly true of early phase intervention development when it is conducted in response to funding opportunity announcements that focus on the inclusion of a novel target/mechanism or novel approach to engaging a known target/mechanism (Hall et al., 2018; Wuchty et al., 2007). It is increasingly recognized that team science approaches are necessary to solve the most complex health issues, including issues related to psychological and behavioral health challenges (Antonucci, 2015; Hall et al., 2018). Research teams responding to this funding opportunity announcement may include collaborators from different personal and cultural experiences and likely different disciplines. For example, a team may include basic behavioral scientists or neuroscientists with expertise in electroencephalogram/event-related potential, or neuroimaging; computer programming and/or computational modeling; or clinicians with varying levels of clinical trial experience or with varying familiarity with defining clinical observations so that they can be integrated into a computational framework. The team may also need to include an experienced clinical trialist to inform and direct the intervention aspects of the project. Team science in its broadest form also includes community members, patient advocates, patients, caregivers, and practitioners to participate in the goals of the project and its eventual dissemination and evaluation (Tebes & Thai, 2018). Optimally, community members will be brought into the team early. It is becoming more common to establish standing and ad-hoc community advisory boards who can consult with research teams. See the Health Equity Resources and Outreach Programs at the University of California, Davis, Clinical and Translational Science Center for an example (University of California, 2022). Note that a functioning team can take months or years to coalesce while they learn to develop a shared understanding and language.
Although team science is increasingly common, challenges associated with successful collaborations are also increasingly recognized (Cooke & Hilton, 2015; Ledford, 2015), as issues with communication, particularly due to differences in content knowledge, interpersonal practices, and concerns regarding credit can arise and hamper progress. The most innovative research may be derived from “transdisciplinary” work teams. These teams go beyond interdisciplinary and multidisciplinary collaborations, to the point that group members are able to generate complex, novel conceptual models by integrating their own knowledge, theory, experience, and methods (Nurius & Kemp, 2019)—a process that may contribute to a sense of joint ownership of and motivation toward the project.
Many resources are available to assist scientists in team science endeavors. The funding opportunity announcement encourages leveraging resources and infrastructure provided by institutions that have Clinical and Translational Science Awards (CTSAs), and the majority of institutions that have CTSAs should have resources in community engagement hubs to assist projects. In addition, CTSAs should have resources to facilitate team science endeavors, including training activities. The updated NIH Collaboration and Team Science: A Field Guide (Bennett et al., 2019) has great practical content, including information on constructing “Prenuptial Agreements for Scientists.” The University of Minnesota “Surviving Group Projects” website features entertaining videos, exercises, and useful documents to facilitate planning and implementing team science (University of Minnesota). In addition, the CRediT (Contributor Roles Taxonomy) is a resource to help think through how different roles contribute to a project and the associated scholarly output (The Casrai Project). Finally, Northwestern University, with funding from NCATS/NIH, provides a web-based tool with exercises to help develop, implement, and evaluate team science practices under the “COALESCE-CTSA Online Assistance for Leveraging the Science of Collaborative Effort” project (Northwestern University). It is likely that team science resources will evolve over time to further accelerate innovation. Investigators may wish to consider these team science issues when forming plans about their project and revisit them after the project is funded.
Decision-Making in an Experimental Therapeutics Framework
Toward the end of the target engagement phase of intervention deployment, the researchers must demonstrate a level of predefined target engagement that justifies further testing of the intervention strategy. The schedule of data collection may permit determinations about the rate of change in the target mechanism, information that may allow for adjustments in the treatment dose necessary for target engagement and, potentially, for clinical outcome improvement. The investigator has designed a set of experiments to determine whether a target is movable; whether a predefined level of target engagement and the boundaries around that target engagement (e.g., standard error of the measurement strategy) demonstrate change. Embedded in these experiments may be competitive target engagement (e.g., comparing alternative strategies intended to target the same mechanism) (Sheeran et al., 2017) or comparative target engagement (e.g., hypothesizing that distinct mediators may contribute to an outcome and seeing which is more movable). If all goes as predicted, then decision-making may support further testing of the intervention.
Yet, there may be alternative outcomes that advance understanding of relevant target mechanisms and their treatment, that, while not as straightforward, may justify further testing. For example, what if the investigator had proposed a primary target that was not movable as predicted but had an alternative hypothesis about a second mechanism that did indeed show change? Given the evidence needed to establish the viability of the target, this alternative mechanism was not fleshed out in the original grant but was mentioned in an alternative strategies section of the grant. Hopefully, the justification and measurement strategy for this alternative was as rigorous as the primary target, and the investigators present this alternative outcome for internal review. One challenge with this outcome is that the original target was not tested—the intervention chosen may not have been optimal to move the target, but the target may have actually been “correct.”
Similarly, what if an intervention impacts more than one target? The schedule of data collection can hopefully inform how these multiple mechanisms function (e.g., independently, synergistically, one as a mediated mechanism of the other). Treatments that are demonstrated to reduce the targeted clinical symptom, but without sufficient evidence of target engagement, offer another interesting dilemma. More concentrated study of alternative mechanisms may be warranted. In the case in which there was target engagement but not clinical change, the researcher has to make determinations about whether the target was invalid for that clinical outcome or whether the hypothesized degree of target engagement was incorrect and a stronger treatment dose is needed (Haller et al., 2021). There are, of course, innumerable permutations related to the resulting outcome data. Strategies to strengthen the researchers’ position are to anticipate threats to the validity argument presented about their purported target mechanism and to demonstrate how those threats were addressed (Weinfurt, 2021).
The Practical Challenges of the Experimental Therapeutics Initiative of the NIMH
Two years goes by quickly in the context of the fast-fail approach. In that time interval, researchers must prepare processes needed to conduct a clinical trial, run the clinical trial, implement an assessment schedule that allows for decision-making regarding treatment dose and target mechanism responses, analyze outcome data, and prepare a progress report that justifies decision-making for the next phase of intervention testing. Not only is this a considerable amount to accomplish in a short period of time, but the decision-making process can be complicated. While decisions are being decided about the next phase of development, personnel from the project may need to be funded through other mechanisms, and researchers may risk losing valuable trained staff. During the workshop, several investigators discussed economic shortfalls during this transition, and several expressed the need for more time to accomplish these goals. Projects that include novel computer programming/coding can also be challenging given this timeframe. Some in the workshop described timelines for software development that were not feasible. Although no concrete recommendations were made, the need for time was a common theme.
Reflections and Summary
This paper reflects on some of the issues and considerations that arose during a workshop sponsored by the NIMH in February of 2020, Novel Target Discovery and Psychosocial Intervention Development Workshop (National Institute of Mental Health, 2020), and on the conversations that workshop inspired. It is notable that there were many back-and-forth discussions and debates during the writing of this manuscript, reflecting the complexity of working through the conceptual and logistical challenges of an experimental therapeutics approach. This paper is by no means meant to reflect an adjudication of these issues, but rather to serve as a thought-provoking exercise to get researchers talking about and advancing research and intervention designs. There are many unmet challenges that this paper has not addressed but which may guide future work. For example, When in the intervention development pipeline should researchers consider the feasibility of dissemination? How can we incorporate the impact of an individual’s context (e.g., family, environment) on target definitions and intervention strategies? How should booster sessions be incorporated or tested? Examining these issues may advance efforts at designing interventions that integrate with people’s lived experiences., The experimental therapeutics framework is one that invites much creativity and collaborative brainstorming. It is an invigorating initiative that holds tremendous promise to advance theory and treatment for mental disorders.
Acknowledgements.
We wish to thank Emily Holmes, Professor in the Department of Psychology at Uppsala University, Uppsala, Sweden, for allowing us to use the work of her laboratory in one of our extended clinical examples. We also wish to acknowledge Pat Arean, Professor of Psychiatry and Behavioral Sciences at the University of Washington, Seattle; Gregory Lewis, Assistant Professor of Intelligent Systems Engineering at Indiana University; Joy Geng, Professor of Psychology at the Mind and Brain Institute at the University of California, Davis; and James Guevara, Professor of Pediatrics at the Children’s Hospital of Philadelphia for their thoughtful contributions throughout the workshop. We wish to thank Dr. Christopher Sarampote, Chief of the Biomarker and Interventions for Childhood Onset Disorders Branch of the National Institute of Mental Health; Dr. Alexander Talkovsky, Program Chief of the Anxiety Disorders and Mood Disorders Programs of the National Institute of Mental Health; Dr. Marjorie Garvey, Program Chief of Novel Strategies for Treatment of Developmental Psychopathology of the National Institute of Mental Health, and Dr. Ann Wagner, former National Autism Coordinator of the National Institutes of Health for their organization of the workshop and their thoughtful comments throughout the writing of this manuscript. We wish to thank Eric Monson for his expert consultation and creative edits to Figure 1. Finally, we wish to thank Jessica Schleider, Assistant Professor of Psychology at Stony Brook University, Gabriel S. Dichter, Professor of Psychiatry at the University of North Carolina, Chapel Hill, and our other anonymous reviewer for their constructive comments that greatly strengthened this submission.
Funding.
Dr. Zucker received funding from the National Science Foundation/National Institute of Mental Health (NIMH), R01-MH-122370 and NIMH R33-MH-097959; Dr. Strauss received funding from NIMH, R61-MH-121560; Dr. Schweitzer received funding from NIMH, 2 R01-MH-091068, R61-MH-110043, R33-MH-110043, and the National Institutes of Clinical and Translational Science (NACTS) Grant UL1 TR0011860; Dr. Scherf received funding from the NIMH, R61-MH-110624; Dr. Boyd received funding from NIMH, R61MH118405; Dr. Choi received funding from NIMH, R33-MH-111850. Dr. Brotman received funding from the National Institute of Mental Health Intramural Research Program, NCT02531893.
Footnotes
Disclosures
Financial Disclosure. The authors have no financial relationships relevant to this article to disclose.
Conflict of Interest. The authors have no conflict to declare.
References
- Antonucci TC (2015). Teams Do It Better! Research in Human Development, 12(3–4), 342–349. 10.1080/15427609.2015.1068035 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Anvari F, & Lakens D. (2021). Using anchor-based methods to determine the smallest effect size of interest. Journal of Experimental Social Psychology, 96, Article 104159. 10.1016/j.jesp.2021.104159 [DOI] [Google Scholar]
- Ashford S, Edmunds J, & French DP (2010). What is the best way to change self-efficacy to promote lifestyle and recreational physical activity? A systematic review with meta-analysis. British Journal of Health Psychology, 15(Pt 2), 265–288. 10.1348/135910709x461752 [DOI] [PubMed] [Google Scholar]
- Baddeley A. (2003). Working memory: looking back and looking forward. Nat Rev Neurosci, 4(10), 829–839. 10.1038/nrn1201 [DOI] [PubMed] [Google Scholar]
- Barnett SM, & Ceci SJ (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128(4), 612–637. 10.1037/0033-2909.128.4.612 [DOI] [PubMed] [Google Scholar]
- Bennett LM, Gadlin H, & Marchand C. (2019). Collaboration and Team Science: A Field Guide. National Cancer Institute, U.S. Department of Health & Human Services, National Institutes of Health. http://teamscience.nih.gov/ [Google Scholar]
- Bisby JA, & Burgess N. (2017). Differential effects of negative emotion on memory for items and associations, and their relationship to intrusive imagery. Curr Opin Behav Sci, 17, 124–132. 10.1016/j.cobeha.2017.07.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bisby JA, Burgess N, & Brewin CR (2020). Reduced Memory Coherence for Negative Events and Its Relationship to Posttraumatic Stress Disorder. Curr Dir Psychol Sci, 29(3), 267–272. 10.1177/0963721420917691 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Breuer J, & Freud S. (1957). Studies on Hysteria (Strachey J, Trans.; Strachey J, Freud A, Strachey A, & Tyson A, Eds.). Basic Books. [Google Scholar]
- Calancie OG, Khalid-Khan S, Booij L, & Munoz DP (2018). Eye movement desensitization and reprocessing as a treatment for PTSD: current neurobiological theories and a new hypothesis. Annals of the New York Academy of Sciences, 1426(1), 127–145. 10.1111/nyas.13882 [DOI] [PubMed] [Google Scholar]
- Chen YR, Hung KW, Tsai JC, Chu H, Chung MH, Chen SR, Liao YM, Ou KL, Chang YC, & Chou KR (2014). Efficacy of eye-movement desensitization and reprocessing for patients with posttraumatic-stress disorder: a meta-analysis of randomized controlled trials. PLoS One, 9(8), e103676. 10.1371/journal.pone.0103676 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chevance A, Ravaud P, Tomlinson A, Le Berre C, Teufer B, Touboul S, Fried EI, Gartlehner G, Cipriani A, & Tran VT (2020). Identifying outcomes for depression that matter to patients, informal caregivers, and health-care professionals: qualitative content analysis of a large international online survey. Lancet Psychiatry, 7(8), 692–702. 10.1016/s2215-0366(20)30191-7 [DOI] [PubMed] [Google Scholar]
- Collins LM (2018). Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST). Springer. [Google Scholar]
- Cooke NJ, & Hilton ML (2015). Enhancing the effectiveness of team science. In Enhancing the Effectiveness of Team Science. http://www.nap.edu/catalog.php?record_id=19007 [Google Scholar]
- Fernandez KC, Jazaieri H, & Gross JJ (2016). Emotion Regulation: A Transdiagnostic Perspective on a New RDoC Domain. Cognitive Therapy and Research, 40(3), 426–440. 10.1007/s10608-016-9772-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gordon J. (2017). An experimental therapeutic approach to psychosocial interventions. NIMH Director’s Messages https://www.nimh.nih.gov/about/director/messages/2017/an-experimental-therapeutic-approach-to-psychosocial-interventions [Google Scholar]
- Grabb MC, Cross AJ, Potter WZ, & McCracken JT (2016). Derisking Psychiatric Drug Development: The NIMH’s Fast Fail Program, A Novel Precompetitive Model. Journal of Clinical Psychopharmacology, 36(5), 419–421. 10.1097/jcp.0000000000000536 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Grabb MC, Hillefors M, & Potter WZ (2020). The NIMH ‘Fast-Fail Trials’ (FAST) Initiative: Rationale, Promise, and Progress. Pharmaceutical Medicine, 34(4), 233–245. 10.1007/s40290-020-00343-y [DOI] [PubMed] [Google Scholar]
- Granholm E, Holden J, Link PC, & McQuaid JR (2014). Randomized clinical trial of cognitive behavioral social skills training for schizophrenia: improvement in functioning and experiential negative symptoms. Journal of Consulting and Clinical Psychology, 82(6), 1173–1185. 10.1037/a0037098 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Grant PM, & Beck AT (2009). Defeatist beliefs as a mediator of cognitive impairment, negative symptoms, and functioning in schizophrenia. Schizophrenia Bulletin, 35(4), 798–806. 10.1093/schbul/sbn008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Grant PM, & Beck AT (2010). Asocial beliefs as predictors of asocial behavior in schizophrenia. Psychiatry Research, 177(1–2), 65–70. 10.1016/j.psychres.2010.01.005 [DOI] [PubMed] [Google Scholar]
- Gross JJ (1998). The Emerging Field of Emotion Regulation: An Integrative Review Review of General Psychology, 2, 271–299. [Google Scholar]
- Gross JJ (2015). The Extended Process Model of Emotion Regulation: Elaborations, Applications, and Future Directions. Psychological Inquiry, 26(1), 130–137. 10.1080/1047840X.2015.989751 [DOI] [Google Scholar]
- Gyurak A, Gross JJ, & Etkin A. (2011). Explicit and implicit emotion regulation: A dual-process framework. Cognition and Emotion, 25(3), 400–412. 10.1080/02699931.2010.544160 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hall KL, Vogel AL, Huang GC, Serrano KJ, Rice EL, Tsakraklides SP, & Fiore SM (2018). The science of team science: A review of the empirical evidence and research gaps on collaboration in science. American Psychologist, 73(4), 532–548. 10.1037/amp0000319 [DOI] [PubMed] [Google Scholar]
- Haller SP, Stoddard J, Botz-Zapp C, Clayton M, MacGillivray C, Perhamus G, Stiles K, Kircanski K, Penton-Voak IS, Bar-Haim Y, Munafò M, Towbin KE, & Brotman MA (2021). A Randomized Controlled Trial of Computerized Interpretation Bias Training for Disruptive Mood Dysregulation Disorder: A Fast-Fail Study. Journal of the American Academy of Child and Adolescent Psychiatry, 61(1), 37–45. 10.1016/j.jaac.2021.05.022 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hawkins M, Elsworth GR, Nolte S, & Osborne RH (2021). Validity arguments for patient-reported outcomes: justifying the intended interpretation and use of data. Journal of Patient-Reported Outcomes, 5(1), 64. 10.1186/s41687-021-00332-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- Holmes EA, Craske MG, & Graybiel AM (2014). Psychological treatments: A call for mental-health science. Nature, 511(7509), 287–289. 10.1038/511287a [DOI] [PubMed] [Google Scholar]
- Holmes EA, Grey N, & Young KA (2005). Intrusive images and “hotspots” of trauma memories in Posttraumatic Stress Disorder: an exploratory investigation of emotions and cognitive themes. J Behav Ther Exp Psychiatry, 36(1), 3–17. 10.1016/j.jbtep.2004.11.002 [DOI] [PubMed] [Google Scholar]
- Holmes EA, & Mathews A. (2005). Mental imagery and emotion: a special relationship? Emotion, 5(4), 489–497. 10.1037/1528-3542.5.4.489 [DOI] [PubMed] [Google Scholar]
- Insel TR (2012, June 12, 2012). Experimental Medicine. [Google Scholar]
- Insel TR (2014). The NIMH Research Domain Criteria (RDoC) Project: precision medicine for psychiatry. American Journal of Psychiatry, 171(4), 395–397. 10.1176/appi.ajp.2014.14020138 [DOI] [PubMed] [Google Scholar]
- Iyadurai L, Blackwell SE, Meiser-Stedman R, Watson PC, Bonsall MB, Geddes JR, Nobre AC, & Holmes EA (2018). Preventing intrusive memories after trauma via a brief intervention involving Tetris computer game play in the emergency department: a proof-of-concept randomized controlled trial. Mol Psychiatry, 23(3), 674–682. 10.1038/mp.2017.23 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Iyadurai L, Hales SA, Blackwell SE, Young K, & Holmes EA (2020). Targeting intrusive imagery using a competing task technique: a case study. Behav Cogn Psychother, 48(6), 739–744. 10.1017/s1352465820000296 [DOI] [PubMed] [Google Scholar]
- James EL, Bonsall MB, Hoppitt L, Tunbridge EM, Geddes JR, Milton AL, & Holmes EA (2015). Computer Game Play Reduces Intrusive Memories of Experimental Trauma via Reconsolidation-Update Mechanisms. Psychol Sci, 26(8), 1201–1215. 10.1177/0956797615583071 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jansen A. (1998). A learning model of binge eating: cue reactivity and cue exposure. Behaviour Research and Therapy, 36(3), 257–272. 10.1016/s0005-7967(98)00055-2 [DOI] [PubMed] [Google Scholar]
- Kane MT (2013). Validating the Interpretations and Uses of Test Scores [Review]. Journal of Educational Measurement, 50(1), 1–73. 10.1111/jedm.12000 [DOI] [Google Scholar]
- Kanstrup M, Kontio E, Geranmayeh A, Olofsdotter Lauri K, Moulds ML, & Holmes EA (2021). A single case series using visuospatial task interference to reduce the number of visual intrusive memories of trauma with refugees. Clin Psychol Psychother, 28(1), 109–123. 10.1002/cpp.2489 [DOI] [PubMed] [Google Scholar]
- Kanstrup M, Singh L, Göransson KE, Widoff J, Taylor RS, Gamble B, Iyadurai L, Moulds ML, & Holmes EA (2021). Reducing intrusive memories after trauma via a brief cognitive task intervention in the hospital emergency department: an exploratory pilot randomised controlled trial. Transl Psychiatry, 11(1), 30. 10.1038/s41398-020-01124-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kara A, & Doğan MT (2021). The Psychopathology, Depression, and Anxiety Levels of Children and Adolescents With Vasovagal Syncope: A Case-Control Study. Journal of Nervous and Mental Disease, 209(8), 547–551. 10.1097/nmd.0000000000001334 [DOI] [PubMed] [Google Scholar]
- Kessler H, Holmes EA, Blackwell SE, Schmidt AC, Schweer JM, Bücker A, Herpertz S, Axmacher N, & Kehyayan A. (2018). Reducing intrusive memories of trauma using a visuospatial interference intervention with inpatients with posttraumatic stress disorder (PTSD). J Consult Clin Psychol, 86(12), 1076–1090. 10.1037/ccp0000340 [DOI] [PubMed] [Google Scholar]
- Krystal AD, Pizzagalli DA, Mathew SJ, Sanacora G, Keefe R, Song A, Calabrese J, Goddard A, Goodman W, Lisanby SH, Smoski M, Weiner R, Iosifescu D, Nurnberger J Jr., Szabo S, Murrough J, Shekhar A, & Potter W. (2018). The first implementation of the NIMH FAST-FAIL approach to psychiatric drug development. Nature Reviews Drug Discovery, 18(1), 82–84. 10.1038/nrd.2018.222 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Krystal AD, Pizzagalli DA, Smoski M, Mathew SJ, Nurnberger J Jr., Lisanby SH, Iosifescu D, Murrough JW, Yang H, Weiner RD, Calabrese JR, Sanacora G, Hermes G, Keefe RSE, Song A, Goodman W, Szabo ST, Whitton AE, Gao K, & Potter WZ (2020). A randomized proof-of-mechanism trial applying the ‘fast-fail’ approach to evaluating κ-opioid antagonism as a treatment for anhedonia. Nature Medicine, 26(5), 760–768. 10.1038/s41591-020-0806-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Langenecker SA, Zubieta JK, Young EA, Akil H, & Nielson KA (2007). A task to manipulate attentional load, set-shifting, and inhibitory control: convergent validity and test-retest reliability of the Parametric Go/No-Go Test. Journal of Clinical and Experimental Neuropsychology, 29(8), 842–853. 10.1080/13803390601147611 [DOI] [PubMed] [Google Scholar]
- Lau-Zhu A, Henson RN, & Holmes EA (2019). Intrusive memories and voluntary memory of a trauma film: Differential effects of a cognitive interference task after encoding. J Exp Psychol Gen, 148(12), 2154–2180. 10.1037/xge0000598 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ledford H. (2015). How to solve the world’s biggest problems. Nature, 525(7569), 308–311. 10.1038/525308a [DOI] [PubMed] [Google Scholar]
- Lilley SA, Andrade J, Turpin G, Sabin-Farrell R, & Holmes EA (2009). Visuospatial working memory interference with recollections of trauma. British Journal of Clinical Psychology, 48(Pt 3), 309–321. [DOI] [PubMed] [Google Scholar]
- McGaugh JL (2000). Memory--a century of consolidation. Science, 287(5451), 248–251. 10.1126/science.287.5451.248 [DOI] [PubMed] [Google Scholar]
- Messick S. (1989). Meaning and Values in Test Validation: The Science and Ethics of Assessment. Educational Researcher 18, 5–11. [Google Scholar]
- Nader K, Schafe GE, & Le Doux JE (2000). Fear memories require protein synthesis in the amygdala for reconsolidation after retrieval. Nature, 406(6797), 722–726. 10.1038/35021052 [DOI] [PubMed] [Google Scholar]
- National Institute of Mental Health. (2015). Strategic Plan for Research (NIH Publication Number 15–6368). Bethesda, Maryland: National Institutes of Health [Google Scholar]
- National Institute of Mental Health. (2020). Novel Target Discovery and Psychosocial Intervention Development Workshop https://www.nimh.nih.gov/news/events/announcements/nimh-novel-target-discovery-and-psychosocial-intervention-development-workshop.shtml. https://www.nimh.nih.gov/news/events/announcements/nimh-novel-target-discovery-and-psychosocial-intervention-development-workshop.shtml [DOI] [PMC free article] [PubMed]
- Nielsen L, Riddle M, King JW, Aklin WM, Chen W, Clark D, Collier E, Czajkowski S, Esposito L, Ferrer R, Green P, Hunter C, Kehl K, King R, Onken L, Simmons JM, Stoeckel L, Stoney C, Tully L, & Weber W. (2018). The NIH Science of Behavior Change Program: Transforming the science through a focus on mechanisms of change. Behaviour Research and Therapy, 101, 3–11. 10.1016/j.brat.2017.07.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Northwestern University. COALESCE-CTSA Online Assistance for Leveraging the Science of Collaborative Effort. https://www.teamscience.net/
- Nurius PS, & Kemp SP (2019). Individual-level competencies for teams collaboration with cross-disciplinary researchers and stakeholders. In Hall KL, Vogle AL, & Croyle RT (Eds.), Strategies for Team Science Success: Handbook of Evidence-Based Principles for Cross-Disciplinary Science and Practical Lessons Learned from Health Researchers (pp. 171–188). Springer [Google Scholar]
- Owens AP, Low DA, Critchley HD, & Mathias CJ (2018). Emotional orienting during interoceptive threat in orthostatic intolerance: Dysautonomic contributions to psychological symptomatology in the postural tachycardia syndrome and vasovagal syncope. Autonomic Neuroscience, 212, 42–47. 10.1016/j.autneu.2018.01.004 [DOI] [PubMed] [Google Scholar]
- Paul SM, Mytelka DS, Dunwiddie CT, Persinger CC, Munos BH, Lindborg SR, & Schacht AL (2010). How to improve R&D productivity: the pharmaceutical industry’s grand challenge. Nature Reviews Drug Discovery, 9(3), 203–214. 10.1038/nrd3078 [DOI] [PubMed] [Google Scholar]
- Pizzagalli DA, Smoski M, Ang YS, Whitton AE, Sanacora G, Mathew SJ, Nurnberger J Jr., Lisanby SH, Iosifescu DV, Murrough JW, Yang H, Weiner RD, Calabrese JR, Goodman W, Potter WZ, & Krystal AD (2020). Selective kappa-opioid antagonism ameliorates anhedonic behavior: evidence from the Fast-fail Trial in Mood and Anxiety Spectrum Disorders (FAST-MAS). Neuropsychopharmacology, 45(10), 1656–1663. 10.1038/s41386-020-0738-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Samara Z, Elzinga BM, Slagter HA, & Nieuwenhuis S. (2011). Do Horizontal Saccadic Eye Movements Increase Interhemispheric Coherence? Investigation of a Hypothesized Neural Mechanism Underlying EMDR. Frontiers in Psychiatry, 2, 4. 10.3389/fpsyt.2011.00004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sawatzky R, Chan EKH, Zumbo BD, Ahmed S, Bartlett SJ, Bingham CO 3rd, Gardner W, Jutai J, Kuspinar A, Sajobi T, & Lix LM (2017). Montreal Accord on Patient-Reported Outcomes (PROs) use series-Paper 7: modern perspectives of measurement validation emphasize justification of inferences based on patient reported outcome scores. Journal of Clinical Epidemiology, 89, 154–159. 10.1016/j.jclinepi.2016.12.002 [DOI] [PubMed] [Google Scholar]
- Schweizer S, Grahn J, Hampshire A, Mobbs D, & Dalgleish T. (2013). Training the emotional brain: improving affective control through emotional working memory training. Journal of Neuroscience, 33(12), 5301–5311. 10.1523/jneurosci.2593-12.2013 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sheeran P, Klein WM, & Rothman AJ (2017). Health Behavior Change: Moving from Observation to Intervention. Annual Review of Psychology, 68, 573–600. 10.1146/annurev-psych-010416-044007 [DOI] [PubMed] [Google Scholar]
- Sierk A, Manthey A, King J, Brewin CR, Bisby JA, Walter H, Burgess N, & Daniels JK (2019). Allocentric spatial memory performance predicts intrusive memory severity in posttraumatic stress disorder. Neurobiol Learn Mem, 166, 107093. 10.1016/j.nlm.2019.107093 [DOI] [PubMed] [Google Scholar]
- Singh L, Espinosa L, Ji JL, Moulds ML, & Holmes EA (2020a). Developing thinking around mental health science: the example of intrusive, emotional mental imagery after psychological trauma. Cogn Neuropsychiatry, 25(5), 348–363. 10.1080/13546805.2020.1804845 [DOI] [PubMed] [Google Scholar]
- Singh L, Espinosa L, Ji JL, Moulds ML, & Holmes EA (2020b). Developing thinking around mental health science: the example of intrusive, emotional mental imagery after psychological trauma. Cognitive Neuropsychiatry, 25(5), 348–363. 10.1080/13546805.2020.1804845 [DOI] [PubMed] [Google Scholar]
- Smith KE, Mason TB, Schaefer LM, Juarascio A, Dvorak R, Weinbach N, Crosby RD, & Wonderlich SA (2020). Examining intra-individual variability in food-related inhibitory control and negative affect as predictors of binge eating using ecological momentary assessment. Journal of Psychiatric Research, 120, 137–143. 10.1016/j.jpsychires.2019.10.017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stevens MC, Gaynor A, Bessette KL, & Pearlson GD (2016). A preliminary study of the effects of working memory training on brain function. Brain Imaging and Behavior, 10(2), 387–407. 10.1007/s11682-015-9416-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tao C, Li X, Tang C, Jin H, & Du J. (2019). Acceleration Index Predicts Efficacy of Orthostatic Training on Vasovagal Syncope in Children. Journal of Pediatrics, 207, 54–58. 10.1016/j.jpeds.2018.10.063 [DOI] [PubMed] [Google Scholar]
- Tebes JK, & Thai ND (2018). Interdisciplinary team science and the public: Steps toward a participatory team science. American Psychologist, 73(4), 549–562. 10.1037/amp0000281 [DOI] [PMC free article] [PubMed] [Google Scholar]
- The Casrai Project. CRediT (Contributor Roles Taxonomy). https://casrai.org/credit. https://casrai.org/credit
- University of California, D. (2022). Health Equity Resources and Outreach Program (HERO) University of California, Davis; Retrieved December 13, 2022 from https://health.ucdavis.edu/ctsc/area/HERO/services.html [Google Scholar]
- University of Minnesota. Surviving Group Projects https://teamwork.umn.edu. https://teamwork.umn.edu
- van Veen SC, Engelhard IM, & van den Hout MA (2016). The effects of eye movements on emotional memories: using an objective measure of cognitive load. European Journal of Psychotraumatology, 7, 30122. 10.3402/ejpt.v7.30122 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Weinfurt KP (2019). Clarifying the Meaning of Clinically Meaningful Benefit in Clinical Research: Noticeable Change vs Valuable Change. Jama, 322(24), 2381–2382. 10.1001/jama.2019.18496 [DOI] [PubMed] [Google Scholar]
- Weinfurt KP (2021). Constructing arguments for the interpretation and use of patient-reported outcome measures in research: an application of modern validity theory. Quality of Life Research. 10.1007/s11136-021-02776-7 [DOI] [PubMed] [Google Scholar]
- Whiteford HA, Harris MG, McKeon G, Baxter A, Pennell C, Barendregt JJ, & Wang J. (2013). Estimating remission from untreated major depression: a systematic review and meta-analysis. Psychological Medicine, 43(8), 1569–1585. 10.1017/s0033291712001717 [DOI] [PubMed] [Google Scholar]
- Wu H, Mata J, Furman DJ, Whitmer AJ, Gotlib IH, & Thompson RJ (2017). Anticipatory and consummatory pleasure and displeasure in major depressive disorder: An experience sampling study. Journal of Abnormal Psychology, 126(2), 149–159. 10.1037/abn0000244 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wu M, Giel KE, Skunde M, Schag K, Rudofsky G, de Zwaan M, Zipfel S, Herzog W, & Friederich HC (2013). Inhibitory control and decision making under risk in bulimia nervosa and binge-eating disorder. International Journal of Eating Disorders, 46(7), 721–728. 10.1002/eat.22143 [DOI] [PubMed] [Google Scholar]
- Wuchty S, Jones BF, & Uzzi B. (2007). The increasing dominance of teams in production of knowledge. Science, 316(5827), 1036–1039. 10.1126/science.1136099 [DOI] [PubMed] [Google Scholar]
- Zucker N, Mauro C, Craske M, Wagner HR, Datta N, Hopkins H, Caldwell K, Kiridly A, Marsan S, Maslow G, Mayer E, & Egger H. (2017). Acceptance-based interoceptive exposure for young children with functional abdominal pain. Behaviour Research and Therapy, 97, 200–212. 10.1016/j.brat.2017.07.009 [DOI] [PMC free article] [PubMed] [Google Scholar]