Preliminary Implementation Outcomes of a Free Online Toolkit to Support Exposure Therapy Implementation for Youth

Abstract

Background

Exposure therapy (“exposure”) for youth anxiety is highly underutilized in clinical practice. Asynchronous, online implementation strategies such as online toolkits hold promise as pragmatic approaches for extending the sustainability of evidence-based interventions, but their long-term usage, perceived utility, and impact are rarely studied.

Objective

This study presents three-year preliminary implementation outcomes for a free, online toolkit to support exposure therapy use with youth: the Resource for Exposure for Anxiety Disordered Youth (READY; www.bravepracticeforkids.com). Implementation outcomes of interest included READY usage statistics, adoption, perceived utility, and clinician exposure use.

Methods

Web analytics characterized usage patterns. A survey of READY users (N = 49; M age = 34.2, 82.9% female, 71% White) assessed adoption, perceived utility, clinician exposure use, and persistent barriers to exposure use.

Results

In its first three years, READY had 13,543 page views across 1,731 unique users; 442 (25.6%) registered as a site user to access specialized content. Survey data suggested variability in usage and perceived utility across toolkit components. Qualitative analyses highlighted persistent exposure barriers that pointed to potential READY refinements.

Conculsions

Overall, READY has been accessed by hundreds of clinicians, but its impact was limited by low return to the site. This study highlights strengths and limitations of standalone online implementation supports and identifies additional steps needed to optimally support clinicians to deliver exposure to youth in need.

Anxiety and related disorders are the most common youth mental health conditions (Merikangas et al., 2010) and are associated with significant functional impairment (Rapee et al., 2009; Swan et al., 2018). Exposure therapy (“exposure”) is the leading cognitive-behavioral therapy (CBT) treatment ingredient that leads to improved youth outcomes in treatment for anxiety (Peris et al., 2015; Higa-McMillan et al., 2016). Exposure refers to structured techniques designed to help youth gradually face their fears in a supported way (Peterman et al., 2015). Despite its empirical support, exposure remains highly underutilized in routine clinical settings. Less than a third of therapists in routine clinical care report using exposures with their anxious clients (Becker-Haimes, Okamura et al., 2017; Whiteside, Deacon, et al., 2016; Whiteside et al., 2016).

There is a robust literature that details the complex implementation barriers that clinicians experience with exposure therapy, including those related to knowledge (Becker-Haimes, Okamura et al., 2017), negative attitudes (Deacon et al., 2013; Lokers, 2020), ethical concerns (Olantunji et al., 2009; Gola et al., 2016), and unique organizational challenges (Becker-Haimes et al., 2020). Recent years have seen innovations in training methodologies that aim to overcome some of these established implementation barriers, including approaches that directly target clinicians’ negative beliefs about exposure (e.g., Farrell et al., 2013; Frank, Becker-Haimes, Rifkin, et al., 2020; Reid et al., 2017). However, there are well-established limitations of traditional workshop-based training and consultation for achieving implementation success (Beidas & Kendall, 2010; Frank et al., 2020) and major cost limitations associated with the more intensive, longitudinal implementation supports shown to be most successful (Okamura et al., 2018). There is a critical need to develop educational implementation strategies to support clinicians in routinely delivering exposures that can augment training and consultation. Given cost limitations of more extensive implementation supports, there is a unique need for strategies specifically designed for under-resourced settings, where exposure therapy is unlikely to be sustained in practice (Chu et al., 2015).

Asynchronous, online toolkit-based implementation supports are a promising and pragmatic approach to supporting implementation of evidence-based interventions (EBIs; Luck et al., 2014; Owens et al., 2020; Yamada et al., 2015). Toolkits typically are defined as “action-oriented compilations of information, resources, or tools, to guide users in organizing information about a specific topic” (Agency for Healthcare Research and Quality, 2016). Implementation toolkits, which bundle information to support use of discrete clinical practices, are an increasingly popular approach to mitigating challenges with dissemination and implementation of EBIs (e.g., Baumann et al., 2018). Despite their growing use in supporting implementation efforts, data on their use and effectiveness has been limited (Yamada et al., 2015), especially for supporting implementation of psychosocial treatments. The purpose of this study was to assess usage and clinician perspectives of an online toolkit that was specifically designed to support the use of exposure therapy with anxious youth – the Resource for Exposure for Anxiety Disordered Youth (READY) – over a three-year period. We were interested in the following implementation outcomes (Proctor et al., 2011; Glasgow et al., 2019): the rates and patterns of READY access, clinician self-reported adoption of READY (defined as the extent to which READY was used), perceived utility of READY (defined for our purposes as the users’ ratings of how useful READY was to their clinical practice), and clinician exposure use (defined as whether use of exposure varied relative to READY toolkit use or not). We also aimed to characterize the persistent barriers clinicians reported with respect to exposure use after accessing the READY toolkit to inform future research in this space.

READY was initially developed as a tangible toolkit (i.e., a binder of resources alongside “challenge cards” with various exposure ideas) designed to augment initial trainings and offer free and accessible resources to address commonly cited barriers to clinician use of exposure with youth. The intended READY user is a clinician who already has some prior training in exposure therapy but needs supports to implement exposure into their practice. Clinical content included in READY has been published previously (Becker-Haimes et al., 2017); briefly, content was developed to provide targeted guidance to clinicians in the application of exposure therapy for the most common anxiety disorders in youth. In the initial evaluation of READY, 70 community mental health clinicians receiving training in the delivery of exposure therapy with youth were asked to provide feedback on READY via survey; an additional subset of clinicians also provided qualitative feedback after using the toolkit for a month. Overall, READY was perceived favorably as a feasible tool to support exposure use in practice. Participants also expressed high openness to the idea of transitioning READY content to a website to facilitate access to resources over time (Becker-Haimes et al., 2017).

Based on this prior research, and in line with literature citing the promise of asynchronous, online implementation strategies as a pragmatic, sustainable approach to improve EBI implementation (e.g., Owens et al., 2020), READY content was transitioned onto a freely accessible online platform (www.bravepracticeforkids.com) in April, 2018. It is important to note that READY is not intended to address all barriers to exposure implementation (e.g., organizational barriers to exposure; Becker-Haimes et al., 2020). Rather, READY’s primary goals are to address clinician-level barriers related to knowledge, attitudes, and skill with respect to sustained exposure delivery. Most importantly, a goal of READY is to address two major barriers: (1) many community clinicians do not have access to an expert supervisor or consultant, and (2) many clinicians do not have the time outside of sessions needed to generate tailored exposure exercises that can address the unique needs of diverse youth with anxiety served in routine outpatient mental health settings.

Currently, the online READY toolkit comprises three sections that aim to address different implementation barriers. Section 1 is an overview of exposure primarily intended to function as a knowledge resource and dispel myths about exposure to improve clinician attitudes toward its use. The content in this section includes step-by-step guides for delivering exposure sessions and includes tip sheets on how to enhance exposures to make them most effective, avoid common pitfalls in conducting exposures with youth, answer frequently asked questions, and more. Section 2 includes targeted tips and tricks and multiple examples of exposure exercises (“challenge cards”) for various common presentations of youth anxiety (separation anxiety, social anxiety, generalized anxiety, panic disorder, and a variety of specific phobias). Challenge cards are developmentally tailored for working with children and adolescents (e.g., by including school-based exposures). Challenge cards can be used by clinicians individually to brainstorm potential exposure ideas or can be printed and brought to the client’s session to form the basis of hierarchy generation by having clients directly rate how anxiety provoking each situation would be (see Becker-Haimes et al., 2017 for all intended potential uses of challenge cards). Challenge cards were intentionally designed to mitigate practical barriers related to the preparation time and reliance on expert supervision needed to effectively tailor exposure practices (Chu et al., 2015), and ensure that there are exposure practice options that require little preparation time or minimal organizational resources to complete. Finally, Section 3 comprises supplemental material that provides resources to reduce preparation time needed for exposures (e.g., the time needed to research recipes for fake vomit to use for vomit phobia).

To date, there has been no systematic evaluation of how the electronic format of READY is utilized nor how it is perceived by clinicians who use the toolkit. Such evaluation would have important implications for understanding the potential role that online toolkits can play in supporting the implementation of complex psychosocial interventions. This study utilized web analytic data from the first three years since READY’s launch and an anonymous survey of toolkit users to assess clinician use and perspectives of READY’s utility to support delivery of exposures to youth. In addition, we aimed to identify key areas for quality improvement of both READY and larger efforts to support exposure implementation. We did not have any a priori hypotheses about anticipated READY usage and adoption rates. However, we did hypothesize that READY would be perceived as useful, consistent with prior work on the non-web version, and that greater READY use would be associated with more exposure use.

Results will inform future refinements of the READY toolkit to maximize its utility with clinicians to ultimately expand the reach of exposure therapy for anxious youth. Results also will produce preliminary generalizable knowledge for the research community about the strengths and limitations of an online toolkit to support clinician delivery of EBIs such as exposure.

Methods

READY Website Description

READY is hosted on the Wix platform (wix.com) and is freely available. In its initial launch phase, clinicians primarily accessed the website via direct web link. As of 2020, search engine optimization was altered to make READY more accessible via leading search engines (e.g., Google). The landing page (www.bravepracticeforkids.com) is accessible to any site visitor and includes a brief overview of the toolkit’s purpose and its contents (e.g., a brief overview of the information within the toolkit, including definitions of the “tips and tricks” sheets and challenge cards). Visitors are prompted to create a free account as a “registered site user” to access any of the toolkit’s content outside of the landing page.

Since the website’s launch in April 2018, READY has been disseminated in a largely naturalistic fashion to practicing clinicians. The most common form of dissemination is direct sharing with community-based or private practice clinicians after local training workshops in exposure therapy and through workshops and presentations at local and national conferences. To date, the READY website was directly referenced or recommended by developers as a resource in approximately 25 workshops and community presentations, across which approximately 600 clinicians were estimated to be present. The development team is also aware of several other exposure trainers who incorporate references to READY in their training materials, although target audience numbers are unknown. Naturalistic peer-to-peer sharing of the site (e.g., postings about READY on social media) by workshop attendees is also known to have occurred.

Participants & Procedures

To examine how READY’s has been accessed, we extracted web analytic data from Wix, the READY toolkit platform. Wix provides a wide variety of analytic data, including web traffic, traffic by time of day, how many of those who visit the page go on to register as a formal user or visit the site again, and average time spent on the site. Analytic data were examined for the first three years after READY’s launch, from April 1, 2018, through March 31st, 2021.

We augmented web analytics with a brief, anonymous electronic survey of site users to assess READY adoption, perceived utility, and association with exposure use, and identify targets for future quality improvement. All registered site users with valid contact information at the time of study start (N = 364) were contacted by email about the survey on four separate occasions between January and March 2021; emails contained a link through which potential participants could opt out of the study and future mailings. There were 15 bounce-backs, and 3 individuals who opted out of survey recruitment emails; 49 individuals completed the survey (14% response rate).

Survey participants (n = 49) had a mean age of 35 (SD = 8.41), and most identified as female (75.5%). Racial identities, reported via open-ended question stems, were White (73.5%), Asian (10.2%), Black or African American (2%), Biracial or Multiracial (2%), or Hispanic (2%). Approximately half (49%) of respondents were doctorate-level and 47% were master’s level. Nearly all (85.7%) respondents endorsed that they provided direct clinical services, and 57% were licensed. Most participants (67.3%) reported a professional discipline of clinical psychology, and the next most prevalent professional discipline was social work (10%). The most common professional settings were private practice (36.7%) and academic medical center/hospital settings (32.7%). Average caseload was 24.5 (SD = 17), and respondents saw an average of 15 clients a week (SD = 10). Most (73.5%) reported that they had previously received formal training or supervised practice in exposure therapy. Forty-two participants (85.7%) reported having at least one youth with anxiety on their current caseload.

Ethical Considerations

All respondents provided informed consent prior to participation. The survey was created and distributed using Qualtrics. Participants were entered into a lottery to win one of ten $35 electronic gift cards. Procedures were approved by the Institutional Review Board at the University of Pennsylvania.

Survey Measures

Self-Reported Toolkit Use (Adoption)

This measure was created for this study to characterize self-reported READY use. Items asked respondents to report on the frequencies and methods by which they interacted with the different elements of the toolkit. Clinicians first reported how often they have interacted with the READY toolkit overall in the past year (Never, 1–3 times, 4–10 times, 11–20 times, 21–30 times or 31 + times) and then were asked to check off the specific ways in which they used the toolkit (yes or no) from a list of items.

Toolkit Feedback Form (Perceived Utility)

Participants completed an adapted version of a brief feedback form that was used to evaluate the earlier, non-web-based version of the READY Toolkit (Becker-Haimes et al., 2017). This form contained three sections. The first two sections asked participants to rate how useful aspects of the toolkit are, using a 4-point Likert scale (Not Useful at All to Very Useful) that was adapted from prior studies assessing usability of new mental health technologies (Bruns et al., 2016) and the original READY pilot. In the first section, questions queried respondents about perceptions of overall content (e.g., how useful the “Tips & Tricks Sheets” are generally); the second section queried respondents on perceptions of disorder-specific content (e.g., how useful the content on Separation Anxiety is to their practice). A third section of this measure asked clients about the potential relevance or utility of new content for READY, to inform future quality improvement efforts. Table 1 presents an overview of the content assessed.

Table 1.

Survey Respondents Perceived Usefulness of Ready Components

	Not Useful at All	Only A Little Useful	Fairly Useful	Very Useful	N/A (Never Used)
	n (%)	n (%)	n (%)	n (%)	n (%)
Ratings of Existing READY Components 1
An overview of basic information about exposure	0 (0)	2 (4.1)	15 (30.6)	16 (32.7)	6 (12.2)
Step by Step guide to exposures	0 (0)	0 (0)	8 (16.3)	23 (46.9)	8 (16.3)
Answers to Frequently Asked Questions	0 (0)	2 (4.1)	12 (24.5)	13 (26.5)	12 (24.5)
Tips & Tricks Sheets	0 (0)	0 (0)	11 (22.4)	20 (40.8)	8 (16.3)
Challenge Cards	1 (2.0)	2 (4.1)	6 (12.2)	24 (49.0)	6 (12.2)
Additional Resources/Supplemental Files	1 (2.0)	2 (4.1)	9 (18.4)	12 (24.5)	15 (30.6)
Ratings of Existing READY Content Areas 1
General Exposure Content	0 (0)	2 (4.1)	10 (20.4)	16 (32.7)	10 (20.4)
Separation Anxiety	0 (0)	0 (0)	13 (26.5)	14 (28.6)	11 (22.4)
Social Anxiety	0 (0)	1 (2.0)	6 (12.2)	23 (46.9)	9 (18.4)
Generalized Anxiety	0 (0)	1 (2.0)	7 (14.3)	23 (46.9)	8 (16.3)
Panic Disorder	0 (0)	3 (6.1)	3 (6.1)	17 (34.7)	15 (30.6)
Specific Phobia – Dark	0 (0)	2 (4.1)	8 (16.3)	9 (18.4)	19 (38.8)
Specific Phobia – Animals/Insects	1 (2.0)	2 (4.1)	5 (10.2)	12 (24.5)	19 (38.8)
Specific Phobia – Blood Injection Injury	0 (0)	1 (2.0)	6 (12.2)	13 (26.5)	19 (38.8)
Specific Phobia – Vomit	1 (2.0)	1 (2.0)	6 (12.2)	14 (28.6)	16 (32.7)
Ratings of Relevance or Utility of Potential Future READY Components 1
Helpful Metaphors for Exposure Therapy	1 (2.0)	2 (4.1)	18 (36.7)	26 (53.1)	--
Obsessive Compulsive Disorder	1 (2.0)	2 (4.1)	10 (20.4)	34 (69.4)	--
Specific Phobia-Choking	5 (10.2)	16 (32.7)	14 (28.6)	12 (24.5)	--
Specific Phobia-Costumed Characters	13 (26.5)	16 (32.7)	12 (24.5)	6 (12.2)	--
Video Examples of Exposure	0 (0)	4 (8.2)	11 (22.4)	31 (63.3)

¹ Some respondents did not complete all items, so percent totals do not sum to 100.

Toolkit Improvement

The feedback form also included an open-ended response option for clinicians to share other content they would like to see included in the READY toolkit. Finally, clinicians were asked to share via an open-ended response item what they experienced as the biggest challenge(s) to exposure delivery with youth in their practice to inform future ideas for exposure support within and beyond READY.

Clinical Practices Questionnaire (Exposure Use)

This questionnaire included several targeted questions asking about the clinicians’ caseload of youth with anxiety and related disorders and their self-reported use of exposure therapy over the past 4 weeks.

Additional Items

Clinicians also reported on demographic and clinical background characteristics (e.g., educational background, professional discipline).

Data Analysis Plan

Descriptive statistics characterized web analytic data and survey responses regarding adoption, perceived utility, and self-reported exposure use. To begin to examine relationships between READY use and clinician exposure practice, we planned to conduct exploratory chi-square analyses to compare whether self-reported frequency of exposure use varied relative to READY use in the past month. Our primary aim with these analyses was to identify what, if any, dosage of READY use was associated with self-reported exposure use in clinical practice.

Finally, open-ended responses to the question about respondents’ biggest challenge(s) to exposure delivery were coded to identify themes. Two independent raters reviewed each of the reported challenges to identify themes of exposure barriers based on the data. Each rater then independently categorized each challenge into one or more of the identified themes. Initial independent coding agreement was high (93%) and coders came to consensus on 100% of all themes. The first author takes responsibility for the integrity of all data and the accuracy of data analyses.

Results

Web Analytics

To assess READY access patterns, web analytic data were pulled from the first three years after the website launch (April 1, 2018, through March 31st, 2021). As of March 31^st, 2021, the READY toolkit had 13,543 unique page views across 2,584 total site sessions from 1,731 unique visitors. Of these unique visitors, 442 (25.5%) registered an account with the site as a clinician user. Site users constituted 22.9% of all page views and 20.5% of site sessions, suggesting that the bulk of site traffic was driven by non-users (i.e., individuals who visited the landing page only but did not register as a site user). The average duration of an individual session for site users was 9 min and 54 s and the average duration of individual sessions for non-registered users was 8 min and 18 s, after removing data outliers and extraordinarily long sessions. Outliers were defined as active sessions longer than the typical 8-hour workday; as analytics do not include inactivity that persists beyond 30 min, this does not reflect someone simply leaving the tab open to the site on their computer and may be reflective of non-human interaction with the site.

Among those who registered as site users, 83% navigated to multiple components (“pages”) within the toolkit, with an average of 4 separate pages per session. Analytics suggested a ratio of approximately 2:1 between site sessions and unique visitors, suggesting repeated usage of the site (our analytics were unable to provide exact estimates of how many times registered users returned to the site). In contrast, among those who visited the site and did not become a site member, only 39% of visitors navigated past the landing page, visiting an average of 2 pages per session.

(n = 4,085) of page views corresponded to views of clinical content and explanatory features (e.g., table of contents) included in READY. The most commonly viewed components of the toolkit were the challenge cards (n = 1,264; 30.9%; M time per page = 1 min, 52 s), followed by the basic overview of exposure (n = 664, 16.3% M time per page = 2 min, 10 s), Tips and Ttricks Sheets (n = 615, 15.1%; M time per page = 1 min, 29 s), Step by Step Guide to Exposures (n = 114, 2.8%, M time per page = 3 min, 6 s) answers to Frequently Asked Questions (n = 50; 1.2%, M time per page 1 min, 12 s), and various additional resources and supplemental files (n = 501; 12.3%. M time per page = 38 s).

Usage was dispersed internationally across the United States, Canada, Europe, the Middle East, and Australia. The largest clusters of users were from the Northeast and Northwest regions of the United States. Most visitors (n = 1,383, 79.8%) accessed the site via desktop or laptop computer, whereas a minority accessed READY via a mobile device (n = 334, 19.3%). Less than 1% of unique visitors accessed READY via tablet. Figure 1 illustrates the number of site sessions over the three-year period. Except for unusually high traffic in the month of September 2019 (potentially due to promoting the website during a large training in that month), site usage has steadily and slightly increased over time.

Fig. 1.

Unique Site Sessions over Time: April 1, 2018-March 31, 2021

Survey Responses

Adoption

Survey respondents reported variability in their frequency of READY use in the past year. Most (n = 35, 71.5%) reported accessing the toolkit less than ten times in the past year, with 53.1% (n = 26) accessing it 1–3 times and 18.4% (n = 9) accessing it 4–10 times, Fewer respondents reported using it 11–20 times (n = 3, 6.1%) or 21 + times (n = 2, 4.1%); 8 respondents (n = 8; 16.3%) reported that they had not used the toolkit in the past year.

The most common ways that READY was used by respondents largely paralleled data from the web analytics: general browsing (69.4%), reviewing challenge cards (i.e., exposure ideas) to prepare for exposure sessions (51.0%), and reviewing tip sheets to prepare for sessions (38.8%). Approximately a quarter of respondents reported downloading material to use directly in session with clients (36.5% and 18.4% respectively for challenge cards and tip sheets) and 10.2% reported sharing the website with the client directly in session. About a third (34.7%) reported that they had shared the website with a colleague and 18.4% reported using the website for training and supervision purposes.

Perceived Utility

Table 1 shows how survey respondents reported on the perceived usefulness of the READY components for their individual clinical practice. Consistent with hypotheses, no element of the toolkit was rated as “Not at All Useful,” except for one respondent rating the Challenge Cards as “Not at All Useful.” The Challenge Cards, Tips & Tricks Sheets, and the Step-by-Step Guide to Exposures were the components that were rated as “Very Useful” the most frequently (by 40.8–49.0% of respondents), which corresponded with the top three most endorsed ways that READY was used. With respect to specific content area, content related to social anxiety and generalized anxiety was rated as “Very Useful” most often (by 46.9% of respondents), followed by content related to panic disorder (34.7%), general overviews of exposure (32.7%), and separation anxiety (28.6%). Content related to specific phobias (of the dark, animals/insects, blood injection injury, and vomit) showed the most variability in perceived utility.

Self-Reported Exposure Use

These analyses were conducted only on the subset of 42 clinicians who reported having at least one youth with anxiety on their current caseload. Within this subset, we looked at both the use of any exposures within the past month, as well as high exposure use, defined as the self-reported use of exposures with over half of anxious youth on their caseload. Of this subsample, 34 (81.0%) of clinicians reported delivering exposure with at least one client over the past month and 24 (57.1%) reported delivering exposures with four or more clients. Given the low number of clinicians who did not deliver exposure, the sample was deemed as having insufficient variability to examine whether READY use had any influence on exposure practice and chi square analyses were not conducted.

Future Quality Improvement Targets for READY

Future targets for READY quality improvement and exposure implementation efforts were assessed using the Toolkit Feedback Form. When asked to rate the perceived utility of potential future READY content (see Table 1), content related to obsessive-compulsive disorder and video examples of exposure implementation were rated most often as likely to be “Very Useful” (69.4% and 63.3% of respondents, respectively), with information on helpful metaphors to use with youth for exposure therapy rated as likely to be “Very Useful” by 53.1% of respondents. There was less enthusiasm for more content related to specific phobias of choking or costumed characters. Twelve clinicians (24.5%) provided responses to the open-ended question about new information content for READY that would be helpful. Information shared varied, and included resources on other potential phobia responses, including medical/dental anxiety, heights and elevators, and phobias related to weather and natural disasters; more resources for social anxiety (beyond what currently exists on the website); and example imaginal scripts for worries that are hard to address via in vivo exposures. Clinicians also reported wanting support for exposures in additional content areas, including those related to fears of death or the devil, selective mutism, and food-related anxiety. One clinician also noted that it would be helpful to have a sortable list or summary spreadsheet of all possible challenge cards and another noted that youth testimonials of their experience with going through exposure therapy for anxiety would be a helpful resource to have to share with clients. Finally, one clinician expressed an interest in specific resources about how to address cultural considerations and variability in anxiety symptom manifestations.

Persistent Barriers to Exposure

Thirty-three participants (70.2%) responded to the open-ended question about perceived challenges to exposure therapy delivery; 66 unique challenges were reported. We identified nine themes of exposure barriers based on the data. Table 2 presents an overview of noted barriers along with the frequencies with each was highlighted across participants, alongside illustrative statements shared by respondents and implications for READY improvements.

Table 2.

Themes Regarding Ongoing Exposure Challenges (N = 66 barriers from 33 respondents)

Theme	Frequency	Examples	Implications for READY
Engagement or attendance	18	“When youth do not follow through with exposure practice outside of sessions” “How to overcome patient resistance/refusal to try any exposures” “Attendance and practice”	Expand on existing content about troubleshooting barriers to engagement
Family factors	14	“Parental accommodations impairing outside practice” “Very accommodating parents unwilling/unable to manage their own distress during the child’s exposure” “Families not understanding the rationale fully, the “why” of exposure”	Create tip sheets about addressing parental accommodation within the context of exposure
Exposure generation or execution	13	“Developing the hierarchy, especially coming up with ways to break the fear into smaller steps” “Delivering the steps in clinic setting (not going to the patients home)”	Create an “Ask an Expert” feature where users can submit questions about specific clinical presentations and get feedback from a clinical expert
Practical or contextual barriers	8	“Time and lack of billing codes/insurance coverage for sessions that go over time due to exposure work” “Transportation to the place when the exposure needs to take place”	N/A – barriers best addressed via other implementation strategies
Obtaining buy-in for exposures	6	“Delivering [psychoeducation] on exposure rationale for patient and parents/caregivers efficiently and effectively” “Youth receptivity to rationale”	Create or link to existing video examples of how to optimally provide psychoeducation about exposure
Youth comorbidities	4	“When anxiety co-occurs with extreme behavior problems or suicidal behavior” “Identifying if exposures are the first step in their treatment process. Eg. If they have hallucinations do we treat that first, simultaneously or second”	Create tip sheets for managing anxiety within common comorbidities; “Ask an Expert” feature
COVID-19 disruptions	4	“COVID can make exposure really hard because a lot of situations related to worries aren’t happening (e.g., separation worries about going to school if a child is only doing remote learning)” “A new problem is planning exposures that can be done remotely via telehealth”	Create or link to existing tip sheets or resources on delivering exposures via telehealth
Clinician self-efficacy	2	“My lack of confidence and competency” “My own inexperience”	Add a “role play clinic” feature, whereby clinicians can pay a small fee to complete a brief exposure role play and get feedback from a READY content expert
Values conflicts	2	“When socially anxious and depressed youth prefer not to put themselves out there though it would benefit them” “When perfectionistic youth prefer perfection over functioning”	Create a READY module designed to guide clinicians in striving for culturally responsive care in exposure practices

Note. Some barriers were classified into more than one theme, so sum total of frequencies does not equal 66.

The top three most cited challenges related to: (1) “engagement or attendance,” defined as barriers related to either engagement (e.g., motivation) in session or poor attendance (27.7% of reported challenges), (2) “family factors,” or barriers related to difficulties with the family system impeding treatment progress such as high levels of parental accommodation (21.2% of reported challenges), and (3) “exposure generation or execution,” defined as difficulties with generating ideas for exposure practices or other challenges related to executing the process of exposure (19.7% of reported challenges). Other themes that emerged included: (1) “practical or contextual barriers,” which included barriers related to practical or contextual factors that interfere with treatment, such as limited insurance reimbursement, lack of transportation, limits of the outpatient setting, and lack of resources needed to do exposure; (12.1% of challenges), (2) “obtaining buy-in for exposures,” which included barriers related to clinicians struggling to provide psychoeducation or obtain buy in from the family (9.1% of challenges), (3) “youth comorbidities,” which included barriers related to non-anxiety related comorbidities or references to the complexity of a youth’s presentation (6.0% of challenges), (4) “clinician self-efficacy,” which included barriers related to one’s own perceived lack of skill or confidence in exposure (3.0% of challenges), and (5) “value conflicts,” which included barriers reflective of a potential mismatch between an exposure practice and a youth’s preference and values (3.0% of challenges). Finally, several clinicians (6.0%) noted challenges related to newer barriers specific to delivering exposures within the context of COVID-19 disruptions in care (e.g., changes in clinical presentations due to COVID-19, telehealth barriers).

Table 2 also describes how an online implementation toolkit like READY could be refined and extended to address the barriers noted above to better support the needs of clinicians working with anxious youth.

Discussion

This study combined web analytic and survey data to conduct a preliminary evaluation of a freely available toolkit to support clinician use of exposure therapy three years after its launch. Findings from web analytic data suggested that READY has had relatively wide and growing use in its first three years; however, usage and adoption are relatively low compared to the broad population of mental healthcare providers (Health Resource & Services Administration, 2020), with especially low “rates of return” to the site. Survey data also indicated that READY was generally perceived positively by users.

  We also intended to examine whether components of READY use were associated with self-reported exposure practice. However, exposure usage rates in this sample were high (> 80%), which did not allow for the variability needed to conduct such analyses. Such high rates of exposure are notable given that typical estimates of exposure usage with anxious youth among clinicians (even those who have received training in exposure) is somewhere between 10 and 30% (Becker-Haimes, Okamura et al., 2017; Chu et al., 2015; Whiteside, Deacon, et al., 2016; Whiteside, Sattler et al., 2016); thus, use in this sample was much higher than typical surveys of practicing mental health clinicians. As most respondents also reported having some prior training in exposure (consistent with the intended user profile), READY may be functioning exactly as intended-- as a supplementary support tool for those who are trained and motivated to use exposure, but who lack experience with or confidence in their execution. We cannot ascertain whether these high rates of exposure use seen here are at all attributable to READY or simply a function of a highly motivated sample of clinicians to deliver exposure. Future work with larger samples should aim to tease apart the potential relationships between READY and exposure use to identify whether there are specific features of READY that are most helpful for promoting exposure use. Future research should aim to dismantle components of online toolkits like READY to determine which features are most conducive to increasing clinical practice use of EBIs. Studies also might aim to identify factors (e.g., clinician knowledge, organizational setting) that moderate the impact of toolkits on practice. This would inform personalization of future implementation supports.

Site users who responded to our survey reported using READY features in varied ways; however, most clinicians reported commonly using READY to prepare for exposure sessions either through reviewing tip sheets or generating exposure ideas via challenge cards; this was supported by larger analytics as well. Both analytic and survey data suggested that some components of the toolkit (e.g., challenge cards) were used much more than others (e.g., the Step-by-Step Guide to Exposure). Interestingly, survey response data also suggested that some clinicians are using READY to support clinical supervision, which was not initially an intended function of the toolkit. It is well-established that gold-standard supervision for exposure therapy is a resource-intensive endeavor (e.g., Dorsey et al., 2018; Reid et al., 2017; Schriger et al., 2022) to support clinicians to achieve competency in exposure delivery. READY may be acting as a supplementary tool for supervisors as well as clinicians; further understanding of READY’s use in supporting clinical supervision and learning how to optimize READY features to benefit this function is an important area to explore in future research.

Findings also pointed to key targets for future quality improvement of READY. First, web analytic data indicated that less than a quarter of visitors went on to register as a site user (i.e., over 75% of those who visited the site landing page and did not access its clinical content). This suggests that the requirement to register as a site user is a barrier to engaging with the site for busy clinicians. Alternatively, toolkits like READY are inherently designed to support clinicians to deliver interventions that they are already motivated to deliver (Davis et al., 2017). READY has a specialized focus and is not intended to be a support tool for novice clinicians learning about exposure for the first time nor is it for non-clinicians who are browsing the internet for youth anxiety resources. Thus, some of this drop-off after individuals visit the READY landing page may be appropriate (e.g., visitors realizing the site is not relevant or appropriate for their needs). However, it does raise the question of whether implementation supports like READY are only “preaching to the choir” (i.e., reaching those clinicians who are already most likely to deliver EBIs; Jensen-Doss et al., 2020). Additional work examining reasons for drop-off among a broad population of clinicians will be critical to understanding which clinicians are most likely to use a tool like READY. Such work could inform enhancements to READY to reach a broader audience (e.g., redesigning the landing page) and identify what alternative implementation supports are needed for clinicians who are unlikely to access an online implementation toolkit.

Second, 70% of clinician survey respondents reported ongoing challenges to their exposure therapy delivery that suggest areas for future development for online implementation supports (see Table 2). Potential areas for development largely fell into three major domains: (1) expanding content on the site (e.g., guidance on delivering culturally responsive exposure treatment, expanding to include content on obsessive-compulsive disorder), (2) creating video-based content to model elements of exposure that can be particularly challenging (e.g., video examples of how to efficiently provide psychoeducation or complete exposure practice), and (3) creating more opportunities for active learning or practice within the toolkit infrastructure (e.g., creating an “Ask an Expert” feature, where users can submit deidentified questions about specific clinical presentations and get feedback from a clinical expert or a “role play clinic,” in which clinicians could schedule and complete a brief exposure role play and get feedback from a content expert). Importantly, addressing such barriers would require more resources and would likely require clinicians to pay small fees to access these features for the site to remain sustainable. This raises questions of how to design online toolkits to optimally support clinicians to deliver psychosocial EBIs like exposure, while remaining freely available; this is an important question for exploration in future research. Furthermore, although the READY toolkit was designed specifically for clinicians who work with youth, future research may consider expanding the toolkit’s content to be relevant for clinicians who provide exposure to adult clients.

Study limitations should be noted. First, the survey had a low response rate (14%). While not atypical for online surveys (Sammut et al., 2021), we cannot ensure generalizability of findings. In addition, while surveying only clinicians who used READY enabled us to obtain rich data on READY utilization and identify areas for improvement, results may differ in a broader sample of clinicians, especially as there is likely selection bias affecting which clinicians sought out a clinical support like READY. Similarly, the clinical background of survey respondents was largely clinical psychology, and nearly half were doctoral level, which contrasts with the largely master’s level mental health workforce (Hoge et al., 2019); this suggests our sample was not reflective of clinicians in clinical practice more broadly. Our survey also was sent out during the COVID-19 pandemic. This may have lowered our response rate and clinicians experiencing higher pandemic-associated distress may have been less likely to participate. Finally, our self-reported exposure use variable did not have sufficient variability to allow for planned exploratory analyses. We also did not assess anything related to a clinician’s quality of exposure delivery.

Our analytic data also had limitations worth noting. Given the naturalistic ways in which READY has been disseminated (e.g., in training, classroom settings, social media, and peer-to-peer sharing), we are unable to assess: (1) the proportion of clinicians who were made aware of READY, and (2) the proportion of clinicians introduced to READY who did and did not engage with the site. Such data would have further contextualized understanding of READY’s reach to date. Analytics also only provided high-level information about usage; we were unable to parse apart common usage patterns for the site (e.g., the different ways that clinicians spend time on the site and how often registered site users end up returning to the site). Finally, given the anonymous nature of our survey, we were unable to link self-reported survey data with actual READY use as measured via analytics.

Findings from this study highlight exciting areas for future research. First, online implementation supports like READY are ripe for testing different dissemination strategies to increase reach and adoption among clinicians to identify the content, format, and timing of messages that are most likely to engage clinicians. Given the relatively low adoption observed in this study, it is likely that multiple pathways of dissemination (e.g., marketing during trainings, conferences, and on social media; embedding it in graduate education for mental health providers) are needed to expand the reach of a tool like READY to its target audience. In addition, insights from behavioral economics (Beidas et al., 2021) could be used to make it easier and more motivating for target users to engage more fully with READY as an implementation support tool (e.g., creating incentive structures for interacting with multiple elements of the toolkit, tailoring messaging to users to support their access of new site content), Second, an important next step for READY evaluation, and evaluation of implementation toolkits broadly, is to move beyond naturalistic evaluation to conducting randomized controlled trials (e.g., randomizing clinicians following standard training and consultation to receive READY access or not, testing the impact of READY versus other non-digitally based implementation supports) to test the incremental benefits of an online toolkit on clinician exposure use. Such work could also test whether the addition of READY can reduce expert consultation resources needed to support clinicians in obtaining competency in exposure delivery and how to optimally combine tools like READY with other implementation strategies (e.g., targeted supervision supports, addressing organizational policies that facilitate exposure delivery, addressing practical or contextual barriers raised by clinicians; Becker-Haimes et al., 2020).

This study has several notable strengths. This is one of the first studies to examine implementation outcomes of a freely available toolkit to support the delivery of a psychosocial treatment for youth. Such implementation supports remain underdeveloped, yet have high potential to address implementation barriers, particularly among clinicians who cannot afford costlier implementation supports (e.g., ongoing expert consultation). Combining web analytic and survey data enabled us to provide both a macro and micro level understanding of READY’s strengths and limitations to date.

Conclusion

The READY online toolkit is a promising implementation resource for supporting clinicians to deliver exposure therapy to anxious youth that can augment more traditional training and consultation activities. Given the sparse data on the extent to which online toolkits are utilized in clinical practice and their potential impact on implementation outcomes, this research makes a unique and novel contribution to the literature. Results of this study highlight both the potential promise and limitations of a freely available online implementation support. Our findings can be harnessed to inform understanding of the strengths and limitations of freely available online implementation supports for youth mental health EBIs.

Funding

Dr. Frank’s work was supported by Grant T32MH019927 (PI: Spirito).

Access to Data

Dr. Becker-Haimes takes responsibility for the integrity of the data and the accuracy of the data analysis.

Declarations

Conflict of Interest

All authors have no conflict of interest to disclose.