Characterizing Clinic Adoption of Clinical and Business Trainings in Child Mental Health in New York State

Abstract

Objective:

This study prospectively examined the naturalistic adoption of clinical and business evidence-informed trainings by all outpatient mental health clinics licensed to treat children, adolescents, and their families in New York State.

Methods:

Using September 2011-August 2013 attendance data from the New York State-funded Clinic Technical Assistance Center, this study classified the adoption behavior of 346 clinics in four ways: by number, type, intensity, and an adopter group category characterizing clinics by the highest training intensity in which they participated. Descriptive statistics on these adoption classifications were examined.

Results:

Among the 268 adopting clinics, a median of five out of 33 trainings were adopted; business and clinical trainings were about equally accessed (82% vs. 78%). Hour-long webinars were most popular (96% participation) compared to 6-18 month-long learning collaboratives (34% participation). Among adopters of business and clinical learning collaboratives, 73-100% sampled a webinar first before they committed to the learning collaboratives, though consistent participation in learning collaborative sessions over time was a challenge. Adopter groups captured meaningful adopter profiles: 41% were low-adopters that selected fewer trainings and only participated in webinars; 34% were high-/super-adopters that accessed more trainings and participated in learning collaboratives.

Conclusions:

More nuanced definitions of adoption behavior can improve the understanding of clinic adoption of trainings and hence promote the development of efficient roll-out strategies by state systems.

Scant literature exists on the adoption of state initiatives in U.S. public mental health systems. Findings from interviews with state mental health directors and child welfare directors on evidence-based practice and quality improvement (QI) adoption suggest that while clinics continuously experiment with new services (1, 2), they rarely implement evidence-based practices and they frequently discontinue the new services (3). Common barriers to adoption of new practices include inadequate financing, pressures for organizational productivity, staff resistance to change, necessary adaptations for the selected practices to be used in usual care, and limited access to information about the practices (1, 2).

Large-scale state roll-outs of evidence-based practices and QIs (3) aim to improve the quality of mental health services (see Appendix 1). They involve substantial financial investments, phasic implementation, and specific target populations (4). State departments tend to drive the focus of initiatives, ranging from trainings on specific clinical practices or services (4, 5) to organizational improvements (6). Roll-out approaches vary from allocating incentive funds to organizations to adopt and implement the initiatives, to using focused methods such as learning collaboratives and modular learning (7).

With limited empirical or conceptual frameworks to guide these state roll-outs, organizations’ adoption behaviors are rarely measured (8). Inconsistent definitions of “adoption” further hinder measurement efforts (9). Adoption measures are often simplistic (e.g., “yes” vs. “no” adoption) and insufficient (e.g., without appropriate denominators of clinics to whom initiatives are offered), with no attention to the type and intensity of initiatives adopted. Basic outcome data related to the adoption process or how to use such data to design future roll-outs are unavailable (10). According to the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework (11), these adoption behavior data not only enhance the understanding of adoption, non-adoption, and de-adoption (8), but they also provide important information prior to the implementation phase.

Clinic Technical Assistance Center (CTAC) in New York State

Considering these limitations in state roll-outs and the paucity of adoption outcomes, the Clinic Technical Assistance Center (CTAC) in New York State provides a unique opportunity to better understand the naturalistic, non-incentivized adoption of trainings on evidence-based practices and QIs. Founded in 2011, the New York State Office of Mental Health (NYSOMH)-funded CTAC (www.ctacny.com) offers statewide trainings and collects clinic attendance data on its offerings. Unlike prior technical assistance efforts that focus on evidence-based clinical interventions (5), CTAC is a training, consultation, and educational center for clinical and business needs that strengthen practitioners’ professional development and clinics’ abilities to meet the financial and regulatory challenges of healthcare reforms. Technical assistance is available to all 346 outpatient mental health clinics that are licensed to serve children, adolescents, and their families. CTAC offers three types of trainings based on NYSOMH feedback and clinics’ expressed needs (see Appendix 2 for details).

Business/Organizational improvement practices

These 12 trainings target clinic and agency administrators (e.g., leaders, executives, finance officers) to address the changing financial drivers of clinic operations and service delivery. They range from nine webinars on developing an effective business model, using financial modeling tools, to an all-day, in-person training on open access, centralized scheduling, and concurrent documentation. Further, two learning collaboratives require participating clinics to commit to a series of webinars, in-person trainings, and ongoing consultations. The Business Effectiveness and Efficiencies Project (BEEP) helps clinics assess, redesign their financial structures, and apply practical strategies towards sustainable business practices over an 18-month period. The Business Effectiveness Assessment Module (BEAM) consists of a series of webinar modules and group phone consultations on financial management over a six-month period.

Clinical Evidence-informed Practices

These 18 trainings target frontline clinicians and clinical supervisors. They include 14 hour-long, lunch-and-learn webinars (e.g., trauma, motivational interviewing, cognitive behavioral therapy) and one in-person training on family engagement. They also include three learning collaboratives that consist of a series of webinars, in-person trainings, and phone consultations. Two learning collaboratives are based on the 4Rs and 2Ss (Rules, Roles and Responsibilities, Respectful Communication, Relationships, Stress, and Social Support) curriculum that integrates evidence-based strategies for children with disruptive behavior disorders (12). One learning collaborative is a 12-month multi-family group model and the other a six-month model for individual families. The third learning collaborative, Practitioner Education and Decision Support (PEDS), adapted from the Managing and Adapting Practice tool (13), offers year-long decision support tools to improve clinical practices, accountability, and outcome monitoring.

Hybrid Practices

These three trainings target both clinic administrators and clinicians to improve business performances and practice outcomes. They include two webinars on outcome measurement. They also include the Training Intervention for the Engagement of Families (TIES), an all-day, in-person training that addresses both business and clinical aspects of patient engagement. TIES offers resources for overcoming organizational barriers to engaging families and trains clinicians on evidence-based family engagement interventions (14).

In summary, CTAC trainings are delivered in three ways — webinar, in-person training, learning collaborative — that represent varying levels of training intensity and clinic commitment. Hour-long webinars are the least intensive. In-person trainings require full-day participation. Learning collaboratives are the most time-intensive because they require ongoing participation in group learning sessions and consultations over a 6-18 month period. Participation in all CTAC offerings is voluntary.

This study aims to use clinic attendance data to describe the naturalistic adoption of CTAC trainings by the population of 346 mental health outpatient clinics that serve children, adolescents, and their families. Adoption behavior is characterized in four ways: by number of trainings accessed; by type of trainings accessed; by intensity of trainings accessed; and by adopter group defined by the highest intensity of training accessed. Examining adoption behavior in multiple ways may provide insight into factors that influence clinic uptake of trainings in evidence-based practices and QIs and could improve future state roll-outs.

Methods

All 346 NYSOMH-licensed outpatient clinics licensed to treat children and adolescents were included in this study, whether or not they attended any CTAC trainings. Between September 2011 and August 2013, clinic attendance at CTAC trainings was examined. Institutional review board approval was waived since the study was not considered human subject research (no interaction/intervention with individuals and no use of identifiable private information). This study classified clinic adoption of trainings in four ways:

By Number of Trainings

Of the 33 trainings offered, adoption can be described by the total number of trainings attended by each clinic.

By Type of Trainings

The three types of trainings offered were business (12 trainings), clinical (18 trainings), and hybrid business and clinical practices (3 trainings). Adoption can be described by the type of trainings selected by clinics.

By Intensity of Trainings

The three levels of training intensity offered were one-hour webinars, full-day in-person trainings, and learning collaboratives. Adoption can be described by the level of training intensity selected by clinics. To be considered a learning collaborative adopter, a clinic must have formally applied to the selected collaborative, and have attended at least one learning collaborative session. While these sessions are typically only open to formal learning collaborative participants, clinics occasionally attended a session as visitors; these clinics were not considered learning collaborative adopters, but were credited with attending the session according to the type and intensity as defined above.

By Adopter Group

Adopters can be categorized by the highest level of training intensity selected by clinics: low-adopters accessed webinars only; medium-adopters accessed at least one in-person training as highest level of intensity selected; high-adopters accessed one learning collaborative; and super-adopters accessed at least two learning collaboratives.

Fisher’s exact tests were used to examine the association between two adoption classifications (number of trainings vs. adopter group) to provide a better understanding of adoption behavior.

Results

Of the 346 clinics, 268 (78%) accessed at least one training, and 78 (22%) accessed none. Table 1 shows the four classifications of adoption among the 268 adopters:

Table 1.

Clinic adoption of CTAC trainings

		268 Adopters
Classification of Adoption		n	%
By Number of Trainings
	1-4 Trainings	130	48
	5-8 Trainings	66	25
	≥9 Trainings	72	27
By Type of Trainings
	Any Business Practice	220	82
	Any Clinical Practice	208	78
	Any Hybrid Practice	120	45
By Intensity of Trainings
	Any Webinar	257	96
	Any In-person Training	124	46
	Any Learning Collaborative	92	34
By Adopter Group
	Low-adopters	109	41
	Medium-adopters	67	25
	High-adopters	59	22
	Super-adopters	33	12

By Number of Trainings

The number of trainings accessed by adopters ranged from one to 26 trainings (median=5; mean=6). However, the adoption distribution was positively skewed, with 48% of the adopters participating in 1-4 trainings, 25% in 5-8 trainings, and 27% in ≥9 trainings.

By Type of Trainings

Clinics participated in business (82%) and clinical (78%) trainings at comparable rates; 45% accessed hybrid trainings, likely reflecting the smaller number of hybrid trainings available. When combinations of training types were examined, 37% of the adopters selected all three types, 24% adopted both business and clinical trainings, 18% focused exclusively on business trainings, and 14% only participated in clinical trainings.

By Intensity of Trainings

Adopters participated in the most readily accessible and least time consuming trainings: 96% participated in webinars, 46% in in-person trainings, and 34% in learning collaboratives. 7 of the 8 most accessed trainings (i.e., attended by 25-34% of all adopters) were webinars on fiscal efficiency and outcome reporting; 4 of the 8 least accessed trainings (i.e., attended by 4-11% of all adopters) were learning collaboratives. Further, 41% of the adopters exclusively accessed webinars, though only 21% selected all three levels of intensity and 22% selected both webinars and in-person trainings.

Adopters of the most intensive type of trainings, learning collaboratives, were likely to have previously sampled less intensive trainings. 22 (73%) of the 30 BEEP adopters and 46 (94%) of the 49 BEAM adopters previously accessed a business webinar. All 18 PEDS adopters and all 14 4Rs and 2Ss (Individual Model) adopters participated in a clinical webinar prior to participating in these learning collaboratives.

Participation in learning collaboratives over time was also examined. In the two completed business learning collaboratives, 14 (47%) of the 30 BEEP adopters, and 12 (25%) of the 49 BEAM adopters, attended at least half of the learning collaborative sessions. The BEAM adopters, however, could revisit missed training modules online. In the two completed clinical learning collaboratives, 21 (84%) of the 25 adopters of 4Rs and 2Ss (Group Model), and 7 (50%) of the 14 adopters of 4Rs and 2Ss (Individual Model), achieved at least 50% participation rate.

By Adopter Group

Based on the highest level of intensity selected, 109 (41%) clinics were low-adopters or webinar-only adopters; 67 (25%) were medium-adopters or adopters whose most intensive training was an in-person training; 59 (22%) were high-adopters or adopters of one learning collaborative; and 33 (12%) were super-adopters or adopters of at least two learning collaboratives.

Table 2 shows that the distribution of the four adopter groups across the number of trainings accessed indicated a statistically significant association (p<.001). Low-adopters not only selected low intensity trainings, but they also participated in fewer number of trainings, with 85% of them adopting only 1-4 trainings. Super-adopters not only signed up for at least two of the most intensive trainings, learning collaboratives, 85% of them also attended ≥9 trainings. High-adopters of one learning collaborative were the next most active group, with 37% of them participating in ≥9 trainings.

Table 2.

Number of trainings accessed by adopter groups^a

	Adopter Group
	Low		Medium		High		Super		Total
Number of Trainings Adopted	n	%	n	%	n	%	n	%	n	%
1-4 Trainings	93	85	28	42	9	15	0	0	130	48
5-8 Trainings	15	14	18	27	28	48	5	15	66	25
≥9 Trainings	1	1	21	31	22	37	28	85	72	27
Total	109	100	67	100	59	100	33	100	268	100

^ap<.001

Discussion

To our knowledge, this is the first study that prospectively examines the naturalistic adoption behavior of an entire population of state outpatient mental health clinics licensed to treat children and adolescents. Given the inconsistent operationalizations of adoption in the field, this study described the uptake of trainings in multiple ways to better understand the nuances of clinic training adoption behavior in New York State. By a simple count of trainings adopted, clinic behavior followed a negative binomial distribution commonly found in the utilization literature (15, 16). Clinics fell into different categories depending on the adoption definition. They appeared to be adopting trainings based on available time and resources. Webinars, the least intensive type of training, were accessed by 96% of the adopters, nearly three times the adopters’ participation in learning collaboratives, the most intensive type of training. Over 40% of adopting clinics exclusively attended webinars. Adopters were equally open to business and clinical trainings, suggesting that clinics were cognizant of the need to improve both fiscal and clinical performances. Further, the adopter groups developed for this study, from low- to super-adopters, conveyed meaningful adopter profiles that reflect both the number and intensity of trainings adopted.

Implications for State Systems

Our findings have important implications for efforts in New York State and other states to improve uptake of trainings:

Increasing the sheer number of trainings is unlikely to improve adoption

That a median of five trainings were accessed by clinics in two years suggests an optimal number of trainings worthy of clinics’ investment regarding lost staff time or lost revenues. Rather than offering more trainings, a state should develop a reasonable number of trainings in the areas of highest demand and highest importance to the state (e.g., improving clinic fiscal health and capacity to report outcomes), in order to improve the depth and quality of those trainings. New or different content areas may also address the needs of non-adopters.

Both business and clinical trainings are critical to clinics’ viability in the current healthcare context

Clinics demonstrated robust and comparable adoption of business and clinical trainings, which suggested that both types of trainings addressed clinics’ needs. In the current healthcare climate of accountability and quality (17), clinics need state technical assistance to adapt to the pre-requisites of healthcare reform. For example, the Behavioral Health Organization (BHO) initiative in New York State (18) is being implemented to transition the public behavioral health system towards managed care. BHOs will be tasked with reviewing and monitoring clinics using standard performance indicators (e.g., outpatient engagement, using electronic medical records to communicate outcomes). Adopters of CTAC business initiatives have shown initial success in financial restructuring and benchmarking productivity to improve their decision-making and monitoring of fiscal health (19). Similarly, adoption of clinical trainings addresses the need to improve patient-centered outcomes.

Intensity and accessibility of trainings are important adoption criteria

Clinics favored the least intensive and the most accessible webinar trainings, which were strategically delivered during the lunch hour. Convenience (i.e., accessing a webinar at one’s desk vs. traveling to an on-site training) and minimal opportunity cost (i.e., one hour vs. full-day) seemed to drive clinics’ adoption behavior. Sequential adoption behavior was evident in both business and clinical trainings, as clinics were more likely to adopt the more intensive learning collaboratives when they had an earlier opportunity to sample lower intensity trainings. This “small-step” preparation for more intensive adoption indicates the influence of trialability on adoption decision-making, and should be considered in introducing important future initiatives (8, 9). Although an intensive application process prepared clinics for the demands of a learning collaborative, clinics had difficulty with consistent participation. In three of the four completed learning collaboratives, less than half of the adopters attended more than 50% of the collaborative training sessions, regardless of the type and length of the learning collaborative. De-adoption is clearly both a reality and risk in such long long-term trainings.

Clear adopter groups communicate meaningful and important adopter profiles

The four adopter groups (low-, medium-, high-, and super-adopters), defined by the highest intensity of trainings adopted, parsimoniously captured critical information about their respective preferences for the number and intensity of trainings. Low-adopters accessed low intensity (webinar-only) and a low number of trainings (1-4 trainings); high and super-adopters selected high intensity (learning collaboratives) and a high number of trainings (≥9 trainings). These groupings can help states characterize clinic adoption behavior in a meaningful way and thus create efficiencies in targeting roll-out efforts. Clinic characteristics of adopter groups could potentially provide critical information to guide the state in the development of sorely lacking tailored roll-out strategies (20).

Depending on the goal, state systems should target engagement efforts at clinics with different adoption patterns

If a state aims to engage all clinics in evidence-based practices and QIs, it must focus its efforts on non-adopters and low-adopters. Understanding the barriers to adoption by these clinics will improve adoption strategies. Alternatively, if the goal is to change clinics from low-/medium-adopters to high-/super-adopter, varying the timing and intensity of training efforts to build on less intensive training is likely to be successful. The low participation in learning collaboratives suggests that engagement of clinics in long-term trainings will be critical.

Ultimately, this study presents nuances of adoption behavior that may help states better understand specific clinic characteristics that may explain variations in clinic adoption behavior. Predictability of different adoption outcomes examined in this study may depend on internal characteristics (e.g., clinic fiscal health, client profiles) and external characteristics (e.g., Medicaid funding, clinic region) (8, 21). Future work by our group will examine these characteristics to understand the facilitators and barriers to adoption, and how the state can use this knowledge to design effective roll-outs.

Limitations

Besides CTAC, New York State has a history of promoting evidence-based practices and QIs through other training mechanisms (5), which may have had an impact on clinic adoption of CTAC-specific trainings. Other non-state related roll-outs (e.g., professional organizations, conferences) that are not captured in our study may target the same population of clinics. It is unknown whether state and non-state related adoption behaviors influence each other. It is also unclear how generalizable our findings are to other states’ efforts to improve uptake, given the differences in relationships between clinics, counties, and the state. New York State clinics, unlike those in other states, tend to operate independently rather than under the directives of their counties. Thus, clinic adoption of trainings in New York State is possibly more heterogeneous than that in other states. Nevertheless, our findings are consistent with existing theories of adoption behavior (8, 21).

Conclusions

This study represents one of only a few attempts to characterize adoption patterns of mental health innovations within an entire state outpatient child mental health system. Its scope in New York State extends beyond the study of evidence-based practices undertaken in Ohio (22) to include QIs and other business improvement practices. While this study describes baseline adoption behaviors, how these behaviors change over time, amidst mounting pressures from state and national healthcare reforms, will be of paramount importance to optimize the design of future states’ efforts. Ideally, adoption data should be used to examine how uptake of trainings translates into more effective clinical practices and ultimately into improved patient outcomes.