Illness management and recovery in community practice

Abstract

Objective

To examine provider competence in providing Illness Management and Recovery (IMR), an evidence-based self-management program for people with severe mental illness, and the association between implementation supports and IMR competence.

Methods

IMR session recordings, provided by 43 providers/provider pairs, were analyzed for IMR competence using the IMR treatment integrity scale. Providers also reported on receipt of commonly available implementation supports (e.g., training, consultation).

Results

Average IMR competence scores were in the “Needs Improvement” range. Clinicians demonstrated low competence in several IMR elements: significant other involvement, weekly action planning, action plan follow-up, cognitive-behavioral techniques, and behavioral tailoring for medication management. These elements were commonly absent from IMR sessions. Competence in motivational enhancement strategies and cognitive-behavioral techniques differed based on the module topic covered in a session. Generally, receipt of implementation supports was not associated with increased competence; however, motivational interviewing training was associated with increased competence in action planning and review.

Conclusions and Implications for Practice

IMR, as implemented in the community, may lack adequate competence and commonly available implementation supports do not appear to be adequate. Additional implementation supports that target clinician growth areas are needed.

Illness management and recovery (IMR) is an evidence-based psychosocial intervention with demonstrated effectiveness in improving illness self-management (McGuire et al., 2013). IMR was created by Mueser and colleagues (2002) who examined the literature to identify common elements of professional-based programs designed to help people manage their mental illness. These elements were used as the foundation for IMR (Gingerich & Mueser, 2004). In a subsequent effort, McGuire and colleagues (2014) utilized an expert survey to establish agreement regarding a total of sixteen critical elements of IMR. Each of these elements has been independently linked with consumer outcomes. It therefore follows that the provision of these elements accounts for some, if not much, of the effects of the IMR program. To this end, fidelity to the IMR model at both the program level (Hasson-Ohayon, Roe, & Kravetz, 2007) and session level (based on rating of audio-recorded sessions) [author cite] have been associated with improvement in consumer outcomes.

Critical Elements of IMR

One common aim of illness self-management programs is providing information regarding the consumers’ conditions, related symptoms, and possible treatments. IMR provides a structured curriculum which is the focus of IMR sessions. Moreover, educational techniques taken from adult learning theory are used to assist consumers in learning and retaining information.

Goal setting is a standard and widely accepted part of illness self-management programs (Levack, Dean, Siegert, & McPherson, 2006) and is considered philosophically essential for recovery-based treatment (Anthony & Liberman, 1992). A robust literature (Locke & Latham, 2002) demonstrates setting goals is associated with increased progress toward those goals. Moreover, increased progress toward goals has been associated with goals which are challenging, important, broken into smaller, proximal goals, and when the consumer feels confident in his ability to accomplish them. To this end, IMR includes goal setting and follow-up—a process in which the consumer and clinician work to identify an overarching recovery goal to guide the clinician and consumer’s work in IMR—as well as action planning/homework and action planning follow-up, in which more specific, short-term goals are set and followed-up on.

Mueser and colleagues (2002) note the promise of motivational enhancement strategies (MESs) and cognitive-behavioral techniques in supporting medication adherence. Moreover, both have utility above-and-beyond medication adherence. MESs are an empirically supported treatment to decrease substance abuse and increase healthy behaviors (Rubak, Sandbæk, Lauritzen, & Christensen, 2005), both of which are important components of illness self-management. Corrigan and colleagues (2001) suggested that motivational enhancement strategies are a promising vehicle by which to explore and establish recovery goals for people with SMI.

Another critical element of IMR is cognitive behavioral therapy (CBT) techniques (McGuire et al., 2014). Numerous clinical trials and some meta-analysis have shown that CBT is an effective and cost-efficient treatment for symptom reduction and management in people with a range of severe mental illnesses (van der Gaag, 2014), including schizophrenia (Wykes, Steel, Everitt, & Tarrier, 2008), major depressive disorders (Cuijpers et al., 2013), and bipolar disorder (Szentagotai & David, 2010). Moreover, there is some evidence to suggest that CBT is efficacious outside of controlled research environments in settings such as community mental health centers (Lincoln et al., 2012; Morrison et al., 2004). In part to increase the accessibility of evidence-based treatments for severe mental illness (van der Gaag, 2014), a wide range of mental health and frontline staff have been successfully trained to utilize CBT techniques, including case managers (Montesano, Sivec, Munetz, Pelton, & Turkington, 2014), peer support specialists (Perry, Murakami-Brundage, Grant, & Beck, 2013), nurses (Turkington, Kingdon, & Turner, 2002), as well as psychiatrists and mental health technicians (Chang, Grant, Luther, & Beck, 2014). Nonetheless, dissemination has lagged, particularly in the United States (Gottlieb et al., 2013; Muerser & Noordsy, 2005). Similarly, in part to increase dissemination of evidence based practices (Mueser et al., 2006), CBT techniques have been incorporated into IMR as a means of applying learning from the IMR curriculum to promote consumers’ illness management and recovery.

A strong working alliance is key to any clinical intervention (Martin, Garske, & Davis, 2000). Moreover, a philosophical pillar of IMR is maintaining a recovery orientation, an attitude consistent with “a process of change through which individuals improve their health and wellness, live a self-directed life, and strive to reach their full potential” (SAMHSA, 2012). Finally, agreement between clinicians and consumer regarding the goals of treatment (see goal-setting above) is a defining element of working alliance [cite Horvath].

IMR Implementation in the Community

Little is known regarding the implementation of IMR in general or of each constituent element outside of effectiveness trials. Implementation in clinical trials differs in important ways from implementation in a naturalistic context. In clinical trials, clinicians are usually highly trained, often included in the study because of demonstrated skill and generally receive implementation supports such as supervision, session feedback, consultation, and fidelity monitoring (Webb, DeRubeis, & Barber, 2010). Consumers and clinical contexts in clinical trials are selected based on assumed success in implementation. This is in contrast to the typical clinical setting in which clinicians, consumers, and the clinical settings vary substantially in their readiness to implement a given practice. For instance, inclusion/exclusion criteria used in clinical trials may or may not be representative of the population of consumers actually receiving services in practice. Additionally, clinicians in the community may receive far less training and supervision than that provided in clinical trials and some research has indicated such implementation supports in the community are often fairly weak (McGuire, White, White, & Salyers, 2013). Poor implementation of a model can lead to a “voltage drop” in which consumer outcomes demonstrated in clinical trials are not realized in actual practice.

In the present study we sought to examine provider IMR competence in community agencies that were implementing IMR as a part of standard practice and were receiving naturally occurring implementation supports. More specifically, we examined the competence with which IMR was provided, in general, as well as the competence in providing each critical element of IMR. For elements with poor competence, we conducted a more granular analysis of the provision of techniques and principles within these elements to discern whether competence deficits were global or tied to under-use of particular aspects of the element. IMR includes ten module topics which target disparate components of illness self-management; each topic may lend itself more or less to the use of a given IMR element. We therefore examined whether provider competence in providing each element varied significantly based on the session’s topic/ module. Our goal was to identify common competence strengths and deficits in IMR in order to better develop and target implementation support efforts. We therefore examined the relationship between implementation supports received by clinicians and their competence in each element.

Methods

Setting and Sample

Mental health agencies from two Mid-Atlantic states and a Midwestern state providing IMR were recruited to participate in the study. Twenty-one agencies decided to participate. Participating agencies included community mental health centers, supportive housing programs, vocational rehabilitation centers, partial care programs, state-operated psychiatric hospitals, and a Veteran’s Administration Medical Center. IMR trainer-consultants at each site have extensive experience in psychiatric rehabilitation and were originally trained in IMR by an IMR model developer. Subsequent trainings are based on the curriculum and materials provided by the model developer. Specifics varied based on location. One of the Mid-Atlantic States maintained two contracts with an academic partner to provide IMR training and consultation with its state-operated facilities and outpatient agencies, respectively. Supports, all of which are optional, included a two-day training, consultation calls, and fidelity visits. Three sites were located in the Midwestern state. One agency provided a two-day IMR training followed by bi-weekly IMR-specific group supervision. The second agency received a one-day IMR training followed by optional, on-going consultation with an IMR expert. The remaining site received a series of one hour in-services with available consultation with a program expert.

All IMR providers at participating agencies were recruited to participate in the current study. Agency leadership identified which providers may be currently offering IMR and forwarded a study e-mail. Providers contacted the study team and informed consent was obtained.

Procedures

All providers were asked to conduct IMR as they normally would, with two modifications: First, providers audio-recorded their IMR sessions for three months. Second, so that all IMR modules could be observed across the study sample, providers were instructed to start with module 1 and then were randomly assigned two to three additional modules from the IMR curriculum to complete. Providers were instructed that, should they complete all assigned modules, they should select additional modules from the IMR curriculum to complete for the remainder of the study period.

Measures

IMR Competence

Provider competence was assessed using the IMR treatment integrity scale (IT-IS). The IT-IS is utilized to assess competence in IMR elements in a particular session (McGuire et al., 2012). The scale contains 16 items rated on a 5-point scale ranging from 1 (unsatisfactory) to 5 (excellent). Each of the 16 items on the scale parallels a critical element of IMR. All raters were trained on the IT-IS protocol. Once trained, raters scored audio-recorded IMR sessions utilizing a scoring rubric containing the 16 elements and indicators of excellence for each element. Indicators of excellence include both characteristics of use of the element (e.g., for recovery orientation, the provider maintains a “non-judgmental” stance) or specific techniques (e.g., “shaping” for cognitive-behavioral techniques). Several items on the scale were only scored when relevant. Specifically, three items were only rated when the clinician covered specific curriculum such as Coping Skills Training, Relapse Prevention Planning, or Behavioral Tailoring for Medication. Regarding the latter, behavioral tailoring includes modifying behavior to integrate medications into one’s daily routine (e.g., placing pillbox next to one’s toothbrush). Additionally, two items were only rated when IMR was given in a group (engagement of group members and mutual support between group members). Finally, Action Plan Review is not rated in the first session. Action plans are activities to be completed by the consumer between sessions (a.k.a., “homework”). Previous psychometric analyses of the IT-IS demonstrated excellent inter-rater reliability, good factor structure validity, acceptable internal consistency, and discriminate validity (McGuire et al., 2012). Data from the current study demonstrated IT-IS scores were positively associated with consumers’ improvement in self-management [author cite].

Session Module

Providers kept logs of session attendance, date, and the topic of the module covered for a particular session. Module topics were confirmed by research staff by listening to the audio-recorded session. In the event that it was unclear what module was covered, a second research staff listened to the recording to confirm the module.

Implementation Supports

Supports were assessed via clinician self-report. Clinicians completed paper-and-pencil or on-line surveys at baseline based on surveys utilized in previous IMR research (Salyers, Rollins, McGuire, & Gearhart, 2009). Clinicians indicated if they had received IMR-specific training (yes/no), motivational interviewing training (yes/no), cognitive-behavioral training (yes/no) and length of their longest IMR, motivational interviewing, and/or cognitive-behavioral training, respectively (One hour, Half day, Full day, Two full days, longer than 2 days), IMR-specific supervision (Never, In the past, Currently) and frequency of IMR-specific supervision (More than weekly, Weekly, Monthly, Less than monthly), and IMR consultation (yes/no). In cases where co-facilitators differed on implementation supports, credit was given if either clinician had received the support.

Analyses

Descriptive statistics were used to examine total IMR competence (IT-IS mean score), each element (IT-IS items), and indicators of excellence for elements demonstrating low competence (mean ≤ 2.0). Separate ANOVA models were used to examine variations in total IMR competence and each element across topical module which was the focus of the session. Finally, the relationship between each implementation support and both total IMR competence and low competence elements was examined with t-tests (categorical) and Spearman’s Rho (rank).

Results

Sample

Providers

There were 43 provider-consumer units led by 53 clinicians. Thirty-seven providers (or co-facilitator pairs) administered IMR in a group format and 6 IMR providers administered IMR individually to multiple consumers separately. IMR providers were predominately younger women (see Table 1) with about ten years of experience in mental health and moderate experience providing IMR in terms of years providing IMR and number of times completing the IMR curricula completed. Reported discipline was predominately psychology or counseling with social work the second most frequent discipline. Highest degree reported was generally a bachelor’s or master’s degree. Most providers had received at least two days of IMR training. Providers reported high importance (M = 4.3 out of 5, SD = 0.6) and confidence (M = 4.5 out of 5, SD = 0.5) in providing IMR according to the model.

Table 1.

IMR Provider Demographics and Background

Characteristic	n	Percent
Age (n = 39)
20–30	12	30.8
31–40	7	17.9
41–50	8	20.5
51–60	8	20.5
61–70	4	10.3
Gender (n = 39)
Male	13	33.3
Female	26	66.7
Education (n = 40)
Associates	4	10.0
Bachelor’s	15	37.5
Masters	15	37.5
Other	6	15.0
Discipline (n = 39)
Psychology/Counseling	19	50.0
Social Work	9	23.7
Peer Specialist	3	7.9
Rehabilitation/Nursing	2	5.3
Addictions Counseling	2	5.3
Other	3	7.9
Characteristic	Mean	SD
Years in Mental Health (n = 37)	9.5	8.2
Years Providing IMR (n = 35)	1.8	2.3
Completed IMR Curricula (n = 8)	3.1	7.0
Providing IMR According the Model
Importance (n = 36)	4.3	0.6
Confidence (n = 35)	4.5	0.5

Implementation support was considered at the group level. Most providers (or co-facilitator pairs) reported having received IMR training, motivational interviewing training, and IMR specific supervision, less than half had not received cognitive-behavioral training, or IMR consultation (see Table 2).

Table 2.

Implementation Supports

Support	n	Percent
IMR Training (n = 34)
Yes	28	82.4
No	6	17.6
Length of Training (n = 28)
1 day	5	17.9
2 days	13	46.4
>2 days	10	35.7
Motivational Interviewing Training (n = 34)
Yes	20	58.8
No	14	41.2
Length of Training (n = 39)
1/2 day	4	17.4
1 day	7	30.4
2 days	6	26.1
>2 days	6	26.1
Cognitive Behavioral Training (n = 34)
Yes	14	41.2
No	20	58.8
Length of Training (n = 20)
1 hour	1	5.0
1 day	10	50.0
2 days	2	10.0
>2 days	7	35.0
IMR Consultation (n = 33)
Yes	8	24.2
No	25	75.8
IMR Supervision (n = 34)
Current or past	20	58.8
Never	14	41.2
Frequency of Supervision (n = 19)
More than weekly	1	5.3
Weekly	10	52.6
Monthly	7	36.8
Less than monthly	1	5.3

Note. In cases where co-facilitators had differing degrees of implementation support, the responses from the co-facilitator with the highest level of support were considered

Sessions/Groups Observed

Providers reported on (N = 224) sessions across the 3-month observation period, of which 176 session were rated on IMR competence by research staff. Providers completed nine sessions on average (M = 9.1, SD = 4.3). Group and individual IMR did not differ on number of sessions completed. Rated sessions were predominately from the first five sessions (n = 119, 68.8%; session number could not be determined for 4 sessions; See Table 3) .Groups covered two to three modules, on average (M = 2.6, SD = 1.4, min = 1, max = 7) during the observed period; the number of modules did not differ based on individual versus group format. A number of sessions (n = 29, 12.9%) did not pertain to any IMR module. The number and percent of non-IMR sessions did not differ based on format (group vs. individual).

Table 3.

Number of Modules Rated

Modules	n
Orientation	6
Module 1: Recovery Strategies	77
Module 2: Practical Facts about Mental Illness	15
Module 3: The Stress-vulnerability Model	6
Module 4: Building Social Supports	27
Module 5: Using Medication Effectively	2
Module 6: Drug and Alcohol Use	2
Module 7: Reducing Relapses	12
Module 8: Coping with Stress	3
Module 9: Coping with Persistent Symptoms	2
Module 10: Getting your Needs Met in the Mental Health System	12
No IMR	12

Element Competence

The average total IMR competence was in the needs improvement range (M = 2.7, SD = .55); a minority of sessions rated in the satisfactory or better range (mean score ≥3.0; n = 53, 30.1%). Elements of IMR competence (averaged across modules) varied substantially (Table 4). Four of the elements were rated at least satisfactory on average (mean score ≥3.0): therapeutic relationship, recovery orientation, enlisting participation from all group members, and educational techniques. In contrast, five elements were rated as unsatisfactory (< 2.0) on average: significant other involvement, weekly action planning, action plan follow-up, cognitive-behavioral techniques, and behavioral tailoring for medication management. Less than a quarter of sessions demonstrated at least satisfactory competence in these five elements. In order to better understand the elements with the lowest competence ratings, we examined use of each indicator of excellence used to guide each element rating (Table 5). Indicators of excellence for these elements were rarely present. Significant involvement competence is demonstrated by the provider eliciting the involvement of a person who is important to the consumer (not a paid mental healthcare provider) in pursing the consumer’s goal and/or attending an IMR session. Significant other involvement was elicited (e.g., “is there someone who can help you reach this goal?” or “would you like your wife to come to our next session?”) in less than one in five sessions. For weekly action planning, indicators of excellence were universally low, occurring in no more than about 1/3 of sessions. Indicators regarding personalizing the action plan to the consumer (matched to client’s preference, tied to client’s goal, and client expresses active agreement) were notably infrequent. Action plan follow-up occurred (at all) in only about one third of sessions and was generally lacking in quality. Regarding cognitive-behavioral techniques, reinforcement was used in over three quarters of session. Modeling and role-playing were used infrequently (about 5% of sessions) and other techniques were used almost never. Finally, behavioral tailoring for medication does not include indicators of excellence, but is rather rated holistically as to its presence/absence and the overall skill with which it was provided.

Table 4.

Implementation of IMR Critical Elements

IT-IS Item	n	M	SD	Min	Max	Percent of Sessions with ≥Satisfactory Rating
Items with Average Rating of Satisfactory or Better
Structured Curriculum	177	4.16	1.12	1.00	5.00	88.6%
Involving All Members	126	3.99	.91	1.00	5.00	92.8%
Therapeutic Relationship	177	3.82	.81	2.00	5.00	94.3%
Recovery Orientation	177	3.36	.88	1.00	5.00	81.3%
Educational Strategies	177	3.34	1.05	1.00	5.00	75.0%
Structure/Use of Time New	137	3.16	.86	1.00	5.00	77.3%
Items with Average Rating of Needs Improvement
Enlisting Support Between Members	126	2.93	1.19	1.00	5.00	59.2%
Structure/Use of Time	177	2.90	.79	1.00	5.00	71.3%
Goals New	177	2.43	1.36	1.00	5.00	43.2%
Coping Skills Training	22	2.42	.47	2.00	3.00	36.4%
Relapse Prevention	13	2.35	.47	2.00	3.00	30.8%
Goals	176	2.17	1.37	1.00	5.00	34.7%
Motivational Enhancement Strategies	177	2.01	.92	1.00	5.00	22.7%
Items with Average Rating of Unsatisfactory
Weekly Action Planning	177	1.94	1.11	1.00	5.00	34.1%
Involving Significant Others	177	1.93	.78	1.00	5.00	13.6%
Behavioral Tailoring	4	1.83	.58	1.00	2.33	0.0%
Action Plan Review	155	1.68	.97	1.00	5.00	17.5%
CBT	177	1.11	.37	1.00	3.33	1.7%

Table 5.

Sessions with Each Indicator of Excellence Present (Lowest Rated Elements Only)

IT-IS ITEM	Indicators	Frequency	Percent
Involvement of Significant Others	Clinician Elicits Involvement	45	17.0
	Clinician Reinforces Involvement	29	10.9
	Specific Plan for Involvement	6	2.3
Weekly Action Planning	Specific	93	35.1
	Measurable	86	32.5
	Achievable between Sessions	107	40.4
	Relevant to IMR	104	39.2
	Matched to client’s preferences	35	13.2
	Matched to client skills, etc.	57	21.5
	Client Expresses Agreement	40	15.1
	Personalizes Information	68	25.7
	Tied to client’s goals/objectives	46	17.4
Action Plan Review	Reviews Action Plan	91	34.3
	Reinforces Efforts to Complete	62	23.4
	Problem Solve Obstacles	35	13.2
	Integrates Material with Goals	40	15.1
	Positive Effect on Recovery	15	5.7
	Complete/Thorough	5	1.9
Cognitive Behavioral Techniques	Educating	3	1.1
	Shaping	1	.4
	Modeling	15	5.7
	Role Playing	13	4.9
	Cognitive Restructuring	2	.8
	Relaxation Training	3	1.1
	Behavioral Experiment	3	1.1
	Reinforcement	179	67.8

Variation in Element Use by Module

Total IMR competence did not differ based on the module covered in the session rated; however, two IMR elements did differ based on module: motivational enhancement strategies (F(9,153) = 2.58, p = .01) and cognitive-behavioral techniques (F(9,153) = 2.25, p = .02; Figure 1). Clinicians displayed the highest competence in motivational enhancement strategies while covering recovery strategies, practical facts about mental illness, and drug and alcohol abuse, while showing the lowest competence when covering reducing relapse and coping with problems and persistent symptoms. Competence in cognitive-behavioral techniques varied less and was about one (unsatisfactory) during all modules except coping with stress and building social support, where it was slightly higher. Notably, competence was not higher in sessions focused on coping with symptoms.

Figure 1.

Average competence in motivational enhancement strategies and cognitive-behavioral techniques by module covered in session

Associations between Implementation Supports and Competence

Types of implementation support include training on IMR, motivational interviewing, cognitive-behavioral training, and IMR supervision and consultation. In cases where co-facilitators had differing degrees of implementation support, the highest level of support was considered. No implementation supports were associated with overall (IT-IS total), CBT, or behavioral tailoring competency. Several implementation supports were associated with action planning and action plan review. Clinicians (or co-facilitators) who reported receiving motivational interviewing training displayed higher competence in weekly action planning (M = 2.21, SD = .86) than those without motivational interviewing training (M = 1.60, SD = .53; t(31.67) = −2.27, p = .03), and higher competence in action plan review (M = 1.79, SD = .73) than those without motivational interviewing training (M = 1.31, SD = .39; t(30.48) = 2.47, p = .02). Clinicians who had ever received IMR supervision had greater competency in weekly action planning (M = 2.17, SD = .90) than those who had never received IMR supervision (M = 1.67, SD = .52; t(31.22) = −2.07, p = .05).

Discussion

The competence demonstrated by these community-based IMR providers leaves substantial room for improvement. While four elements (therapeutic relationship, recovery orientation, enlisting participation from all group members, and educational techniques) were provided competently, on average, overall, IMR competence was low; moreover, several elements of the IMR model were infrequently implemented at all. The competent use of motivational enhancement and cognitive-behavioral techniques occurs more frequently in sessions focused on particular topical modules. However, with the exception of some prior motivational interviewing training and IMR-specific supervision, implementation supports showed little association with IMR competence. We consider several explanations for these findings below.

First, it should be noted that IMR has a rich and extended curriculum, which could provide a challenge in terms of true mastery for clinicians. Moreover, the standards set for “competence” in the current study are not empirically-derived. Although the competence scale used (the IT-IS) has proven reliability and has been positively linked with consumer outcomes, the specific thresholds were not established based on known relationships to consumer outcomes. Findings should be considered in light of this limitation.

Clinicians rarely elicited significant other involvement in IMR; this finding is consistent with an analysis of IMR implementation in the Veteran Health Administration (McGuire, White, White, & Salyers, 2013). Clinicians’ infrequent attempts may be attributable to lack of perceived importance or confidence in providing IMR according to the model. Regarding the former, in an expert survey regarding the critical elements of the IMR model, significant other involvement was not considered “essential” or “defining” of IMR (McGuire et al., 2014), despite its demonstrated effects in supporting consumer recovery (Dixon et al., 2001). Regarding the latter, involving significant others is a particularly time-intensive and a difficult endeavor (Dixon et al., 2001); therefore, providers may give up after repeated failures in this realm. Behavioral tailoring for medication management is similar to eliciting significant other involvement in that it was not considered essential or defining of IMR by IMR experts (Alan B McGuire, Luther, et al., 2014) and yet has been supported as an effective solo intervention (Velligan et al., 2008) and the only medication adherence intervention in Mueser and colleagues review (2002) to be tested in more than one randomized-controlled trial. An important limitation of the current study, though, is the lack of indicators of excellence explicating competence in behavioral tailoring. However, behaviorally tailoring was generally not attempted at all, thus minimizing this limitation.

Cognitive-behavioral techniques and action planning and review are complex clinical interventions (Morgenstern, Morgan, McCrady, Keller, & Carroll, 2001). Indeed, to achieve competency in formal cognitive-behavioral therapy, community-based clinicians often engage in targeted multi-day training with accompanying multi-month consultation and evaluation of work samples (e.g., therapy session reviews; case-write ups (Stirman, Buchhofer, McLaulin, Evans, & Beck, 2009; Stirman et al., 2010); thus, it may be that competent provision of these techniques requires additional targeted training or implementation supports, particularly for staff with little training in psychology or psychopathology. Further, it is notable that several other IMR elements that are not part of the CBT item are often considered CBT techniques such as creating and following an agenda (Beck, 2011), setting personal recovery goals (Grant, Reisweber, Luther, Brinen, & Beck, 2014), and conducting weekly action planning or homework (Flach et al., 2015). Indeed, these techniques are included on some CBT competence scales (Haddock et al., 2001; Young & Beck, 1980), suggesting that more general CBT competence may be higher if these techniques were included. Nevertheless, both these techniques and the CBT techniques on the IT-IS were utilized very infrequently. Moreover, the aforementioned IMR expect survey found that IMR experts deemed CBT techniques as a non-defining element of IMR (McGuire et al., 2014). Taken together, IMR providers might not utilize CBT techniques because of a lack of perceived relevance or uncertainty regarding how to incorporate CBT techniques into IMR sessions. Future work should re-evaluate the role of CBT techniques within IMR and develop methods to enhance clinician competence with CBT techniques.

Some model elements may be better suited to support certain topics in the IMR curriculum; to this end, clinicians varied in their competent application of two elements based on the topic covered in the session: cognitive-behavioral techniques and motivational enhancement strategies. In some cases, this variation appears to make clinical sense. For instance, clinicians used motivational enhancement strategies more when discussing alcohol and drug use and cognitive-behavioral techniques such as role-playing and modeling more when discussing building social support. However, clinicians used motivational enhancement strategies less when discussing using medication effectively. Motivational enhancement strategies were included as an element of IMR to a great degree due to their demonstrated effectiveness in increasing medication adherence (Mueser et al., 2006). Some prior research has indicated providers believe that consumers served by IMR are largely medication adherent (McGuire, Kean, Bonfils, Presnell, & Salyers, 2014); therefore, providers may not utilize techniques aimed at increasing medication adherence due to a lack of perceived need. It should be noted that there is a difference between a consumer agreeing to take medication in principle and actual adherence. It is also important to note that raters are given latitude in rating the skill with which elements are provided (i.e., if motivational-enhancement strategies are used correctly or incorrectly in a given session); however, the IT-IS does not provide credit for a provider abstaining from the use of an element when it may be clinically indicated. In other words, if motivational-enhancement techniques are not used at all in a given session, a rating of one is given, even if a rater might believe it was appropriate to not use motivational-enhancement techniques in the session. Additional research is needed regarding appropriate clinical decision-making regarding the use of each element.

An important issue is whether IMR providers are regularly attempting under-implemented elements without success or just not attempting to include them. Indicators of excellence used in the IT-IS scale range in difficulty; therefore, if the provider attempts to implement the element, but does so with little skills, some indicators of excellence will still be present. However, our findings indicate the indicators of excellence for poor implementation elements were almost universally absent. These findings suggest that the gap is not one of provider skill, but a failure to implement the element at all.

The implementation supports examined in the current study demonstrated little association with IMR competence. Some motivational interviewing training did show promise as it was associated with action planning and review. Few details were obtained regarding the extent or quality of the implementation supports provided and it possible that these supports were failed to meet minimal levels to have any effect on implementation. No empirical investigations have demonstrated an adequate level of training or consultation to obtain and maintain competence; however, some models merit examination (Rychener, Salyers, Lariola, & Little,2009). Alternatively, training, supervision, and consultation may not be sufficient supports for competent IMR provision. Moreover, the quality and focus of the supervision may not be appropriate. For instance, although supervision is IMR specific, it may focus on administrative matters (e.g., completing appropriate progress notes) rather than on clinical matters or fidelity to the model. Supervision and consultation in the community can occur without direct practice observation, which still allows less competent provision to go unobserved. Implementation supports which provide specific feedback to clinicians regarding the competence during training or later, such as audit-and-feedback (Ivers et al., 2014) and fidelity monitoring (Bond, Drake, McHugo, Rapp, & Whitley, 2009; Lu et al., 2012) may be particularly useful. Also, it may be that some variation is explained by provider factors. For instance, in our analyses provider experience in IMR was positively associated with IMR competence (as rated using the IT-IS; McGuire et al., N.d.).

Findings from the current study extend beyond IMR. Cognitive-behavioral techniques, action planning and review, significant other involvement, and behavioral tailoring are common to many other psychosocial interventions for people with severe mental illness (Drake et al., 2001) as well as self-management programs for a variety of chronic conditions (Barlow, Wright, Sheasby, Turner, & Hainsworth, 2002). Examinations of implementation of these practices, like IMR, are generally done in the context of effectiveness research, thus leaving open the possibility that their implementation lags in community practice (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005).

In some agencies participating in the current study there are no expectations to implement evidenced-based practice, but if a provider chooses to implement IMR they are subject to additional fidelity monitoring. Therefore, providers may experience “performance punishment;” i.e., by providing an evidence-based practice they are subjected to additional requirements. Despite the fact that clinicians in this study demonstrated low competence in the provision of IMR, it is also likely that these clinicians provided services that were proportionally more rigorous than services provide by them and their colleagues not based on an evidence-based model. In a study on the effect of IMR on readmission to a state hospital, IMR provided with relatively low programmatic fidelity still produced a reduction in the risk of readmission compared to treatment as usual (Bartholomew & Zechner, 2014). A second, and related issue, is that programs rarely have requirements about monitoring outcomes or assessing the quality of treatment provision (Kilbourne, Keyser, & Pincus, 2010) because of this many clinicians may not distinguish EBPs from other agency programs that have no demonstrable effectiveness.