Getting to Uptake: Do Communities of Practice Support the Implementation of Evidence-Based Practice?

Abstract

Introduction

Practitioners are increasingly encouraged to adopt evidence-based practices (EBP) leading to a need for new knowledge translation strategies to support implementation and practice change. This study examined the benefits of a community of practice in the context of Ontario’s children’s mental health sector where organizations are mandated to adopt a standardized outcome measure to monitor client response to treatment.

Method

Readiness for change, practice change, content knowledge, and satisfaction with and use of implementation supports were examined among practitioners newly trained on the measure who were randomly assigned to a community of practice (CoP) or a practice as usual (PaU) group. CoP practitioners attended 6 sessions over 12 months; PaU practitioners had access to usual implementation supports.

Results

Groups did not differ on readiness for change or reported practice change, although CoP participants demonstrated greater use of the tool in practice, better content knowledge and were more satisfied with implementation supports than PaU participants.

Conclusion

CoPs present a promising model for translating EBP knowledge and promoting practice change in children’s mental health that requires further study.

Introduction

In children’s mental health (CMH) services research, evidence-based practice (EBP) refers to a body of scientific knowledge about service practices including treatment, referral, assessment, case management, and clinical outcomes (Hoagwood & Johnson 2003). In Ontario, as in many American jurisdictions (e.g., Hawaii, Michigan, New York State, Ohio), governments have systematically implemented standardized outcome measurement tools to monitor client response to treatment and service outcomes. This policy direction stems from a practical need to understand the functional outcomes of children who receive service and to apply this information to their care and to system planning and service delivery more generally. The evidence base in support of outcome management and measurement comes from research showing that practitioner attention to treatment response is related to improved patient outcomes, and that providing practitioners with a reliable standardized tool for assessing client response to treatment can have a significant impact on the quality of care delivered (Barlow, Hayes & Nelson, 1984; Ogles, Lambert & Master, 1996). Outcome measurement leads to improved treatment, enhances clinical science, provides accountability, and maintains the ethical responsibility of practitioners to examine service quality. Thus, measuring outcome has important implications for the quality and impact of MH service delivered to children.

Practitioners are increasingly being encouraged to develop their practice knowledge, to adopt EBPs, including empirically supported treatments, prevention programs, and assessment methods (Garland, Kruse, & Aarons, 2003). They face important challenges in applying this knowledge, determining how best to implement EBPs, increasing organizational and practitioner readiness or receptivity for change, addressing the clinical utility and efficiency of these practices, and demonstrating the impact following uptake. The field continues to rely on practices that have little supporting evidence or, at worst, have poor outcomes (Busch, 2002; Dishion, McCord & Poulin, 1999) despite evidence that most children who receive an empirically supported treatment get significantly better and do so more quickly than with other treatments or no treatment (Chambliss & Ollendick, 2001; JCCP, 1998).

In reality, clinicians do not change their practice readily (Haines & Donald, 1998) and the prevention and treatment approaches with the best empirical support are rarely used in typical clinical practice (Kazdin, Bass, Ayers, & Rodgers, 1990; Weersing & Wiesz, 2002). Few ‘toolkits’ or practical examples detailing how to get evidence into practice in a way that leads to practice change are available to guide the adaptation or translation of evidence-based treatments to real-life clinical practice (Conner-Smith & Wiesz, 2003).

Strategies for teaching evidence-based interventions have fallen short, with little evidence of transfer between the classroom and the workplace (Soden & Halliday, 2000). While clinicians can learn the rudiments of an intervention in training, it is in the practice context that they will develop their knowledge and begin to shift their approach. Traditional approaches to improve uptake of research findings more generally have focused on better availability and presentation of evidence by identifying, synthesizing, and disseminating evidence to practitioners in practical accessible formats –e.g., reviews in clinical journals, clinical guidelines, better access to electronic sources of information, continuing medical education or professional development and conferences, but even these efforts are insufficient in and of themselves (Grol & Grimshaw, 2003).

Methods of teaching evidence-based practice need to change, to become more routed in the practice environment and to be situated within a community of learners. Knowledge is expanded through discussion (Bielaczyc & Collins, 1999), and communities of practitioners help to foster discussion. The sum of the community’s knowledge is greater than the sum of individual practitioner’s knowledge (Gherardi & Nicolini, 2000), and as the collective of the group advances, so too does the individual’s knowledge (Bielaczyc & Collins, 1990). Knowledge acquisition within the context of a practice communication helps to foster continuous learning and shape learning organizations that will more readily adapt to new practices and approaches as they emerge from discovery research (Senge, 1990). A key challenge before us is to explore innovative strategies that can reduce uptake barriers and facilitate practice change and the implementation of evidence-based practices. This study explores the use of communities of practice as a method for achieving this goal in the implementation of an outcome measurement tool (Hodges, 2003; Child and Adolescent Functional Assessment Scale) in child and youth mental health.

Community of Practice

The community of practice (CoP) model first emerged in education (Lave & Wenger, 1990) and later in management (Brown & Duguid, 1991) as a useful and effective method of knowledge management. In the last decade, it has begun to diffuse slowly into the health sector. Defined as a group of people who share knowledge, learn together, and create common practices (Wenger, McDermott & Snyder, 2002) communities of practice are shaped by three fundamental elements: a domain of knowledge, which creates common ground, a sense of common identity, and inspires members to contribute and participate; a community of people who care about the domain, thus creating the social fabric for learning, sharing, inquiry, and trust; and the shared practice made up of frameworks, tools, references, language, stories, documents, that community members share. Members of a community of practice are bound together by common interests and a desire to continually interact. CoPs are often informal, with fluctuating membership, and differ from formal work groups, project teams or informal networks in emphasizing the development of members’ capabilities and exchanging knowledge. While they are traditionally described as organic (Wenger, McDermott & Snyder, 2002) emerging from within the membership rather than through top down hierarchical mechanisms, our interest here is in developing intentional or deliberate communities that can support knowledge exchange among practitioners seeking to bring research evidence into their practice.

Since communities of practice can change their agenda to suit the needs of members, they are well positioned to address the knowledge requirements of a broad range of health practitioners. CoPs can exist both in real time, face-to-face format, and virtually on the Internet, thereby transcending limitations imposed by geography and time. Case studies have shown they can assist with the development of professional skills, assist in the transfer of good practice, and have long-term benefits on organizational practice and productivity (Mitchell, 2003).

Communities of practice have found their niche in many private sector organizations for managing sharing of tacit knowledge across divisional and/or geographic boundaries (e.g., British Petroleum, The World Bank, World Health Organization, Clarica, Daimler-Chrysler). Studies of the CoP model applied in health are just beginning to emerge. A review of community of practice studies for the dissemination and uptake of evidence-based practices reported 26 papers in the health care literature (Li et al., unpublished), of which 13 were primary studies. Typically studied via action research and case study methods, these studies have shown that CoPs can transcend the barriers and limitations inherent in traditional continuing medical education (Parboosingh, 2002); have positive effects on the acquisition and maintenance of knowledge in cardiovascular care (Paquet, Leprohon, & Cantin, 2004); reduce time to treatment, improve waiting times, and increase patient and staff satisfaction in emergency care (Huckson, 2001); and improve operating-room practices and effective techniques among leading heart surgeons resulting in a twenty-four percent drop in overall mortality rate for coronary bypass surgery (see Davenport & Prusak, 1998). CoPs have also been shown to promote evidence-based practice in nursing (Tolson, Booth & Lowndes, 2008) and to facilitate quality improvement initiatives in surgical oncology (Fung-Kee-Fung et al., 2008).

Strategies for Practice Change

Both continuing education (CE) and knowledge translation (KT) strategies have been used to reduce the gap between evidence and practice and to alter practitioner behaviour. Continuing medical and professional education incorporate much of the theory and practice of adult learning (Knowles, Holton, Swanson & Holton, 1998), self directed learning (Candy, 1991), and reflective practice (Schön, 1990). CE can improve knowledge, attitudes, professional behaviour, and patient health status, with change most likely to occur in practitioner knowledge and competence, followed by performance changes and patient outcomes. These findings are strongest for CE that is ongoing, interactive, contextually relevant, and based on needs assessment (Robertson, Umble & Cervero, 2003). Programs that supplement knowledge and skill teaching with cues to action, protocols, reminders, counseling guides and materials, administrative and peer support, supplies and equipment and that attend to policy incentives or disincentives change behaviour more effectively than programs seeking to change knowledge, attitudes and skills alone (Robertson, Umble, & Cervero, 2003). Strategies found to promote behaviour change include assessment of learning needs by physicians and/or CME planners; two-way communication over time between learners and faculty; hands-on skills practice and interaction with faculty and other learners; and sequenced and multifaceted interventions (Mazmanian & Davis, 2002; Robertson et al., 2003; Umble & Cervero, 1996). It has been said that no single strategy is best (Oxman, Thomson Davis & Haynes, 1995; Grant & Stanton, 2000). Rather, the context of professional practice and the specific needs of the situation should be considered when determining the most effective strategy.

Professional education alone cannot close the gap between evidence and practice and so other strategies are needed. Here is where knowledge translation can be an important tool because it offers the possibility of examining issues more comprehensively than can typically occur in CE (Davis et al., 2003). According to Davis et al. (2003), one way KT differs from CE is in the target. While CE focuses on practitioner alone, KT allows attention to be given to a range of participants in healthcare practices, including consumers, policy makers, and managers. KT strategies target the health system more broadly. Content in traditional CE focuses on practice based behaviours whereas KT builds on these areas by using evidence-based research. In CE, the major driver can be the teacher or it can be self directed and focused on organizational learning principles. Both are predicated on a simple linear model linking learning to re-licensing and recertification and only tangentially to performance or healthcare outcomes. In contrast, KT reflects the consideration of both the practitioner-learner and the educational or clinical policy provider or health-care system (CIHR, 2002). This more holistic approach makes it easier to close the gap between evidence and practice.

The focus of KT has become the process more than product, with KT conceptualized as a reciprocal process of interaction and exchange among the producers and users of knowledge (Oh & Rich, 1996; knowledge exchange model). Interactive models of KT emphasize the personal nature of the process; that KT is facilitated when knowledge producers and knowledge users are known to one another and are familiar with one another’s needs, preferences, objectives, and circumstances (Huberman, 1994; Lomas, 2000; Jacobsen, Butterill & Goering, 2003).

To date, KT approaches have focused on better availability and presentation of evidence by identifying, synthesizing, and disseminating evidence to practitioners and other stakeholders in practical accessible formats, e.g., reviews in clinical journals, clinical guidelines, better access to electronic sources of information, conferences. While all KT strategies have the potential of helping with effective transfer of evidence to practice, there is no singular recipe that will be effective in all contexts, with all audiences, and for all KT goals. A synthesis of 54 reviews of KT interventions to change health clinical practice concluded that change is possible when a well-designed intervention is used, and most interventions studied had some effects, however, none of the interventions is superior for all changes in all settings (Grol & Grimshaw, 2003).

KT is complex, requiring a variety of strategies to target multiple obstacles at different levels (Grol & Grimshaw, 2003). Although bringing about change is difficult in any organization or practice, it is particularly so for health centres, hospitals, and universities, where highly trained and autonomous professionals are largely in control of the core processes. Often, their deeply ingrained patterns of beliefs and behaviours can impede their willingness to change. This is why, within the organizational change literature, the concept of “readiness for change” is also gaining increasing attention. Effective KT will require attending to the variability in individuals’, organizations’, and communities’ interest, willingness, and ability to acquire and adopt new knowledge (Barwick et al., 2005).

Context for this Study

In Ontario, 117 CMH organizations comprising over 5,000 practitioners have been mandated since 2000 to adopt an electronic version of a standardized outcome measurement tool to monitor client response to treatment and measure service outcomes. The Child and Adolescent Functional Assessment Scale, CAFAS (Hodges, 2003) is a clinician-rated global measure of functional impairment in children aged 6–18 years who have or may have emotional, behavioral, substance use, psychiatric, or psychological problems. An overview of its use in Ontario is provided elsewhere (Barwick, Boydell, Cunningham & Ferguson, 2004).

Nine years of CAFAS training and implementation have identified a pressing need to develop practitioners’ knowledge beyond that which is imparted during training. Recognizing that what is needed is a range of KT and educational supports, our team developed an implementation support infrastructure (Barwick, Boydell, & Omrin, 2002) that includes reliability and software training; web, wiki1, email, and telephone supports; site visits for individualized consultation; and regional face-to-face CoPs that have been well attended and valued. In 2007, the province expanded the outcome measurement initiative by adding 14 CMH organizations to the user group, thereby creating a new cohort of CAFAS-user practitioners with no experience with the tool or the supports already in place. This provided a unique opportunity to examine the impact of communities of practice as a model of collaborative learning leading to practice change, improved content knowledge, and quicker uptake into practice.

In this study we examined whether practitioners in a community of practice changed their practice more readily than practitioners given access to the implementation supports typically available (practice as usual; PaU). We also looked at whether CoP practitioners demonstrated greater knowledge of the tool. We looked at whether practitioners in a CoP environment reported greater satisfaction with implementation supports compared to practitioners in PaU environments who had access to typical supports provided by the CAFAS implementation team. Readiness for change was measured to control for organizational factors that play an important role in practice change. An analysis of how learning and knowledge sharing occur in a CoP environment, based on interviews, field notes and commitment-to-change statements tracked over time are to be reported elsewhere (Barwick, Boydell, Peters & Barwick, in preparation).

Objective

The objective of this study was to explore clinician practice, practice knowledge (of the CAFAS tool), and use of and satisfaction with CAFAS implementation supports among clinicians participating in community of practice sessions versus clinicians engaging in usual practice. The primary outcome was practice change as measured by clinician self-report and use of the CAFAS tool in practice.

Method

Participants

The target population were children’s mental health practitioners (e.g., frontline social workers and child and youth workers) working in service provider organizations newly mandated to use the CAFAS outcome tool who were trained on the CAFAS scale. Fourteen CMH service provider organizations newly added to the provincial CAFAS user group were invited to participate in the study when they registered their clinical staff for CAFAS reliability and software training. Clinicians from 6 consenting organizations were randomly assigned, clustered by organization, to either the CoP (n= 18 from 3 organizations) or PaU (n=19 from 3 organizations) support conditions. All clinicians were eligible to participate in the study if they were trained and achieved inter-rater reliability on the CAFAS tool. All participants were reimbursed for their travel and funds were provided to the participating CMH organizations to secure clinical back-up to cover clinicians’ absences.

Procedures

Organizations participated in 2-day reliability and 1-day software orientation training between March and September 2006. CoP practitioners met as a ‘community’ of new CAFAS users 6 times over an 11 month period, in October and December 2006 and then again in February, April, June and September 2007. Meetings were held in the same location and were hosted and facilitated by the CAFAS Trainer. The CAFAS Supports Questionnaire was administered at each meeting, while the CAFAS Knowledge Questionnaire and Practice Change Questionnaire were completed at baseline, mid-point (approximately 4 months from baseline and after the first 3 sessions) and end-point (after the last 3 sessions or 11 months from the first session). PaU practitioners completed questionnaires at baseline, mid-point, and end-point.

Measures of Practice Change

Practice Change Questionnaire

A 10-question Likert scale questionnaire was developed based on best practice behaviours for CAFAS use and rated by each participant at baseline, mid-point, and end-point to assess the degree of self-reported change. The items tapped CAFAS rating behaviours that are indicative of optimal CAFAS tool use, and each rating captured the last 3 months of practice, e.g., Has your clinical interviewing become more comprehensive? Have you changed your approach to gathering client information? Have you changed your CAFAS scoring practices? Have you used the Treatment Plan? Have you used the CAFAS to assess the client’s treatment progress? Have you provided the client/family with a copy of the CAFAS Client Assessment Report? Have you looked at the CAFAS outcomes for your ‘closed’ clients? Has CAFAS improved your clinical practice? Has use of CAFAS improved your confidence as a clinician? Items were rated as ‘very much’, ‘somewhat’, ‘very little’ or ‘not at all.’ Data were reduced to a Total Practice Change Score.

CAFAS Measure

The number of times clinicians rated the CAFAS in practice was used as in indicator of actual change in practice. Data were reduced to the total number of CAFAS ratings per organization – a sum of all the CAFAS ratings conducted by participating clinicians. The CAFAS or Child and Adolescent Functional Assessment Scale (Hodges, 2003) is designed to rate impairment in children and youth who have or may have emotional, behavioral, substance use, psychiatric, or psychological problems. It consists of behavioral descriptions, (e.g., expelled from school) arranged into four levels of impairment - severe, moderate, mild, and no or minimal impairment – across eight domains of functioning (subscales): school or work, home, community, behavior towards others, moods and emotions, self-harmful behavior, substance use, and thinking. The rater reads the items in each subscale, beginning with the severe items, until a description of the client’s functioning is found. The score on each subscale is determined by the level of impairment under which the item appears: severe, 30; moderate, 20; mild, 10; no or minimal, 0. Subscale scores are combined to form a Total CAFAS score. Training can be accomplished independently by rating 10 reliability vignettes from the CAFAS Self-Training Manual. This ensures that all raters use the same “rules’ and definitions of terms. In Ontario, rating of these vignettes is conducted in 2-day training workshops. Software and train-the-trainer workshops are also provided. To guard against “rater drift,” booster exercises are completed annually after achieving reliability.

Measure of Content Knowledge

CAFAS Knowledge Questionnaire

Participants answered 20 true/false questions at baseline, mid-point, and end-point measuring specific knowledge related to clinical use of the CAFAS scale. Questions were based on frequently asked questions collected over 5 years and their importance for accurate, reliable use of the tool in clinical practice. Total score ranges from 0 to 20. Data were reduced to a total CAFAS knowledge score.

Measure of Satisfaction with Supports Available

Satisfaction with CAFAS Supports Questionnaire

Satisfaction with CAFAS implementation supports was measured at the end-point of the study (month 12) using a twenty item questionnaire. The questions captured the extent to which participants were satisfied with: information provided on the CAFAS website, reliability training, communications with CAFAS staff, support from their organization and from colleagues, quarterly agency CAFAS reports, support from a regional community of practice, and CAFAS software training. Responses were Likert scale, ranging from ‘very much’, ‘somewhat’, ‘very little’, ‘not at all’ or ‘not applicable. Total score ranges from 0 to 20. Data were reduced to a total satisfaction score.

Use of Implementation Supports

Use of CAFAS Supports Questionnaire

Participants completed a 20-item questionnaire regarding their use of CAFAS implementation supports. Responses were ‘yes’, ‘no’, ‘don’t know/does not apply’. Questions pertained to whether they attended refresher reliability training, completed a booster reliability exercise, attended a regional CAFAS community of practice meeting, attended a train-the-trainer session, returned to the CAFAS self training manual to verify scoring procedures, accessed information from another CAFAS manual or document, visited the CAFAS website, contacted the CAFAS office, whether their agency received a site visit from the CAFAS trainer, participated in reviewing their agencies quarterly CAFAS report, sought support from colleagues, downloaded documents from the website, or attended a presentation about CAFAS in the community. Data were reduced to a total CAFAS supports score.

Measure of Readiness for Change

Modified Organizational Readiness for Change Scale

The original ORC ( Lehman, Greener, & Simpson, 2002) includes scales from four major domains—motivation, resources, staff attributes, and climate that have been shown to be key features of organizational change. There are 115 Likert-type items (5-point Likert response) on 18 content domains that take 10 minutes to complete. The scale has satisfactory reliability and validity (Lehman et al., 2002) with clinicians and managers in the addictions field. A modified version used with CMH practitioners demonstrated moderate but lower reliability (Cronbach alpha .60) due to the deletion of several original items deemed inappropriate to the CMH context (Barwick et al., 2005). Scale items cover four major areas: motivation and readiness for change, institutional resources, personality attributes of the staff, and organizational climate. Data were reduced to total subscale scores.

CoP Format and Content

The structure of our CoP was developed according to certain key principles: sustainable CoPs need to be assisted in their creation and development, and their evolution has to be purposefully and systematically nurtured (St-Onge & Wallace, 2003); CoP success requires a shared sense of purpose and ownership, self-initiated view of learning and a readiness to learn from each other, overall climate of trust and involvement, a partnering mindset and corresponding skills, supportive context and leadership endorsement, and realistic expectations on return on investment (McDermott, 2000). Specific agendas for each CoP evolved naturalistically following a rudimentary structure.

CoP Session 1

Following introductions the facilitator communicated the rational and intended benefits of CAFAS community of practice, e.g., the purpose of the CoP is to support and develop the practice surrounding the use of the CAFAS tool. Participants were oriented to the various roles that help set-up, develop, nurture, and sustain the community, and set the stage for its sustainability. As community members, each person had the responsibility to bring practice issues to the community, participate in the productive inquiry conversations, develop ‘best practices’, and engage in personal and professional development. All members collaborated to set agendas and participate actively in discussion. The role of the sponsoring organizations – the CMH service providers - was to openly support the project and provide the necessary resources (e.g., release time). The facilitator’s role was to facilitate the community’s development, liaise with participants, manage problem resolution, steward meetings (co-creating agenda, arrange facilities, lunch, reimbursements), facilitate community discussions/activities/presentations, and guide the development of the CoP through community building activities that focus on knowledge sharing (i.e., reflective practice, commitment to change activities). There was also a key role for the content expert, who acted as a resource to the community when needed. The logistics for loading the CAFAS software were discussed, followed by a discussion of why each of the members chose to participate. Members were asked to discuss what outcome measurement means to them, and this was followed by an ‘expert’ led discussion on the topic. The commitment-to-change exercise was introduced and the baseline questionnaires completed. The session concluded with questions being posed by the community members, leading to discussion amongst the group.

CoP Sessions 2–6

At the start of each session, the group was invited to shape the agenda for the meeting using a process of ‘productive inquiry’ - the catalyst that drives knowledge exchange. Productive inquiry is the ‘need to know’ posted to the CoP in the form of questions that result in conversations. For example, a participant encountered an issue or a situation in applying the CAFAS that s/he has never experienced. Other members of the community were asked, “Have you ever seen this before?” The productive inquiry is not satisfied by information – a simple yes or no was not going to address the depth of need. Instead, the responses were offered from positions of experience with the subject matter. While there may be a straightforward reply, conversation built in which advice, opinions, and information were offered, again situated in practice. Productive inquiry initiated the actions of knowledge access, knowledge exchange, and knowledge creation. The knowledge needed and shared was triggered by a real situation connected to practice. Some providers presented solutions or examples from their experience which served to stimulate the development of ‘best practices’ for CAFAS use. Time was allotted for questionnaires and to review previous CTCs, comment on the degree to which they were realized in practice, and to write new ones.

Analyses

All outcome variables were examined to establish a normal distribution with a skewness value of less than 3. All outcome variables in our study satisfied this criterion and no transformations were required. Independent t-tests were used to detect significant differences between the two groups (community of practice versus practice as usual) on all variables that could potentially affect the results related to our variables of interest in this study. T-tests were also used to determine group differences on one of our outcome variables (Satisfaction), measured at 12-months post-baseline. T-tests were two-tailed due to the exploratory nature of this study. Outcome variables of interest were: knowledge change, use of CAFAS supports, satisfaction with CAFAS supports, and practice change. A two-way repeated measures ANOVA was used to examine the difference between 3 outcome variables (knowledge change, practice change, and use of implementation supports) over three time points (baseline administration of the questionnaire, mid-point/4-months and end-point/12-months post-baseline).

Results

Demographics

Six of 14 organizations newly introduced to the CAFAS initiative in 2006–07 participated in the study. Eight organizations did not participate for a variety of reasons: one agency had only one clinician attend their agency’s CAFAS reliability training; one agency primarily serves juvenile justice clients; one agency was too busy to participate; one agency did not reply to the request; 3 agencies were too far to commute to the community of practice sessions, and one agency provided only Francophone services.

Of 92 clinicians trained in the 6 organizations, 37 consented to participate. Group assignment was based on cluster randomization by organization, with 18 people in the CoP group and 19 in the practice as usual group (PaU). Prior to baseline administration three participants dropped out the study. Of the 34 (17 CoP, 17 PaU) remaining participants who completed the baseline assessments, 20 participants (11 CoP, 9 PaU) completed the questionnaires at 12 month follow-up (42% attrition rate).

Participants were mostly female (89.2%), and had on average 9 years of experiences as a clinician (7 years among PaU group; 10.8 years among CoP group). Four participants had graduate level education, 8 had bachelors level training, 14 had diplomas or certifications in social work, social service work, child and youth care, or early childhood education, and there was one registered nurse (7 participants did not provide level of education data). Of the 3 PaU agencies, 56 clinicians trained for reliability of which 39 (69.6%) consented/participated. Among the 3 CoP agencies, 36 clinicians trained for reliability and 24 (66.7%) participated/consented. Sixty-six percent of the clinicians randomly assigned to the CoP group participated in 4 or more sessions of CoP. Overall, clinicians participated in an average in 3.7 ± 2.1 CoP sessions.

Organizational Readiness for Change

Results of independent t-tests showed no group differences on the organizational readiness (ORC) scale (p>0.05, see Table 1), suggesting that any changes evident on other measures were not a consequence of differences in perceived organizational readiness for change. Although a detailed analysis of ORC data is beyond the scope of this paper, t-tests conducted on specific sub-scales of the ORC were included in this paper to determine the presence of any additional factors that could influence the adoption of the CAFAS tool by the two groups of practitioners.

Table 1.

Group Differences on Readiness for Organizational Change

	Community of Practice (N=17)		Practice as Usual (N=17)
	Mean	SD	Mean	SD	t*
Motivation for Change
Program needs for improvement	35.80	5.38	34.62	8.23	.631
Immediate training needs	33.44	8.00	32.06	7.72	.618
Pressures for Change	33.19	3.01	33.28	4.57	.950
Adequacy of Resources
Offices	30.21	5.51	29.22	7.02	.656
Staffing	28.68	7.02	30.78	7.89	.424
Training	39.02	5.98	35.24	5.81	.086
Computer Access	33.95	5.75	34.12	4.63	.926
E-communications	41.62	6.25	41.92	4.64	.877
Staff Attributes
Growth	28.24	6.47	30.78	8.46	.331
Efficacy	37.94	6.63	39.12	5.37	.574
Influence	40.59	4.60	38.63	4.09	.198
Adaptability	38.63	4.09	38.04	4.09	.678
Organizational Climate
Mission	40.29	4.50	36.18	7.61	.066
Cohesion	32.06	9.85	35.29	8.92	.323
Autonomy	31.12	5.52	32.16	6.56	.640
Communication	36.18	10.24	36.76	7.69	.851
Stress	35.10	10.68	31.18	10.80	.295
Change	33.06	5.25	34.00	4.64	.583

^*Two-tailed; none were significant.

Note that there were no self-reported differences between the two groups among items that measured ‘levels of work stress’ or ‘role overload’ (t (32) = 1.07, p =0.29), suggesting that the study outcomes were not due to differences between the groups with respect to work load or work induced stress. Examination of the ‘staff attributes’ subscale of the ORC showed that there were no differences in the measures of “growth” (t (32) = −0.98, p =0.33 and efficacy (t (32) = −0.56, p =0.57), both of which can generally be used as an individual measures of readiness for change. The mean scores showed that participants in both the CoP and the PaU group scored high on measures of growth (^CoP=28.2, and ^PaU =30.7 respectively) and measures of efficacy (^CoP = 37.9, and ^PaU = 39.1 respectively) suggesting high levels individual readiness for change in both groups.

Satisfaction with CAFAS Implementation Supports

Comparison of participants’ satisfaction with CAFAS implementation supports revealed the CoP participants were significantly more satisfied with the supports and resources provided compared with PaU practitioners at 12 months, t(17) = 2.74, p=0.01), ^CoP = 20.18, ^PaU =11.37) (See Table 2).

Table 2.

CAFAS Use in Practice and Satisfaction with Implementation Supports

		Community of Practice (N=18)			Practice as Usual (N=19)			t
# CAFAS Ratings (Mar/06-Oct/07)†		# clients rated	# ratings	Mean # ratings per client	# clients rated	# ratings	Mean # ratings per client
(CoP-org1)		12	66	5.5
(CoP-org2)		7	22	3.1
(CoP-org3)		55	64	1.2
(PaU-org1)					8	28	3.5
(PaU-org2)					0	0	-
(PaU-org3)					10	37	3.7
Total # Ratings			152			65
		(n=11)			(n=8)
Satisfaction with Implementation Supports	M	20.18			11.38			2.74**
	(Sd)	5.14			8.86

^*p <.01

^†Number of clients rated reflects all cases in the CAFAS database for each organization. Anecdotal report from clinicians suggests these cases reflect the number of clients admitted to each agency during the study time period; however no client information system data were collected at the time to substantiate this claim. The number of CAFAS ratings per organization is calculated on number of clients in the exported CAFAS data for each organization.

Practice Change

Practice change was captured by the number of CAFAS ratings conducted by practitioners in both groups during the 12 month study period. This data revealed that CoP practitioners used the tool more frequently, conducting a total of 152 ratings over the year compared to 65 by PaU participants (see Table 2 for all group comparisons). One CoP organization did not conduct any CAFAS ratings because they experienced technical problems with servers that could not be quickly resolved.

Two-way repeated measures ANOVAs examined changes in study outcomes between the two groups (CoP and PaU) over time (11-months) (See Table 3). Analysis of the primary outcome variable, practice change, showed an overall significant difference in reported practice (Practice Change Questionnaire) over time for both groups, F(1, 17)=11.7, p=0.001. Specifically, clinicians reported little practice change at time 1 followed by a large increase in practice change at time 2. Pair-wise comparisons showed a significant difference in practice change between time 1 and 3, (p=0.002) and time 1 and 2 (p=0.05). Results of the between subject effects showed no significant main effect for group indicating that there was no overall difference in clinician practice between the two groups, F(1, 17)=0.20, p=0.65. There were no significant interactions.

Table 3.

Group Differences on Outcome Variables

	X̄	X̄	X̄	F	F	F
	Time 1*	Time 2*	Time 3*	Group (pvalue)	Time (pvalue)	Group × Time (pvalue)
Practice Change Questionnaire
CoP	3.00	23.2	8.81	0.20 (0.65)	11.7 (0.001)a	1.49 (0.25)a
PaU	1.33	11.6	1.80
CAFAS Knowledge Questionnaire
CoP	12.1	11.0	14.1	2.37 (0.14)	2.25 (0.13)a	10.23 (0.002)a
PaU	10.4	11.5	10.8
Implementation Supports
CoP	4.88	3.44	6.55	0.02 (0.87)	1.24 (0.31)a	8.66 (0.01)b
PaU	4.88	5.22	4.22

^*Time 1, 2 and, 3 are equivalent to measures taken at baseline, 4-months post-baseline and 11 months post-baseline

^aWilks’ Lambda (Multivariate Tests).

^bTest of Within Subject Contrasts

CAFAS Knowledge

Multivariate tests revealed no significant main effects for group (CoP versus PaU), F (1,15)=2.37, p=0.14, or time, F (1,15)=2.25, p=0.13 for the CAFAS Knowledge Questionnaire. An examination of the within subject contrasts did show a marginal quadratic main effect for time (p=0.07), indicating a possible increase in knowledge over the 12 month study period. A significant Time × Group interaction was observed, F (1,15)=10.23, p=0.002. Examination of the cell means indicated there was a large increase in knowledge for clinicians in the CoP group between time 2 (^CoP.2 =11.0) and time 3 (^CoP.3 =14.1) whereas there was a small decrease in knowledge scores in the PaU group from Time 2 (^PaU.2 =11.5) to Time 3 ((^PaU.3 =10.8). Closer inspection of the knowledge scores at 12-months indicated that the CoP group scored significantly higher on the knowledge measure at 12 month than the PaU group, t (19) =1.98, p=0.01 (See Table 3).

Use of CAFAS Implementation Supports

Results of the repeated measures ANOVA revealed no significant main effects for the Use of CAFAS Implementation Supports Questionnaire, indicating no difference in implementation support use between the two groups and no significant increase in supports use over time (p>0.05). Examination of the cell means, however, did show the CoP group to have higher implementation support use over all time periods with the largest difference at 12-months (^CoP =6.55, ^PaU =4.22). Within subject contrasts indicated a significant Time × Group interaction, F (1,15) =8.66, p=0.01. Examination of the cell means indicated that there was a large increase in implementation in the CoP group from time 2 to time 3 as compared to the PaU group whose implementation support use scores remained consistent, only dropping slightly between time 2 and time 3 (See Table 3).

Discussion

This study examined whether communities of practice support the uptake of an outcome measurement tool in child and youth mental health community-based practice. Frontline children’s mental health clinicians were randomly assigned to a community of practice or a practice as usual group, and followed over the course of a year. Results suggest that communities of practice may be a useful strategy for promoting the implementation of this evidence-based practice. Although practitioners supported by the community of practice did not report their practices to have changed any more than their colleagues in the practice as usual environment, they did demonstrate greater use of the tool in the real world practice context. Practitioners supported by the community of practice also demonstrated better knowledge of the tool at the end of the year than did their counterparts, and they were much more satisfied with the implementation supports provided to them than were practitioners who had access to the multiple supports provided to all CAFAS users in the province.

This is the first known study of the community of practice model within the child and youth mental health sector. The random assignment to groups improves upon previous case study research on communities of practice, which was further strengthened by the inclusion of several variables of interest including actual practice change in the real world setting.

As this study is a preliminary examination of the use of community of practice in support of EBP implementation, there were also some limitations. Firstly, with fewer than 20 participants per group who were followed for only one year, caution is required in generalizing findings to other groups of child and youth mental health practitioners working in publicly funded community based service provider organizations. This small sample size could also have had an impact on the stability of the results presented in this paper. Second, the frequency of CAFAS ratings does not take into account variation in the number of patients entering into treatment in each organization. Anecdotal report from the organizations suggests the clinicians rated all new clients as they came into treatment at the organization during this period of time, but no data were collected from organizations’ client information systems to substantiate this. The mean number of ratings per client demonstrated that clinicians from both groups were rating at periodic intervals during treatment which is an indicator of ‘best practice’ – that is to say, more than just the first rating at entry to treatment. At least one CoP agency reported they would have used the tool more had they had more new clients during the year of the study. Nevertheless, it is informative that one of the PaU organizations did not manage to get going during the year and ended up with no cases rated on the CAFAS despite have admitted new clients to treatment. We believe this demonstrates how much handholding some organizations need to take up and implement an EBP in the real world. Third, it is worth noting that the supporting organizations were provided with financial support to backfill clinicians’ absences ($33CDN per hour per practitioner for each of the 6 CoP sessions) and CoP clinicians were paid a stipend of $20CDN per session to cover their transportation costs. Anecdotal reports from CoP participants at the last session indicated they found great value in participating, and their executive directors were in support of CoP involvement beyond the study. Nevertheless, the urgency of addressing service delivery for long wait lists competes with non-service time for professional development, networking, and practice reflection. This is a conflict the field will likely have to address as they move toward adopting evidence-based practices in the future.

Overall, we can conclude that communities of practice are worthy of further study as a means of supporting EBP implementation. With fewer than 20 participants in each group and a study period of only one year, continued research is important to evaluate the knowledge translation and professional development potential of the community of practice model for the health and mental health sectors. Such studies should provide control conditions and follow participants over a minimum of two years. The capacity of CoPs to transcend geographic and organizational boundaries, disciplines, and practitioners’ time limitations may prove this model to be as superior a strategy for knowledge translation in health and mental health as it has proven to be in business.

Communities of practice will continue to be a feature of our provincial CAFAS support strategy based on evidence that it was very well received among the clinicians involved and produced some encouraging results in practice change, knowledge, and practitioner satisfaction. Moving forward, a wiki based community of practice has been developed and launched with the intention of augmenting existing uptake supports and face-to-face regional communities of practice. Future research will examine how virtual CoP environments support knowledge translation in the child and youth mental health sector.

Implications for Knowledge Translation in Child and Youth Mental Health

The CoP model makes training of evidence-base practices more than the transmission of explicit, abstract knowledge from one who knows to one who does not in an environment that excludes the complexities of practice and the community of practitioners (Brown & Duguid, 1991). Transfer models that isolate knowledge from practice are ineffective. Knowledge translation needs to be situated within the context in which it will be used and developed. This is especially relevant in mental health where clinicians typically deal with clients one-on-one and may not have the benefit of team input.

Mental health practitioners are obliged to keep abreast of emerging research in their field, despite having limited access to the research literature (Barwick et al., 2005). Even when access is addressed, they must do more than read journals or attend training workshops. Training to become a reliable rater of the CAFAS measure is only the first step to using it competently in the field. Our experience in the last decade suggests that training is not a one-off event; continuous support from experts and implementers, community learning in the form of shared discussions and problem solving, and experience with new skills in the field is a long term and complex process.

Training can never address all possible clinical situations. Rather it offers the basic knowledge underlying one’s competency in applying a certain intervention. Communities of practice can provide practitioners with a forum for exchanging knowledge and collaborative problem solving situated within cumulative real clinical experiences. In a sector that is plagued by a lack of networking, communication and collaboration across organizations, CoPs serve to break down these silos and extend the range of contacts and knowledge. Communities of practice may also benefit child and youth mental health organizations by helping new hires to develop their knowledge and clinical skills more quickly; a huge benefit in a sector beset by high staff turnover.

Getting research into practice in child and youth mental health will require multiple changes on many levels. As a first step, important shifts need to happen in university training to include basic knowledge of EBPs. Also needed is the development of a professional learning culture within provider organizations to transform them into learning organizations, as well as changes at the government policy level to support the roles necessary to incorporate science-to-practice within provider organizations. Creating dedicated positions for supporting EBP training and program evaluation, for instance, staffed by individuals with the necessary competencies as opposed to available staff who are not prepared but rather take these functions on as add-ons to their job responsibilities. There must also be a policy shift toward supporting practice development, training, reflective practice, and networking activities alongside direct service. These changes are needed if Ontario is to realize its policy goal to “shift our focus away from outputs and towards the achievement of better outcomes” (Government of Ontario, 2008).

Previous Presentations

Barwick, M. A., Peters, J., Boydell, K. M., & Barwick, A. K. (December 8–9, 2008). Reaching for Knowledge in Unison: Using Communities of Practice to Support Practice Change. Paper presented at the UK Conference on Evidence-Based Health Care, Centre for Evidence Based Medicine, Oxford, UK.

Barwick, M. A., Peters, J., Boydell, K. M., & Barwick, A. K. (October 16–17 2008). Communities of Practice in Support of Practice Change. Knowledge in Motion, The Harris Centre, Memorial University, St. John’s Newfoundland.

Barwick, M. A., Peters, J., Barwick, A. K., & Boydell, K. M. (August 5–7 2008). Reaching for Knowledge in Unison: Do ‘Communities of Practice’ Support Practice Change? Findings from a Pilot Study. National Collaborating Centres for Public Health, Annual Summer Institute in Knowledge Translation, Synthesis, and Exchange. Kelowna BC.