Knowledge Transfer and Exchange: Review and Synthesis of the Literature

Abstract

Knowledge transfer and exchange (KTE) is as an interactive process involving the interchange of knowledge between research users and researcher producers. Despite many strategies for KTE, it is not clear which ones should be used in which contexts. This article is a review and synthesis of the KTE literature on health care policy. The review examined and summarized KTE's current evidence base for KTE. It found that about 20 percent of the studies reported on a real-world application of a KTE strategy, and fewer had been formally evaluated. At this time there is an inadequate evidence base for doing “evidence-based” KTE for health policy decision making. Either KTE must be reconceptualized, or strategies must be evaluated more rigorously to produce a richer evidence base for future activity.

Knowledge transfer and exchange (KTE) is an interactive interchange of knowledge between research users and researcher producers (Kiefer et al. 2005). The primary purposes of KTE are to increase the likelihood that research evidence will be used in policy and practice decisions and to enable researchers to identify practice and policy-relevant research questions. Even though there are many strategies for KTE, it currently is not clear which ones should be used in which contexts (Lavis et al. 2003a). To date, the complete literature on KTE as it pertains to health policy has not been reviewed in a single study. One reason is the challenges in adequately defining KTE across different literatures that tend to use varying terminology in articulating the underlying concept of information and evidence exchange between researchers and health policy decision makers. Accordingly, no summary of the current evidence regarding KTE strategy effectiveness in relation to health policy is available.

Our review was the first part of a larger study designed to find evidence-based KTE practices to inform the design of a specific KTE platform for a series of research projects referred to collectively as the “Alberta Depression Initiative” (ADI). This article reports on the review's findings. Another, separate article reports on the findings of a series of key informant interviews on KTE issues relevant to the ADI research program and also outlines the implications of both the review and interview findings for KTE in research programs like the ADI.

Background

With the growing demands on health care resources and a general culture of accountability, greater emphasis is being placed on generating knowledge that can have a practical impact on the health system (Lomas 1997). To this end, “knowledge transfer” emerged in the 1990s as a process by which research messages were “pushed” by the producers of research to the users of research (Lavis et al. 2003b). More recently, “knowledge exchange” emerged as a result of growing evidence that the successful uptake of knowledge requires more than one-way communication, instead requiring genuine interaction among researchers, decision makers, and other stakeholders (Lavis et al. 2003b).

While the value of and need for KTE has received wide support, both researchers and decision makers also acknowledge that they are driven by demands that may not be conducive to successful KTE. For researchers, these demands include challenges such as adapting the research cycle to fit real-world timelines, establishing relationships with decision makers, and justifying activities that fit poorly with traditional academic performance expectations (CHSRF 1999). A perceived lack of knowledge of the research process, the traditional academic format of communication, research that is not relevant to practice-based issues, and a lack of timely results are often cited by those charged with making policy decisions as being barriers to using research findings (CHSRF 1999). Both parties also frequently lament the lack of time and resources to participate in KTE.

Noting these challenges, a variety of mechanisms to facilitate KTE have been proposed, such as joint researcher–decision maker workshops, the inclusion of decision makers in the research process as part of interdisciplinary research teams, a collaborative definition of research questions, and the use of intermediaries that understand both roles known as “knowledge brokers” (CHSRF 1999). In addition, interpersonal contact between researchers and decision makers is an oft-cited fundamental ingredient in successful KTE initiatives (Thompson, Estabrooks, and Degner 2006). To date, however, “gold standard” approaches to KTE seem to be based, at best, on anecdotal evidence but mostly on experience and even rhetoric rather than on rigorous evidence. Our primary aim for this review was to examine and summarize the current evidence base for KTE in relation to health policy, resulting in an evidence-based resource for planning KTE processes.

Methods

We based our review on adaptations of systematic review methods used commonly for clinical research questions. In our case, we used a review process to address questions at the health policy level. Our methods were intended to be transparent, to include appraisal and validation steps in accordance with the principle of replicability but also to involve comparative and thematic synthesis rather than quantitative analysis, and to use gray literature sources to illuminate contextual issues identified from peer-reviewed studies (Adair et al. 2006; Lavis et al. 2004).

Our literature review had four steps: (1) searching for abstracts, (2) selecting articles for inclusion through a relevancy rating process, (3) classifying and rating the selected articles, and (4) synthesizing and validating them. The steps of the review are shown in appendix A. Our initial goals were to ensure a broad capture of a relatively new and poorly defined field and then to identify a final set of the highest-quality and most relevant articles through a consensus screening of abstracts and a selection of articles.

The principal investigator and a medical research librarian developed and ran the search strategy in January 2006. They searched eight databases for English-language abstracts from 1997 to 2005: Medline, EMBASE, Cinahl, PsycINFO, EconLit, the Cochrane Database of Systematic Reviews, sociological abstracts, and social sciences abstracts. The gray literature also was reviewed, including the University of York HTA database, University of Laval KUUC database, New York Academy of Medicine Gray Literature Reports, and ABI Inform (ProQuest dissertations and theses).

We then reviewed reference lists in the identified papers and reports, as well as publication lists of international research centers and researchers known to us to have an interest in KTE. The primary search terms looked for were (the following and variations of) knowledge generation, knowledge translation, knowledge transfer, knowledge uptake, knowledge exchange, knowledge broker, and knowledge mobilization. We also used substitutes for knowledge, such as evidence, information, and data. Our focus in the review was on studies of KTE that could have either an impact on or implications for health care policies at an organizational, regional, provincial, and/or federal level. We attempted to keep the diffusion of innovation literature separate (e.g., Greenhalgh et al. 2004; Rogers 1995), although of course in some cases, the two literatures overlapped. Our strategy also resulted in some but not extensive overlap with “implementation research,” which has as its focus the set of activities created to carry out a given program (e.g., Fixsen et al. 2005). In this sense, the KTE literature can be viewed as a subset of a more broadly defined notion of implementation research that includes activities with practitioners, consumers, and policymakers but also has a greater focus on “transfer” then on information “exchange.”

The initial search yielded 4,250 abstracts. The research team drafted a relevancy criteria statement, tested it on a subset of one hundred abstracts, and discussed with the reviewers the differences in interpretation. They reached a high level of agreement (kappa = 0.78), indicating both clarity and consistency in the researchers' understanding. They talked about the discrepancies and settled on a final relevancy definition that they applied to the abstracts. The team's operational definition was “research conducting/implementing KTE and evaluating KTE between researchers and policy and decision makers.” We specifically excluded publications reflecting exchanges between researchers and clinicians, between providers, or between providers and consumers. One research team member screened the remainder of the identified abstracts and retrieved 150 full articles. A subsequent review of reference lists and KTE websites and research centers found an additional nineteen peer-reviewed articles.

We reviewed these 169 papers and assessed their relevancy, resulting in eighty-one studies that we sorted into implementation studies (i.e., the implementation of specific KTE strategies or evaluations of KTE approaches, n= 18) and nonimplementation papers and reports (i.e., reviews, commentaries, and surveys of relevant stakeholders pertaining to KTE but not reporting on implementation of an actual KTE strategy, n= 63). We then gave these studies a quality rating, using a fifteen-point scale for implementation studies that separately assessed the quality of the literature review, research design, data collection, analysis, and reporting of results, and a ten-point scale for the nonimplementation papers that qualitatively assessed the fit of the paper into the context of the literature, including the date of the paper, journal, and evidence of critical thought (see appendix B). Two members of the team rated a subset of articles (n= 20), resulting in a high level of agreement (kappa = 0.78). Discrepancies were discussed, and a consensus was reached in all cases. One member of the research team then rated the remaining papers. In order to limit the pool of studies to those perceived to be of higher quality, we had decided earlier to include only those studies that had an overall score higher than 7/10 or 10/15 on the respective rating scale. The first two subsections of the results focuses on these “higher-rated” studies.

After this, we identified a number of relevant, nonduplicative reports from the gray literature following the preceding strategies, noting that we continued to review all relevant gray literature reports until spring 2007. These reports were not formally rated as the peer-reviewed literature was, and we used them solely to supplement information that did not appear elsewhere. As such, we included information that in our view was (1) a novel addition to the peer-reviewed literature and (2) made a substantial contribution to the knowledge base of KTE as a whole. Key messages from these reports are outlined in the third section of the results.

Results

As indicated, of the eighty-one papers that were quality rated, sixty-three were classified as nonimplementation studies. That is, they were opinion pieces, reviews, or surveys of stakeholders concerning KTE issues. Conversely, slightly more than 20 percent (n= 18) of the studies reported on a real-world application of a KTE strategy. About 70 percent (n= 56 of 81) were published from 2003 to 2005, with the remaining (n= 25) published from 1997 to 2002, suggesting that the field is growing in interest and importance. Overall, thirty-four (of 63) nonimplementation studies were scored 7/10 or greater, and ten (of 18) implementation studies were scored 10/15 or better. Of these “higher-rated” articles (n= 44), the lead author was located in Canada in 55 percent (n= 25 of 44) of the studies. The study originated from the United Kingdom or Europe in 23 percent (n= 10 of 44) of the cases, while 11 percent (n= 5 of 44) were from the United States, and four studies were from elsewhere. Four reports were identified in the gray literature that, in our view, provide substantial additional information about or insight into KTE. The remaining sections of this article are based on these forty-eight studies or reports, that is, thirty-four of sixty-three nonimplementation studies, ten of eighteen implementation studies, and four gray literature reports.

We also should note the influence of the Canadian Health Services Research Foundation (CHSRF) on our review. Over the last decade, CHSRF has promulgated the use of the terms knowledge transfer and knowledge transfer and exchange. This use, paralleled by many prominent Canadian researchers, some with links to the CHSRF, has resulted in a preponderance of KTE-labeled papers in Canada. As such, on one level, the CHSRF's “marketing” of KTE evidently has been remarkably effective. On another level, however, as is described in the following sections in some detail, without an evaluation of KTE, the CHSRF's efforts may have been premature.

Nonimplementation Studies

The nonimplementation literature identified four major themes: (1) organizing frameworks for applying KTE strategies, (2) barriers and facilitators to KTE, (3) methods and issues for measuring the impact of research studies, and (4) perspectives from different stakeholder groups on what works and what does not work with respect to KTE. These themes were identified on the basis of the highest frequency of appearance in the literature as well as, in our view, the greatest importance for making decisions about the development of KTE strategies and how best to implement them.

Organizing Frameworks for Applying KTE Strategies

We chose five frameworks that had been developed to guide KTE initiatives, which are summarized in table 1. Dobbins and colleagues (2002) proposed a framework that uses Rogers's Diffusion of Innovations theory (1995) to illustrate the adoption of research into clinical and policy decision making. In a review of the use of research in policymaking, Hanney and colleagues (2003) explored the factors enhancing this use, emphasizing the importance of actions at the interfaces between research producers and users while at the same time highlighting the relevance of “receptor capacity.” Such interaction should occur at various stages in the research process, for example, setting priorities, commissioning research, and communicating findings. Ebener and colleagues (2006) proposed using a knowledge map, or visual association of items, that includes both the knowledge type (i.e., what, how, why, where, and who) and the recipient (i.e., individual, group, organization, or network). This information is similar to components in Lavis and colleagues' (2003a) framework, which recommends five elements to consider when organizing KTE: message, target audience, messenger, knowledge transfer process and support system, and evaluation strategy. In regard to the fifth step, an objective for policymakers would be to inform debate, which is often more realistic than the objective to change decision-making outcomes. Finally, Jacobson, Butterill, and Goering (2003) developed a framework to increase researchers' familiarity with the intended user groups and context.

Table 1.

Organizing Frameworks for KTE Application

Authors	Underlying Theory and Literature	Main Components or Issues Raised
Dobbins et al. 2002	Synthesis of management, knowledge utilization, and evidence-based practice literatures	Basic premise that passive diffusion does not necessarily result in behavior (or policy) change
		Characteristics related to innovation, organization, environment, and individual influence research uptake
Hanney et al. 2003	Weiss's (1979) belief that research use is a process of interaction between research inputs and decision outputs	Importance of actions at the interface between research producers and users at various stages in the research process
		Receptive capacity of research user critical
Ebener et al. 2006	Not clear	Visual association of items or knowledge map used to bridge gap between knowledge generation and use across different levels of the health system
		Results in actionable information to enhance understanding of complex processes, resources, and stakeholders
Lavis et al. 2003a	Authors' experience	Five main elements: messages should be “actionable” for decision makers (the what); messages should be audience specific (to whom); credibility of messenger is important (by whom); engagement of stakeholders across multiple stages is interactive (the how); measurement must be suitable to audience and objectives (did it work?)
Jacobson, Butterill, and Goering 2003	Authors' experience and knowledge transfer literature	Five domains: user group, issue, research, knowledge transfer relationship, dissemination strategies
		Key issues: level of rapport or trust among stakeholders, and mode of interaction (i.e., written, oral, formal, informal)

Barriers and Facilitators

The barriers and facilitators for KTE are well recognized as a result of dozens of studies and perhaps are the most frequently addressed topic area in the KTE literature on health policy decision making. These factors can be classified on individual and organizational levels and pertain to relationships between researchers and decision makers, modes of communication, time and timing, and context. Table 2 summarizes these factors. Owing to the attention paid to them in the literature, we discuss a number of the studies and concepts here in more detail.

Table 2.

Main KTE Barriers and Facilitators

Barriers	Facilitators
Individual Level	Individual Level
Lack of experience and capacity for assessing evidence	Ongoing collaboration Values research
Mutual mistrust	Networks
Negative attitude toward change	Building of trust Clear roles and responsibilities
Organizational Level	Organizational Level
Unsupportive culture	Provision of support and training (capacity building)
Competing interests	Sufficient resources (money, technology)
Researcher incentive system	Authority to implement changes
Frequent staff turnover	Readiness for change Collaborative research partnerships
Related to Communication	Related to Communication
Poor choice of messenger	Face-to-face exchanges
Information overload Traditional, academic language	Involvement of decision makers in research planning and design
Traditional, academic language	Clear summaries with policy recommendations
No actionable messages (information on what needs to be done and the implications)	Tailored to specific audience
	Relevance of research
	Knowledge brokers
	Opinion leader or champion (expert, credible sources)
Related to Time or Timing	Related to Time or Timing
Differences in decision makers' and researchers' time frames	Sufficient time to make decisions
Limited time to make decisions	Inclusion of short-term objectives to satisfy decision makers

In Norway, Innvaer and colleagues (2002) systematically reviewed twenty-four surveys of facilitators and barriers to the use of research evidence by health policymakers. The most frequently reported facilitators were personal contact between researchers and policymakers, clear summaries of findings with recommendations for action, good-quality research, and research that included effectiveness data. Other studies have also supported the use of face-to-face encounters as being key to KTE (Greer 1988; Jacobson, Butterill, and Goering 2003; Lomas 2000a; Roos and Shapiro 1999; Soumerai and Avorn 1990; Stocking 1985).

In a participatory evaluation of Manitoba's The Need to Know project, Bowen, Martens, and the Manitoba Need to Know Team (2005) interviewed community partners to identify the characteristics of effective KTE and found that the most important factors were based on relationships. The quality of relationships and the trust developed between the research partners were critical components. The mutual mistrust between policymakers and researchers has been noted elsewhere as a barrier to the use of research (Choi et al. 2005; Trostle, Bronfman, and Langer 1999).

In their examination of pharmaceutical policymaking, Willison and MacLeod (1999) suggested that to improve the use of research, researchers must first decide who their audience is. Similar to what Lavis and colleagues (2003a) recommended, Willison and MacLeod emphasized that each audience has different information needs and communication styles and therefore the information must be appropriately tailored. Research should be presented in summary format, in simple language, and with clearly worded recommendations (Reimer, Sawka, and James 2005; Willison and MacLeod 1999). Acceptable evidence for decision makers can be less rigorous than that for researchers and includes gray literature (i.e., government publications, consultants' reports, monographs, and conference proceedings) (Hennink and Stephenson 2005; Weatherly, Drummond, and Smith 2002). One study noted that decision makers persistently valued experience more than they did research (Trostle, Bronfman, and Langer 1999).

Another frequently recommended facilitator is the inclusion of key individuals, either decision makers or opinion leaders, in the research planning and design stages (DeRoeck 2004; Lomas 2000b; Ross et al. 2003; Vingilis et al. 2003; Whitehead et al. 2004; Willison and MacLeod 1999). Timeliness and the relevance of research also are important (Dobbins et al. 2001; Frenk 1992; Hemsley-Brown 2004; Hennink and Stephenson 2005; Jacobson, Butterill, and Goering 2004; Mubyazi and Gonzalez-Block 2005; Stewart et al. 2005; Trostle, Bronfman, and Langer 1999). Since researchers tend to have longer time horizons than decision makers do, Willison and Macleod (1999) suggested that shorter-term objectives be included to address policymakers' needs. British researchers noted the potential for a “sleeper effect,” in which evidence is stored and not used until a more encouraging political climate develops (Whitehead et al. 2004). Similarly, Martens and Roos (2005) referred to the importance of keeping information on hand until a favorable context prevails, a notion also alluded to by Roos and Shapiro (1999) regarding research on the use of prenatal care and the delivery of mental health services in Manitoba.

Bogenschneider and colleagues (2003) suggested that seminar series with different stakeholder groups be used to facilitate the exchange. The EUR–ASSESS project concluded that personal contact with policy staff was more effective than printed material (Granados et al. 1997). This conclusion coincided with reviews by Grimshaw, Eccles, and Tetroe (2004) and Grimshaw and colleagues (2001), which examined interventions used to influence the uptake of knowledge to change clinical practice. Educational outreach visits and interactive meetings were generally effective, and printed material and didactic meetings were the least effective. Although these reviews reveal important dissemination activities, on the surface there appears to be limited evidence regarding specifically how strategies should be applied to different stakeholder groups.

Although systematic reviews can inform policymaking (Dobbins et al. 2001; Lavis et al. 2004; Lavis et al. 2006), it also is clear that factors aside from “evidence” (as traditionally defined by researchers) affect decision making. Evidence seldom has a rationally linear impact, given the complexity of the decision-making context (Whiteford 2001). For example, both Frenk (1992) and Lomas (2000b) noted the importance of formal and informal institutional structures for decision making. This can include the distribution of responsibility and accountability as well as the roles of interest groups and policy networks in determining what information will be used according to particular values.

The researcher incentive system in universities also has been cited as a barrier. Fraser (2004) commented that the current professional incentive system (i.e., including publishing in peer-reviewed journals and acquiring grants for academic, as opposed to applied or translational research) is “diametrically opposed” to the needs of potential research users. Researchers, of course, are acutely aware of this challenge and may find themselves asking, with no clear answer, Whose responsibility is KTE? and Who will fund these KTE activities?

Waddell and colleagues (2005) examined the use of research in the context of competing influences on the Canadian policy process. Although policymakers used and valued research evidence, they also described three prevailing influences on the policy process that Waddell and colleagues termed inherent ambiguity, institutional constraints, and competing interests. When describing the process of policymaking, one interviewee commented that “facts and logic aren't deciding factors given that decision makers are faced with an immense amount of competing information on an immense range of subjects.” Institutional constraints include fragmentation across state and local levels of government, as well as across the health, education, social service, and justice sectors.

One recently proposed mechanism to facilitate KTE between researchers and decision makers is a knowledge broker, who is trained specifically in information exchange and has set aside time for the process. Vingilis and colleagues (2003) used the term connector to refer to people who help potential knowledge users determine their knowledge needs and help researchers translate, influence, and initiate KTE. Research on the use of knowledge brokers has been limited, however, and has prompted calls to examine the costs and benefits of this KTE strategy (Pyra 2003) and also the quality of information resulting from knowledge broker–based KTE initiatives (CHSRF 2000).

Finally, the use of health services research in policymaking may be enhanced by a government culture that nurtures an interest in and the value of research (Bowen, Martens, and the Manitoba Need to Know Team 2005; Hennink and Stephenson 2005; Roos and Shapiro 1999; Whitehead et al. 2004).

Learning organizations move beyond employee training into organizational problem solving, innovation, and learning (Stinson, Pearson, and Lucas 2006). This means graduating from simply updating a few practices or implementing initiatives to changing the organization's culture to instill the value of mutual learning (Dowd 1999).

Measuring the Impact of the Research

Research organizations and funders are increasingly recognizing the importance of measuring the impact of health research on policies and practices. Documentary analysis, in-depth interviews, and questionnaires have been used to assess the impacts and outcomes of research knowledge. Several researchers have attempted to measure or score the impact of research on the development of public policy. In an exploratory study examining the role of health services research in Canadian provincial policymaking, Lavis and colleagues (2002) interviewed policymakers in Ontario and Saskatchewan. These interviews highlighted three avenues to the use of citable research: (1) the policymakers read printed material; (2) the policymakers interacted with the researchers; and (3) the researchers were involved with the working groups.

Landry, Amara, and Lamari (2001) looked at the use of social science research by interviewing Canadian university faculty members. They defined the use of research as a six-stage cumulative process from transmission to application, with intermediate stages of cognition, reference, adoption, and influence. Nearly half the respondents indicated that they transmitted findings to practitioners, professionals, and decision makers. However, when moving through the six stages, there was a marked increase in research findings rarely or never used. The most important determinants of utilization were the mechanisms linking researchers to users and the users' context. Knowledge utilization depended more heavily on factors regarding the behavior of the researchers and the users' context, or receptive capacity, than on attributes of the research products themselves (Landry, Amara, and Lamari 2001).

Lavis and colleagues (2003b) devised an assessment tool for funders and research organizations to measure the impact of research. They described the following stages: (1) identify target audiences for research knowledge, (2) select appropriate categories of measures (e.g., producer–push, user–pull, or exchange measures), (3) select measures given resources and constraints, and (4) identify the data sources and/or collect new data, analyzing whether and, if so, how research knowledge was used in decision making. They recommended intermediate outcome measures such as whether a policy changed if resources were available to conduct case studies determining whether knowledge was used in the context of competing influences on decision-making processes.

In relation to this last point, examining how knowledge is used moves beyond whether it was used. Almost three decades ago, research knowledge was identified as being used in one of three ways: instrumental, conceptual, or symbolic (Weiss 1979). An instrumental use is research knowledge that directly shapes policies and results in action; a conceptual use refers to a change in awareness or understanding of certain issues; and a symbolic use merely legitimizes existing policies or positions.

Finally, von Lengerke et al. (2004) suggested that health promotion policy using public health research is associated with the policy's impact if both strong social strategies and the political will to support a given policy are present at the same time. To test this assumption, they analyzed data from a survey of policymakers concerning four prevention and health promotion policies in six European countries. These authors found that research use was positively associated with policy output (i.e., the implementation of programs) and outcome (i.e., effectiveness) in contexts in which political interference was minimal.

Stakeholders' Perspectives

The fourth main theme identified in the nonimplementation studies has to do with perspectives on the relationship between KTE stakeholders and the potential effectiveness of KTE strategies. Table 3 outlines six studies from our review and the perspectives offered. While some of the key points raised can be found elsewhere in the KTE literature (e.g., the importance of relationship building and rapport between the researcher and decision maker), the main issue here is that different stakeholders across different contexts came to similar conclusions.

Table 3.

Stakeholders' Perspectives on KTE Relationships

Authors	Stakeholders	Perspectives
Ross et al. 2003	Researchers and decision makers funded by the Canadian Health Services Research Foundation (CHSRF)	Benefits of decision maker's involvement in the research process included an increased focus of the research on its application to users, a greater understanding of and appreciation for the realities of the decision maker's world, and the enhancement of the decision maker's research skills
Sibbald and Kossuth 1998	Ontario Health Care Evaluation Network (OHCEN)	Key strategies to facilitate communication among stakeholders included interactive workshops, access to a research librarian, and a database providing information on health services and research papers
Weatherly, Drummond, and Smith 2002	Coordinators of the Health Improvement Programs in 102 English health authorities	Questionnaires with health authorities, qualitative interviews, and document review: if evidence was used, it was a mixture of internal or experiential evidence (i.e., clinical opinion, public opinion, and health care management opinion) and external, empirical evidence
Trostle, Bronfman, and Langer 1999	Researchers and policy- makers in Mexico	Both formal and informal communication useful to the development of the relationship between researcher and policymaker
Goering et al. 2003	Mental health policy branch of Ontario government and the Centre for Addiction and Mental Health research unit	Four tiers of linkage and exchange in research-based policy development: (1) interorganizational relationships, (2) interactive research projects, (3) dissemination, and (4) policy formation
CHSRF 2000	Research funders	Important for research funders to assess the optimal ways to take issues and priorities from decision makers and transmit them to researchers

Implementation Studies

Our review identified eighteen studies in which a specific KTE mechanism was employed or implemented, and ten of these were rated at 10/15 or higher on our quality index. As table 4 shows, there are numerous approaches to KTE. The focus of many of these interventions is on generating two-way communication, which is not surprising given the emphasis on this in the nonimplementation literature. The notion of a phased intervention to build relationships followed by facilitated meetings also arose.

Table 4.

Key KTE Strategies Identified in the Literature

Face-to-face exchange (consultation, regular meetings) between decision makers and researchers

Education sessions for decision makers

Networks and communities of practice

Facilitated meetings between decision makers and researchers

Interactive, multidisciplinary workshops

Capacity building within health services and health delivery organizations

Web-based information, electronic communications

Steering committees (to integrate views of local experts into design, conduct, and interpretation of research)

The ten implementation studies synthesized here varied in topic area and context: five focused on health promotion and prevention; two looked at workplace health safety; and the remaining three involved mental health, child health, and cancer pain management, respectively (see table 5). The message communicators included researchers, decision makers, and knowledge brokers, and three studies did not specify the communicator. In all these studies, the amount of information available to assess the given KTE strategy varied widely.

Table 5.

Summary of KTE Implementation Studies Identified in the Literature

Reference	Purpose, Objectives, Participants	Design and Methods	Key Results
Dobbins et al. 2001	One hundred forty-one decision makers from thirty-five public health units in Ontario participated	Single-group posttest	1. Program planning: percentage of retrieved articles read in a month, number of years since graduation, value organization placed on using research evidence for decision making, ongoing training in critical appraisal of research literature
Ontario, Canada	Objective 1: determine extent to which systematic reviews of public health interventions influence public health decisions	Cross-sectional follow-up telephone survey	2. Program justification: value that organizations placed on using research evidence for decision making, ongoing training in critical appraisal, expectation of using systematic reviews in future, perception that systematic reviews would overcome barrier of not having enough time to use evidence
Factors of the innovation, organization, environment, and individual that predict the influence of five systematic reviews on public health decisions	Objective 2: examine individuals' perceptions of organizational, environmental, innovation, and individual characteristics that influence impact of reviews on decisions	Topics of systematic reviews chosen in collaboration with provincial advisory group to ensure relevance to current policy and program decisions	3. Program evaluation: existence of mechanisms that facilitated transfer of new information into public health unit
Rating: 14/15		Multiple logistic and linear regression analyses used to identify predictors of overall use of systematic overviews and influence of overviews on policy decisions	4. Policy development: value that organizations placed on using research evidence in decision making, access to online database searching, age
			5. Staff development: making decisions in collaboration with other community organizations
			Forty-one percent of respondents perceived systematic reviews as having a great deal of influence on program planning and 49 percent on program justification; a greater perception that organization valued use of research and ongoing training in critical appraisal; a greater perception of influence of systematic review.
Jacobson, Butterill, and Goering 2005	Consulting can be viewed as strategy for transfer of knowledge between researchers and decision makers.	Multiple-case study	First project: consultant conducted informant interviews with administrators and clinical managers, took snapshots of inpatient characteristics over a defined period of time, and compared the facility's utilization patterns with those of similar institutions; final report recommended several organizational and programmatic changes to improve effectiveness and efficiency of bed use
Ontario, Canada	Clients were contracted for consultants' knowledge or expertise and skills to develop policy or practice recommendations in mental health system.	First project: examined psychiatric bed use at a rural inpatient facility	Evaluation: consultant's report became “source of authority”; report was used to argue against bed closures, but ministry ultimately rejected this; report was used to promote internal changes, such as the designation of several existing facility beds as holding and crisis beds, and this was adopted
Consulting as a strategy for knowledge transfer	Consultants were research associates or scientists employed by the Health Systems Research and Consulting Unit at the Centre for Addiction and Mental Health (CAMH).	Second project: organization of court-based mental health services in large urban area reviewed by team of consultants	Second project: consultant team reviewed organization of court-based mental health services in large urban area of Ontario; team conducted informant and focus group interviews with stakeholder groups; consultants held daylong workshop to present preliminary findings for all stakeholders
Rating: 14/15		Third project: examined provincewide regional assessment projects developed for Ontario's mental health system reform	Evaluation: consulting team and steering committee drew up recommendations and final report that focused on recommendations for which there had been agreement at stakeholder workshop, but the consultants' recommendation that the organizations providing court-based services adopt an integrated lead agency model was rejected
			Third project: meetings with groups of local stakeholders (with the facilities and programs that would be the sites for much of the data collection) to inform them about the study and to seek support
			Evaluation: assessment project findings and recommendations were incorporated into the reports, and recommendations were issued by the regional task forces
Kothari, Birch, and Charles 2005	Aim: to determine whether interaction between users and producers of research is associated with greater level of adoption of research findings in design and delivery of health care programs	Two-group posttests	Interacting teams had increased understanding of report's analysis and attached greater value to report. The interaction was not associated with increased levels of utilization in terms of application within the time frame of the study.
Ontario, Canada		Responses to dissemination of a research report: all public health units received report, but only subset involved in development; three units that contributed to production compared with three not involved	A large difference was found between interacting and comparison teams regarding their intent to use the research findings in future activities. Interacting teams expected to use local data in report for presentations, media communications, the development of educational materials, and strategic and program planning. The comparison teams made little mention of the report's future use.
“Interaction” and research utilization in health policies and programs: Does it work?		Data collection involved group interviews and document review (i.e., annual reports)	Both interacting and comparison teams used research findings to confirm that ongoing program activities were consistent with research findings and to compare program performance with that of other units.
Rating: 14/15
Kramer and Cole 2003	Healthy workplace constructed two key messages that were conveyed and evaluated against specific criteria, including how the information was used (i.e., conceptual use, effort to use, procedural use, and structural use).	Multiple-case study	Conceptual use of thematic messages and increased awareness: workplace parties began to refer to concepts within thematic messages.
Ontario, Canada		Three medium-size manufacturing companies in southern Ontario combined knowledge broker (KB) and observer (the primary author)	Making effort to use thematic messages: effort was evidenced by self-reflection and goal setting.
Sustained, intensive engagement to promote health and safety knowledge transfer to and utilization by workplaces		KT intervention had two phases: first focused on relationship and partnership development: one-on-one conversations between KB and members of workplaces; second focused on more active engagement with two types (i.e., levels) of KB-facilitated meetings on the thematic messages.	Procedural use of thematic messages: action plans, performance monitoring, job satisfaction surveys, procedural changes including daily plant walk around focusing on safety, weekly plant team meetings, and discussion of team development at monthly management meeting all were used.
Rating: 14/15		Qualitative approaches for data collection (observation, field notes, meeting notes) and analysis	Structural use of thematic messages: a group of key personnel was created to investigate accidents, changed notice board with one that highlights achievements, implemented team-focused safety bonus scheme.
			Promoted knowledge utilization by being sustained, intensive, and interactive (not only in building trust and credibility but also in giving KB opportunity to learn language and cultures of different workplaces; informal as well as formal contacts with numerous people in order to achieve a critical mass of people who knew about thematic messages)
			Prince Edward Island
Robinson et al. 2005	Examine utility of linking systems between public health resource and user organizations for health promotion dissemination and capacity building in three Canadian provinces	Parallel-case study	System structure and linking roles: coalition; joint community–project team linking body and regional health staff and volunteer co-chairs as linking agents
Ontario, Canada		Three of the eight provincial heart health projects under study in the Canadian Heart Health Initiative (CHHI). Prince Edward Island and Manitoba interactions were two-way exchanges between resource and user groups; owing to large number of user groups, Ontario was not actively involved in evidence appraisal, program development, and transfer.	Linking activities: regular communication, training and retreats, collaboration, cosponsorship, networking, facilitation, informal training, advocacy, research information, volunteer development
			Ontario
Using linking systems to build capacity and enhance dissemination in heart health promotion: a Canadian multiple-case study		Outcome measures of linking system: capacity enhancement and implementation of comprehensive heart health promotion	System structure and linking roles: provincial advisory group and recourse system; linking agents via medical officers of health (MOH) and managers
Rating: 14/15		Thirty provincial reports selected to reflect content related to capacity and dissemination, a range of time periods in each project, and different audiences; key informant interviews and content analysis of reports of dissemination phase	Linking activities: research monitoring and feedback, provincial resource center collaboration, regular communication, research dissemination, training, technical support, networking
			Manitoba
			System structure and linking roles: committees as linking groups for broad communities, committee facilitators as linking agents
			Linking activities: regular communication, resource provision, modular training, collaboration, facilitation, informal training, technical support
			Improvements in capacity enhancement and implementation of heart health programs across all three provincial systems were observed; difficult to draw causal relationships between specific mechanisms used and outcome measures reported
Kramer and Wells 2005	Transfer body of knowledge on participative ergonomics to number of consultants and ergonomists in Health and Safety Prevention systems (HSAs)	Case study	Categories of knowledge utilization: conceptual, political, and instrumental (Weiss 1979)
Ontario, Canada	Goals included: (1) to have HSAs adopt principles of research to individual consulting model, (2) to incorporate case studies from own sector to increase contextual specificity, and (3) to have them become knowledge brokers	Consultants and ergonomists from practitioner-based associations in Ontario's HSAs acted as knowledge brokers	Conceptual use: HSAs' receptivity to process, agreement from HSA consultants regarding compatibility of models used, commitment from executives to adopt
Achieving buy-in: building networks to facilitate knowledge transfer	Considered the potential users and multipliers of the research; role was to link to long-term audience of Ontario's workplaces	Conceptual framework reflects knowledge transfer and networking theory: establish goodwill, achieve reciprocity, knowledge utilization, long-term alliances	Political use: consultants' use of process as “moral weight” to persuade executives to permit them to change consulting model to a more intensive one
Rating: 12/15		Seventeen meetings with twelve HSAs (n= 150 participants) held over one year	Instrumental use: creation of specific tool that used principles, adoption of principles in documentation, adoption of more participatory approach to ergonomic practice
		Data were collected and analyzed from notes taken at meetings using qualitative methods.	Barriers to adoption: confusion adopting new information, complexity of research overwhelming, intensive process prohibitively expensive
Loiselle, Seminic, and Cote 2005	Dissemination of Health Canada (HC) funded research (2001) on breast-feeding in immigrant community in Montreal	Case report	No formal evaluation, but regional breast-feeding committee members pooled resulting priorities and formed integrated promotion and action plan
Quebec, Canada	Capitalized on newly formed government- sponsored regional breast-feeding committee	Multidisciplinary workshops in eight target settings (three teaching hospitals and five regional centers)	Three joint priorities were selected by hospitals, community health centers, volunteer groups, and policymakers:
Sharing empirical knowledge to improve breast-feeding promotion and support: description of a research dissemination project	Ninety participants included perinatal health care professionals, decision makers, lay feeding-support volunteers, and representatives from Regional Health Board.	Two-hour dissemination activity; 45-minute oral presentation on study findings; 30-minute discussion; 45-minute brainstorming for improving promotion of breast-feeding	1. Creation and implementation of common breast-feeding policy across region's hospitals and community health centers
Rating: 11/15		Communications tailored to stakeholders	2. Development of prenatal breast-feeding education programs tailored to multiethnic population
			3. Enhancement of breast-feeding education and support across hospital- and community-based services
Stewart et al. 2005	Twenty-seven policymakers, practitioners, and researchers from seven southern African countries	Case report	The authors reflected on changes observed: participants had increased understanding of purposes and processes of research, but for research to make a difference, research community needs to emphasize publication of research findings written for potential users.
London, UK	Residential workshops for training in evidence-based decision making for HIV prevention	Intensive intervention has potential to reduce evidence-practice gap for HIV prevention in southern Africa by training nonresearchers to engage with research while providing opportunity for researchers to engage with policymakers and practitioners.	Drawing on feedback and observations, workshops may have addressed the evidence-practice gap in three areas: access to research, understanding of research, and relevance of research.
Exploring the evidence-practice gap: a workshop report on mixed and participatory training for HIV prevention in southern Africa	Mixed and participatory training in accessing and appraising research	Workshop facilitators were four researchers.	Workshops enabled small group of people to access relevant research in timely manner and were most successful in influencing researchers to consider bridging evidence-practice gap by producing more relevant research, applicable to policymakers and practitioners.
Rating: 11/15
Philip et al. 2003	User fellowship in UK	Case report	Authors state that workshops were full, thus suggesting dissemination was successful
Aberdeen, Scotland	Economic and Social Research Council Health Variations Program (ESRC) designed to improve understanding of causes of socioeconomic inequalities in health	Practitioners, researchers, and decision makers involved in production and distribution of newsletters regarding introduction to project and emerging findings/progress, short articles, presentations, posters at conferences, practitioner groups, practitioner seminar collaborative with research team, gatekeeper seminar collaborative with health education board-research reports and workshops	Difficulty experienced in reaching practitioners not already “networked”
Practicing what we preach? A practical approach to bringing research, policy, and practice together in relation to children and health inequalities			Demand for seminar in rural area suggests success of targeting rural practitioners
Rating: 10/15
Philip et al. 2003 Aberdeen, Scotland Practicing what we preach? A practical approach to bringing research, policy, and practice together in relation to children and health inequalities Rating: 10/15	User fellowship in UK Economic and Social Research Council Health Variations Program (ESRC) designed to improve understanding of causes of socioeconomic inequalities in health	Case report Practitioners, researchers, and decision makers involved in production and distribution of newsletters regarding introduction to project and emerging findings/progress, short articles, presentations, posters at conferences, practitioner groups, practitioner seminar collaborative with research team, gatekeeper seminar collaborative with health education board-research reports and workshops	Authors state that workshops were full, thus suggesting dissemination was successful Difficulty experienced in reaching practitioners not already “networked” Demand for seminar in rural area suggests success of targeting rural practitioners
Rashiq et al. 2006	One hundred thirty participants from fourteen health and administrative disciplines attended eleven workshops; test new model of Health Technology Assessment.	Single-group posttest	Postworkshop survey: 99 percent indicated workshops useful in linking research to practice. Action planning component least satisfactory: 30 percent of respondents reported that no action plan was developed, mainly because of insufficient time in session, and 50 percent reported making plans to improve CNCP management in some form where none existed previously. Impact of workshops on participants' perception of knowledge of CNCP was assessed and found to rise after the workshop.
Alberta, Canada	Increase awareness of chronic noncancer pain (CNCP) management in Alberta rural practitioners, and change attitudes	Experts in CNCP management and HTA specialists (ambassadors) travel to AB health regions to translate systematic review findings; ambassadors held eleven two-hour interactive sessions on CNCP.
Rating: 10/15		Session was case study discussion in which participants proposed different treatments.

For example, some studies used more rigorous designs and/or based their assessment on predefined outcome measures. Kothari, Birch, and Charles (2005) used a quasi-experimental study design (i.e., one that had a comparison group) and qualitative methods in determining whether the uptake of information contained in a research report hinged on being involved in developing the report itself. Outcomes measured were the decision maker's understanding of the analysis and intent to use the research findings. Their findings suggested that in the study's time frame, interaction with the report was not associated with the decision makers' greater use. Kramer and Cole's (2003) multiple-case study applied qualitative methods to determine the impact of KTE based on a set of predefined outcome measures. In this study, the mode of communication was a plain-language booklet entitled the “Participative Ergonomic Blueprint.” The “Blueprint” is a facilitator's guide to implementing a successful participative ergonomics program as part of an employer's health and safety program. Finally, the work by Robinson and colleagues (2005) also included a wide range of outcome measures. The challenge here is that the use of multiple linking activities and outcomes makes it difficult for the reader to discern the individual impact of any one KTE strategy. The remaining studies were a mix of posttest studies, case studies, and case reports offering varying levels of evidence with respect to the strategy's effectiveness.

These studies also examined the presence of a formal, planned evaluation of the KTE strategy. From the information reported in these ten studies, only five seemed to have intended to formally evaluate their KTE strategies in advance, and an even smaller set (n= 3), as alluded to earlier, had clearly defined outcome measures. It is also notable that not a single randomized controlled study of KTE was identified. While some studies reported observations of or reflections on the impact of the KTE strategy, generally these observations were not based on a formal evaluation including a research study design with already identified outcome measures. Of course, the intent of these studies was not necessarily to evaluate the KTE activity; in most cases, the emphasis was on the description of the transfer and exchange of information itself. However, owing to the lack of rigorous evaluation, there is little foundation for transferring findings from these studies to other or even similar contexts.

In short, based on these studies, we did not find an “off the shelf” set of recommendations for developing and implementing KTE strategies. This difficulty is due in part to the relatively small number of implementation studies across fields in health care and also to the even less formal and/or rigorous evaluation of these strategies.

Gray Literature

We included the gray literature in this review to supplement our findings from the peer-reviewed papers. We decided that four reports made a significant contribution to the literature being studied, and they are summarized in table 6. The fourth report listed is particularly important, as it is the only study in our entire review that used a randomized control trial (RCT) design to assess KTE strategies.1Dobbins and colleagues (2007) conducted an RCT to test the effectiveness of KTE strategies in Canadian public health decision making on programs related to the promotion of physical activity and healthy body weight in children. The three progressively more active interventions were access to an online registry of systematic reviews evaluating public health interventions, targeted evidence messages, and knowledge brokering. The targeted messaging was significantly more effective in promoting evidence-informed decision making compared with the website and knowledge-brokering groups. The extent to which decision makers in public health organizations valued research evidence affected the knowledge-brokering activity. Knowledge brokering was more effective in those organizations that placed less value on research evidence and was less effective in those organizations that already recognized the importance of evidence-based decision making.

Table 6.

Summary of Gray Literature Reports

Author	Objective/Topic	Key Points
Landry et al. 2006	Overview of factors to consider in developing KT strategies	Six guiding questions: (1) What are the most important aspects to consider when establishing a KT strategy? (2) What are the outputs of research (findings, concepts, methodologies, tools)? (3) Who are the potential users of the research outputs? (4) What is the most effective way to make contact and interact with users? (5) How can potential users be involved in meaningful ways throughout the project? (6) What do users need to know about research in order to understand it and assess its value for potential uptake?
Birdsell et al. 2006	Use of health research results in Alberta	Sources of knowledge most commonly used by decision makers: (1) reports internal to their organization or produced by other government agencies or regional health authorities, (2) management staff of their organization, and (3) evaluation reports Determinants of research utilization including the decision makers' attitude toward research, increased person-to-person contact with researchers, exposure to research-oriented events, and research experience; however, seldom did decision makers report instrumental use of research
		Determinants of research utilization including the decision makers' attitude toward research, increased person-to-person contact with researchers, exposure to research-oriented events, and research experience; however, seldom did decision makers report instrumental use of research
CIHR 2006	Case book of twenty-four health services and policy research knowledge translation stories	Effective KT requires long-term, sustained relationships, and KT activities involving face-to-face interaction are the most effective. Supportive organizational climates also were emphasized, and specifically “executive-level buy-in” was identified as being critical.
Dobbins et al. 2007a	RCT of KTE strategies in public health decision making	Targeted messaging significantly more effective in promoting evidence-informed decision making compared with website and knowledge-brokering groups

Note:^aThis study was presented at a conference but not yet published at the time of the review; and it was identified after the initial search.

Discussion

The bulk of the literature on KTE in regard to health policy pertains to barriers and facilitators to implementation, as well as frameworks that can be used to organize and design KTE strategies, perspectives of KTE from various stakeholder groups, and ways to measure the impact of research on health policy. A smaller subset of the literature pertains to the implementation of KTE strategies for health policy decision making. Overall, these studies have undergone only limited evaluation, and for those that have been evaluated, generalizing findings to other contexts is extremely difficult.

Some of the key messages in the nonimplementation papers, which clearly reiterate Innvaer and colleagues' (2002) findings, are the importance of having personal contacts and building trust through quality relationships over time, in order to have a genuine exchange of information that results in some form of change. It also is clear that no one size fits all, with specific messages needing to be tailored to each audience. That said, what is not as clear from the literature is what works in what contexts and even where the responsibility for KTE rests. In addition, important advances have been made in measuring the impact of research (Allen Consulting Group 2005; Kuruvilla et al. 2006). As a recent report from the Canadian Institutes of Health Research (CIHR 2005) pointed out, indicators of the effect of health research are building capacity for training, informing policy, offering health benefits such as life expectancy and quality of life, and providing key economic benefits such as workplace productivity. This broader view of outcomes fits nicely with the perspectives of understanding organizational change and policy development more broadly and, as discussed later, in examining this question by drawing on information across multiple disciplines (Greenhalgh et al. 2004).

Combining these insights with those from the implementation studies, the major finding of this review is that despite the rhetoric and growing perception in health services research circles of the “value” of KTE, there is actually very little evidence that can adequately inform what KTE strategies work in what contexts. Of course, existing studies can provide insight into KTE activity, but in our view, with the current state of the literature, there is insufficient evidence for conducting “evidence-based” KTE for health policy decision making.

This conclusion may explain why researchers and research funders have recently produced informative papers based on years of experience with KTE (e.g., Lomas 2007). As most would likely acknowledge, this experience falls short, however, in meeting the criteria as research evidence. We do not mean to suggest that those in the KTE field do not have important insights; rather, we are drawing a parallel to how many health services researchers have been critical of clinicians for drawing conclusions from case series reports, particularly since the advent of the evidence-based medicine movement. Viewed in this light, one call from our findings is the need for the greater application of formal and rigorous research designs to assess and evaluate the success of KTE strategies in specific contexts. Reporting on aspects of KTE such as barriers and facilitators does reveal the relevant issues to those interested in an interchange between researchers and decision makers. But in the end, primary research on KTE itself would be required in order to produce the evidence necessary to decide how best to allocate dedicated KTE resources.

Another of our findings is that KTE, at least as conceptualized to date, simply does not fit with the underlying politics of health policymaking. That is, noting limited rigorous implementation and evaluation of KTE strategies at the policy level, could it be that the concept of KTE in this context has been inappropriately transferred from clinical decision making? For the positivist, cause-and-effect translates in this context into knowledge that is transferred between researcher and decision maker in order to influence change. Of course, many scholars have recognized the many factors resulting in a particular action at the policy level (e.g., Mitton and Patten 2004; Whiteford 2001) and, indeed, the importance of context in knowledge utilization (Dobrow et al. 2006). But more fundamentally, as Lewis (2007) pointed out, evidence-based decision making has not had the intended effect due to a myriad of reasons, not the least of which is that decisions at the policy level do not fit into neat little boxes that can be informed by technically oriented inputs. As such, it is important to ask whether those on the “KTE path” will ever be able to make significant strides forward. One may argue that KTE strategies need to be evaluated and refined, but the complexities of real-world policymaking and the misalignment between the evidence producers and the decision makers suggest that other literatures and disciplines, and indeed, other ways of thinking need to be given greater weight in these discussions.

Noting these arguments, introducing an evaluation component into KTE exercises, as has been called for elsewhere (Pyra 2003), should be even more important, as it is not only the refinement but, perhaps more fundamentally, the justification of KTE that may be required. For example, it may be more beneficial to conduct an evaluation based on whether and how policy was informed, rather than simply the extent to which research was used (Eager et al. 2003; Lavis et al. 2003a). If there is to be a demonstrable impact, researchers must learn about the challenges and environment in which decision makers operate and determine how to present the information in a manner appropriate to the real-world environment (Aaserud et al. 2005). But if there is consistently “no impact” through rigorous KTE endeavors across different contexts, it may be that KTE should not continue to be pursued. At this point, we do know, as stated by Lavis and colleagues (2003a, p. 240) in their survey of research organizations mentioned earlier, that “the directors [in funding organizations] were remarkably frank about not evaluating their knowledge transfer activities.”

If KTE activities continue to be pursued, then one way to conceptualize development, implementation, and evaluation is to consider what is under the control of the researchers and what falls under the influence of the policymakers. While this may magnify the perceived gap between these two main stakeholder groups, it can also serve to highlight which aspects of KTE should receive research grant funding and which contextual factors need to be addressed to the decision maker receiving the information. Similarly, much more effort is needed to articulate how knowledge is best transferred from decision makers to researchers and who is responsible for ensuring that this interaction and ultimate exchange takes place. With limited time and the ever-increasing demands on both groups, it is difficult to justify expending resources on ineffective strategies that ultimately are outside the control of either or both parties.

In our view, funding bodies can play a major role here, not only in requiring that detailed KTE plans be incorporated into research grants, but also in funding innovative fellowships such as residency placements in academic and health service delivery organizations and in funding more primary research on the evaluation of specific KTE strategies. On this last point, funders should support more rigorous study designs of KTE strategies, perhaps as a component of larger programs of research that include multiple studies with subsequent designs building directly on previous work. On the other hand, as Greenhalgh and colleagues (2004) point out, controlling for potential confounders in organizational and systems-level research may not be the most prudent way forward. While our study does not speak directly to this debate, we do suggest that at a minimum funders consider allocating resources to KTE endeavors as a pursuit in itself rather than as an “add-on,” as is so often the case at present.

Our review has several limitations. First, our method of including and excluding studies in our review could be criticized. Agreement on inclusion is not the same as the “worthiness” of scientific endeavor but rather merely suggests that individual research team members agree that a given paper “fits” with a relevancy statement outlined for the review. Furthermore, we based our synthesis on studies that, in our view, were of higher quality, but again, we relied on our own values in these judgments. Similarly, the search terms we selected likely reflect our understanding of the topic and our own biases. The fact that more than half the articles retrieved were by Canadian authors may suggest that we have brought a specific lens to this review that excluded other important and related works. Nonetheless, the review was intended to describe how KTE and closely related concepts are currently used, and it did not investigate broader notions such as knowledge diffusion, implementation research, or policy decision making. Nor was our intent to summarize the evolution of related theory. Thus we would argue that the review did address the field of KTE but does not suggest that others who label their work differently have not made important contributions.

Conclusion

Although KTE is not a new concept, it seems to be growing more important. Nonetheless, KTE as a field of research is still in its infancy. It is not hard to find opinion pieces and anecdotal reports about how to use KTE, but the limited reporting of KTE implementation and the even more limited formal evaluation of it leave those wanting to develop their own KTE efforts at a loss for evidence-based strategies. Relationships and institutional knowledge are clearly important themes, as is quality interaction with a few individuals, as opposed to a mass barrage of information to many. Even though no one size fits all, what is needed is more work to inform the application of KTE strategies across contexts to enable evidence-based practice. In our view, the only way to reach this is to conduct primary research on KTE, rather then seeing KTE as an “add-on” to other projects.

Appendix A

Steps in Literature Review

Appendix B

Quality Rating Sheets

Ref ID: ________

KTE Empirical Article Quality Rating Sheet

0 – not present or reported anywhere in the article

1 – present but low quality

2 – present and midrange quality

3 – present and high quality

________ 1. Literature Review: Directly related recent literature is reviewed and research gap(s) identified.

________ 2. Research Questions and Design: A priori research questions are stated, and hypotheses, a research purpose statement, and/or a general line of inquiry is outlined. A study design or research approach is articulated.

________ 3. Population and Sampling: The setting, target population, participants, and approach to sampling are outlined in detail.

________ 4. Data Collection and Capture: Key concepts/measures/variables are defined. A systematic approach to data collection is reported. Response or participation rate and/or completeness of information capture is reported.

________ 5. Analysis and Results Reporting: An approach to analysis and a plan to carry out that analysis is specified. Results are clear and comprehensive. Conclusions follow logically from findings.

________ /15 = Total Score

Ref ID: ________ KTE Nomempirical Article Quality Rating Sheet

1	2	3	4	5	6	7	8	9	10
– Barely relevant		– one or two interesting ideas but not innovative		– Relevant and a few interesting ideas		– Quite good		– Preeminent, groundbreaking paper by leading researcher in field
– Middate range						– Highlights interesting ideas
– Poor writing		– Fairly unknown journal or authors		– Middate range				– Directly on topic
– Poor logic		– “Stale” or ideas covered in more recent material		– Authors' credentials uncertain		– Raises new ideas		– Strong conceptualization
– Local experience								– Progressive
– Redundant				– Of average interest		– Good journal
				– Uncertain about journal		– Quite recent (2000 to present)		– Evidence of critical thought
– Obscure journal								– Prestigious
– Commentator with low-level, non-research-related credentials								– Very recent (2003+)
– At old age of date range Best not to include		Will not be missed		May reinforce ideas; perhaps should include		Definitely include		Must include

Source: Adapted from Adair et al. 2006.