Integrated Knowledge Translation and Grant Development: Addressing the Research Practice Gap through Stakeholder-informed Research

Abstract

Objective:

We describe our stakeholder engagement process for grant application development that occurred as part of our integrated knowledge translation plan and make recommendations for researchers.

Method:

In phase 1, a stakeholder consultation group was developed. In phase 2, surveys regarding knowledge gathering, research agenda, and research collaboration preferences were sent to 333 cross-sectoral youth-serving organizations in Ontario, including family and consumer organizations.

Results:

In phase 1, 28 stakeholders from six sectors participated in the consultation group and provided input on multiple aspects of the proposal. Through this process, 19 stakeholders adopted formal roles within the project. In phase 2, 206 surveys were received (response rate = 62%). Survey responses supported the grant focus (concurrent youth mental health and substance use problems). Respondents also prioritized project goals and provided specific feedback on research and knowledge translation. Finally, although some stakeholders chose greater involvement, most survey respondents indicated a preference for a moderate level of participation in research rather than full team membership.

Conclusions:

Despite short timelines and feasibility challenges, stakeholders can be meaningfully engaged in and contribute to the grant proposal development process. Consideration is needed for the practical challenges that stakeholder organizations face in supporting and participating in research.

Substantial literature highlights the gap between knowledge produced by research regarding mental health and application of that knowledge in mental health services (e.g., McHugh & Barlow, 2010; National Institute of Mental Health (NIMH), 1999; Weisz, Weiss, & Donenberg, 1992). Research has enhanced our understanding of psychopathology, providing both information for clinicians regarding services to provide to clients, and information for policy-makers to consider in the structuring and funding of mental health services. The impact of this literature, however, has been limited by a gap in the exchange of knowledge between researchers and these various stakeholders.

Awareness of this research-practice gap, concern regarding its impact on effective service provision, and recognition of the resulting reduction in returns on public funds invested in research have led granting bodies to place greater emphasis on knowledge translation and exchange activities. The mandate of the Canadian Institutes of Health Research (CIHR, 2006), a funding body for health researchers in Canada, emphasizes both knowledge creation and the translation of that knowledge into improved health for Canadians. Other granting bodies are similarly placing greater emphasis on knowledge translation and exchange activities (e.g., NIMH, 1999). As such, researchers applying for funding face increased pressure to develop effective knowledge translation plans for sharing their findings not only with the research community, but also with the broader community of stakeholders and citizens.

Recently, recognition of the limitations of traditional knowledge translation models that operate in a unidirectional or hierarchical manner from researchers to stakeholders has increased. Knowledge translation activities based on these traditional models can increase resistance among recipients and have had limited success in promoting uptake of empirically supported treatments (Herschell, McNeil, & McNeil, 2004). Accordingly, to address these challenges, there has been a move toward integrated models of knowledge translation and exchange involving reciprocal exchanges and enhanced collaboration between researchers and stakeholders over the course of an initiative or project (Becker, Stice, Shaw, & Woda, 2009; Bowen & Martens, 2005; Graham et al., 2006; Green & Mercer, 2001; Hatgis, Addis, Krasnow, Khazan, & Jacob, 2001; Jansson, Benoit, Casey, Phillips, & Burns, 2010; Miller, Sorensen, Selzer, & Brigham, 2006). Various approaches to establishing collaborative researcher-stakeholder relationships have been described in the literature, including participatory action research (PAR; e.g., Kidd & Kral, 2005; Southam-Gerow, Hourigan, & Allin, 2009), community-partnership research (e.g., Becker et al., 2009), and community-based participatory research (e.g., Macauley & Nutting, 2006). Each of these approaches aims to involve the potential users of research in the formulation and application of the research (Green & Mercer, 2001) with general acknowledgement that actual implementation of research-stakeholder collaborations may vary from very little involvement to full and equal involvement by stakeholders.

The reported benefits of these collaborative models include increases in stakeholders’ perceptions of the relevance of research findings and in turn, increases in their application of research findings (Macaulay & Nutting, 2006; Walter, Davies, & Nutley, 2003). Other reported benefits include gains in practical and conceptual knowledge for both researchers and stakeholders (Hatgis et al., 2001; Walter et al., 2003), and improved quality and external validity of research (Bartunek, Trullen, Bonet, & Sauquet, 2003; Herschell et al., 2004). Indeed, the involvement of stakeholders in the development of research plans has been noted as essential for successful community-based, externally valid, research (Bartunek et al., 2003; Reback, Cohen, Freese, & Shoptaw, 2002); if the research questions are not relevant to stakeholders, neither will the answers be (e.g. Jansson et al., 2010). Notably, integrated knowledge translation in research planning and project development is now a CIHR requirement for many funding opportunities (Graham & Tetroe, 2009).

The grant development phase of the research process, however, is characterized by practical limitations, most notably short timeframes and limited funding for research assistance. As such, researchers may perceive a number of constraints in the extent to which they are able to engage stake-holders in consultation and collaboration during this phase, and may feel limited to strategies such as obtaining letters of support from stakeholders (Frazier, Formoso, Birman, & Atkins, 2008). This strategy, however, is unlikely to result in the level of collaboration necessary for these benefits nor to address the research-practice gap (Frazier, et al., 2008). Accordingly, as part of an integrated knowledge translation plan, we implemented a multi-step process of engaging a broad cross-section of stakeholders during the grant application development phase of a CIHR Emerging Team grant focused on understanding and addressing concurrent disorders in children and youth. We aimed to develop a program of research that would involve knowledge users in its formulation, thereby increasing its likely relevance and utility to stakeholders. We illustrate these integrated knowledge translation methods here to demonstrate an approach that was found to be feasible within the practical constraints of the grant development process, and to encourage the development and discussion of other methods of engaging stake-holders during this important phase of the research process. We also report on the results of these consultations, including stakeholders’ perspectives on researcher-stakeholder collaboration.

Method

We conducted a two-phase stakeholder consultation during development of a grant application focused on child and youth concurrent disorders (CD). In phase one, a stakeholder consultation group was developed; phase two consisted of surveying a large cross-section of stakeholders from youth-serving agencies across Ontario, Canada.

Phase One: Consultation Group

Consultation group members (N=28) included representation from 28 organizations from six youth sectors: mental health, addictions, justice, child welfare, community health, and youth engagement/advocacy; and four roles (e.g., consumers, frontline service providers, agency decision-makers, policy-level decision-makers) from across Ontario. Participants were recruited via emails sent to distribution lists of networks of youth-focused services. Additional targeted recruitment occurred to ensure representation from consumers and policy-makers based on suggestions from early consultation group members.

Consultation group members were asked for input regarding: 1) project objectives; 2) study hypotheses; 3) project methods (including sample, design, measurement and data collection strategies); 4) feasibility; and, 5) strategies for researcher-stakeholder collaboration. Their input was sought through: 1) providing materials via email for review; 2) teleconferences; and, 3) in-person meetings.

Phase Two: Survey

In addition, we sought input on research goals, methods, and preferred researcher-stakeholder collaboration strategies from a larger body of stakeholders through a web-based survey of youth-serving organizations and consumer-related (family and youth advocacy/engagement) organizations in Ontario. Through this we also sought to expand our stakeholder network.

Participants

We developed a comprehensive list of youth-serving organizations across sectors by obtaining organizational information from provincial sources, such as Children’s Mental Health Ontario, Drug and Alcohol Registry of Treatment, Ontario Association of Community Health Centres, and Ontario Association of Children’s Aid Societies.

Through this process we identified 412 stakeholder organizations. We phoned each organization (three calls maximum) and obtained the email addresses of 282 program managers (or equivalent); the remainder were not obtained due to out-of-service numbers or lack of response. For unreachable organizations, general email addresses from the internet were used. In order to increase representation of consumer-related groups (family and youth advocacy/engagement), we used a snowball sampling approach, asking existing stakeholders to identify consumer-related organizations to include in survey sampling. Twenty additional organizations were identified using this method.

Survey Content

Informed by the consultation group the survey included the following domains: 1) preferences for knowledge exchange including current strategies for accessing clinical and research evidence and preferences for receiving clinical and research information; 2) preferences for research collaboration; and, 3) perspectives on the proposed research agenda, including the perceived presence of concurrent disorders amongst the children, youth and families they serve and opinions regarding the most important issues to be addressed by the project. Service provider and consumer versions of the survey differed in wording and on the inclusion of some service-specific questions.

Procedures

Invitations for survey participation (with survey link) were emailed successfully (i.e. did not bounce back) to 333 stakeholder organizations; 206 survey responses were received (62% response rate; n=188 service provider version; n=28 consumer version). Service providers who indicated their organization’s primary service sector were from mental health (38%), health (25%), addictions (19%), child welfare (7%), education (3%), justice (3%), and other, such as housing/shelters, outreach, or employment services (5%).

Results

Consultation Group

The stakeholder group contributed significantly to the grant development process and their input was reflected in the grant proposal in numerous ways. For example:

1. sample size goals were reduced to improve feasibility;

2. youth engagement strategies to minimize attrition were broadened to include electronic strategies and incentives;

3. stakeholder samples were expanded to include missing sectors;

4. stakeholder engagement strategies were broadened to include incentives;

5. data collection strategies (e.g., survey) were revised in terms of content and wording; and,

6. specific research questions were generated.

In addition, some researcher-stakeholder collaboration roles were formalized through members’ participation as grant co-applicants (n=3) or collaborators (n=3), through volunteering to be considered as potential study sites (n=5), or through membership on an Advisory Committee for project implementation (n=8).

Survey

Stakeholder Knowledge Exchange Preferences

Through this survey we sought information about stake-holder knowledge gathering practices and preferences to allow us to match our knowledge exchange activities to stakeholder preferences. Results revealed that workshops/presentations and discussions with colleagues were the most commonly used strategies for obtaining clinical information across both respondent groups (see Table 1). Similarly, workshops/presentations were the most commonly preferred method for receiving research-based knowledge and evidence, far outweighing any other strategies.

Table 1.

Preferences for methods of obtaining clinical and research-based knowledge among service provider and consumer groups

	Service providers				Consumer groups
	Clinical knowledge (n = 117)		Research-based knowledge (n = 114)		Clinical knowledge (n = 16)		Research-based knowledge (n = 14)
	n	%	n	%	n	%	n	%
Workshops/presentations	101	86	104	91	13	81	14	100
Discussions with colleagues	73	62	32	28	12	75	2	14
Clinical literature/professional journals	58	50	37	32	4	25	3	21
Meetings with experts	46	39	38	33	7	44	5	36
Internet websites	44	38	52	46	10	63	6	43
Provincial guidelines	44	38	21	18	2	13	2	14
Program evaluations	40	34	28	25	4	25	0	0
Journal articles	33	28	28	25	4	25	2	14
Treatment manuals	32	27	36	32	3	19	3	21
Community needs assessments	30	26	19	17	3	19	2	14
Other	17	15	9	8	2	13	1	7
Newsletters	14	12	31	27	5	31	7	50
Media	4	3	8	7	1	6	1	7
Listserves, blogs	0	0	6	5	2	13	1	7

Note: The item instructions requested that respondents indicate their top 3 preferred methods.

Stakeholder Research Collaboration Preferences

We also sought to examine stakeholder preferences regarding models of researcher-stakeholder collaboration to inform efforts to work meaningfully with stakeholders. Table 2 shows responses to the question, “How can we best involve you and your organization in helping us to identify the most important research questions and the best approaches to use in future clinical research projects?” The most commonly endorsed service provider responses were surveys (67%) and focus groups (55%). Substantial involvement such as full research team membership was only endorsed by 11% of service providers, even though multiple responses were permitted. Among respondents from consumer-related organizations who responded to this item (n=12) surveys were the preferred option (50%) for being involved in research; full research team membership was endorsed by 25%.

Table 2.

Preferences for methods of involvement in research

	Service providers (n = 111)		Consumer groups (n = 12)
	n	%	n	%
Survey	74	67	6	50
Focus group	61	55	4	33
Interview	39	35	4	33
Consultation to a research team	29	26	4	33
Web-based discussion forum	29	26	0	0
Full membership on a research team	11	10	3	25

Note: Multiple responses were permitted.

Stakeholder Research Agenda Perspectives

Table 3 summarizes responses from both service providers and consumer-related groups regarding the proposed research agenda. Both groups identified improving access, promoting the coordination or integration of services across service sectors, and enhancing youth and family engagement in services as the most important goals for the team’s proposed project. Indeed, 83% of the consumer respondents ranked improving how easily people can access services as the most important issue to address through this project. Among service provider respondents, improving access was also commonly chosen as the most important issue to address (58%), but overall there was greater variability in responses, with a number of topics rated as important.

Table 3.

Opinions on the most important issues for the project to address

	Service providers (n = 125)		Consumer groups (n = 18)
	n	%	n	%
Improving access to service	72	58	15	83
Promoting cross-sectoral coordination or integration	60	48	9	50
Maintaining child, youth and family engagement in interventions	58	46	4	22
Ensuring service continuum across different levels of severity	48	38	7	39
Strategies for engaging children, youth and families in service	47	38	4	22
Helping organizations to implement evidence-based interventions	39	31	4	22
Finding new ways to understand which interventions work	37	30	6	33
Determining CD screening strategies	36	29	3	17
Providing education regarding CD	33	26	4	22
Promoting cross-agency coordination or integration	26	21	3	17
Developing new intervention approaches	23	18	3	17

Note: Respondents were asked to indicate the top 3 issues that should be addressed.

Consistent with these priorities, almost two thirds (63%) of service provider organizations reported that children and youth with clinically significant concurrent mental health and substance use concerns present for service at their agencies while only approximately half (55%) of those agencies, or 35% of all participating agencies, reported that they offer services to address these co-occurring concerns. Moreover, respondents who reported that youth with CD present to their agencies for service indicated that, on average, approximately 45% of their total volume of youth clients have clinically significant problems with both substance use and mental health concerns.

Similarly, according to consumer-related responders 43% of people involved with their organizations are ‘extremely impacted’ by CD and almost half of those individuals (47%) are ‘not at all’ able to access services to address all their concerns through one service provider. Notably, this is consistent with service providers’ reports that only 55% of agencies who see youth with CD offer services to directly address youth CD concerns.

Discussion

Greater collaboration between researchers and stakeholders has been identified as an important strategy for closing the research-practice gap (Hatgis et al., 2001; Herschell et al., 2004). Such collaborations may take many forms, with varying levels of stakeholder involvement at different times during the research process. One of the points at which collaboration is particularly important, yet often lacking, is during the development of research or grant proposals (Green & Mercer, 2001). Notably, there are a number of obstacles to stakeholder collaboration at this step of the research process. First, timeframes are often short between funding announcements and application deadlines and may not lend themselves to meaningful collaboration with stakeholders who have numerous and usually pressing clinical and administrative commitments. Indeed, research has cited time and resource demands on community-based stakeholders to be an obstacle to stakeholder collaboration in research (Bowen & Martens, 2005). Second, the activities described here are very time and resource intensive.

Some grants provide some funds for proposal development; however, many grants offer no funding prior to the grant being awarded. This can limit the collaborative activities that are feasible during this phase, including meaningful stakeholder collaboration.

In this paper, we describe ways in which we collaborated with stakeholders to inform grant application development. Of course, these activities represent only a small proportion of the work required to prepare a full grant application. Given the considerable workload associated with application preparation and the short timeframe, it was necessary to tailor our stakeholder collaboration activities to those that were feasible during this timeframe, yet would also be meaningful.

Our multi-level stakeholder consultation process informed our research proposal development in many valuable ways. For example, the consultation group provided insight into a number of issues relevant to the feasibility of our research methods, which allowed us to modify our methods to reflect the practical issues related to working with community-based agencies and the clinical population of interest. As well, our consultation group activities and survey allowed us to obtain stakeholder perspectives regarding the relative importance of potential issues to address in our proposed research which we were then able to integrate with information from literature reviews to produce research questions and hypotheses that were both evidence-informed and of practical importance to the field. Furthermore, stakeholders provided information about how they would like to be involved in research, and how they prefer to receive information allowing us to tailor our proposed knowledge translation and exchange activities.

These findings are limited by the survey response rate (62%) and the lack of capacity to gather detailed information from stakeholders who provided input through the stakeholder survey. As well, the practical aspects of preparing a grant application, especially with a relatively short turn-around time constrained our capacity to document in detail the consultation process (e.g., number of emails, number of responses) with the stakeholder consultation group. Future efforts should record such details as the process unfolds.

Conclusions

This study confirms that stakeholder participation can provide valuable input to research planning and is possible even within tight timelines and budgets. Given its importance in shaping useful research, even limited involvement within such tight constraints should be considered. Notably, survey and focus group participation are preferred methods of involvement for many stakeholders perhaps because many agencies report working at or beyond capacity and substantial involvement in research may constitute a significant resource burden. Online surveys are relatively inexpensive, providing an attractive option where no or very limited funding is available to support the grant development process.

Based on our experiences we recommend the following to researchers: 1) begin planning stakeholder involvement from project conceptualization; 2) consider and minimize the burden on participating stakeholders; and, 3) include different collaboration options, allowing different degrees of stakeholder involvement, recognizing that more intensive collaboration options (e.g., focus groups; larger meetings) with fewer stakeholders allow for more detailed input and less intensive options, such as surveys allow for input from a broader range of stakeholders.

Although higher levels of involvement have been advocated to achieve sustainable capacity building in community-based organizations (Spoth, Greenberg, Bierman, & Redmond, 2004), effective and meaningful researcher-stakeholder partnerships requiring full stakeholder participation in all aspects of a research project may not be necessary nor feasible. Instead, identifying effective strategies to ensure that stakeholders have meaningful opportunities to shape research questions, methods and interpretations of findings (Green & Mercer, 2001), while at the same time allowing for achievement of project goals within the confines of the practical constraints of research and clinical contexts may be our most pressing challenge.