Does It Matter? Decision-maker Perceptions on the Impact of the Regional Training Centres

Abstract

A key feature of the Regional Training Centres (RTCs) is the scope and nature of their engagement with decision-makers. While the RTCs may believe that they have an excellent association with decision-makers, is that belief shared? The authors of this paper draw on the results of a survey of decision-makers undertaken by the Canadian Health Services Research Foundation (CHSRF) as part of the preparation for the fourth-year assessment and evaluation of the RTCs. The discussion encompasses three substantive issues: (1) decision-makers' assessment of the added value of the RTCs, (2) the RTCs' ability to effect cultural change in decision-maker organizations and (3) the experience and value of the internship/placement to the decision-maker organization.

Key messages

• The RTCs have significantly changed the landscape of applied health services research.

• The RTC intern brings to the organization a set of skills that it either currently lacks or needs to develop.

• The RTCs should develop indicators jointly with decision-makers that mark progress and measure change.

The Regional Training Centres (RTCs) were established by the Canadian Health Services Research Foundation (CHSRF) as part of the CADRE initiative. CADRE – Capacity for Applied and Developmental Research and Evaluation – was designed to create the conditions to enhance health services research capacity in Canada, and to re-orient and re-focus the work of health researchers towards the application and use of research in health services decision- and policy making. (See Conrad 2008 for a discussion of the CADRE program.)

A key feature of the RTCs is the scope and nature of their engagement with decision-makers. This engagement takes many forms, including decision-maker membership on RTC Advisory Boards, participation in RTC seminars and workshops, provision of placements for RTC students and, on occasion, membership on student thesis committees. While the RTCs may believe that they have an excellent association with decision-makers, is there any evidence to indicate that such a belief is shared? This paper draws on the results of a survey of decision-makers undertaken by CHSRF as part of the preparation for the fourth-year assessment and evaluation of the RTCs.

The first four years of each training centre were dedicated to setting up infrastructure and educational programs. The purpose of the fourth-year review, under the assumption that the full impact of the RTCs would appear over the remaining six years of CHSRF funding, was to gain enough information and confidence to continue each award. In this regard, the review examined the RTCs' progress against their stated objectives and assessed each program's future viability. The focus of the fourth-year review was on quality improvement and strengthening the centres over the remaining (six) years of the award. (See Davey and Altman 2008 for a discussion of the review and its findings.) By means of this process, CHSRF sought to reassure all stakeholders that the investment in this component of the CADRE program would, both in terms of time and money, demonstrate significant impact by the year 2011 and have a major effect on health services and nursing research capacity within the next two decades.

Survey Methods

As part of the fourth-year review, four distinct online surveys were developed and analyzed by CHSRF staff. The online surveys were distributed by each RTC to key stakeholders – students, participating faculty, principals and decision-makers. An e-mail was sent to a purposive sample of participants identified by each RTC, inviting them to complete the online survey (Survey Monkey). The surveys were distributed during 2004 and 2005. The reason for the difference in distribution times was that the Ontario Regional Training Centre was established a year after the others. Responses to each survey were submitted to a secure site to which only CHSRF had direct access. This paper discusses the responses of the decision-maker (DM) group.

The invitation to complete the survey was sent to a total of 71 DMs with 37 responding, a completion rate of 52%. The breakdown per RTC was: Atlantic Regional Training Centre 8/16 – 50%; Centre FERASI 6/18 – 33%; Ontario Training Centre 8/10 – 80%; and Western Regional Training Centre 15/27 – 55%. The overall response rate is good and is well within acceptable limits for online surveys of this nature. Nevertheless, surveys of this type have limitations. The respondents may not have represented the range of DMs within any one RTC. Those who responded may have had closer links with the RTCs than those who chose not to complete the survey. The selection of DMs by the RTCs may have been biased towards those who would give a favourable response. While none of these constraints can be discounted entirely, there is no evidence to suggest that the views expressed by the DMs in the survey are not a valid reflection of their association with the RTCs.

CHSRF Survey Results

Although the DM survey represented only one aspect of the information collected by CHSRF as part of its fourth-year review, the collected opinions and results do provide an interesting picture of the DMs' linkage with the RTCs. The survey posed a series of questions about the DMs' association and involvement with, perception of and satisfaction with the RTCs. The three most substantive issues are reported here: assessment of the added value of the RTCs; ability of the RTCs to effect cultural change in the DM organizations; and the experience and value of the internship/placement to the DM organization. Each question is discussed in turn.

Added value of the RTCs

The CHSRF survey demonstrated clearly that the RTCs had an impact on the decision-makers, and that generally the impact was positive and significant (Figure 1). Approximately 70% of those DMs who responded stated that the impact of the RTCs on the training of applied health and nursing services researchers has been either “good” or “very good.”

FIGURE 1.

Decision-makers' assessment of the value of the RTCs (percentage; N=37)

Respondents to the survey had an opportunity to provide additional comments in assessing the added value of the RTC. The following are typical of the views expressed:

The Training Centre helps to create and support a culture of evidence-based decision-making and, most importantly, provides graduates who bridge the worlds of research and service. Strengthening these links is critically important.

(Atlantic DM)

Through the diverse interests of the faculty, the good connectedness to the health delivery field and to government, and through the biannual educational sessions they host, the ARTC positions evidence from research as an important tool for service delivery and policy decisions. By modelling and teaching it, the ARTC helps to normalize and routinize the regular use of evidence for decisions, large and small.

(Atlantic DM)

Le Centre permet la formation de chercheurs en gestion des soins infirmiers qui développent de nouvelles connaissances pouvant expliquer les phénomènes d'organisation du travail et leurs effets sur la qualité des soins. [The Centre enables healthcare researchers of the future to develop new knowledge that enables them to explain workplace phenomena and their effects on the quality of care.]

(Quebec DM)

An opportunity to access the latest trends in management with the presence of a FERASI student. Access to the management conferences all over the world. Discussion groups and participation in research forums with other organizations.

(Quebec DM)

The capacity building is the biggest value. The OTC prepares many for research, and to use research in their decision-making.

(Ontario DM)

Creates students that are well prepared to work in decision-maker organizations with a strong understanding of the research applications.

(Western DM)

In short, the RTCs appear to have had a significant influence on increasing the capacity of and for health services researchers in Canada.

Ability to effect cultural change

Respondents were asked to assess the degree of cultural change that occurred within their organization as a result of their association with the RTC. While respondents clearly indicated that the RTCs had a positive impact, the degree to which the RTCs were successful in effecting cultural change in DM organizations was not as encouraging. (Figure 2). Half the respondents rated the RTCs' ability to effect cultural change as “some.” Very few DMs indicated that the RTC had a more positive impact in influencing and enhancing the organizational culture of evidence-informed decision-making.

FIGURE 2.

Impact of RTCs on decision-maker organizational culture (percentage; N=18)

Respondents were asked to indicate why the RTCs were less effective in influencing cultural change. Most of those who stated that their association with the RTC had not resulted in any cultural change said the reason was that their organization was already aware of the importance of evidence-based decision-making. The following comments illustrate this perspective:

We are already sensitive to these issues.

(Atlantic DM)

Je ne crois pas qu'un changement de culture peut être fait par l'assistance à des conférences. Je suis cependant convaincue que la diplomation des premiers étudiants et leur implication macroscopique dans des projets d'organisation des soins et services infirmiers feront une différence. [I do not believe that a change in culture may be brought about through conferences. I am convinced that the graduation of the first students and their long-term involvement in the organization of care and nursing services will make a difference.]

(Quebec DM)

Our organization has been committed to evidence-based decision-making, so the participation with OTC has not made a significant cultural shift. Our organization was already there.

(Ontario DM)

The organization's strong adoption of a more evidence-based culture predated involvement in WRTC.

(Western DM)

The following comments were made by decision-makers who indicated that an association with the RTC had led to a shift in organizational culture:

We have significantly improved our analysis of complex decisions/issues – as well, it has increased our involvement in all types of research both locally and at a national level, in collaboration with other partners.

(Atlantic DM)

More reliance on evidence both in discussions and decision-making. There is a growing movement to include literature in the discussions.

(Quebec DM)

There is an increased awareness and capacity among practitioners for using, understanding and promoting the use of evidence in delivering programs and services.

(Ontario DM)

Linking research activity back to the field practitioners in a meaningful way has resulted in increased participation in research.

(Western DM)

There is an increased awareness of skills brought by those in a training program – increased openness to outside views.

(Western DM)

It is not surprising that the fourth-year review noted considerable variation among decision-makers about the ability of the RTCs to influence the culture of evidence-informed decision-making. Changing organizational culture is a long and difficult process, and considerable resistance is always encountered throughout the journey. Nevertheless, as the quotations above illustrate, some decision-makers were quick to realize the potential contribution of the RTCs to enhancing their organization's approach to and utilization of evidence-informed decision-making.

Experience and value of internships/placements

The opportunity for students as part of their learning experience to apply their developing research skills in a decision-making environment is a key component of the RTC programs. Students undertake a residency with a DM partner. This experience is designed to develop an understanding between both parties (the student and the DM organization) of how knowledge is transferred between the academic community and decision-makers, and to provide the student with hands-on research and decision-making experience. The CHSRF survey asked respondents who had provided an internship or placement to first rate the experience from their own perspective (Figure 3) and then to indicate the extent to which hosting a student provided added value to the organization (Figure 4).

FIGURE 3.

Decision-makers' rating of internship/placement experience (percentage; N=12)

FIGURE 4.

Decision-makers' rating of added value of internship/placement (percentage; N=18)

The healthcare organizations that hosted an RTC intern or resident generally found the experience very beneficial. Figure 3 indicates that on the whole, DMs were positive about the value of hosting an intern, with 75% giving the experience a rating of “good” or “very good.”

The following comments from DMs indicate the mutual benefits of the internship or placement to both the RTC student and the DM organization:

It was a great experience to have [the student] with us – research often assists us in our work. However, it is often more anecdotal than pure research. The learning was mutual for both [the student] and our organization. We learned more about health research … .

(Western DM)

[The student] has been an asset to [our team] and to the work of the department during her placement. She has demonstrated a willing and positive attitude, has contributed actively to [the] work of [the team] beyond her immediate assignment … .

(Western DM)

Hosting an RTC internship or placement was not a one-sided arrangement for decision-makers and their organization. There was clear added value to the organization in doing so. As Figure 4 demonstrates, nearly 60% of the respondents stated that the added value of hosting an intern was “good” or “very good.”

The respondents' comments about their experiences with interns illustrate the contribution that the RTC interns made to the organizations:

Introducing a research-focused and analytical mind to public policy initiatives supported our efforts to be more evidence-based. Besides, the individual in question was a good, hard-working employee.

(Atlantic DM)

Par son apport aux discussions dans le cadre des réunions, par sa recherche d'étude avec données probantes pouvant alimenter nos décisions ou recadrer nos changements en organisation du travail en soins infirmiers. [Through his/her contributions to discussions within the group, through his/her research studies with probing questions, [the intern] was able to influence our decisions about where to refocus the organization of our nursing care.]

(Quebec DM)

The intern helped us with key initiatives that needed research/evidence to support them. The individual was extremely helpful in finding research to support decision-making.

(Ontario DM)

The student was able to provide us with several pertinent papers and summaries of evidence that supported work in progress.

(Western DM)

Discussion

It has been several years since the CHSRF fourth-year survey was conducted. Since then, as the papers in this special journal issue attest, the RTCs have made considerable process in engaging and enhancing ties with decision-makers. One of the more notable initiatives that have strengthened the linkages between the RTCs and DMs has been the Executive Training in Research Application (EXTRA) program. (See Conrad 2008.) The RTCs act as regional mentoring centres for EXTRA by providing its Fellows with both academic and organizational mentors and facilitating the development and implementation of the Fellows intervention project. Through this facilitating role, the RTCs not only help decision-makers understand the significance of evidence-informed decision-making, but through the interactions among RTC students, EXTRA Fellows and DMs, the RTCs influence the processes that local health services organizations use to address current healthcare issues.

Successes

Faced with persistent gaps in research and evidence in a number of policy and program files, a main objective for decision-makers in working with the RTCs involved increasing capacity in applied health services research in targeted areas. Capacity building is a dynamic process. By working directly with student researchers in underserved research areas, decision-makers have an opportunity to gain knowledge in the short term while generating interest in key topics over the longer term. The expectation on the part of the RTCs is that this first-hand exposure will lead researchers to pursue these areas of inquiry further and contribute to an overall understanding of key aspects of applied health services.

The impact of the RTC initiative on organizational culture is more challenging to quantify, and generalizations are difficult. Where organizational participants were more familiar with research, the initiative reinforced positive attitudes and behaviours related to both the use of evidence and the link between researchers and decision-makers.

Some organizations are much less experienced as participants in and users of research. For many of these, their work alongside the RTC students represented their first close involvement in research activities. Here, the initiative played a key role in laying a foundation for better understanding and use of research.

The RTCs have seen the interrelationship with decision-makers grow and evolve since the conclusion of the CHSRF fourth-year review. Although certain DMs are more open to accepting RTC students than others, more and more DMs contact their respective RTCs to enquire about hosting a student intern or resident. This willingness is an acknowledgement by the DM organization that there is added value for the institution in hosting an RTC intern or resident. Those organizations that may not be overly strong in evidence-informed decision-making nonetheless still request an intern or resident because they appreciate that the intern brings to the organization a set of skills that they either currently lack or need to develop. The following example illustrates how one healthcare organization utilized RTC interns.

The organization engaged OTC students in two distinct projects. For the first, a team of students conducted key informant interviews, document reviews, analysis and reporting related to the implementation of a high-profile initiative. In the second, a student undertook a review of and provided presentations on instruments and processes used internationally to measure the strength of “system integration” in the human services context.

From an organizational perspective, both exercises resulted in positive impacts for decision-makers and planners, particularly in terms of the knowledge gained. The students applied their research and analytical skills to a variety of key challenges and, in each case, provided insight into the issue at hand. The students' work has been important in shaping the organizations' understanding of issues, and to this day endures as a basis for analysis.

The students clearly benefited most in terms of their exposure to and knowledge gained about the machinery of government and the application of research in a real-world context. As one of the organization's decision-makers noted,

“Their actual research skills were improved”

(Ontario DM)

.

Challenges

The ongoing challenge for the RTCs is how to ensure that the gains made so far are sustained. For example, have the RTCs succeeded in increasing health services researcher capacity and decision-maker receptivity? There is anecdotal evidence to indicate that progress has been made, as DM organizations are employing RTC graduates. However, this indicator is not the only measure of success, nor should it be.

Decision-maker participation in RTC workshops and related activities, such as the OTC's Summer Institute, has the objective of modelling activities that build stronger relationships between DMs and researchers. In some cases, DM organizations have only a basic understanding of how to shape and sustain these interactions. As one Ontario decision-maker put it: “My work with the OTC has provided an opportunity to understand how we can make these relationships work better.”

Such experiences are extremely valuable to those involved. However, for the RTCs, the challenge remains to determine whether there is a meaningful way to measure the impact beyond the individual value of each of the activities.

For example, many DMs do not have a clear sense of whether the RTCs have made progress on the objective of leveraging these opportunities to build much-needed capacity in underserved research areas. The connection with the EXTRA Program is a step in the right direction, but more should and could be done. Do we know how many more researchers are choosing to pursue specific organizational topics as a result of the RTCs? Have the various efforts of the RTCs succeeded in establishing better relationships between researchers and decision-makers?

I absolutely support the work of [the] RTCs but think there may be some value in developing joint measures that allow all of us to mark progress and measure change.

(Ontario DM)

Conclusion

It is clear from the CHSRF decision-makers' survey that the RTCs have had an impact. RTC faculty and students have contributed to the better understanding among decision-makers of the importance and value of evidence-informed decision-making.

Perhaps the greatest compliment that DM organizations have paid to the RTCs is that many have employed the graduates. It is not unusual for a student to be offered employment with the organization that provided his or her work-based placement.

There is no question that CHSRF has significantly changed the landscape in terms of strengthening the relationship between decision-makers and researchers. However, within that overall agenda, it is less clear, in the absence of indicators/targets of success, whether the RTCs have produced more researchers involved in health services research or whether the gaps in evidence are less significant than they were prior to the RTCs' creation.

It is abundantly clear that the RTCs cannot afford to become complacent. They are making an impact, both in terms of increasing the pool of applied health services researchers in Canada and of influencing and strengthening the culture of evidence-informed decision-making. However, as the RTCs mature they must exert greater effort and emphasis towards developing the “metric” that will enable decision-makers to see the collective impact of their substantive activities.