Participants’ Perceptions of a Multidisciplinary Training Program for Graduate and Postgraduate Students in Drug Use Management and Policy

Abstract

Objective. To determine graduate and postgraduate students’ perceptions of a drug use management and policy program that applied wide-ranging policy research skills to inform pharmaceutical decision-making.

Design. Nine cohorts of graduate and postgraduate students from diverse academic and professional backgrounds were paired with health-system preceptors for 4 months, and supported by faculty advisors and administrators, to complete research projects that generated evidence to inform policy decisions.

Assessment. A self-administered survey instrument was sent to all alumni of the program over the previous 10 years. The majority of respondents indicated: their prior academic coursework could be applied to everyday life; service-learning projects complemented university programs; participation led to greater awareness of decision-makers’ needs and appreciation of their tacit knowledge; and communication abilities were enhanced with decision-makers, and academics. Many also reported personal desire to fulfill healthcare-system research needs; personal belief in their ability to make a difference; and increased postgraduation marketability.

Conclusion. A drug use management and policy program allowed graduate students from various disciplines to develop new skills and collaborate with experts to produce research evidence that was relevant to drug policy that addressed real-world problems.

INTRODUCTION

Drug use management and policy research is of increasing importance to federal governments in order to improve the financing, delivery, and management of health care. Conducting research that addresses these issues requires increased human capacity (ie, more people with the appropriate skills engaged in this research). To respond to this need, the College of Pharmacy at Dalhousie University in Halifax, Canada, developed the Drug Use Management and Policy Residency Program in 2000.¹ (Although not a residency program in the US professional education sense, this service-learning experience was referred to as a residency program because the graduate and postgraduate students were placed or matched with preceptors and hosted by the healthcare organization for the duration of the program.) This program, open to learners from a wide spectrum of graduate programs (MSc, PhD students, and postdoctoral fellowships), used a service-learning paradigm to coproduce relevant pharmaceutical policy and management research with community partners from provincial government departments and health services delivery organizations.

The program was distinct because not all participants were licensed pharmacists and the focus was on developing research evidence to inform drug policies (eg, findings from systematic reviews, quality improvement initiatives, or evaluation of Nova Scotia hospital or federal/provincial public drug insurance programs). Consequently, residents did not provide clinical care to patients or deliver services to patients during the program. About a third of the participants were pharmacists and the remainder were nonpharmacists with skills relevant to the research conducted.

Most graduate programs encompass a single disciplinary approach with specific content and research methods taught in controlled environments and involve traditional dissemination approaches to academic audiences.^2-5 Although the transdisciplinary and multi-sector nature of applied health services research is recognized, many graduate programs provide limited opportunities for conducting research in applied settings, which allows students to gain an understanding of how research is accessed, appraised, and applied in the “real world” of decision-making.^2,5 Because many health services research graduate students are finding employment with government and health care organizations and not academia, the lack of hands-on experience with real world health policy is of concern.^6,7

More exposure of students from undergraduate and graduate pharmacy and health profession programs to community-based clinical, program management, and policy environments is needed, and this has been promoted in guidelines, standards, and discussion papers. For example, the Canadian Interprofessional Health Collaboration developed a nationwide competency framework which requires incorporation of knowledge, skills, attitudes, and values needed to make judgements and apply them to practice settings.⁸ Competencies identified by the Accreditation Council for Pharmacy Education (ACPE)⁹ and the Center for the Advancement of Pharmaceutical Education (CAPE),¹⁰ although designed for pharmacy education, can be applicable to graduate programs. Items of particular relevance include: establishing linkages between decision-makers and current and future researchers, developing population-specific, evidence-based disease management programs, and facilitating students’ commitments to become independent, self-initiated life-long learners.^8-10 Morgan and colleagues identified a framework of attributes for master’s programs in health services and policy research which included the need to formulate important research questions and apply knowledge to complex environments.¹¹

A common element to the many definitions of health services research is producing evidence to inform health system improvements.¹² Engaged scholarship is one approach where researchers and their graduate students are linked with decision-makers in mutually beneficial partnerships which can be transdisciplinary often integrating multiple forms of scholarship.¹³ Service-learning, one of the educational pedagogies through which engaged scholarship is implemented, is defined as “an educational methodology which combines community service with explicit academic learning objectives, preparation for community work, and deliberate reflection”¹⁴ and “integrate[s] the scholarship of teaching, application, and engagement…”.¹³ Engaging decision-makers, academics, and students provides participants with a deep understanding of the health care context and helps them to identify the most fruitful research directions. The student becomes the bridge between those who need policy-relevant research and academics who oversee its production in a rigorous manner.

The service-learning literature describes various program models.^14-18 Most programs target undergraduate students who provide direct and indirect service to the community. There are however graduate programs, such as health services administration, health services and policy research, and specific health professional training programs involving residency placements within the health care sector where students work alongside preceptors and/or conduct specific projects.^11,19,20

Research about the potential impact on students’ learning from participation in service-learning programs continues to be needed.¹⁴ Some evaluations have focussed on understanding critical success factors, and benefits and challenges encountered by all participants.^17,21,22 Short-term outcomes (eg, changes in attitudes and skills) and intermediate and longer-term outcomes (eg, the impact of research methods used, and impact on career development) are being assessed.¹⁴

Eyler and colleagues¹⁷ categorized evaluation research related to students’ service-learning under 2 major headings: (1) the effects of service-learning experiences on students and (2) the effects of particular program characteristics on students. Effects of the student-learning experience included positive individual outcomes, such as personal development, interpersonal growth, leadership ability and communication skills; positive societal outcomes, such as responsibility and commitment to service; and mostly positive effects on reducing stereotypes and facilitating cultural and racial understanding. In terms of students’ learning outcomes, service-learning has demonstrated a positive effect on academic knowledge and skills in areas of understanding complex issues, problem solving, critical thinking and cognitive development, and allowing students to apply learning in “real world” settings. There were positive results on career development and relationships with participants’ host organizations; however, the effect of service-learning measured by course grades or grade point average was mixed.¹⁷

Key program characteristics identified by Eyler and colleagues¹⁷ included quality of placement, quality and quantity of reflective activities, application of service experience to academic content (and vice versa), duration and intensity of service, exposure to diversity, community voice, quality of feedback, and an academic linkage with applied learning including requirements for oral and written reflection. The quality of the service-learning placement was seen as a predictor of students’ learning, their personal and interpersonal skill development, and development of close connections to faculty members. In quality placements, students were actively involved, performed a wide variety of tasks, and had important levels of responsibility.¹⁷ Research is limited about long-term implications for graduate students’ in health sciences participating in service-learning, especially with pharmacy and drug policy education. The drug use management and policy program described here was developed as a component of the 10-year Canadian Health Services Research Foundation (CHSRF)/Canadian Institutes of Health Research (CIHR) Research Scientist Chair Award granted to Dr. Ingrid Sketris at the College of Pharmacy.²³ Its conceptual framework and program elements have been previously described.¹

A critical feature of this award was the expectation that these research chairs capitalize on their established research credibility to fast track methods aimed at decision-maker engagement, while focusing on sustained interactions through innovative linkage and exchange activities.²³

According to Denis and colleagues²⁴ the practice of evidence-based medicine had a precipitating effect in identifying the need for a parallel approach in evidence-informed health system management. To support this cultural shift, applied health services researchers need to acquire skills in how to transfer research findings and engage with health-system policy and decision-makers, while decision-makers need to gain a greater appreciation for the role of policy-relevant research.³ It is within this context that the program is situated.

This paper describes findings from a retrospective evaluation of the 9 cohorts of residents who completed the program. We evaluated the implications of the residency program from the graduate students’ perspective using a survey instrument that specifically focused on their experience with service-learning, engagement with the health care community, and potential influences on their academic and career development. The goal for this study was to assess students’ perspectives on their residency experience, attitudes toward their community involvement, and the influence of the residency on their academic and career decisions.

DESIGN

Service-learning guided the design of the program, where graduate students from various disciplines gained experience in coproducing research.¹ By adhering to service-learning principles,¹⁴ the program was specifically intended to ensure the needs of 2 key stakeholders, the Dalhousie University academic community and community-based organizations, were satisfied. The residency consisted of a 4-month placement at organizations such as government departments and health care delivery organizations, where preceptors offered guidance. Because the program was built on the foundation of engaged scholarship, this residency provided students with an opportunity to apply theory, knowledge, and skills to “real world” problems, and differed from a traditional investigator-driven approach to knowledge creation, which continues to dominate graduate training programs.² Because community-engaged scholarship and service-learning are synergistic, each research project was identified and guided by the decision-maker’s/preceptor’s knowledge needs, coupled with methodological advice from academic mentors. Service-learning fits with the teaching methodology known as andragogy because it is learner focused and emphasizes learning by doing. The instructors (faculty advisors and the program director) complemented this approach by each functioning as a coach for a resident/preceptor team that carried out the policy project.

The program attracted graduate students attending Dalhousie University and other Atlantic Canada universities enrolled in a variety of disciplinary programs including community health and epidemiology, economics, health informatics, health services administration, law, medicine, nursing, occupational therapy, and psychology. It provided interdisciplinary education in a “real world” setting where research skills learned in disciplinary-based graduate education were applied to the design and implementation of pharmaceutical policy projects. Residents from diverse academic programs and health care-related experiences conducted research projects to inform decision-makers about drug therapies. Each resident became part of a team that included preceptors from the host organization; academic advisors from their graduate program who provided guidance in research methods design and theories and data analysis; and the program director, program coordinator, and administrative assistant.

A key feature of service-learning programs is the inclusion of multiple ways for student learning to occur.^{14, 21} We used various learning strategies in the program.¹ Based on Fink’s framework,²⁵ we identified that Bloom’s taxonomy, which describes hierarchical sequencing of 6 kinds of learning (evaluation, synthesis, analysis, application, comprehension, and recall knowledge) under the cognitive domain, had limitations as an educational framework in explaining our program design. Fink maintained that individuals and organizations involved in higher education have expressed a need for important kinds of learning that do not emerge easily from the Bloom taxonomy. These include learning how to learn, leadership and interpersonal skills, ethics, communication skills, character, tolerance, the ability to adapt to change, etc. We believed Fink’s taxonomy of significant learning²⁵ provided a better fit with the key elements of the program. Foundational knowledge was built through skills seminars, application was achieved through individualized research projects, integration was attained through placements in host organizations with unique drug policy questions, the human dimension was completed by working as part of a multidisciplinary team, caring was reached through becoming aware of decision-makers’ knowledge needs, and learning how to learn was achieved through the reflective aspects of the residency.

Given the intricacies and risks associated with placing graduate students in practice and policy settings, numerous best practices coupled with tacit knowledge of program staff members, directed program development, and implementation.^1,20,23 In accordance with service-learning best practices, various types of supports were built into the program to ensure success. For example, the graduate students, preceptors, and faculty advisors were oriented to the residency program by the program director and program coordinator.

A robust educational component developed residents’ and host organizations’ knowledge and skills through exposure to broader academic and health care knowledge and skills. Each group of residents participated in 5 skills workshops (eg, searching drug-related literature, management of references, reflective learning, writing briefing notes for decision-makers, media training) and 8-10 relevant drug policy seminars/presentations (eg, applying for ethics approval, systematic literature reviews, Canada’s and Nova Scotia’s health care systems, the legislative process in Nova Scotia, ethical issues and pharmaceuticals, etc). The majority of these workshops and seminars/presentations were delivered by Dalhousie University faculty and staff members and by decision-makers from the Nova Scotia government.

Participants in the residency program received ongoing guidance from the program director and program coordinator. Methodological expertise was provided by the program director and faculty/academic advisors who either had experience and/or interest in community-engaged scholarship. Residents kept learning journals and submitted a learning portfolio upon completion. A critical component involved the organization of a final invitational workshop on an annual basis where the research results from each cohort were presented to decision-makers and academic audiences. Following completion of the program, most residents were assisted by the program director and program coordinator to disseminate results from their projects using appropriate knowledge translation approaches including submission of abstracts, posters, and journal publications to relevant audiences.

The residents experienced a bird’s-eye view of the political and organizational context in which policy is developed and decisions are made. They documented their reflections in “learning journals” and gained confidence from applying their research skills, talking with decision-makers, and even recommending policy options by assisting with the preparation of briefing notes for government. The residency program illustrated the reciprocal aspects of the service-learning model well: while the host agencies provided research input including data and other practical resources, the university provided scientific knowledge, methods, and access to specialized expertise. The residents developed and presented relevant and applicable evidence – the kind of evidence that could be used by the host organizations to inform practice, policy, and program development, while at the same time being of sufficiently high quality that in most cases the projects resulted in peer-reviewed presentations at scientific conferences and publications in academic journals.

Over a 10-year period, 38 students conducted 38 unique projects, including syntheses of previous research, surveys of national practices, development of conceptual frameworks, and conducting primary research. Regardless of the type of project, each resident worked one-on-one with a decision-maker and learned how to apply the skills and values of health services research to authentic questions and issues in the field of drug policy. (A summary of residency projects is available from the corresponding author). Various research methods were used in the residents’ projects (eg, surveys, systematic review and meta-analysis, time series analysis, qualitative analysis, economic evaluation, secondary analysis of administrative databases) which focused on various therapeutic research topics (eg, diabetes care, controlled drug prescription monitoring, prescription cost-drivers, immunization, chronic obstructive pulmonary disease [COPD], drug information resources, physician profiling, electronic medical records) and various health care environments (eg, provincial government, family medicine, hospital and long-term care). Given these complexities, challenges were encountered, such as: determining the scope of the project to be undertaken by each resident so that it could be completed within the timeframe of the residency; assessing the skills and educational backgrounds of each resident and determining a match between the resident and a host organization’s knowledge needs; providing extra guidance for those residents without health care or other work experience or pharmacy background, and for residents whose first language was not English; obtaining ethics approval for each project; and gaining access to data for each project in a timely manner.

EVALUATION AND ASSESSMENT

Evaluation was a key component of the residency program design. Each cohort of residents, preceptors, and faculty advisors completed various post-residency evaluation forms included in the program manual. Although completed annually, these evaluations, along with a formative evaluation¹ conducted in the early stages of the program, did not provide us with appropriate data to assess the long-term program benefits and outcomes for residents.

A self-assessment survey was used to evaluate the overall impact of the program from the resident’s perspective. Ethics approval was initially received from Dalhousie University Health Sciences and Humanities Human Research Ethics Board on September 12, 2008, with annual renewals obtained thereafter. The key research questions that guided the study of post-program resident outcomes included: (1) What knowledge and skills did residents gain from participating in the residency program? (2) Did the residency program affect residents’ perception of self and others (decision-makers and faculty advisors)? (3) Did the residency program promote a focus on evidence-informed attitudes and behaviors? (4) Did the residency program influence the development of research capacity among residents and the host organization? (5) How did factors of age, gender, academic discipline, and presence/absence of work experience affect the outcomes of the residency program?

Two existing self-report instruments^14,26 for evaluating students’ service-learning experiences were adapted to the Nova Scotia context and the program. Our modified 36-item survey included 6 sections: demographic characteristics; residents’ perceptions of the residency experience; residents’ attitudes toward working with community-based host organizations to coproduce research evidence; the influence of the residency experience on graduate programs and career choices; residents’ reflections on the personal benefits they derived from this experience; and an opportunity where residents could provide personal comments.

For the closed-ended statements there were 5 response options: strongly disagree, disagree, neutral, agree, and strongly agree. In accordance with survey research procedures, selected statements were repeated using both negative and positive approaches. Because the survey instrument was adapted, it was pilot tested using 2 volunteers outside the study population.

Between 2001 when the program began and 2010, 9 cohorts (38 residents) were trained. There was no cohort in 2007 because the program director was on a sabbatical.

The residents were hosted at 4 distinct Canadian health policy and decision-making organizations located in Halifax: the Nova Scotia Department of Health and Wellness (formerly the Department of Health, n=20 residents); Capital District Health Authority (n=15 residents); IWK Health Center (n=2 residents); and Northwood Inc. (n=1 resident). Each resident received a stipend of $10,000. Over the 10-year period, the placements involved 31 preceptors from the host organizations and 26 faculty advisors, none of whom received any monetary compensation outside of their regular salaries. Grant-paid staff members included a program coordinator and an administrative assistant; the program director held a tenured faculty appointment at Dalhousie University.

All former residents were contacted by mail, e-mail, or phone during the fall of 2009 to determine their interest in participating in an evaluation of their residency experience. Confirmation of interest was received from all residents. A consent to participate form was sent, and following receipt of the signed form, the survey instrument was distributed electronically in a Microsoft Word document during July 2010. Up to 3 reminders were sent to nonresponders, and in January 2011, the survey was closed.

Thirty-five of 38 surveys were returned (92% response rate); all 9 cohorts were represented. Based on the size of the study, the 5 response choices were collapsed into 3 categories for ease of analysis: neutral responses were kept as a standalone category, while strongly disagree and disagree responses and agree and strongly agree responses were each combined.

The demographics of the respondents varied significantly (Table 1). One-third was over the age of 35 years, with a range of 20 to 50 years. Although the residents’ disciplinary backgrounds varied widely and reflected a multidisciplinary group, 50% had health professional and/or clinical experience. In the program, 23.7% (9/38) of participants were Canadian trained pharmacists (8 of 9 were licensed); 10.5% (4/38) were foreign-trained pharmacists not licensed in Canada; and the remaining 65.8 % (25/38) were nonpharmacists. The demographics of the respondents were only minimally different from those of the overall resident population.

Table 1.

Demographic Characteristics of Survey Respondents

Statistical analyses did not identify any obvious trends except for age. Using the age dichotomy (under age 35 years and age 35 years and over), 1 item differed significantly: “I was able to work directly with a residency decision-maker”. One hundred percent of older residents agreed with this statement, while only 64% of younger residents agreed with it (p=0.05).

Participants’ responses to 26 survey statements are provided in Table 2. Fifteen residents provided qualitative comments, including specific benefits received (eg. “hands-on experience”, “broader perspective and insights”, “furthered career aspirations”, “opportunity to confront weaknesses in my skills set”, “moral support”).

Table 2.

Responses to the Resident Survey: Service-Learning Assessment (n=35 )

DISCUSSION

The perspectives on service-learning provided by the graduate students in the program support the positive outcome findings reported in the literature.^18,27 What is different is that the graduate students in our residency program came from a wide spectrum of academic programs, had a range of health care backgrounds and experiences, and their ages were widespread. Some had no work experience, while others had professional designations and had spent years working in health care.

The study found that those older and more experienced residents came into the program with well-established career goals and academic expertise. For these participants, the residency program provided an opportunity to refine and hone existing skills by designing a research project that generated evidence to solve current policy problems. On the other hand, for younger residents who had limited life and work experience, the program was a steep learning curve about health care issues in general and drug policy specifically. However, the program design allowed for this learning to occur within the context of a “safety net” provided by the resident’s preceptor, faculty advisor, program director, and program coordinator.

Residents’ responses indicated they recognized the importance of doing policy-relevant research in accordance with the needs of identified decision-makers. Residents’ liked the experiential learning adventure, a fundamental aspect of the service-learning paradigm. The positive response (agree/strongly agree) of 80% for statement (“…the residency helped me better understand my graduate work, courses and academic training”) supports the complementarity of service-learning with graduate-level education. Respondents also assigned a high value to their residency placement, despite the widely divergent nature of the community-based organizations and program settings where they were hosted, and the differences in preceptors’ backgrounds.

Those residents inexperienced in working with decision and policy makers at the outset of the program increased their confidence and took calculated risks during their residency. They were adventurous and became actively engaged. This attitude assisted with developing skills in how to communicate with decision and policy makers in tandem with building a desire to make a difference in the health care sector. Other respondents commented that they were able to build new skills or augment existing skills. A few residents faced personal challenges during the residency. One-third of the respondents agreed/strongly agreed they had become aware of some of their own biases and prejudices during the residency. Recognition of this limitation provided an opportunity for the resident to develop strategies with the program team to overcome these challenges. The majority of residents reported the residency helped them communicate their ideas in a decision-making environment and academic setting. When reflecting on their residency experience, most thought they could make a difference in the health care sector.

In the survey, over three-quarters of the residents acknowledged a responsibility to see that their research was used to inform policy and practice, and many observed that they could make a difference in the health care sector by coproducing policy-relevant knowledge. The residents were not only supported in communicating their findings with decision-maker audiences through presentations and verbal and written briefings, but also were mentored in disseminating their project results through traditional academic channels. Following the completion of the 4-month residency, many of the residents continued with their research, publishing 4 chapters/monographs and 18 publications, and making 39 professional presentations (details are available from the corresponding author).

The program played a key role in addressing several important concerns of drug policy decision-makers. New projects were often designed to build upon past residency project results, which applied residents’ unique research skills, backgrounds, and/or relational capital and synergies developed through ongoing linkages between the program director/faculty advisors and preceptors. This opportunistic approach resulted in an accumulation of outcomes in some therapeutic areas that were of greater value than the outcomes from any single project would have been.

The following case study illustrates how this coproduction and accumulation of scientific evidence from residents’ projects influenced decision-making. In this case, the drug policy problem involved changing how respiratory medication is administered. Each associated project produced evidence that served as a building block in support of this change (additional details are available from the corresponding author). Decision-makers first identified the problem: many patients presenting with asthma and COPD were unnecessarily receiving respiratory medications delivered by nebulization machines. For most patients, a portable inhaler is far less difficult to handle and is as effective as a nebulizer machine. Switching to portable inhalers could save money and potentially prevent problems such as bacterial infections that have been linked with regular use of nebulizers. The Drug Evaluation Alliance of Nova Scotia supported this change in treatment approach through education and development of specific reimbursement criteria, which altered how these medications were paid for by the publically funded Nova Scotia Pharmacare drug plan. Several residents contributed both quantitative and qualitative evidence that was used to inform and subsequently evaluate this policy decision.^28-33

Our findings confirm the conclusion by Gelmon and colleagues that “service-learning is clearly a powerful pedagogy with timely relevance to the new competencies demanded for future health professionals.”²² We would expand on this statement to include relevance for future health services researchers. Our results confirm the importance of integrating academic programs and community-based work in a seamless manner. Also, the inclusion of a series of targeted seminars provided a common base relevant to conducting research in drug policy. Regular meetings were a “safe space” for residents to critically reflect upon their learning and how well they were assimilating into the particular culture of their host organization.^34-36

This retrospective evaluation study is relevant as the findings confirm the feasibility and applicability of service-learning to graduate-level education in general and pharmaceutical policy in particular. Regardless of their age, academic discipline, level of graduate training, extent of health care experience or clinical knowledge, or presence/absence of a background in pharmacy, the respondents confirmed the value of the program. They were able to successfully transfer research skills and academic knowledge to complete pharmaceutical projects, thereby producing research evidence that could be used to solve decision-maker policy problems.

The findings from this study do have some limitations. The data gathered were self-reported and there was a significant time lag for some residents between completion of the residency and responding to the survey. Thus, earlier cohorts may have been better able to assess any influence their residency experience had on their career choice and marketability than more recent cohorts. Also, no comparison or control group was used in the study.

The Drug Use Management and Policy Residency Program, developed with grant funding, built research capacity for pharmaceutical policy through: graduate students in Atlantic Canada universities who subsequently found employment in academia, health care institutions, and research-oriented public and private organizations; graduate programs with involvement of multidisciplinary faculty members who served as research advisors; project outcomes within NS health care organizations that informed decisions about practice, programs and policies; and preceptor involvement in research that meets their organization’s or practice’s needs. Now that the funding has ended, it remains to be determined which elements of the program will be retained and which will have to be modified or discontinued.

SUMMARY

Participating graduate and postgraduate students believed the Drug Use Management and Policy Residency Program provided them with opportunities for community-engaged scholarship in a service-learning environment. The program provided access to research evidence needed by host organizations to address current issues in drug policy. They recognized that their projects provided a transdisciplinary interface for health care managers/professionals and faculty advisors to come together, which exemplified the value of engaged scholarship in a service-learning environment. We discovered that graduate students from various disciplines could produce research evidence relevant to drug policy; however, the need to “up skill” nonpharmacy students was established. All residents remarked on how new knowledge and skills were acquired and that the program provided the opportunity to address decision makers’ real-world problems.

Many projects informed decisions about clinical practice, programs, policies, and/or provided opportunities for ongoing research in some identified areas of concern.