Assessing Outcomes and Perceived Benefits of a Professional Development Seminar Series

Abstract

Objective. To evaluate the outcomes of alumni who were enrolled in a professional development seminar series during their doctor of pharmacy program.

Design. A weekly development seminar series was administered over 5 semesters with the goal of bringing academic advisees together to help develop performance-based abilities, prepare them for entry into the profession after graduation, and provide exposure to different career opportunities.

Assessment. A survey instrument containing 39 Likert-type scale items, 2 open-ended questions, and a 10-item demographic survey was created and content-validated to assess the effect of the seminar series on alumni advisees’ perceived outcomes and professional development since their graduation. The survey was electronically forwarded to advisees from the graduating classes of 2005 to 2012, and response data was collected with Qualtrics, a web-based survey service. A total of 36 percent of alumni responded to the survey. Respondents cited exposure to career alternatives and opportunities, development of presentation and communication skills, networking, and the importance of advisor/mentor relationships as benefits of the seminar series.

Conclusion. The professional development seminar series has demonstrated a positive impact on alumni advisees’ career development and professional outcomes, most notably relating to career path exposure, communication skills, and advisor/mentor relationships.

INTRODUCTION

In Background Paper II: Entry-level, Curricular Outcomes, Curricular Content, and Educational Process, the Commission to Implement Change in Pharmaceutical Education recommended that AACP member schools immediately commit themselves to curricular change.¹ Specifically, the schools would need to commit to engendering competencies and outcomes essential to pharmaceutical care and strengthening the effectiveness of pharmaceutical education. Additionally, the commission identified 10 general outcomes and competencies that underlie the education of a professional person and citizen. Within Background Paper II, the commission challenged faculty members to create a comprehensive curriculum that would provide, in part, experiences to develop communicative, interpersonal, and problem solving skills.¹ In December 2002, the University of Illinois at Chicago’s College of Pharmacy adopted 10 performance-based abilities that its students should possess at the time of graduation.

Several guidelines in ACPE’s Standards 2007 highlight the importance of academic advising and counseling in pharmacy education. Guideline 16.1 instructs colleges and schools of pharmacy to “provide academic advising and career-pathway counseling…coordinate the availability of personal counseling and …plan and participate in activities that support the development of students as professionals.”² Reinforcing the importance of these requirements, ACPE also suggests in Guideline 26.6 that the faculty evaluation process should recognize and value the important roles faculty members play as academic advisors and career pathway counselors.² Unfortunately, several barriers to fulfilling these guidelines exist, including time constraints related to teaching, research, clinical service, and scholarship obligations. Further, academic advising is an activity often perceived as not valued or rewarded, and the framework for providing such student services effectively and efficiently is often underdeveloped or not developed at all.

This curricular deficit inspired the design and implementation of the professional development seminar series.³ Additional inspiration for the series came from (1) the observed discrepancy between the wide variety of opportunities that exist for doctor of pharmacy graduates and the advisees’ limited knowledge of these opportunities, and (2) the observed failure of most curricula to address professional development adequately during the didactic years. Enrollment in the series was restricted to 2 faculty members’ advisees (up to approximately 18 students per faculty member per graduating class) to allow adequate time and resources for the facilitation of a mentor-mentee relationship with each student. The seminar series was developed based on Guideline 16.1 from ACPE’s curricular requirements. The objective of this study was to explore the overall perceived benefits and outcomes of the professional development seminar series on alumni who participated as academic advisees in the series from the classes of 2005 through 2012.

DESIGN

The professional development seminar series was originally created as a novel forum to provide academic advisees with thorough academic advising and career pathway counseling simultaneously in an efficient manner.³ The initial evaluation of the seminar series successfully demonstrated student development of various performance-based skills.³ Subsequent research demonstrated significant improvement in the development of self-efficacy among seminar participants, as well as the value of a microteaching exercise to help students develop their communication, critical-thinking, and problem-solving skills.^4-5 Zorek et al demonstrated the value of guest speakers in this series.⁶ Typically, time commitment for the instructor was less than 1 hour per week for preparation and 1 hour with the advisees in each session. Prior to each course, however, considerable time was devoted to contacting possible guest speakers and agreeing on their availability, purpose of presentation, and presentation dates. Having evaluated several components of the series, it was logical in this study to evaluate the overall program and its effectiveness in terms of alumni outcomes.

A survey instrument consisting of 39 Likert-type scale items with 2 open-ended questions and a demographic survey consisting of 10 questions was created. Likert-type scale items were divided into five categories: (1) relevance of the series for my development, (2) skills/confidence, (3) entry into the profession, (4) in reflection, and (5) other. The 2 open-ended questions asked the respondent to (1) list 3 main “take aways” of value gained from the series and (2) share any additional thoughts or reflections regarding the series. The short demographic portion at the end of the survey asked the respondent’s gender, class year, age range, number of semesters enrolled in the series, academic advisor name, previously earned degrees, postgraduate training completed, degrees and/or certifications earned after graduation, desire to remain connected to the development seminar series, and postgraduate involvement in local, state, and national organizations.

Prior to administering the survey, an external review of the survey items and open-ended questions for content validity was conducted by 9 external pharmacy faculty members with instructional design experience in the doctor of pharmacy curriculum. The panel was asked to evaluate each proposed item on a 5-point scale ranging from “highly pertinent/valid” to “highly invalid.” For item acceptance, a 70% decision rule was instituted, ie, 7 experts had to indicate the item was either highly pertinent or valid. A Claim of Exemption was submitted to the UIC Institutional Review Board and subsequently approved.

A total of 207 of the 2 authors’ academic advisees from the graduating classes of 2005 to 2012 were contacted via e-mail and asked to participate voluntarily in the survey. Instructors requested advisees to submit their personal e-mail addresses before graduation, or e-mails were supplied by the college’s department of pharmacy advancement. Follow-up e-mails were sent to all advisees at 1 month and at 4 months after the original e-mail as a reminder to complete the survey. Responses to the survey were kept confidential, and respondents were informed that data would only be reported in aggregate for purposes of statistical analysis and subsequent scientific manuscripts that may emanate from the research. Survey response data were collected and descriptive statistics were generated using Qualtrics survey software, (Qualtrics LLC, Provo UT) (Table 1).

Table 1.

Survey to Assess the Impact of the Professional Development Seminar Series on the Professional Career of UIC^a College of Pharmacy Graduates (response values in %, n=74 unless otherwise noted)

EVALUATION AND ASSESSMENT

Thirty-six percent of advisees (N=74) completed the survey. Of the survey respondents, 64% were female, 75% percent were between the ages of 23-32, and 59% reported having previously earned a bachelor’s degree. These characteristics were similar to those of the general college of pharmacy student population of the 2005-2012 graduating classes. Additionally, 77% of respondents had participated in the seminar series for either 4 or 5 semesters and 34% had completed postgraduate training. Sixty-four percent of respondents reported extracurricular involvement postgraduation, with 39% reporting being active in the local community, 27% reporting involvement in a national pharmacist association, and 16% reporting involvement in a state pharmacist association.

The vast majority of respondents from the surveyed classes appeared to perceive a benefit from participation in the professional development seminar series (Table 1). With regard to the relevance toward alumnus development, the seminar series provided skills directly applicable to the alumnus’ career (95%), contributed to the alumnus’ professional development (99%), and provided useful information on career opportunities (96%). With regard to developing the skills and confidence of the alumnus, the series helped alumni improve their oral presentation skills through the microteaching assignment and improve their oral presentation skills in other settings (91% and 89%, respectively). Further, the series helped alumni develop critical thinking skills by enabling them to think on their feet during the microteaching assignment (82%) and helped alumni learn the art of providing constructive feedback through peer evaluations of classmates’ microteaching presentations (88%).

With regard to entering the profession, the seminar series helped alumni identify their career interests (84%). When asked to reflect on the professional development seminar series, respondents stated it introduced them to different career opportunities (93%), enhanced their personal interaction skills (86%), helped them develop a positive and satisfying relationship with their academic advisor (96%), and instilled the belief that the series should become a core offering in the pharmacy curriculum (89%).

Several themes emerged from the open-ended section of the survey. Responses to the survey item asking alumni to list the 3 most important things they took away from the seminar series were categorized into discrete, mutually exclusive groups based on the general theme of the response. The most frequently cited benefits of the seminar series included: (1) exposure to career alternatives and opportunities (58%), (2) development of presentation and communication skills (53%), (3) development of networking skills (33%), and (4) the importance of advisor and mentor relationships (28%). Other alumni cited improvements in their ability to construct curriculum vitae and résumés, give and receive criticism, and embody professionalism. Responses regarding additional thoughts and reflections on the seminar series were overwhelmingly positive. Many alumni commented on the practical skills they gained from having a close relationship with an advisor that they would not have gained otherwise. Other alumni expressed that this close advisor-advisee relationship was unique to the UIC doctor of pharmacy program and to their experience with advisors in general. Alumni were also grateful for the opportunity to be exposed to new areas of pharmacy practice that they were previously unaware existed. Finally, several alumni described how they have incorporated components of the seminar series into their own precepting of pharmacy students.

DISCUSSION

The survey results suggested that many of the goals and objectives that served as impetus for the creation and development of the seminar series were successfully addressed. The most commonly cited benefits of the series included exposure to new career opportunities, development of communication and interpersonal skills, and the formation of a strong advisor-advisee relationship. All of these are important aspects of professional development that Background Paper II considered previously lacking in some capacity.¹ An additional noteworthy educational outcome was the consistent improvement in characteristics of professionalism reported by seminar participants. Embodying and demonstrating professionalism is an outcome cited in the mission statements and objectives of numerous colleges and schools of pharmacy nationwide; however, little direction is provided on how this is to be achieved. Brown and Ferrill defined professionalism as it relates to pharmacy student development using Bloom’s Taxonomy of Educational Objectives.⁷ They described professionalism as relating to attributes such as interpersonal, communication, and critical-thinking skills, as well as empathy and personal values germane to development as a health care professional.⁷ Based on this definition and the responses to our survey, it would appear that the professional development seminar series provided a useful approach for developing this trait.

From a faculty member perspective, the seminar series appeared to succeed in addressing the problem of providing high-quality academic advising to modest-sized groups of students simultaneously. This is evidenced by the vast majority of alumni advisees expressing the belief that the seminar series should be a core offering in the college of pharmacy curriculum. As the college revises its core curriculum, hopefully, it will provide a professional development offering of at least one hour. In addition, an online, 1-credit hour course is being created by one of the faculty members. It is intended to allow students to participate in discussion boards/chat rooms and view previously recorded presentations by guest speakers.

The survey results also demonstrated that the seminar series helped more than 85% of participants create a plan for selection of advanced pharmacy practice experiences (APPEs) and helped over 90% of participants develop an effective “road map” to achieve professional goals. Prior to implementation of the seminar series, these outcomes were also identified as existing pharmacy curriculum deficiencies needing to be addressed or resolved. Given the number of guest speakers providing insight into different career opportunities over the course of 5 semesters, it was of interest, too, to determine if the students’ professional goals would change during this series. Survey results demonstrated, however, that 68.9% of participants did not change their professional goals, and 60.3% of participants did not change their decision regarding the pursuit of postgraduate training as a result of the seminar series.

Academic advising on professional development facilitates effective mentor-mentee relationships crucial to identifying students’ interests and career objectives. In turn, this provides a framework for student assessment outside of traditional areas such as content knowledge of a particular discipline. The ability to assess student character and professionalism is becoming increasingly valued by both employers and postgraduate training programs. For example, the Pharmacy Online Residency Centralized Application Service (PhORCAS) now requires those serving as applicant references to assess the candidate’s professionalism, leadership/mentoring skills, and initiative as part of a standard letter of recommendation. Since its implementation, this series has enhanced the faculty’s ability to complete the PhORCAS and create letters of recommendation for their advisees seeking PGY1 residencies, regardless of practice setting.

The skills and abilities gained from the series were crucial in helping students develop into effective health care professionals. In addition, the series allowed students to explore a wide variety of career opportunities and potentially move into expanding clinical roles that aid in the ongoing shift of the profession from product-focused to patient-focused care. Finally, the seminar series was structured to allow advisees to learn from and mentor each other, which may translate to an ability in practice settings to educate future students and help them develop skills effectively.

There are several limitations regarding our findings. Instead of being randomly selected to participate, students learned of the series through word of mouth and personally selected the investigators to be their academic advisors. Students may have also chosen to enroll in the series because it provided credit for elective professional course requirements. Additionally, sampling bias may have been present in our study if series participants were composed primarily of highly motivated students who wished to gain the additional development and mentoring afforded by the series. This study also did not utilize a control group to assess confounding factors that may have been present in the college of pharmacy curriculum outside the seminar series. For example, students not participating in the seminar series may have been exposed to curriculum vitae writing workshops, public speaking opportunities, and alternative career pathways via student organizations and APPE/IPPE experiences. It is unknown whether such factors would have had a confounding effect on the positive study results we specifically attributed to the professional development seminar series.

The relatively low response rate to the survey may have also introduced non-response bias into the study. Aside from alumnus disinterest in the authors’ research and unwillingness to take time to complete and return the survey, the possibility that the invitation to participate was sent to inactive e-mail addresses could have contributed to the non-response. Recall bias may have also affected our results as the alumni surveyed had graduated from the program 1 to 8 years ago. Alumni graduating recently may have been able to reflect on the impact of the seminar series more accurately compared with those graduating in earlier years. Additionally, there was a higher number of respondents from more recently graduated classes than from earlier classes, which could have caused the results to not be generalizable across all classes studied. However, we don’t believe this significantly impacted the results because responses to survey questions were similar across all classes.

SUMMARY

The authors observed that providing academic advising and career-pathway counseling mandated by ACPE in Standards 2007 was frequently neglected in doctor of pharmacy programs. Additionally, mechanisms to develop student professionalism consistently and intentionally are often lacking during the on-campus, didactic portion of the doctor of pharmacy curriculum. This aspect of pharmacy education is crucial to the evolution of the profession from product- to patient-focused care. A professional development seminar series conducted by a few dedicated faculty members can help to overcome deficiencies in the college/school advising strategy. This research demonstrates the value of such a series on multiple tiers of student development. It addresses neglected aspects of pharmacy education while simultaneously providing student services to a large group of advisees in a time-efficient manner.