Utilization and Perceived Effectiveness of a Web-Based Faculty Development Seminar for International Medical Science Educators

Abstract

The goal of this study was to examine the utilization and perceived effectiveness of the International Association of Medical Science Educators (IAMSE) Webcast Audio Seminar Series (WAS) by participants at the individual and institutional levels. The Webcast Audio Seminar Series User Survey (WASUS) included multiple quantitative and qualitative measures of user perceptions of their experiences and overall quality. Data was collected using a 42-item survey that examined user identification, utilization, and perceived effectiveness of the IAMSE WAS as a faculty development tool. Quantitative measures were summarized using descriptive statistics, including frequencies, means, and standard deviations. Qualitative data was investigated using an iterative, inductive thematic coding method. Qualitative themes were summarized and applied to quantitative trends as explanatory mechanisms with the intent to provide a more nuanced narrative of the data. The survey was sent to all 2012–2017 WAS participants which provided a cross-sectional snapshot of WAS user perceptions over a substantial period of time. Fifty-two participants responded. Survey participants were asked to rate WAS sessions on twelve different quality components. Quality ratings were explored by user subscription types and user experiences with other web-based conferences. Users rated the WAS program very highly on all components from 2012 to 2017. The high level of perceived quality by users is likely an important reason why WAS participation has continued to grow since its implementation. Since the quality ratings were consistently high over a 5-year period in which the number of users also grew and organizers continue to add new interactive features for users, it is expected that this growth is sustainable.

Introduction

Training medical school faculty in educational pedagogy and practice requires significant resources, including time and money. The International Association of Medical Science Educators (IAMSE) webcast Audio Seminar Series (WAS) was established to assist in addressing those constraints. On April 18, 2002, IAMSE unveiled a new feature for members and friends, an Audio Seminar Series. As the Executive Director, Roger Koment, wrote in his announcement email to the DR-ED listserv, “This modality was created as another means (along with the Annual Association Meeting, peer-reviewed journal, and expanding website) for addressing our professional society’s goal of sharing information on teaching and learning the fundamental sciences of medicine.” The inaugural seminar was presented by Thomas Viggiano, MD, MEd, and focused on curriculum evaluation as a quality improvement process. The delivery mechanism was described as “essentially a conference call” which enabled participants to view slides on a website and hear the speaker via telephone. Members of the DR-ED listserv were encouraged to contact IAMSE to be considered for presenting future sessions.

From the first few standalone sessions, the seminars evolved into a 5–6 session series occurring twice a year and centered on a theme in medical education. The themes included topics of interest to medical educators including teaching and assessment modalities, use of technology, and trends in basic science education. In 2011, the theme-based series offerings increased to three series per year. In 2012, “Evolution and Revolution in Medical Education” became a regular yearly series that allowed for offerings of loosely related topics.

Seminars were originally created as a benefit of membership for members of IAMSE on a first-come first-serve basis. If spaces remained, they were offered to individuals within the professional community, primarily as a means of publicizing IAMSE and attracting new members. As time and the reputation of the web seminars progressed, schools began to set up speakerphones in conference rooms and request booking of “season tickets” as a means to deliver faculty development.

While the initial sessions were organized by IAMSE’s Executive Director and Membership Committee, the success of the program necessitated the creation of an ad-hoc committee whose charge was to formulate recommendations on issues such as the number of sessions per year, the process for determining topics and themes, whether a fee should be charged and who should pay it, and should subscriptions be individual or institutional. With the growth of the web seminar participation, IAMSE charged a standing committee, which assumed responsibility for selection of themes, topics, and presenters as well as financial support and evaluation of the program. Although early seminars were provided at no cost to participants, the implementation of an honorarium for presenters and increased administrative costs required that IAMSE charge a fee for both members and non-members.

The field of faculty development has grown substantially in the last decade [9]. The evidence for online faculty development is sparse and insubstantial [2]. Attempts to address faculty development challenges of expense and distance have included utilization of Facebook as a social media tool, which demonstrated its acceptability and accessibility [5]. Traditional journal clubs have been limited by the geography of participants, and an online medical education journal club can provide a valuable opportunity for continuing education and faculty development for both the participant and presenter [3]. Chiswell et al. [1] have shown webinars to be an effective, acceptable, accessible, and sustainable vehicle for delivering information and support to health professionals and cancer patients to reduce the impact of cancer. Hoke et al. [4] used webinars to provide a viable method of instruction and education for school personnel interested in strategies for improving school’s wellness environment and indicated that further investigation is necessary to explore the link between webinar participation and positive changes in school wellness environments.

Martin et al. [6] addressed the need for training clinician-educators, by designing and implementing an asynchronous, interactive webinar series detailing a systematic approach to medical education research and scholarship. The series introduced fundamental concepts in medical education scholarship to inspire self-directed study for motivated learners. A study exploring advantages and disadvantages of online context from the perspectives of 10 faculty, indicated that control of pace and continued access to resources were benefits of the online context, while lack of social interaction, intrinsic motivation, and accountability were challenges for faculty in completing professional development online [11]. Faculty development interventions have the ability to build community of practice among program participants and in the work place [10]. Peuler and McCallister [7] have demonstrated that planning and implementing online professional development opportunities is challenging but attainable.

The goal of this study is to examine the utilization and perceived effectiveness of the IAMSE WAS seminar series by participants at the individual and institutional levels.

Materials and Methods

Instrumentation

The Webcast Audio Seminar Series User Survey (WASUS) included multiple quantitative and qualitative measures of user perceptions of their experiences and overall quality. The original survey was developed by three charter members of IAMSE serving on the Webcast Audio Seminar committee. Data was collected using a 42-item survey that examined perceived effectiveness (18 items), user utilization (18 items), and user identification (6 items) of the IAMSE WAS as a faculty development tool. Cronbach’s alpha was used to evaluate the reliability of the 11-item perceived effectiveness Likert scale (α = 0.8), and five of the perceived effectiveness items were qualitative or open-ended. The user utilization items included a range of questions with topics ranging from how WAS sessions were organized at various institutions to WAS’s role in faculty development to its role in providing faculty credit and consideration for promotion and tenure decisions.

Quantitative measures were summarized using descriptive statistics, including frequencies, means, and standard deviations [8]. Qualitative data was investigated using an iterative, inductive thematic coding method. Qualitative themes were summarized and applied to quantitative trends as explanatory mechanisms with the intent to provide a more nuanced narrative of the data. The survey was sent to all 2012–2017 WAS participants which provided a cross-sectional snapshot of WAS user perceptions over an extended period of time. Fifty-two participants responded. Data was collected at a single time point which precluded any longitudinal inferences. Hence, the summaries in this analysis were entirely descriptive in nature.

IAMSE WAS Participants

To appropriately frame survey results with context, a summary of WAS registration by membership types, obtained from the WAS Committee, is presented below (Fig. 1). The data shows that the year-over-year variation in registration trend occurred similarly for all membership types. In 2016, however, there were substantial increases in both non-member and individual WAS registrations.

Fig. 1.

IAMSE WAS registration summary

Results

Sample

A total of 39 out of 52 (75%) of Webcast Audio Seminar Series (WAS) survey participants indicated that they used an institutional WAS subscription instead of an individual account. Although 96% of participants worked in medical schools, chiropractic schools and physician assistant schools were also represented in the survey sample. Moreover, 92% of participant institutions were located in the USA and Canada. The Dominican Republic, Dutch Caribbean, St. Vincent, and the Grenadines were also represented. Most (67%) respondents indicated that their institutions participated in the WAS sessions at a single site/location; however, some (33%) respondents indicated that their institutions hosted WAS sessions at two or more sites.

The average WAS session group was approximately eight participants, but group sizes ranged from one to 25 participants. Moreover, the average group was composed of approximately 51% basic science faculty participants, and although all groups included at least 10% basic science faculty, some groups were composed of as high as 100% basic science faculty. Almost all groups included some clinical faculty members and the average group included approximately 27% clinical faculty members. Finally, the average group was composed of about 14% administrators and 8% staff. Table 1 shows the average length of subscription time disaggregated by group composition type. The participants in groups with predominantly basic science faculty participants and the participants in groups with equal compositions had subscribed to WAS longer than participants in groups with other compositions. Since data was collected at a single point in time and not on an ongoing basis, it was not clear whether this was indicative of a recent increase in clinical faculty and administrator subscriptions or whether belonging to a particular group composition (i.e., predominantly basic science faculty) increased the likelihood of a prolonged subscription. Nevertheless, the descriptive data clearly showed that the survey sample represented a wide range of WAS participants in terms of geographic locations, subscription types and lengths, group sizes, and compositions.

Table 1.

Group compositions and subscription length

	Subscription length
	1 series**		1–2 years		2–3 years		5 +		All
WAS group participant compositions	N	%	N	%	N	%	N	%	N
Predominantly basic science faculty	1	4	6	25	10	42	7	29	24
Predominantly clinical faculty	3	75	1	25	0	0	0	0	4
Predominantly administrators	1	50	0	0	0	0	1	50	2
Equal compositions*	1	6	2	13	4	25	9	56	16
All responses	6	13	8	17	15	33	17	37	46

*Groups were considered “equal compositions” if survey participants indicated no clear predominance of one type of professional status over the others in participating in their WAS session groups

**To estimate subscription length, participants were asked the following question: “How long has your school participated in the WAS program?”

Results

Summary of Utilization

While 17 out of 52 (33%) of survey respondents indicated that they participated in ongoing sessions as a formal faculty development program, approximately 32% indicated they had actually extended the WAS sessions by engaging in topic discussions with colleagues at their local site following each session. This was higher than expected, given that approximately 25% of survey participants reported that they were using individual subscriptions as opposed to an institutional subscription which would likely have provided more opportunity for such discussions. Surprisingly, fewer than half of the participants who indicated that they engaged in regular post-session discussions also indicated that their participation in WAS was part of a formal faculty development program. This important finding suggests that although some of the “post-session discussions” were formally implemented as part of a professional development program by institutional faculty development coordinators (34%), to a fairly great extent, engagement in WAS was intrinsically motivated in nature. Furthermore, very few participants indicated that they were receiving external recognition, such as continuing education credit (4%) or consideration for recognition in promotion and tenure (20%), for participating in WAS, which supported this finding. The qualitative data also supported the idea that those individuals choosing to participate in the WAS series likely did so for an intrinsic interest in professional development as opposed to participating to receive some extrinsic interest like external credit or some financial incentive. Almost half of respondents described exactly how their institution used the WAS program for professional development and a range of responses were provided. One theme that emerged was that institutions were using the sessions for a more general purpose to “…introduce concepts and ideas to new faculty.” Another theme included the use of WAS sessions to target individual or institutional objectives such as “…increasing educational scholarship.”

Approximately half of all survey respondents expressed interest in the proposition of IAMSE offering continuing education credit for WAS participation, and approximately a quarter indicated that they would be willing to pay for such credit. Moreover, survey participants commonly reported that they had benefited from more than one WAS session, having learned about a variety of topics. The most commonly identified beneficial topics related to educational research, evaluation systems, remediation plans, wellness, and entrustable professional activities (EPAs). In addition to the ongoing WAS sessions, approximately 75% of participants indicated that an awareness of WAS session archives and almost all of the actual WAS archive session users reported retroactively using archived sessions on more than one occasion. In total, approximately 66% of participants had viewed multiple archived sessions.

Perceived Quality, WAS Satisfaction by Component, and Associated Factors

To better understand participant perceptions of WAS quality, survey participant satisfaction with each of the major components of the program was investigated using frequencies, percentages, and a measure of central tendency and spread for summary purposes only. Further, a scaled composite quality scale was created as a measure of overall perceived quality and this construct was used to explore associations with various other factors. Participants were asked to indicate their general satisfaction with each WAS component using the following rating scale: 1 = very dissatisfied; 2 = dissatisfied; 3 = satisfied; 4 = very satisfied.

Most WAS components were very positively received with very little disparity between component ratings (Table 2). Components relating the quality and usefulness of content/topics were among the highest rated. Participants also expressed high satisfaction with the session themes. This finding was supported by the fact that over 75% of participants answered “no” when asked if they would prefer individual sessions independent of a common series theme. Those who indicated that they would prefer individual sessions were asked to explain why they would prefer individual sessions. Thematic coding of these responses indicated that participants were interested in individual sessions in addition to the themed sessions as opposed to instead of the themed sessions. One suggestion that was provided was to “reserve one of the five-session series for independent topics.” As previously noted, in 2012, “Evolution and Revolution in Medical Education” became a regular yearly series that allowed for offerings of loosely related topics.

Table 2.

Perceived quality and WAS satisfaction by component

	1 very dissatisfied (%)	2 dissatisfied (%)	3 satisfied (%)	4 very satisfied (%)	% satisfied and very satisfied
1 Visual presentation	2	12	55	31	86
2 Audio presentation	2	10	60	28	88
3 Ease of use of technology	2	14	49	35	79
4 Interactivity of the sessions	6	22	61	12	73
5 Quality of the content of the sessions	2	6	52	40	92
6 Usefulness of the content of the sessions	0	8	52	40	92
7 Topics presented	0	4	53	43	96
8 Session format	0	8	47	45	92
9 Cost for the WAS series	0	4	67	29	96
10 Day of week schedule	2	2	69	27	96
11 Time of day schedule	0	8	61	31	92
12 Session theme	0	4	45	51	96
Overall satisfaction rating					Mean (SD)
Composite quality scale rating*					3.2 (0.48)

*Composite quality scale was created by taking the overall mean. Hence, since each component had an equal number of observations, each component equally contributed (i.e., had equal weight) to the final estimate. Cronbach’s alpha was used to assess the reliability of this construct. Typically, an inter-item correlation of 0.7 is acceptable. The reliability of the composite construct was found to be excellent (α = 0.9)

**n = 52

Although participants were satisfied with the ease of use of technology and session formats, a small percentage of participants expressed interest in favor of increased interactivity of the sessions as this item was rated slightly lower than other components. Since at the end of each WAS session presenters included interactive question and answer forums, it was not immediately clear whether these participants wanted to extend the time for this already existing interactive activity or whether they had alternative ideas for how to more effectively remotely engage geographically separated participants. It was noteworthy that WAS expansion and implementation of new, additional interactivity features was not evaluated in this survey since implementation occurred at approximately the same time the survey was administered. However, to further understand this finding, a qualitative item was used to better understand participants’ past experience with web-based conferences. Participants were asked to identify the other web-based conferences in which they had participated. Several other web-based conferences were identified including the Association of American Medical Colleges, American Medical Association, Association for Medical Education in Europe, Center for Disease Control, MedEdWorld, and ExamSoft. Qualitative cross-tabs were then used to identify which web-based conferences were most commonly perceived to be of the same or higher quality, and follow-up analyses were used to investigate trends in the specific favorable characteristics associated with other quality web-conferences. Although there were no particular trends in terms of which specific web-conferences were identified to be of the same quality as WAS, there was a trend in the qualitative data in terms of the favorable characteristics that were associated with other quality web-conferences. A substantial number of participants were satisfied with the web-delivery mechanism for the WAS. However, as was consistent with the finding in Table 1 regarding the “interactivity of the sessions” for several participants who identified other conferences to be of similar or higher quality, a commonly identified favorable characteristic was web seminar “session interactivity.” Not surprisingly, the participants who identified a preference for increased interactivity happened to be the same individuals who rated WAS interactivity lower than other participants. By contrast, several participants appreciated the interactive question delivery mechanism provided by WAS and others requested adding to this an actual full discussion board to increase interactivity. One additional suggestion was to incorporate guidelines, activities, or applications for continued post-session continued discussion at the local level, and potentially adding some unspecified mechanism for sharing those experiences with the larger audience. It was worth noting that those participants who had reported engaging in local post-session discussions did not rate the WAS quality any higher than those who did not participate in such a session. In fact, the average rating for this group was exactly the same for both groups. However, it was not clear from the data whether the perceived benefit from the overall experience that included such sessions was different than the perceived benefit from the experience that did not include such sessions. Overall, however, participants were very satisfied with the interactivity of the WAS sessions as 73% of participants indicated being “satisfied” or “very satisfied.”

Additional measures were explored to triangulate these general findings regarding perceived quality. Almost all respondents indicated that they had participated in other, non-IAMSE web-based conferences, and approximately 70% had participated in web-based conferences related to medical education. As previously discussed, participants were asked to compare the quality of WAS sessions to other web-based conferences. A total of 11 participants indicated the WAS sessions were of “inferior quality,” 29 indicated the “same quality,” and nine indicated a “superior quality.” Interestingly, however, the average composite quality scale was quite high within each sub-distribution (Fig. 2). More specifically, the average composite quality scale among those participants who reported “inferior quality” was 2.7; the average rating for those participants who reported the same quality was 3.4; and lastly, the average rating for those participants who reported superior quality was 3.5. Hence, even the participants who had experience with what they perceived to be higher quality web-based conferences perceived a high WAS programming quality on average.

Fig. 2.

Composite quality scale observations by comparison

Almost all (88%) survey participants indicated that the WAS series enhanced their professional activities/teaching as a medical educator. Moreover, over 57% of these respondents indicated that WAS had “moderately” or “greatly” enhanced their professional activities or teaching. One additional important qualitative theme to emerge was that participants felt that WAS had helped them broaden their understanding of how other institutions approached similar curricular, instructional, and assessment challenges to those that their home institution faced. Comments focused on the immediate usefulness of WAS because the topics addressed common and relevant challenges in medical education. Many others commented on the impact that the WAS sessions had on professional development at their institution. For example, one theme to emerge was the value provided by WAS given the low costs associated with providing a similar level of program quality. One participant described the development of a monthly medical education journal club implemented at that institution which focused on articles relating to the most recent WAS session and the additional cost-effective faculty development benefits that provided.

Discussion

The goal of this study was to examine the utilization and perceived effectiveness of the Webcast Audio Seminar Series by participants at the individual and institutional levels. Data was collected using a 42-item survey that examined user identification, utilization, and perceived effectiveness of the IAMSE WAS as a faculty development tool. Quantitative measures were summarized using descriptive statistics, including frequencies, means, and standard deviations. The descriptive data showed that the survey sample represented a wide range of WAS participants in terms of geographic locations, subscription types and lengths, group sizes, and compositions. Moreover, WAS users included early career and experienced medical education professionals, including academic deans, faculty, administrators, and staff, and the group compositions varied tremendously. Most participants were not engaging in WAS sessions as a part of a formal faculty development program, but rather, they chose to invest their own professional development resources and efforts into the WAS series. This was especially noteworthy given that almost all survey participants indicated that they had engaged in more than one WAS series. Taken together, this spoke to the value that users placed on the WAS as a professional development tool. This was supported by the fact that very few participants received external credit for such participation. Participants had indicated that they participated in WAS sessions only to receive some type of external credit, or if the majority of participants had only participated in one series, it may have indicated that participants did not find as much value in such participation. However, this was not the case.

As WAS continues to provide high-quality and convenient content to a global audience, it may be worth considering new technologies and methods to continue to increase user interactivity. It was noteworthy that WAS expansion and implementation of new, additional interactivity features was not evaluated in this survey since implementation occurred at approximately the same time the survey was administered. If perceptions change as a result of the new interactive features, future surveys will likely capture changes in user perceptions, though study design will still preclude causal inferences about the impact. In addition, individuals and institutions have already discovered ways to enhance WAS interactivity by engaging in discussion sessions at the local level. This localized commitment was not associated with increased perceived quality of the WAS programming, but the increased engagement and interactivity of users is expected to increase the popularity of the sessions over time.

The observed overall growth and popularity of the series since its inception in 2002 could be explained by a number of factors, but certainly, the positive user perceptions of all components of the series has contributed to its overall success. The percentages of satisfied users were similarly high across all components as all WAS components were very positively received with very little disparity between component ratings. Components relating the quality and usefulness of content/topics were among the highest rated. If WAS continues to expand, it will be increasingly important to monitor the uses and perceived quality of the programming, both in terms of the immediate value and the longitudinal value perceived by users.

Participants identified a variety of uses for the WAS program and were able to describe either ideas that they had implemented at the local level or ideas that they were interested in pursuing or seeking assistance with to further enhance the impact of the sessions. For example, participants described more general uses such as introducing new faculty to important topics in medical education and using WAS as a professional development mechanism. Alternatively, they described more specific applications of WAS topics and strategies, such as learning about and implementing an institutional evaluation system. The wide scope of general and specific uses identified by participants combined with the fact that both institutional and individual users perceived a high program quality suggested that there is likely a potential to continue the expansion of the WAS program.

Conclusion

In summary, users rated the WAS program very highly on all components from 2012 to 2017. The high level of perceived quality by users is likely an important reason why WAS participation has continued to grow since its implementation. Since the quality ratings were consistently high over a 5-year period in which the number of users also grew and organizers continue to add new interactive features for users, it is expected that this growth is sustainable. However, this descriptive study is intended to be a description of how various individuals and institutions utilize WAS and how they perceive its effectiveness. Hence, this descriptive study is not intended to imply causality or infer beyond the scope that the design permits. Moreover, given the limited sample size, follow-up data collection will permit further and fuller descriptive insights into user perceptions of the interactivity of WAS sessions.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

NA

Informed Consent

NA