Quantifying Published Scholarly Works of Experiential Education Directors

Abstract

Objective. To determine the amount and potential impact of scholarly works that directors of experiential education in US colleges and schools of pharmacy have published since 2001.

Methods. A search in Web of Science was used to identify publications and citations for the years 2001-2011 by experiential education directors as identified by the American Association of Colleges of Pharmacy (AACP) Roster of Faculty and Professional Staff in 2011. Publication productivity was analyzed by position title, faculty rank, and type of institution (public vs private, research vs nonresearch-intensive). Types of published works were characterized, related citations were identified, and a reported h-index was collected for each person who published during this period.

Results. Ninety-seven of 226 (43%) experiential education directors published 344 scholarly works which had received 1841 citations, for an average of 1 publication every 3 years and an average citation rate of 5.3 per publication. Directors at publicly funded and research-intensive institutions published slightly more than did their counterparts at private and nonresearch-intensive schools. Publications were concentrated in 6 journals with a weighted mean publication impact factor of 1.5.

Conclusion. Many experiential education directors have published scholarly works even though their titles and ranks vary widely. While the quantity of such works may not be large, the impact is similar to that of other pharmacy practice faculty members. These results could be used to characterize the scholarly performance of experiential education directors in recent years as well as to establish a culture of scholarship in this emerging career track within pharmacy education.

INTRODUCTION

Since 2001, when the doctor of pharmacy (PharmD) degree was implemented, the number of pharmacy colleges and schools and the amount of experiential education required for the degree have increased. The most recent accreditation guidelines mandated a greater emphasis on experiential education in pharmacy curricula.¹ With these increases, the number of faculty members and administrators in experiential education has grown substantially in recent years, fostering interest in career development among experiential education faculty members. As the number and scope of experience of faculty members in this sector of pharmacy education develops, the desire for a culture of scholarship in experiential education is growing. Measuring the current state of scholarship among faculty members and administrators of experiential education programs would be useful in understanding the impact of their contributions and establishing expectations for scholarship in this evolving career path in pharmacy academia.

Scholarship among faculty members in colleges and schools of pharmacy as measured by publication productivity has been reported.^2-12 However, no such reports have been published for experiential education faculty members or administrators. Scholarship for research-intensive faculty members is frequently measured by amount of funding received from National Institutes of Health or other funding sources. As pharmacy academia has broadened the definition of scholarship and creativity to include integration, application, and teaching,^13-15 the scholarly contributions that experiential education faculty members disseminate are not likely to involve randomized, placebo-controlled trials or million dollar grants. Productivity in administering experiential education programs is rarely accomplished with extramural funding. Most experiential education faculty members are not in tenure-track positions^16,17 wherein such funding and publication are a consistent expectation. Many experiential education faculty members are in junior academic ranks¹⁸ and thus have had neither the time nor the opportunity to publish. Additionally, universities have varying organizational structures that put some of these individuals into faculty ranks and others into administrative roles. As such, expectations for scholarship are likely to vary based on the type of institution and job descriptions for individual experiential education positions. Consequently, the true contribution and productivity of scholarship in experiential education are still emerging. A baseline measurement of scholarly activity in experiential education could serve as a guide to deans, other administrators, and faculty members in setting appropriate performance and promotion criteria as well as providing professional development for individuals seeking successful academic careers.

Directors of experiential education have a unique role because their positions emphasize administration and service. They are unlike other practice faculty members who often split their time between practice and teaching. They also do not typically apply for research grants or operate laboratories with graduate students. The demands of program administration can limit traditional opportunities for practice, teaching, and research. While expectations for experiential education directors may seem to contradict scholarly endeavors, they are usually faculty members, rather than exclusively administrators, who aspire to advance and disseminate aspects of scholarship that focus on learning, assessment, and administration in experiential education. The purpose of this project was to describe and measure the quantity and impact of scholarly works by experiential education directors using scholarship indices and surrogate markers found in a citation and publication database.

As suggested in a review of scholarship indices,¹⁸ analysis of published scholarly works using citation counts and the h-index may help to quantify publishing productivity and the impact of scholarly works. The h-index combines the number of publications an author has produced and the corresponding citation count into a whole number to provide an assessment of both productivity and impact.¹⁹ For example, an author has an h-index of 10 when he has published 10 papers, each of which has been cited at least 10 times. Because the h-index favors senior faculty members who have published frequently over many years and experiential education faculty members tend to be in junior ranks, an m-quotient can also be used to quantify publication productivity. The m-quotient¹⁸ allows comparison of the h-index across faculty members with differing years of experience. It is calculated by dividing the h-index for an individual by the number of years since that individual first published. In effect, the m-quotient puts publishing productivity in the context of an individual’s career history. This analysis is the first published report of such indices for experiential education directors in pharmacy. The resulting measures of scholarship and indices of publication productivity could serve as a baseline for assessing scholarly productivity over time and setting realistic expectations as this sector of pharmacy education matures.

METHODS

The American Association of Colleges of Pharmacy (AACP) Roster of Faculty and Professional Staff in 2011 was used to identify experiential education directors for this analysis. Even though the experiential education Academic Section had a membership representing over 1000 faculty and staff members at the time of this search, this section of membership was found to include some staff members, practice faculty members, department chairs, and deans with only tangential involvement in experiential education programs. Therefore, the definition of an experiential education director (ie, the individual or individuals with primary responsibility for administering experiential education) was narrowed. Experiential education directors were, therefore, identified as those who had “experiential education” selected under “special area of responsibility.” Upon review, the list generated was determined to represent most individuals directly involved in administering the experiential program at their respective institutions. Individuals in the database who were from colleges or schools of pharmacy outside of the United States were excluded from the study, which narrowed the study group to 226 people.

The AACP Roster of Faculty and Staff did not provide details about the tenure-track status, years in current position, or years of experience of the individuals listed in the database, but it did provide data regarding the institution where the individuals were employed and their position titles. The Carnegie Classification of Institutions of Higher Education¹⁹ was used to characterize the types of institutions at which study individuals were employed. Institutions were classified as (1) privately or publicly funded and (2) research intensive or nonresearch intensive. Only universities classified with the Basic Classification Methodology for Doctorate-Granting Universities as DRU (doctoral/research), RU/H (research [high research activity]), and RU/VH (research [very high research activity]) were categorized as research-intensive. Study individuals were also categorized as having faculty appointments or not. Individuals with faculty appointments were further separated into the following categories: traditional appointments (professor, associate professor, assistant professor), clinical appointments (clinical professor, clinical associate professor, clinical assistant professor), and dual administration/faculty appointments (administrator/professor, administrator/associate professor, administrator/assistant professor). All administrative titles (ie, associate or assistant dean, director, and associate or assistant director) were combined into the administration/faculty appointment category.

A search in the Web of Science bibliographic database (Web of Knowledge by The Thompson Corporation, www.thomsonisi.com) was used to identify publications and citation records. A search for publications by each author, limited to the years 2001-2012, was conducted in June 2012. Faculty names were entered using the full last name, first initial*, and middle initial* (if available). Each publication found was individually evaluated to ensure the validity of the search results. When the search parameters entered identified publications for multiple individuals or did not identify publications for the correct individual, up to 4 additional letters were added to the name until the correct faculty member could be confirmed with their associated educational institution, as listed in the roster. When an individual was found to have changed employment and publications originating from multiple institutions were found, curricula vitae available through school Web sites were referenced for work and education history to confirm that all publications for the individual had been found. Because 1 of the authors has worked in experiential education for many years, personal knowledge of individuals and their work history was also used to confirm that publications for an individual were correctly identified. Only confirmed correct publications were included in the results. In an attempt to best characterize all types of scholarly work that experiential education faculty members produced, all types of publications that are typically peer reviewed (ie, articles, abstracts/proceedings, reviews, books, and book chapters) were included. Letters, news articles, editorials, and corrections were excluded, as these publications are not typically peer reviewed. Except for the inclusion of abstracts, the types of publications included in this analysis were similar to those included in other published analyses of scholarship by pharmacy faculty members.

The “Create Citation Report” feature in Web of Science was used to identify subsequent citations, years in which citations occurred, the reported h-index, and the journal titles in which works were published. The m-quotient was then calculated for each individual who had published during the study period. Because this search was conducted in June 2012, the reported h-index in Web of Science reflects 11 complete years and partial results from 2012. The clinical impact factor as reported in Journal Citation Reports within ISI Web of Knowledge by Thompson Reuters was recorded for all journals in which publications were found. All data were collected and results computed using spreadsheet and calculation functions in Microsoft Excel.

RESULTS

During the time period studied, 97 (43%) of 226 experiential education directors published scholarly works, as identified by the parameters of this search. Thirty-two directors published 1 time, 19 published 2 times, 13 published 3 times, 9 published 4 times, and 24 published 5 or more times during this time period (Table 1). Five individuals (<1% of total) published 78 (35%) of all works identified. The average publication rate for all experiential education directors whose works were identified by this search (N=226) was approximately 1 publication every 3 years. Experiential education directors at publicly funded institutions published slightly more than privately funded institutions, and experiential education directors at research-intensive institutions published slightly more than those at nonresearch-intensive institutions (Table 2).

Table 1.

Number of Publications per Experiential Education Faculty Member, January 2001-June 2012^a

Table 2.

Publications per Experiential Education Director, Based on Institution Type, January 2001-June 2012^a

Twenty different combinations of titles and ranks were identified in the roster for individuals included in this analysis. The titles included traditional professorial ranks, clinical appointments, and administrative titles alone and in various combinations. Individuals with traditional faculty ranks alone (professor, associate professor, assistant professor) produced the most publications in rank order, followed by those with clinical titles (clinical professor, clinical associate, clinical assistant professor) and those with combined administrative (associate dean, assistant dean, director of experiential education) and faculty titles. In general, those with higher academic and administrative ranks published more than did those at lower ranks.

This search identified 344 publications, which were cited by others a total of 1841 times. Experiential education directors were the primary author on 102 (30%) of these publications. In aggregate, the average citation rate per publication was 5.4, and the average number of citations per year was 175. Per person who published, the average number of citations per year was 1.8. The mean h-index for experiential education directors was 1.8 (range 0-8). The average h-index for the 5 authors who published most was 6.2. Removing the h-index for these 5 authors reduced the average h-index for all others to 1.5 (range 0-6). The average m-quotient was 0.16 (range 0-0.75).

Of the 344 publications found, 287 (83%) articles, 31 (9%) abstracts, 26 (8%) reviews, and no books or book chapters were identified. When the identified abstracts were examined more closely, no abstracts from poster presentations at the AACP Annual Meeting were found, as would have been expected. This finding was confirmed with representatives from Thompson Reuters directly to ensure the search was conducted correctly. A subsequent search of International Pharmaceutical Abstracts (IPA), now owned by Thompson Reuters as well, also yielded incomplete results. Abstracts from the Journal of Pharmaceutical Education (AJPE) were found in IPA only prior to 2008. Consequently, scholarly works in experiential education from live presentations as posters or in meetings were underrepresented in this analysis.

Over the study period, 92 journals were identified as containing published works by experiential education directors. The top 6 journals in which articles appeared contained 65% of all articles identified (Table 3). Scholarly works for experiential education faculty members seemed to be concentrated in these 6 journals, as no other journal contained more than 9 articles. The Journal contained publications for 61 authors, which accounted for 63% of all experiential education directors who published during this time. The mean publication impact factor for the top 6 journals was 1.8, and the weighted average impact factor was 1.6.

Table 3.

Top 6 Most Common Journals in Which Experiential Education-Directors Published, 2001-2011^a

DISCUSSION

While this analysis, the first of its kind in experiential education, provided much needed information, there were study limitations. The criteria used to define and identify experiential education directors in the AACP Roster of Faculty and Staff were imperfect. Also, various educational institutions are organized differently, which makes it difficult to precisely identify those responsible for administering experiential education difficult. The search parameters and resulting list of experiential education faculty members used in this analysis, therefore, may have omitted individuals in the larger AACP experiential education section who have produced important or influential scholarly works. Also, this search does not include works of other faculty members who are not members of the AACP experiential education section but are involved in experiential education through practice and precepting. Further, this analysis included works by some faculty members who were published before they became experiential education directors. Because topics of publications found in this search were not collected, results were likely to include publications unrelated to experiential education. This analysis, therefore, is not a comprehensive analysis of all experiential education-related publications. In some cases, this analysis may actually inflate contributions because articles that multiple experiential education directors authored were counted multiple times (1 for each author). However, this same duplication has occurred in other published reports of publication metrics for pharmacy faculty members as well.^3-5

Comparing these results to previously published data, publication productivity of experiential education directors was found to be lower than in other categories of pharmacy faculty members. Using previously published results to calculate an average publication rate over time, pharmacy practice faculty members published an average of 0.4-0.5 publications per person per year^5,12 compared with experiential education directors, who published an average of 0.1 publications per person per year. However, this analysis was limited to a subset of experiential education faculty members (directors only), whereas previous analysis of pharmacy practice faculty members included all members of the AACP pharmacy practice section. The larger pharmacy practice section represents a broad number of individuals, many of whom are likely to be tenure track and have different expectations for scholarship. Comparing all members of the AACP Experiential Education Section membership to the AACP Pharmacy Practice Section membership might be a more direct comparison that warrants further analysis. Some other analyses^3,12 have used publication searches in both PubMed and Web of Science, whereas this analysis only used the latter. A broader search of the literature using PubMed may have yielded a higher publication rate for these experiential education directors.

Department chairs, the only other pharmacy faculty member category for which such indices have been published, had an average h-index of 8 (0-32) and mean m-quotient of 0.36 (0-1.1).³ Given that experiential education faculty members have significantly fewer years of experience in academia than do department chairs, the difference in the h-index for experiential education directors (1.5) would be expected. The m-quotient, which removes the influence of highly cited works by senior faculty, is lower for experiential education directors than for chairs, which suggests that differences in publication productivity may be attributable to differences in expectations for publishing rather than professional experience. Even with lower publication rates, however, the average impact factor for journals in which experiential education directors have published was similar to that for practice faculty members (1.8 vs 1.9, respectively).⁵ These results indicate that while experiential education directors do not publish as often as other faculty members, their potential for producing quality, influential works is strong.

Even though experiential education directors published fewer scholarly works than did other categories of pharmacy faculty members, such comparisons may not be a legitimate comparison. The role in pharmacy education that experiential education faculty members fulfill is quite different from the respective roles of practice faculty members, researchers, deans, or department chairs. Publication counts alone may not be sufficient to characterize experiential education scholarly endeavors or expectations. As described in the Report of the AACP Professional Affairs Committee on Experiential Education Delivery in 2005, “The individual in charge of the experiential education program is called upon to be an administrator and manager of programs, resources, and personnel; to assume leadership for the academic program; and, depending on the nature of the academic appointment, to function as a faculty member with contributions and effort devoted to instruction, scholarship and the service mission of the institution.” ²⁰ This job description involves a significant and essential administrative component. Few other categories of pharmacy faculty members have such a mix of administrative duties with expectations for teaching and practice. The variety of faculty ranks and titles found for a relatively limited number of individuals in this analysis would suggest that standardized job descriptions may not exist for experiential education directors. Expectations for scholarship are, therefore, likely to vary. Without reviewing individual job descriptions, it is difficult to know which individuals in this analysis were expected to generate publications as a form of scholarship. The individuals in this analysis who had not produced any publications within the years searched (over half of those studied) may not have any expectations for publishing. Additionally, the process used to define and identify experiential education directors used in this analysis may not have included all individuals with primary responsibility for administering experiential education and may still have included some with only tangential involvement. Inconsistent expectations for scholarship and publication make it difficult to use publication counts alone to assess scholarly performance of experiential education faculty members. Further research is needed to determine common expectations for scholarship and publication productivity for faculty and staff members of experiential programs.

This analysis also did not differentiate experiential education directors who work for newer schools of pharmacy from those at established schools. At newer colleges and schools, faculty might not have produced many publications because the opportunities for scholarship may have been constrained by the demands of building an experiential education program from ground up and the limited number of students who have completed such programs. However, all full-time faculty members in colleges and schools of pharmacy have some level of obligation to participate in the generation or application of new knowledge and to disseminate it.²¹As such, experiential education faculty members — especially junior ranking faculty members — seek clarification about their expectations for scholarship as they build their careers, approach promotion opportunities, and strive for a culture of scholarship in the experiential education field.

Comparing publication productivity based on institution type, this analysis found that experiential education directors at public and research-intensive institutions publish slightly more often than do those at private or nonresearch-intensive schools. However, even a college or school with a high volume of research could have an experiential education director who has purely administrative responsibilities with little expectations for publication. The small difference in publication rates based on type of institution suggests that volume of scholarly works in experiential education is more likely to be associated with individual job descriptions than with an institution’s funding model or emphasis on research. Therefore, objective measures of currently published scholarly works in experiential education may provide a sound, evidence-based framework for establishing publication productivity expectations instead of suppositions imposed by other research faculty members. Experiential education is a unique field of scholarship that is not necessarily comparable to that of other traditional faculty roles.

At the outset of this analysis, the Web of Science database was thought to be an ideal source for identifying published works as it had been for pharmacy practice faculty in 2007.⁶ While this search identified numerous articles published by experiential education directors, it did not fully characterize the scholarly contributions of these individuals. Although a search using additional databases, such as PubMed, would likely have yielded a more comprehensive list of publications generated, the nature of experiential education scholarship is unlikely to conform to the norms set by traditional faculty expectations. As described by Kennedy and colleagues, the defining elements of scholarship are “that the work is original and creative, that it stems from the application of individual or collaborative intellect, and that the work is composed in a manner that is subject to peer review and effective communication.”¹⁶ Therefore, an article count alone may not demonstrate the full scope of scholarly contributions of experiential education faculty members. For instance, few abstracts were found and no books or book chapters were identified, even though some experiential education faculty members have authored and disseminated these types of work. Poster abstracts from AACP annual meetings would be expected to highlight scholarship in experiential education given that AACP has dedicated poster sessions to this specialty in pharmacy education, and poster abstracts are often where scholarly work of junior faculty members first appear. Considering that AJPE contained most of the publications for experiential education directors, it is unfortunate that neither Web of Science nor IPA was found to include a complete and reliable source for such abstracts. Future research should be conducted to fully elucidate the scholarly works from live presentations and meeting abstracts relating to experiential education. Only after including that body of work and considering other types of works can the state of scholarship in experiential education be accurately characterized.

Rather than arbitrarily applying performance criteria from other disciplines, perhaps objective measures of scholarly work that experiential education directors have already published should form the basis for assessing performance and setting future goals. The results in this analysis would support that a publication rate of 1 work every 3 years is an appropriate starting point for discussions about setting expectations. For colleges and schools that choose to incorporate scholarship as a part of experiential education directorships, multiple publications per year may be unrealistic, based on these findings. As the years of experience for the numerous junior faculty members in experiential education grows over time and the culture of scholarship matures, this publication rate would be expected to rise.

CONCLUSION

A significant number of experiential education directors have produced scholarly works even though the large number of ranks and titles these individuals hold suggest that standard expectations for scholarship do not exist. Still, publications by experiential education directors have a similar impact (factor) as those of pharmacy practice faculty members. Results from this analysis could be used to characterize the productivity of experiential education faculty members to set realistic expectations for future scholarship, and to guide administrators in establishing a culture of scholarship in this emerging career track within pharmacy education.