New Indices in Scholarship Assessment

Abstract

Faculty members' contributions to research and scholarship are measured by a variety of indices. Assessment also has become an integral part of the Accreditation Council for Pharmacy Education's accreditation process for professional programs. This review describes some of the newer indices available for faculty scholarship assessment. Recently described metrics include the h-index, m-quotient, g-index, h(2) index, a-index, m-index, r-index, ar index, and the creativity index. Of the newer scholarship metrics available, the h-index and m-quotient will likely have the most widespread application in the near future. However, there is no substitute for thoughtful peer review by experienced academicians as the primary method of research and scholarship assessment.

INTRODUCTION

Strategic planning and assessment are essential elements of academic pharmacy. Measurement of outcome parameters in a strategic plan can help determine how well goals are being met. The Accreditation Council for Pharmacy Education (ACPE) has expanded the nature of assessment in academic pharmacy with its 2007 guidelines.¹ The ACPE changes have recently prompted the American Association of Colleges of Pharmacy to devote their 11^th annual Institute to evaluation, assessment, and outcomes.² However, it is often difficult to know how to measure a particular area or goal and which parameter more accurately measures change than others.^3,4 For example, scholarship can be measured by such parameters as the amount and type of grant support for research, number of books, book chapters, or abstracts published, and most classically, the number of journal articles published.⁵ It is this last parameter of measuring scholarship that has undergone a recent renaissance. Interesting new measures of the depth, breadth, and creativity in journal article publishing have been developed in recent years. The purpose of this paper is to describe these new scholarship metrics.

Journal article publishing, as a measure of faculty scholarship, has historically been tracked as simply the number of papers published by a faculty member. Further refinement of this evaluation process has involved differentiating peer-reviewed and non-peer-reviewed publications, discriminating between research publications and other types of scholarship (eg, review articles, case reports, letters to the editor), and counting the number of papers on which the faculty member was the first or senior author. Citation analysis expanded the evaluation of journal article publishing to include the impact or usefulness of a faculty member's work by measuring the surrogate marker of how many times a paper has been cited by other authors.⁶ The Institute for Scientific Information (ISI) began compiling and publishing citation data and, as a result, the number of times a faculty member's articles had been cited in the literature became part of the metrics of scholarship assessment. In the last 5 years, a host of new parameters have been introduced to analyze and quantify a faculty member's impact and standing in a particular discipline. Table 1 lists these various parameters and describes their calculation, advantages, disadvantages, and their normative values in available disciplines.^7-19 While there are too many of these parameters to fully describe all of them here, we highlight several of the newer indices that hold promise as useful scholarship metrics.

Table 1.

NEW INDICES

h-index and m quotient

Created by Hirsch in 2005, the h-index combines the quantitative aspects of the number of published papers with the impact features of citation counts.¹⁴ The definition of the h-index is listed in Table 1. If a faculty member has an h-index of 20, the faculty member has published 20 papers, each of which has ≥20 citations. These “h” papers are considered a group of high performance publications and have been given the name “Hirsch core.”²⁰ The h-index is easily calculated and is now available as a regular feature of the ISI Web of Science. The h-index is a “balanced” metric that is insensitive to an extensive body of work that is largely uncited or a small number of publications that have abnormally high citation counts. The h-index correlates well with peer assessment and can be predictive of future academic success.^21,22 One drawback of the h-index is that it favors scholars who have consistently published papers over many years.^23,24 Therefore, it is difficult to compare the h-index of a junior faculty member to that of a senior faculty member. The m quotient was introduced to normalize the h-index by taking into account the number of publishing years.¹⁴ The m quotient allows comparisons of faculty members' h-indices across a wide range of tenure situations. The h-indices and m quotients for pharmacy practice chairs have recently been published.⁸ Data also are available in the field of physics, where these metrics were originally developed.¹⁶ These parameters are usually discipline specific and it is not generally useful to compare them among faculty members from different disciplines.^23-28

a-index

A variation of the h-index, the a-index is calculated as the average number of citations received by the Hirsch core publications. The a-index provides an assessment of the impact of the most productive core publications of a faculty member. The a, m, r, and ar indices (see Table 1 for definitions) all similarly measure the impact of the Hirsch core publications in various ways. Bornmann and colleagues¹⁶ found that peer assessments correlated better with indices that measure the quality of the productive core rather than indices that measure the quantity of the productive core (h, g, h(2) indices and m-quotient, Table 1).

Creativity Index

The creativity index (C_a) was developed recently by Soler and is the only metric that claims to measure a faculty member's creativity.¹⁹ This metric involves counting the number of citations a paper receives and the number of references the paper cited, normalized by the number of authors of the paper. Soler has described it as follows:

…imagine that two scientists, Alice and Bob, address independently an important and difficult problem in their field. Bob takes an interdisciplinary approach and discovers that a method developed in a different field just fits their need. Simultaneously, Alice faces the problem directly and reinvents the same method by herself (thus making less references in her publication). All other factors being equal, both papers will receive roughly the same number of citations, since they transmit the same knowledge to their field. But it may be argued that Alice's work was more creative in some sense, and that her skills might possibly (but not necessarily) be more valuable in a given selection process. Eventually, the usefulness of different merit indicators will depend on how well they correlate with real human made selections.¹⁹

The calculation of the creativity index is somewhat complicated. The everyday use of this calculation for assessing departmental faculty members is not very practical. However, Soler has a Web page with a downloadable program that will calculate the index from an ISI Web of Science file. The instructions for this are available in a paper.¹⁹

DISCUSSION

The ISI has historically been the exclusive provider of citation analysis data. However, other providers of scholarly analysis of citations have emerged. Scopus (Elsevier B.V., New York, NY) and Google Scholar (Google, Inc, Mountain View, CA) have challenged ISI as the only source of citation data. Unfortunately, the same search done on all 3 databases would probably generate 3 different results from the available citation networks.²⁹ This must be kept in mind when evaluating faculty citation counts using different sources.

Citation analysis, as a technique to measure impact, has limitations.^30-34 Methodological issues, such as misspelled author names, homographs (ie, scientists with the same names in different disciplines), inconsistent use of author initials, or author name changes are potential problems.^30,34 Databases, such as ISI, Scopus, and Google Scholar, have limitations and differences in their journal coverage.³³ Additionally, publication errors in source journals and publication errors in ISI, Scopus, or Google Scholar databases can confuse results. Finally, self-citations, negative citations, and lack of citations from emerging new sources (ie, Web metrics) can further complicate results. However, taken in context, citation analysis is a useful addition to the overall assessment of a faculty member's scholarly work.^33,35 Many of the metrics outlined in this paper utilize citation analysis as part of the calculation and the drawbacks of this method must be understood.

The new indices for measuring scholarship can provide unique insights into evaluating journal scholarship. However, no parameter, on its own, fully represents all aspects of a particular scholar's work. Each metric has strengths and weaknesses as described in Table 1. Further, nothing replaces the thoughtful review of unbiased senior peers and colleagues in assessing scholarship.^36,37 Peer review, although imperfect, is the gatekeeper of scholarly publishing and remains the foundation of the academic tenure and promotion system. Of the available new indices, the h-index and its derivative, the m quotient appear to be finding the most acceptance among academic and research establishments. The h-index is a simple, whole number that is easily calculated, correlates well with peer assessments, and has some predictive value for future success.^21,38-40 In addition, the h-index takes into account both quantitative (number of papers) and qualitative (citations received) aspects of scholarship. The original Hirsch paper describing the h-index was cited 385 times as of January 2009 (Google Scholar, January 23, 2009). The majority of the newer indices do not have a sufficient track record to warrant routine calculation. An intriguing aspect of the new indices is the possibility of calculating departmental or college-wide parameters.⁴¹ For example, h-index data from a sufficient number of specific departments may be a marker to compare departments across a wide range of colleges of pharmacy. Of course, these data would need to be normalized by the number of departmental faculty members and type of college (eg, public, private, health-sciences based, non-health-sciences based). Caution should be exercised when comparing metrics across colleges with varying missions and goals. The skewed nature of publication data should also be taken into account. As a rough guide, approximately 20% of faculty members from a given college or department will normally account for 80% of the publications.⁸ Therefore, measures of central tendency should be evaluated with caution. Appropriately normalized data may eventually be helpful in assessing the amount and quality of scholarship being conducted at various colleges of pharmacy and establishing institutional goals.

CONCLUSION

While there are many new indices with interesting, unique attributes, none is a perfect metric for measuring scholarship. Nothing replaces thoughtful peer assessment of the works in making judgments concerning the quality of faculty scholarship. However, the new metrics, combined with discipline-specific normative values, may aid administrators and department chairs in evaluating individual faculty members in the larger realm of their scientific discipline. These measures can also be applied to individual faculty members, departments, research groups, or an entire college of pharmacy. To be more useful for these purposes, normative values for academic pharmacy would need to be generated.