Suitability of h- and x-indices for evaluating authors’ individual research achievements in a given short period of years

Abstract

Background:

The h-index of a researcher refers to the maximum number h of his/her publications that has at least h citations via the concept of the square area. The x-index is determined by the maximum area of a rectangle under the curve to interpret authors’ individual research achievements (IRAs). However, the properties of both metrics have not been compared and discussed before. This study aimed to investigate whether both metrics of h- and x-index are suitable for evaluating IRAs in a short period of years.

Methods:

By searching the PubMed database (Pubmed.com), we used the keyword “PLoS One” (journal) and downloaded 50,000 articles published in 2015 and 2016. A total of 146,346 citations were listed in PubMed Central and 27,035 authors(with h-index ≥1) were divided into 3 parts. Correlation coefficients among metrics (ie, AIF, h, g, Ag, and x-index) were examined. The bootstrapping method used for estimating 95% confidence intervals was applied to compare differences in metrics among author groups. The most cited authors and topic burst were visualized by social network analysis. The most prominent countries/areas were highlighted by the x-index and displayed via choropleth maps.

Results:

Results demonstrated that, first, the h-index had the least relation to other metrics and failed to differentiate authors’ IRAs among groups, particularly in a short time period. Second, the top 3 highest x-index for countries were the United States, China, and the UK but with the productivity-oriented feature. Third, the most cited medical subject headings (ie, MeSH terms) were genome, metabolome, and microbiology, and the most cited author was Lori Newman (whose x-index = 13.52, and h = 2) from Switzerland with the article (PMID = 26646541) cited 291 times. The need for the x-index combined with a visual map for displaying authors’ IRAs was verified and recommended.

Conclusions:

We verified that the h-index failed to differentiate authors’ IRAs among author groups in a short time period. The x-index combined with the Kano map is recommended in research for a better understanding of the authors’ IRAs in other journals or disciplines, not just limited to the journal of PloS One as we did in this study.

Highlights

• We verified the h-index not able to differentiate author IRAs in a short time period and suggested using x-index combined with the Kano map in the future research.

• We addressed the x-index that combined with a Kano map can disclose the inner characteristics of the feature toward the productive or the influential in nature.

• The way using the citations and the degree centralities together in a social network is merit to highlight the most cited MeSH terms in the past.

1. Introduction

Bibliometric indices have become an important tool of modern assessment of authors’ individual research achievements (IRAs). These metrics are used to evaluate authors (on publication count, citation count, h-index, m-quotient, hc-index, e-index, g-index, and i-10 [i-n] index) and journals (on impact factor, Eigenfactor, article influence score, SCImago journal rank, and source-normalized impact per article).^[1] Among metrics, the best-known author-based indicator is the h-index^[2] proposed in 2005, which is simple and easy to use for describing the IRA of a researcher who has the maximum number h of publications with at least h citations in a career. However, the authors’ IRAs are often assessed in a given recent short period of years. The use of the h-index is limited to produce identical values and rankings for authors (or academic institutes) when applied.^[3] An empirical study is, thus, required to verify that the h-index is not suitable for differentiating authors’ IRAs. By contrast, the x-index^[4] illustrated in 2018 is determined by the maximum area of a rectangle under the curve to interpret authors’ IRAs. However, the properties of both metrics evaluated in a particularly short period of years have not been compared and discussed before. Whether both metrics of h- and x-index are suitable for evaluating IRAs in the short run needs further verification and study.

1.1. Requirement of author metrics based on weighted contributions

The most cited articles in a discipline have been reported in many authors.^[5–7] Few focused their studies on the most cited authors. The main reason is that it lacks an author-weighted scheme (AWS) for fairly computing the coauthor credits in an article byline. An AWS^[8,9] proposed in 2018 was applied to calculate author impact factor (AIF)^[10] and other relevant metrics for solving the problem of coauthors not necessarily sharing equal contributions in an article.^[11,12]

1.2. Requirement of classification for author groups using the Kano model

Authors’ IRAs have been classified into three groups: the influential, the one-dimensional, and the productive for use in management.^[4,13] However, no such definite criteria were defined in their studies. They merely used excess citations divided by tail publications beyond (or below) the h-core (eg, >1, ≈1, and <1).^[13] The assumption of 1-dimension based on the h-index (ie, the ratio [citations: publications = 1 based on each article]) cannot be held in a short run (eg, from 2015 to 2016 due to few articles published and cited in a giving period).

The Kano model^[14,15] is a theory for product development and customer satisfaction developed in the 1980s by Professor Noriaki Kano. This model classifies customer preferences into 3 main categories (ie, must-be quality, 1-dimensional quality, and attractive quality) displayed by a diagram. The 3 partitions can be appropriately applied to the 3 author groups (ie, the productive, the 1-dimensional, and the influential) classified by previous articles.^[4,13]

1.3. 1.3 Properties for citation indices

The Ag-index^[16] can be defined as AIFg at the g-core by the equation of g ≤ (ΣCi)/g. Similarly, the Ah-index (=[ΣCi]/h)^[17] and Ax-index (=[ΣCi]/x) result from the concept of AIFh and AIFx, respectively. As such, the properties of h≥g≤x and Ah≥Ag≥Ax were illustrated in the literature.^[4,17] The correlation between h and g has been proven to be higher than that with the x-index (=Σ√max.i × Ci, I from 1 to x).^[4] Whether the relations among the x-index, Ag, and AIF are closer than those with h and g requires further studies and examinations.

Furthermore, the x, g, and h indices are scale-invariant but not independent^[4,17] because adding a new article with the same number of citations may change their relative value and ranking. For example, the x-indices of both (2, 2) and (1, 1, 1, 1) are 2. The x-index of (2, 2, 1) is still 2, but the x-index of (1, 1, 1, 1, 1) is √5. Thus, the x-index is more sensitive toward publications than the h(or g)-index, particularly on the academic institute (or country/region) basis due to more accumulative articles in publication, even in a short period of years. The x-index is affected by publication outputs toward a higher value via the formula (=√max.i × Ci, i from 1 to x).^[4] Accordingly, whether the x-index can be combined with the Kano diagram using 2 axes of the number of citable publications(on axis x) and cited articles(on axis y) to appropriately evaluate authors’ IRAs is concerned.

1.4. Objectives

The purpose of this study was to verify whether both metrics of h- and x-index are suitable for evaluating IRAs in a short period of years. We attempted to inspect the relations among studied indices, plot the Kano diagram for classifying the types of authors’ IRAs using x-index, verify the effectiveness of h-index to differentiate authors’ IRAs, visualize the most cited authors and topic burst on a visual representation, and demonstrate the most prominent countries/areas in a discipline using the choropleth map to interpret the property of the x-index.

2. Methods

2.1. Data source

By searching the PubMed database (PubMed.org), we used the keywords “PLoS One” (Journal) on October 7, 2018, and downloaded 50,000 articles published in 2015 and 2016. It is because no any journal has a large number of articles (>21,000 per year since 2012) enough as PloS One that can be substantially reflect the characteristics of the citation analysis in a short period of years. Otherwise, the results of individual research achievements for authors might be identical or near to equivalent if h-/x-index are applied in a given short period of years.

An author-made Microsoft Excel visual basic for application (VBA) module was used to analyze the data. All the downloaded abstracts were based on the type of journal article. A total of 146,346 citations were matched to the artiles in PubMed Central. A total of 27,035 authors with h-index ≥1 were included in this study and then equally divided into 3 groups by AIF,^[10] see Supplemental Digital Content 1 and 2.

All the data used in this study were downloaded from PubMed Central (PMC), which means that the study required no ethical approval according to the regulation promulgated by the Taiwan Ministry of Health and Welfare.

2.2. AWS for quantifying coauthor contributions

We applied the AWS^[8,9] based on the Rasch rating scale model^[18] as Eq. (1). The sum of authorships equals 1 for each article.

$$W_{ij}=\frac{\exp ({\gamma }_{ij})}{\sum _{j=0}^{m-1}\exp ({\gamma }_{ij})}=\frac{2.72^{\gamma }_{ij}}{\sum _{j=0}^{m-1}2.72^{\gamma }_{ij}},$$ (1)

where the power (${\gamma }_{ij}$) is the ordered author name (i) and the article (j) from m-1 to 0, and the author number is m. More importance is given to the first (=exp [m-1], primary) and last (=exp [m-2]) corresponding or supervisory authors. We assume that the others (the middle authors) have made smaller contributions to articles.

2.3. AIF and other indices used to evaluate IRA

The AIF used to evaluate IRA is expressed as Eq. 2:^[10]

$$AIF=\frac{\sum Cited.papers.based.on.W_j.}{\sum Citable.papers.\times W_j.in.the.given.yrs},$$ (2)

Other author-level bibliometric indices, such as g, Ag, h, and x, were calculated in this study.

2.4. Comparing differences in metrics among author groups

All authors’ AIFs were present using the three parts of authors (ie, influential, 1-dimensional, and productive) as Kano model's classification on a diagram. Authors were grouped based on the cutting points to obtain almost equal observations at AIF>14 (n = 10232), >8 (n = 8690), and ≤8 (n = 8113) (Fig. 1).

Figure 1.

Three types of authors dispersed on a scatter plot (n = 27,035).

The bootstrapping method^[19,20] was used to examine differences in metrics among author clusters. A total of 1000 median metrics were retrieved from the random samples of 100 repetitions on mean values for each metric and cluster. As such, the median and 95% confidence interval (CI) were obtained to compare differences in metrics among author clusters by inspecting whether 2 95% CI bands were overlaid.

2.5. Social network analysis using Pajek software

In keeping with the Pajek guidelines^[21] used for social network analysis (SNA), we applied SNA to generate the control file and defined an entity (or medical subject headings [MeSH] terms defined in PMC) as a node that is connected to another one through the edge of a line. In general, the relation valued by the weight is defined by the number of connections between 2 nodes.^[22,23] The clusters can be determined by a specific algorithm referred to as centrality in SNA.

Three main centrality measures (ie, degree, closeness, and betweenness) are frequently used to evaluate the influence (or power) for an entity (eg, the author).^[22,23] Centrality is an important index to analyze the network. Any individual authors in the center of the social network will be considered to have the most influential role on the network and own the speed to gain information.^[24] In this study, the top 500 articles with the most citations ranging from 7 to 291 were included to explore the burst topics with equal weights via clustering analysis in SNA. The size of the bubble indicates its influence; a large bubble denotes a major influence as mostly cited topics in recent years. The top 500 authors in the influential group were plotted against the 2 axes of citable and cited number on a dashboard to highlight the eminent authors with a high x-index.

2.6. Creating dashboards on Google Maps

The metrics and partitioned clusters were yielded by author-made modules in MS Excel and the SNA algorithms in Pajek. We created pages of Hyper Text Mark-up Language used for Google Maps. All relevant bibliometric indices were linked to dashboards on Google Maps. Choropleth maps^[25,26] were provided to readers to interpret the most prominent countries/areas with high x-index.

2.7. Statistical Tools and Data Analyses

SPSS 15.0 for Windows (SPSS Inc, Chicago, IL), MedCalc 9.5.0.0 for Windows (MedCalc Software) and author-made Excel VBA were used to draw the scatter plot, the box plot, and the choropleth map.

3. Results

3.1. TASK 1: the relations among studied indices

Correlation coefficients among metrics are shown in Table 1. The lowest was the h-index, whereas the highest was the x-index (see the bottom in Table 1). The h and g indices showed a closer relation than the other 3, indicating that both h and g indices might be harder to differentiate authors’ IRAs under a short period of years.

Table 1.

Correlation coefficients among metrics.

	h	g	x	AIF	Ag
H		0.50	0.23	0.11	0.10
G	0.50		0.36	0.24	0.07
X	0.23	0.36		0.86	0.91
AIF	0.11	0.24	0.86		0.85
Ag	0.10	0.07	0.91	0.85
Log odds	−1.12	−0.87	0.46	0.05	−0.18

Log odds = average (log[corr/{1-corr}]). AIF = author impact factor.

3.2. TASK 2: Whether the h-index can differentiate authors’ IRAs

Regarding the 3 types of authors separated by colors in Figure 1, we examined whether the h-index can differentiate authors’ IRAs. All metrics, but the h-index with the lowest coefficient of variance = 0.15 in Figure 2, could discriminate author groups. The influential authors exhibited higher metrics (ie, x, Ag, and AIF) than their counterparts (Fig. 2) because all types of 95% confidences were separated.

Figure 2.

Comparisons of metrics among groups.

3.3. TASK 3: visualizing the most cited authors and topic burst on a map

The author Lori Newman from Switzerland earned the highest x-index (=13.52) at the left top corner in Figure 3, and she has published 4 articles in PLoS One. One article (PMID: 2664654)^[27] has been cited 291 times on June 9, 2019. Readers are invited to scan the QR-code in Figure 3 to examine any of the detailed metrics while clicking the author of interest on the specific bubble.

Figure 3.

Most cited authors in PLoS One.

The top 3 MeSH terms with the most frequent occurrences and citations in PloS One were genome, metabolome, and microbiology (Fig. 4). These terms exhibited the highest interest and had more citations in recent years.

Figure 4.

Highly cited MeSH terms and the articles in PLoS One.

The top 3 cited articles were authored by Lori Newman (PMID: 2664654, 291 times),^[27] C Ritz from Denmark (PMID: 26717316,123 times),^[28] and KA Jablonski from the United States (PMID: 26699615,122 times).^[29] Their MeSH terms were global health and epidemiology; data interpretation, statistical dose–response relationship, and drug; and metabolism, methods, metabolism, physiology, and transcriptome, respectively.

3.4. TASK 4: the most prominent countries/areas displayed on a choropleth map

The 3 countries (ie, the US, China, and the UK) presented the highest x-index among author-affiliated nations around the world. Readers are invited to scan the QR-code on Figure 5 for detailed information on the choropleth map.^[25,26]

Figure 5.

Author-based x-index dispersed with high author impact factor level in countries/areas.

We plotted all those countries/areas using the x-index to display on a 2-axes map (Fig. 6). The biggest bubbles were dispersed at the bottom (ie, in the productive zone), indicating that this zone (at the right side) was not influential with a high x-index (Fig. 3). Many countries with a high x-index are productivity-oriented rather than influential-oriented.

Figure 6.

Using the x-index on a diagram to show the feature of entities.

4. Discussion

4.1. Principal findings

We found that the h-index failed to differentiate authors’ IRAs among author groups in a short time period; the top 3 countries with the highest x-index were the United States, China, and the UK affected by a large number of publications (Fig. 6); the most cited MeSH terms were genome, metabolome, and microbiology; and the most cited author was Lori Newman (whose x-index = 13.52, AIF = 160.71, h = 2, g = 2) from Switzerland with one article (PMID = 26646541) cited 291 times.^[27]

4.2. What we have known from this study

The correlation between h and g in Table 1 was proven to be higher than that with the x-index(=√max.i × Ci, I from 1 to x), similar to a previous study.^[4] The diagram using the Kano model^[14,15] could be applied to classify authors’ IRAs on a diagram (Figs. 1 and 3, and 6), which was rarely applied to classify author features in the literature.

The most frequently cited articles have been studied by many authors.^[5–7] Few focused their studies on the most cited authors in the past, and few combined articles with their citations and MeSH terms using SNA to highlight the results. The AWS is the preliminary condition for quantifying coauthor contributions in an article byline.^[11,12] The assumption of coauthors earning equal sizes of credits in an article for calculating metrics, such as the h- and x-index,^[2,4] is problematic. We proposed the use of the AWS to fairly quantify coauthor contributions in this study.

Authors’ IRAs can be classified into 3 groups in features of the influential, the one-dimensional, and the productive for use in management.^[4,13,30] Regrettably, no such definite criteria were evident in their studies, but only a concept using the excess citations divided by tail publications beyond (or below) the h-core (eg, >1,≈1, and <1) was reported.^[13,30] We proposed fitting to the Kano model to classify authors into 3 types: high efficiency(or citation-oriented), intermediate efficiency(or the neutral), and low efficiency(or productivity-oriented) (Fig. 3). Thus, the authors’ characteristics could be easily understood via the Kano diagram and the x-index.

4.3. What implications provided in this study

We confirmed the following findings: the h-index was suitable for authors on a career basis instead of in a short period of years; the x-index should combine the Kano diagram to partition authors’ IRA features (ie, high efficiency, intermediate efficiency, and low efficiency); and the dominant countries/regions with high x-index are oriented toward productivity because more publications led to higher x-index.

In particular, we suggested that the x-index should be combined with the Kano diagram to ensure author classification in efficiency, which will move their tendencies toward the influential or the productive. In general, the x-index shows difficulty in displaying the inner characteristics (ie, the tendency toward the influential or the productive) of the feature to readers.

Identifying author types in IRAs is essential to manage scholar authors in universities or research institutes. No study has provided an effective and scientific way to classify authors’ IRA types. In the present work, we proposed a way to show authors in three types of efficiencies (ie, high, intermediate, and low) on a Kano diagram, which is unique and innovative to classify authors’ IRAs.

We combined article citations and article PMID with MeSH terms using SNA to display results on a dashboard. Doing so can easily highlight the most outstanding entities in a network, such as the functionality of a word cloud to visualize uncoded text responses and questions with numerous frequency conveniently shown on a diagram. With the SNA technique ( eg, in Fig. 4), the most cited terms and articles with the highest number of centrality degree were presented on a diagram to facilitate further investigate their features on a dashboard through Google Maps. Readers are invited to click on the bubble and refer to details on the dashboard.

4.4. Study limitations and suggestions

Although our findings were based on the above analysis, several limitations are worth mentioning and necessitate further research. First, all data were downloaded from PMC. There might be some biases resulting from matched authors given that different authors with the same name or abbreviation are affiliated with different institutions. Therefore, the results of the author's IRAs were influenced by the accuracy of the indexing author.

Second, 161,451 authors were extracted from the downloaded data. Only 27,035 authors with h-index ≥1 were included in this study, and the top 500 with high AIFs are shown in Figure 3. Similarly, the top 500 articles with more citations ranging from 7 to 291 are illustrated in Figure 4. The article space limitations did not allow us to demonstrate/display many details in our figures. Readers can scan the QR-code on the Figures to view more information after clicking on the bubble of interest. One MP4 video for introducing how to use it was provided at the reference^[31] and Supplemental Digital Content 3.

Third, the preliminary condition used to evaluate authors’ IRAs was fairly quantifying coauthor contributions in an article. We did not introduce details about the AWS applied in this study. Readers are encouraged to read our previous articles on the issue.^[8,9]

Fourth, the data extracted from PMC cannot be generalized to other major citation databases, such as the Scientific Citation Index (Thomson Reuters, New York, NY) and Scopus (Elsevier, Amsterdam, The Netherlands). As such, the most cited authors or MeSH terms are barely determined by the publications indexed in PMC.

5. Conclusions

We verified that the h-index failed to differentiate authors’ IRAs among author groups in a short time period. The x-index combined with the Kano map is recommended in research for a better understanding of the authors’ IRAs in other journals or disciplines, not just limited to the journal of PLoS One as we did in this study.

Acknowledgments

The authors thank Enago (www.enago.tw) for the English language review of this manuscript.

Author contributions

KY developed the study concept and design. CH, YT, and JCJ analyzed and interpreted the data. FJ monitored the process of this study and helped in responding to the reviewers’ advice and comments. TWC drafted the manuscript, and all authors provided critical revisions for important intellectual content. The study was supervised by FJ. All authors read and approved the final manuscript.

Conceptualization: Tsair-Wei Chien.

Data curation: Tsair-Wei Chien, Chieh-Hsun Chen, Yu-Tsen Yeh, Jui-Chung John.

Formal analysis: Kyent-Yon Yie, Jui-Chung John.

Investigation: Feng-Jie Lai.

Methodology: Kyent-Yon Yie, Chieh-Hsun Chen.

Resources: Yu-Tsen Yeh.

Supervision: Feng-Jie Lai.

Validation: Kyent-Yon Yie.

Writing – original draft: Tsair-Wei Chien.

Writing – review & editing: Tsair-Wei Chien.