An Evaluation of h-Index as a Measure of Research Productivity Among Canadian Academic Plastic Surgeons

Abstract

Background:

Evaluation of research productivity among plastic surgeons can be complex. The Hirsch index (h-index) was recently introduced to evaluate both the quality and quantity of one’s research activity. It has been proposed to be valuable in assessing promotions and grant funding within academic medicine, including plastic surgery. Our objective is to evaluate research productivity among Canadian academic plastic surgeons using the h-index.

Methods:

A list of Canadian academic plastic surgeons was obtained from websites of academic training programs. The h-index was retrieved using the Scopus database. Relevant demographic and academic factors were collected and their effects on the h-index were analyzed using the t test and Wilcoxon Mann-Whitney U test. Nominal and categorical variables were analyzed using χ² test and 1-way analysis of variance. Univariate and multivariate models were built a priori. All P values were 2 sided, and P < .05 was considered to be significant.

Results:

Our study on Canadian plastic surgeons involved 175 surgeons with an average h-index of 7.6. Over 80% of the surgeons were male. Both univariable and multivariable analysis showed that graduate degree (P < .0001), academic rank (P = .03), and years in practice (P < .0001) were positively correlated with h-index. Limitations of the study include that the Scopus database and the websites of training programs were not always up-to-date.

Conclusion:

The h-index is a novel tool for evaluating research productivity in academic medicine, and this study shows that the h-index can also serve as a useful metric for measuring research productivity in the Canadian plastic surgery community. Plastic surgeons would be wise to familiarize themselves with the h-index concept and should consider using it as an adjunct to existing metrics such as total publication number.

Introduction

Determining the academic success of plastic surgeons can be a complex and convoluted subject. Several factors can be taken into account to measure one’s academic success, such as research productivity, contribution to medical education, clinical service, and grant funding, with research productivity being the most emphasized.^1,2 However, the traditional approach of using the total number of publications and citations does not adequately evaluate research output regarding its overall impact.² A novel bibliometric measure, the Hirsch index (h-index), has become popular in the field of academic plastic surgery in recent years.

The h-index was first proposed by physicist Jorge Hirsch to measure research output, which is assigned a value “h” if “h” number of articles were published and each of these articles were cited “h” times.³ In his subsequent study, Hirsch demonstrated that the h-index is a more accurate predictor of future research productivity than any of the traditional bibliometric measures.⁴

The h-index can also address inherent problems with simply using total number of publications as an indicator of research productivity. Presently, there is an increasing publication access to numerous online, open access, and nontraditional journals, and using solely the number of publications provides little to no insight on the quality of the articles published. Arguably, higher quality articles are more likely to be cited and this approach itself is the basis for determining the impact factor of scientific journals.^5,6

Since its conception, the h-index has been validated in several fields of academic medicine, such as urology, anesthesiology, neurosurgery, radiology, emergency medicine, and plastic surgery.^7–12 Within the field of academic plastic surgery, the h-index is shown to predict academic rank and research grant funding.^13–15 To date, several studies have identified the academic utility of the h-index among academic plastic surgeons in the United States.^13–20 However, an equivalent study has not been conducted in Canada, and such a study may shine light upon the Canadian research practices in comparison to international colleagues. The objective of this study was to examine the association of the h-index and various academic attributes among Canadian academic plastic surgeons.

Methods

Compilation of List of Academic Plastic Surgeons

A list of 11 Canadian plastic surgery training programs was obtained from the Canadian Society of Plastic Surgery website in June 2016. Faculty lists provided by each program’s website were used to identity our study population. If the list was not available online, we contacted the program director or the program’s administrative assistant by e-mail on 2 attempts. Programs that did not respond to our e-mails to provide faculty lists were excluded from the study.

Data Collection

Program websites were used to collect various demographic characteristics and academic attributes of the plastic surgeons included in our study. Information collected included: gender, fellowship training, graduate degree, subspecialty, training institution, academic rank (instructor/lecturer, assistant professor, associate professor, and professor), and year of certification as a Fellow of the Royal College of Surgeons of Canada. In addition, missing information from faculty websites were supplemented by Google search (Google, Mountain View, California) using surgeons’ names and keywords “plastic surgery.” Results in the first 20 pages were viewed and appropriate data were extracted.

Collection of the h-Index as a Measure of Academic Productivity

Similar to previous studies in the United States, the h-index for each surgeon was collected using the Scopus database (www.scopus.com, Elsevier, Philadelphia, Pennsylvania), data acquired from 1996 to 2016. Each surgeon’s first and last names were used as search terms, and middle name initials were occasionally entered to improve our search strategy. Bibliometric measures collected included h-index, h-index after removing self-citations, total number of publications, total number of citations, and total number of citations after removing self-citations. Data were collected by 2 independent evaluators (A.G. and N.S.), and any discrepancy was resolved by consulting the primary author (J.H.).

Statistical Analysis

The association between nominal and categorical variables and the h-index was calculated using χ² test and 1-way analysis of variance (2-tailed) where appropriate. The effect of self-citation on the h-index and total number of citations was assessed using student t test and Wilcoxon Mann-Whitney U test (2-tailed) where appropriate. In addition, linear regression models were built to examine the predictors of h-index. An Omnibus F test determined if there was a significant association between the outcome variables and at least 1 predictor.²¹ Univariate analysis was first performed and a final multivariable model was built incorporating relevant confounders identified from the literature. Finally, The Spearman correlation was calculated between the h-index and the total publication number, as well as the total citation number, given the nonnormal nature of the data. All analyses were performed using SAS Version 9.4 (SAS Institute Inc, Carey, NC, USA). An α value of 0.05 was set as the threshold for statistical significance.

Results

We obtained a list of faculty members from all 10 English speaking training programs. One French speaking program (University of Montreal) did not respond to our request and was excluded from the study. After excluding 11 surgeons whose names were not found in the Scopus database, our study sample included 175 faculty plastic surgeons from the 10 programs. Study demographics are shown in Table 1. Most surgeons were male (81%) and received fellowship training (66%). The average h-index is 7.6 and the bibliometric values are shown in Table 2. Self-citation did not significantly affect the h-index (inflation = 2.7%, P = .87) or mean total number of citations (inflation = 4.9%, P = .87).

Table 1.

Study Demographics of 175 Surgeons From 10 Training Programs.

Variables	Frequency (%)
Academic rank
Instructor/lecturer	31 (18)
Assistant professor	58 (33)
Associate professor	40 (23)
Professor	34 (19)
Not specified	11 (6)
Fellowship training
Yes	115 (66)
No	60 (34)
Graduate degree
Yes	69 (39)
No	106 (61)
Gender
Male	142 (81)
Female	33 (19)
Mean years in practice	18.9 (SD = 11.7)

Abbreviation: SD, standard deviation.

Table 2.

Bibliometric Values of Canadian Academic Plastic Surgeons.

Bibliometric Measure	Value (SD)
h-index	7.6 (7.5)
h-index (self-citation removed)	7.4 (7.3)
Mean total publication number	22.2 (32.0)
Mean total citation number	450.6 (889.2)
Mean total citation number (self-citation removed)	429.3 (841.7)

Abbreviations: h-index, Hirsch index; SD, standard deviation.

Higher academic rank and possession of a graduate degree were associated with higher h-index (P = .0001 and P = .003, respectively), whereas fellowship training and gender showed no association (Table 3). There was no gender-based difference with respect to subspecialty of training, institution-based work, total number of publications, or the h-index (Table 4). However, significantly more female surgeons have a graduate degree in comparison to males (61% vs 35%, P = .006). Males were shown to be in practice for a longer period of time (mean = 20.2 years vs 13.2, P = .0019).

Table 3.

Association of Academic Variables and the h-Index.

Variables	h-index (SD)	P Value
Academic rank		.00001
Instructor/lecturer	2.1 (1.4)
Assistant professor	4.2 (2.8)
Associate professor	10.1 (7.5)
Professor	12.2 (9.9)
Fellowship training		.42
Yes	7.7 (6.9)
No	6.7 (8.5)
Graduate degree		.0003
Yes	10.0 (8.6)
No	5.7 (6.1)
Gender		.08
Male	7.7 (7.9)
Female	5.8 (5.0)

Abbreviations: h-index, Hirsch index; SD, standard deviation.

Table 4.

Academic Variables Between Genders.

Academic Variables	Male	Female	P Value
Subspecialties			.06
Institutions			.09
Graduate degree (%)	35	61	.006
Years in practice (95% CI)	20.2 (18.2-22.1)	13.2 (9.5-16.9)	.0019
Mean total publication, n	23.4	17.1	.37
Mean total citation, n	457.7	205.3	.24
h-index	7.7	5.8	.08

Abbreviations: CI, confidence interval; h-index, Hirsch index.

Univariate and multivariate analyses, which took into account potential confounders, demonstrated that graduate degree, higher academic rank, and more years in practice were predictors of the h-index (Tables 5 and 6). The Omnibus F test showed the overall multiple linear regression model was significant (P < .0001). The Spearman correlation coefficient between h-index and total publication number is 0.90 (P < .0001), and for total citation number it is 0.91 (P < .0001; Figures 1 and 2).

Table 5.

Univariate (Unadjusted) Analysis of h-Index and Predictors.

Variables/Predictors	Unadjusted β Coefficient (95% CI)	Test Statistic	P Value
Gender (reference: male)	−1.94 (−4.79 to 0.91)	−1.34	.18
Graduate degree	4.38 (2.18 to 6.58)	3.93	.0001
Academic rank	0.88 (0.37 to 1.38)	3.41	.0008
Years in practice	0.22 (0.13 to 0.31)	4.85	<.0001

Abbreviations: CI, confidence interval; h-index, Hirsch index.

Table 6.

Multivariate (Adjusted) Analysis of h-Index and Predictors.^a

Variables/Predictors	Unadjusted β Coefficient (95% CI)	Test Statistic	P Value
Omnibus F test (df)		18.59 (4)	<.0001
Gender (reference: male)	−1.61 (−4.12 to 0.90)	−1.26	.21
Graduate degree	6.34 (4.30 to 8.38)	6.13	<.0001
Academic rank	0.51 (0.06 to 0.97)	2.22	.03
Years in practice	0.26 (0.17 to 0.34)	5.82	<.0001

Abbreviations: CI, confidence interval; h-index, Hirsch index.

^a R ² = 0.29.

Figure 1.

Correlation between h-index (horizontal axis) and total publication number (vertical axis). h-index, Hirsch index.

Figure 2.

Correlation between h-index (horizontal axis) and total citation number (vertical axis). h-index, Hirsch index.

Discussion

Since the introduction of the h-index to measure both quality and quantity of research output, this landmark metric has been tested in a variety of disciplines.^7–11 It is a practical and relatively novel concept that has been well studied in the field of academic plastic surgery in the United States. Higher h-index is associated with academic rank, NIH funding, and research experience.^{1,12,13,15,18,20} This study offers insights on the applicability of using the h-index as a predictor of research productivity metrics in the Canadian plastic surgery academic community.

Similar to the US studies, this study also showed that graduate degree, academic rank, and years in practice were predictors of higher h-index. The average h-index is 7.6 and lower than the US counterpart, which has been reported to range from 8.5 to 11.7.^2,12,18 Interestingly, Gast et al reported that Canadian trained plastic surgeons working at US academic centers have a mean h-index of 9.1.¹⁷ This could be explained by a much more affluent research community in the United States. In addition, the US training programs are longer in duration and potentially offered more time and opportunities for research. Another factor is that the volume of clinical work that Canadian plastic surgeons are expected to carry out in contrast to their US counterparts. In their study, Cheung et al referenced 2 self-reported surveys that suggest Canadian plastic surgeons believe there is a shortage of plastic surgeons in Canada, while US plastic surgeons believed there to be an excess of plastic surgeons in the United States.²² Thus, a shortage of surgeons may translate into more clinical volume and less protected research time for Canadian plastic surgeons.

In our study, male surgeons had a higher h-index than females (mean = 7.7 vs 5.8, P = .08), although this was not statistically significant. Curiously, the association between gender and the h-index has not been clearly elucidated in previous studies.^1,2,12,17,19 Male surgeons often had a higher h-index than females, although statistical significance was not uniformly present in each of the studies. This phenomenon may be attributed to the fact that male surgeons had been in practice significantly longer than females, which has been observed in a previous study.¹⁷ Recent studies have looked at the gender gap in research productivity of female academic clinicians. Studies evaluated female publications among medical faculty, academic surgeons, and academic plastic surgeons.^23–25 A study by Raj et al reported that despite having equal grant funding compared to their male counterparts, female staff had fewer referenced publication and a lower h-index.²⁵ These studies attributed this disparity to the disproportionate burden of family responsibility, as well as a lack of research mentorship for the female staff.^23–25 Although a greater proportion of females had graduate degrees, we did not distinguish between the different types of degrees in our analysis. Certain graduate degrees, such as Master of Business Administration and Master of Education, do not yield the same level of research productivity as Master of Science (MSc).

Historically, there has been a gender disparity in Canadian medical school enrolment in which women are under-represented. Having said that, female representation in medical schools has been steadily increasing since 1970s to 2011 (from 15.7% to 57.7%).²⁶ Similar to the United States, there has also been an increase in the proportion of female plastic surgeons in Canada from the year 2000 (10.1%) to 2016 (22.4%).²⁷ In fact, a trend of increasing female authorship has been observed in the Journal of Plastic and Reconstructive Surgery, and so the h-index gender disparity may be expected to continue diminishing in the near future.²³

One of the limitations of the h-index is that it could be inflated by self-citation. Strategic self-citation as a means to manipulate the h-index has been reported.²⁸ Two studies have examined such phenomenon in academic plastic surgery. Lopez et al reported the inflation of h-index to be 2.2% among academic hand surgeons, although this was not statistically significant.¹ In contrast, the h-index experienced a 2.8% (statistically significant) decrease after self-citation removal in the study by Swanson et al.²⁹ In our study, the effect of self-citation on the h-index is of a similar magnitude at 2.7%, although this was not statistically significant. Overall, self-citation minimally influences the h-index of plastic surgeons, especially considering the h-index concept is relatively new with the first study published in 2014 in academic plastic surgery.^2,13,17

Despite being a novel concept, the h-index has rapidly gained widespread popularity among academic physicians and has been shown to predict academic rank and research funding.^13–15 It certainly could be utilized as an adjunct measure for academic promotion and career advancement. For junior trainees, the h-index could facilitate one’s decision-making process of selecting research supervisors and/or collaborators. For these reasons, every effort should be made to maximize the impact of our scholarly activities. Based on the findings of this study, we recommend that recruitments to academic centres should not be based on candidates with clinical fellowships alone but rather on candidates who have graduate degrees, such as MSc or PhD, in addition to clinical fellowships.

There are several limitations of our study. Firstly, the Scopus database inherently lags behind real time publications and citations. Having said that we believe this bias is equally applied to all plastic surgeons analyzed in this study and that the h-index is valid to demonstrate research productivity. In fact, we encourage all academic plastic surgeons to manually update their research records on the Scopus database. Secondly, faculty websites of training programs may not be up to date. Perhaps future studies should investigate self-reporting of academic attributes and research productivity by individual programs. Moreover, the cross-sectional nature of our study warrants longitudinal studies to examine dynamic changes of research productivity over time. In addition, Reed et al determined that women in academic medicine tend to achieve greater levels of academic productivity (ie, publication rates) later in their careers. Therefore, assessment of female plastic surgeons in the middle stages of their careers may underestimate their h-index values.³⁰ Other criticisms of the h-index include not discriminating between favourable versus unfavourable citations, not discriminating between author positions or contributions, and not being dynamic, in that the h-index does not diminish for inactive researchers.

Conclusion

The h-index is a novel tool for evaluating research productivity in academic medicine, and this study shows that the h-index can also serve as a useful metric for measuring research productivity in the Canadian plastic surgery community. Plastic surgeons would be wise to familiarize themselves with the h-index concept and should consider using it as an adjunct to existing metrics such as total publication number.