Impact factor of medical education journals and recently developed indices: Can any of them support academic promotion criteria?

Abstract

Journal Impact Factor (JIF) has been used in assessing scientific journals. Other indices, h- and g-indices and Article Influence Score (AIS), have been developed to overcome some limitations of JIF. The aims of this study were, first, to critically assess the use of JIF and other parameters related to medical education research, and second, to discuss the capacity of these indices in assessing research productivity as well as their utility in academic promotion. The JIF of 16 medical education journals from 2000 to 2011 was examined together with the research evidence about JIF in assessing research outcomes of medical educators. The findings were discussed in light of the nonnumerical criteria often used in academic promotion. In conclusion, JIF was not designed for assessing individual or group research performance, and it seems unsuitable for such purposes. Although the g- and h-indices have demonstrated promising outcomes, further developments are needed for their use as academic promotion criteria. For top academic positions, additional criteria could include leadership, evidence of international impact, and contributions to the advancement of knowledge with regard to medical education.

Introduction

Research is a fundamental aspect of academic life. It also represents an aspect of scholarship in medical education. Each month, approximately 60,000-65,000 new health-related research articles are published and indexed in the PubMed portal.[1] In most journals, however, the quality of the publications varies. Some papers are not clearly written, have poorly described methods, or use tools of low validity and reliability in spite of the Journal Impact Factor (JIF).[2] In academia, there is a need for introducing new indices to define the quality of research publications.

Academic departments, research centers, and funding bodies are increasingly interested in ways to assess academics’ research production and the quality of individuals’ research outcomes. In most universities, promotion and tenure systems reward individual achievements using general citation-based journal rankings. Although JIF is meant for journal rankings, several institutes let the ranking of journals where researchers published their work influence the academic career progression and the funding of grants.[3,4,5,6]

Medical educators, like other academics, are under pressure to publish their work in top-ranking journals listed in the Science Citation Index (SCI), Social Sciences Citation Index (SSCI), and Journal Citation Reports (JCR). For the preceding year, the JIFs are published in the JCR each June. The JIF of a scientific journal is the ratio of the number of citations found for the two preceding years of articles published and divided by the number of citable items published in the same two years.[7,8]

In a competitive research environment, alternative citation tracking allows researchers and universities to:

1. Identify the number of times a paper has been cited, and

2. Trace the development of research concepts or ideas over time by tracking them backward and forward.

This would enable researchers to work on the quality of their research to match the standards required by top journals in their field.[9,10,11]

Several studies have examined journal rankings in journals of different disciplines including nursing,[12,13] nutrition,[14] public health,[15] neurosurgery,[16] dermatology,[17] forensic science and toxicology,[3] psychology,[18] orthopedics,[19] radiation oncology,[20] and medical informatics.[21] For medical education, however, no studies have assessed the impact factor or discussed possible new tools for citation analysis. In the same vein, the h- and g-indices and the Article Influence Score (AIS) have not been studied in relation to medical education.[22,23,24]

The first part of this paper aims to review data sources and approaches for citation analysis. This knowledge is then applied to the assessment of 15 medical education journals to define highly regarded medical education titles by gathering data for each tool for these journals from Web of Science. We also aim to examine the strengths and limitations of using the JIF and other indices: h- and g-indices and the AIS.

The second part aims to assess whether any of these indices would add more evidence to support the policies and criteria of academic promotion and grant assessments and their current use in medical education.

First Part: Assessing The JIF and Recently Developed Indices

Journal Impact Factor (JIF): A critical review

The JIF has emerged as a tool for ranking, evaluating, categorizing, and comparing scientific journals.[3,8,25] The Institute for Scientific Information (ISI), a component of Thomson Scientific, was behind this development.

A listing of journals’ citations and their JIFs is made available by the ISI (Philadelphia, PA, USA), and it is also included in the JCR. It is important to note that the citation data of a single year and the citation data from only the two previous years’ articles constitute a significant limitation of the JIFs.[26] Considering the fact that the average paper is not cited in the first year after publication, data gathered for 1-2 years post publication is likely to provide an unrepresentative low snapshot of the Impact Factor. However, other researchers have shown that the relative short-term citation impact measured in the window underlying the JIF is a good predictor of the citation impact of the journals in the years to come.[6]

Another criticism of JIF is related to its calculation. JIF depends on which article types Thomson Scientific deems “citable”. Another limitation of the JIF is that the quality of the articles varies within a journal; the distribution of citations is skewed by only a few articles close to the population mean.[27,28,29,30] Therefore, the publication of review articles (which usually acquire far more citations than research articles) or the publication of just a few very highly cited research papers can improve a journal's JIF. It has been shown that less than 20% of the articles published in a journal account for more than 50% of the total number of citations. Many articles are not cited at all, or they are cited because some readers disagree with the authors.[24,31,32] Accordingly, a single publication cannot be judged by the JIF. Added to this is the bias that may occur due to self-citations.[33] However, the JIF may be misused or abused by journals with the aim to improve their impact factor. For example:

1. The journal may publish a larger percentage of review articles, which generally attract more citations than research articles;

2. The editor of a journal may set a submission policy that certain sections or articles be “by invitation only,” with the aim to invite exclusively senior scientists in the field to submit their work and ensure that the published papers are citable;

3. The journal may decline to publish articles such as “case reports” in medical journals because they are unlikely to attract citations;

4. “Abstract” or “biography” may not be allowed for certain articles and hence such articles will not be counted by Thomson Scientific as citable items, but these articles may attract citations and contribute to the rise of the JIF; and

5. The editor may publish accepted papers early online, before they are published in paper format, by about 4-6 months.

More on recently developed indices

To resolve the problems related to self-citations, Eigenfactor™ Metrics (http://www.eigenfactor.org/) was created by Carl Bergstrom, Jevin West, and Marc Wiseman at the Information School, University of Washington, Seattle, Washington, United States.[32,33,34,35] The Eigenfactor Score is somewhat similar to a JIF but is corrected for the journal's self-citations. Therefore, references from one article in a journal to another published in the same journal are removed during the calculation of the Eigenfactor.

Google Scholar and Scopus

Google Scholar was launched in 2004 as a gateway to scholarly literature.[36] The database is readily available free of charge and shows the number of citations of and details about the journals citing each paper. However, the contents are not organized under subject headings. This makes it difficult to assess a researcher's publication outcomes. In addition, it shows a broader range of sources than JCR or Scopus, resulting in the inclusion of nonjournal sources. Scopus is an indexing database built by Elsevier Co. and launched in 2004. The database claims 4600 health sciences titles and shows 100% coverage of the databases MEDLINE/PubMed, Embase, and Compendex. More details about Scopus have been highlighted elsewhere.[12,36,37] However, neither Google Scholar nor Scopus have addressed the limitations of JIF.

The h-index

In 2005, JE Hirsch proposed the h-index to assess the impact of an individual author.[22,23,36] The h-index has been shown to be of no value in journal ranking. To determine the h-index of an author, papers are ranked in a decreasing order of their received citations; the h-index is the (unique) highest number of papers that received h or more citations.[22,23] The h-index may have several advantages, as outlined in Table 1. However, the h-index is not sensitive enough to indicate changes even if the paper receives 5, 50, or 500 more citations: The index does not capture such changes in citations over time.[23,38]

Table 1.

Key information – Strengths and weaknesses of different indices

IndicesReference	What does it measure?	How is it calculated?	Strengths	Weaknesses
Journal Impact Factor (JIF)[3,8,25,26,27,28,29,30,31,32,33]	The average number of times articles from the journals published in the past two years have been cited in the JCR year.	Calculated by dividing the number of citations by the total number of articles published in the two previous years.	Can be used to compare journals within a particular field.	Cannot be used to compare journals across disciplines.
				The use of the arithmetic mean in the calculation of JIF is a statistically inappropriate measure.
				Is not recommended for assessing the work outcomes of researchers.
				Editorial policy may affect the JIF.
The h-index[22,23,36]	It measures the productivity and the impact of the published work.	Calculated on the basis of the researcher's obtained citations ranked in a descending order; the h-index is the (unique) highest number of papers that received h or more citations.	It is intended to measure simultaneously the number of publications (quantity) and the citations (quality) obtained by a researcher.	It does not consider information about the placement of the author in the author's list.
				It does not take into consideration the context of citations made (e.g., citations made in a negative context).
			It is not affected by time lag between publication and citation of material as it is the case with JIF.	It does not capture the changes in the number of citations attracted over time (insensitive to highly cited articles).
The g-index[23]	It is a mesure of the quality of research outcomes produced by a researcher and is sensitive to highly cited articles.	Calculations based on the highest number of papers that together received g2 or more citations.	It is more accurate in assessing the quality of research outcomes.	May not be suitable for assessing researchers in institutes with poor quality research.
			It is sensitive to highly cited articles.
Eigenfactor[35]	Similar to the JIF but corrected for the journal self-citation.	Calculated from the number of citations obtained by articles published in a journal in the last five years. In this matric the journal self-citation are corrected.	The Eigenfactor is not influenced by journal self- citation.	It also considers which journals have contributed to these citations. In this way highly cited journals will influence the network more than the lower cited journals.
Article Influence Score (AIS)[39]	The average influence of a journal's articles over the first five years after publication.	Calculated by dividing a journal's Eigenfactor Score by the number of articles in the journal, normalized as a fraction of all articles.	It is roughly analogous to the 5-Year JIF.	Cannot be applied to recently published articles/journals.

The g-index

Because of the limitations of the h-index and its insensitivity to highly cited articles, Egghe proposed the g-index.[23] The g-index is sensitive to the most cited articles. The g-index is defined as the highest number of papers that together received g² or more citations. In other words, the higher the number of citations received for an article, the higher the g-index.[23]

To explain the differences between the h- and g-indices and the sensitivity of the latter to highly cited articles, let us look at two examples. Researcher A has published five articles with 5 citations. This researcher has an h-index of 5. Researcher B has published 5 papers; four of them attracted 5 citations each, and the remaining one attracted 15 citations. The h-index for researcher B is also 5, while the g-index will vary depending on the number of citations attracted by the best article he/she has published. If the citations attracted by the best article were 15, 25, or 50, the g-index would be 6, 7, and 9, respectively. Therefore, the g-index is more sensitive in assessing a researcher's productivity than the h-index and far more accurate than the JIF in assessing individual researchers.

The Article Influence Score (AIS)

This index calculates the relative importance of the journal on a per-article basis. The AIS is obtained by dividing the Eigenfactor Score by the number of articles published in the journal and normalized to make the overall AIS of all journals 1.0. It is roughly analogous to the 5-year JIF; it is the ratio of the journal's citation influence to the size of the journal's article contribution over a period of 5 years.[39] Table 1 summarizes key information, strengths, and weaknesses of different metrics.

Second Part: Academic Promotion in Medical Education and Citation Indices

Academic promotion

For staff promotion, the universities often count such parameters as:

1. Number of papers published in peer review journals; 2. Number of papers published in top-ranking journals[7] ; 3. Number of citations and cites per paper; 4. Other scholarly work such as the number of patents, the number of graduate students supervised, conference papers at national and international levels, research books, chapters of books, and monographs; and 5. The number of grants and research projects with the applicant as the principal researcher or associate investigator.[40]

Interestingly, there has been limited discussion in the literature about academic promotion, but extensive documentation on university webpages. The existing literature criticizes such bibliometrics in decision-making. Notably, this has resulted in a discussion concerning the academic nursing profession,[41] similar to that seen in medical education: The amount of research is limited, but there is also considerable diversity in the research methodology.

The wide use of JIF in academic appointments and promotions takes two forms: The “quality” of the journals in which the applicant is publishing and the “quality” of the papers as measured by the number of citations.

Citation indices and staff promotion

Table 2 shows 16 highly regarded medical and allied health education journals with the JIF scores from 2000 to 2011. The total cites in 2011 under the category “Education, Scientific Discipline” were 42,997, and the Median Impact Factor was 0.902 for a total of 33 journals indexed under this category. Only 16 journals were selected for this study as the other journals covered other disciplines.

Table 2.

Changes in journal impact factor (JIF) from the year 2000 to 2011 for selected journals on medical and health related education (adopted from ISI Journal Citation Report, last updated in August 8, 2012)

Journal abbreviation	2000	2001	2002	2003	2004	2005	2006	2007	2008	2009	2010	2011
Acad Med	1.554	1.401	1.302	1.104	2.304	1.940	2.607	2.571	1.867	2.338	2.631	3.524
Adv Health Sci Educ	–	–	–	0.821	1.219	1.244	1.065	0.885	1.254	1.412	1.416	2.089
Adv Physiol Educ	–	–	–	–	1.291	1.043	1.260	0.984	1.483	1.542	1.382	1.547
Am J Pharm Educ	0.852	0.270	0.479	0.632	0.101	0.807	0.743	0.663	0.936	1.067	1.265	1.205
Anat Sci Educ	–	–	–	–	–	–	–	–	–	–	2.976	–
Biochem Mol Biol Edu	–	0.300	0.409	0.637	0.513	0.646	0.368	0.504	0.635	0.292	0.619	0.840
BMC Med Educ	–	–	–	–	–	–	–	–	–	–	1.201	1.152
CBE Life Sci Educ	–	–	–	–	–	–	–	–	–	–	1.182	1.191
Eur J Dent Educ	–	–	–	–	–	–	–	–	–	1.024	1.237	1.183
Indian J Pharm Educ	–	–	–	–	–	–	–	–	–	0.150	0.163	0.106
J Biol Educ	0.278	0.356	0.358	0.281	0.255	0.262	0.267	0.211	0.400	0.360	0.367	0.391
J Contin Educ Health	–	–	–	–	–	–	–	0.435	1.468	1.000	2.575	1.521
J Surg Educ	–	–	–	–	–	–	–	–	–	–	1.351	1.376
Med Educ	1.078	1.367	1.525	1.188	1.919	–	–	2.562	2.181	2.696	2.639	3.176
Med Teach	0.785	1.089	1.047	0.893	0.891	–	0.974	1.229	1.121	1.333	1.494	1.217
Teach Learn Med	0.586	0.523	0.797	0.479	1.108	0.867	0.727	0.825	0.731	0.741	0.679	0.748

Interestingly, Advances in Health Sciences Education, which was indexed for the first time in 2003, has demonstrated progressive increases in its JIFs over the following years. Other journals, such as Teaching and Learning in Medicine, which was indexed in 2000, have failed to demonstrate significant improvement in its JIFs over these past years. The recently published journal Anatomical Sciences Education, however, was indexed for the first time in 2010, with a JIF of 2.976.

The largest increase was found for Academic Medicine and Medical Education, whose JIF scores increased from 1.554 and 1.078 in 2000 to 3.524 and 3.176 in 2011, respectively. Two other journals with noteworthy performance were Advances in Health Sciences Education and Advances in Physiology Education. Although Medical Teacher has shown progressive increases in its JIF scores over the years, the improvement in the JIF values has been small.

Table 3 shows that 10 journals indexed in 2011 had 5-year JIF scores ranging from 3.189 (Medical Education) to 0.600 (Journal of Biological Education). The correlation between the 2-year JIF and 5-year JIF for these journals was high (r = 0.89, P < 0.001), which is consistent with other studies.[25]

Table 3.

Journal Impact Factor (JIF) of selected journals on medical and health related education before and after adjusting for the number of self-citations-JIF (2 years) and JIF (5 years)

Journal abbreviation	ISSN	Number of issues per year	Impact Factor* (2 year)	Impact factor (5 year)	Articles published in 2009 and 2010	Total cites	Number of self-cites (%)	Minus self-cites	Impact without self-cites
Acad Med	1040-2446	12	3.524	3.076	2257	7955	1111 (13)	6844	3.032
Adv Health Sci Educ	1382-4996	5	2.089	2.059	373	780	67 (8)	713	1.912
Adv Physiol Educ	1382-4996	4	1.547	1.825	416	643	195 (30)	448	1.076
Am J Pharm Educ	0002-9459	10	1.205	1.301	1080	1301	772 (59)	529	0.489
Biochem Mol Biol Edu	1470-8175	6	0.840	0.712	408	343	135 (39)	208	0.509
BMC Med Educ	1472-6920		1.152	–	580	668	58 (8)	610	1.051
CBE Life Sci Educ	1931-7913	4	1.191	–	246	293	110 (37)	183	0.744
Eur J Dent Educ	1396-5883	4	1.183	–	425	503	180 (35)	323	0.760
Indian J Pharm Educ	0019-5464	4	0.106	–	425	45	4 (8)	41	0.096
J Biol Educ	0021-9266	4	0.391	0.600	905	354	41 (11)	313	0.345
J Contin Educ Health	0894-1912	4	1.521	2.416	534	812	166 (20)	646	1.209
J Surg Educ	1931-7204	6	1.376	–	246	338	54 (15)	284	1.154
Med Educ	0308-0110	6	3.176	3.189	1720	5462	726 (13)	4736	2.753
Med Teach	0142-159X	12	1.217	1.653	2124	2585	575 (22)	2010	0.946
Teach Learn Med	1040-1334	4	0.748	1.007	969	725	45 (6)	680	0.701

*Median Impact Factor for the Category Education, Science Discipline is 0.902

Table 4 summarizes additional information about medical and allied health journals indexed in the JCR. For each journal, the table shows the number of citable articles and citable reviews in 2011 for 15 journals (no information available on Anatomical Sciences Education) as well as the number of references and the ratio of references to total citable items (articles and reviews). The number of citable reviews varied widely.

Table 4.

Citable items and number of citations for 15 medical education journals in 2011

Journal abbreviation	Number of citable items			Number of citations			Number of citations to total citable items
	Articles	Reviews	Combined	Articles	Reviews	Combined
Acad Med	184	8	192	5783	487	6270	32.7
Adv Health Sci Educ	96	3	99	1498	253	1751	17.7
Adv Physiol Educ	52	1	53	1375	7	1382	26.1
Am J Pharm Educ	148	6	154	3064	262	3326	21.6
Biochem Mol Biol Edu	60	0	60	996	0	996	16.6
BMC Med Educ	101	4	105	2663	164	2827	26.9
CBE Life Sci Educ	37	1	38	1242	21	1263	33.2
Eur J Dent Educ	43	0	43	1022	0	1022	23.8
Indian J Pharm Educ	57	1	58	1546	47	1593	27.5
J Biol Educ	30	1	31	580	46	626	20.2
J Contin Educ Health	43	2	45	1389	84	1473	32.7
J Surg Educ	79	4	83	1321	166	1487	17.9
Med Educ	103	11	114	3408	648	4056	35.6
Med Teach	210	7	217	5480	376	5856	27.0
Teach Learn Med	50	3	53	1042	123	1165	22.0

From Table 4 it appears that the mean number of references in the citable articles varied widely. It ranged from a low of 16.6 (Biochemistry and Molecular Biology Education) to a high of 35.6 (Medical Education).

Table 5 shows the ranking of medical and allied health education journals and the AIS of each journal. As is the case with JIF, only a few manuscripts enhance this score, while most manuscripts have not acquired a sufficient number of citations.

Table 5.

Ranking of medical education journals on the basis of JIF and the article influence score of each journal

Journal abbreviation	Country	Journal ranking*	Article influence score
Acad Med	United States	1	1.135
Adv Health Sci Educ	United States	3	0.796
Adv Physiol Educ	United States	4	0.416
Am J Pharm Educ	United States	8	0.163
Biochem Mol Biol Edu	United States	12	0.165
BMC Med Educ	England	11	–
CBE Life Sci Educ	United States	9	–
Eur J Dent Educ	Denmark	10	–
Indian J Pharm Educ	India	15	–
J Biol Educ	United States	14	0.130
J Contin Educ Health	United States	5	0.740
J Surg Educ	United States	6	–
Med Educ	England	2	1.011
Med Teach	England	7	0.485
Teach Learn Med	United States	13	0.416

*Journal ranking in its subject category is based on Impact Factor for the 15 journals included in the study. Some of these journals are listed in Health Care Sciences & Services category as well

Discussion

In this paper, JIF has been analyzed and compared with later developments in the use of citations for the evaluation of research quality in general, and the journals addressing medical education have been explored in some depth.

The introduction of JIF in 1997 was a major milestone. Today, however, the limitations of JIF are clearly felt by many,[24,31,32,34,42] and there is a growing need for additional, more sophisticated tools in all stages of scientific endeavor to optimize future success in research funding and academic recruiting. The development of medical education is today ever more guided by research,[43,44] but so far, no citation analysis of the JIF in comparison to the AIS, h-indices, and g-indices has been made. The ranking of medical education journals will probably fill an information gap within the health sciences. In this analysis, a number of well-regarded medical and allied health journals listed in JCR have been selected, analyzed, and compared.

From the analyses of the citation indices, the realization emerges with some strength that the current use of JIF does not serve the best of academic interests; an unjustifiable discrepancy between the journal ranking and the author ranking can be considerable. Moreover, there is a JIF bias in favor of publications within fields having a rapid turnover. JIF does not have the sensitivity and specificity to adequately meet the current needs and expectations for advances in the academic community across research fields.

Accordingly, when the funding of individual researchers or groups is to be decided or when making decisions about academic promotions, the use of the h- and g-indices together with the AIS is more likely to result in better assessments. The San Francisco Declaration on Research Association (DORA) recommends that JIF should not be used as a surrogate measure of the quality of an individual research article.[45]

Another important issue is the growing realization that JIFs are biased toward certain fields of research. For example, JIF is strongly in favor of high-profile disciplines with a rapidly cycled field of discoveries and turnover, such as molecular biology and biochemistry. This does injustice to low-profile disciplines such as health education, nursing, and midwifery.[46] The speed of turnover makes it difficult for medical educators to compete with colleagues from some other disciplines. It is also important to realize that the highest impact factors for journals covering medicine, biochemistry and molecular biology, biochemical research methods, and biology are 53.298, 34.317, 19.276, and 11.452, respectively, while the highest impact factor for medical education journals is only 3.524 (for Academic Medicine).

Furthermore, the numbers of journals in the area of medicine (general and internal), biochemistry and molecular biology, biochemical research methods, and biology indexed in the JCR are 155, 200, 72, and 85, while only 14 journals are indexed under medical education, and one for dentistry education, and another one for pharmacy and pharmaceutical education. This situation leaves limited opportunities for medical and allied health educators to publish their work in high-impact journals. As another example, consider that a medical educator publishes an article in Academic Medicine, a journal with a JIF of 3.524, and another colleague from the Department of Medicine at the same institute publishes in Annals of Medicine, a journal with a JIF of 3.516. Both journals have nearly the same JIF, but Academic Medicine is the top journal in medical education, while Annals of Medicine is ranked #19 in its own field. This major difference is totally ignored if only the JIF is considered in the academic assessment of research outcomes.

Nevertheless, better indices provide vital support in decision-making for research for funding, recruitment, and improved teaching in the competitive environment of academia. In certain ways, a change in the current use of citation indices will sharpen the competition in wholesome ways. More importantly, it is likely to enable better decisions and more fairness with regard to assessments of the publication output of individuals and research groups across disciplines and methodologies. In addition to these metrics, a battery of other indices should form the basis for academic promotion, particularly for top positions, including the following: 1. Invitations to speak internationally about research, 2. A sustained record of being the principal investigator in funded research, 3. Services as an editor and/or editorial board member of medical education journals and scientific journals, and years as peer reviewer to top international journals in the field, 4. Leadership roles on national and international committees of major medical education societies, and major conferences on medical education, 5. Prestigious national and international awards for research and innovations in medical education, 6. Leadership in international collaboration in research and publication as principal investigator, and 7. Leadership and accumulated achievements in specific areas in medical education.

Each of these indices could be standardized by a numerical system. For example, invitations as a keynote speaker may be evaluated by using the following scoring system: 0 = not invited, 1 = invited to speak in a meeting held within their own university, 2 = invited to speak at a national conference, 3 = invited to speak at an international university ranked lower than their own, 4 = invited to speak at an international university ranked higher than their own, 5 = invited to speak at a major international conference. Indices such as these could enhance assessment for academic promotion.

Conclusions

Given the need for tighter links between research quality and funding as well as recruitment practices, it is time to revise the scientific evaluations also within medical teaching; institutional decisions should preferably be evidence-based and favor individuals with solid scientific merit rather than be driven by coincidental or ideological motives. In the absence of better tools, rough approximations of scientific quality were derived from the JIF in the past. Although AIS and the g- and h-indices have shown promising outcomes, further developments are needed. Other key indices, particularly for top academic positions, should also be considered.

Financial support and sponsorship

This work was funded by the College of Medicine Research Center, Deanship of Scientific Research, King Saud University, Riyadh, Saudi Arabia.

Conflicts of interest

The authors declare that they have no conflict of interest and that the whole manuscript has been created by the authors.