Impact factors of orthopaedic journals between 2000 and 2010: trends and comparisons with other surgical specialties

Abstract

Purpose

The impact factors (IF) of orthopaedic journals is an important component in determining the future of orthopaedic research funding. We aim to characterise the trend in journal IF over the last decade and draw comparisons with other surgical specialties.

Methods

We conducted an analysis of impact factors from Journal Citation Reports between 2000 and 2010.

Results

Between 2000 and 2010 the number of orthopaedic journals increased from 24 to 41, more than any other surgical specialty and the mean IF increased from 0.842 to 1.400. Journals printed in the English language had a significantly higher IF in the year 2010 (1.64 vs. 0.33, p = 0.01) than those printed in other languages. English language journals published in the US had significantly higher mean 2010 IF (1.932 vs. 1.243, p = 0.025) than those published in Europe, and this had changed compared with 2000 mean IF (0.978 Vs. 0.704, p = 0.360). Orthopaedics was ranked sixth out of 11 surgical subspecialties in 2000 but dropped to seventh out of 11 in 2010.

Conclusions

The quality of orthopaedic journals has significantly increased over the last decade and this has been accompanied by a rise in mean IF. It is important that orthopaedics continues to improve the quality of research, which may help orthopaedic researchers secure funding in the future.

Introduction

The publication of work in a reputable journal has become almost mandatory for progression in every surgeon’s career. Indeed ‘publish or perish’ has become accepted wisdom for those trying to enter and exit specialist training. In parallel with the increasing necessity for non-career academics to publish research there has been a large increase in the number of scientific journals. Between the years 1999 to 2010 the number of journals registered as ‘clinical medicine journals’ in the Journal Citation Report (JCR) – Thomson Reuters increased from 1,291 to 1,986.

The reputation of any journal is partly reliant on the frequency of citation of the articles that it has published, as this implies academic acceptance of published work by peers. It is often stated that “journals have prestige, but their prestige is only derived from the usefulness of the articles they publish” [1]. Impact factor (IF) is determined in an attempt to quantify and rank the quality of journals and has been published since 1961 by the Institute for Scientific Information (ISI). Invented in 1951 by Eugene Garfield [2] the 2010 IF for a given journal can be calculated as follows:

IF is a measure of the frequency that articles from a journal are cited by a larger group of journals, whilst controlling for the fact that some journals publish more citable articles than others. The impact factor attempts to eliminate some of the bias favouring large, frequently issued or older journals which have a larger citable body of literature than those which are smaller, newer or published less frequently.

Thomson Reuters states that the primary utility of the JCR reports is to assist librarians and researchers in selecting and managing journal collections. It acknowledges that the use of IF has been extended to evaluating academic work but states that whilst this may give an approximation of the prestige of journals it should not be used in isolation [3].

Impact factors have been widely criticised for various reasons [4–7], not least that they are open to manipulation using various techniques such as self citation (referring to articles from the same journal) [8], reducing the number of articles the JCR includes as citable (some types of articles such as editorials and letters are not counted as citable) and the inclusion of review articles, which are traditionally cited more frequently than other types of articles [9]. These and other problems mean that the impact of papers published in multidisciplinary journals are substantially overestimated while the impact of papers in more specialized journals may be significantly underestimated [10].

Despite the wealth of published literature exploring IF in general, there is a relative paucity of articles assessing the IF of orthopaedic journals. Studies published to date have noted a significant correlation between self-citation rate and IF and that specialist orthopaedic journals had a higher self citation rate than general orthopaedic journals [11, 12]. Siebelt et al. studied both the JCR and SCImago Journal Rank (SJR; based on Scopus® data) and demonstrated that SJR IF showed a strong relation with the JCR IF [13]. Bosker and Verheyan demonstrated that between 2000 and 2004 the United States accounted for the largest number of publications during that period [14].

This study aimed to investigate the trends in IF of orthopaedic journals over time and draw comparisons with other surgical specialties.

Materials and methods

IF from Journal Citation Reports [8] were analysed for the years 2000–2010. In keeping with previous studies, only journals deemed primarily to be orthopaedic journals by all authors were included [11, 13]. Two authors (RM and AS) then categorised the journals into general or subspecialty groups (Table 1).

Table 1.

Journals in 2010 primarily focussing on the practice of orthopaedic surgery, subdivided into general and specialist (accepted JCR abbreviations are used)

General orthopaedic journals, n = 23			Specialist orthopaedic journals, n = 18
Title	Language	Published	Title	Language	Published
ACTA ORTHOP	English	Scandinavia	ARTHROSCOPY	English	US
ACTA ORTHOP BELG	English	Belgium	EUR SPINE J	English	US
ACTA ORTHOP TRAUMATO	Turkish	Turkey	FOOT ANKLE INT	English	US
AM J SPORT MED	English	US	HAND CLIN	English	US
ARCH ORTHOP TRAUM SU	English	Germany	HIP INT	English	Italy
BMC MUSCULOSKEL DIS	English	UK	J ARTHROPLASTY	English	US
CLIN ORTHOP RELAT R	English	US	J FOOT ANKLE SURG	English	US
EKLEM HAST CERRAHISI	Turkish	Turkey	J HAND SURG-AM	English	US
EUR J ORTHOP SURG TR	English	France	J HAND SURG-EUR VOL	English	UK
INDIAN J ORTHOP	English	India	J PEDIATR ORTHOP B	English	US
INT ORTHOP	English	Germany	J PEDIATR ORTHOPED	English	US
J AM ACAD ORTHOP SUR	English	US	J SHOULDER ELB SURG	English	US
J BONE JOINT SURG AM	English	US	KNEE	English	Netherlands
J BONE JOINT SURG BR	English	UK	KNEE SURG SPORT TR A	English	Germany
J ORTHOP RES	English	US	MED CHIR PIED	French	France
J ORTHOP SCI	English	US	MINERVA ORTOP TRAUMA	Italian	Italy
J ORTHOP TRAUMA	English	US	SPINE	English	US
OPER ORTHOP TRAUMATO	English	Germany	SPINE J	English	US
ORTHOP CLIN N AM	English	US
ORTHOP TRAUMATOL-SUR	French	France
ORTHOPEDICS	English	US
REV CHIR ORTHOP	French	France
Z ORTHOP UNFALLCHIR	German	Germany

Impact factors were taken directly from the JCR 2000–2010. Differences in mean IF between groups were analysed by an independent sample t-test using SPSS for Macintosh Version 17.0; a p value of < 0.05 was considered significant.

Results

The number of journals focussing primarily on trauma and orthopaedic surgery (Table 1) has increased from 24 in 2000 to 41 in 2010. The mean impact factor has also increased over this period from 0.842 (range 0.099–2.233) to 1.400 (0.000–3.821). The majority of journals from 2010 are printed in the English language (n = 34, 81 %) and 50 % are published in the United States (n = 21). The 2010 impact factors ranged from 3.821 for the American Journal of Sports Medicine to 0.000 for the Minerva Ortthopedica E Traumatology (Fig. 1).

Fig. 1.

2010 Impact factors of journals listed in the JCR deemed to be primarily orthopaedic journals (as per Table 1). Listed by accepted JCR abbreviation

Journals printed in the English language had a significantly higher mean 2010 IF (1.64 vs. 0.33, p = 0.01) than those printed in other languages; there was no significant difference in 2000 IF (0.860 vs. 0.859, p = 0.231) (Fig. 2). English language journals published in the US had significantly higher mean 2010 IF (1.932 vs. 1.243, p = 0.025) than those published in Europe, and this had changed compared with 2000 mean IF (0.978 vs. 0.704, p = 0.360).

Fig. 2.

2000 Impact factors of journals listed in the JCR deemed to be primarily orthopaedic journals (as per Table 1). Listed by accepted JCR abbreviation

Both the specialist and general orthopaedic journals increased their IF over the study period (Figs. 3, 4 and 5); at no stage between 2000 and 2010 were the differences in mean IF significant.

Fig. 3.

Change in mean impact factor of orthopaedic journals 2000–2010

Fig. 4.

Change in impact factor 2000–2010 for orthopaedic journals

Fig. 5.

Change in impact factor 2005–2010

All journals increased their impact factor over the period 2000–2010 (range 0.204–2.100). Between 2005 and 2010 two journals experienced a decline in impact factor, the Journal of Paediatric Orthopaedics (British) (−0.033) and Orthopaedic Clinics of North America (−0.309). All other journals increased their impact factor between 2005 and 2010 (range 0.012–1.897).

In 2000, orthopaedics was ranked six out of the 11 surgical subspecialties we analysed according to mean IF (Table 2). Orthopaedics added 17 (ten general and seven specialist) journals to the JCR between 2000 and 2010, more than any other surgical specialty. This was accompanied by an increase in mean IF from 0.842 to 1.400; however, it dropped a place in the rankings to seven out of 11 (Table 3).

Table 2.

2000 mean impact factors for each surgical subspecialty

Category in 2000	Mean	Maximum	Minimum	Number of journals
Transplant surgery	2.322	4.035	0.678	4
Cardiovascular surgery	1.604	3.276	0.224	10
Oncology surgery	1.543	2.799	0.293	5
General surgery	1.339	5.987	0.159	30
Urology	1.175	2.896	0.151	13
Orthopaedic surgery	0.842	2.233	0.099	24
Neurosurgery	0.820	2.918	0.154	16
ENT	0.794	1.917	0.083	14
Pediatric surgery	0.686	1.216	0.350	3
Plastic surgery	0.653	1.423	0.159	9
Maxillofacial surgery	0.649	0.771	0.541	3

Table 3.

2010 mean impact factors for each surgical subspeciality

Category in 2010	Mean	Maximum	Minimum	Number of journals
Oncology surgery	2.677	4.182	1.118	5
Transplant surgery	2.650	6.051	0.495	7
Urology	1.942	8.843	0.274	20
Cardiovascular surgery	1.880	3.853	0.557	11
General surgery	1.583	7.474	0.053	42
Neurosurgery	1.567	4.791	0.116	18
Orthopaedic surgery	1.400	3.821	0.146	41
Maxillofacial surgery	1.376	1.890	0.772	4
Pediatric surgery	1.184	1.825	0.521	6
Plastic surgery	1.085	2.647	0.113	13
ENT	0.961	2.182	0.038	18

Discussion

In the last decade the number of orthopaedic journals has increased more than in any other surgical specialty. At the same time it is encouraging that the mean IF of orthopaedic journals has increased in line with trends seen in other surgical specialties (Tables 2 and 3). Orthopaedics has seen an increase in the number of journals published in languages other than English over the last decade and these have been shown to have significantly lower impact factors than those published in English. It is perhaps fair to assume that a new journal will take time to become established and have its articles cited. This taken together with an increase in non-English language journals is perhaps the reason why the overall IF of orthopaedic journals has not improved relative to other surgical specialties.

Both general and specialist orthopaedic journals in our study improved their IF between 2000 and 2010 with the means following a similar trend. At no time during the study period were the IF of specialist and general journals significantly different. This is perhaps surprising, as other studies have shown general orthopaedic journals to have higher IF than specialist orthopaedic journals because of the larger pool for citation [15–17]. Some of the journals with the highest improvement in impact factor between 2000 and 2010 were from the specialist category (Arthroscopy, European Spine Journal, Journal of Elbow and Shoulder Surgery).

Journals published in the United States had a higher mean IF than their European counterparts with a widening of the gap over the period of the study, a finding that is consistent with studies in other areas of medicine [9, 18]. This may be partially attributable to citing behaviour which favours journals from the United States in some fields, i.e. European researchers tend to frequently cite more papers from the United States, whereas US-based researchers rarely cite work from other nations [9].

To our knowledge this study represents the most wide-ranging review of the trends of orthopaedic journal impact factors over the last decade. Unlike existing studies we have chosen to analyse all orthopaedic journals irrespective of the language or country published. Analysing data for other surgical specialties proved troublesome as the JCR groups certain journals together i.e. Urology and Nephrology, Cardiology and Cardiothoracic surgery. It could be argued that the assignment of individual journals into one category or another was arbitrary. Furthermore, the country of publication could be ambiguous; for example, Acta Orthopaedica, although currently contracted for printing, etc., to Williams Willkins, is owned by the Combined Scandinavian and Dutch Associations.

Competition for research funding remains extremely fierce and there is evidence to suggest that funding bodies and academic assessment institutions have adopted IF for the evaluation of individual researchers or research groups when allocating funding [19]. The Research Excellence Framework (REF), scheduled for 2014, primarily aims to provide comprehensive ratings for research in all disciplines to inform UK higher education funding bodies in their allocation of grants for research [20]. It will in part use citation metrics (although this will not necessarily be JCR IF) to assess research output and impact [21].

The British Orthopaedic Association has noted a lack of support from funding bodies for trauma and orthopaedic academic units over the last decade and is developing a focused strategy to ensure that orthopaedics receives a share of funding from bodies such as the Medical Research Council (MRC) and the National Institute for Health Research (NIHR) [22]. Continued improvement in the IF of orthopaedic journals could be viewed as a vital aid to assist in securing future funding for orthopaedic research.

As orthopaedics continues to increase its research output it is vital that the quality of publications are maintained and improved. Studies have highlighted widespread weaknesses in published orthopaedic research [23–25]. Parsons et al. recently noted poor compliance to published guidance for the reporting of clinical research (CONSORT and STROBE) and a general lack of statistical rigour in orthopaedic research [24]. Okike et al. showed that studies with a high level of evidence, large sample size, representation from multiple institutions and conflict of interest disclosure are associated with higher citation rates, on which IF depends [26]. These factors do not necessarily indicate better research; rather they act as surrogate markers. Therefore if an editor considers it desirable to increase the impact factor of a journal then improving the quality of the research published would seem a plausible way to achieve this. This is encouraging for orthopaedics as we are seeing a trend towards larger national clinical trials and assimilation of clinical effectiveness data, along with utilisation of resources such as the National Joint Registry and the National Hip Fracture Database.

Conclusion

We acknowledge that impact factors have weaknesses and opponents; nevertheless, they are widely used and may be considered an important measure of research quality. This is the largest published study of orthopaedic journal impact factors to date and we have demonstrated an increase in both the number of orthopaedic journals and mean impact factor over the last decade. It is important that as a speciality we continue to strive to improve the quality of our research output, which should improve the impact factors of our journals.