Factors associated with citation rates in the orthopedic literature

Abstract

Introduction

Investigators aim to publish their work in top journals in an effort to achieve the greatest possible impact. One measure of impact is the number of times a paper is cited after its publication in a journal. We conducted a review of the highest impact clinical orthopedic journal (Journal of Bone and Joint Surgery, American volume [J Bone Joint Surg Am]) to determine factors associated with subsequent citations within 3 years of publication.

Methods

We conducted citation counts for all original articles published in J Bone Joint Surg Am 2000 (12 issues). We used regression analysis to identify factors associated with citation counts.

Results

We identified 137 original articles in the J Bone Joint Surg Am. There were 749 subsequent citations within 3 years of publication of these articles. Study design was the only variable associated with subsequent citation rate. Meta-analyses, randomized trials and basic science papers received significantly more citations (mean 15.5, 9.3 and 7.6, respectively) than did observational studies (mean retrospective 5.3, prospective 4.2) and case reports (mean 1.5) (p = 0.01). These study designs were also significantly more likely to be cited in the general medical literature (p = 0.02).

Conclusion

Our results suggest that basic science articles and clinical articles with greater methodological safeguards against bias (randomized controlled trials and meta-analyses) are cited more frequently than are clinical studies with less rigorous study designs (observational studies and case reports).

Investigators conducting surgical research aim to disseminate their findings to the widest possible audience. Citation rate of a study after publication is one measure of the study's exposure. Having a research paper cited by multiple sources after its original publication can facilitate the translation of its findings to patient care. That is, the quicker the information about important surgical research is spread to the surgical community, the more likely it is to be applied to patient care. Although the application of new techniques and research findings among surgeons is more complex than simply disseminating information to a vast audience, identifying factors that are associated with increased citation rates may assist authors in improving the impact of their work.

We focused on the orthopedic surgery literature, with the understanding that our results would likely be relevant to other areas of surgery. The Journal of Bone and Joint Surgery, American volume (J Bone Joint Surg Am) is currently the most frequently cited clinical orthopedic journal, with a reported impact factor of 2.14. The purpose of our study was to evaluate the factors associated with increased citation rates of individual studies. We hypothesized that study designs, such as randomized trials, would achieve higher citations than would study designs of lower levels of evidence, such as case series.

Methods

We included all original scientific articles published in the J Bone Joint Surg Am in 2000. Two investigators conducted hand searches of the journal, from January through December 2000. From each eligible article, we collected the following baseline information:

• study design (retrospective observational study, prospective observational study, randomized trial, meta-analysis, and basic science);

• sample size;

• continent where the study was conducted; and

• area of orthopedic specialty (spine, hip and knee reconstruction, pediatrics, trauma, sports medicine, upper extremity and oncology).

Citation counts

Using the first author's name, 2 of us queried the Institute for Scientific Information (ISI) Web of Science database (http://isiknowledge.com) to ascertain, as of March 31, 2003, the number of subsequent citations for each article after publication. If entering the first author's name failed to yield any citations for an article, we searched for the second and last author to limit misclassification of an article as having zero subsequent citations. We chose a 3-year period after publication (2000–2003) to assess citations, on the basis of previous reports (see Fassoulaki and colleagues¹).

For each article identified in the ISI database, we collected the number of citations in orthopedic journals, the number of citations in nonorthopedic journals and the number of subsequent self-citations by one or more authors of the original published paper. A journal was considered orthopedic if the publication routinely included articles pertinent to orthopedic surgeons (i.e., those characterizing studies involving patients with musculoskeletal problems relevant to orthopedic surgical practice). We subdivided the number of citations in nonorthopedic journals into those published in the 5 highest impact medical journals (Journal of the American Medical Association impact factor = 17.6, New England Journal of Medicine impact factor = 29.1, British Medical Journal impact factor = 6.6, Annals of Internal Medicine impact factor = 11.1 and Lancet impact factor = 13.3). Gami and others² suggested that bias may result from author self-citation. To determine whether a subsequent citation was a self-citation, we compared the complete authorship of the original journal articles with the complete authorship of all citing articles.

Data analysis

We used analysis of variance to compare the number of citations among journals, across different years, and between orthopedic and nonorthopedic journals. We used t tests to conduct post hoc comparisons. To account for multiple comparison tests. We corrected the alpha level, using the least squares difference technique.³ We also conducted multivariable regression to examine whether study design, sample size, geographical location of study or study topic predicted the number of subsequent citations. Residuals from the regression analysis were plotted to ensure that assumptions of normality were maintained.

Results

Study characteristics

We identified 137 original articles in J Bone Joint Surg Am published in 2000. Two were meta-analyses (level 1 evidence), 3 were randomized trials (level 1 evidence), 22 were prospective observational studies (level 2 evidence), 79 were retrospective observational studies (level 4 evidence), 20 were basic science reports and 11 were case reports. Sample sizes ranged from 1 to 2367 patients (mean 109). Most articles focused on joint arthroplasty and upper extremity disorders (70/137, 51%) and were conducted in North America (109/137, 80%) (Table 1).

Table 1

Subsequent citations

We identified 749 citations of the 137 original articles. The number of citations after publication ranged from 0 to 38 (mean 5.8); of these, 24 articles (17.5%) had received no citations up to March 31, 2003.

In our regression analysis adjusted for study design, sample size, self-citation, geographical location of study and study topic, study design was the only variable significantly associated with the number of citations after publication (p = 0.045). Meta-analyses, randomized trials and basic science papers received significantly more citations (mean 15.5, 9.3 and 7.6, respectively) than did observational studies (mean retrospective 5.3, prospective 4.2) and case reports (mean 1.5) (analysis of variance, p = 0.01). (Fig. 1) The overall rate of self-citations among the 749 citations was 12.2% (92/749). Study design was not associated with self-citations.

FIG. 1. Mean number of citations by study design. Randomized trials, meta-analyses and basic science papers received significantly more citations than did other study designs. *p < 0.05 when compared with meta-analyses.

Citations within orthopedic versus nonorthopedic journals

Of 749 citations, 225 (30%) appeared in nonorthopedic journals. Study design was significantly associated with citations in nonorthopedic journals (p = 0.02). Meta-analyses, randomized trials and basic science reports yielded significantly greater citations than did other study designs (p < 0.05) (Fig. 2). Nine articles in the J Bone Joint Surg Am were cited in 1 of the 5 highest impact general medical journals. Of these, 4 were prospective case series, 1 was a randomized trial and 4 were retrospective case series.

FIG. 2. Mean number of citations in orthopedic and nonorthopedic journals. Meta-analyses, randomized trials and basic science papers received significantly more citations in nonorthopedic journals than did other study designs (*p < 0.05 when compared with case reports).

Discussion

In a review of 137 scientific publications in J Bone Joint Surg Am, we found that authors can expect, on average, to have 5 citations within 3 years of publication; we also found that meta-analyses, randomized trials and basic science reports are significantly more likely to be cited, and cited in nonorthopedic journals, than are other study designs. Further, although self-citation rate was substantial, it was not associated with overall citation rates.

The impact factor for a journal is the number of citations to articles published in the journal in the past 2 years divided by the number of articles published during that period. Impact factor is the frequency with which an average article in a journal has been cited in a particular year. ISI also reports the number of citations that individual articles receive. The use of citations to measure relative impact has been criticized for the following reasons: 1) the 2-year period set by the ISI for citations is arbitrary, 2) the ISI's database doesn't contain all available journals, and 3) the impact factor does not take into account self-citations. Although the use of the impact factor is recognized to have limitations, it is presently the preferred measure of an article's exposure.^4,5

Strengths and limitations

One strength of our study is that we evaluated citations from all scientific articles over a complete year. In addition, we assessed citation counts for each article in duplicate. Our findings provide some vindication for using levels of evidence as a general measure of the hierarchy of study design. Papers with higher levels of evidence were more likely to be cited after publication than were lower-evidence publications (level 4). Our findings are limited by the fact that we did not assess the quality of the citation or whether it supported the index paper, nor did we point out flaws in the index paper.

Relevant literature

An important finding of our study is the 3.6-fold and 2.2-fold increases in citations with meta-analyses and randomized trials, respectively, when compared with observational studies. Our findings are supported by Weale and Lear⁶ who identified 139 surgical journals and 112 medical journals in the ISI database in 2001. These authors searched Medline to identify any randomized trials published in 1999 and 2000 in these 251 journals. Randomized trials accounted for a median of 2% of all published studies.⁶ The number of randomized trials published in a journal over the study period was significantly correlated with impact factor (r = 0.38, p = 0.01).

Our results are discrepant with those of Callaham and colleagues,⁷ who used the Science Citations Index database for all citations of published articles originally submitted to a 1991 emergency medicine specialty meeting. Of 204 included articles, the mean citations per year was 2.04 (95% confidence interval 1.6–2.4; range 0–20.9). The strongest predictors of citations per year was the impact factor of the original publishing journal and the study sample size. However, features of the study design (i.e., randomization, blinding, retrospective vs. prospective) had little influence on the number of citations. In another study, West and McIllwaine⁸ identified 417 citations in papers published in 1995–1997 in an addiction journal. No association between the number of citations and quality rating of the index paper was identified (r = -0.01, p > 0.05).

Studies evaluating the influence of self-citation rates on journal impact factors have reported conflicting results.^1,6–10 This finding may be due to the differences in self-citation rates among journals. Fassoulaki and colleagues¹ examined the influence of self-citation rates on journal impact factor in 1995 and 1996 issues of 6 different anesthesia journals. Self-citation rates ranged from 17%–35% in these journals and were positively correlated with journal impact factor (r = 0.90, p = 0.015).¹ Motamed and colleagues⁹ identified self-citation rates of 4%–11.9% across 6 otolaryngology journals in 1997 and 1998; however, these investigators reported no significant correlation with journal impact factor (r = –0.31, p = 0.56). The self-citation rate (12.2%) in J Bone Joint Surg Am was similar to that reported by Motamed and colleagues.⁹

Conclusions

The publication of randomized trials, meta-analyses and basic science reports in J Bone Joint Surg Am led to significantly more citations and more citations in nonorthopedic journals than other study designs. Authors involved in orthopedic research may improve the impact of their work by pursuing basic science research or by designing clinical studies with greater methodological safeguards against bias. Increased dissemination of research findings may lead to quicker adoption of clinically important research findings to patient care.

Dr. Bhandari is funded, in part, by a Clinical Scientist Fellowship, Department of Clinical Epidemiology and Biostatistics, McMaster University. Dr. Busse is funded by a Canadian Institutes of Health Research Fellowship Award. Dr. Devereaux is funded by a Heart and Stroke Foundation of Canada/Canadian Institutes of Health Research Fellowship Award. No funding was received in preparation of this manuscript.

Accepted for publication Jan. 27, 2006.

Competing interests: None declared.

Dr. Mohit Bhandari, Department of Clinical Epidemiology and Biostatistics, McMaster University Medical Centre, 1200 Main St. W., Rm 2C9, Hamilton ON L89 3Z5; fax 905 524-3841