Assessing research impact with Google Scholar: The most cited articles in the journal 2008–2010

Do you care about if and how you and your colleagues’ research articles are cited by other researchers? Which articles published in this journal are the most cited? Is it possible to calculate the impact factor of a journal by yourself? I often go to Google Scholar to check out the citations of articles. And being new as editor I wanted to study how the articles in our journal are being cited.

Google Scholar has the advantage of being free to use as compared with Web of Science owned by Thomson (who publish the Thomson Reuters Impact Factor), and Scopus, owned by Elsevier. Subscription fees for Web of Science are secret, but considered to be higher than Scopus fees which in 2005 were $US20 000–120 000 per year depending on institution size [1]. Google Scholar has increased its capacities and features in recent years and is now updated twice a week. Google Scholar used to yield fewer articles published before 1996, while today it readily retrieves earlier research. Bibliometric research comparing Google Scholar with Web of Science and Scopus shows differences but the correlation is high [2–5]. In addition to Thomson journal articles, Google Scholar searches show science reports, dissertations, books, and articles from journals not indexed by Thomson.

Lacking access to Web of Science or Scopus, I used Google Scholar to track citations for all 111 original publications in the Scandinavian Journal of Primary Health Care (SJPHC) in 2008–2010. The median number of Thomson journal citations according to a manual Google Scholar search for the 111 articles was two. I then calculated a 2010 impact factor by summing all Thomson journal citations published in 2010 that I found in Google Scholar for articles published in the SJPHC in 2008 and 2009. The sum of 144 citations was then divided by 77, which is the number of original articles published in the SJPHC during 2008 and 2009. My own 2010 impact factor calculation for SJPHC was 1.87. The 2010 impact factor delivered by Thomson is 1.909. This means that Google Scholar search missed three citations, i.e. 2% of those that make up the impact factor.

In Table I the titles of the most cited articles in each of the 12 issues published in the SJPHC 2008–2010 are presented plus the cumulated number of citations they have received both Google Scholar and Thomson journal citations; the number of cumulated citations from the SJPHC, 18% of all Thomson citations, as well as the share of Thomson journal citations that make up the 2010 impact factor. Obviously, articles published in 2008 and 2009 have been cited more than the 2010 articles. And, evidently, the most cited articles in 2008 contribute twice as much (17%) to the 2010 impact factor as do the most cited in 2009 (9%). This is because they have on average twice as much time available to be cited for the impact factor algorithm.

Table I.

Original articles published in the Scandinavian Journal of Primary Health Care (SJPHC) with the highest number of citations per issue in 2008–2010.¹

SJPHC issue	Article title	Google Scholar cumulated citations	Thomson journals cumulated citations (SJPHC)	Impact factor (IF) citations/share of SJPHC 2010 IF	Author country
2008–1	Problems in sickness certification of patients: A qualitative study on views of 26 physicians in Sweden [7]	35	21 (5)	10/6.9%	Sweden
2008–2	Pharmacologically inappropriate prescriptions for elderly patients in general practice: How common? [8]	24	18 (2)	7/4.9%	Norway
2008–3	Secular trends in cardiovascular risk factors with a 36-year perspective: Observations from 38- and 50-year-olds in the Population Study of Women in Gothenburg [9]	17	14 (4)	6/4.2%	Sweden
2008–4	Cancer rehabilitation: Psychosocial rehabilitation needs after discharge from hospital? [10]	18	13 (3)	1/0.7%	Denmark
2009–1	Falls risk among a very old home-dwelling population [11]	16	10 (0)	5/3.5%	Finland
2009–2	Does patient education facilitate diabetic patients’ possibilities to reach national treatment targets? [12]	9	6 (2)	2/1.4%	Sweden
2009–3	The Norwegian General Practice (NORGEP) criteria for potentially inappropriate prescriptions to elderly patients [13]	20	17 (2)	5/3.5%	Norway
2009–4	GPs’ antibiotic prescription patterns for respiratory tract infections – still room for improvement [14]	11	8 (1)	1/0.7%	Norway
2010–1	Early detection of COPD combined with individualized counseling for smoking cessation: A two-year prospective study [15]	6	6 (0)		Finland
2010–2	What do GPs feel about sickness certification? A systematic search and narrative review [16]	9	5 (2)		England
2010–3	Associations between successful palliative trajectories, place of death and GP involvement [17]	3	3 (1)		Denmark
2010–4	Influence of CRP testing and clinical findings on antibiotic prescribing in adults presenting with acute cough in primary care [18]	4	4 (0)		Norway

Note: ¹SJPHC citations in brackets. Impact factor citations are citations in Thomson journals during 2010. The articles’ percentage share of the 2010 impact factor follows.

Looking at the article titles shows that the journal publishes a wide variety of research subjects, which goes with our diverse area of medicine. I wanted to study whether any substantive fields of research were more cited than others. Therefore, I further analysed the article titles and tested a possible correlation between article title content and number of citations. No significant correlations between article title content and number of Thomson citations were found after repeated non-parametric tests using different title content categorizations.

Many have criticized the impact factor for various reasons [6]. Nevertheless, the success of research and its implementation requires articles to be read, spread, and cited. And one easy way to measure research impact is counting citations.