RE: Vanner S.“Fair is foul, and foul is fair”. Can J Gastroenterol 2011;11:601–2.
To the Editor,
A bibliometric analysis of digestive health research in Canada: “If you’re not taking flak, you’re not over the target”
We are writing to respond to some of the comments from Dr Stephen Vanner in the editorial he wrote regarding our article in the November 2011 issue of the Journal, and to the letter from Dr William Depew published in the current issue. Many of Dr Depew’s comments overlapped with those of Dr Vanner.
Dr Vanner stated that our paper had a “fundamental flaw” related to the database of digestive health researchers we used, as he felt that it “does not accurately reflect the digestive health research community in Canada”, that “a number of defined research ‘groups’ are artificial” and that our selection of 106 investigators from across Canada “represents a serious bias, failing to include some high profile investigators” while “including others who are not part of active research groups”. First, as detailed in our paper, we attempted to collect bibliometric data on as many Canadian digestive health researchers as possible, using the definition “researchers who devote a substantial portion of their research efforts to digestive health and disease, and generally, who are affiliated with digestive health research groups and/or gastroenterology or hepatology divisions”. We also explained in the paper that individuals within their first three years of their academic appointment were excluded (so that the analysis was focused on researchers who have had a chance to establish their research programs). Not stated in our paper was the fact that we only included researchers in the analysis who were based at a university in Canada (researchers outside of academia were not included). We recognize that we may have missed some researchers, and for this we apologize. We encourage readers to e-mail us the names of any digestive health researcher who they feel was overlooked, for inclusion in future analyses.
We grouped the investigators according to their university affiliation because this was the simplest and least ambiguous mechanism. We recognize that some researchers belong to research groups within their universities (often hospital-based, for example) but have no way of determining whether such groups are “artificial”, as Dr Vanner suggested. We also believe that it would be very difficult, if not impossible, to determine whether a particular researcher’s group is “active”, as Dr Vanner has suggested.
Dr Vanner also indicated that our use of the Hirsch factor as one of the two metrics we employed was a “major flaw”. We acknowledge (and did so in the paper) that the Hirsch factor is not a perfect metric for determining the impact and influence of a researcher. However, at this point in time, it is widely regarded as the best available metric, and it is widely employed (indeed, we have previously received funding from a Canadian funding agency to perform analyses of the Canadian digestive health research community). We would argue that the H factor does provide a good index of the impact a researcher has had on their field. In contrast to what Dr Vanner stated, we did make mention in the paper of some of the shortcomings of the H factor, including the impact of collaborative studies and co-authorship. We did not at any point state or imply that someone with an H factor of 32 is ‘twice as good’ as someone with an H factor of 16, but we did mention that junior investigators would generally have lower H factors than senior members (though we saw clear exceptions to this in our analysis). As stated in the paper, senior researchers have had more time to publish and to be cited, so they will tend to have higher H factors than younger investigators. Dr Vanner claimed that our use of the H factor for the most recent decade was done “to correct for seniority” and that it “represents a substantial bias towards groups that are top heavy with senior investigators”. We would argue the opposite. Restricting the analysis to the past decade gives a more accurate index of recent productivity of the researchers, and would largely favour younger investigators who are in their most productive years (as opposed to more senior investigators who were more productive prior to 2000). This was borne out by the presence in the top 25 rankings of many more junior researchers when the 2000 to 2010 period was evaluated as opposed to the full database (1976 to 2010).
The Influence factor that we also employed did, in fact, prove useful in differentiating researchers who have the same or similar H factor, but have different levels of influence in this field. There were researchers in our study who had identical H factors, but with one of the researchers having three times as many citations as the other. The Influence factor, though imperfect, separates those two researchers where the H factor would not.
Dr Vanner suggested that we had a “scientific obligation to describe the nature of the data and to provide some sort of statistical treatment”. We believe that readers of the Canadian Journal of Gastroenterology can judge the data for themselves. We displayed the raw data in figures 1 and 2 along with the average scores for each university and for the entire database.
Finally, Dr Vanner was critical of the efforts we took to reduce error when calculating the metrics used in our study, referring to the fact that we contacted the 30 top-ranked researchers to ask if they agreed that our metrics for them were correct. The bibliometric analyses we performed were done separately by three different analysts. The results were compared, and where there were differences, we determined the reasons for the differences and made the appropriate corrections. As stated in our article, in addition to using the ISI Web of Science, we also a used a number of bibliographic websites and search engines to try to reduce the chance of errors in the bibliometric data. Contacting the 30 top-ranked researchers was our final quality control step. Our reasoning was that the highest ranked researchers have the most extensive lists of publications, so if our data were correct for those researchers, they likely would also be correct for researchers with fewer publications. However, we take Dr Vanner’s point and, in the future, we will use a random sample of researchers for this final quality control procedure.
The portion of Dr Depew’s letter that did not overlap with the Vanner editorial focused largely on ethical issues, the main one being related to the role of the first author (DT) in the study and a perceived bias toward McMaster University.
Several statements in Dr Depew’s letter are incorrect. The data that were put on the REM website in January 2011 were not from the study published in the Canadian Journal of Gastroenterology. Indeed, the study published in the November issue of the Journal had not even initiated in January of 2011. We did not issue any press release or put information on our website regarding the study until it had been published, in November 2011. The data Dr Depew referred to were from an earlier, less extensive study that focused on a different time frame and only used H factor as a metric. It was THAT study that was highlighted on the Farncombe Institute’s website at the time, not the one recently published in CJG. The first author of the paper (DT) was employed by the Department of Medicine at McMaster University and had been working with Dr Wallace on a project related to cannabinoids as a potential therapy for inflammatory bowel disease. This affiliation with McMaster was declared in the authorship line, even though the work at McMaster was unrelated to bibliometric analyses. With respect to the claim that the study was biased in favour of McMaster University, Dr Depew does not mention the fact that the University of Toronto ranked better than McMaster University in the study, and that REM has done work for that University as well.
It is not correct, as Dr Depew claims, that the REM website “promotes McMaster and by default the Farncombe Institute as ‘most dominant’ in terms of digestive disease research impact”. In fact, it states that Toronto and McMaster were the most dominant, followed by the University of Calgary, Manitoba and Western.
With respect to his comment regarding “the ethical bounds of subjects consent”, one needs to bear in mind that the data presented in our article are entirely in the public domain, and are based entirely on publications. Authors voluntarily publish work, and those works are publicly cited by other authors.
With respect to Dr Depew’s comment about the ‘coding’ of researchers in our data tables, it is important to note that this was done in response to a request from the editors of the Journal.
As mentioned in our paper, the analysis we performed provides a ‘snapshot’ of digestive health research in Canada. We believe that the analysis shows a vibrant research community that is well represented in universities across the country.
Désirée Tuitt,
Tara Lipman