RE: Tuitt D, Knight F, Lipman. A bibliometric analysis of digestive health research in Canada. Can J Gastroenterol 2011;11:609–614.
To the Editors, Caveat emptor
I have read with disappointment the article ‘A bibliometric analysis of digestive health research in Canada’ by Tuitt et al, published in the November 2011 issue of the Journal. The publication of this article raises concerns about ethical matters and scientific issues bearing on the very essence of evidence-based reporting. Some of the problems were exposed in the accompanying editorial by S Vanner in the same issue of the Journal. These matters alone are substantive. I raise additional and potentially serious irregularities that further undermine the worth of this contribution.
I will first address the troubling ethical issues of consent and conflict of interest. The authors are associates or employees of Research Excellence Metrics (REM). This commercial enterprise lists the Farncombe Family Digestive Health Research Institute and McMaster University (Hamilton, Ontario) as clients, among others. Both of these organizations are presented in a most favourable manner in the article. This affiliation was promoted publically on the REM website for months before the receipt and acceptance of this manuscript. On their website, REM published graphic illustrations of the rankings of their artificially constructed digestive diseases research groups in January 2011. Professor John Wallace, Director of the Farncombe Institute, cited the Institute’s REM-defined ‘top ranking’ as a promotional piece on the Farncombe website at the same time. To sharpen the focus on this issue, it is necessary to point out that Desiree Tuitt is listed as a research analyst (part time) in the research group of Professor Wallace at the Farncombe Family Digestive Health Research Institute, an affiliation that is not declared. In these days of uncompromising full disclosure, this apparent oversight is unacceptable.
In a blighted effort to achieve a degree of anonymity, the authors code the list of investigator subjects. One can only speculate why they did not code the institutional sites. They now provide, post facto, a complete list of names of the subjects on their website! If this action does not transgress the ethical bounds of subject consent, it comes perilously close. That the authors have taken unfair advantage of the Journal is evident.
I will deal next with the scientific problems. A primary role of evidence-based scientific publications is advancement of knowledge in a discipline. For the Canadian Journal of Gastroenterology, such publications should serve the national interests in digestive disease research, diagnosis and treatment. Submitted manuscripts must be scientifically sound. There must be a worthy purpose. Methods must be carefully elaborated. The data must be complete and accurate. Data manipulation must conform to analytical standards. I submit that none of these elements are properly preserved in this instance.
The title of the article suggests that the research described will analyze Canadian digestive health research. What is the purpose of such an analysis? Will the analysis provide evidence showing that Canadian digestive disease research is thriving or has improved over time? Will it show that digestive disease research is keeping pace with research in other fields? Will it show that such research compares favourably with similar research in other countries? The only questions identified by the authors relate to potential differences between males and females, and between clinician and basic scientists. These are partially addressed, but the inexplicable comparisons among different Canadian sites leaves this reader puzzled as to how such comparisons relate to the stated objective. Why have they chosen to make such comparisons? How do such comparisons benefit the national interests of Canadian digestive disease researchers? I argue that the impetus for these comparisons has less to do with an analysis of Canadian digestive disease research and more to do with the commercial nature of the organization responsible for the manuscript and the relationships the company has with its clients.
The authors chose to use the Hirsch Index (h–I) as a metric. They include their own ‘Influence factor’ for good measure. Vanner identified several limitations of h–I. There are many significant problems with the h–I. None of these were addressed adequately in the article. Hirsch himself cautioned about the use of the h–I or any other single number in isolation in making determinations about productivity and impact. He professed that “a single number can never give more than an approximation to an individual’s multifaceted profile and that many other factors should be considered in combination in evaluating an individual”. He also opined that “Although I argue that a high h is a reliable indicator of high accomplishment, the converse is not necessarily always true”. He warned users to correct the index for scientific ‘age’. The authors’ attempt to deal with this major issue is naive and misleading. Manuscript opportunity depends directly on investigator-based infrastructure, which changes over time. For bench scientists, this includes the acquisition of equipment, technicians and graduate students. For clinical investigators, it depends on the acquisition of ancillary research personnel such as study nurses and trial networking among like-minded investigators and pharmaceutical partners. Investigators gain manuscript momentum during their careers. It is not scientifically justifiable to use h–I for comparative purposes in a heterogeneous group of investigators with widely disparate scientific ages without normalizing for such age-determined momentum. Hirsch also drew attention to the matters of variations in the magnitudes of h determined solely by subfields of study and he recommended correction for coauthorship. None of these crucial issues was addressed properly. Several scholarly publications have raised real concern about the actual utility of h–I in these research impact measurements. The limitations of h–I demand careful integration by anyone using it to measure individual contributions. The treatment in this particular article is inadequate.
Examination of the scatter plots shows considerable overlap. It is clear that more than 50% of the subjects across all sites have values below the mean for the entire cohort. Are these investigators unworthy? I think not. It is more likely that the variation is related to those confounding issues Hirsch warned about – the very things the authors failed to adequately address! As a member of a Canadian Institutes of Health Research grants review panel for several years, I can state unequivocally that, despite its shortcomings, real-time peer review by knowledgeable individuals trumps bean counting under any circumstance.
If the h–I has limitations, there should be great consternation at the uncertainties attached to the REM-conceived ‘Influence factor’ variable. At least h–I has been submitted to some independent scrutiny. The REM Influence factor has not. It is axiomatic that the performance characteristics of any measurement tool must be carefully defined and validated before it is used in a protocol. There is no such validation apparent.
The authors offer no statistical treatment of the data whatsoever. Regrettably, there is no discussion on this point. Data handling and presentation in this article are totally unfettered by scientific convention.
The authors’ data collection was seriously flawed. On their own website, the authors present a graphic illustration of the careers of two Canadian investigators. One of these, Brett Finlay at the University of British Columbia (Vancouver, British Columbia), is shown as a scientist with major impact and influence. I agree that Dr Finlay is a recognized leader in research. However, his research is in the domain of digestive diseases. He does not appear in the authors’ list of investigators. Why is this? Estimation of Dr Finlay’s h–I for the 1976 to 2010 interval at 84 would place him first overall. It is unreasonable to suggest that the authors forgot Dr Finlay. It is unacceptable to suggest that they weren’t aware that his research was in digestive diseases. Furthermore, an investigator at one of the implied underachieving centres has an h–I of at least 54. This would place this individual in the top 5! Why was this investigator excluded, forgotten, not identified or deliberately left out? The authors are solely responsible for these critical errors.
The authors have arbitrarily reported their results, many in artificial groups linked with specific institutions. Plots show mean values for these so-called groups. The implication is that groups with high means are better than those with low means. The quote by Tuitt et al on the REM website provides unequivocal confirmation of this intent. The truth is that there are few such functional groups. From the perspective of supporting the national interests of the Canadian Association of Gastroenterology, the Canadian Association for the Study of the Liver, the Canadian Digestive Health Foundation and all those Canadian scientists engaged in digestive disease research, the negative impact of this method of presentation is potentially stunning.
Following publication in the Journal, the REM website now promotes McMaster and, by default, the Farncombe Institute as ‘most dominant’ in terms of digestive disease research impact. Indeed, the site proclaims “The study’s lead author, Desiree Tuitt, suggested that the data can be useful in optimizing investments in research. ‘There is tremendous competition for medical research funding in Canada. Our analysis has identified the researchers who are having the greatest impact in this field’ – in other words, those who are achieving the greatest bang for each research buck”! The authors are claiming, for the advantage of their clients, that they can help to identify Canadian researchers who should be funded and, by inference, who should not! There is no cost-benefit analysis to support this reckless assertion. Based on the evidence presented by the authors, it is simply not possible to reach any such conclusion. It is clear that the real objective of the manuscript was the establishment of a foundation for promotional client-related purposes, not unbiased scientific discovery.
I am saddened by this series of events. Regrettably, I believe the Journal has been duped. In doing so, the authors have successfully manipulated a scientific journal into a promotional platform for the dissemination of biased information and wholly unsupportable conclusions. As a consequence, the Journal has failed our digestive diseases community and a considerable proportion of highly capable scientists who labour with skill and enthusiasm to maintain digestive diseases research in Canada. The potential contributions to Canadian digestive disease research by several capable Canadian scientists have been put at risk. The abuse of a legitimate scientific forum for the purpose of promotional advantage is inexcusable. In my opinion, the sole solution for this malfeasance is unilateral retraction. Res ipsa locitur.
William T Depew BSc MD FRCPC FACG
Queen’s University School of Medicine,
Kingston, Ontario