Open issues for education in radiological research: data integrity, study reproducibility, peer-review, levels of evidence, and cross-fertilization with data scientists

Abstract

We are currently facing extraordinary changes. A harder and harder competition in the field of science is open in each country as well as in continents and worldwide. In this context, what should we teach to young students and doctors? There is a need to look backward and return to "fundamentals", i.e. the deep characteristics that must characterize the research in every field, even in radiology. In this article, we focus on data integrity (including the “declarations” given by the authors who submit a manuscript), reproducibility of study results, and the peer-review process. In addition, we highlight the need of raising the level of evidence of radiological research from the estimation of diagnostic performance to that of diagnostic impact, therapeutic impact, patient outcome, and social impact. Finally, on the emerging topic of radiomics and artificial intelligence, the recommendation is to aim for cross-fertilization with data scientists, possibly involving them in the clinical departments.

We are currently facing extraordinary changes not only in science and biomedicine but in the human society, due to the impact of at least three major drivers: pandemic diseases, technological development (including computer science and artificial intelligence), and loss of confidence in peaceful globalization processes. A harder and harder competition in the field of science is open in each country as well as in continents and worldwide.

In this context, what should we teach to young students and doctors? There is a need to look backward and return to "fundamentals", i.e. the deep characteristics that must characterize the research in every field, even in radiology.

Ethics, of course, represents a primary topic, whit multiple implications, such as the respect for patients (their privacy, as now defined by the European rules of the GDPR and its applications) and the full compliance with the regulations relating to the approvals of the Ethics Committees for all studies including humans, human tissues or cells (or animal models), with whatever design.

However, beside these issues, albeit certainly relevant, we are convinced that we have to educate young people to pay the greatest attention to the integrity and quality of research, focusing on the general method of science. First of all, we must highlight that the study design is the most important issue, absolutely more impacting the quality of research than statistical methods. Even though a statistical plan should be defined at the beginning of a study proposition, data analysis can be always redone from the scratch, but if the data suffer from unrecoverable bias due to a poor study design, it is very difficult to find out solutions [1].

We would like to highlight here the crucial concept of data integrity, respect for the “holiness” of data, when the temptations of the competitive context of bibliography metrics operate, showing disturbing similarities with “likes” on social media. The recent text by Enrico Bucci, Bad Scientists [2], has set the theme starting from the numerous cases of articles, also published in high-impact journals such as Nature and Science, withdrawn after the discovery of the complete fabrication or falsification of data. There was no shortage of dramatic episodes, including the guilt-ridden suicide of the falsifier's mentor (in Japan) and death sentences based on 95% error methods used in forensics (in the USA). Bucci reports worrying data. He cites Daniele Fanelli (Stanford University and CNR Ethics Commission), according to whom 2% of researchers admit fraudulent behavior and 14% have witnessed fabrication or falsification of data [3].

Another very hot topic regards the reproducibility of study results. The attempts to replicate the results of 67 studies related to new drugs according to Prinz et al. [4] were successful in 20 − 25% of cases. According to Begley and Ellis [5], the reproducibility success rate for 53 studies considered fundamental in oncology was even lower (11%). Consider that the number of clinical trials published and available on the PubMed in 2021 alone: 36,273. Of them 28,466 were randomized controlled trials. The verification by replication of these studies is, given these high numbers, is simply… impossible. Moreover, with the increasing volume of scientific papers produced during the COVID-19 outbreak, the possibility to check the published data has become more and more difficult, leading to a retraction and widowing of more than two hundred of articles [6]. To reduce this phenomena, journals’ editors and the publishing staff have to make important efforts to promote and maintain data integrity by following the international publishing standards [7].

One could argue that the peer-review process should protect the community from fraud in research. However, the process of peer-review, firstly adopted by the Royal Society of Philosophical Transactions in 1600 in order to evaluate what has already been discussed [8, 9], is today in difficulty because of being subjective, variable, and often inconsistent [10], unable to discover the fabrication of data, especially if is “well done” [2]. Good news comes from the automatic detection of counterfeit laboratory images and the existence of dedicated websites, such as Retraction Watch [11].

It is necessary to educate young people taking into account these issues; this process must begin as earlier as possible, to spread the passion and the importance of a robust scientific culture and sense of responsibility in the young medical students that approach the course for graduation in medicine.

First, we must pay more attention to the apparently "bureaucratic" aspects of scientific publications because, instead, they constitute the ethical basis: approval by Ethics Committee; definition of the contribution of each author; declaration of funding; declaration of conflicting/competing interests; declaration of overlap with cohorts of patients already subject of previous publications; availability of data (public or on request). This last aspect refers to the interesting perspective of "data sharing", i.e. an open social view of the use of scientific data [12].

Starting from these "fundamentals", many other themes must be discussed, in particular with residents in radiology and young radiology researchers. One is that of the need to raise the level of radiological research objectives from the estimation of diagnostic performance to that of diagnostic impact, therapeutic impact, patient outcome, social impact (typical of population screening) [13]. This means to be able to get higher value from radiological procedures and to increase the role of radiologists in the multidisciplinary teams. In this process, radiologists need to look around in an investigative and innovative way, with openness to new frontiers coming from elsewhere as, for example, that proposed by Tagliafico et al. [14] with the application of blockchain in the radiological research. A parallel topic is the preference to be accorded to prospective, multicenter studies, with a preliminary calculation of power and sample size, possibly including the evaluation of reproducibility of radiological results. The intra-patient design (if possible) should be preferred for the comparison of diagnostic performance, the randomized one for screening and for evaluating patient outcome, as is in interventional radiology.

Finally, on the emerging topic of radiomics and artificial intelligence, the recommendation is to aim for cross-fertilization with data scientists, possibly involving them in the clinical departments. They can understand, for example, that even a good "area under the curve” at receiver operating characteristics analysis may be accepted for publication but useless in certain clinical settings, where different balances between sensitivity and specificity are needed. Having data scientists next to us, we will be able to approach deep learning and convolutional neural networks with less awe and more technical competence [15, 16], for example opening the black box  of their operation that often seems to be stubbornly closed [17].

These are some suggestions to promote a strong next generation of researchers in radiology, being able to shape a brilliant future for our discipline.

Funding

No funding.

Declarations

Conflict of interest

No competing/conflicting interests to be declared.