Abstract
Retractions serve a crucial role in maintaining the integrity and accuracy of scientific literature. There has been growing interest in understanding the patterns behind retractions. This bibliometric study analyzed retracted articles published between 2003 and 2022, indexed by the Science Citation Index Expanded of the Web of Science Core Collection database. A total of 8466 retracted articles were identified, revealing an overall increase up to 2019, followed by a decline. A total of 109 countries contributed to the retracted articles, with China and the United States having the highest absolute numbers. In addition, the articles were published in 2347 different journals, with Tumor Biology recording the largest number of retracted articles. The top 10 most cited retracted articles indicated that data and image integrity issues were the main reasons for retraction. The primary reasons for retractions, identified by linking the retracted articles to the Retraction Watch Database, were data and results issues followed by plagiarism and duplication. In conclusion, the present bibliometric study offered an overview of the status of retracted articles indexed by the Web of Science Core Collection over the past two decades. These findings provide insight into areas where scientific integrity may be compromised and serve as a guide to foster a responsible research environment.
Keywords: Retractions, Retraction of publication, Scientific misconduct, Publishing, Bibliometric analysis, Bibliometrix
1. Introduction
Ensuring the integrity and accuracy of published findings is of paramount importance for maintaining the credibility and reliability of scientific knowledge [1]. Retractions serve as a formal mechanism within the scientific publishing process, designed to correct or remove research articles identified as containing significant flaws, inaccuracies, or instances of misconduct. Retractions uphold the integrity of the scientific literature by addressing issues such as data fabrication, plagiarism, methodological errors, duplicate publication, or ethical breaches.
A retraction may be initiated by the authors of the papers or their institution if they discover errors, inaccuracies, or misconduct in their published work. In such cases, they are expected to contact the journal editor, provide an explanation of the issue, and request a retraction or correction. In addition, when journal editors become aware of potential issues in a published article through notifications from readers, reviewers, or their own detection methods, they can conduct preliminary investigations and contact the authors to address the concerns. If a retraction is deemed necessary, the journal will issue a retraction notice, providing specific reasons for the retraction. The journal's online and print records will be updated to reflect the retraction and ensure that future readers are aware of the article's status [2,3].
While retractions can create substantial citation penalties [4] and may have negative implications for the authors involved, such as reputational damage or loss of credibility [5], they serve to prevent the dissemination of false or misleading information. This can help to avert potential negative impacts on future research, wasted resources, or misguided clinical practices [6]. An analysis of 180 retracted English-language papers published from 2000 to 2010, along with their 851 secondary studies citing them, showed that the retracted papers were cited over 5000 times. These retracted studies put over 428,000 enrolled participants and 79,690 at-risk patients [7]. Studies examining the nature of citations of retracted articles revealed a lack of awareness among investigators, journal editors, and peer reviewers about the perpetuation of retracted work [8,9].
The number of retracted scientific articles has been increasing in recent years. Some authors suggested that this trend indicates a rise in scientific misconduct. A review of 2047 biomedical and life science research articles indexed by PubMed as retracted on May 3, 2012, revealed that 67.4 % of retractions were attributable to misconduct, including fraud or suspected fraud (43.4 %), duplicate publication (14.2 %), and plagiarism (9.8 %) [10]. A comprehensive survey of retracted articles from 42 large bibliographic databases across major scholarly fields showed that the number of articles retracted per year increased by a factor of 19 from 2001 to 2010. The factor decreased to 11.4 when excluding repeat offenders and adjusting for the growth of the published literature [11].
However, other authors offered an alternative interpretation, suggesting that the rise in retractions could also result from growing scientific integrity and increasing efficiency of the detection system rather than a surge in scientific misconduct [6,12]. A comprehensive survey spanning 42 bibliographic databases and publisher websites across various scholarly disciplines identified 4449 scholarly publications retracted between 1928 and 2011. According to the authors, the proportion of published scholarly literature affected by retraction remained relatively small, and most retracted articles did not contain flawed data [11]. Overall, while the increasing number of retracted scientific articles is a concerning trend, it is essential to consider multiple possible explanations for the phenomenon.
There has been a growing interest in understanding the patterns and reasons behind retractions in recent years. This interest is driven by a desire to improve research integrity and the overall quality of scientific literature. Investigations into retractions have been conducted across various fields, including anesthesiology [13], cancer [14], cardiovascular disease [15], dentistry [16], neurosurgery [17], nursing [18,19], obstetrics and gynecology [20], orthopedy [21], and veterinary medicine [22].
In addition to systematic analysis [23], bibliometric methodology has emerged as an important tool to understand further the patterns and trends of scientific research [24]. Bibliometrics, as a field of research, applies quantitative methods to evaluate the impact of scientific publications, such as identifying highly cited papers, influential authors, and top-ranked journals [25]. Several studies have used bibliometric analysis to examine the characteristics of retractions in specific domains, such as COVID-19 [26], oncology [27], rehabilitation [28], and schizophrenia [29]. Nevertheless, there remains a need for a comprehensive analysis of retractions across multiple fields. To address this gap, the present study used bibliometric analysis to investigate the patterns of retractions in scientific research across various disciplines. Retracted articles published between 2003 and 2022 were identified using data from the Science Citation Index Expanded (SCI-Expanded) of the Web of Science (WoS) Core Collection database. Country-level, journal-level, article-level, and keyword co-occurrence analyses were conducted. Moreover, retraction reasons were analyzed by linking retracted articles to the Retraction Watch Database using Digital Object Identifiers (DOIs).
The objectives of this study are as follows: (1) To analyze the trends and patterns of retracted articles in scientific literature from 2003 to 2022, focusing on the distribution of retractions across various disciplines and journals, citation trends, and the geographic distribution; (2) To identify the primary reasons for article retractions during this period, with a particular emphasis on issues related to data integrity, plagiarism, and ethical concerns; (3) To provide insights into the factors contributing to retractions and to offer recommendations for improving research integrity and publication practices.
2. Methods
The SCI-Expanded of the WoS Core Collection database (Clarivate Analytics, Philadelphia, PA, USA) was used to identify articles in this bibliometric analysis (Fig. 1). According to Clarivate, the SCI-Expanded indexes over 9500 of the world's most impactful journals across 178 scientific disciplines, with more than 53 million records dating from 1900 to the present [30]. While Scopus, a major commercial bibliographic database introduced by Elsevier in 2004 [31], is frequently utilized in bibliometric studies [32], our choice of WoS for the current study was driven by its unique inclusion of a dedicated document type for “retracted publication.” This specific feature facilitates the straightforward identification and exclusion of retracted articles, thereby enhancing the reliability and accuracy of our search results.
Fig. 1.
Study flowchart.
The search was conducted on the same day (July 1, 2023) to avoid changes in the content due to updates in the database. For analysis, the document type “retracted publication” was used to identify retracted articles. In WoS, this document type indicates that the article in question has been formally withdrawn. Conversely, the document type “retraction” was not used because it represents merely a formal notice in WoS, which lacks data on citations and references.
The time frame of the search was limited to 2003 to 2022 to provide a two-decade snapshot of retractions in scientific literature. This allows for the identification of patterns in the retraction landscape while maintaining relevance to the current research environment. To ensure proper interpretation of the results, the publication language was restricted to English using the WoS field tag LA.
To conduct the bibliometric analysis, we utilized Bibliometrix 4.1 (Naples, Italy) [33] for country-level and journal-level analyses, and VOSviewer version 1.6.19 [34] for co-occurrence analysis of article keywords. Our keyword analysis used KeyWords Plus, which is a feature in WoS that augments author-generated keywords for a given article by adding additional index terms, thereby revealing more relevant terms potentially overlooked by the authors [35]. To maintain readability, only the top 200 terms were included to generate the co-occurrence network. In the network figure, the size of the circle reflects the number of articles in which the term occurs. The proximity between two linked terms indicates the relatedness of the terms based on their number of co-occurrences in documents.
To explore the reasons for retraction, the articles identified in the WoS were matched with the records in the Retraction Watch Database [36] by linking with DOIs. In the Retraction Watch Database, a total of 107 reasons for retraction were listed [37]. To improve interpretability, these reasons were reclassified into 12 categories: (1) Data and Results Issues, (2) Plagiarism and Duplication, (3) Investigations and Findings, (4) Authorship and Ethical Concerns, (5) Misconduct and Fraud, (6) Image Manipulation and Fabrication, (7) Peer Review and Editorial Issues, (8) Withdrawal and Retraction Notices, (9) Institutional and Policy Issues, (10) Complaints and Objections, (11) Miscellaneous, and (12) Procedural and Legal Issues (Table S1). In addition, as each retracted article can have multiple reasons for retraction, with no specific order of importance, the denominator for calculating the percentage of each retraction reason is the total number of retraction reasons rather than the number of articles.
To assess whether specific subject categories are more susceptible to certain types of retraction reasons, we focused on the three most prevalent retraction reasons—“Data and Results Issues,” “Plagiarism and Duplication,” and “Investigations and Findings” —along with their combinations. A contingency table was constructed to cross-tabulate the reasons for retraction with the main subject categories of the retracted articles. The subject category refers to the field of study most likely to be referenced or search for the information provided in the article. While many articles may overlap across different subject categories, only the most relevant subject category, as identified by the Retraction Watch Database, was assigned to each article in the present study. To determine the association between retraction reasons and subject categories, a Chi-square test was performed. Furthermore, post-hoc comparisons of column proportions were conducted using the Bonferroni correction to identify statistically significant differences between subject categories.
3. Results
During the 20-year period from 2003 to 2022, 8466 retracted articles were identified from the SCI-Expanded of the WoS Core Collection database. Fig. 2 shows the annual trend of retracted articles over this time. The trend in the number of retracted articles exhibited apparent fluctuations from 2003 to 2022. Initially, there was a consistent increase, rising from 137 articles in 2003 to a peak of 787 in 2019. However, after reaching the peak in 2019, the trend sharply reversed, with the number of retracted articles falling to 239 by 2022.
Fig. 2.
Plot of annual number of retracted articles from 2003 to 2022.
A total of 109 countries contributed to the articles analyzed in the present study. Table 1 shows the top 10 countries, based on the affiliation of the corresponding author, of retracted articles. China (n = 3,669, 43.3 %) represented the largest share of retracted articles, followed by the United States (n = 1,246, 14.7 %), and India (n = 529, 6.2 %). The United States had the highest average citation number (51.9), followed by the United Kingdom (51.7). In addition, China showed the highest ratio of multiple-country publications to total publications (9.0 %), indicating a high inter-country collaboration. In contrast, India showed the lowest ratio at 1.2 %.
Table 1.
The top 10 countries, based on the affiliation of the corresponding author, of retracted articles from 2003 to 2022 (N = 8466).
| Rank | Country | Number of Articles (%) | Average Article Citations |
MCP/Total Ratio (%) |
|---|---|---|---|---|
| 1 | China | 3669 (43.3) | 16.7 | 9.0 |
| 2 | The United States | 1246 (14.7) | 51.9 | 2.2 |
| 3 | India | 529 (6.2) | 22.7 | 1.2 |
| 4 | Japan | 440 (5.2) | 27.0 | 1.4 |
| 5 | Iran | 374 (4.4) | 18.8 | 2.0 |
| 6 | Korea | 232 (2.7) | 19.2 | 1.6 |
| 7 | Italy | 174 (2.1) | 23.6 | 3.0 |
| 8 | The United Kingdom | 171 (2.0) | 51.7 | 3.9 |
| 9 | Germany | 160 (1.9) | 41.0 | 3.3 |
| 10 | Egypt | 120 (1.4) | 23.8 | 2.6 |
MCP: multiple country publication.
The retrieved articles were published in 2347 different journals. A total of 1137 retracted articles (13.4 %) were published in the top 10 journals. Tumor Biology, the journal with the largest number of articles (n = 153), had a Journal Impact Factor of 3.7 in 2016 but was no longer indexed by WoS afterward. Among the top 10 journals, the Journal of Clinical Anesthesia had the highest Journal Impact Factor of 6.7. Regarding WoS categories, oncology was the most prevalent, and most journals belonged to either quartile 2 or 3 (Table 2).
Table 2.
The top 10 journals based on the number of retracted articles from 2003 to 2022 (N = 8466).
| Rank | Journal | Number of Articles (%) | Journal Impact Factora | Web of Science Category [Quartile] |
|---|---|---|---|---|
| 1 | Tumor Biology | 153 (1.8) | 3.7 (last indexed in 2016) | Oncology [2] |
| 2 | Journal of Cellular Biochemistry | 142 (1.7) | 4.0 | Biochemistry & Molecular Biology [2] Cell Biology [3] |
| 3 | Journal of Biological Chemistry | 133 (1.6) | 4.8 | Biochemistry & Molecular Biology [2] |
| 4 | RSC Advances | 128 (1.5) | 3.9 | Chemistry, multidisciplinary [2] |
| 5 | European Review for Medical and Pharmacological Sciences | 124 (1.5) | 3.3 | Pharmacology & Pharmacy [3] |
| 6 | PLoS One | 117 (1.4) | 3.7 | Multidisciplinary Sciences [2] |
| 7 | Molecular Medicine Reports | 102 (1.2) | 3.4 | Oncology [3] Medicine, Research & Experimental [3] |
| 8 | Oncology Reports | 83 (1.0) | 4.2 | Oncology [2] |
| 9 | Cluster Computing | 80 (0.9) | 4.4 | Computer Science, Theory & Methods [2] Computer Science, Information Systems [1] |
| 10 | Journal of Clinical Anesthesia | 75 (0.9) | 6.7 | Anesthesiology [1] |
Journal Impact Factors were obtained from the 2022 Journal Citation Reports, unless otherwise indicated.
Fig. 3 represents the number of retracted articles for the top 10 journals over time. The lines showed a general increase in the number of articles published after 2011, with occasional fluctuations. Tumor Biology experienced a sharp rise in 2012–2014. The curve discontinued after 2016 as the journal was delisted by WoS in July 2017 [38]. Journal of Cellular Biochemistry, European Review for Medical and Pharmacological Sciences, RSC Advances, and Cluster Computing also exhibited sharp rises in 2017, followed by decreases. Journal of Biological Chemistry displayed a steady number of retracted articles, while other journals showed an increase from 2014 to 2018, then declined.
Fig. 3.
Plot of annual number of retracted articles in the top 10 journals with the highest num-ber of retracted articles from 2003 to 2022 (N = 8466).
Table 3 showcases the top 10 leading retracted articles based on the total number of citations during the study period. The top article, a large multicenter trial in Spain investigating the protective effect of a Mediterranean diet supplemented with extra-virgin olive oil or nuts, was first published in 2013 in the New England Journal of Medicine [39] and received a total of 2363 citations. However, due to errors in methods, the article was retracted [40] and republished in 2018 [41] after excluding study sites that violated randomization principles. BMC Evolutionary Biology retracted the article by Jobb et al. [42] due to the decision by the corresponding author that the software could not be made available publicly, which breached the journal's editorial policy on software availability [43]. The remaining eight articles were retracted due to either data or image integrity issues.
Table 3.
The top 10 retracted articles based on the number of citations from 2003 to 2022 (N = 8466).
| Rank | First Author (No. of Total Authors) | Title | Journal | Year of Publication | Total Citationsa |
Year of Retraction | Reason for retraction |
|---|---|---|---|---|---|---|---|
| 1 | Estruch R (18 & study team) | Primary prevention of cardiovascular disease with a Mediterranean diet | New England Journal of Medicine | 2013 | 2362 | 2013 | Errors in methods |
| 2 | Fukuhara A (22) | Visfatin: a protein secreted by visceral fat that mimics the effects of insulin | Science | 2005 | 1490 | 2007 | Data integrity issue, result issue |
| 3 | Voinnet O (4) | An enhanced transient expression system in plants based on suppression of gene silencing by the p19 protein of tomato bushy stunt virus | Plant Journal | 2003 | 1336 | 2015 | Image integrity issue |
| 4 | Erler JT (9) | Lysyl oxidase is essential for hypoxia-induced metastasis | Nature | 2006 | 1094 | 2020 | Image integrity issue |
| 5 | Jobb G (3) | TREEFINDER: a powerful graphical analysis environment for molecular phylogenetics | BMC Evolutionary Biology | 2004 | 1007 | 2015 | Could not make software available |
| 6 | Bolli R (20) | Cardiac stem cells in patients with ischaemic cardiomyopathy (SCIPIO): initial results of a randomised phase 1 trial | Lancet | 2011 | 1000 | 2019 | Data integrity issue |
| 7 | Mehra MR (5) | Cardiovascular disease, drug therapy, and mortality in Covid-19 | New England Journal of Medicine | 2020 | 854 | 2020 | Data integrity issue |
| 8 | Raj L (14) | Selective killing of cancer cells by a small molecule targeting the stress response to ROS | Nature | 2011 | 849 | 2018 | Image integrity issue |
| 9 | Rubio D (7) | Spontaneous human adult stem cell transformation | Cancer Research | 2005 | 803 | 2010 | Data integrity issue |
| 10 | Nakao N (6) | Combination treatment of angiotensin-II receptor blocker and angiotensin-converting-enzyme inhibitor in non-diabetic renal disease (COOPERATE): a randomised controlled trial | Lancet | 2003 | 742 | 2009 | Data integrity issue |
Total citation included citation before and after retraction.
The retracted articles belonged to 204 subject categories of WoS. Table 4 lists the 20 most frequently occurring subject categories of WoS in retracted articles from 2003 to 2022. Oncology (12.3 %), biochemistry & molecular biology (10.0 %), cell biology (8.5 %), medicine research & experimental (7.3 %), and pharmacology & pharmacy (6.7 %) represented the top five categories, accounting for a total of 3798 articles (44.9 %) of the 8466 articles.
Table 4.
The top 20 subject categories of Web of Science with retracted articles from 2003 to 2022 (N = 8466).
| Rank | Subject categories | n (%) | Cumulative n (%) |
|---|---|---|---|
| 1 | Oncology | 1044 (12.3) | 1044 (12.3) |
| 2 | Biochemistry & Molecular Biology | 848 (10.0) | 1892 (22.3) |
| 3 | Cell Biology | 717 (8.5) | 2609 (30.8) |
| 4 | Medicine, Research & Experimental | 619 (7.3) | 3228 (38.1) |
| 5 | Pharmacology & Pharmacy | 570 (6.7) | 3798 (44.9) |
| 6 | Chemistry, Multidisciplinary | 427 (5.0) | 4225 (49.9) |
| 7 | Materials Science, Multidisciplinary | 425 (5.0) | 4650 (54.9) |
| 8 | Multidisciplinary Sciences | 343 (4.1) | 4993 (59.0) |
| 9 | Engineering, Electrical & Electronic | 337 (4.0) | 5330 (63.0) |
| 10 | Neurosciences | 266 (3.1) | 5596 (66.1) |
| 11 | Physics, Applied | 243 (2.9) | 5839 (69.0) |
| 12 | Chemistry, Physical | 240 (2.8) | 6079 (71.8) |
| 13 | Biotechnology & Applied Microbiology | 225 (2.7) | 6304 (74.5) |
| 14 | Immunology | 200 (2.4) | 6504 (76.8) |
| 15 | Computer Science, Information Systems | 187 (2.2) | 6691 (79.0) |
| 16 | Nanoscience & Nanotechnology | 177 (2.1) | 6868 (81.1) |
| 17 | Medicine, General & Internal | 168 (2.0) | 7036 (83.1) |
| 18 | Surgery | 163 (1.9) | 7199 (85.0) |
| 19 | Telecommunications | 161 (1.9) | 7360 (86.9) |
| 20 | Environmental Sciences | 155 (1.8) | 7515 (88.8) |
The co-occurrence analysis conducted using VOSviewer revealed distinct thematic clusters, as depicted in Fig. 4. Four interconnected clusters were observed. The green cluster focused on expression with terms related to cancer research, such as proliferation, invasion, metastasis, migration, and progression. The red cluster is composed of terms associated with molecular studies, such as apoptosis, activation, inflammation, inhibition, and pathway. The yellow cluster focused on terms related to pathogenesis, such as disease, therapy, survival, management, risk, and epidemiology. The blue cluster presents a broader array of terms, such as model, prediction, identification, mechanism, and protein.
Fig. 4.
A co-occurrence network of keywords in retracted articles from 2003 to 2022 (N = 8466).
To examine the reasons for retractions, the 8164 articles identified using the WoS were linked to records in the Retraction Watch Database based on DOI of the articles. Of the 7375 articles (90.3 %) successfully linked, the total count of the reasons was 21,422 cases were obtained because each article can be assigned with multiple reasons according to the Retraction Watch Database (Table 5). The most prevalent issue, accounting for 28.8 % (6160 cases), is related to data and results issues. Plagiarism and duplication follow with 19.1 % (4085 cases), and investigations and findings constitute 16.8 % (3588 cases). Authorship and ethical concerns represent 9.0 % (1918 cases), while misconduct and fraud account for 6.0 % (1288 cases). Image manipulation and fabrication are noted in 5.1 % (1092 cases) of the issues. Peer review and editorial issues, withdrawal and retraction notices, and institutional and policy issues make up 4.6 % (985 cases), 3.6 % (763 cases), and 3.0 % (643 cases) respectively. Complaints and objections constitute 2.0 % (418 cases), miscellaneous issues are 1.9 % (412 cases), and procedural and legal issues are the least frequent at 0.3 % (70 cases).
Table 5.
Distribution of retraction reasons of retracted articles from 2003 to 2022.
| Retraction Reason | Count (%) |
|---|---|
| Data and Results Issues | 6160 (28.8) |
| Plagiarism and Duplication | 4085 (19.1) |
| Investigations and Findings | 3588 (16.8) |
| Authorship and Ethical Concerns | 1918 (9.0) |
| Misconduct and Fraud | 1288 (6.0) |
| Image Manipulation and Fabrication | 1092 (5.1) |
| Peer Review and Editorial Issues | 985 (4.6) |
| Withdrawal and Retraction Notices | 763 (3.6) |
| Institutional and Policy Issues | 643 (3.0) |
| Complaints and Objections | 418 (2.0) |
| Miscellaneous | 412 (1.9) |
| Procedural and Legal Issues | 70 (0.3) |
| Total | 21422 (100) |
Of the 8466 articles identified in the Web of Science, 7375 were successfully linked with records from the Retraction Watch Database. As each article may have multiple retraction reasons, a total count of 21,422 for the various reasons was documented.
The cross-tabulation of the top three retraction reasons and main subject categories for articles retracted between 2003 and 2022 revealed distinct patterns in the distribution of retraction causes across different fields, with an overall Chi-square p-value of <0.001. A total of 6883 articles were analyzed, excluding those with missing data in subject categories. “Data and results issues” were significantly more prevalent in Basic Life Sciences (24.8 %) and Health Sciences (24.4 %) compared to Business and Technology (18.0 %). “Plagiarism and duplication” were significantly more common in Physical Sciences (35.9 %) and the "Other" category (34.6 %), which includes humanities, social sciences, and environmental sciences, compared to Basic Life Sciences (18.1 %) and Business and Technology (26.4 %). Retractions due to “Investigations and findings” were most prevalent in Health Sciences (22.7 %), significantly exceeding the proportions in Business and Technology (7.2 %) and Physical Sciences (7.2 %). Combinations of any of the top three retraction reasons were prevalent in Business and Technology (36.6 %) and Basic Life Sciences (35.6 %) compared to other categories. Lastly, other reasons and their combinations were more common in the "Other" category (18.5 %), Physical Sciences (17.6 %), and Health Sciences (14.6 %) compared to other categories (Table 6).
Table 6.
Cross-tabulation of main retraction reasons and subject categories of retracted articles from 2003 to 2022.
| Reason |
Subject category, n (column %) |
|||||
|---|---|---|---|---|---|---|
| Business and Technology | Basic Life Sciences | Health Sciences | Physical Sciences | Other | Row Total |
|
| Data and Results Issues | 110a | 1034b (24.8) | 197b (24.4) | 243a,b (20.9) | 29a,b (22.3) | 1613 (23.4) |
| (18.0) | ||||||
| Plagiarism and Duplication | 161a,b | 756c | 186b | 417d | 45a,d | 1565 |
| (26.4) | (18.1) | (23.0) | (35.9) | (34.6) | (22.7) | |
| Investigations and Findings | 44a | 502b | 184c | 84a | 11a,b | 825 |
| (7.2) | (12.0) | (22.7) | (7.2) | (8.5) | (12.0) | |
| Combinations of the above three reasons | 223a | 1487a | 124b | 213b | 21b | 2068 |
| (36.6) | (35.6) | (15.3) | (18.3) | (16.2) | (30.0) | |
| Other reasons and combinationsa | 72a,b | 394b | 118a,c | 204c | 24a,c | 812 |
| (11.8) | (9.4) | (14.6) | (17.6) | (18.5) | (11.8) | |
| Column Total | 610 | 4173 | 809 | 1161 | 130 | 6883 |
| (100) | (100) | (100) | (100) | (100) | (100) | |
Each subscript letter denotes a subset of subject category whose column proportions do not differ significantly from each other at the 0.05 level, adjusted using the Bonferroni correction.
Other subject categories included humanities (n = 6), social sciences (n = 14), and environmental sciences (n = 110).
4. Discussion
Overall, this bibliometric study analyzed retracted articles published between 2003 and 2022 and indexed by the SCI-Expanded of WoS Core Collection database. The results revealed the patterns and implications of article retractions, contributing to the theoretical understanding of research integrity and publication ethics. The study highlighted the significance of regional and institutional factors, as well as disciplinary vulnerabilities, indicating that systemic pressures and the complexity of research methodologies in certain fields are closely linked to the incidence of retractions. Regarding practical implications, this study showed the importance of strengthening peer review processes, as illustrated by the Tumor Biology case, and suggests the need of new technologies to detect fraudulent reviews. Moreover, standardized practices for linking retraction notices directly to the retracted articles in databases should be implemented to mitigate the spread of invalidated research findings. In addition, the correlation between competitive pressures in specific fields and retractions signifies the need to address systemic issues within the scientific community. Targeted educational programs in research ethics for fields more prone to retractions is recommended.
During the 20-year period, a total of 8466 retracted articles were identified. There has been an increase in the number of retracted articles up to 2019. While the increase could correspond to the growth in the total number of publications during the same period, it does not explain the drop after 2019. The decline after 2019 might be attributed to the time delay required for retractions to occur, considering that retractions may take several years. Based on the top 10 cited retracted articles, the period before retraction ranged from less than a year to 14 years, with a median of 6.5 years. A study of retracted articles from 1960 to 2020 indicated that at least half of the retractions occurred no more than three years after publication [9]. Therefore, a drop in the number of retracted articles after 2019 might be attributed to the time delay required for retractions to take place. However, it is also possible that the scientific community has become more stringent in peer-review processes as well as the recent initiatives to increase transparency [44] might have contributed to this effect. Moreover, the introduction of new tools for the detection of plagiarism might have enabled earlier detection and reduced fraudulent activities leading to retractions later on.
The present study identified China and the United States as the countries with the largest number of retracted articles, accounting for 58 % of the 8466 retracted articles. Similar findings have been reported in other studies that analyzed different periods and databases. A study analyzed retracted articles indexed by WoS between 1981 and 2020 revealed that authors from China have higher retractions (25.7 %), followed by the United States (16.1 %), and India (5.3 %) [45]. Another PubMed-based study on articles published between 2008 and 2012 showed that the United States retracted the most articles, whereas China retracted the most articles for plagiarism and duplicate publication [46]. An early study based on 834 WoS retractions also found that the number of retracted papers from Chinese authors had increased between 1997 and 2016, with approximately three-quarters of which were triggered by fake peer review, plagiarism, or falsification [47]. Nevertheless, these results should be interpreted within the context of the size of the body of literature. Both China and the United States are among the largest producers of scientific research globally. Their substantial publication volume might contribute to a ‘‘publication inflation’’ that partially explains the higher number of retractions [11]. When evaluating with a different index, such as retractions per paper published, Iran takes the top spot with 14 retractions per 10,000 papers, followed by Romania with 10.4 retractions per 10,000 papers. China became the seventh spot with five retractions per 10,000 papers [48]. It should be noted that while focusing on specific countries provides significant insights, understanding retraction patterns requires a broader, global perspective.
Tumor Biology, a medical journal covering clinical and experimental oncology, had the highest number of retracted articles. Starting in 2012, the journal showed a sharp rise in the number of articles that were subsequently retracted. In 2017, its publisher Springer retracted 107 articles after discovering they were accepted based on fake peer reviews. As a result of the mass retraction, the journal was delisted by WoS starting July 2017 [38]. The incident with Tumor Biology underscores the importance of authentic peer review and indicates the need for rigorous measures to prevent potential lapses in the process. To help editors choose credible reviewers and close peer review loopholes, it is imperative to create an expert database with verified information on experts' credentials. In addition, new technologies should be explored to assist the detection of suspiciously positive reviews [49].
Findings from top 10 cited retracted articles indicated that most were retracted due to image and data integrity issues. The former often involved duplication of images, whereas the latter was often linked to the implausibility of data, where the results were statistically or scientifically inconsistent with previous findings. Articles were retracted because the authors failed to provide raw data when requested, which can be due to poor data management practices or intentional concealment of fraudulent or flawed data. Moreover, the high number of citations received by these articles is another important concern. Previous studies have demonstrated that articles continued to be cited years after retraction [50]. One contributing factor to this issue is that retraction notices often lack clear indication and linkage to the retracted articles [9]. Putting flawed data in the scientific literature can mislead both fellow scientists and the broader community. As a result, the scientific community needs to enhance the clarity and transparency of retraction notices and uphold the integrity of the scientific literature.
Findings from the analysis of WoS subject categories and co-occurrence network analysis of keywords showed that articles in the field of oncology, biochemistry, molecular biology, and cell biology appeared to be particularly prone to retractions. This pattern might reflect underlying challenges and complexities within these disciplines or broader systemic issues within the scientific community. Studies in these fields are characterized by intricate experimental designs and advanced technologies, which might increase the likelihood of errors. The competitive nature and pressure to produce novel findings in these fields may also inadvertently encourage behaviors leading to retractions. A systematic survey of 571 retracted cancer publications revealed that academic misconduct in the form of plagiarism, duplicate publication, and fraud accounts for 61 % of retracted cancer publications [14]. A recent bibliometric study on retracted publications in oncology used Retraction Watch Database to explore the reasons for retractions. The top three reasons included investigation by journal/publisher (30.3 %), duplication of image (24.9 %), and fake peer review (22.6 %), accounting for a combined total of 77.8 % of the retractions analyzed [51]. In addition, the prominence of these fields may lead to higher levels of scrutiny by peers, leading to more retractions being detected and acted upon. Therefore, the findings might reflect a combination of underlying complexities and systemic issues within these disciplines.
The analysis of retraction reasons showed that the top two most prevalent reason for retractions are “data and results issues” and “plagiarism and duplication”, accounting for 48 % of the cases. Approximately 29 % of the retraction reasons are related to the category “data and results issues”. This finding indicates that important role of data integrity in scholarly research. Issues such as unreliable or error in data and results can adversely affect the credibility of research findings and are critical areas for scrutiny and improvement. The category “plagiarism and duplication” was the second most common reasons for retraction. Despite the widespread availability of plagiarism detection tools, their current application is insufficient to address the problem. Journals need to adopt a more vigilant approach to plagiarism detection at multiple stages of the publication process, including initial submission, post-peer review, and pre-publication to uphold the standards of academic publishing and ensure the integrity of scholarly research.
The cross-tabulation of subject categories and retraction reasons revealed distinct patterns that may be attributed to the nature and demands of the respective disciplines. “Data and results issues” were more prevalent in Basic Life Sciences and Health Sciences. This trend possibly arises from the complex and often intricate methodologies required in these fields, where experimental design, data collection, and analysis are highly susceptible to error, misinterpretation, or even manipulation. In contrast, “plagiarism and duplication” are more common in the Physical Sciences. This observation suggests variations in ethical standards across disciplines, where the rigor of plagiarism detection and enforcement might differ. The structure of research in Physical Sciences, which often involves collaborative and cumulative work, might also make it easier for such ethical breaches to occur unnoticed. In addition, the higher rate of retractions observed in Health Sciences may reflect the intense scrutiny and rigorous investigative processes applied to research that has direct implications for public health.
This study has some limitations worth mentioning. First, the articles analyzed in this study were retrieved from the WoS. While it is well-known that various databases have different strengths and weaknesses [52], WoS was chosen for this study because it contains citation metadata, particularly the availability of a dedicated document type for “retracted publication.” Second, an author-level analysis was not conducted due to the significant issue of author name ambiguity, particularly the Romanized names of Chinese authors [53]. Third, 9.7 % of the retracted articles identified in the WoS could not be linked to the Retraction Watch Database using DOI. The absence of these articles may potentially affect the magnitude of the counts. However, given the substantial difference in the frequency of the top three retraction reasons, it is unlikely that this limitation will alter their ranking.
5. Conclusion
Retractions play a vital role in maintaining research integrity by ensuring the accuracy of the scientific record, promoting transparency and accountability, deterring misconduct, and fostering a culture of continuous improvement within the scientific community. Our bibliometric analysis of 8466 retracted articles from 2003 to 2022, indexed by the SCI-Expanded of the WoS Core Collection, offered an overview of the current state of retractions and reasons for retractions.
The study reveals a notable increase in retractions until 2019, followed by a decline, indicating evolving awareness and response to scientific misconduct and errors in the research community. China and the United States were found to lead in the number of retractions, reflecting their large contributions to global scientific output. Specific journals and subject areas, particularly oncology, biochemistry, and cell biology, were more prone to retractions, highlighting the vulnerability of these fields.
Our study emphasizes the importance of robust peer review processes and the use of advanced tools for detecting misconduct. The significant impact of retracted articles, evidenced by their high citation counts even after retraction, suggests the need for better dissemination of retraction notices and greater transparency in the retraction process. Promoting a culture of integrity and accountability within the scientific community can reduce the adverse effects of retractions and enhance the reliability of published research.
Our findings provide a basis for developing targeted educational programs and policies to improve research ethics. As the scientific landscape evolves, maintaining vigilance and commitment to ethical standards is crucial to ensure the progression of credible scientific literature.
Funding
This work was supported by Tzu Chi University (formerly Tzu Chi University of Science and Technology, TCCT-1111A08).
Data availability statement
Data will be made available on request.
CRediT authorship contribution statement
Malcolm Koo: Writing – review & editing, Writing – original draft, Methodology, Funding acquisition, Formal analysis, Conceptualization. Shih-Chun Lin: Writing – review & editing, Writing – original draft, Conceptualization.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.heliyon.2024.e38620.
Appendix A. Supplementary data
The following is/are the supplementary data to this article:
References
- 1.Resnik D.B., Shamoo A.E. The Singapore statement on research integrity. Account. Res. 2011;18:71–75. doi: 10.1080/08989621.2011.557296. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Atlas M.C. Retraction policies of high-impact biomedical journals. J. Med. Libr. Assoc. 2004;92:242–250. [PMC free article] [PubMed] [Google Scholar]
- 3.COPE. Retraction guidelines. (Retrieved September 9, 2023) https://publicationethics.org/files/retraction-guidelines.pdf.
- 4.Lu S.F., Jin G.Z., Uzzi B., Jones B. The retraction penalty: evidence from the web of science. Sci. Rep. 2013;3:3146. doi: 10.1038/srep03146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Stern A.M., Casadevall A., Steen R.G., Fang F.C. Financial costs and personal consequences of research misconduct resulting in retracted publications. Elife. 2014;3 doi: 10.7554/eLife.02956. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Fanelli D. Why growing retractions are (mostly) a good sign. PLoS Med. 2013;10 doi: 10.1371/journal.pmed.1001563. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Steen R.G. Retractions in the medical literature: how many patients are put at risk by flawed research? J. Med. Ethics. 2011;37:688–692. doi: 10.1136/jme.2011.043133. [DOI] [PubMed] [Google Scholar]
- 8.Hamilton D.G. Continued citation of retracted radiation oncology literature-do we have a problem? Int. J. Radiat. Oncol. Biol. Phys. 2019;103:1036–1042. doi: 10.1016/j.ijrobp.2018.11.014. [DOI] [PubMed] [Google Scholar]
- 9.Hsiao T.K., Schneider J. Continued use of retracted papers: temporal trends in citations and (lack of) awareness of retractions shown in citation contexts in biomedicine. Quant. Sci. Stud. 2022;2:1144–1169. doi: 10.1162/qss_a_00155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Fang F.C., Steen R.G., Casadevall A. Misconduct accounts for the majority of retracted scientific publications. Proc. Natl. Acad. Sci. U. S. A. 2012;109:17028–17033. doi: 10.1073/pnas.1212247109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Grieneisen M.L., Zhang M. A comprehensive survey of retracted articles from the scholarly literature. PLoS One. 2012;7 doi: 10.1371/journal.pone.0044118. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Steen R.G., Casadevall A., Fang F.C. Why has the number of scientific retractions increased? PLoS One. 2013;8 doi: 10.1371/journal.pone.0068397. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Nair S., Yean C., Yoo J., Leff J., Delphin E., Adams D.C. Reasons for article retraction in anesthesiology: a comprehensive analysis. Can. J. Anaesth. 2020;67:57–63. doi: 10.1007/s12630-019-01508-3. [DOI] [PubMed] [Google Scholar]
- 14.Bozzo A., Bali K., Evaniew N., Ghert M. Retractions in cancer research: a systematic survey. Res. Integr. Peer Rev. 2017;2:5. doi: 10.1186/s41073-017-0031-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Audisio K., Robinson N.B., Soletti G.J., Cancelli G., Dimagli A., Spadaccio C., Olaria R.P., Chadow D., Rahouma M., Demetres M., Tam D.Y., Benedetto U., Girardi L.N., Kurlansky P., Fremes S.E., Gaudino M. A survey of retractions in the cardiovascular literature. Int. J. Cardiol. 2022;349:109–114. doi: 10.1016/j.ijcard.2021.12.021. [DOI] [PubMed] [Google Scholar]
- 16.Faggion C.M., Jr., Ware R.S., Bakas N., Wasiak J. An analysis of retractions of dental publications. J. Dent. 2018;79:19–23. doi: 10.1016/j.jdent.2018.09.002. [DOI] [PubMed] [Google Scholar]
- 17.Wang J., Ku J.C., Alotaibi N.M., Rutka J.T. Retraction of neurosurgical publications: a systematic review. World Neurosurg. 2017;103:809–814.e1. doi: 10.1016/j.wneu.2017.04.014. [DOI] [PubMed] [Google Scholar]
- 18.Al-Ghareeb A., Hillel S., McKenna L., Cleary M., Visentin D., Jones M., Bressington D., Gray R. Retraction of publications in nursing and midwifery research: a systematic review. Int. J. Nurs. Stud. 2018;81:8–13. doi: 10.1016/j.ijnurstu.2018.01.013. [DOI] [PubMed] [Google Scholar]
- 19.Joaquim S., Longhini J., Palese A. What can we learn from retracted studies in the nursing field in the last 20 years? Findings from a scoping review. Acta Biomed. 2022;93 doi: 10.23750/abm.v93iS2.12954. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bennett C., Chambers L.M., Al-Hafez L., Michener C.M., Falcone T., Yao M., Berghella V. Retracted articles in the obstetrics literature: lessons from the past to change the future. Am. J. Obstet. Gynecol. MFM. 2020;2 doi: 10.1016/j.ajogmf.2020.100201. [DOI] [PubMed] [Google Scholar]
- 21.Yan J., MacDonald A., Baisi L.P., Evaniew N., Bhandari M., Ghert M. Retractions in orthopaedic research: a systematic review. Bone Joint Res. 2016;5:263–268. doi: 10.1302/2046-3758.56.BJR-2016-0047. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Christopher M.M. Comprehensive analysis of retracted journal articles in the field of veterinary medicine and animal health. BMC Vet. Res. 2022;18:73. doi: 10.1186/s12917-022-03167-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Hwang S.Y., Yon D.K., Lee S.W., Kim M.S., Kim J.Y., Smith L., Koyanagi A., Solmi M., Carvalho A.F., Kim E., Shin J.I., Ioannidis J.P.A. Causes for retraction in the biomedical literature: a systematic review of studies of retraction notices. J. Korean Med. Sci. 2023;38:e333. doi: 10.3346/jkms.2023.38.e333. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Donthu D., Kumar S., Mukherjee D., Pandey N., Lim W.M. How to conduct a bibliometric analysis: an overview and guidelines. J. Bus. Res. 2021;133:285–296. [Google Scholar]
- 25.Ninkov A., Frank J.R., Maggio L.A. Bibliometrics: methods for studying academic publishing. Perspect. Med. Educ. 2022;11:173–176. doi: 10.1007/s40037-021-00695-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Khan H., Gupta P., Zimba O., Gupta L. Bibliometric and altmetric analysis of retracted articles on COVID-19. J. Korean Med. Sci. 2022;37:e44. doi: 10.3346/jkms.2022.37.e44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Pantziarka P., Meheus L. Journal retractions in oncology: a bibliometric study. Future Oncol. 2019;15:3597–3608. doi: 10.2217/fon-2019-0233. [DOI] [PubMed] [Google Scholar]
- 28.Bordino M., Ravizzotti E., Vercelli S. Retracted articles in rehabilitation: just the tip of the iceberg? A bibliometric analysis. Arch. Physiother. 2020;10:21. doi: 10.1186/s40945-020-00092-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Chen P., Li X.H., Su Z., Tang Y.L., Ma Y., Ng C.H., Xiang Y.T. Characteristics of global retractions of schizophrenia-related publications: a bibliometric analysis. Front. Psychiatry. 2022;13 doi: 10.3389/fpsyt.2022.937330. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Clarivate. Web of science: Science citation index expanded. (Retrieved September 9, 2023) https://clarivate.com/webofsciencegroup/solutions/webofscience-scie/.
- 31.Baas J., Schotten M., Plume A., Côté G., Karimi R. Scopus as a curated, high-quality bibliometric data source for academic research in quantitative science studies. Quant. Sci. Stud. 2020;1:377–386. [Google Scholar]
- 32.Pranckute R. vol. 9. 2021. p. 12. (Web of Science (WoS) and Scopus: the Titans of Bibliographic Information in Today's Academic World). Publications. 2021. [Google Scholar]
- 33.Aria M., Cuccurullo C. Bibliometrix: an r-tool for comprehensive science mapping analysis. J. Informetr. 2017;11:959–975. [Google Scholar]
- 34.van Eck N.J., Waltman L. Software survey: vosviewer, a computer program for bibliometric mapping. Scientometrics. 2010;84:523–538. doi: 10.1007/s11192-009-0146-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Zhang J., Yu Q., Zheng F., Long C., Lu Z., Duan Z. Comparing keywords plus of WOS and author keywords: a case study of patient adherence research. J. Assoc. Inf. Sci. Technol. 2015;67:967–972. [Google Scholar]
- 36.Retraction Watch Database. (Retrieved May 20, 2024)http://retractiondatabase.org/.
- 37.Retraction Watch Database. User Guide Appendix B: Reasons. (Retrieved May 20, 2024) https://retractionwatch.com/retraction-watch-database-user-guide/retraction-watch-database-user-guide-appendix-b-reasons/.
- 38.Watch R. When a journal retracts 107 papers for fake reviews, it pays a price.(Retrieved May 20, 2024) https://retractionwatch.com/2017/08/16/journal-retracts-107-papers-fake-reviews-pays-price/.
- 39.Estruch R., Ros E., Salas-Salvado J., Covas M.I., Corella D., Aros F., Gomez-Gracia E., Ruiz-Gutierrez V., Fiol M., Lapetra J., Lamuela-Raventos R.M., Serra-Majem L., Pintó X., Basora J., Muñoz M.A., Sorlí J.V., Martínez J.A., Martínez-González M.A., PREDIMED Study Investigators Primary prevention of cardiovascular disease with a Mediterranean diet. N. Engl. J. Med. 2013;368:1279–1290. [Google Scholar]
- 40.Estruch R., Ros E., Salas-Salvado J., Covas M.I., Corella D., Aros F., Gomez-Gracia E., Ruiz-Gutierrez V., Fiol M., Lapetra J., Lamuela-Raventos R.M., Serra-Majem L., Pintó X., Basora J., Muñoz M.A., Sorlí J.V., Martínez J.A., Martínez-González M.A. Retraction and republication: primary prevention of cardiovascular disease with a Mediterranean diet. N. Engl. J. Med. 2018;378:2441–2442. doi: 10.1056/NEJMc1806491. [DOI] [PubMed] [Google Scholar]
- 41.Estruch R., Ros E., Salas-Salvado J., Covas M.I., Corella D., Aros F., Gomez-Gracia E., Ruiz-Gutierrez V., Fiol M., Lapetra J., Lamuela-Raventos R.M., Serra-Majem L., Pintó X., Basora J., Muñoz M.A., Sorlí J.V., Martínez J.A., Fitó M., Gea A., Hernán M.A., Martínez-González M.A., PREDIMED Study Investigators Primary prevention of cardiovascular disease with a Mediterranean diet supplemented with extra-virgin olive oil or nuts. N. Engl. J. Med. 2018;378 doi: 10.1056/NEJMoa1800389. [DOI] [PubMed] [Google Scholar]
- 42.Jobb G., von Haeseler A., Strimmer K. Treefinder: a powerful graphical analysis environment for molecular phylogenetics. BMC Evol. Biol. 2004;4:18. doi: 10.1186/1471-2148-4-18. [DOI] [PMC free article] [PubMed] [Google Scholar] [Retracted]
- 43.Jobb G., von Haeseler A., Strimmer K. Retraction note: treefinder: A powerful graphical analysis environment for molecular phylogenetics. BMC Evol. Biol. 2015;15:243. doi: 10.1186/s12862-015-0513-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Elliott K.C., Resnik D.B. Making open science work for science and society. Environ. Health Perspect. 2019;127 doi: 10.1289/EHP4808. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Sharma K. Team size and retracted citations reveal the patterns of retractions from 1981 to 2020. Scientometrics. 2021;126:8363–8374. [Google Scholar]
- 46.Amos K.A. The ethics of scholarly publishing: exploring differences in plagiarism and duplicate publication across nations. J. Med. Libr. Assoc. 2014;102:87–91. doi: 10.3163/1536-5050.102.2.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Lei L., Zhang Y. Lack of improvement in scientific integrity: an analysis of wos retractions by Chinese researchers (1997-2016) Sci. Eng. Ethics. 2018;24:1409–1420. doi: 10.1007/s11948-017-9962-7. [DOI] [PubMed] [Google Scholar]
- 48.Oransky I. Volunteer watchdogs pushed a small country up the rankings. Science. 2018;362:395. doi: 10.1126/science.362.6413.395. [DOI] [PubMed] [Google Scholar]
- 49.Xiao Y., Chen J., Wu X.H., Qiu Q.M. High retraction rate of Chinese articles: it is time to do something about academic misconduct. Postgrad. Med. J. 2022;98:653–654. doi: 10.1136/postgradmedj-2021-140853. [DOI] [PubMed] [Google Scholar]
- 50.Bar-Ilan J., Halevi G. Temporal characteristics of retracted articles. Scientometrics. 2018;116:1771–1783. [Google Scholar]
- 51.Gholampour B., Gholampour S., Noruzi A., Arsenault C., Haertlé T., Saboury A.A. Retracted articles in oncology in the last three decades: frequency, reasons, and themes. Scientometrics. 2022;127:1841–1865. [Google Scholar]
- 52.AlRyalat S.A.S., Malkawi L.W., Momani S.M. Comparing bibliometric analysis using PubMed, scopus, and web of science databases. J. Vis. Exp. 2019;152 doi: 10.3791/58494. [DOI] [PubMed] [Google Scholar]
- 53.Strotmann A., Zhao D. Author name disambiguation: what difference does it make in author-based citation analysis? J. Am. Soc. Inf. Sci. Technol. 2012;63:1820–1833. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data will be made available on request.