An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature

Abstract

Objective

Analyses of the impact of a body of clinical trial reports subject to research misconduct have been few. Our objective was to examine the impact on clinically relevant research of a group of researchers’ trial reports (‘affected trial reports’) affected by research misconduct, and whether identification of misconduct invoked a reappraisal.

Design

In 2016, we used five databases and search engines to identify ‘citing publications’, that is, guidelines, systematic and other reviews, and clinical trials citing any of 12 affected trial reports, published 1998–2011, eventually retracted for research misconduct. The affected trial reports were assessed more likely to have had impact because they had hip fracture outcomes and were in journals with impact factor >4. Two authors assessed whether findings of the citing publications would change if the affected trial reports were removed. In 2018, we searched for evidence that the citing publications had undertaken a reassessment as a result of the potential influence of the affected trial reports.

Results

By 2016 the affected trial reports were cited in 1158 publications, including 68 systematic reviews, meta-analyses, narrative reviews, guidelines and clinical trials. We judged that 13 guidelines, systematic or other reviews would likely change their findings if the affected trial reports were removed, and in another eight it was unclear if findings would change. By 2018, only one of the 68 citing publications, a systematic review, appeared to have undertaken a reassessment, which led to a correction.

Conclusions

We found evidence that this group of affected trial reports distorted the evidence base. Correction of these distortions is slow, uncoordinated and inconsistent. Unless there is a rapid, systematic, coordinated approach by bibliographic databases, authors, journals and publishers to mitigate the impact of known cases of research misconduct, patients, other researchers and their funders may continue to be adversely affected.

Strengths and limitations of this study.

• A wide-ranging literature search examined the impact of 12 randomised clinical trial reports affected by misconduct.

• A detailed examination of the extent and effect of these trial reports on guidelines, systematic and other reviews, and clinical trials was undertaken.

• We only examined the impact of 12/27 retracted trial reports, and assessing the impact on citing publications would have been strengthened if we had contacted the authors of the 68 citing publications.

• We only examined the effect on published research we were able to identify, and probably have not found all publications, especially guidelines. We did not examine impact on other forms of influence, for example, grant applications, drug company documents.

Background

We raised concerns about 33 randomised controlled trial (RCT) reports, ‘affected trial reports’, from one research group in Japan (see online supplementary appendix for list of 33 RCTs).^{1 2} Our systematic review published in November 2016 examined these affected trial reports published in the field of osteoporosis over 15 years. The affected trial reports ostensibly involved large numbers of older patients with significant comorbidities, such as stroke, Alzheimer’s disease and Parkinson’s disease.¹ In September 2016, the editor of the journal that published our systematic review conveyed the results of its investigations to all the journals with affected trial reports. By May 2019, 27/33 of these affected trial reports had been retracted for reasons including fabrication, plagiarism, authorship misconduct and unresolved concerns about data integrity.

Retraction of a research paper may have important implications for clinical practice and present and future research initiatives. Patients and research participants may be put at risk if decisions are based on findings that are later retracted because they were incorrect or unreliable.^{3 4} It is therefore important to determine the extent of a retracted paper’s influence, for example, through citations in other influential publications, such as systematic reviews and guidelines, and its use in initiating new research. There is evidence that authors of publications that cite retracted work remain unaware of the retraction,⁵ and this has potentially important consequences for their work, that of subsequent researchers, and for clinical practitioners and patients.

Analyses of the impact of a body of clinical trial reports subject to research misconduct have been few. Our objective was to examine the impact and influence of a selection of the published affected trial reports most likely to affect clinical guidance and practice and further research. We focused on affected trial reports with hip fracture outcome data in influential journals.

Methods

Search criteria

We studied the impact of a subgroup of the 33 affected trial reports whose integrity was analysed in our systematic review.¹ This subgroup of trial reports was used because these trials had hip fracture as an outcome, arguably the most important consequence of osteoporosis, and affected trial reports on this outcome are likely to have the greatest impact. We included all affected trial reports with hip fracture outcomes that had also been published in higher impact journals (ISI Web of Knowledge impact factor >4).

Evidence identification

In August 2016, we used Scopus and Web of Science to find citations of each affected trial report and the type of publication that cited each report (‘citing publications’—guidelines, systematic and other reviews, and clinical trials). We also searched Google Scholar, PubMed and personal databases to identify systematic reviews, meta-analyses, narrative reviews and guidelines relating to hip fracture prevention, which potentially would include these affected trial reports. Finally, we sought other types of publications that cited the affected trial reports, through an iterative process, for example, using the following search command in Ovid MEDLINE: (sato.tw) and ((letter or comment$).pt). We excluded self-citing publications by authors of affected trial reports from our evaluations.

Assessment of impact

Where possible, meta-analyses which included data from affected trial reports were reanalysed to investigate whether the quantitative findings, such as summary risk ratios in forest plots, would change without the inclusion of those data. In the case of reviews in which data from affected trial reports were not included in quantitative synthesis, we used our judgement. One investigator (FS) initially assessed all citing publications for the influence of affected trial reports, which were then discussed in depth with a second investigator (AA). Agreement was reached between AA and FS on all affected publications, apart from two where AG and MB provided input leading to consensus. We categorised affected publications according to the likelihood of a change in findings if the affected trial reports were excluded:

1. Findings likely to change.

2. Uncertain if findings would change.

3. Findings unlikely to change.

In November 2018, we searched again Web of Science, Scopus or guideline websites to see if the affected systematic reviews, meta-analyses, narrative reviews and guidelines, identified in August 2016, had published any notice, update, correction or retraction on publishers’ websites resulting from recognition that the publication was potentially influenced by the affected trial reports. We searched Web of Science, or Scopus if not included in Web of Science, to identify the number of times the citing publications we had judged likely or possibly to have been influenced by affected trial reports had themselves ever been cited, and the date of the most recent citation.

In July 2019, we searched Web of Science for any publication that cited the affected trial reports after they had been retracted, to examine whether these publications mentioned that the affected trial reports had been retracted.

Patient and public involvement

We did not involve patients or the public in our work.

Results

Twelve trial reports from the original 33 were identified by us for evaluation. These 12 affected trial reports all had hip fracture outcomes and were published between 1997 and 2011 in journals with impact factors>4, with 3182 reported participants (table 1).^6–17 They were published in journals with a median impact factor of 5.8 (range from 4.5^{7 9 10} to 30¹⁴). All 12 affected trial reports were retracted between June 2016 and April 2019, but by July 2019 only 7 (58%) were marked as retractions on both Ovid Medline and PubMed,9 10 12–15 17 and two further affected trial reports were marked as retracted on PubMed but not on Medline.^{11 16}

Table 1.

Affected trial reports in journals with impact factor >4 and hip fracture outcome data

Citation	Intervention	Journal impact factor	Hip fracture data		Times cited by any of the affected publications*	Google Scholar total citations August 2016
			Control	Intervention
(Retracted April 2019) Sato Y, et al. Amelioration of hemiplegia-associated osteopenia more than 4 years after stroke by 1 alpha-hydroxyvitamin D3 and calcium supplementation. Stroke 1997;28:736–9.6	Alphacalcidiol v placebo	6	4/39	0/45	8	80
(Retracted October 2018) Sato Y, et al. Menatetrenone ameliorates osteopenia in disuse-affected limbs of vitamin D- and K-deficient stroke patients. Bone 1998;23:291–6.7	Vitamin K v nil	4.5	1/54	0/54	6	94
(Retracted August 2017) Sato Y, et al. Amelioration of osteopenia and hypovitaminosis D by 1alpha-hydroxyvitamin D3 in elderly patients with Parkinson’s disease. J Neurol Neurosurg Psychiatry 1999;66:64–68.8	Alphacalcidiol v placebo	5.6	6/43	1/43	11	105
(Retracted September 2017) #Sato Y, et al. Amelioration of osteoporosis by menatetrenone in elderly female Parkinson's disease patients with vitamin D deficiency. Bone 2002;31:114–8.9	Vitamin K v nil	4.5	8/60	1/60	4	93
(Retracted October 2017) #Sato Y, et al. Menatetrenone and vitamin D2 with calcium supplements prevent nonvertebral fracture in elderly women with Alzheimer's disease. Bone 2005;36:61–8.10	Vitamin K/ vitamin D/ calcium v nil	4.5	15/100	2/100	12	55
(Retracted July 2016) Sato Y, et al. Risedronate therapy for prevention of hip fracture after stroke in elderly women. Neurology 2005;64:811–6.11	Risedronate v placebo	8.3	7/187	1/187	13	88
(Retracted June 2016) #Sato Y, et al. Risedronate sodium therapy for prevention of hip fracture in men 65 years or older after stroke. Arch Intern Med 2005;165:1743–8.12	Risedronate v placebo	13.3	10/140	2/140	19	139
(Retracted June 2016) #Sato Y, et al. The prevention of hip fracture with risedronate and ergocalciferol plus calcium supplementation in elderly women with Alzheimer disease: a randomized controlled trial. Arch Intern Med 2005;165:1737–42.13	Risedronate v placebo	13.3	19/250	5/250	12	61
(Retracted June 2016) #Sato Y, et al. Effect of folate and mecobalamin on hip fractures in patients with stroke: a randomized controlled trial. JAMA 2005;293:1082–8.14	B12/folate v placebo	30	27/314	6/314	25	323
(Retracted June 2016) #Sato Y, et al. Alendronate and vitamin D2 for prevention of hip fracture in Parkinson's disease: a randomized controlled trial. Mov Disord 2006;21:924–9.15	Alendronate v placebo	5.4	14/144	4/144	11	44
(Retracted July 2016) Sato Y, et al. Risedronate and ergocalciferol prevent hip fracture in elderly men with Parkinson disease. Neurology 2007;68:911–5.16	Risedronate v placebo	8.3	9/121	3/121	9	62
(Retracted April 2017) #Sato Y, et al. Once-weekly risedronate for prevention of hip fracture in women with Parkinson's disease: a randomised controlled trial. J Neurol Neurosurg Psychiatry 2011;82:1390–3.17	Risedronate v placebo	5.6	15/136	33/136	1	14

#Marked as retracted on both PubMed and Ovid Medline by July 2019. Retraction dates relate to online posting.

*Publications of interest: 68 systematic reviews, meta-analyses, narrative reviews, guidelines and clinical trials citing at least one of the 12 affected trial reports.

We examined 40 publications in July 2019 that cited any of the 12 affected trial reports after they were retracted. Thirty-four publications (85%) expressed no concern about the affected trial reports, and six (15%) cited the affected trial reports but discounted their findings as a result of misconduct.

Citations of affected trial reports

By August 2016, the 12 affected trial reports were cited a total of 1158 times in publications of any kind, identified by our literature searches. The median number of citations for affected trial reports was 84 (range 14 to 323).

Sixty-eight systematic reviews, meta-analyses, narrative reviews, guidelines and clinical trials cited at least one of the 12 affected trial reports. Each affected trial report was cited by a median of 11 of the 68 publications (range one to 25). Five citing publications, including Agency for Healthcare Research and Quality (AHRQ) comparative effectiveness reviews, were not listed on Web of Science or Scopus. Of the 68 citing publications indexed on Ovid Medline, 27 were systematic reviews, meta-analyses and narrative reviews, 9 effectiveness reviews and guidelines, and 32 clinical trial reports.

Reviews and meta-analyses

The 12 affected trial reports were included in 23 systematic reviews, meta-analyses and narrative reviews, covering a broad spectrum of topics, including prevention of falls and fractures, treatment of psychiatric symptoms and the role of homocysteine in disease.^18–43 Four further reviews and meta-analyses cited but did not include any data from affected trial reports in their analyses.^44–47

Fracture reviews and meta-analyses

Nine reviews and meta-analyses relating to hip fracture prevention were identified that cited at least one affected trial report. The findings of four were likely to change following the removal of the affected trial reports (see table 2).18 21 22 24

Table 2.

Number of reports citing affected trial reports, with assessment of impact of trial reports

Topic	Number of affected publications	If affected trial reports removed
		Findings likely to change	Unclear if findings would change	Findings unlikely to change
Fracture reviews and meta-analyses	9	4	4	1
Falls reviews and meta-analyses	2	1	0	1
Other reviews and meta-analyses	12	3	3	6
Effectiveness reviews and guidelines	9	5	1	3
Total	32	13	8	11

Two systematic review authors did not express caution that their findings were derived from one group of investigators. The systematic review by Zhang et al ²¹ (three citations, most recent December 2016) only included affected trial reports.²¹ However, the authors noted the lack of generalisability from Japanese-only populations. The systematic review by Zhao et al ²² focused on hip fracture and bone mineral density outcomes in Alzheimer’s disease; affected trial reports were the only sources of bone mineral density data.

Cockayne et al ¹⁸ undertook a meta-analysis of vitamin K for fracture prevention (217 citations, August 2018) which influenced Japanese osteoporosis guidelines. The reduction in hip fractures was statistically and clinically significant with an OR of 0.23 and narrow CI (95% CI 0.12 to 0.47). However, Cockayne et al ¹⁸ also included a sensitivity analysis to investigate the effect of removing the three affected trial reports.^8–10 This analysis changed the result to a statistically non-significant result with wide confidence intervals (OR 0.30, 95% CI 0.05 to 1.74). The reason given for conducting the sensitivity analysis was that the trial populations were from a single centre and included participants at much higher risk of fractures than other trials. The authors expressed some caution when interpreting the main findings of their review because of the uncertainty introduced by this sensitivity analysis and their conclusions—that vitamin K helps to prevent hip fractures—would be different if the affected trial reports were omitted. Importantly, this 2006 meta-analysis, without any caveat related to the sensitivity analysis, is the sole evidence cited for vitamin K preventing vertebral and non-vertebral fractures in the journal publication of the 2011 Japanese guidelines for osteoporosis²³ (122 citations, October 2018). In 2018, in response to retractions, Cockayne’s group published a letter of explanation and corrected article,^{19 20} removing the three affected trial reports, with the revised OR for hip fracture of 0.30 (95% CI 0.05 to 1.74).

One affected trial report¹⁴ was judged to have influenced the strength of a review’s conclusions. This was a narrative review of B vitamins and bone health²⁴ (eight citations, September 2018). The affected trial report¹⁴ showed that B vitamins significantly reduced hip fractures, contrary to the evidence cited that most studies did not demonstrate reduced fracture risk. The authors noted that the results of one affected trial report¹⁴ were unusual and speculated that improvements in neurological and cognitive function from B vitamins would prevent fall-related fractures. We judged that without the affected trial report the review’s conclusions of lack of efficacy of the intervention would be stronger.

Cases where we were unable to reanalyse meta-analyses after removal of affected trial reports would have been facilitated by authors providing open access to all their data. For four meta-analyses, it was unclear if omission of the affected trial reports would alter the findings (Richy et al ²⁵ 79 citations, October 2018; Richy et al ²⁶ 91 citations, February 2018; Murad et al ²⁷ 26 citations, August 2018; Yang et al ²⁸ 54 citations, October 2018). Clarification of the impact of the affected trial reports requires the reviews’ authors to repeat their meta-analyses with and without the affected trial reports. The citation of one affected trial report¹¹ in the review by McCarus et al ²⁹ is little more than a passing reference and data from the trial report were not used.

Falls reviews and meta-analyses

Two affected reviews and meta-analyses related to the prevention of falls were identified, since the affected trial reports also provided data on falls.

The results from one affected trial report¹⁰ changed the findings for a combined treatment (calcium, vitamin D and vitamin K for falls prevention). One Cochrane review on the prevention of falls in the community³⁰ (756 citations, November 2018) included an unpooled meta-analysis of data from one affected trial report¹⁰ and one other trial of calcium alone, relating to the number of fractures caused by falling. The analysis shows a large, statistically significant, reduction in fracture risk in the intervention group from the affected trial report (risk ratio 0.13, 95% CI 0.04 to 0.43), and a null effect in the other trial⁴⁸ (risk ratio 0.90, 95% CI 0.69 to 1.16).

Data from two affected trial reports were included in unpooled meta-analyses in the review by Batchelor et al ³¹ (65 citations, September 2018), in which the affected trial report data were not outlying.

Other reviews and meta-analyses

Twelve other affected reviews and meta-analyses were identified. Removing affected trial reports from three would likely alter their conclusions. The conclusion of one systematic review on interventions for osteoporosis (Hermann et al,³² 65 citations, 2018) that B-vitamins were likely to reduce the risk of osteoporosis was supported by data from an affected trial report,¹⁴ which showed a reduction in hip fractures in the intervention group. The review’s authors note several limitations in the affected trial report, but commented on its ‘very promising’ results.

In their review of vitamin D and Parkinson’s disease, Peterson et al ³³ (16 citations, 2017) base their conclusions almost entirely on data from four affected trial reports.8 9 15 16

Three affected trial reports^{8 15 16} were cited in the review by Binks and Dobson³⁴ (one citation, 2017) as evidence for the benefit of vitamin D and bisphosphonates in people with Parkinson’s disease. Although Binks and Dobson were careful to draw attention to the limitations of the trial reports, nonetheless their conclusions would be substantially different without these data.

Affected trial reports were included in three reviews (Alibhai et al ³⁵ 118 citations, October 2018; Carda et al ³⁶ 24 citations, September 2018; Simpson et al ³⁷ 14 citations, April 2018) where it was unclear if findings would be altered by the omission of the affected reports’ data. The conclusions of six systematic reviews were unlikely to change if data from affected trial reports were omitted.^38–43

Systematic reviews excluding affected trial reports

A further four systematic reviews cited but did not include affected trial reports in their reviews as a result of existing concerns with data,^44–46 or awaiting responses to enquiries about data.⁴⁷ One was a Cochrane review by one of the authors of this paper, with concerns dating back to 2006.⁴⁴ Another Cochrane review, whose authors corresponded with AA, excluded trials for not fitting study inclusion criteria.⁴⁵ Latham et al ⁴⁶ appeared to exclude one trial⁸ because of its poor quality from their review of vitamin D for falls prevention and other outcomes. Verheyden et al ⁴⁷ categorised two affected trial reports as awaiting assessment^{11 12} in their Cochrane review of falls prevention after stroke.

Effectiveness reviews and guidelines

Affected trial reports were cited in nine effectiveness reviews and clinical guidelines (one published in Scotland, the others in the USA), for stroke,⁴⁹ fracture prevention^50–55 and fall and injury prevention.⁵⁶ Removing these affected trial reports would likely alter findings in five reviews and guidelines.^{51–54 56}

The effectiveness review from the US AHRQ in 2007 on fracture prevention⁵¹ (no citation count available) included six affected trial reports in their Table 56,6 8 11–13 15 which are the only trials cited for bisphosphonates preventing fractures in high risk falls patients. In addition, three affected trial reports^11–13 are the only evidence used to support the 2.5 mg dose of risedronate for preventing hip fracture. This dose of risedronate does not have marketing approval in the USA (https://www.fda.gov/downloads/drugs/developmentapprovalprocess/ucm071436.pdf), but does in Japan (https://www.ajinomoto.com/en/presscenter/press/detail/g2009_07_31.html).

The publication in the Annals of Internal Medicine ⁵² from this AHRQ review has been cited 346 times, including September 2018, and it references four of the six above-mentioned affected trial reports, with these reports being the sole sources of data evidencing the reduction in fractures from bisphosphonates in patients with Parkinson’s disease, Alzheimer’s disease or stroke. The linked guideline from the American College of Physicians⁵³ (114 citations, March 2018) references the same four affected trial reports as evidence for bisphosphonate use in populations at increased risk of falls.

When the AHRQ review was updated in 2012⁵⁴ (no citation count available), it included evidence from five affected trial reports,11–13 15 16 with no new trials from other authors providing data for risedronate 2.5 mg/day in the prevention of hip fracture. The effectiveness review also states that this dose is equivalent to higher doses of risedronate.

A 2008 evidence-based handbook for nurses⁵⁶ (no citation count available) contains the statement that risedronate is effective in preventing fractures in older women, older men who have had a stroke and older women with Alzheimer’s disease, based entirely on two affected trial reports.^{12 13}

It was unclear whether exclusion of the affected trial reports would alter findings in one report. American stroke guidelines⁴⁹ (1230 citations, November 2018) used evidence from one report¹⁴ of vitamin B₁₂ and folate supplementation as the only evidence when discussing fracture prevention among patients with a recent ischaemic stroke. However, ‘routine’ supplementation of vitamins was not recommended, so we judged that it was unclear if findings, related to higher risk patients, would change without this one report.

Findings of three reviews were unlikely to change following exclusion of affected trial reports. The updated 2017 American College of Physicians’ guidelines⁵⁵ (74 citations, October 2018) includes two of the affected trial reports on 2.5 mg daily risedronate^{13 16} in its overview of the evidence for the use of risedronate from the AHRQ review,⁵⁴ but does not discuss the specific issue of the lower dose of risedronate. Guidelines from Scotland relating to osteoporosis and fractures⁵⁰ express caution about using the affected trial report on vitamin B₁₂ and folate supplementation¹⁴ in recommendations: ‘As this was a Japanese population that had suffered a stroke, it is not certain how relevant the findings are to a Scottish population.’ A guideline from AHRQ⁵⁷ excluded one trial report⁸ from its review of interventions to prevent falls in older people. The reason for exclusion was that the report did not focus on the outcome of interest, that is, the rate of falls or number of fallers, despite what appeared to be relevant falls data in the affected trial report.

Trials

We identified 32 clinical trial reports (including 27 RCTs) which cited affected trial reports. In eight cases,^58–65 affected trial reports contributed to the rationale for undertaking further RCTs. These RCTs are listed in table 3. Seven trials discussed one or more of the affected trial reports in their introduction sections, and five trials in their discussion sections. The strongest suggestion of influence in study design or rationale comes from the RCT by van Wijngaarden et al,⁶⁵ published in 2014, which discusses two RCTs in people at risk of cardiovascular disease or with cerebrovascular disease which had been unable to demonstrate B vitamins preventing fractures. These RCTs were contrasted with the affected trial report,¹⁴ which reported a reduction in hip fractures in stroke survivors. van Wijngaarden et al then state that ‘Given the conflicting results and low generalizability to the general older population, further investigation is needed.’ van Wijngaarden et al’s trial randomised 2919 participants to B vitamins or placebo for 2 years, and found no treatment effect on osteoporotic fractures.⁶⁵

Table 3.

RCT reports in which affected publications by Sato and colleagues are included in the justification for the trial

RCT	Affected trial report cited	Intervention, patient group and outcome	Sample size	Follow-up
Bauman 200558	6	1 alpha-hydroxyvitamin D2 for reducing bone loss in spinal cord injury patients	40	24 months
Berendsen 201359	14	Vitamins D, B12 and folate for slowing functional decline in people over 65 years	1250	12 months
Binkley 200960	7 10	Vitamin K for bone density and biochemical markers in postmenopausal women	381	12 months
Emaus 201361	7 9 10	Vitamin K for bone density and biochemical markers in postmenopausal women	334	12 months
Grieger 200962	14	Multivitamins for improving bone quality, falls and nutritional status in care home residents	92	6 months
Hermann 200763	14	B-vitamins for bone density and biochemical markers in people with osteoporosis	47	12 months
Rucklidge 201264	14	Multivitamins and minerals for stress in adults	91	2 months
Van Wijngaarden 201465	14	Vitamins B12 and folate for preventing fractures in people ≥65 years with elevated homocysteine status	2919	24 months

RCT, randomised controlled trial.

In another eight RCTs (not shown in table 3), the authors cited affected trial reports to draw attention to the disparities between their own findings and those reported.^66–73 It appeared unlikely that the affected trial reports contributed to the rationale for these trials.

Discussion

Our analysis suggests that affected trial reports are likely to have had an adverse impact on clinical care and other research. By 2016, affected trial reports were widely cited in the published literature of particular relevance to older people with Parkinson’s disease, stroke or Alzheimer’s disease, where, despite their generally small sample size and number of events, they dominated the literature for fracture prevention. Despite recommendations for caution in deriving conclusions from data from a very limited number of authors and centres,^{74 75} authors of reviews that included affected trials rarely expressed caution.^{21 22} We were unable to identify published or registered (ClinicalTrials.gov) RCTs of bisphosphonates in these patient groups by other research groups. Thus, other researchers (and funders) may have been dissuaded from undertaking further trials by evidence from these affected trial reports. It was apparent that some systematic reviews and guidelines, particularly for the earlier three patient groups, would be different for vitamin K and risedronate 2.5 mg/day if the affected trial reports were removed, and that some affected systematic reviews and guidelines have themselves been widely disseminated.18 23 30 49 52 However, we do not know which parts of these systematic reviews and guidelines have been influential. With one exception,^18–20 authors and/or journals of citing publications have either not identified that their publications have been compromised, or decided no action is required, although the latter seems unlikely. To our knowledge, bibliographic database/journal/publisher/guideline developer structures are not established that permit systematic identification and correction of publications that are affected by the inclusion of research with compromised integrity. Even if removing the affected trial reports did not influence their conclusions, citing authors should publish an update. This should give details of their examination of the impact of the correction or retraction on their own work, and confirm that changes are not required or have been made. This would remove uncertainty in the interpretation of their work.^{76 77} This could be aided by publishing an amended article, with an updated version number, as has been suggested by Barbour and colleagues.⁷⁸

Our assessment in August 2016 relates to publications up to that time. New, affected publications have continued to accumulate. We only assessed the impact of the 12 likely most influential affected trial reports (based on hip fracture outcomes and publication in journals with impact factor >4) from the 33 we originally investigated.¹ The remaining 21 affected trial reports may also have been influential. For example, the 2007 AHRQ report by MacLean and colleagues⁵¹ on treatments to prevent fractures includes six affected trial reports that we did not assess. It was not always possible to fully assess the impact of affected trial reports, because published data in affected publications were insufficient to allow us to replicate analyses after excluding affected trial reports. Examining impact in a network meta-analysis such as that by Murad and colleagues²⁷ would be difficult, even if data were available. Narrative reviews can be particularly vulnerable to studies with research misconduct,⁷⁵ and assessing impact in narrative reviews was often more challenging, as others have found.⁷⁴

We only investigated affected trial reports’ impact on published research. They could also have influenced grant applications, educational events, media coverage and social media, evaluation of which require a very broad range of information sources. Most importantly, we could not directly establish the effect on patients from clinical practices informed by the unreliable research. We did not examine the impact of reviews and systematic reviews authored by the group of researchers who published the affected trial reports, which includes more than 30 reviews and meta-analyses. Such active dissemination by self-citation in cases of prolific misconduct also occurred in the Reuben and Fujii cases.^{74 79}

We have probably missed guidelines in our evaluation of citing publications, since these are poorly covered by indexing databases. AHRQ full guidelines^{51 54} were identified through linked journal articles, and Scottish Intercollegiate Guidelines Network (SIGN) guidelines from personal databases.⁵⁰ Thus, we have probably underestimated the impact of these 12 trial reports.

Our findings are consistent with those of others who have investigated the impact of publications affected by research misconduct on subsequent publications and systematic reviews.74 76 80–83 In the Scott Reuben case, almost half of Reuben’s articles on perioperative analgaesia were still being cited more than 5 years after their retraction,⁸¹ and his reports widely infiltrated literature in this area.⁸⁴

Retractions of affected trial reports examined here started only in 2016, but concerns about research by this Japanese research group had been expressed as early as 2004–2007 by other groups, so that delays in investigation also increased the impact of this misconduct.^85–89 Mott and colleagues found a 46% reduction in citations of randomised clinical trial reports in the first year after retraction,⁸³ and retractions also reduce subsequent publication by authors associated with misconduct.⁹⁰

It seems systems have not changed to mitigate the impact of misconduct, once it is identified, more than 10 years since these issues were highlighted by Sox and Rennie.⁷⁶ van der Vet and Nijveen⁹¹ argued on the basis of a single, preclinical case study that indirect citations did not contribute to the propagation of research misconduct. However, for randomised trials in clinical areas affecting systematic reviews and guidelines further propagation is likely, as we show in the case of the systematic review by Cockayne et al ¹⁸ and its influence on Japanese osteoporosis guidelines.²³ In the case of Fujii’s extensive publications, the effect of his misconduct on the management of postoperative nausea and vomiting appears to have only been minimised by the large volume of publications from other authors.⁷⁹ In a recent paper, analyses by Fanelli and Moher⁹² suggested that meta-analyses may overestimate their summary effect sizes when they include studies later retracted for issues with data, methods or results.

Delays in the processes of investigating, correcting or retracting research misconduct add to the impact on patients, funders and other researchers. Delays in retraction by journals, even in response to official notification by investigating authorities, continue to be problematic and contribute to the impact of retracted work.⁹³ Once a retraction is posted by a journal all bibliographic databases and search engines should be swiftly updated. This was not the case with affected trial reports, which were retracted but not always listed as retracted on Ovid Medline and PubMed, in some cases more than 2 years later. Journals and their publishers could help to prevent the citation of retracted studies by themselves checking or requiring authors to check their reference list against Retraction Watch’s database (http://retractiondatabase.org/RetractionSearch.aspx?) before submission.⁷⁷ Zotero software that is linked to Retraction Watch’s database,⁹⁴ or ReTracker linked to Retractions in PubMed,⁹⁵ might facilitate authors’ awareness of retractions. Clearly marked, retracted articles and properly informative retraction notices should be linked on journals’ websites and both should be freely accessible.^{76 80}

Research misconduct can have widespread detrimental effects on subsequent research initiatives and clinical practice. Some possible solutions to minimise the impact of retracted publications are given in box 1, but there remains no overarching body with the commitment to coordinate managing the consequences of proven research misconduct.

Box 1. Some possible solutions for minimising the impact of retracted research reports.

• Journals and publishers should ensure that expressions of concern, retractions or corrections are appropriately flagged so that they are immediately available to be listed as such on bibliographic databases, including that of Retraction Watch, and search engines.

• Publishers should sign up to The CrossMark (https://www.crossref.org/services/crossmark/), an initiative to take readers to the current version of the paper, which should include expressions of concern, retractions or corrections.

• After institutional investigations have found that misconduct has taken place, institutions could notify corresponding, first authors and senior authors of citing publications.

• Listing an expression of concern, retraction or correction on bibliographic databases should generate automatic alerts to corresponding, first authors and senior authors of citing publications.

• Retraction Watch’s database of retractions, linked to reference management software, should be used to regularly scan researcher’s personal reference libraries.⁹⁴

• Journals and their publishers could help to prevent inappropriate citations by themselves checking or requiring authors to check their reference list for expressions of concern, retractions or corrections.

• Organisations responsible for publications, which are not usually listed on bibliographic databases, e.g. clinical guideline groups, should regularly check Retraction Watch’s database against their reference lists, or ensure their guidelines are listed on bibliographic databases.

• Authors of citing publications should publish an amendment, or a reassurance that the publication is unaffected, with a link to the affected publication.