Identifying and managing problematic trials: a Research Integrity Assessment (RIA) tool for randomized controlled trials in evidence synthesis

Abstract

Evidence synthesis findings depend on the assumption that the included studies follow good clinical practice and results are not fabricated or false. Studies which are problematic due to scientific misconduct, poor research practice, or honest error may distort evidence synthesis findings. Authors of evidence synthesis need transparent mechanisms to identify and manage problematic studies to avoid misleading findings.

As evidence synthesis authors of the Cochrane COVID-19 review on ivermectin, we identified many problematic studies in terms of research integrity and regulatory compliance. Through iterative discussion, we developed a Research Integrity Assessment (RIA) tool for randomized controlled trials (RCTs) for the update of this Cochrane review. In this paper, we explain the rationale and application of the RIA tool in this case study.

RIA assesses six study criteria: study retraction, prospective trial registration, adequate ethics approval, author group, plausibility of methods (e.g., randomization), and plausibility of study results. RIA was used in the Cochrane review as part of the eligibility check during screening of potentially eligible studies. Problematic studies were excluded and studies with open questions were held in awaiting classification until clarified. RIA decisions were made independently by two authors and reported transparently. Using the RIA tool resulted in the exclusion of >40% of studies in the first update of the review.

RIA is a complementary tool prior to assessing ‘Risk of Bias’ aiming to establish the integrity and authenticity of studies. RIA provides a platform for urgent development of a standard approach to identifying and managing problematic studies.

1. Introduction

Systematic reviews aim to identify all studies that meet the eligibility criteria for the review. Studies that challenge the principles of good clinical practice and scientific integrity can mislead and corrupt the findings of a systematic review, and hence mislead guidelines and official recommendations that use the reviews. Public health laws may be developed based on the findings of systematic reviews, which has been particularly important in responding to the COVID-19 pandemic.¹ This causes a dilemma for review authors: Ioannidis et al. pointed out that authors should become aware that false and fatally flawed trials are very common in the field of medicine and suggests toning down the confidence in their conclusions.² Avenell et al. found evidence that trials retracted due to misconduct distorted the evidence base including systematic reviews, meta-analyses, narrative reviews and clinical guidelines citing such trials and concluded that many of those guidelines, systematic or other reviews would likely change their findings if the affected trial reports were removed.³

The COVID-19 pandemic threw these dilemmas into sharp focus. We systematically synthesized the evidence for the Cochrane review “Ivermectin for preventing and treating COVID-19”.⁴ During the second pandemic year and after publication of the Cochrane review, several clinical trials were retracted due to critical concerns on trustworthiness.^5–8 After their retraction, studies claiming to prove ivermectin’s huge beneficial effect for treating this disease remain considered in published evidence syntheses.^9–12 Additionally, it has to be considered that even if evidence syntheses are retracted as well¹³ or corrected for such distortions^14,15 the initial effect on patient demand, public beliefs, clinical practice, and research priorities was difficult to reverse. Whilst most of the retracted studies actually did not meet our inclusion criteria, we felt uneasy about the full extent of the problematic study pool investigating ivermectin for COVID-19, and in preparing the review update, we developed a tool for our Cochrane review to help us identify studies that were potentially “problematic” in relation to whether they had been fabricated, data had been altered, or were not in accordance with good clinical practice.

Mechanisms for identifying studies with fabricated or false data, or where problems with research conduct amounting to failure of research integrity, is an active area of research. A group of researchers recently reported results from a qualitative international interview study with the aim to better understand the views of research integrity experts about what might make a study problematic or untrustworthy and what warning signs could be used in a practical screening tool to identify potentially problematic studies.¹⁶ Cochrane defines a ‘problematic study’ as “any published or unpublished study where there are serious questions about the trustworthiness of the data or findings, regardless of whether the study has been formally retracted. Scientific misconduct will not be the only reason that a study might be problematic; problems may result from poor research practices or honest errors.”.¹⁷ Cochrane has also published guidance to facilitate research integrity checks in the reviews it publishes,^18,19 but these checks have not routinely formed part of evidence synthesis or guideline development processes to date. We drew on a few non-validated tools available at the time of updating our Cochrane review to develop our approach (such as the ‘REAPPRAISED’ checklist for evaluation of publication integrity²⁰ and the data extraction tool from the Cochrane Pregnancy and Childbirth Group²¹ that addresses various aspects of scientific integrity). The latter tool was recently used for the systematic exploration of trustworthiness of published trial data in 10 RCTs from one author investigating psychological interventions for the treatment of chronic pain.²² With this screening tool the authors identified concerns about research governance, data plausibility at baseline, the results, and apparent data duplication.

In this paper we describe the Research Integrity Assessment (RIA) tool to assess randomized controlled trials (RCTs) and how we have used it for the Cochrane review on ivermectin.²³ This case study is another important step for the urgent development and adoption of a standard approach to sifting out problematic studies.

2. Methods

2.1. Development of the new research integrity assessment tool

We developed a tool for the assessment of research integrity of RCTs focusing on investigational medicinal products (IMPs^24–26). The tool was developed prior to preparing the first update of the Cochrane review “Ivermectin for preventing and treating COVID-19”²³ and is guided by specific questions of how to identify and deal with problematic studies in the context of this systematic review. Study characteristics to assure research integrity of RCTs have already been discussed and considered in various other publications and places,^18,20,21 and some characteristics are legally required, such as ethics committee approval.²⁷ Based on study characteristics and specific items reported in existing screening tools^20,21 and legal requirements,²⁷ we used iterative discussions, piloting preliminary forms, and web-conferences among our team of six authors to agree on critical and important study characteristics and handling of identified problematic studies in this systematic review. Selection of study characteristics and criteria is based on the content expertise and subjective assessment of the authors involved. We are content experts who were either authors on the Cochrane ivermectin review (SW, MP, SR, NS), part of the editorial process (PG), or had previously developed strategies for dealing with problematic studies in Cochrane Review Groups (ES). The assessment tool was developed in an Excel-based format and contains questions to critical and important criteria that help to identify problematic RCTs when deciding on inclusion into a systematic review. The tool was used in a case study of the Cochrane review update on ivermectin and it was not validated.

3. Results

3.1. RIA: Critical and important study characteristics to assess research integrity of RCTs investigating IMPs

We achieved consensus among the authors on six domains considering critical and important study characteristics to assure research integrity of RCTs investigating IMPs based on adherence to good clinical practice and scientific integrity: Retraction notices, prospective trial registration, ethics committee approval and written informed consent, author group, sufficient reporting and plausibility of methods (e.g. study design/randomization), and plausibility of study results. In the workflow of RIA through domain 1 to 6, three decisions on a studýs eligibility for the evidence synthesis are possible at any hierarchical step from 1 to 6: RCTs may be either included or excluded from the evidence synthesis, or moved to awaiting classification (Figure 1). The ‘awaiting classification’ category is typically used in Cochrane reviews for studies about which an inclusion or exclusion decision cannot be made because insufficient information is currently available.²⁸ In the following paragraphs, we introduce the Research Integrity Assessment (RIA) tool based on critical and important criteria of study characteristics summarized in key domains, explain their rationale, and provide methodological guidance. Critical and important criteria of the RIA tool are summarized in Table 1. An Excel-based format of the tool with critical and important signalling questions to the domains, is available online (https://doi.org/10.5281/zenodo.7024699).

Figure 1. Hierarchical work flow and decision tree of the Research Integrity Assessment (RIA) tool.

Potentially eligible RCTs identified during screening should be assessed for research integrity hierarchically considering domain 1 to 6. Retraction, lack of prospective registration, lack of adequate ethical approval with informed written consent, inconsistencies in the author group and the location of the study, lack of proper randomization, implausible study results should lead to exclusion of a RCT. Concerns with the RCT in any domain put the study in ‘awaiting classification’ and should lead to further investigations. If no concerns appear through all domains or could be clarified, e.g. in correspondence with study authors, the RCT meets criteria for inclusion in the review and can be processed further. In living systematic reviews, included RCTs and RCTs ‘awaiting classification’ must be reassessed for retraction notices.

Table 1.

Critical and important criteria for a Research Integrity Assessment (RIA) of RCTs investigating IMPs for evidence syntheses

	Domain 1: Retraction or expression of concern	Domain 2: Trial registration	Domain 3: Ethics approval	Domain 4: Author group	Domain 5: Methods	Domain 6: Results
Critical criteria	Retraction of the study	Registry number not reported Not prospectively registered	Ethics committee approval not reported Name and location of the ethics committee not reported Ethics approval number not reported Written informed consent not obtained	Inconsistency of authors’ affiliations and countries the study is reported to have taken place in Inconsistency in different parts of the article, e.g. country and location of study conduct	Insufficient reporting of the study design (e.g. randomization) and unclear that the study was properly randomized
Important criteria	Expression of concern published elsewhere	Inconsistency in details of study dates reported in the publication and in the registration documents	Ethics committee approval not obtained by a nationally recognised ethics committee as defined in the country’s clinical trial regulations	Implausible number of authors for the study design (e.g. a single author article reporting a randomized control trial is unrealistic)	Number of participants in each group inconsistent with the reported randomization method (e.g. block randomization) Baseline details not reported in sufficient detail to assess whether randomization worked properly Incomplete or missing study flow diagram (optional)	Implausible number of patients with the condition recruited within the timeframe Unrealistic response rate or loss of follow-up Plagiarism Excessive similarity or difference in the characteristics of the study participants between groups Discrepancies between data reported in figures, tables, and text Calculation errors Implausible study results (e.g. massive risk reduction)

Domain 1: Retracted studies or studies with published expression of concern

The problem:

Journal editors should retract a published study if they have clear evidence that the findings are unreliable.²⁹ Consequently, retracted studies should not be included in evidence syntheses as they can distort the evidence base.³

Assessment:

Cochrane has published detailed guidance on how to search for retraction notices and handle retracted studies in Cochrane reviews.^18,19 Retracted RCTs can be identified by review authors as such through a search for post-publication amendments in the systematic search for studies or on the Retraction Watch Database (http://retractiondatabase.org/RetractionSearch.aspx?). Retracted RCTs should simply be excluded. An expression of concern can be published by a journal to raise awareness of a possible problem in a published study²⁹ and may also announce a full retraction. According to the Cochrane problematic studies guidance, review authors should take RCTs with an expression of concern depending on the nature of the concern and either exclude from the review, or put in awaiting classification category, or include in the review (e.g. for reasons that do not affect the validity of the data).¹⁸

Domain 2: Prospective trial registration

The problem:

The WHO declares that registration of all intervention trials is a scientific, ethical and moral responsibility and expects that all clinical trials are prospectively registered in a WHO Registry Network approved registry.³⁰ The WHO regards trial registration as the publication of an internationally-agreed set of information about the design, conduct and administration of clinical trials. These details should be published on a publicly accessible website managed by a registry conforming to WHO standards. Registries checking data as part of the registration process may lead to improvements in the quality of clinical trials by allowing identification of potential problems early in the research process.³⁰ One of the minimum standards set out for trial registries in the International Standards for Clinical Trial Registries (Item 2.3) is that registries must obtain written third-party confirmation of a trial’s existence as part of the registration process.³¹ Prospective trial registration can be a proxy for trial quality as investigators and authors of high quality trials know and follow these responsibilities, and prospectively registered studies have been shown to be at lower risk of bias.³² The exclusion of non- and retrospectively registered RCTs in systematic reviews may shrink the study pool dramatically because there is poor compliance with trial registration^33,34 despite the fact that it may be legally required as defined in clinical trial regulations world-wide. On the other hand, this approach could bring us closer to the truth about the effectiveness of an intervention and gives us more confidence in the conclusions of our systematic reviews.³⁵

Assessment:

Whether or not a RCT has been prospectively registered can be proven in the trials register. The trial registry number should be reported in the study publication.³⁶ Prospective registration is defined as registration of a trial in a recognized national or international trials register before enrolment of the first participant.³¹ Due to the increase in trial initiation early in the COVID-19 pandemic, a delay between submission of a trial registration to and the actual publication on the register web site may have occurred. Extraordinary circumstances such as this pandemic, however, do not release investigators from submitting their registration prospectively or justify a total lack of registration. To avoid an unfair and unreasonable judgement, the registration’s first submission date should be considered and deemed prospective if occurring before enrolment of the first study participant. In case of doubt, review authors should contact the authors for the submission date of the trial protocol and the RCT should be moved to the ‘awaiting classification’ category. Review authors should also search for any inconsistencies in details of study dates (for example, study start, start of recruitment, registration date) reported in the publication and in the registration documents. In case of inconsistency, review authors should contact the investigators for clarification. Review authors should exclude non-registered and retrospectively registered RCTs.

There is no empirical evidence that inclusion of only prospectively registered RCTs in a review guarantees the inclusion of trustworthy studies only,¹⁸ therefore, additional study criteria are critical and considered in the following.

Domain 3: Adequate ethics approval

The problem:

Adherence to ethical principles in clinical studies is compulsory to protect the dignity, rights and welfare of research participants.²⁷ As such, all clinical trials involving human beings have to be reviewed by an ethics committee to ensure that the appropriate ethical standards are being upheld.²⁷ It is further good clinical practice that the study investigators obtain written informed consent from all participants before randomization.³⁷ Details on ethics approval and written informed consent are included in the WHO trial registration guidelines.³¹

Assessment:

Review authors should check the trials registry record, the study protocol and the published study report for a published statement on whether approval from an ethics committee was granted. This statement should include the name of the ethics committee granting the approval, and an approval number.¹⁸ A statement on whether written informed consent was obtained, or a justification for its absence, should also be included in the study records. Lack of such statements in the article does not necessarily mean that a study did not have ethics approval or did not obtain participantś written informed consent. Therefore, if a study did not report the name of the ethics committee and/or the approval number and information on consent, review authors should send a request to the authors and the RCT should be moved to the pool of studies awaiting classification until clarified. If the authors cannot provide any component of the above, the RCT should be excluded. Moreover, it should be assured that a nationally recognised ethics committee as defined in the country’s clinical trial regulations gave the approval. The ethics committee can be searched for on the WHO list of national ethics committees (https://apps.who.int/ethics/nationalcommittees/nec.aspx) or searching the national responsible authority’s list of recognised ethics committees. Unfortunately, there is no international list of responsible authorities available and an individual web search might be necessary. An English language summary of the specific regulations for some countries can be found on the NIH Clinical Trials Regulation website (https://clinregs.niaid.nih.gov/).

Domain 4: Author group

The problem:

Authorship confers credit and has important academic, social, and financial implications, but authorship also implies responsibility and accountability for published work.³⁸ When authorship is abused, accountability and responsibility can be questionable and the potential for manipulated analysis and conclusions may increase. The domain ‘author group’ is a proxy for whether it is plausible that the study has actually taken place with a focus on authors’ details, number of authors, and the location where the study was conducted. There are certainly many more details regarding authorship that may be important to address, e.g. author contributorship, ghost authorship, and funding details, which are currently not part of the RIA tool. However, when applying our tool, other review authors are welcome to examine studies in more extensive detail in this domain. The REAPPRAISED list examines research governance, authorship, and research conduct, and details (unfortunately without further guidance) can be found there.²⁰

Assessment:

Review authors should focus on the authors’ details and the location where the study was conducted. Review authors should check whether an article is authored primarily by individuals with affiliations different from the country(ies) where the study was conducted without sufficient explanation, though information about trial sponsorship, ethics committee approval, funding and regulatory oversight included in trial registry details can assist in evaluating such cases. Moreover, inconsistencies in the article regarding different countries specified in different parts of the article or as compared to the trial registry entry should also flag a study as ‘potentially problematic’. Finally, the review authors should check whether the number of authors is plausible for the study design. In its most extreme form, one single author article reported an RCT, and may indicate a fabricated study, since it is impossible for one person to have managed such a complex study design alone.¹⁸ Li et al. suggested a low author-to-study size ratio or a number of authors less than three should trigger concern.³⁹ If there are concerns regarding a study’s author group, a request should be sent to the authors and the RCT should be moved to the pool of studies awaiting classification until clarified. If the authors cannot justify any component or inconsistency of the above, the RCT can be excluded. However, exclusion of the RCT should be weighed against judgments from the other domains.

Domain 5: Sufficient reporting and plausibility of methods (for example, randomization methods)

The problem:

RCTs that report very sparsely on their methods (e.g. study design/randomization) immediately raise alarms, particularly when study authors provide insufficient information to be able to make an adequate assessment of the risk of bias using the Cochrane Risk of Bias (RoB) tools 1 or 2.0 in evidence synthesis.⁴⁰ Cochrane Reviews include an assessment of the risk of bias for each included study. The RoB tools 1 or 2.0 are structured into a fixed set of domains of bias, focusing on different aspects of trial design, conduct and outcome reporting.⁴⁰ With incomplete methods the RoB assessment frequently returns to ‘unclear’ or ‘some concerns’ of bias which is actually misleading. Conversely, study authors who applied deficient study methods or introduced bias into their studies could avoid a poor risk of bias rating simply by underreporting. Moreover, it is important to check that the study was properly randomized. Identical numbers allocated to each group in the absence of a block approach to randomization are considered as cause for concern.²²

Assessment:

The most critical criteria for domain 5 is a sufficient reporting on study design methods (e.g. randomization) in the study report to consider that the study was properly randomized. The method used for the randomization must be described and the process must lead to a random allocation of the participants. The sole designation ‘randomized study’ is not sufficient. Identical numbers of participants allocated to each group without use of a block randomization was considered as cause for concern. Baseline details must also be provided in sufficient detail to estimate whether the randomization worked. If it turns out that a trial declared as ‘randomized’ in the article was not properly randomized, the review authors should exclude the study. Considering details on the study flow diagram such as the number of participants being randomized, receiving the intervention, and being analysed, is optional for the RIA, as those aspects will be covered by the Cochrane RoB tool 1 and 2.0 (and missing information may be adjudicated for with a high risk of bias judgement). Therefore, with domain 5 it should be assured that only RCTs with sufficient reporting of their methods and proper randomization are to be included in the study pool of the review, and studies with insufficient details are held in awaiting classification until the authors provide further details upon request.

Domain 6: Plausible results

The problem:

There are alarming numbers of fabricated and false data or trials published each year^2,41 and this has major consequences for the entirety of the health research ecosystem and for naïve systematic reviewers who assume that the published studies are real. How can systematic reviewers identify and deal with false or fabricated data and trials when looking at published articles? In-depth checks of baseline details and individual participant data are time consuming and require specific statistical training – and both may be limited for most systematic reviewers. Therefore, we achieved expert consensus on a few criteria, which are used already in existing screening tools,^20,21,42 to be warnings for implausibility of reported results in RCTs.

Assessment:

Review authors should assess for plausibility 1) the number of patients recruited within the timeframe with the condition; 2) the response rate or number of participants lost to follow-up; 3) excessive similarity or difference in the characteristics of the study participants between groups; and 4) results that could be implausible (e.g. massive risk reduction, unexpected outlier data, unusual frequency of a rare outcome). Furthermore, review authors should note any data error (e.g. number of participants or events that did not add up), calculation error, and discrepancies between data reported in figures, tables, and text. In addition, when multiple reports of the RCT are available, review authors should check for overlap in text and data between published articles by the same or different authors without explanation.

When working through those criteria, and inconsistencies or implausible data are identified, the review authors should send an information request to the study authors to allow for comments and clarification. Until resolution, the RCT should be held in awaiting classification. This domain should be handled with care as for the above mentioned complexity and should not lead to exclusion of a trial without clear evidence.

3.2. When should the RIA be used during evidence synthesis?

It is important to assess RCTs that pass the PIC(O) (participants, intervention, comparator, and (outcomes)) eligibility screening as early as possible for research integrity. Exclusion of problematic RCTs cleans the whole study pool, not only the estimated effects in meta-analyses and conclusions thereof, but also qualitative analyses, summaries of baseline characteristics, and conclusions regarding evidence gaps. Therefore, early exclusion of problematic RCTs or movement to the awaiting classification category of inconclusive RCTs is preferred.

3.3. How should the RIA be used during evidence synthesis?

In the hierarchical workflow of RIA through domain 1 to 6, three decisions on a studýs eligibility are possible at any step from 1 to 6: RCTs may be either included, excluded or moved to awaiting classification (Figure 1). Whenever it is concluded from the decision on one domain that a RCT has to be excluded, the following domains no longer apply, can be omitted and do not have to be answered. The first three domains, e.g. retraction, prospective registration, and ethics approval, can frequently be answered with certainty and direct decisions without sending an author request (e.g. for retraction, retrospective registration, verbal informed consent). Retraction, lack of prospective registration, lack of adequate ethics approval with informed written consent, should lead to exclusion of a RCT. For domains 4 to 6, a definite decision on a studýs research integrity is much more difficult to find and correspondence with the study authors can be necessary. Decisions can be subjective. For performance of the RIA, it has to be kept in mind that RCTs should be included only if there is no obvious doubt regarding any of the critical criteria included in the tool. However, inconsistencies in the author group and the location of the study, lack of proper randomization, or implausible study results are sometimes difficult to answer with certainty. We suggest to be careful in excluding a RCT for a single domain concern regarding domains 4 to 6, especially if there are no other concerns raised when using this tool – it should apply in dubio pro reo. If there is any inconsistency, insufficient information or serious concerns, requests should be sent to the authors. Each study author must have the opportunity to respond and clarify the questions. When authors do not respond or questions remain unanswered, the RCT should be held in the awaiting classification category when the evidence synthesis is published. Systematic reviews in a living mode have to re-evaluate all included RCTs and RCTs held in awaiting classification for published retraction notices or expression of concern for each update.

For the RIA, similar to a standard systematic review approach, all documents of a study should be used for the assessment (e.g. all reports of a study including full text publication, registration, and protocol).

The Excel version of the tool is based on critical and important signalling questions to the domains and includes columns to summarise a conclusion for each domain and an overall conclusion, which justifies the decision on research integrity and eligibility (https://doi.org/10.5281/zenodo.7024699). The tool offers a transparent way to document what the review authors have done (e.g. correspondence with the trial authors) and their judgements. For transparency, the table should be published as a supplement to the systematic review or deposited in an online repository with permanent digital object identifier (doi). The consequence of the RIA on the study pool should also be documented in the PRISMA flow diagram of the review using a new reason for exclusion ‘failed research integrity assessment’.

3.4. Who should apply RIA?

Each study should be independently assessed by two review authors and discrepancies should be resolved by discussion. It is important that the review author team consists of researchers who have knowledge in clinical trial design and regulation, systematic review methodology, and clinical content expertise. Checks for plausibility of results (domain 6) require some knowledge of the topic area of the RCT.

3.5. Working example: Cochrane ivermectin review

For the Cochrane ivermectin review, we evaluated 25 RCTs in our first review update²³: 14 included studies and three studies with results awaiting classification from the original review version⁴ were re-evaluated; and eight studies with results identified by the updated search were evaluated. In the review update, we excluded 11 out of 25 assessed studies, moved three to awaiting classification, and included eleven studies meeting all criteria for inclusion into the review update (Table 2, Supporting information 1). The most frequent reason for exclusion was a lack of prospective trial registration in nine studies; one study was retracted and another turned out not to be a randomized trial. The RIA excel sheet was published as a supplement to the updated Cochrane review and deposited in an online repository with permanent digital object identifier.⁴³ Changes to the RIA tool according to peer review of this method paper which appeared after the Cochrane review had been published are listed in Supporting Information 2.

Table 2. Research integrity assessment of RCTs applied to the Cochrane ivermectin review update.

25 RCTs were evaluated (14 included studies and three studies with results awaiting classification from the original review version, and eight studies with results identified by the updated search). The assessment took place hierarchically from domain 1 to 6 for all studies. Whenever it is concluded from the decision on one domain that a study has to be excluded, the following domains were not answered.

	Domain 1: Retraction or expression of concern	Domain 2: Trial registration	Domain 3: Ethics approval	Domain 4: Author group	Domain 5: Methods	Domain 6: Results
Number of studies evaluated	25	24 (−1)	15 (−9)	15 (−0)	15 (−0)	14 (−1)
Result	1 retracted study (awaiting classification 1st review); identified via retraction watch database and post-publication amendment	9 studies without prospective trial registration (6 included and 1 awaiting classification 1st review, 2 updated search); 4 non- and 5 retrospectively registered	1 study posted results on a trials register without reporting ethics approval and number (awaiting classification 1st review); all studies with written informed consent	2 studies posted results on a trials register and could not be assessed (1 awaiting classification 1st review, 1 updated search)	1 non-randomized study (included 1st review); 3 study authors did not report/clarify sufficient detail on randomization (1 awaiting classification 1st review, 2 updated search)	3 study authors did not clarify questions on study results (1 awaiting classification 1st review, 2 updated search)
Decision/Action	1 study excluded 24 studies without concerns	9 studies excluded 15 studies without concerns	no study excluded 1 study awaiting classification 14 studies without concerns	no study excluded 2 (+1) study awaiting classification 13 studies without concerns	1 study excluded 3 (+1) study awaiting classification 11 studies without concerns	no study excluded 3 (+0) study awaiting classification 11 studies without concerns

4. Discussion

Whilst we know that problematic studies create challenges, the number of problematic studies and their influence has not been clearly quantified.^3,44 The example of studies included in, then later excluded from, the Cochrane ivermectin review demonstrates the need to manage problematic studies. Every study included in a systematic review has an influence on the results and conclusions of the review and this has far-reaching consequences for the population treated on the basis of the review.⁴⁵ As the number of systematic reviews being part of or being considered in evidence ecosystems continues to grow, we believe ensuring research integrity of studies included in the study pool of a systematic review is an approach that provides evidence closer to the unbiased truth and respects human rights. We hope the RIA tool, consisting of six domains to assess the research integrity of RCTs included in systematic reviews, will be viewed as a new transparent option to include the concept of research integrity in evidence synthesis as part of the eligibility screening and serves as a platform for urgent developments in this direction.

We believe the RIA tool represents a considerable expansion over previous efforts. The strength of our tool is that it was developed in accordance with recent Cochrane guidance on identifying problematic trials^17,18 using an international team of (Cochrane) editors and content experts in systematic reviews methodology, clinical trial regulations, and evidence ecosystems. Using this tool for evaluation of research integrity of potentially eligible RCTs should be performed as part of the review’s eligibility screening. Documentation of reviewers’ decisions on in- and exclusion will be transparent for evidence users increasing systematic reviewers’ accountability. We view that we need to implement an integrity assessment process quickly, but see this tool as the first pilot version that can and should be adapted, modified, tested and validated over time. We have already launched a meta-epidemiological study to assess the impact of RIA on a larger pool of interventional RCTs included in systematic reviews on COVID-19.⁴⁶

This RIA tool is not a ‘Risk of Bias’ assessment. Critical aspects of research integrity assessed with the RIA are not considered in established RoB assessments such as the Cochrane RoB 1 or 2.0 tool. RIA aims to establish the integrity and authenticity of the studies. Cochrane’s research integrity department points out that the Cochrane RoB tool operates on the basis that the data are true.⁴⁷ Research integrity of the trials is acknowledged as being a separate issue that should be handled prior to RoB assessment.

There may be several limitations to our approach. The example presented here describes the development of the RIA tool used in the Cochrane review ivermectin for COVID-19. Therefore, this tool was used in one case study and has not yet been validated. In addition, using this tool on trials in standard evidence synthesis work, where data are used from trials conducted in any country at any point in time, will not be as straightforward as in this example. Depending on the clinical question, in some cases, the majority of trials may have been conducted before trial registries even came into existence. Using this tool on trials is challenging in these circumstances, as there may not be any public records of trial governance, and alternative approaches may need to be developed, e.g. including older studies in secondary analysis only. A larger group of experts with different backgrounds and a Delphi panel process could have led to a different final set of critical domains. Future application of this tool may lead to refinement or adjustment of some domains, e.g. for different scenarios with intervention studies which are not IMPs. Another area in need of development for the RIA tool is the number of key criteria that may be needed to warrant study exclusion. Identification and handling of false and fabricated data or trials by in-depth checks of baseline details and individual participant data requires specific statistical training – and both may be limited for most systematic reviewers. Statistical experts for individual participant data (IPD) analysis and detection of fabricated data may not be well represented by our sample of experts. Therefore, the checking domain for plausibility of study results may still be under development and may change when new methods for identification of false and fabricated data become available. Research in this area is still in its infancy. Bordewijk et al. recently published a scoping review on methods to assess research misconduct in health research and concluded that tools to investigate are rudimentary and labour-intensive, and automatic tools and routine validation of these methods is needed.⁴² We plan to apply the RIA tool to a larger sample of systematic reviews to verify performance. With these findings, we plan to further develop the tool within the next three years in a Delphi process with a subsequent update.

We believe that this tool and the ivermectin review example has important teaching capabilities to raise awareness of the concept of research integrity for systematic reviewers and clinical study investigators. Clinical trial regulations are designed to ensure patient safety and must be adhered to. It is unfortunate that research was identified in the context of the ivermectin review that does not meet the required standard, and the only way forward is to improve access to Good Clinical Practice training for everyone involved in clinical trials, health research and evidence synthesis worldwide, with the aim of achieving full compliance with the regulations. We hope that research integrity criteria will lead to a rethinking so that the results of lawfully conducted studies are given prominence rather than treating all data equally in evidence synthesis. In addition, we hope that ensuring research integrity of studies included in the study pool of a systematic review is an approach that helps to provide evidence closer to the unbiased truth and improves respect of human rights in evidence synthesis. It is a moral duty that problematic studies are not included in systematic reviews used to inform guidelines or as information for health professionals or the public.

Highlights.

What is already known

Including problematic studies in terms of research integrity in evidence syntheses can lead to misleading conclusions, and harm human health. Cochrane has published guidance to facilitate research integrity checks in the reviews it publishes. However, these checks have not routinely formed part of evidence synthesis or guideline development processes to date.

What is new

We developed the RIA (‘Research Integrity Assessment’), a tool to assess the integrity of research and adherence to good clinical practice reported in RCTs of investigational drugs for an update of a Cochrane COVID-19 review. The assessment uses signalling questions to identify problematic RCTs and is used when studies are being considered for inclusion in an evidence synthesis. Problematic studies are excluded or held in an awaiting classification category.

Potential impact for Research Synthesis Methods readers outside the authors’ field

Ensuring research integrity of studies included in evidence synthesis is an approach that helps to provide evidence closer to the unbiased truth and improves respect of human rights in evidence synthesis.

Data availability statement

All data relevant to the study are included in the article, uploaded as supporting information, or available in the Cochrane ivermectin review with link to a public, open access repository.