Systematic review protocol assessing the processes for linking clinical trial registries and their published results

Abstract

Introduction

Clinical trial registries are an important source of information for tracking clinical trials from their inception through to their reporting, and have been used to measure publication bias and outcome reporting bias. Our aim is to survey and quantify the processes that have been used to identify links between clinical trial registries and published trial reports in studies that rely on these links to evaluate the completeness and accuracy of trial reporting.

Methods and analysis

We will identify studies that describe a process for identifying the links between a trial registry included in the WHO International Clinical Trial Registry Platform and published trial results, and use those links to evaluate the completeness and accuracy of trial reporting. Information extracted from the studies will include the purpose and application domain of the study, registries used or searched, processes by which the links were identified, the study period and proportions for which links were found. We will summarise what is known about the number and availability of links between clinical trial registries and published results, and examine how automatic linking, inference and inquiry processes have been used to identify links since the introduction of trial registries.

Ethics and dissemination

The systematic review is focused on the analysis of secondary data and does not require ethics approval. The results of the systematic review will be used to inform standard processes used to identify links to and from clinical trial registries in studies that evaluate the completeness and accuracy of clinical trial reports, as well as systematic reviews. Our findings will be disseminated by publishing the systematic review in a peer-reviewed journal, and by engaging with stakeholders from clinical trial registries and bibliographic databases.

Strengths and limitations of this study.

• This systematic review will quantify the processes used to link clinical trial registries to clinical trial results and determine how these may have changed since the introduction of clinical trial registries.

• The processes used to link clinical trial registries to published reports of clinical trials vary across studies that rely on those links to evaluate the accuracy and completeness of trial reports, and this systematic review will quantify these differences to inform the way this is performed in the future.

• By producing a baseline measurement of the availability of automatic links and the number of other links that must be identified through inquiry or inference, the systematic review will help determine the potential value of using clinical trial registries to augment current methods used to identify trials for systematic reviews of clinical interventions.

• As studies linking clinical trial registry data to published results are designed for a range of different purposes, the processes used to identify links are not always reported completely, making information extraction difficult.

Introduction

Clinical trial registries were designed to provide information to researchers, clinicians and the public about trials that are underway or for which the results have not been reported.^{1 2} Since their introduction, their use has increased substantially following changes in requirements for journal publication and changes to the law in several countries,^3–5 and a number of studies have examined publication bias^6–17 and outcome reporting bias,^18–27 using one or more of the registries.

Studies that use clinical trial registries to examine the completeness and accuracy of clinical trial reporting rely on being able to establish links between registries and reports of clinical trials. A proportion of those links can be accessed automatically,^{28 29} but the remainder must be determined by inference or inquiry. The manner in which these processes are used vary from study to study and are known to be time-consuming.^{9 26 27} It is not yet known whether differences in the way links are established by these processes have influenced the results of studies examining publication bias or outcome reporting bias. As a consequence, there is a current need to survey the studies that have used these processes to identify links between clinical trial registries and their published results.

Clinical trial registries are sometimes used to identify trials for inclusion in systematic reviews.^30–32 Systematic reviews benefit from clinical trial registries because they can be used to quantify reporting bias for an intervention or condition, can be used to assist in scheduling updates,^33–36 also could be used as an external corpus in machine learning methods that automate or assist in searching and screening methods³⁷ and in some cases as a source of trial results that have not been published in peer-reviewed literature.^{31 32} By understanding the processes that have been used to establish links between trial registries and published trial reports, we may be able to provide guidance on how each of these processes can be used to identify a complete set of trials, supporting new methods that use clinical trial registries in systematic reviews.

The objective of this systematic review is to quantify the processes that have been used to link clinical trial registrations to their published results in studies that examined the completeness and accuracy of clinical trial reporting.

Methods

Inclusion and exclusion criteria

We will include all English-language studies that use one or more of the clinical trial registries included in the WHO International Clinical Trial Platform (WHO ICTRP),³⁸ to compare what was registered with what was published, determine the proportion of published trial reports that have been registered or the proportion of registered trials that have been published. Studies will be excluded if they do not report the number of clinical trials for which they identified links or if the study is describing a trial or reviewing clinical evidence.

Search strategy

Relevant articles will be identified by searching PubMed and EMBASE for studies that meet the inclusion criteria. These databases were selected because they are known to have good coverage of clinical research,^{39 40} and other databases typically used in systematic reviews of clinical evidence (such as the Cochrane Central Register of Controlled Trials) were irrelevant to the topic of the review. Owing to the atypical nature of the included studies, we designed the search strategy with the support of a medical research librarian, and it was designed to balance the number of articles returned by the broad terms covering the clinical trial registries by constraining the search using terms that were common to the set of relevant studies. We considered a set of 50 articles we knew met the inclusion criteria and used their titles, abstracts and keywords to define a search strategy that returned all 50 articles without returning an unmanageable number of irrelevant articles (table 1). We will additionally hand-search the reference lists of all included studies to identify any other articles that may have been missed by our searches. The complete steps for searching both databases are included in the online supplementary material.

Table 1.

Search strategy used for PubMed and EMBASE bibliographic databases

	Search terms
#1	Any registry from WHO ICTRP OR “trial registry”[Title/Abstract] OR “trial register”[Title/Abstract] OR “trial registries”[Title/Abstract] OR “trials registry” [Title/Abstract] OR “registry of clinical trials” [Title/Abstract]
#2	(“trial registration”[Title/Abstract] AND (discrepancy[Title/Abstract] OR discrepancies[Title/Abstract] OR consistency[Title/Abstract] OR inconsistency[Title/Abstract]))
#3	#1 OR #2
#4	(unregistered[Title/Abstract] OR non-publication[Title/Abstract] OR nonpublication[Title/Abstract] OR unpublished[Title/Abstract] OR published[Title/Abstract] OR (registered[Title/Abstract] AND (publication[Title/Abstract] OR clinical trial as topic [MeSH Terms]))) Note: For EMBASE, equivalent EMTREE is “clinical trial (topic)”.
#5	#3 AND #4
#6	(“outcome reporting bias”[Title/Abstract] OR “selective reporting”[Title/Abstract] OR “selective outcome reporting”[Title/Abstract] OR “missing outcome data”[Title/Abstract] OR “publication bias”[MeSH Terms] OR (“reporting quality”[Title/Abstract] AND publications[Title/Abstract])) Note: For EMBASE, the publication bias MeSH Term has no equivalent and is removed.
#7	#5 OR #6

ICTRP, International Clinical Trial Registry Platform; MeSH, Medical Subject Headings.

Two reviewers will evaluate the articles returned by the searches against the inclusion and exclusion criteria. Duplicate studies will be removed by automatically comparing digital object identifiers (DOIs) across the search results where possible, and by manually evaluating titles, authors and journal names for the remainder. In each of the two phases of screening for eligibility (title/abstract and then full text review), disagreements about inclusion will be resolved by a third author and by discussion, as needed.

Data extraction process

Data from studies will be extracted independently by both reviewers and then compared, reporting the level of agreement for each information element. The information to be extracted includes the following: (1) the number of trial registry entries examined or identified in the study; (2) the number of published trial reports examined or identified in the study; (3) the trial registry or registries used; (4) the purpose of the study (such as measuring publication bias, outcome reporting bias or the number of published trials that were registered); (5) the application domain; (6) the processes used to identify the links and (7) the proportions of the links found for each method if available.

In relation to the method for identifying the links, we categorise links as one of three types—automatic, inferred or inquired. Automatic links are those for which the unique identifier from the trial registry entry is used to identify links to published results without requiring any inference or manual work. Inferred links are those for which the investigators used the characteristics of the trials to search and reconcile links to or from published trial reports. Inquired links are those confirmed by contacting trial investigators or authors to identify the location of published results.

Data synthesis

Using the information extracted from the articles, we will pool the overall proportions of trials for which links were identified. As they represent different types of links between trial registries and published reports, studies that start from a cohort of trial registry entries and identify published results will be pooled separately from studies that start from a cohort of published trial reports and identify trial registry entries.

Heterogeneity in the overall number and proportion of links identified in these studies is expected to come partially from differences in the processes being used to identify links, the period in which the trials were completed and published (reflecting temporal changes in the policy and practice of trial registration and reporting) and the specific application domains (some conditions or interventions may be more likely to have registered trials published or published trials registered). To account for these differences in the overall pooled estimates, we will estimate the contributions of each of the three categories of linking processes to the overall estimates wherever the information is available. The result will include estimates of the proportions of links that can be automatically captured, the larger proportion that can be reliably identified when investigators search for and infer links, and the larger proportion that can be identified when investigators contact trial investigators for more information.

We are also interested in examining whether the processes for identifying links between registry entries and trial results have changed over time. To measure the differences over time, we will examine the trend in the proportion of links identified overall—as well as using each of the three categories of linking processes—by applying linear regression relative to the midpoints of the data collection periods specified in each of the studies.

Discussion and dissemination

To the best of our knowledge, this is first systematic review surveying the processes used to identify links between clinical trial registries and published clinical trial results. By aggregating the results of many trials in the area to estimate the proportions of links that can be identified through automatic linking, inference and inquiry, this systematic review is expected to advance the field in several ways. First, the systematic review will be used to determine whether differences in the processes for identifying links between registry entries and trial reports can partially explain differences in the results of existing studies of publication bias and outcome reporting bias, which in turn may be used to help standardise the way these studies are undertaken in the future. Second, by determining the proportions of trial registry entries that can be automatically linked to their results, we can help guide new systematic review technologies that rely on links to improve methods used in the identification of trials.

One limitation of the systematic review process is the exclusion of studies that are not published in English, which may mean that we miss some articles describing registries based in non-English speaking countries. As a substantial proportion of the studies that will be included are designed for purposes other than simply identifying the links between registries and published results, the description of the processes used to link the two may be limited, and this may limit our ability to determine the proportions of links captured automatically, by inference or by inquiry.

Our findings will be reported on the basis of guidelines from Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The results of this review will be submitted for publication in a peer-reviewed medical journal. Other forms of dissemination will include direct engagement with clinical trial registry developers.