Evaluating adherence to the International Committee of Medical Journal Editors’ policy of mandatory, timely clinical trial registration

Abstract

Objective

To determine whether two specific criteria in Uniform Requirements for Manuscripts (URM) created by the International Committee of Medical Journal Editors (ICMJE)—namely, including the trial ID registration within manuscripts and timely registration of trials, are being followed.

Materials and methods

Observational study using computerized analysis of publicly available Medline article data and clinical trial registry data. We analyzed a purposive set of five ICMJE founding journals looking at all trial articles published in those journals during 2010–2011, and data from the ClinicalTrials.gov (CTG) trial registry. We measured adherence to trial ID inclusion policy as the percentage of trial journal articles that contained a valid trial ID within the article (journal-based sample). Adherence to timely registration was measured as the percentage of trials that registered the trial before enrolling the first participant within a 60-day grace period. We also examined timely registration rates by year of all phase II and higher interventional trials in CTG (registry-based sample).

Results

To determine trial ID inclusion, we analyzed 698 clinical trial articles in five journals. A total of 95.8% (661/690) of trial journal articles included the trial ID. In 88.3% the trial-article link is stored within a structured Medline field. To evaluate timely registration, we analyzed trials referenced by 451 articles from the selected five journals. A total of 60% (272/451) of articles were registered in a timely manner with an improving trend for trials initiated in later years (eg, 89% of trials that began in 2008 were registered in a timely manner). In the registry-based sample, the timely registration rates ranged from 56% for trials registered in 2006 to 72% for trials registered in 2011.

Discussion

Adherence to URM requirements for registration and trial ID inclusion increases the utility of PubMed and links it in an important way to clinical trial repositories. This new integrated knowledge source can facilitate research prioritization, clinical guidelines creation, and precision medicine.

Conclusions

The five selected journals adhere well to the policy of mandatory trial registration and also outperform the registry in adherence to timely registration. ICMJE's URM policy represents a unique international mandate that may be providing a powerful incentive for sponsors and investigators to document clinical trials and trial result publications and thus fulfill important obligations to trial participants and society.

Background and significance

Clinical trial registries attempt to increase transparency of research enterprise by documenting all past and current clinical trials.¹ All clinical research stakeholders, including subjects, funders, scientists, publishers, clinicians, patients, and taxpayers, have an interest in seeing the results of research published. As a result, explicit links between trials and resulting journal articles are becoming increasingly important and an important topic for the field of clinical research informatics. Efforts to document all clinical trials are greatly reinforced by journal requirements for submitted manuscripts. The International Committee of Medical Journal Editors (ICMJE) maintains a policy for Uniform Requirements for Manuscripts (URM), which states that all articles reporting a clinical trial must register the trial in a trial registry and clearly state the trial registry identifier within the abstract of the manuscript.^{2 3} In addition to registration itself, the URM policy requires timely registration, meaning that trial registration must occur before enrolling the first participant.⁴ The URM policy was initially created by 14 founding journals of ICMJE; as of August, 2012, the ICMJE website listed 1209 other journals that also officially endorse it.

Several other initiatives aim to increase transparency in the conduct of clinical trials.⁵ Viergever and Ghersi suggested introducing standards for clinical trial registries.⁶ The WHO created a meta-registry platform to enable a single search point for several federated registries.⁷ Journal editors and investigators continue to evaluate reporting practices of existing clinical trials^{8 9} and develop new ones, such as the CONSORT statement.¹⁰ In our own work investigating the accuracy of the trial-publication link,¹¹ we have shown that published articles often fail to properly reference appropriate trial IDs.

Objective

We set out to investigate, on a purposive sample of journals, the rate of adherence to two aspects within the URM policy. We measured the degree to which trial ID is included within a journal article and we examined the frequency with which clinical trial articles mention a registry trial ID (criterion: inclusion of trial ID). For trials referenced in these articles, we assessed whether the trial registration was carried out before participant recruitment (criterion: timely registration). To our knowledge, this is the first study to look at the publishers’ aspect of the URM policy and analyzing the clinical trial registration using a journal-based sample based on the set of all articles published by a set of journals. The scope of our study was also influenced by our emphasis on the informatics of structured trial–publication links and adherence to existing policy aspects that aim to generate such links.

Methods

Sample selection

To obtain a set of trial journal articles for analysis, we first selected a number of journals to investigate and then identified all clinical trial articles published in those journals during a restricted period of time. Ideally, we would pursue a heterogeneous set of journals and compare factors associated with URM policy adherence; however, our scope was to arrive at a generic journal-based methodology and produce a benchmark assessment of adherence for a group of exemplary journals.

We chose to investigate ICMJE's founding journals since they created the URM policy and presumably have had the most time to adopt and follow the guidelines. We further limited journal selection to include only those with a 2011 impact factor of ≥10.0. Owing to differences in the number of trials articles and their proportion to all published articles, we analyzed the journals as a group and we did not seek to compare the performance of individual journals.

We retrieved all articles published in these journals between January 1, 2010 and December 31, 2011 and then selected only articles that reported clinical trial results (trial article). We refer to this set of articles throughout this manuscript as a journal-based sample.

Although some journals have a designated ‘Clinical Trial Article’ category, we found these categorizations to be inconsistent across the journals studied. We therefore used medical subject headings (MeSH) assigned by the National Library of Medicine in the Medline database. Specifically, we included all articles for which the Medline field of ‘Publication Type’ included the MeSH term ‘Journal Article’ and either the MeSH term ‘Clinical Trial’ or a more specific MeSH term (see online supplementary appendix A for the full set of terms used); however, we excluded publications indexed with the MeSH term ‘Clinical Trial Phase I’ since some journals and the US Food and Drug Administration Amendments Act of 2007 (FDAAA) do not require trial registration for articles reporting such trials. We also excluded articles for which the only publication type registered was ‘Multicenter Study’, since a preliminary analysis showed that such articles do not necessarily report clinical trials. Finally, we also excluded journal articles indexed with the publication type ‘Comment’. Online supplementary appendix A contains the actual queries used.

To evaluate the degree to which our automated trial article selection criteria indeed select trial articles which would warrant trial registry registration, we evaluated all articles without a trial ID link (using the full text of the article) to determine whether the studies which they report were interventional human-subject studies that would normally warrant registration in a trial registry (human trial article type).

Criterion 1: inclusion of trial ID

To evaluate the rate of articles which properly include a trial ID, we had designed a method to extract the trial ID linked to a given trial article. Since July 2005, the Medline article indexing process stores trial IDs in the Medline field called secondary identifier (SI). The secondary identifier field is used for references to trial IDs and may also store accession numbers to various databases of molecular sequence data, gene expression, or chemical compounds. The Medline database extracts trial IDs for the two largest trial registries: ClinicalTrials.gov (CTG) and the International Standard Randomized Controlled Trial Number Register (ISRCTN). We used automated methods to identify trial IDs within articles, supplemented by manual review if no trial ID was found initially. We used the PubMed application programming interface (API) to extract the SI fields for all articles in our journal-based sample.

From the SI field, we obtained the registry name and trial ID (eg, ISRCTN25072883 or NCT00493922). We use the term trial article with SI-linked trial ID for a trial article that has one or more a trial IDs recorded within Medline's SI field. For articles that did not contain any trial links within the SI field, we manually investigated first, the article abstract for references to registries not covered by the Medline indexing process (eg, the Australian New Zealand Clinical Trials Registry). Although the URM policy states that the trial ID must be included in the abstract, we also looked for cases in which the article mentions a trial ID by manually reviewing the full text of the article. This manual review gave us additional opportunity to determine whether the article described a human trial. Articles with trial ID links discovered during manual review were classified as trial articles with free-text-linked trial ID. Adherence to the inclusion of trial ID criterion was determined as a percentage of trial publications with SI-linked or free-text-linked trial ID out of all trial publications in the journal-based sample.

Criterion 2: timely registration

While the analysis of registration adherence depended on the journal articles with links to trial registration records as the unit of analysis, the assessment of timely registration considers the trial registration records themselves. We therefore examined the trial registration records using the journal-based sample and expanded the study to include a much larger set of trial registration records.

Journal-based sample

In the journal-based sample, we used all trial registrations that were referenced in the SI fields of the Medline citations of the journal article set described above with some additional restriction: (1) trial articles that have exactly one valid trial ID, (2) the linked trial is registered in a registry that uses an API that supports automated retrieval of trial registration date and trial start date, and (3) the corresponding trial registration records have valid trial start dates. The only trial registry that provides a comprehensive API is CTG. As a consequence, we limited our analysis to articles linking to trials within CTG.

Registry-based sample

In the second sample, we used a much larger set of trials that we analyzed for timely registration.

Once again, we turned to CTG because of its support for automated access to trial attributes. Our registry-based sample comprised all interventional trials of phase II or higher with a trial start date present within CTG and with dates of registration between January 1, 2006 and December 31, 2011. For trial registration date, we used the CTG first received date field, which is system generated and represents the date when the record was first given to CTG. CTG also provides the trial start date, defined as the month and year when enrollment to the protocol began.

In both the journal-based and registry-based samples, the adherence to timely registration was determined as the percentage of trials that were registered before enrolling their first participants. Although USA law (FDAAA) allows a 21-day grace period,¹² we allowed a 60-day grace period for multiple reasons. First, to account for current month-based reporting of trial start date within CTG (eg, December 2007 instead of an exact start date; in our analysis we converted registration month into the last day of the month: December 31, 2007). Second, to allow additional time for existing data quality assurance processes applicable to trial registration and to analyze substantial trial registration delays. The 60-day boundary was determined by a pilot analysis of trial registration delays and expert consensus.

Results

Five journals met our initial selection criteria: New England Journal of Medicine, The Lancet, JAMA, Annals of Internal Medicine, and the British Medical Journal. These five journals contained a total of 6504 articles in 2010 to 2011, of which 698 met our criteria for being considered trial articles. Table 1 lists total article counts, trial article counts (using MeSH article type keywords) and trial articles with SI-linked trial ID by journal. It shows the initial counts based on automated methods and before any manual review of the abstract or full text was performed.

Table 1.

Journals investigated and counts of all articles in years 2010–2011, trial articles (using MeSH keywords) and trial articles with SI link

Journal	Count of all articles	Count of trial articles (using MeSH keywords)	Count of trial articles with SI link*
Annals of Internal Medicine	754	47	38
British Medical Journal	1649	111	80
JAMA	1081	91	80
The Lancet	1577	190	176
New England Journal of Medicine	1443	259	242

*The count reflects the number of articles where trial ID is present in the Medline SI field.

SI, secondary identifier.

Criterion 1: inclusion of trial ID

Our automated method, which used the Medline SI field, identified trial ID links in 616 of the 698 trial articles (88.3%). All the links were either to the CTG or ISRCTN Registry (eight articles contained links to both registries). Manual review of the abstracts of the remaining 82 identified an additional 15 articles in which a CTG or ISRCTN trial number was present in the abstract but had not been included in the Medline SI field (eg, PMIDs: 20124231, 21315441). These 15 articles were classified as articles with a free-text linked trial ID. In 29 other articles, we found trial IDs from other ICMJE-approved registries (eg, Australian New Zealand Clinical Trials Registry or Chinese Clinical Trial Registry) and those articles were also classified as trial articles with free-text-linked trial ID. In the remaining articles we proceeded to full-text manual review seeking additional valid registry trial IDs and evaluation of human trial article type. In one article we found a valid trial ID in the body of the article that was not present in the abstract which we also added to the category of trial articles with free-text-linked trial ID. The compliant articles (totaling 661 articles) consisted of 616 trial articles with SI-linked trial ID plus 45 trial articles with free-text-linked trial ID. We found a total of 29 trial articles that did not contain any trial ID (eg, PMIDs: 21109302, 21612469)

Looking at the denominator, we found eight articles that did not describe human trials, despite being tagged as trial articles by PubMed. These included, for example, N-of-1 trials (PMIDs: 21148220, 22187187), trials investigating non-human subjects (PMIDs: 22108262, 21081600) or articles on a statistical topic relevant to clinical trials (PMID: 21300711). We removed those eight articles from the original denominator of 698 for calculations of the adherence to the first criterion. The overall rate of trial ID inclusion was therefore 95.8% (661/690).

Criterion 2: timely registration

Journal-based sample

The journal-based sample for assessing timely registration started with a set of articles with trial ID links within the Medline SI field (616 articles) and applied additional inclusion criteria such as (1) linked to the CTG registry with an advanced API, (2) linked to exactly one and valid trial ID, and (3) valid trial start date. Within the initial set of 616 trial-linked articles, we found that 489 had links to CTG, with 459 articles having links to exactly one trial. Of those 459 linked trials registrations, eight trials were excluded because of invalid linked trial ID (four trials) or blank start dates (four trials). Therefore, the final analyzed sample contained 451 article–trial pairs. Of the 451 trials, 272 (60%) were classified as having timely registration (trial registration date before 60 days after the trial start date).

Because publication of clinical trial results may take several years, timely registration requirement was only introduced in September 2005, and because trial registration cannot be enforced or corrected retroactively, we analyzed all trials in the journal-based sample (articles published in 2010–2011) by start date of the linked trial in order to better understand the results that would be less affected by these constraints. Table 2 shows the percentage of timely registered trials in the journal-based sample, arranged by the starting year of the associated trial. The table shows increasing adherence to timely registration of trials published in the five analyzed journals from 46% in 2005 (based on 57 trials) to 89% in 2008 (based on 57 trials). Subsequent years have inadequate numbers of articles (19 in 2009 and four in 2010) to assess whether this trend is continuing.

Table 2.

Timely registration rate in the journal-based sample arranged by trial start year

Trial start year*	Total trials	Trials with timely registration (%)
2000	9	2 (22)
2001	14	3 (21)
2002	11	4 (36)
2003	34	5 (15)
2004	42	9 (21)
2005	57	26 (46)
2006	89	71 (80)
2007	87	72 (83)
2008	57	51 (89)
2009	24	19 (79)
2010	4	4 (100)

*The journal-based sample included trial result articles about 23 trials that started between 1987 and 1999. Because ClinicalTrials.gov registry was first available in early 2000, trials initiated before registry release could not have been timely registered and are therefore not shown in the table.

Registry-based sample

As of May 2012, the CTG registry contained a total of 126 191 trials. Of those, 43 791 trials met our inclusion criteria (interventional trials of phase II or higher, non-empty start date, registered between January 1, 2006 and December 31, 2011). Table 3 shows the timely registration adherence by trial registration year. The adherence ranged from 56% (of all 6431 trials registered in 2006) to 72% (of all 7243 trials registered in 2011). Again, owing to the introduction of the timeliness requirement in September, 2004, we also analyzed the registry-based sample by trial start year. Table 4 shows timely registration adherence by trial start year, which again increased from 58% in 2006 (based on 5505 analyzed trials) to 93% in 2011 (based on 5719 analyzed trials) and even 100% in 2012 (based on 874 trials). The analysis in table 4 starts with year 2006 because the registry-based sample was limited to trials registered during 2006–11 and trials starting before January 1, 2006 would not be expected to be registered in a timely way.

Table 3.

Timely registration adherence in the registry-based sample (arranged by trial registration year)

Trial registration year	Total trials	Registered within 60 days (%)
2006	6431	3604 (56.04)
2007	7013	3990 (56.89)
2008	8285	4899 (59.13)
2009	7523	5063 (67.30)
2010	7296	5210 (71.41)
2011	7243	5243 (72.39)

Table 4.

Timely registration adherence in the registry-based sample (arranged by trial start year)

Trial start year	Total trials	Registered within 60 days (%)
2006	5505	3208 (58.27)
2007	5947	3831 (64.42)
2008	6500	4695 (72.23)
2009	6421	4949 (77.08)
2010	6129	5041 (82.25)
2011	5719	5305 (92.76)
2012	874	874 (100)
2013	9	9 (100)
2015	2	2 (100)

Discussion

Adherence to URM policies

In our study of a purposive sample of journals that were the founding members of the ICMJE that introduced the URM policy, we found that automated methods could detect trial registration in 88% of cases; manual review showed that the rate was closer to 96%. While this rate is laudable, our study points out that increased vigilance is required by authors and publishers to assure that article–trial links are properly captured, so as to support a variety of automated techniques that attempt to process such information on a large scale. The resulting trial–publication link, created at the time of manuscript submission, is peer-reviewed, more precise¹¹ and greatly increases the value of clinical trial registries for evidence-based medicine reviews.

Although our results indicate a high compliance with trial ID inclusion in five high-impact journals, accurate trial–publication data depend on numerous other journals, not analyzed in this study. A prior study of ours,¹¹ estimating the negative predictive value of (absent) trial–publication links, showed that 44% of trials with no linked result publications had unlinked relevant publications. Taken together with the results of this study, we can conclude that trial-ID inclusion compliance in journals outside our investigated set of journals is probably unacceptably low. This study differs from our previous study in the following points: the previous study¹¹ (focused on precision and negative predictive value) examined the accuracy of trial–publication links present either in the article abstract (abstract trial–publication link) or trial registry record (registry trial–publication link). It used two random samples (405 trials for the assessment of precision and 50 trials for negative predictive value) from a pool of 14 260 trials in the CTG registry. The study conclusions were that (1) the abstract link is more prevalent and more precise than the registry link; and that (2) that many trials have unlinked result publication.

Our present study analyzes only abstract trial–publication links and focuses on trial ID inclusion and timely registration. This study reaches its conclusions from a sample of journal articles (as opposed to trial registry records). The registry-based sample does partly use clinical trial registry data; however, the sample size is larger (43 791 trials) and uses different inclusion criteria (interventional, phase 2+ trials registered during 2006–2011, regardless of whether a linked publication exists) as opposed to the inclusion criteria used in the link accuracy study (14 260 trials with completion date between September 2005 and December 2008).

The result of our analysis of timely trial registration (overall 60% adherent trials in the journal-based sample) must be viewed from the perspective of the associated trial start year. The trend of increasing timely registration (table 2) shows good progression towards full compliance. The five analyzed ICMJE founding journals are clearly diligent in enforcing the URM policies and achieving higher timely registration rates in comparison with the registry-based sample (tables 2 and 4). Our primary objective was to measure the adherence of the five journals as a group; however, while doing so, we also considered general automated methods and whether the existing informatics infrastructure supports automation of such assessment in any journal.

Two previous studies have examined timely registration using registry-based samples,^{9 13} but ours is the first to investigate a journal-based sample. Our results differ somewhat from the results reported by Califf et al,⁹ who found that 48% of all interventional trials registered between October 2007 and September 2010 were registered in a timely manner. This difference may be because we limited our analysis to include only interventional trials of phase 2 and higher (Califf's study included phase 1 studies) and we allowed a 60-day grace period to account for the limitation of the month-based trial start date reporting.

Limitations

Some aspects of our methods might have biased our finding of 96% adherence to trial registration. First, we only analyzed five journals that were founding members of ICMJE and have a high impact factor. Registration adherence in journals that (1) do not formally adopt the URM policy of mandatory and timely trial registration or (2) journals with lower impact factor may be different. Possibly, higher registration in the five analyzed journals may simply reflect a correlation between the high caliber of investigators who produce high- quality studies worthy of submission to journals with a high impact factor, rather than an effect of the URM policy itself.

It was outside the scope of our study to investigate in detail non-adherence factors at the article, journal or trial level. Instead, the focus was to produce benchmark results for a group of exemplary journals.

Second, we relied on Medline MeSH article type keywords to identify trial articles. It is possible that not all trial articles were properly assigned a clinical trial article type and thus such articles would not be included in our analysis. Also the reverse problem is possible—namely, that not all articles tagged as clinical trial article type were indeed reports on clinical trials. The latter problem can be partially clarified by data from our manual review of the full text of 38 articles with missing trial ID in the abstract, which showed that eight articles (out of 698 total) did not actually represent human clinical trials. This represents a rate of 1.1% (8/698) of falsely predicted trial articles (using trial article prediction based on Medline classification) from all predicted trial articles, which we consider acceptable.

Third, in our automated methods, we relied on the Medline citation process to provide the associated trial ID. While the SI field contains trial IDs for only two registries (CTG and ISRCTN), our manual review was able to identify other registrations in an additional 4% of articles. The manual review might have missed some references in articles without such a link or the SI field might have contained a false-positive link. The latter was evaluated in our previous study¹¹ and was found to be 100% precise. We also limited our analysis to mere presence of the trial registration link. It was outside the scope of our study to analyze discrepancies between the registry study record and the publication study description. This has been examined by previous studies.^{8 14}

Fourth, our registry-based study of registration timeliness was limited to a single registry (CTG). While this registry is the largest by far, with over 130 000 entries, the next four largest registries (the EU Clinical Trials Registry, ISRCTN, the Japan Primary Registries Network and the Australian New Zealand Clinical Trials Registry), when taken together, contain 46 000 entries (counts are as of August 2012 and may contain duplicate entries). Online supplementary appendix B contains counts of registered trials in the registries listed in the URM policy. We did not examine the timeliness of registration in these latter registries owing to the inaccessibility of data through automated means. Thus, we cannot comment on compliance of trials in these registries. However, CTG was by far the most frequently used registry referenced by the articles in our sample and thus the compliance rate found there provides a general estimate of the adherence of all articles.

Finally, we conclude that adherence to timely registration is higher in the journal-based sample than in the registry-based sample; however, this comparison (tables 2 and 4) is limited since the trial inclusion criteria used in those two samples have some differences, such as allowing phase 1 interventional and observational trials in the journal-based sample and limiting the registry-based sample only to trials registered during 2006–2011.

Implications

The requirement for timely trial registration serves three purposes: (1) other investigators are informed about newly initiated trials (to avoid conducting redundant studies), (2) patients can learn about trials in which they might participate, and (3) trials are not preferentially registered based on their outcomes. Our results show that ICJME founding journals reasonably enforce the URM policy requiring trial ID inclusion and timely trial registration. Our methodology points to a possible automated quality control mechanism once Medline indexing of trial articles has occurred. Further, existing Medline indexing processes could be augmented by natural language processing techniques targeting clinical trial IDs and extended to cover all WHO primary clinical trial registries. Structural link between trials and articles could also be achieved by advanced natural language processing targeting large subsets of the Medline database such as semantic Medline¹⁵ or Biomedical Knowledge Repository.¹⁶

Adherence to URM requirements for registration and trial ID inclusion increases the utility of PubMed and links it in an important way to clinical trial repositories. This new integrated knowledge source (which we refer to as the trialome¹¹) can facilitate research prioritization, creation of clinical guidelines, and precision medicine.

Conclusion

Our study investigated the degree to which editors of a limited set of journals hold their authors accountable to the URM policies. While the requirement to register trials is largely being met in the sample of journals we investigated (96%), the requirement for timeliness of registration, which may be influenced by post-study decisions by investigators about whether to publish trial results, requires additional reinforcement. ICMJE's URM policy represents a unique international mandate that may be providing a powerful incentive for sponsors and investigators to document clinical trials and trial result publications and thus fulfill important obligations to trial participants and society.