Issues regarding the Indexing of Adaptive Clinical Trial Articles

Abstract

Accurate, consistent indexing of biomedical publications is important for retrieving and evaluating evidence, that in turn can affect clinical decisions, evidence syntheses and research directions. Adaptive trials are arguably among the most complex clinical trial designs, being defined as trials in which the allocation or intervention given to subjects is not fixed in advance, but can vary according to the responses or characteristics of the participants. In the course of preparing to tackle the automated indexing of adaptive clinical trial articles, we manually examined the PubMed articles bearing the Adaptive Clinical Trial publication type to date and identified a surprisingly high proportion of erroneous and questionable indexing decisions. If one agrees that each article should be indexed according to its own study design (rather than the study design of the larger trial that it is linked to), then 46.7% of the articles were mis-assigned. If we consider that articles which presented partial results (e.g., one arm of a multi-arm trial) might be broadly deserving of being indexed as adaptive trials, then the adjusted error rate would still be 24.2%. These errors are worth discussing in detail because they have more general lessons to teach us – not only for biomedical end-users who want to utilize indexing terms for efficient retrieval, but for indexers who need to annotate articles accurately and consistently.

Introduction

Accurate, consistent indexing of biomedical publications is important for retrieving and evaluating evidence, that in turn can affect clinical decisions, evidence syntheses and research directions. Articles are indexed by the National Library of Medicine (NLM) according to one or more Publication Types, e.g., Review, Case Report, or Clinical Trial, and can be described using Medical Subject Headings (MeSH, MH) according to the study design(s) employed in the article, e.g. Case-Control Studies, Random Allocation, or Retrospective Studies (https://www.nlm.nih.gov/mesh/meshhome.html). Our team has carried out machine learning based automated indexing of biomedical articles for more than 50 different publication types and study designs (collectively, PTs) [1] and we are continuing to consider additional PTs to achieve comprehensive indexing. Having created methods for indexing an article as likely to be a Clinical Trial [2], and specifically indexing Randomized Controlled Trials [3, 4], we are now examining methods for indexing articles according to specific subtypes of clinical trials, e.g., phase I, II, III, IV, pragmatic trial, equivalence trial, and adaptive trial.

Adaptive trials are arguably among the most complex clinical trial designs, being defined as trials in which the allocation or intervention given to subjects is not fixed in advance, but can vary according to the responses or characteristics of the participants [5]. Such trials are increasingly common, having been stimulated by the Covid-19 pandemic and the need to be flexible as new possible treatments emerged and as the disease itself evolved in real time [6]. A type of adaptive trial design is the platform trial, in which multiple interventions are all compared against the same control group; as a practical matter, platform trials are considered adaptive, insofar as the list of possible interventions is often flexible, some dropped according to subject responses and new ones added during the trial. NLM newly added Adaptive Clinical Trial as a formal indexing term in 2018, and as of the time of our study, only 182 articles were indexed with that term.

A single clinical trial may give rise to a wide range of multiple publications – some report the overall clinical outcome of the trial, but some articles may only report a small subset of the overall trial. Others may carry out interviews with participants, measure genetic or biochemical markers, re-analyze data across multiple trials, and so on. Trial protocol articles describe the design of a trial in advance. In order to make sense of these diverse papers, two strategies are needed: First, one must have a way to link a given trial to all of its publications, which has motivated several investigators to devise automated linking methods [7–9]. Second, on the other hand, each article should be indexed on its own terms, that is, according to the publication type and study design mentioned in the article itself, regardless of the study design of the underlying clinical trial. This agrees with NLM’s indexing policy for publications.

In the course of preparing to tackle the automated indexing of adaptive clinical trial articles, we manually examined the PubMed articles bearing the Adaptive Clinical Trial publication type to date and identified a surprisingly high proportion of erroneous and questionable indexing decisions, which are worth discussing because they have more general lessons to teach us – not only for biomedical end-users who want to utilize indexing terms for accurate retrieval, but for indexers who need to annotate articles accurately and consistently.

Methods

Two of us (NRS and AMT) independently annotated all PubMed articles indexed as Adaptive Clinical Trial [Publication Type; PT] as of February 32, 2025. Annotations were resolved to consensus through discussion. Each was scored as either:

1. an Adaptive Clinical Trial article (i.e., presenting trial outcome results that involved adaptive or platform design);

2. a clinical trial article that presented partial results from an adaptive trial, but which did not itself reflect adaptive design (e.g., one arm of a multi-arm study), or that discussed a specific adaptive trial but did not present trial outcome data;

3. a trial protocol or overview of a specific adaptive trial;

4. an article which discussed adaptive trials more generally, e.g., simulated trials, discussions of trial methodology, reviews, or re-analyses of multiple trials;

5. a frank indexing error: articles which were not related to clinical trials at all;

6. a probable indexing error: authors claimed that the article presented results from an adaptive or platform trial, but the available text did not appear to support that claim.

Results

Out of 182 articles indexed as Adaptive Clinical Trial [PT], only 97 were actually scored as being an adaptive clinical trial. 41 were presenting partial results or discussing specific trials, 14 were protocol or overview articles, 11 were more general discussions of adaptive trials, 14 were frank errors, and 5 were probable errors. If one agrees that each article should be indexed according to its own study design (rather than the study design of the larger trial that it is linked to), then (182–97)/182 = 46.7% of the articles were mis-assigned! This is a surprisingly high error rate. If we consider that the 41 articles which presented partial results might be broadly deserving of being indexed as adaptive trials, then the adjusted error rate would be reduced to (182–97-41)/182 = 24.2%, still very high.

Many of the mis-assigned articles should have properly been indexed as “Clinical Trials as Topic”[MH] or “Clinical Studies as Topic”[MH], reflecting the fact that they are discussing trials rather than presenting clinical outcomes. This includes trial protocols, for whom NLM indexing policy would assign “Clinical Trial Protocol”[PT], the vast majority of which are also indexed as “Clinical Trials as Topic”[MH] and “Clinical Studies as Topic”[MH]. We found that a total of (14+11) = 25 articles should have been indexed “as topic” instead, whereas only 6 of the 182 articles were actually dual indexed as “Clinical Trials as Topic”[MH] or “Clinical Studies as Topic”[MH], and none at all were dual indexed as “Adaptive Clinical Trials as Topic”[MH].

The frank errors were disproportionately indexed in the first few years after introduction of the indexing term (e.g. 9 of the 14 were published in or before 2019) which may indicate some uncertainty as the indexing process was developed. However, as shown in Table 1, these articles include surveys given to faculty, genetic sequencing studies, and even materials science! It is hard to imagine how such a crazy mix of articles received assignments as Adaptive Clinical Trial.

Table 1.

Frank errors in indexing of Adaptive Clinical Trial articles.

PMID	Title	Pub Date
27437657	Incidence and Determinants of Traumatic Intracranial Bleeding Among Older Veterans Receiving Warfarin for Atrial Fibrillation.	2016
27601076	Functional mechanisms underlying pleiotropic risk alleles at the 19p13.1 breast-ovarian cancer susceptibility locus.	2016
29271472	The diversification of inequality.	2019
29439746	Antimicrobial Stewardship Program Implementation of a Quality Improvement Intervention Using Real-Time Feedback and an Electronic Order Set for the Management of Staphylococcus aureus Bacteremia.	2018
30221411	The interplay between chemical speciation and physiology determines the bioaccumulation and toxicity of Cu(II) and Cd(II) to Caenorhabditis elegans.	2019
30550598	Comparison of classical and transgenic genetic sexing strains of Mediterranean fruit fly (Diptera: Tephritidae) for application of the sterile insect technique.	2018
30624488	Psychiatric care for abnormal children at the Hospital Col√¥nia Sant’Ana (Santa Catarina, Brazil, 1940s).	2018
30978187	The relative contribution of color and material in object selection.	2019
31007114	How are clinician-educators evaluated for educational excellence? A survey of promotion and tenure committee members in the United States.	2019
34229803	Direct Writing on Paper Substrate to Prepare Silver Electrode Structures for Flexible Sensors.	2021
34643460	Type 2 diabetes mellitus potentiates acute acrylonitrile toxicity: Potentiation reduction by phenethyl isothiocyanate.	2021
35210402	Quantitative electrophysiological assessments as predictive markers of lower limb motor recovery after spinal cord injury: a pilot study with an adaptive trial design.	2022
36951095	Identifying regions of interest in mammogram images.	2023
37767576	An unusual case of thalassemia intermedia with inheritable complex repeats detected by single-molecule optical mapping.	2024

These are errors of commission. What about errors of omission, when articles that deserve to be indexed as Adaptive Clinical Trial fail to receive that assignment? Given that the indexing term was only introduced in 2018, the fact that earlier articles may not have been indexed as such is not necessarily an oversight nor an error. However, to estimate informally how many adaptive clinical trial articles may be in PubMed overall, we carried out a PubMed search to identify any PubMed articles indexed as Clinical Trial[PT] which mentioned either “adaptive trial” OR “platform trial” and were not already indexed as “Adaptive Clinical Trial”[PT]. Only 34 articles were retrieved, the majority published before 2018, suggesting there are not a lot of relevant articles missed in the current indexing effort. Retrospective indexing of publications, as carried out by NLM in many cases, may hopefully assign the earlier articles in due course.

As an update, before submitting this paper, we double-checked the latest status of indexing (as of March 9, 2025) and found that three of the articles that we had judged “as topic” had their indexing changed -- removing Adaptive Clinical Trial[PT] and adding “as topic” MeSH terms. This indicates that NLM is carrying out at least a limited extent of error correction on articles.

Discussion

Adaptive clinical trials have complex designs, and the publications that are linked to them appear to represent a challenge for indexers, since our analyses show a surprisingly high error rate for NLM indexing of these articles. Depending on whether one makes a strict or narrow interpretation of indexing policy, about a quarter to one-half (24.2% to 46.7%) of assignments were inappropriately made. This is in contrast to our earlier estimates of NLM indexing accuracy of randomized controlled trials (RCTs), in which only about 5% of articles marked as RCTs were judged not to be indexed appropriately, and another 3% additional RCT articles failed to receive RCT indexing [3].

The largest category of indexing errors involved articles which discussed trials (e.g., describing trial designs, proposing trial design methodology, or reviewing multiple trials) but did not present the clinical outcomes of a specific adaptive clinical trial. These articles would properly be indexed as “Adaptive Clinical Trials as Topic”[MH]. An additional category were frank errors (Table 1) for which no obvious explanation is forthcoming. There are several lessons to be learned here:

First, for biomedical end-users seeking to retrieve the full range of published articles related to a specific adaptive clinical trial, or related to this class of trials in general: Is it really so bad to include extra articles in a PubMed search, as long as most of the appropriate articles are retrieved? Perhaps not when the class includes only 182 articles, but indexing schemes must apply consistently to all article types, which may include 1,000, 5,000 or 50,000 publications. Better that indexing is not overly inclusive. However, it is indeed important to be able to identify the entire assemblage of articles that are generated by, or related to, a given trial. For that purpose, it would suffice for authors to mention the trial registry number in each publication. As well, one can employ linking tools such as Trials to Publications [9] which attempt to identify publications linked to registered trials even when registry trial numbers are not explicitly given, or missing from the abstract and only given within the full-text of the article [10].

Second, for those of us interested in automated indexing of adaptive trials, one must be careful to remove the “as topic” articles and the frank errors from the overall set of PubMed indexed articles, before using them as a training set for “Adaptive Clinical Trial”. On the other hand, we note that there are currently 552 PubMed articles indexed as “Adaptive Clinical Trials as Topic”[MH], several times more than are indexed as Adaptive Clinical Trial [PT]. This suggests that is may be worthwhile to carry out automated indexing for “Adaptive Clinical Trials as Topic” as well.

Finally, a limitation of this study is that the indexing criteria and notes given to NLM indexers are not public, so at present we are left to infer these from the observed indexing behavior. Definitions for publication types and study designs are given in the MeSH database (https://www.nlm.nih.gov/mesh/meshhome.html) but these are clearly not detailed enough for indexers. For those who maintain bibliographic databases and annotate articles with indexing terms and other metadata, permitting and incorporating public feedback regarding detection of errors would improve the accuracy of indexing. Perhaps current bibliographic databases resemble static data resources, but in the future, they could instead interface dynamically with the users and become interactive.

Abbreviations

NLM

National Library of Medicine

MH

Medical Subject Heading

MeSH

Medical Subject Heading

PT

publication type

Funding Statement

Supported by NIH grant 1R01LM014292-01. Funder had no influence on the study, its design, or its publication. Neither the authors nor their institutions at any time received payment or services from a third party for any aspect of the submitted work.