An interactive dashboard to track themes, development maturity, and global equity in clinical artificial intelligence research

Interest in the application of artificial intelligence (AI) to human health continues to grow, but widespread translation of academic research into deployable AI devices has proven more elusive.^1,2 There is increasing recognition of limitations in how AI research is carried out, from methods of model validation that do not emulate real-world conditions,³ to characteristics of data⁴ and inadequate inclusion of researchers and populations from diverse global regions.⁵ Systematic reviews of clinical AI criticise widespread risk of bias and lack of downstream clinical utility, and research waste is an increasing concern.^6,7

One problem is the lack of a unifying perspective over the colossal-sized landscape of global AI research. Continual quantification of research characteristics can enable identification and monitoring of shortcomings in this heterogeneous landscape. However, the sheer quantity of published research (>1509000 papers on MEDLINE under broad terms; appendix p 1) makes this a substantial challenge. Literature database searches have poor specificity, and cannot directly identify original research in model development, or pinpoint research representing advanced stages of model validation. Literature reviews only describe a portion of research at a single timepoint, are laborious to conduct and reproduce, and are quickly outdated in a rapidly changing landscape.

In response to these requirements, we produced an end-to-end Natural Language Processing (NLP) pipeline that performs real-time identification, classification, and characterisation of AI research abstracts extracted from MEDLINE, outputting results to an interactive dashboard, creating a live view of global AI development. We identified four primary aims: first, to directly discover original research in clinical AI model development; second, to identify research at more advanced development stages using mature evaluation methodology, ie comparative evaluation of AI algorithms versus a reference standard⁸ or prospective real-world testing (appendix p 13); third, to map, in real-time, global distribution and equity in AI research production on a per-author basis; and fourth, to track the main active research themes across clinical specialties, diseases, algorithms, and data types.

Development was done using Python (version 3.8) and Tensorflow (version 2.5). To achieve the required performance, we employed transfer learning, using state-of-the-art Bi-directional Encoder Representations from Transformers NLP models with pre-training on medical corpuses.⁹ Models were fine-tuned on manually labelled abstracts indexed on MEDLINE before 2020 and tested prospectively on abstracts indexed after pipeline completion. The final pipeline and methods described are available in the appendix (pp 1–5, 13). In a prospective evaluation, the classifier for research discovery achieves an F1 score of 0·96 and Matthews correlation coefficient (MCC) of 0·94. The classifier for maturity achieves an F1 of 0·91 and MCC of 0·90. The multi-class classifier for labelling themes achieves a macro-average F1 of 0·97. When evaluated against publications discovered by recent systematic reviews, the pipeline correctly classified 98% for inclusion and maturity. Performance metrics are reported in the appendix (pp 8–12).

The dashboard allows all discovered research to be visualised by development maturity, medical specialty, data type, algorithm, research location, publication date, or different combinations of attributes. Datasets containing labelled abstracts and metadata are refreshed every 24 h and made available to download, as an aid to literature reviewers, or for reproducible analysis of research progress across any cross-section of characteristics.

Using dashboard datasets, we illustrate heterogeneity in research maturity across major specialties and diseases over the past decade using a horizon chart (appendix p 17).¹⁰ Respiratory medicine, breast cancer, and retinopathy demonstrate greatest production of mature research relative to total research production. Distribution of data type usage across major subspecialties are shown as heatmaps (appendix p 14), showing increased prevalence of mature validation methodology using radiomics (and other computer vision tasks) across all specialties. Notably, only 1·3% of research, and 0·6% of mature research, involved an author from a low to low-middle income country (as per World Bank definitions), with 93·6% of such research published after 2016 (appendix p 15). Live visualisations are found on the dashboard website.

While demonstrating state-of-the-art NLP performance, classifier limitations include imperfect accuracy compared with careful human reviewers (the trade-off against time required for manual characterisation). We use only MEDLINE due to their unique application programming interface. Finally, prediction using full articles could increase performance, but this was hindered by a paywalled access to most publications.

The interactive dashboard was published in November, 2021. Given its popularity and utility to date, we plan to continue enhancement of this resource. We consider immediate downstream use-cases to be analysis of drivers for AI maturity and translation, reviewing features of mature AI research, and ongoing characterisation of AI development in developing countries. Codes and data are made public, with the hope that functionality can be expanded in collaboration with the global AI community.