Research impact analysis of international funding agencies in the realm of allergy and immunology

Abbreviations

AMED‐PPAI

Practical Research Project for Allergic Diseases and Immunology of the Japan Agency for Medical Research and Development

EU

European Union

FA

Funding agency

FWCI

Field‐Weighted Citation Impact

MeSH

Medical Subject Headings

METI

Ministry of Economy, Trade and Industry of Japan

MEXT

Ministry of Education, Culture, Sports, Science and Technology of Japan

MHLW

Ministry of Health, Labour and Welfare of Japan

MRC‐HIU

Human Immunology Unit of the Medical Research Council

NIH‐HAI

Hypersensitivity, Autoimmune, and Immune‐mediated Diseases Study Section of the National Institutes of Health

R&D

Research and development

UK

United Kingdom

UMAP

Uniform Manifold Approximation and Projection

US

United States of America

CONFLICT OF INTEREST

TF and AI are employees of the AMED. YO and KA are scientific advisors of AMED. MTa is the Program Officer, and HI is the Program Supervisor of the AMED‐PPAI. TA (2015–2020) and YO (2018–2020) were employees of the AMED. SNi was the former Program Supervisor of the AMED‐PPAI (2015–2018). The other details about competing interests are provided separately.

To the Editor,

A longitudinal approach should be employed for research and development (R&D) on allergic and immunological diseases across all life stages. To strategically use limited public funds in promoting such R&D, their characteristics of long‐term research support and societal implementation should be considered. ¹ However, outcomes of the funding research evaluation have focused on conventional, shortsighted indicators. To determine the kind of indicators needed for the funding strategy, we compared the research impact of funding agencies (FAs) in the UK, US, and Japan, utilizing indices related to research substantiality ² and analyzing index words/abstracts connected with the national strategy for allergy and immunology. ³

We used AMEDfind—an open database of top‐down R&D projects funded by AMED—and selected 53 awards for a Practical Research Project for Allergic Diseases and Immunology (AMED‐PPAI) (Figure S1). 1053 papers with verified PubMed IDs were included. As the controls, we selected the Hypersensitivity, Autoimmune, and Immune‐mediated Diseases Study Section (NIH‐HAI), an immunology‐focused project in the Americas, and Human Immunology Unit (MRC‐HIU), that in Europe, extracting 373 US papers and 118 UK papers, published in 2015–2019, respectively (see Appendix S1 for all methods).

The Field‐Weighted Citation Impact (FWCI)—evaluating research paper quality—was highest for MRC‐HIU following NIH‐HAI and AMED‐PPAI (Table 1, Figure 1A). Although the international co‐authorship rate was lowest in the AMED‐PPAI, the annual trend showed a gradual increase (Figure 1B, Table 1). The number of top 10% most cited papers ² /value, evaluating funding efficiency, was highest for MRC‐HIU (Table 1).

TABLE 1.

Comparison of publications and societal impact of research funded by international funding agencies

	AMED‐PPAI	MRC‐HIU	NIH‐HAI	UK comparator	Mann–Whitney–Wilcoxon distribution p values
FWCI	1.92	3.45	2.48
No. of publications (2015–2019)	1053	118	373
Value of awards (million USD, 2015–2019)	23.6	8.86	35.3
No. of publications/value	37.3	9.35	7.09
International co‐authorship percentage	1.36	4.17	1.60
No. of top 10% most cited papers	10.2	5.3	3.9
No. of top 10% most cited papers/value	0.432	0.598	0.110
No. of awards	53			1376	N/A
Value of awards (million USD, 2013–2018)	20.9			521	N/A
Mean value/award	0.393			0.379	.71
Mean funding peirod (years)	2.90			3.00	.071
No. of intellectual properties/value	0.00473			0.00155	.039(*)
No. of publications/value	0.509			0.118	2.43E−13(***)
Ratio of CC–BY papers	0.350			0.486	N/A
Value of further funding/value	1.58			2.97	.0019(**)
No. of engagement activities/value	0.0909			0.100	.71

Abbreviations: AMED‐PPAI, Practical Research Project for Allergic Diseases and Immunology of the AMED; FWCI, Field‐Weighted Citation Impact; MRC‐HIU, Human Immunology Unit of the MRC; N/A, not applicable; NIH‐HAI, Hypersensitivity, Autoimmune, and Immune‐mediated Diseases Study Section of the NIH; No, number; USD, United States Dollar.

*p < .05, **p < .01, ***p < .001.

FIGURE 1.

Comparison of research impact of international funding agencies (FAs). Scholarly outputs from AMED‐PPAI, MRC‐HIU, and NIH‐HAI were analyzed using FWCI (A) and international co‐authorship ratio (B). UMAP with spectral clustering for dimension reduction of top 50 FWCI papers from three FAs (C, D) and relevance of each FA against twelve clusters (E). Cross‐tabulation of MeSH headings' list of Actions of Strategy 2030 with list of top 50 papers from each FA (F) (box size: the percentage of papers with related MeSH; box color: standardized Pearson residuals). Objectives of engagement activities were listed for AMED‐PPAI and UK comparator (G)

To characterize these outputs, we performed natural language analyses of the top 50 FWCI papers from three FAs and top 100 papers on this topic during 2015–2019 (Figure 1C–E). ⁴ Although all FAs produced mainly basic allergy/immunology study papers (e.g., clusters 0, 1, 2, and 9 in Figure 1D), AMED‐PPAI produced relatively more clinically relevant outputs (clusters 3 and 8). Further, we analyzed MeSH headings for relevance to national unique Strategic Outlook toward 2030 formulated by the Ministry of Health, Labour and Welfare of Japan (Figure 1F, Table S1). ³  While most projects were pathogenic studies, AMED‐PPAI funded research published more papers on precision medicine and host‐extrinsic factor relations (Action I‐2, I‐3). The Action II and III groups were not strongly represented.

To assess societal impact, we utilized ResearchFish—a widely used database of the FAs in Europe ⁵ —to randomly select 1376 UK‐funding awards and their reports, trend‐matched for the AMED‐PPAI (UK comparator) (Table S2). The number of intellectual properties and publications per value was higher for AMED‐PPAI, while the open access rate and new funding amount obtained were higher in the UK comparator (Table 1). Despite the similar number of engagement activities, their outreach targets differed (AMED toward public and media; the UK towards students, patients, and industry; Figure 1G).

The purpose of this study is not to compete for superiority among countries or FAs but to expand the possibilities for multidimensional interpretation of trends and characteristics of funded outcomes using multiple indicators rather than uniform one. Due to the limitation of open databases, we focused on three countries for the funding impact analysis, whereas scholarly output in this realm is also increasing in other countries and jurisdictions, including the EU (Figure S2). China's growth is particularly remarkable, and additional analysis is desirable with public funding status. ⁶ Furthermore, the indices used have different trends among countries, and their balance should be carefully considered to reflect each country's science and technology policies.

In conclusion, we conducted impact analyses from multiple perspectives, including indicators related to substantiality ² and index words profiling/clustering based on the national strategy. ³ These findings may inform international collaborative long‐term research that strategically leverages each research funding institution's strengths.

Supporting information

App S1