Skip to main content
NCBI home page
Resuscitation Plus logoLink to Resuscitation Plus
. 2024 Feb 22;18:100584. doi: 10.1016/j.resplu.2024.100584

Using generative artificial intelligence in bibliometric analysis: 10 years of research trends from the European Resuscitation Congresses

Nino Fijačko a,b,c,⁎, Ruth Masterson Creber d, Benjamin S Abella e, Primož Kocbek a,f, Špela Metličar a,g, Robert Greif b,h,i, Gregor Štiglic a,j,k
PMCID: PMC10899017  PMID: 38420596

Abstract

Aims

The aim of this study is to use generative artificial intelligence to perform bibliometric analysis on abstracts published at European Resuscitation Council (ERC) annual scientific congress and define trends in ERC guidelines topics over the last decade.

Methods

In this bibliometric analysis, the WebHarvy software (SysNucleus, India) was used to download data from the Resuscitation journal's website through the technique of web scraping. Next, the Chat Generative Pre-trained Transformer 4 (ChatGPT-4) application programming interface (Open AI, USA) was used to implement the multinomial classification of abstract titles following the ERC 2021 guidelines topics.

Results

From 2012 to 2022 a total of 2491 abstracts have been published at ERC congresses. Published abstracts ranged from 88 (in 2020) to 368 (in 2015). On average, the most common ERC guidelines topics were Adult basic life support (50.1%), followed by Adult advanced life support (41.5%), while Newborn resuscitation and support of transition of infants at birth (2.1%) was the least common topic. The findings also highlight that the Basic Life Support and Adult Advanced Life Support ERC guidelines topics have the strongest co-occurrence to all ERC guidelines topics, where the Newborn resuscitation and support of transition of infants at birth (2.1%; 52/2491) ERC guidelines topic has the weakest co-occurrence.

Conclusion

This study demonstrates the capabilities of generative artificial intelligence in the bibliometric analysis of abstract titles using the example of resuscitation medicine research over the last decade at ERC conferences using large language models.

Keywords: Emergency medicine, European Resuscitation Council, Congress, Bibliometrics analysis, Generative artificial intelligence

Introduction

Bibliometric analysis has been previously conducted to evaluate research trends in resuscitation science. It is defined as the application of mathematical and statistical methods to critically assess publication activity in different research areas.1 Over the past decade, management of ventricular fibrillation and targeted temperature management have received considerable research attention, but recently palliative care, extracorporeal membrane oxygenation and brain injury have generated increased activity.2 Identifying relevant and quality information is challenging as the quantity and volume of scientific literature is increasing.3 Chat Generative Pre-trained Transformer (ChatGPT) (Open AI, USA) can be used as a tool for conducting bibliometric analysis.4

ChatGPT is an artificial intelligence tool that incorporates a sophisticated language model, trained on a large amount of data from the web and other sources, which enables it to automatically generate text for various purposes - answering questions and performing different tasks that require understanding the context and using natural, human like language.5

The aim of this study was to demonstrate “proof of concept” to perform bibliometric analysis using ChatGPT to extract quantitative information from large amounts of text and present research trends in resuscitation science from the European Resuscitation Council (ERC) annual scientific congresses over the past decade.

Methods

This study was conducted in four phases between April and May 2023. In the first phase, we used a licensed version of web scraping software called WebHarvy (SysNucleus, India).6 This software allows users to download data from various websites of interest in a systematic fashion. We downloaded abstract titles, types of presentation, and first author affiliation from the ERC annual scientific congresses available on the website of the journal Resuscitation. In the second phase, two experts in emergency medicine created definitions for all eleven ERC 2021 guidelines topics.7 These definitions were formulated by using keywords from the ERC 2021 guidelines unique to each ERC guideline topic,8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 as detailed in Supplement A. Subsequently, in the phase three, we used programming language Python (Python Software Foundation, Wilmington, USA), version 3.9, where formulated definitions were used as an input for ChatGPT-4 application programming interface (API)20 with the goal to classify abstract titles into one or more ERC 2021 guideline topics. The function call details are in Supplement B which includes the used prompt for the ChatGPT-4 API. During the fourth phase, the categorization of ten abstract titles, randomly selected and aligned with ERC 2021 guidelines topics by the ChatGPT-4 API, was compared against classifications conducted manually by an expert as described above. In the last phase, statistical analysis and visualization of the results were conducted in Microsoft Office Excel 2021 and software R (R Foundation for Statistical Computing, Vienna, Austria), version 4.1.0.21 During this phase, we manually grouped results into three groups: (Group 1: Basic life support, Adult advanced life support, Paediatric life support, Newborn resuscitation and support of transition of infants at birth; Group 2: Post-resuscitation care, Epidemiology of cardiac arrest in Europe, Cardiac arrest in special circumstances, First Aid; Group 3: Systems saving lives, Education for resuscitation, Ethics of resuscitation and end-of-life decisions) with the goal of enhancing the presentation of the results.

Results

From 2012 to 2022 a total of 2491 abstracts were published at ERC congresses. Published abstracts ranged from 368 (2015) to 88 (2020). Due to the Coronavirus disease 2019 pandemic, ERC congress was held virtually in 2020—the only year it was a fully virtual conference. Each year abstracts were submitted by teams of authors from 29 to 43 countries, with a total of 68 countries represented. The UK had the most accepted abstracts (11.5%; 285/2491), followed by Spain (10.5%; 262/2491), Germany (6.4%; 158/2491), and Japan (5.7%; 141/2491). The largest number of studies came from Europe (73.4%; 1827/2491), followed by Asia (18.6%; 462/2491) (Table 1). Posters were the most common type of presentation (80.2%; 1997/2491). A comparative analysis between expert manual classification of abstract titles into ERC guidelines topics and the ChatGPT-4 API revealed an average misclassification rate of fewer than 2 out of 11 ERC guidelines topics. The most common ERC guidelines topic was Basic life support (50.1%; 1247/2491), followed by Adult advance life support (41.5%; 1034/2491) and Systems saving lives (40.1%; 1000/2491). Accepted abstracts in the Education for resuscitation ERC guidelines topic (36.0%; 898/2491) have declined from 52.3% in 2016 to 25.0% in 2022, whereas as Epidemiology of cardiac arrest in Europe (24.3%; 605/2491), and Paediatric life support (9.3%; 231/2491) ERC guidelines topics increased, the first one for 13.5% (22.7% in 2012 to 36.2% in 2022) and second one for 10.9% (6.1% in 2012 to 17.0% in 2020). An interesting interaction can be observed for Post-resuscitation care (29.4%; 733/2491) and First Aid (7.4%; 185/2491) ERC guidelines topics in 2016 – showing a similar increase and decrease, more precisely the first decreases by 13.4% from 37.5% the previous year, whereas the second increases by 9.2% from 6.5% (Fig. 1). Fig. 2 illustrates that the Basic life support and Adult advanced life support ERC guidelines topics have the strongest co-occurrence to all ERC guidelines topics, where the Newborn resuscitation and support of transition of infants at birth (2.1%; 52/2491) ERC guidelines topic has the weakest co-occurrence. Paediatric life support ERC guidelines topic in most of them focuses on Education for resuscitation, Systems saving life, and First aid topics, while Ethics of resuscitation and end-of-life decisions (4.6%; 1142491), Epidemiology of cardiac arrest in Europe, Special circumstances (13.4%; 334/2491), and Post-resuscitation care ERC guidelines topics research is lacking. Linear trend analysis at a standard 0.05 significance level for each ERC guidelines topic showed, that only two ERC guidelines topics - the Epidemiology of cardiac arrest in Europe (t = 2.553, p = 0.034) and Paediatric life support (t = 2.323, p = 0.049) had a statistically significant positive linear trend.

Table 1.

Features of the abstract’s proceedings in the ERC congresses.

Year of ERC congress Host country Number of abstracts Number of participating countries The country with the most contributions
2022 Belgium 196 41 United Kingdom (31/196; 15,8 %)
2021 / / / /
2020 Online 88 29 United Kingdom (15/88; 17,1 %)
2019 Slovenia 253 43 United Kingdom (24/253; 9,5 %)
2018 Italy 324 40 Italy (40/324; 12,4 %)
2017 Germany 230 31 Germany (34/230; 14,8 %)
2016 Island 216 39 United Kingdom, Poland (19/216; 8,8 %)
2015 Czech Republic 368 40 Spain (47/368; 12,8 %)
2014 Spain 278 36 Spain (73/278; 26,3 %)
2013 Poland 229 38 United Kingdom (24/229; 10,5 %)
2012 Austria 309 37 United Kingdom (39/309; 12,6 %)
Total / 2491 68 United Kingdom (285; 11,5 %)
Open in a new tab

Fig. 1.

Fig. 1

Open in a new tab

Proportion of categorized abstract titles using ChatGPT-4 API with respect to ERC guidelines topics grouped into three districts groups.

Fig. 2.

Fig. 2

Open in a new tab

Visualisation of co-occurrence of ERC guidelines topics.

Discussion

In this proof-of-concept project, we demonstrated the ability of ChatGPT-4 API to classify abstract titles from ERC congresses into eleven discrete ERC guidelines topics. Our results showed that the Basic life support ERC guidelines topic is a well-established domain that connects to all other ERC guidelines topics. This can be attributed to the history of the ERC guidelines, where the Basic life support ERC guidelines topic was present for the first time in 1992.21 On the other hand, the ERC guidelines topic of Newborn resuscitation and support of transition of infants at birth was added more recently, in 2010.22 For the Basic life support ERC guidelines topic, the most frequent ERC guidelines topics at ERC congresses were related to out-of-hospital cardiac arrest, safety, symptom recognition, chest compressions and ventilation, and automated external defibrillator use. A study of the fifty most cited articles in emergency medicine journals found that ERC guidelines topics related to cardiac arrest, pain and toxicology were the most common. The creation and development of a unique knowledge base depends on the publication of scientific papers in peer-reviewed journals and abstract presentations at conferences. However, in journals not specifically related to emergency medicine, the range of topics in this area were broader. Among the fifty most cited nonemergency medicine articles, most of them were related to cardiology, sepsis, neurology, and a smaller proportion were related to the medical fields like psychiatry, endocrinology, hematology or radiology.23 Another bibliometric study presented that the focus was on topics related to Adult advanced life support, such as therapeutic hypothermia and extracorporeal membrane oxygenation.2

This study also demonstrates the effectiveness of large language model (LLM) based approaches like GPT-45, 24 in performing bibliometric analysis. However, it should be noted that using LLMs also reduces the credibility of results as the level of trustworthiness in the results produced from LLMs is still relatively low.25 Therefore, it is important to ensure that manual inspection of the results is performed, at least in the first few iterations of the task being targeted for automation. Like other research applications, the training data available to ChatGPT-4 API represents a limitation. In the case of bibliometric analysis, there are many papers that are not freely accessible, which may be the reason why information from these papers cannot be extracted, at least not in a time-saving, automated manner.26 However, the increasing amount of freely available scientific literature will not only open new opportunities to analyze vast amounts of literature, but also improve the extraction and other bibliometric analysis related tasks of the LLMs.

There are also some limitations that were observed during the study and should be mentioned. First, we only used abstract titles and not full abstracts. This problem occurred because all abstracts were available online in separate PDF files on Resuscitation website. In the future, full abstracts might be more readily available for download, enabling full abstract analysis. The second limitation is that we were only able to download data for the first author. With this limitation we could not perform additional analyses in the form of bibliometrics parameters related to the author as was done in other studies.2, 26, 27 This type of analysis requires processing significant amounts of non-structured data and other relevant metrics28 beyond the scope of the current work.

Funding

Nino Fijačko, Primož Kocbek, and Gregor Štiglic are supported by Slovenian Research Agency grants ARRS P2-0057, ARRS N3-0307, ARRS BI-US/22-24-138, NextGenerationEU and MVZI (C3330-22-953012). Benjamin S. Abella has received research funding from the National Institutes of Health, the Department of Defense, and Becton Dickinson. He has served as a paid consultant to Becton Dickinson, Zoll and Stryker. He holds equity in MDAlly and VOCHealth. Ruth Masterson Creber receives research funding from the National Institutes of Health (R01HL161458, R01NS123639, R01HL152021) and the Patient-Centered Outcomes Research Institute (PCORI).

CRediT authorship contribution statement

Nino Fijačko: Writing – review & editing, Writing – original draft, Visualization, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization. Ruth Masterson Creber: Writing – review & editing. Benjamin S. Abella: Writing – review & editing. Primož Kocbek: Writing – review & editing, Writing – original draft, Visualization, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Špela Metličar: Writing – review & editing, Writing – original draft, Data curation, Conceptualization. Robert Greif: Writing – review & editing. Gregor Štiglic: Writing – review & editing, Writing – original draft, Visualization, Software, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation.

Declaration of competing interest

Nino Fijačko is a member of the ERC BLS Science and Education Committee. Robert Greif is ERC Director of Guidelines and ILCOR, and ILCOR Task Force chair for Education Implementation and Team. Other authors declare that they have no conflict of interest. Benjamin S. Abella reported serving on the American Heart Association Resuscitation Science Symposium committee.

Footnotes

Appendix A

Supplementary data to this article can be found online at https://doi.org/10.1016/j.resplu.2024.100584.

Appendix A. Supplementary material

The following are the Supplementary data to this article:

Supplementary data 1
mmc1.docx (17.1KB, docx)
Supplementary data 2
mmc2.docx (14.1KB, docx)

References

  • 1.Pritchard A. Statistical bibliography or bibliometrics? J Doc. 1969;25:348–349. [Google Scholar]
  • 2.Jia T., Luo C., Wang S., et al. Emerging trends and hot topics in cardiopulmonary resuscitation research: a bibliometric analysis from 2010 to 2019. Med Sci Monit. 2020;26:e926815. doi: 10.12659/MSM.926815. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Rodrigues S.P., van Eck N.J., Waltman L., Jansen F.W. Mapping patient safety: a large-scale literature review using bibliometric visualisation techniques. BMJ Med. 2014;4:e004468. doi: 10.1136/bmjopen-2013-004468. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Nakaya Y., Higaki A., Yamaguchi O. ChatGPT's ability to classify virtual reality studies in cardiology. Eur Heart J Digit Health. 2023;4:141–142. doi: 10.1093/ehjdh/ztad026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Kleesiek J., Wu Y., Stiglic G., Egger J., Bian J. An opinion on ChatGPT in health care - written by humans only. J Nucl Med. 2023;64:701–703. doi: 10.2967/jnumed.123.265687. [DOI] [PubMed] [Google Scholar]
  • 6.Webharvy. (Accessed 30 January 2024, at: https://www.webharvy.com/).
  • 7.Perkins G.D., Graesner J.-T., Semeraro F., et al. European Resuscitation Council guidelines 2021: executive summary. Resuscitation. 2021;161:1–60. doi: 10.1016/j.resuscitation.2021.02.003. [DOI] [PubMed] [Google Scholar]
  • 8.Gräsner J.T., Herlitz J., Tjelmeland I.B., et al. European Resuscitation Council guidelines 2021: epidemiology of cardiac arrest in Europe. Resuscitation. 2021;161:61–79. doi: 10.1016/j.resuscitation.2021.02.007. [DOI] [PubMed] [Google Scholar]
  • 9.Semeraro F., Greif R., Böttiger B.W., et al. European Resuscitation Council guidelines 2021: systems saving lives. Resuscitation. 2021;161:80–97. doi: 10.1016/j.resuscitation.2021.02.008. [DOI] [PubMed] [Google Scholar]
  • 10.Olasveengen T.M., Semeraro F., Ristagno G., et al. European Resuscitation Council guidelines 2021: basic life support. Resuscitation. 2021;161:98–114. doi: 10.1016/j.resuscitation.2021.02.009. [DOI] [PubMed] [Google Scholar]
  • 11.Soar J., Böttiger B.W., Carli P., et al. European Resuscitation Council guidelines 2021: adult advanced life support. Resuscitation. 2021;161:115–151. doi: 10.1016/j.resuscitation.2021.02.010. [DOI] [PubMed] [Google Scholar]
  • 12.Lott C., Truhlář A., Alfonzo A., et al. European Resuscitation Council guidelines 2021: cardiac arrest in special circumstances. Resuscitation. 2021;161:152–219. doi: 10.1016/j.resuscitation.2021.02.011. [DOI] [PubMed] [Google Scholar]
  • 13.Nolan J.P., Sandroni C., Böttiger B.W., et al. European Resuscitation Council and European Society of Intensive Care Medicine guidelines 2021: post-resuscitation care. Resuscitation. 2021;161:220–269. doi: 10.1016/j.resuscitation.2021.02.012. [DOI] [PubMed] [Google Scholar]
  • 14.Zideman D.A., Singletary E.M., Borra V., et al. European Resuscitation Council guidelines 2021: first aid. Resuscitation. 2021;161:270–290. doi: 10.1016/j.resuscitation.2021.02.013. [DOI] [PubMed] [Google Scholar]
  • 15.Madar J., Roehr C.C., Ainsworth S., et al. European Resuscitation Council guidelines 2021: newborn resuscitation and support of transition of infants at birth. Resuscitation. 2021;161:291–326. doi: 10.1016/j.resuscitation.2021.02.014. [DOI] [PubMed] [Google Scholar]
  • 16.Van de Voorde P., Turner N.M., Djakow J., et al. European Resuscitation Council guidelines 2021: paediatric life support. Resuscitation. 2021;161:327–387. doi: 10.1016/j.resuscitation.2021.02.015. [DOI] [PubMed] [Google Scholar]
  • 17.Greif R., Lockey A., Breckwoldt J., et al. European Resuscitation Council guidelines 2021: education for resuscitation. Resuscitation. 2021;161:388–407. doi: 10.1016/j.resuscitation.2021.02.016. [DOI] [PubMed] [Google Scholar]
  • 18.Mentzelopoulos S.D., Couper K., Van de Voorde P., et al. European Resuscitation Council guidelines 2021: ethics of resuscitation and end of life decisions. Resuscitation. 2021;161:408–432. doi: 10.1016/j.resuscitation.2021.02.017. [DOI] [PubMed] [Google Scholar]
  • 19.R Foundation for Statistical Computing, Vienna, Austria.
  • 20.Generative Pre-trained Transformer 4 (GPT-4) application programming interface. (Accessed 30 January 2023, at: https://aiapp.org/).
  • 21.Holmberg S., Handley A., Bahr J., et al. Guidelines for basic life support: a statement by the Basic Life Support Working Party of the European Resuscitation Council. Resuscitation. 1992;24:103–110. [PubMed] [Google Scholar]
  • 22.Nolan J.P., Soar J., Zideman D.A., et al. European Resuscitation Council guidelines for resuscitation 2010 section 1: executive summary. Resuscitation. 2010;81:1219–1276. doi: 10.1016/j.resuscitation.2010.08.021. [DOI] [PubMed] [Google Scholar]
  • 23.Barbic D., Tubman M., Lam H., Barbic S. An analysis of altmetrics in emergency medicine. Acad Emerg Med. 2016;23:251–268. doi: 10.1111/acem.12898. [DOI] [PubMed] [Google Scholar]
  • 24.Kirtania DK. ChatGPT as a tool for bibliometrics analysis: interview with ChatGPT (March 17, 2023). Available at SSRN: https://ssrn.com/abstract=4391794 or 10.2139/ssrn.4391794. [DOI]
  • 25.Farhat F., Sohail S.S., Madsen D.Ø. How trustworthy is ChatGPT? the case of bibliometric analyses. Cogent Eng. 2023;10:2222988. [Google Scholar]
  • 26.Arif T.B., Munaf U., Ul-Haque I. The future of medical education and research: is ChatGPT a blessing or blight in disguise? Med Educ Online. 2023;28:2181052. doi: 10.1080/10872981.2023.2181052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Choudhri A., Siddiqui A., Khan N.R., Cohen H.L. Understanding bibliometric parameters and analysis. RadioGraphics. 2015;35:736–746. doi: 10.1148/rg.2015140036. [DOI] [PubMed] [Google Scholar]
  • 28.Xu L., Tang F., Wang Y., et al. Research progress of pre-hospital emergency during 2000–2020: a bibliometric analysis. Am J Transl Res. 2021;12:1109–1124. [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary data 1
mmc1.docx (17.1KB, docx)
Supplementary data 2
mmc2.docx (14.1KB, docx)

Articles from Resuscitation Plus are provided here courtesy of Elsevier

RESOURCES