Evaluation of the Diagnostic Capabilities of Artificial Intelligence (GPT‐4) in a Cardiology Department in Sub‐Saharan Africa: Cross‐Sectional Study

ABSTRACT

Context/Objective

“Generative Pre‐trained Transformer” type 4 or GPT‐4 is an artificial intelligence (AI) developed by OpenAI. It can be used to generate positive and differential diagnoses. However, its effectiveness in a cardiology department in Africa has not been studied. The aim of this study is to evaluate the diagnostic accuracy of GPT‐4 in the cardiology department.

Methods

This was a retrospective descriptive cross‐sectional study conducted from April 1, 2024 to May 30, 2024 in the cardiology department of the Bogodogo University Hospital. Our primary endpoint was whether the main diagnosis of GPT‐4 corresponded to the final diagnosis made by the cardiologists.

Results

Out of 50 patients collected, GPT‐4 found the diagnosis on the basis of clinical data in 35% (18/50). In 82% (41/50) of cases, GPT‐4's diagnosis was one of the cardiologist's three hypotheses and in 64% (32/50) of cases the diagnosis was found with certainty by GPT‐4. The differential diagnosis listed by GPT‐4 was scored 5 in 46 patients. All the diagnoses of the aetiological groups were found by GPT‐4 in 100% (50/50) of cases, except in the hypertensive and ischaemic cardiomyopathy groups.

Conclusion

GPT‐4 demonstrated a variable ability to generate accurate diagnoses, with a significant improvement when paraclinical data were included.

1. Introduction

GPT‐4 is an artificial intelligence (AI) developed by OpenAI [1]. It is a large language model based on automatic natural language processing, also known as “generative pretrained transform” (GPT) [2]. GPT‐4 is capable of generating textual responses that sound human in response to queries written by users [2]. Previous studies have reported that the diagnostic accuracy of differential diagnosis lists generated by GPT‐4 for clinical vignettes ranged from 64% to 83% [3]. Extensive research is currently being conducted in a variety of areas, including cardiovascular disease, using GPT‐4 [4]. In our African context, the findings of these studies could be beneficial due to the inadequacy of medical equipment and the obvious shortage of medical specialists. However, there is a lack of studies addressing the competence of GPT‐4 in the diagnosis of cardiovascular disease, based on clinical and paraclinical data in a black African population. The aim of this study was to assess the diagnostic accuracy of GPT‐4 when provided with clinical and paraclinical data and to compare its performance with that of cardiologists in a cardiology department in Burkina Faso.

2. Materials and Methods

2.1. Study Design

This was cross‐sectional study conducted from April 1, 2024 to May 30, 2024 in the cardiology department of the University Hospital Center of Bogodogo, Burkina Faso.

2.2. Data Sources

Clinical and paraclinical data from 50 consecutive patients admitted to the cardiology department was reviewed. The source documents were hospital registers, medical records. The patient files contained clinical data with the three diagnostic hypotheses suggested by the cardiologists' consensus upon each patient's admission, as well as the specific diagnosis made after additional tests were performed.

2.3. Inclusion and Exclusion Criteria

All patients hospitalised in the department during the study period for specific cardiovascular disease were included. Patients with multiple problems and those for whom the department's cardiologists were unable to reach a consensus on a specific diagnosis were excluded from this study.

2.4. Evaluation Procedure

We used the Chat GPT 4 template application (June 14 version; GPT‐4, OpenAI LLC). We first recorded the clinical signs and then supplemented them with the paraclinical signs for each patient on GPT‐4, followed by the following prompt:

• 1.Given these clinical signs, what are the three most likely differential diagnoses in order of priority?
• 2.Given these clinical and paraclinical signs, what is the definitive diagnosis?

For echocardiography, electrocardiogram, and chest X‐ray, we included only the results of the interpretation given by the specialist. Identifiers were assigned to each patient during the collection process, so that no names are on our database, thus preserving anonymity and confidentiality. Table 1 presents two examples of applications of the evaluation procedure used with GPT‐4.

Table 1.

Two examples of applications of the engineering technique used with GPT‐4 showing good diagnostic correlation.

Number of cases	Evaluation procedure with GT‐4	GPT‐4 diagnosis(es)	Cardiologist diagnosis(es)
Case 1	Given the following clinical signs: a 64‐year‐old hypertensive woman with a history of hypertensive heart disease, presenting with dyspnoea and noted on examination to have normal consciousness, BP 144/95 mmHg, oedema of the lower limbs, tachycardia with left gallop, mitral insufficiency murmur, bilateral crackles, muffled heart sounds and ascites; what are the three main diagnostic hypotheses in order of priority?	1. Congestive heart failure due to decompensated hypertensive heart disease	1. Decompensated hypertensive heart disease in heart failure
		2. Congestive heart failure due to significant mitral insufficiency	2. Ischaemic heart disease decompensated in heart failure
		3. Congestive heart failure due to ischaemic dilated cardiomyopathy	3. Valvular heart disease decompensated in heart failure
	Given these clinical signs (…) with atrial fibrillation on the electrocardiogram, atrial and left ventricular hypertrophy, left ventricular ejection fraction of 35% on the echocardiogram, dilatation of the left cavities, pulmonary arterial hypertension and increased filling pressures; chest X‐ray and BNP not done, hyponatremia of 133/mm3: what is your definitive diagnosis?	Decompensated congestive heart failure with impaired LVEF (35%) in advanced hypertensive heart disease complicated by atrial fibrillation and functional mitral insufficiency	Secondarily dilated hypertensive cardiomyopathy with impaired LVEF (35%) decompensated into heart failure complicated by atrial fibrillation and functional mitral insufficiency.
Case 2	In the presence of the following clinical signs: 75‐year‐old man with hypertension, type 2 diabetes and active smoker, presenting with midthoracic precordialgia which has been present for 4 months and is less intense. Examination reveals normal consciousness, BP 132/82 mmHg, oedema in the lower limbs, mitral insufficiency murmur, bilateral crepitus rales and muffled heart sounds: what are the three main diagnostic hypotheses, in order of priority?	1. Chronic ischaemic heart disease decompensated into heart failure on ischaemic heart disease)	1. Ischaemic heart disease decompensated into heart failure
		2. Congestive heart failure in advanced hypertensive heart disease	2. Hypertensive heart disease decompensated in heart failure
		3. Heart failure due to organic or functional mitral insufficiency	3. Valvular heart disease decompensated in heart failure
	Given these clinical signs (…) with a Q wave from V1–V4 on the electrocardiogram with negative T waves, hypokinesia of the anteroseptal and apical segments on the echocardiogram, left ventricular ejection fraction at 55%, troponins at two times normal, chest X‐ray, BNP and coronary angiography not performed: what is your definitive diagnosis?	Chronic ischaemic heart disease (former extensive anterior infarction) with segmental hypokinesia, complicated by heart failure with preserved ejection fraction, in a high‐risk cardiovascular setting.	Ischaemic cardiomyopathy with preserved ejection fraction decompensated into congestive heart failure in a high‐risk cardiovascular setting.

2.5. Outcomes

Our predefined primary endpoint was whether the main diagnosis in GPT‐4 matched the final diagnosis made by cardiologists. The secondary endpoints were whether the final diagnosis matched by providing clinical data only. Then the presence of the final diagnosis in the possible differential and the differential quality score of the model using a 5‐point ordinal scoring system previously published by Bond et al. [5]. This score is based on accuracy and utility (5 = the actual diagnosis was suggested in the differential; 4 = the suggestions included something very close, but not exact; 3 = the suggestions included something closely related that might have been helpful; 2 = the suggestions included something related, but unlikely to be helpful; 0 = no suggestions close to the target diagnosis). The scale does not contain a score of 1 [5].

2.6. Statistical Analysis

Categorical variables, including the exact diagnoses found by GPT‐4 using clinical data alone, and then the combination of clinical and paraclinical data, were expressed as frequencies and percentages. We also calculated the proportions of the final diagnosis as a function of the differential diagnosis scores. Means and standard deviations (SD) were used for normally distributed data, and medians and ranges or interquartile ranges (IQR) for data that were not normally distributed. The data were entered into an Excel database and all the analysis was carried out using SPSS software version 20.0. Missing data were treated as missing data during analysis.

3. Results

In this study, out of 54 patients received during the study period, 4 were excluded (2 were excluded because clinical and paraclinical information was incomplete; 2 did not have a precise diagnosis). Figure 1 shows the flow of our patients. GPT‐4 found the diagnosis solely on the basis of clinical data in 36% (18/50) of cases. In 82% (41/50) of cases, GPT‐4's diagnosis was one of the three diagnostic hypotheses put forward by the clinicians. After inclusion of the paraclinical data, he reported 92% (46/50) certainty with the physicians' diagnoses. Figure 2 illustrates the results of the differential diagnosis score against Bond et al. The main diagnoses, such as ischaemic heart disease, hypertensive heart disease, toxic heart disease, and valvular heart disease, were found in 24% (n = 12), 20% (n = 10), 16% (n = 08) and 14% (n = 07), respectively. All the diagnoses of the aetiological groups were found by GPT‐4 in 100% of cases, except in the hypertensive and ischaemic cardiomyopathy groups. These results are shown in Table 2.

Figure 1.

Study flow chart.

Figure 2.

Graph of Bond et al. performance score for differential diagnoses found by GPT‐4.

Table 2.

Diagnostic accuracy of GPT‐4 compared to cardiologist consensus.

Cardiovascular conditions admitted during the study	Consensus diagnosis by cardiologists	Diagnostic accuracy of GPT‐4		Incorrect diagnosis of GPT‐4
		With clinical data only	With clinical + paraclinical data
Ischemic heart disease	12	03	09	03
Hypertensive heart disease	10	02	09	01
Toxic heart disease	08	04	08	00
Valvular heart disease	07	03	07	00
Chronic pulmonary heart disease	05	03	05	00
Myocarditis	03	01	03	00
Peripartum cardiomyopathy	02	02	02	00
Pulmonary embolism	02	00	02	00
Pericarditis	01	00	01	00

4. Discussion

Artificial intelligence (AI) occupies a prominent place in contemporary medical practice [3]. The aim of our study was to evaluate the performance of GPT‐4 in the context of African cardiology practice. To our knowledge, this is the first study to evaluate the diagnostic capabilities of GPT‐4 in a cardiology department in sub‐Saharan Africa.

Our study in cardiology revealed that GPT‐4 correctly identified patients' diagnoses in 64% of cases, a significantly better result than that observed by Stoneham et al. in dermatology, where GPT‐4 correctly identified the diagnosis in 56% of cases [6]. In terms of differential diagnoses, GPT‐4 listed the cardiologist's diagnosis among its hypotheses with a score equal to 5 in 92% of cases, compared with 100% in the dermatology study [6]. This disparity can be attributed to several factors. Firstly, our study used the latest version of GPT‐4, version 4, which benefits from significant improvements in terms of accuracy and analytical capabilities. In contrast, the study by Stoneham et al. used an earlier version of the AI, which could explain the lower performance [6]. In addition, it may also be related to the fact that dermatology is strongly influenced by vision, which was not tested in the study of Stoneham et al. [6]. In cardiology, GPT‐4 has the advantage of being able to draw on a multitude of paraclinical tests, such as electrocardiograms, echocardiograms and laboratory analyses, to refine its diagnoses. This wealth of clinical data enables GPT‐4 to produce more accurate and reliable diagnoses. However, in our study, the diagnostic accuracy of GPT‐4 compared to clinical data was lower than the results of retrospective studies on AI conducted by Yavuz et al. in 2024 [7]. In their study of 20 clinical cases produced by cardiology experts, GPT‐4 reported a diagnostic accuracy of 80% (18/20) based on diagnostic data. GPT‐4.0 thus appears to be an excellent and valuable tool for clinical decision‐making in cardiology, especially when provided with refined clinical semiology in complex or specific cases, as was the case in the study by Yavuz et al. However, it must be acknowledged that, compared to the score obtained by Blond et al., the Likert and Global quality scales are capable of providing greater accuracy but require the use of clinical and paraclinical elements specific to each disease being evaluated [8, 9].

GPT‐4's capabilities are proving to be an invaluable asset for the future of medical practice. Indeed, the integration of generative AI such as GPT‐4 can offer immediate support to doctors in complex cases, reducing diagnostic errors and improving patient outcomes [4]. In an educational context, GPT‐4 could play a crucial role in the training of future cardiologists, in particular by refining clinical reasoning and the acquisition of medical knowledge [4]. Interaction with generative AI exposes learners to a variety of diagnoses, preparing them for complex clinical situations.

GPT‐4, is becoming increasingly important in cardiology, as demonstrated by several Western studies. For example, the study by Gunay et al. concluded that GPT‐4 outperformed cardiologists on common questions, while its performance closely aligned with that of cardiologists as the complexity of the questions increased [10]. Similarly, the study by Guo et al. found that the application of AI in the diagnosis and treatment of cardiac arrhythmias was superior to that of medical specialists [11].

The use of GPT‐4 in the diagnosis of cardiovascular disease presents both risks and challenges. Risks include concerns about confidentiality, ethics, bias and discrimination. GPT‐4 can be used intentionally or unintentionally to create false evidence and material, thanks to its impressive ability to produce information with a high degree of plausibility [12]. This includes “hallucinations”, where the content generated is not based on reality, creating entirely fabricated facts [12]. Another major risk is the reproduction of biases present in training data. In the field of health, where precision is crucial, errors or inaccuracies can be catastrophic [12]. To minimise these risks, rigorous human evaluation is essential, as is compliance with standards of accuracy, reliability and interpretability. In addition, security measures must be put in place to protect patient information, including encryption, access control, secure data storage and compliance with confidentiality regulations.

Challenges include the need for GPT‐4 to have the medical expertise to understand the complex relationships between conditions and treatments. Being limited to data, it lacks recent medical advances, which may affect its clinical utility [13]. Our study showed that in 81.48% of cases, the diagnosis of GPT‐4 matched the cardiologists' assumptions, highlighting its potential usefulness despite the challenges mentioned. Another challenge is the potential help of GPT‐4 for non‐cardiologists in the triage and referral of patients, especially in countries such as ours, where there is a critical shortage of specialist physicians, but also with the security challenge of not being able to evacuate certain patients.

Our study of the diagnostic capabilities of GPT‐4 in cardiology has several important limitations. Firstly, as with any retrospective study, it suffers from missing data in some patients, which may affect the representativeness of the results. Secondly, although the cases studied provide valuable insight into diagnostic scenarios, they may not reflect the full range of clinical presentations, including atypical cases or diagnostic challenges encountered in the cardiology department. In addition, some diagnoses could have been refined if GPT‐4 had access to the clinical course of patients, as a clinician would. These limitations must be taken into account when interpreting the results, and underline the need for further studies to fully assess the potential and limitations of GPT‐4 in the diagnosis of cardiovascular disease.

5. Conclusion

In conclusion, our study of the diagnostic capabilities of GPT‐4 in a cardiology department in Africa reveals promising results. In 64.81% of cases, GPT‐4 established a diagnosis concordant with that of the cardiologist, particularly in the nosological groups associated with valvular and hypertensive cardiomyopathies. These results demonstrate the potential of GPT‐4 as a diagnostic aid, while underlining the importance of continuing research to refine these results, particularly in developing countries such as ours. Future multicentre, prospective studies evaluating the diagnostic capabilities of Large Language Models would make a substantial contribution to patient care, especially in areas where human resources are limited.

Author Contributions

Martin Wendlassida Nacanabo: conceptualization, methodology, data curation, writing – review & editing, writing – original draft, project administration. André Arthur Taryètba Seghda: investigation, methodology, project administration. Anna Tall/Thiam: investigation, supervision. Georges Christian R Millogo: supervision, investigation. Yannick Laurent Tchenadoyo Bayala: project administration, investigation, conceptualization, data curation. Valentin Nobila Yameogo: supervision, investigation. André Koudnoaga Samadoulougou: supervision, validation. Patrice Zabsonre: supervision.

Ethics Statement

Approval was obtained from the ethics committee of Bogodogo University Hospital. The procedures used in this study adhere to the tenets of the Declaration of Helsinki.

Consent

We have obtained informed consent of the subjects concerned. All measures are taken to preserve the confidentiality of information concerning them.

Conflicts of interest Interest

The authors declare no conflicts of interest.

Transparency Statement

The corresponding author, Martin Wendlassida Nacanabo, affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.

Data Availability Statement

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.