Detection of hypertension from pharyngeal images using deep learning algorithm in primary care settings in Japan

Abstract

Background

The early detection of hypertension using simple visual images in a way that does not require physical interaction or additional devices may improve quality of care in the era of telemedicine. Pharyngeal images include vascular morphological information and may therefore be useful for identifying hypertension.

Objectives

This study sought to develop a deep learning-based artificial intelligence algorithm for identifying hypertension from pharyngeal images.

Methods

We conducted a secondary analysis of data from a clinical trial, in which demographic information, vital signs and pharyngeal images were obtained from patients with influenza-like symptoms in multiple primary care clinics in Japan. A deep learning-based algorithm that included a multi-instance convolutional neural network was trained to detect hypertension from pharyngeal images and demographic information. The classification performance was measured by area under the receiver operating characteristic curve. Importance heatmaps of the convolutional neural network were also examined to interpret the algorithm.

Results

This study included 7710 patients from 64 clinics. The training dataset comprised 6171 patients from 51 clinics (460 positive cases), and the test dataset comprised 1539 patients from 13 clinics (130 positive cases). Our algorithm achieved an area under the receiver operating characteristic curve of 0.922 (95% CI, 0.904 to 0.940), significantly improving over the baseline prediction model incorporating only demographic information, which scored 0.887 (95% CI, 0.862 to 0.911). Our algorithm had consistent classification performance across all age and sex subgroups. Importance heatmaps revealed that the algorithm focused on the posterior pharyngeal wall area, where blood vessels are mainly located.

Conclusions

The results indicate that a deep learning-based algorithm can detect hypertension with high accuracy using pharyngeal images.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• There were some previous studies that sought to diagnose hypertension using electronic health records and/or physiological data such as photoplethysmography. However, no study attempted to diagnose from pharyngeal images.

WHAT THIS STUDY ADDS

• In this diagnostic study which included 7710 patients, the deep learning-based algorithm that incorporated demographic information and pharyngeal images demonstrated high diagnostic performance with an area under the receiver operating characteristic curve of 0.922.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• Given that the deep learning-based algorithm can accurately detect hypertension using pharyngeal images, it holds potential for future utilisation in providing medical accessibility and diagnosis through consumer accessible devices, such as smartphones.

Introduction

Hypertension is a major global disease burden, with more than one billion people—or approximately 40% of adults aged 25 and above—diagnosed with hypertension worldwide.¹ If it is not appropriately diagnosed and treated, hypertension can lead to major complications such as myocardial infarction, stroke and chronic kidney disease.2,4 However, given that hypertension is often asymptomatic, it is often unidentified and undiagnosed. Even when it is appropriately diagnosed, many patients are untreated or have insufficient control of their blood pressure.¹ Complications of hypertension account for approximately 10 million deaths every year worldwide¹; this issue is particularly salient in low- and middle-income countries where there is limited access to healthcare.¹

Underdiagnosis is a major issue in hypertension because untreated patients are at increased risk of both all-cause and other disease-related mortality.⁵ Globally, the proportion of people with undiagnosed hypertension is estimated to be 41% in women and 51% in men.⁶ This large discrepancy between the prevalence and awareness of hypertension is mainly caused by a lack of access to healthcare and/or insufficient resources for diagnosing hypertension, such as blood pressure monitoring devices.7,9 A US survey reported that only 55% of patients with hypertension use blood pressure monitoring devices at home¹⁰; this number is estimated to be far lower in the general population (including those individuals without diagnosed hypertension). Easier and more accessible diagnosis without the use of medical devices can be an effective solution to the underdiagnosis of hypertension. Given that the number of mobile phone users is 7.3 billion, which is 91.5% of the world’s population as of 2022,¹¹ we hypothesised that a new diagnostic technique for hypertension that uses the cameras on mobile phones has the potential to reduce the underdiagnosis of hypertension and ultimately lower hypertension-related morbidity and mortality.

Hypertension causes vascular damage and remodelling.^{12 13} Previous studies on the use of retinal fundus images to predict hypertension¹⁴ and other diseases¹⁵ have reported that geometrical changes in blood vessels are important for diagnosis. Pharyngeal images contain vascular information; we hypothesised that vascular changes caused by hypertension may be visible on such images. With the possibility of detecting hypertension through images, smartphone-based diagnosis can be cheaper, faster and more accessible compared with the conventional sphygmomanometer-based method. It also has the potential to improve the quality of telemedicine by addressing the current difficulties of diagnosing hypertension without additional medical devices. In this study, we therefore developed and investigated a deep learning-based algorithm for the accurate detection of hypertension using pharyngeal images, with a view to its future application in telemedicine.

Methods

Study design

We conducted a secondary analysis of data from a clinical trial in patients with influenza-like illness (registered to the Japan Registry of Clinical Trials: jRCTs032190120). Patients with influenza-like symptoms were recruited from 64 primary care clinics in Japan between November 2019 and January 2020. The inclusion and exclusion criteria of the clinical trial were shown in online supplemental table S1.

As shown in figure 1, after removing samples with missing data or labels, patients were split into training (80%) and test (20%) datasets. The training dataset was used to develop the deep learning algorithm, and the test dataset was used for evaluating the algorithm. To exclude the influence of clinic-specific biases (eg, positive rates of patients) during evaluation, the test dataset used patients from different clinics than the training dataset.

Figure 1. Patient flowchart.

Data collection

Pharyngeal images were captured using a specially designed medical camera with a light source and a tool that depressed the patient’s tongue. Pharyngeal images and demographic information—including disease history, medication history and blood pressure measured with a sphygmomanometer—were obtained at the clinic.

We defined hypertensive patients as: (1) patients with a history of hypertension or (2) patients taking antihypertensive medications. Although we had the data on measured blood pressure, given that it was measured only once during the physician visit, to account for potential measurement error^{16 17} and potential bias associated with elevated body temperature/heart rate in patients with influenza-like illness,¹⁸ we did not use it to define hypertension in our main analysis. Instead, we used this variable in our sensitivity analysis. In the sensitivity analysis, we reanalysed the data using an alternative definition of hypertension as patients with a systolic blood pressure ≥140 mm Hg or a diastolic blood pressure ≥90 mm Hg¹⁹ (we did not use the information on medical history and medications for this definition).

Development of hypertension detection model

We developed a neural network-based algorithm to detect hypertension (figure 2). In designing the neural network, there were two major challenges: controlling the quality of the input images (brightness, blurriness, etc) and efficiently extracting the three-dimensional information of the pharynx.

Figure 2. Model architecture and training configurations. FC, fully connected; MI-CNN, multi-instance convolutional neural network; ReLU, rectified linear unit.

To address the issue of input image quality, we constructed an image quality screening module consisting of a lightweight convolutional neural network (CNN)—MobileNetV2.²⁰ The image screening module was trained on a randomly sampled subset of training dataset, with labels annotated by a physician in the authors (MF), based on several quality criteria (eg, visibility of pharynx, focus, brightness, motion blur and fog). The performance of the image screening module was assessed on another subset of training dataset consisting of 535 images, resulting in a sensitivity of 0.914 and a specificity of 0.988. Only the images that passed the module were used for analysis.

To overcome the difficulties of obtaining three-dimensional information about the pharynx from a single image, we designed a multi-instance convolutional neural network (MI-CNN) that accepted multiple input images from different locations and angles. The MI-CNN architecture consisted of a CNN-based encoder that generated feature maps from pharyngeal images, and a multilayer perceptron classifier that pooled features and output a hypertension probability score. To further improve the model, the output of the neural network was concatenated with demographic information (age and sex) and used as input for the second-stage logistic regression model.

Binary cross entropy loss and the Adam²¹ optimizer (learning rate=0.0005; β1=0.9; β2=0.999) were used to train the neural network. A fivefold cross-validation was performed on the training dataset and hyperparameters were selected based on the results. Based on our preliminary experiments (online supplemental table S2), EfficientNet-B5²² architecture was used as the encoder backbone, the maximum number of input images was set to three, and each input image was resized to 384×384×3 pixels. Transfer learning²³ was performed using parameters pre-trained on the ImageNet²⁴ dataset. All hyperparameters and training configurations are shown in figure 2. Our algorithm is an ensemble of five models from the five folds, and the final hypertension probability score was calculated by averaging the predictions from each fold.

Statistical analysis

The performance of the algorithm was evaluated by calculating the area under the receiver operating characteristic curve (AUROC) for the training and test dataset. Considering its potential clinical use in patient screening, the sensitivity when specificity=90% was also calculated. P values of <0.05 were considered significant. To investigate the influence of pharyngeal images on the predictions, we also examined the performance of an MI-CNN model using pharyngeal image only and a logistic regression model using demographic information only.

To assess the model performance in different patient groups, analyses stratified by age (under 40 years, 40–59 years and 60 years and over) and sex were performed.

Moreover, to further explore the mechanism of the deep learning model, we used gradient-weighted class activation mapping (Grad-CAM)²⁵ (ie, heatmaps) to visualise the important areas that contributed to the final prediction results. We selected the 50 samples that the model judged to be most positive-looking (ie, samples with the highest hypertension possibility scores) and 50 negative-looking samples (ie, samples with the lowest hypertension possibility scores). Three physicians (MF, SO and YT) examined the heatmaps and qualitatively assessed the differences between positive-looking and negative-looking samples. Consensus was obtained between the three physicians for the qualitative assessment results. The study followed the Transparent Reporting of Multivariable Prediction Model for Individual Prognosis or Diagnosis reporting guidelines (online supplemental table S3).

Results

Data collection

Of the 9029 patients who were eligible (figure 1), 9003 patients completed the study. This study included 7710 patients with no missing values, of whom 590 patients were defined as having hypertension. The baseline characteristics of the patients are shown in table 1.

Table 1. Dataset characteristics. P value is for the difference between the training dataset and test dataset.

	Training dataset (N=6171/51 clinics)		Test dataset (N=1539/13 clinics)		P value
	Hypertension	No hypertension	Hypertension	No hypertension
Participants, n (%)	460 (7.45)	5711 (92.6)	130 (8.45)	1409 (91.6)	0.21
Female, n (%)	184 (40.0)	2924 (50.2)	46 (35.4)	710 (50.4)	0.40
Age, mean (SD)	60.7 (13.3)	31.8 (17.2)	60.3 (13.9)	32.2 (15.6)	0.19
Age subgroups, n (%)
0<age<40	21 (4.57)	3715 (65.0)	6 (4.62)	955 (67.8)	0.18
40≤age<60	193 (40.2)	1653 (28.9)	61 (46.9)	384 (27.3)	0.46
60≤age	246 (53.5)	343 (6.01)	63 (48.5)	70 (4.97)	0.30
Comorbidity, n (%)
Diabetes	107 (23.3)	68 (1.19)	26 (20.0)	17 (1.20)	0.99
Dyslipidaemia	200 (43.5)	158 (2.77)	58 (44.6)	29 (2.06)	0.87

In total, 24 779 images from 7710 patients passed the quality screening module. The deep learning model was trained using the training dataset of 19 611 pharyngeal images from 6171 patients. The model performance was evaluated using the test dataset of 5168 images from 1539 patients. A stratified group split was performed so that there were no overlapping clinics, and the hypertension rates were very similar in both datasets.

Model evaluation

The model in the main analysis exhibited high diagnostic accuracy on the test dataset, achieving an AUROC of 0.922 (95% CI, 0.904 to 0.940) and a sensitivity when specificity=90% of 0.700 (95% CI, 0.621 to 0.779) (figure 3). In the sensitivity analysis using the measured blood pressure data to define hypertension, the model demonstrated high accuracy as well, yielding an AUROC of 0.783 (95% CI, 0.744 to 0.822).

Figure 3. Results of the main analysis and sensitivity analysis. P value is for the difference between the demographic information-only model and the combined model. AUC, area under the curve; AUROC, area under the receiver operating characteristic curve; ROC, receiver operating characteristic.

Figure 4 shows the model performance across different age and sex subgroups. The predictions consistently had significant capability to detect hypertension in all subgroups. In particular, a high AUROC was observed in the subgroup under 40 years old, but this was expected to be due to the small number of positive cases.

Figure 4. Results of the subgroup analysis on the test dataset. AUC, area under the curve; ROC, receiver operating characteristic.

Due to the disease characteristics of hypertension, high accuracy was observed using demographic information alone. However, the addition of pharyngeal image resulted in statistically significant improvements of AUROC in both analyses. Assuming a scenario in which a 90% specificity is maintained for screening in primary care clinics, the addition of pharyngeal images can be expected to yield an absolute improvement of 10% in sensitivity. According to a study that examined the trends in hypertension, there were approximately 600 million undiagnosed individuals with hypertension as of 2019.⁶ An improvement of 10% in sensitivity would mean a reduction in missed diagnoses by approximately 60 million, which has noteworthy clinical relevance.

Interpretation of the hypertension detection model

Qualitative analysis of the heatmaps revealed that the intensity of attention to the posterior pharyngeal wall area and palatal arch, where blood vessels were visible, was much higher in positive-looking samples than in negative-looking samples. Some Grad-CAM heatmaps are provided in online supplemental figure S1.

Discussion

Through this study, we have discovered both the high accuracy of our algorithm in diagnosing hypertension and the contribution of pharyngeal images in improving the accuracy of hypertension diagnosis. Although studies using pharyngeal images to diagnose influenza²⁶ and identify sex²⁷ have already been reported, to the best of our knowledge, this is the first study to demonstrate the utility of pharyngeal images in diagnosing hypertension. It has also been reported that chest radiographs²⁸ and retinal images¹⁵ can be used to diagnose lifestyle-related diseases. However, pharyngeal images can be captured with much lower invasiveness compared with these images and without the need for special devices, making them highly compatible with digital therapeutics and telemedicine. Our findings suggest the potential for the future development of contactless diagnostics incorporating pharyngeal images would be immensely helpful.

Based on the results of our analysis using Grad-CAM heatmaps, we hypothesised that the deep learning model may diagnose hypertension by identifying specific blood vessel patterns (eg, stenosis and tortuosity) and arteriosclerosis caused by hypertension. Given that the pharynx is one of the few body parts in which blood vessels can be observed from the outside, images of the pharynx may provide similar information to the retinal images reported in a previous study.¹⁴

Our study was built on prior research that sought to diagnose hypertension without using blood pressure cuffs. Recent studies have used physiological data, such as photoplethysmography (PPG) or electrocardiography, to detect hypertension. Liang et al converted PPG signals into a scalogram using continuous wavelet transform and then predicted hypertension from the scalogram using a CNN.²⁹ The authors reported an F1-score of 92.55% for binary classification between normotension and hypertension. Similarly, other studies have explored the possibility of detecting hypertension from PPG signals using the open PPG-BP dataset,³⁰ which contains only 219 patients.31,33 While informative, these studies are limited by their small dataset sizes, making evaluation metrics difficult to generalise. In contrast, our approach to diagnosing hypertension using pharyngeal images was validated using a relatively large dataset.

Limitations

The present study has some limitations. First, a certain degree of misclassification may occur in the definition of hypertension. To address the various issues associated with blood pressure measurements, hypertension was defined based on history and medication information in the main analysis. However, it should be noted that we cannot identify people with undiagnosed hypertension using this method. Additionally, if disease history or medication history is not available at the clinic, it becomes necessary to obtain this information from the patients through verbal communication, which may be subject to the uncertainty of the patients’ recollections. To address this issue, we conducted a sensitivity analysis to define hypertension using the measured blood pressure data and confirmed that our findings were not sensitive to how we define and identify hypertensive individuals. Second, potential biases can be considered in the patient population of this study. The patients in this study were recruited from primary care clinics in Japan, and it is likely that they share a high degree of ethnic homogeneity. Also, the subjects of this study were patients exhibiting influenza-like symptoms, and it is possible that several background factors differ from those of the general population. Therefore, the external validity of the study’s findings may be limited.

Conclusions

We developed a deep learning-based algorithm for diagnosing hypertension from pharyngeal images and basic demographic information, demonstrating the utility of pharyngeal images in the diagnosis of hypertension. Although we used a specially designed medical camera in this study, our findings further suggest the feasibility of developing an algorithm that can diagnose hypertension using pharyngeal images captured by smartphones in the future, thus addressing one of the major limitations of telemedicine: the difficulty of collecting data that allow a diagnosis to be made. Such technology will be particularly useful during pandemic situations, such as the COVID-19 pandemic, when diagnoses need to be made without physical interactions between patients and healthcare providers.

Data availability statement

Data are available upon reasonable request.