A study of using cough sounds and deep neural networks for the early detection of Covid-19

Abstract

The current clinical diagnosis of COVID-19 requires person-to-person contact, needs variable time to produce results, and is expensive. It is even inaccessible to the general population in some developing countries due to insufficient healthcare facilities. Hence, a low-cost, quick, and easily accessible solution for COVID-19 diagnosis is vital. This paper presents a study that involves developing an algorithm for automated and noninvasive diagnosis of COVID-19 using cough sound samples and a deep neural network. The cough sounds provide essential information about the behavior of glottis under different respiratory pathological conditions. Hence, the characteristics of cough sounds can identify respiratory diseases like COVID-19. The proposed algorithm consists of three main steps (a) extraction of acoustic features from the cough sound samples, (b) formation of a feature vector, and (c) classification of the cough sound samples using a deep neural network. The output from the proposed system provides a COVID-19 likelihood diagnosis. In this work, we consider three acoustic feature vectors, namely (a) time-domain, (b) frequency-domain, and (c) mixed-domain (i.e., a combination of features in both time-domain and frequency-domain). The performance of the proposed algorithm is evaluated using cough sound samples collected from healthy and COVID-19 patients. The results show that the proposed algorithm automatically detects COVID-19 cough sound samples with an overall accuracy of 89.2%, 97.5%, and 93.8% using time-domain, frequency-domain, and mixed-domain feature vectors, respectively. The proposed algorithm, coupled with its high accuracy, demonstrates that it can be used for quick identification or early screening of COVID-19. We also compare our results with that of some state-of-the-art works.

1. Introduction

According to the global database maintained by John Hopkins University, more than 270 million COVID-19 (and its variants) cases and 5.3 million deaths have been reported till December 13, 2021 [1]. Social distancing, wearing masks, widespread testing, contact tracing, and massive vaccination are all recommended by the World Health Organization (WHO) to reduce the spreading of this virus [2]. To date, reverse transcription-polymerase chain reaction (RT-PCR) is considered the gold standard for testing coronavirus [3]. However, the RT-PCR test requires person-to-person contact to administer, needs variable time to produce results, and is still unaffordable to most global populations. Sometimes, it is unpleasant to the children. Not least, this test is not yet accessible to the people living in remote areas, where medical facilities are scarce [4]. Alarmingly, the physicians suspect that the general people refuse the COVID-19 test in fear of stigma [5].

Governments worldwide have initiated a free massive testing campaign to stop the spreading of this virus, and this campaign is costing them billions of dollars per day at the average rate of $23 per test [6]. Hence, easily accessible, quick, and affordable testing is essential to limit the spreading of the virus. The COVID-19 detection method, using human audio signals, can play an important role here.

Researchers and clinicians have suggested using the recordings of speech, breathing, and cough sounds to detect various diseases. The results published in the literature show that the speech samples can help clinicians to detect several diseases, including asthma [7], [8], [9], [10], Alzheimer's disease [11], [12], [13], Parkinson's disease [14], [15], [16], depression [17], [18], [19], schizophrenia [20], [21], [22], autism [23], [24], head or neck cancer [25], and emotional expressiveness of breast cancer patients [26]. A comprehensive survey on these works can be found in [27]. Among these diseases, respiratory diseases like asthma have some similarities to COVID-19. An extensive investigation on the detection of asthma using audio signal processing can be found in [7], [8], [9], [10]. These works show that asthma causes swollen and inflamed vocal folds, which do not vibrate appropriately during voice generation. Hence, the voice samples of asthma patients differ from that of healthy (i.e., control) subjects. For example, it is shown in [7] that asthmatic subjects show longer pauses between speech segments, produce fewer syllables per breath, and spend a more significant percentage of time in voiceless ventilator activity than their healthy counterpart.

Recently, researchers have been suggesting using cough sounds for the early detection of the COVID-19. However, there are still some challenges as the cough is also a symptom of 30 other diseases [28], [29]. Hence, it is very challenging to discriminate the cough sound of the COVID-19 patients from that of other patients. In [28], the authors considered three diseases: bronchitis, pertussis, and COVID-19. Investigation of 247 normal cough samples and 296 pathological samples was performed. The authors used a convolutional neural network (CNN) to implement a binary classifier and a multiclass classifier. The binary classifier discriminates pathological cough sounds from normal cough sounds, and the multiclass classifier categorizes the pathologies into one of the three pathology types. In a similar work [30], the authors considered bronchitis, bronchiolitis, and pertussis. They used a CNN to discriminate against these pathologies.

Various human audio samples, namely, sustained vowel ‘/a/’, counting (1-20), breathing, and cough samples, have been used in [31]. The authors considered nine acoustic voice features: spectral contrast, mel-frequency Cepstral Coefficients (MFCCs), spectral roll-off, spectral centroid, mean square energy, polynomial fit, zero-crossing rate, spectral bandwidth, and spectral flatness. They used a random forest (RF) algorithm to discriminate the COVID-19 samples from the control/healthy samples, and they have achieved an accuracy of 66.74%.

In [32], the authors used large samples (5320 samples) selected from the MIT open voice COVID-19 cough dataset [33]. They extracted the MFCC features from the cough sounds and classified them by using a CNN. The network consists of one Poisson biomarker layer and three pre-trained ResNet50s. The results showed that their proposed system achieved an accuracy of 97%.

Cough and breathing sounds have also been used in [34]. In that work, the authors used eleven acoustic features: RMS energy, spectral centroid, roll-off frequencies, zero-crossing rate, MFCC, Δ-MFCC, ∆^2-MFCC, tempo, duration onsets, and period. In addition, they used VGGish (a pre-trained CNN from Google) to classify the samples into COVID-positive/non-COVID, COVID-positive with cough/non-COVID with cough, and COVID-positive with cough/non-COVID asthma with cough. The achieved accuracy of that system was 80%, 82%, and 80%, respectively, for the classification tasks mentioned above.

In [35], the authors used Computational Paralinguistic Challenge (COMPARE) [36] features and extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [37] to discriminate the COVID-19 samples from the healthy samples. These features were extracted by using the OpenSMILE [38] tool kit. The voice samples were collected by using five sentences uttered by the patients. The authors classified the COVID-19 patients into three categories, namely, high, mild, and low. In that study, they used 260 samples, including 52 COVID-19 samples. The authors have used a support vector machine (SVM) and achieved an accuracy of 69%.

Three acoustic feature sets have been used in [39]. The first one was the COMPARE acoustic features, which were collected by using the OpenSmile software. The second one was a combination of acoustic feature sets extracted by freely available software, PRAAT [40] and LIBROSA [41]. The third one was an acoustic feature set consisting of 1024 embedded features extracted by a deep CNN. The samples used in the investigation comprised of three vowels (i.e., ‘/a/’, ‘/s/’, and ‘/z/’), cough sounds, six symptomatic questions, and counting from 50 to 80. The authors have used the SVM with radial basis function (RBF) and RF as the classifiers. Experimental results showed an average accuracy of 80% in discriminating the COVID-19 positive patients from the COVID-19 negative patients based on the features extracted from the cough sound and vowel ‘/a/’ recordings. They achieved even more accuracy (83%) by evaluating six symptomatic questions.

In [42], the authors used voice features, namely, cepstral peak prominence (CPP), harmonic-to-noise ratio (HNR), first and second harmonic (H1H2), fundamental frequency and its variations (F0SD), Jitter, Shimmer, and maximum phonation time (MPT) to discriminate the voice samples of the COVID-19 patients from that of the healthy subjects. The authors collected the sustained vowel sample ‘/a/’ from 70 healthy and 64 COVID-19 patients of Persian speakers. They revealed significantly higher F0SD, Jitter, shimmer, H1H2, and voice break numbers in the COVID-19 patients than the control/healthy group.

Vowels in ‘/ah/’, snoring consonants in ‘/z/’, cough sound, and counting samples from 50 to 80 have been used in [43]. The authors have used a recurrent neural network (RNN) based expert classifier in work. The authors have used three techniques: pre-training, bootstrapping, and regularization to avoid the over-fitting problem of RNN. They also used the leave-one-speaker-out validation technique to achieve a recall of 78%. In a similar work [44], the authors used the RNN algorithm with long short-term memory (LSTM) to detect the COVID-19 patients. In that investigation, the authors used several features, including spectral centroid, spectral roll-off, zero-crossing-rate, MFCCs, and ΔMFCCs from the cough sound, breathing sound, and voice samples of the COVID-19 patients. The authors used 60 healthy and 20 COVID-19 patients in the work. To improve accuracy, they removed the silence part from the samples using the PRAAT software. As a result, the authors achieved an accuracy of 98.2%, 97.0%, and 77.2% by using breathing, cough, and voice samples, respectively.

In [45], the authors have used the MFCC features of cough, breathing, and voice sounds to discriminate the COVID-19 patients from the non-COVID-19 patients. The authors concluded that the MFCCs of cough and breathing sounds for the COVID-19 patients and non-COVID-19 patients are similar. However, the MFCCs of voice sounds are very distinct between the COVID-19 and non-COVID-19 patients.

A cloud computing and artificial intelligence-based early detection of the COVID-19 patients have been presented in [46]. The authors used three-voice features, namely, HNR, Jitter, and Shimmer. In addition, they used the RBF algorithm as a classifier. The authors suggested that the HNR, Jitter, and Shimmer can be used to differentiate between healthy and asthma patients. They also indicated that the same parameters can be used to discriminate the healthy and COVID-19 patients.

Recurrence quantification measures in the MFCCs have been introduced in [47] to detect the COVID-19 patients using sustained vowel ‘/ah/' and cough sounds. The authors have used several classifiers, namely, decision trees, SVM, k-nearest neighbor, RF, and XGBoost. Among these classifiers, they achieved the best results with the XGBoost classifier. That model achieved accuracies of 97% (with an F1 Score of 91%) and 99% (with an F1 Score of 89%) for coughs and sustained vowels, respectively.

In [48], the authors used crowdsourced cough audio samples that were acquired on a smartphone from around the world. They collected three acoustic features: MFCCs, mel-frequency spectrum, and spectrogram from the cough sounds. The authors used an innovative ensemble classifier model (consisting of three networks) to discriminate the COVID-19 patients from the healthy subjects. The highest accuracy achieved was 77.1%.

This work is a preliminary investigation of Artificial Intelligence's (AI's) capability to detect COVID-19 by using acoustic features. The proposed algorithm has been developed based on the available data which is limited. Rigorous testing of the algorithm is required with more data before deploying the algorithm in practice for COVID-19 pre-screening. The main contributions of this paper are as follows:

• •To develop a novel algorithm based on signal processing and a deep neural network (DNN).
• •To compute the acoustic features and compare their uniqueness for the cough sound samples of control (i.e., healthy) and COVID-19 patients.
• •To form the feature vectors using three domains: time-domain, frequency-domain, and mixed-domain, to investigate the efficacy of these feature vectors.
• •To achieve a high classification accuracy (compared to other related works) without overwhelming computation burden on the system.
• •To use a dropout strategy in the proposed algorithm to make the training process faster and to overcome the overfitting problem.
• •To provide a detailed performance analysis of the proposed system in terms of the confusion matrix, Accuracy, Precision, Negative predictive value (NPV), and F1-Score.

The rest of the paper is organized as follows. The related background is presented in Section 2. The models, materials, and methods are explained in Section 3. Simulation results and discussions are presented in Section 4. Research applicability is explained in Section 5, and the paper is concluded with Section 6.

2. Background

The human voice generation system mainly consists of lungs, larynx, and articulators. Among them, the lungs are considered the power source of the voice generation system. Respiratory diseases prevent the lungs from working properly and hence affect the human voice generation system. Respiratory diseases can be classified into two main classes, namely, (a) obstructive and (b) restrictive [49]. Obstructive lung diseases make the pulmonary airways narrow and affect a patient's ability to expel air from the lungs completely. Hence, a significant amount of air remains in the lungs all the time. On the other hand, people suffering from restrictive lung diseases cannot fully expand their lungs to fill them with air. Hence, the lungs fail to fully expand. Some patients may suffer from a combination of both obstructive and restrictive respiratory diseases. Cough is the common symptom of obstructive, restrictive, and combined lung diseases. Hence, cough sounds are considered useful for detecting lung diseases caused by respiratory issues [50].

COVID-19 is also considered a respiratory disease. Like other respiratory diseases, COVID-19 can cause the lungs to fill with fluid and get inflamed. As a result, patients can suffer from breathing difficulty and need treatment at the hospital with severe onset. Untreated COVID-19 can progress and lead to acute respiratory distress syndrome (ARDS), a form of lung failure [51]. Although coughing is a common symptom of any respiratory illness, including COVID-19, recent studies suggest that the COVID-19 cough is characterized by dry, persistent, and hoarse at the earliest stage of coronavirus infected patients. Hence, the cough sound samples of COVID-19 patients differ from those of other patients suffering from some other respiratory diseases. Human cough samples contain three phases: explosive phase, intermediate phase, and voiced phase [52], as shown in Fig. 1 . These phases represent the glottal airflow variation in the vocal cord, and they differ depending on the pathological conditions of the patients.

Fig. 1.

A typical cough sound signal [52].

Two segmented cough sound samples are randomly selected from the Virufy database [53] to investigate the differences between the cough sound samples of a COVID-negative (i.e., healthy/control) subject and a COVID-positive patient. The cough sound samples of a healthy subject and a COVID-positive patient are shown in Fig. 2 . This figure demonstrates that the healthy sample is similar to the typical human cough sound signal presented in Fig. 1. However, the cough sound sample of the COVID-19 patient varies significantly from the typical human cough sound sample. For example, both the intermediate and voiced phases are longer for the COVID-positive patient than for the healthy subject.

Fig. 2.

Comparison of the cough sounds for a healthy subject and a COVID-19 patient collected from the Virufy database [53].

Moreover, the signal amplitude during the voiced phase is higher for the COVID-positive patient than for the healthy subject. The amplitudes in the explosive phase also differ between these two cough sound samples, as depicted in Fig. 2. The differences mentioned above indicate that the cough sound can be used as a valuable tool to discriminate the COVID-positive patient from the healthy subject. The power spectral densities (PSDs) of these two samples are plotted in Fig. 3 . It is observed in the figure that the healthy cough sound has prominent frequencies of continuously decreasing magnitudes. On the other hand, the COVID-positive cough sound samples do not contain very distinct frequencies.

Fig. 3.

Comparison of the power spectral densities (PSDs) of the cough sounds for a healthy subject and a COVID-19 patient.

3. Models, materials, and methods

The proposed system model is presented in Fig. 4 . It consists of four major steps: pre-processing, feature extraction, formation of feature vectors, and classification. The main functions of the pre-processing stage are audio segmentation and windowing. Afterward, the frames are formed. In the next step, the features are extracted from the framed samples. The extracted features are then grouped to form the feature vectors. Finally, the feature vectors are applied as the input to the classifier. The most crucial component of the proposed system is feature extraction (also called the data reduction procedure). It involves extracting features from the cough sound of interest. The main advantage of using features is that the analysis algorithm (i.e., classifier) needs to deal with small and transformed data compared to original voluminous cough sound data.

Fig. 4.

Block diagram of the proposed algorithm.

In practice, acoustic features are extracted, and a feature vector is formed, representing the original data. However, the selection of features and the formation of the appropriate feature vector is an open issue for ongoing research in pattern recognition. In this investigation, 33 acoustic features are considered to form three feature vectors. The acoustic features used in this work can be broadly classified into two major classes: time-domain and frequency-domain features. In this investigation, the cough sound samples are divided into small frames using a rectangular window, and the features are extracted from these frames. These features are explained in the following subsections.

3.1. Time-domain features

In this investigation, we consider the following time-domain features: (i) short-term energy, (ii) zero-crossing rate, and (iii) entropy of energy [54]. The short-term energy of the $i$th frame is calculated by

$$E\left(i\right)=\sum _{n=1}^{W_L}{\left|x_i\left(n\right)\right|}^2,$$ (1)

where, $x_i\left(n\right)$ is the $i$th frame, with $W_L$ being the length of the frame. The energy expressed in (1) is normalized as

$$E_n\left(i\right)=\frac{E\left(i\right)}{W_L}=\frac{{\sum }_{n=1}^{W_L}{\left|x_i\left(n\right)\right|}^2}{W_L},$$ (2)

The normalized energy contents of the COVID-positive and healthy cough sounds are plotted in Fig. 5 (a). This figure shows that the energy contents of both samples are concentrated in a few frames, and they exhibit a high variation over successive frames. However, the energy content of the COVID-positive patient is much higher than that of the healthy subject. It indicates that the cough sound sample of the COVID-positive patient contains weak phonemes and a short period of silence between two coughs. Hence, the energy content also varies rapidly between two successive frames.

Fig. 5.

The time-domain features (a) Short time energy distribution, (b) Short time zero-crossing rate, and (c) Energy entropy.

The zero-crossing rate of a cough sound signal can be defined as the rate of sign changes of the movement over the frames. It is calculated by using the following equation

$$Z\left(i\right)=\frac{1}{2W_L}\sum _{n=1}^{W_L}\left|sgn\left[x_i\left(n\right)\right]-sgn\left[x_i\left(n-1\right)\right]\right|,$$ (3)

where, $sgn(•)$ is the sign function defined by $\mathit{sgn}\left[x_i\left(n\right)\right]=1$, when $x_i\left(n\right)\ge 0$ and $\mathit{sgn}\left[x_i\left(n\right)\right]=-1$, when $x_i\left(n\right)<0$. The zero-crossing rates of the COVID-positive patient and the healthy subject are plotted in Fig. 5(b), which shows that the healthy cough sound sample has a more zero-crossing rate than that of the COVID-positive patient. Since the zero-crossing rate measures the noisiness of a signal, it exhibits a higher zero-crossing rate for the unvoiced part of the cough sound sample and a lower zero-crossing rate for the voiced samples. As shown in Fig. 2, the voiced phase of the cough sound samples for the COVID-positive patient is longer than that of the healthy subject. Hence, the zero-crossing rate is lower for the COVID-positive patient than for the healthy subject, as depicted in Fig. 5(b). The short-term entropy of energy can be interpreted as a measure of the abrupt changes in the energy level of an audio signal. To compute it, we first divide each short-term frame into $K$ sub-frames of fixed duration. Then, for each sub-frame, $j$, the energy is calculated by using (1) and divide it by the total energy, $E_{short\_fram{e}_i}$ of the short-term frame. Then, the sub-frame energy values, $e_j$, for j =1,2, …, K, is computed as a sequence of probabilities and is defined as

$$e_j=\frac{E_{sub\_fram{e}_j}}{E_{short\_fram{e}_i}},$$ (4)

where, $E_{short\_fram{e}_i}=\sum _{k=1}^K{E}_{sub\_fram{e}_k}$. At the final step, the entropy, $H\left(i\right)$, is calculated from the sequence $e_j$ by

$$H\left(i\right)=-\sum _{j=1}^K{e}_j\mathit{lo}{g}_2\left(e_j\right),$$ (5)

The short-term entropy of energy for the COVID-positive patient and the healthy subject are plotted in Fig. 5(c). The short-term entropy of energy for the COVID-positive patient is greater than that of the healthy subject for most of the frames. Since the energy content of the COVID-positive patient varies more abruptly than that of the healthy subject, the energy entropy tends to be higher for the COVID-positive patient after frame 20, as shown in Fig. 5(c).

3.2. Frequency domain features

Frequency-domain acoustic features are extracted from the discrete Fourier transform (DFT) of a signal. The DFT of a frame of audio signal can be expressed as

$$X_i\left(k\right)=\sum _{n=1}^{N-1}{x}_i\left(n\right)e^{-j\frac{2\pi }{N}nk},$$ (6)

where, $N$ is the size of the DFT, $X_i\left(k\right)$ is the value of the DFT coefficients, and $k=1,2,...W_L$. The spectral centroid dictates a noise-robust estimate of the dominant frequency for the cough sound signal that varies over time. It is also called the center of gravity of the spectrum. The value of the spectral centroid, $C_i$, for the $i$th audio frame is calculated by

$$C_i=\frac{{\sum }_{k=1}^{W_L}k{X}_i\left(k\right)}{{\sum }_{k=1}^{W_L}{X}_i\left(k\right)},$$ (7)

The spectral centroids of the COVID-19 positive and the healthy person are shown in Fig. 6 (a). It is shown in the figure that the spectral centroids of the cough sound for the healthy person are higher compared to those of the COVID-19 cough sound samples until approximately frame number 50. The highest value corresponds to the brightest sound practically. Usually, the existence of noise, silence, etc. signifies the lower values of the spectral centroid. This is noticeable for COVID-positive patient as opposed to the healthy person for the range mentioned above. From nearly 50-80 frames, the COVID-positive patient exhibits higher values of the spectral centroid. After frame number 80, both the samples show insignificant spectral components.

Fig. 6.

The frequency-domain features (a) Spectral centroid, (b) Spectral entropy, and (c) Spectral flux

Spectral entropy is a measure of irregularities in the frequency domain. The spectral entropy features are computed from the short-time Fourier transform (STFT) spectrum. Spectral entropy is widely used to detect the voiced regions of an acoustic signal. The flat distribution of silence or noise induces high entropy values. The spectral entropy is computed with the same method that follows to calculate the cough signal's energy entropy. First, the spectrum of the short-term frame is divided into $L$ sub-bands. The energy, $E_f$, of the $f$th sub-band, where $f=0,1,2,\dots ,\left(L-1\right)$, is normalized by the total spectral energy. The normalized energy is defined as $n_f=\frac{E_f}{{\sum }_{f=0}^{L-1}{E}_f}$, $f=0,1,2,\dots ,\left(L-1\right)$. Finally, the entropy of the normalized spectral energy, $n_f$, is computed by

$$H=-\sum _{f=0}^{L-1}{n}_f\mathit{lo}{g}_2\left(n_f\right),$$ (8)

The spectral entropies of the COVID-positive and the healthy person are shown in Fig. 6(b). This figure shows that the spectral entropy of the healthy person is higher than that of the COVID-positive patient for most of the frames. The reason is that the voiced part of the signal contains less spectral entropy than the unvoiced one.

The spectral flux measures the spectral change between two successive frames. The spectral flux is computed as the squared difference between the normalized magnitudes of the spectra for the two subsequent short-term windows. It is defined by

$$F{l}_{\left(i,i-1\right)}=\sum _{k=1}^{W_L}{\left[E{N}_i\left(k\right)-E{N}_{i-1}\left(k\right)\right]}^2,$$ (9)

where, $E{N}_i\left(k\right)=\frac{X_i\left(k\right)}{{\sum }_{l=1}^{W_L}{X}_i\left(l\right)}$. The spectral fluxes of the cough sound sample for the COVID-positive and the healthy person are plotted in Fig. 6(c). The magnitudes of the spectral flux are higher for the healthy person compared to the COVID-positive patient for most frames. The reason is the more frequent local spectral changes in the healthy cough sound samples than in the COVID-positive ones. This indicates more rapid spectral alternation among phonemes in the healthy cough sound sample than in the COVID-positive patient.

The spectral roll-off is the frequency below which a certain percentage (usually around 90%) of the magnitude distribution of the spectrum is concentrated. Therefore, if the $m$th DFT coefficient corresponds to the spectral roll-off of the $i$th frame, then it satisfies the following equation

$$\sum _{k=1}^m{X}_i\left(k\right)=C\sum _{k=1}^{W_L}{X}_i\left(k\right),$$ (10)

where, $C$ is the adopted percentage (user parameter). The spectral roll-off frequency is usually normalized by dividing it with $W_L$, so that it takes values between 0 and 1. The spectral roll-offs of the cough sound samples for the healthy person and the COVID-positive patient are shown in Fig. 7 (a). It can be easily observed that the cough sound samples of the healthy person show a higher spectral roll-off value than that of the COVID-positive patient for most of the frames. It means that the cough sound sample of the healthy person has a wider spectrum compared to that of the COVID-positive patient.

Fig. 7.

The frequency-domain features (a) Spectral roll-off, (b) MFCC coefficient, (c) Chroma vector, and (d) Feature harmonics.

We also include the MFCCs to form the feature vector. The MFCCs have been widely used in respiratory disease detection algorithms for a long time [55], [56], [57]. The main advantage of the MFCCs over other acoustic features is that they can completely characterize the shape of the vocal tract configuration. Once the vocal tract is accurately characterized, we can estimate an accurate representation of the phonemes being produced by the vocal tract. The shape of the vocal tract manifests itself in the envelope of the short-time power spectrum, and the MFCCs accurately represent this envelope [58]. The following procedure is used to compute the MFCCs [59]. The voice sample, $x\left[n\right]$ is first windowed with an analysis window $w\left[n\right]$ and the STFT, $X\left(n,{\omega }_k\right),$ is computed by

$$X(n,{\omega }_k)=\sum _{m=-\infty }^{\infty }x\left[m\right]w[n-m]e^{-j{\omega }_{k}m},$$ (11)

where, ${\omega }_k=\frac{2\pi k}{N}$, with $N$ being the DFT length. The magnitude of $X\left(n,{\omega }_k\right)$ is then weighted by a series of filter frequency responses whose center frequencies and bandwidth are roughly matched with the auditory critical band filters called mel scale filters. The next step is to compute the energy using the STFT, weighted by each mel scale filter frequency response. The energy for each speech frame at time, $n$ and for the $l$th mel-scale filter is given by

$$E_{mel}\left(n,l\right)=\frac{1}{A_l}\sum _{k=L_l}^{U_l}{\left|V_l\left({\omega }_k\right)X\left(n,{\omega }_k\right)\right|}^2,$$ (12)

where, $V_l\left(\omega \right)$ is the frequency response of the $l$th mel-scale filter, and $L_l$ and U_l are the lower and upper-frequency indices, respectively, over which each filter is nonzero, while $A_l$ is defined as

$$A_l=\sum _{k=L_l}^{U_l}{\left|V_l\left({\omega }_k\right)\right|}^2,$$ (13)

The cepstrum, associated with $E_{mel}\left(n,l\right)$, is then computed for the speech frame at time, $n$ by

$$C_{mel\left[n,m\right]}=\frac{1}{R}\sum _{l=0}^{R-1}log\left(E_{mel}\left(n,l\right)\right)cos\frac{2\pi ml}{R},$$ (14)

where $R$ is the number of filters. In this work, we consider 13 MFCCs. The plots for the arbitrarily chosen 7^th coefficient of the MFCCs for both the healthy cough sound samples and COVID-positive cough sound samples are shown in Fig. 7(b). It is shown in the figure that the magnitude of the 7^th MFCC coefficient is higher for the COVID-positive cough sound sample compared to that of the healthy cough sound for most of the frames.

The chroma vector used in this work is a 12-element representation of spectral energy. The chroma vector is computed by grouping the DFT coefficients of a short-term window into 12 bins. Each bin represents the 12 equal-tempered pitch classes of semitone spacing. Also, each bin produces the mean of the log-magnitudes of the respective DFT coefficients, defined by

$$v_k=\sum _{n\in S_k}\frac{X_i\left(n\right)}{N_k},kϵ0,\dots ,11$$ (15)

where, $S_k$ is a subset of the frequencies that correspond to the DFT coefficients and $N_k$ is the cardinality of $S_k$. In the context of a short-term feature extraction procedure, the chroma vector $v_k$ is usually computed on a short frame basis. This results in a matrix $V$, with elements $V_{k,i}$, where indices $k$ and $i$ represent pitch-class and frame-number, respectively. The chroma vector plots of the healthy and the COVID-positive cough sound samples are shown in Fig. 7(c). It is shown that the chroma vector of the healthy person shows one dominant coefficient, and the rest of the coefficients are of small magnitudes. On the other hand, the chroma vector of the COVID-positive cough sound sample is noisier and does not have any dominant coefficient. In addition, the chroma vector of the cough sound sample for the COVID-positive patient does not contain any nonzero coefficient.

The autocorrelation function for the $i$th frame is computed by

$$R_i\left(m\right)=\sum _{n=1}^{W_L}{x}_i\left(n\right)x_i\left(n-m\right),$$ (16)

Actually, $R_i\left(m\right)$ is the correlation of the $i$th frame with itself at time lag, $m$. Then the autocorrelation function is normalized as

$$r_i\left(m\right)=\frac{R_i\left(m\right)}{\sqrt{{\sum }_{n=1}^{W_L}{x_i}^2\left(n\right){\sum }_{n=1}^{W_L}{x_i}^2(n-m)}},$$ (17)

Afterward, the maximum value of $r_i$, i.e., the harmonic ratio is calculated as

$$H{R}_i=\max \left\{r_i\left(m\right)\right\},$$ (18)

where $T_{\min }\le m\le T_{\max }$, $T_{min}$ and $T_{max}$ are the minimum and maximum allowable values of the fundamental period. Here, $T_{max}$ is often defined by the user, whereas $T_{min}$ usually corresponds to the lag in time for which the first zero crossing of the $r_i\left(m\right)$ occurs. The plots for the harmonic ratio of the healthy and the COVID-positive patients are shown in Fig. 7(d). It is depicted in the figure that the harmonic ratio of the cough sound sample for the healthy person is higher for most of the frames. However, the harmonic ratio shows nonzero values for all analysis frames of the COVID-positive cough sound samples. On the other hand, the harmonic ratio of the healthy person has zero values for some of the analysis frames.

In this work, the cough sound samples collected from the Virufy database [53] are used. The Virufy is a volunteer-run organization, which has built a global database to identify the COVID-19 patients using AI. The database contains both clinical data and crowdsourced data. The clinical data is accurate because it was collected and reprocessed at a hospital following a standard operating procedure (SOP). Qualified physicians monitored the whole process of data collection. The subjects were confirmed as healthy persons (i.e., COVID-19 negative) and COVID-19 patients (i.e., COVID-positive) by using the RT-PCR test, and the data was labeled accordingly. The database also contains the patients’ information, including age, gender, and medical history. Virufy provided 121 segmented cough samples from these 16 patients. The Virufy database contains both the original cough audio recordings and the segmented version of the cough sounds. The segmented cough sounds were created by identifying the periods of relative silence in the recordings and separating cough sound samples based on those silences. The segments with no coughing sound or having too much background noise were removed. The crowdsourced data, maintained by Virufy, is diverse and donated by patients from multiple countries. This database is significantly increasing in volume over time as more people are contributing their cough samples. In this work, only the clinically collected cough samples are used as they are more authentic than crowdsourced data and, also, the segmented cough samples are used.

A DNN discriminates the COVID-19 cough sound samples from the healthy cough sound samples, as shown in Fig. 4. The DNN model presented in [60] is used and modified to implement the proposed system. The DNN used in the network consists of three hidden layers. Each hidden layer consists of 20 nodes. The network has 500 input nodes for the matrix input. It has only one output node as the decision is binary. The output node employs the softmax activation function, whereas the hidden nodes consist of the sigmoid function. One of the limitations of the DNN is that they are vulnerable to overfitting. This problem worsens as the network includes more nodes. To solve the overfitting problem, we employ a dropout algorithm. This algorithm trains only some of the randomly selected nodes rather than all the entire network nodes. The dropout effectively prevents overfitting as it continuously alters the nodes and weights in the training process. In this work, a dropout ratio of 10% and 20% are used for the input and hidden layers, respectively.

4. Simulation results and discussion

For biomedical signals classification, findings are made in the context of medical prognosis [61]. Therefore, in COVID-19 cough sound sample detection, we need to provide a clinical or diagnostic interpretation of the rule-based classifications made with the acoustic features pattern. The following terminologies and performance parameters are used [55,62]:

True positive (TP) occurs when the predicted test is positive for COVID while the subject is also COVID-positive. True negative (TN) occurs when the predicted test is negative for COVID and the subject is COVID-negative as well.

Sensitivity or Recall is denoted by $S^{+}$ and it is defined by

$$S^{+}=\frac{\text{number}\text{of}\mathit{TP}\text{decision}}{\text{number}\text{of}\text{the}\text{actual}\text{pathological}\text{subjects}}.$$ (19)

Specificity is denoted by $S^{-}$ and it is given by

$$S^{-}=\frac{\text{number}\text{of}TN\text{decision}}{\text{number}\text{of}\text{the}\text{actual}\text{healthy}\text{subjects}}.$$ (20)

False-negative (FN) occurs when the test is negative for a subject who possesses the COVID. The probability of this error, known as the false-negative fraction (FNF), is given by

$$FNF=\frac{\text{number}\text{of}FN\text{decision}}{TP+\text{number}\text{of}FN\text{decision}}.$$ (21)

False-positive (FP) is defined as the case when the predicted result is COVID-positive, but the individual is COVID-negative. The probability of this type of error or a false alarm, known as the false-positive fraction (FPF), is given by

$$FPF=\frac{\text{number}\text{of}FP\text{decision}}{TN+\text{number}\text{of}FP\text{decision}}.$$ (22)

Accuracy is simply a ratio of the correctly predicted observations to the total number of observations. The accuracy is defined by

$$Accuracy=\frac{TP+TN}{TP+FP+FN+TN}.$$ (23)

Precision or Positive Predictive Value (PPV) is the ratio of the correctly predicted positive observations to the total predicted positive observations. The precision is defined by

$$Precision=\frac{TP}{TP+FP}.$$ (24)

F1 Score is the weighted average of the Precision and Recall. Therefore, this score takes both false positives and false negatives into account. The F1 Score is defined by,

$$F1Score=\frac{2*Recall*Precision}{Recall+Precision}.$$ (25)

NPV (Negative Predictive Value) represents the percentage of the cases labelled as truly negative. The NPV is defined by

$$NPV=\frac{TN}{TN+FN}.$$ (26)

The samples are distributed into three parts: 70% are for training the DNN, the remaining 30% into validation, and testing with a ratio of 2:1. Five-fold validation is used. The data samples and patient information are listed in Table 1 . The proposed system's training, validation, and testing results with the three feature vectors are listed in Table 2 .

Table 1.

Data Samples

Sample	Corona test	Age	Gender	Medical history	Reported symptoms	Cough file name
1	Negative	53	Male	None	None	neg-0421-083-cough-m-53.mp7
2	Positive	50	Male	Congestive heart failure	Shortness of breath	pos-0421-084-cough-m-50.mp3
3	Negative	43	Male	None	Sore throat	neg-0421-085-cough-m-43.mp3
4	Positive	65	Male	Asthma/chronic lung disease	Shortness of breath, new or worsening cough	pos-0421-086-cough-m-65.mp3
5	Positive	40	Female	None	Sore throat, loss of taste, loss of smell	pos-0421-087-cough-f-40.mp3
6	Negative	66	Female	Diabetes with complication	None	neg-0421-088-cough-f-66.mp3
7	Negative	20	Female	None	None	neg-0421-089-cough-f-20.mp3
8	Negative	17	Female	None	Shortness of breath, sore throat, body aches	neg-0421-090-cough-f-17.mp3
9	Negative	47	Male	None	New or worsening cough	neg-0421-091-cough-m-47.mp3
10	Positive	53	Male	None	Fever, chills, or sweating, shortness of breath, new or worsening cough, sore throat, loss of taste, loss of smell	pos-0421-092-cough-m-53.mp3
11	Positive	24	Female	None	None	pos-0421-093-cough-f-24.mp3
12	Positive	51	Male	Diabetes with complication	Fever, chills, or sweating, new or worsening cough, sore throat	pos-0421-094-cough-m-51.mp3
13	Negative	53	Male	None	None	neg-0422-095-cough-m-53.mp3
14	Positive	31	Male	None	Shortness of breath, new or worsening cough	pos-0422-096-cough-m-31.mp3
15	Negative	37	Male	None	None	neg-0422-097-cough-m-37.mp3
16	Negative	24	Female	None	New or worsening cough	neg-0422-098-cough-f-24.mp3

Table 2.

Training and Testing Accuracy of the Feature Vectors

Feature Vector	Training Accuracy (%)	Validation Accuracy (%)	Testing Accuracy (%)
Time-domain feature vector	100	93.27	89.20
Frequency-domain feature vector	100	98.50	97.50
Mixed feature vector	100	96.37	93.80

First, the time-domain feature vector is used that has three acoustic features, namely, zero-crossing rate, energy, and energy entropy. Then, the DNN (with five-fold cross-validation) is trained, and the system is tested with the time-domain feature vector. The results are shown in Table 2, with an average training accuracy of 100%, validation accuracy of 93.27%, and testing accuracy of 89.20%. The classification matrix [61] of the time-domain feature vector is provided in Table 3 . Based on the data presented in Table 3, it can be concluded that the DNN can correctly detect the COVID-positive cough sound samples with a sensitivity of 86.67% by using the time-domain features. On the other hand, it can accurately identify healthy cough sound samples with a specificity of 91.67%.

Table 3.

The Classification Matrix of the Time-Domain Feature Vector

Actual	Prediction (%)
	Healthy	COVID-19
Healthy	91.67% ($S^{-})$	8.33% ($FPF)$
COVID-19	13.33% ($FNF)$	86.67% ($S^{+})$

Simulations are repeated by using the frequency-domain feature vector. As mentioned before, the features considered are spectral centroid, spectral entropy, spectral flux, spectral roll-offs, MFCCs, and chroma vector. The training, validation, and testing results are also listed in Table 2. The data shows that the DNN achieves training accuracy of 100%, validation accuracy of 98.50%, and testing accuracy of 97.50% by using the frequency-domain feature vector. It can be concluded that the testing accuracy of the frequency-domain feature vector is higher than that of the time-domain feature vector. The classification matrix of the frequency-domain feature vector is presented in Table 4 , which shows that the frequency-domain feature vector boosts the DNN's ability to detect the COVID-positive cough sound samples with a sensitivity of 95%. Moreover, the DNN can accurately identify the healthy samples with a specificity of 100%. Both parameters are higher than those of the time-domain feature vector.

Table 4.

The Classification Matrix of the Frequency-Domain Feature Vector

Actual	Prediction (%)
	Healthy	COVID-19
Healthy	100.00% ($S^{-})$	0.00% ($FPF)$
COVID-19	5.00% ($FNF)$	95.00% ($S^{+})$

Lastly, time-domain and frequency-domain features are combined to form a mixed-feature vector. The training, validation, and testing accuracies for the mixed feature vector are listed in Table 2. The achieved training, testing, and validation accuracies are 100%, 96.37%, and 93.80%, respectively. The classification matrix of the mixed-feature vector is presented in Table 5 . The DNN can detect COVID-positive cough sound samples with a sensitivity of 93.34%. On the other hand, it can accurately identify the healthy cough sound samples with a specificity of 94.17%.

Table 5.

The Classification Matrix of the Mixed Feature Vector

Actual	Prediction (%)
	Healthy	COVID-19
Healthy	94.17% ($S^{-})$	5.82% ($FPF)$
COVID-19	6.67% ($FNF)$	93.34% ($S^{+})$

The performances of the proposed system in terms of Accuracy, Precision, F1 Score, and NPV for the time-domain feature vector, frequency-domain feature vector, and mixed domain feature vector are listed in Table 6 . This table shows that the proposed system achieves the highest accuracy of 97.5% using the frequency-domain feature vector. On the other hand, the lowest accuracy of 89.2% is achieved using the time-domain feature vector. The other performance scores, including Precision, F1 Score, and NPV, are the highest for the frequency-domain feature vector.

Table 6.

The Performance Comparison

Measures	Time-domain feature vector	Frequency- domain feature vector	Mixed feature vector
Accuracy	0.892	0.975	0.938
Precision/ PPV	0.912	1.000	0.941
F1 Score	0.889	0.974	0.937
NPV	0.873	0.952	0.934

Cough is regarded as a natural defense mechanism of some respiratory disorders, including COVID-19. The human audible hearing range impaired existing subjective clinical approaches of cough sound analysis [63]. Exploration of noninvasive diagnostic approaches well above the audible frequency range (i.e., 48000 Hz) used for sample data can overcome this limitation as demonstrated in this study. The non-stationary characteristics of cough sound samples impose additional challenges for signal processing-based approaches. Also, cough patterns show variability in human subjects under the same pathological state. The cough features that are closely tied to the intensity levels as in the time domain can have dissimilarity for the identical pathology. The cough sound is characterized by the fundamental frequency and significant harmonics when pathology is involved. The restriction of airways causes turbulence in the cough sound that constitutes the harmonics [52]. More realistically, a method that captures both time and frequency changes over the cough sound samples should associate the investigated respiratory disorder, i.e., COVID-19, with greater accuracy. The best diagnostic performance with the frequency-domain feature vector as in Table 6 justifies that the cough sound features distributed in the frequency domain should possess greater significance.

Finally, the performance of the proposed system is compared with other related works available in the literature, as listed in Table 7 . The comparison table shows that the proposed system achieves a higher accuracy of 97.5% with the frequency-domain feature vector using the cough sound samples compared to [44]. The system achieves even higher accuracy with the time-domain and mixed-feature vector than the works published in [31, [34], [35], 64].

Table 7.

The Performance Comparison with Existing Works

Research Work	Samples	Phonemes	Features	Classifier	Accuracy
N. Sharma [31]	Healthy and COVID-positive: 941	Cough, Breathing, Vowel, and Counting (1-20)	Spectral contrast, MFCC, Spectral roll-off, Spectral centroid, Mean squareenergy, Polynomial fit, zero-crossing rate, Spectralbandwidth, and Spectral flatness	RF	66.74%
C. Brown et al. [34]	COVID-positive: 141, Non-COVID: 298, COVID-positive with Cough:54, Non-COVID with Cough:32, Non-COVID asthma: 20	Cough, and Breathing	RMS energy, Spectral centroid, Roll-off frequencies, Zero-crossing, MFCC, Δ-MFCC,${\Delta }^{\Lambda }2$ -MFCC, Duration, Tempo Onsets, Period	CNN	80%
J. Han [35]	COVID-Positive: 52, Healthy: 208	Voice	COMPARE, and eGeMAPS	SVM	69%
A.Hassan [44]	COVID-Positive: 20, Healthy: 60	Breathing, Cough, and Voice	Spectral centroid, Roll-off frequencies, Zero-crossing, MFCC, and Δ-MFCC	RNN	98.2% (Breathing), 97% (Cough), 88.2% (Voice)
V. Espotovic [64]	COVID-Positive: 84, COVID-Negative: 1019	Voice, Cough, and Breathing	Wavelet	Ensemble Boosted	88.52%
Proposed System (time-domain)	COVID-Positive: 50, Healthy: 50	Cough	zero-crossing rate, energy, and energy entropy	DNN	89.2%
Proposed System (Frequency-domain)	COVID-Positve:50, Healthy: 50	Cough	Spectral centroid, spectral entropy, spectral flux, spectral roll-offs, MFCC, and chroma vector	DNN	97.5%
Proposed System (Mixed- feature)	COVID-Positve:50, Healthy: 50	Cough	zero-crossing rate, energy, energy entropy, spectral centroid, spectral entropy, spectral flux, spectral roll-offs, MFCC, and chroma vector	DNN	93.8%

5. Research applicability

Since the publicly available databases are restricted to COVID-positive and COVID-negative (i.e., healthy/control) cases, this study focuses on discriminating COVID-19 cough sound from the healthy cough sound. However, the proposed algorithm can have a possibility to differentiate pathological cough sounds into distinct pulmonary/respiratory diseases, including COVID-19, asthma, bronchiectasis, etc. The pathophysiology and acoustic property of cough sounds can provide significant information in the frequency domain to characterize them for multi-classification purpose. Asthma causes the airways of the patient to be inflamed and narrower. On the other hand, bronchiectasis damages the airways and widens them abnormally. Few randomly selected cough sound samples of some respiratory disorders are investigated in [65]. The samples available in [66] are not sufficient to apply the proposed deep learning-based algorithm. One sample of asthma and bronchiectasis cough sound each is shown in Fig. 8 to demonstrate their uniqueness in the time domain. Bronchiectasis cough sound has longer cough sequences compared to asthma cough sound.

Fig. 8.

The cough sound samples of asthma and bronchiectasis.

Additionally, the bronchiectasis cough sound demonstrates more flow spikes than the asthmatic cough sound [52]. These flow spikes indicate more severe inflammation in bronchiectasis patients than in an asthmatic patient. Comparing Fig. 2 and Fig. 8, it can be concluded that the explosive, intermediate, and voiced phases are very distinct in the COVID-19 cough sample; however, these phases are hardly visible in asthma and bronchiectasis cough sounds.

As demonstrated in this study, some of the frequency-domain features of COVID-19, asthma, and bronchiectasis cough samples are plotted in Fig. 9 to show their uniqueness. The spectral entropy of the bronchiectasis sample is much higher for most of the frames compared to COVID-19 and asthma cough samples. The other features including spectral flux, MFCC, and feature harmonics are also non-identical for the mentioned three respiratory disorders. The distinct differences for the frequency domain features indicate that the proposed algorithm can also be applied to differentiate COVID-19 from asthma and bronchiectasis cough samples, provided a good number of datasets are available for each class.

Fig. 9.

The frequency domain features of (a) Spectral entropy, (b) Spectral flux, (c) MFCC coefficient (6^th), and (d) Feature harmonics for COVID-19, asthma, and bronchiectasis cough samples.

6. Conclusion

In this paper, a DNN-based study for the early detection of COVID-19 patients has been presented using cough sound samples. This study proposed a system that extracts the acoustic features from the cough sound samples and forms three feature vectors. A rigorous, in-depth investigation has been provided in this work to show that the cough sound samples can be a valuable tool to discriminate the COVID-19 cough sound from other healthy cough sound samples for preliminary noninvasive assessment. In this work, it has been shown that some acoustic features are unique in the cough sound samples of the COVID-19 patients and hence can be used by a classifier like DNN to discriminate them from the healthy cough sound samples successfully. However, there has always been an argument about selecting the appropriate acoustic features for the classifications. The major challenges are (a) to decide whether to use a single feature (like MFCC, spectrogram, etc.) or feature vector, (b) to select the appropriate combination of acoustic features to form the feature vector, and (c) to choose the appropriate domain (i.e., time-domain, frequency-domain, or both). Three feature vectors have been investigated in this work to address this issue. It is shown and justified that the frequency-domain feature vector has provided the highest accuracy compared to the time-domain or mixed-domain feature vector.

The performance of the proposed system has been compared with those of other existing state-of-the-art methods that are presented in the literature for the diagnosis of COVID-19 patients from audio samples. The proposed noninvasive pre-diagnosis technique can enhance the screening of COVID-positive cases, including asymptomatic and pre-symptomatic patients. Also, early diagnosis can help them to stay in touch with healthcare providers for a better prognosis to avoid the critical consequences of COVID-19.

For future work, more focus will be given to investigate the progression level of the COVID-19 patients by using the cough sound analysis. Furthermore, since some other respiratory diseases produce similar cough sounds, it is imperative to compare the cough sound features of the COVID-19 patients with those of the other respiratory diseases. Currently, we are actively seeking data to investigate the mentioned issues.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.