Skip to main content
PMC home page
Lippincott Open Access logoLink to Lippincott Open Access
. 2022 Jun 7;43(6):1816–1823. doi: 10.1097/AUD.0000000000001246

Random Forest Classification to Predict Response to High-Definition Transcranial Direct Current Stimulation for Tinnitus Relief: A Preliminary Feasibility Study

Emilie Cardon 1,2,✉, Laure Jacquemin 2, Martin Schecklmann 3, Berthold Langguth 3, Griet Mertens 1,2, Olivier M Vanderveken 1,2, Marc Lammers 1,2, Paul Van de Heyning 1,2, Vincent Van Rompaey 1,2, Annick Gilles 1,2,4
PMCID: PMC9592176  PMID: 35666538

Objectives:

Transcranial direct current stimulation (tDCS) of the right dorsolateral prefrontal cortex has been hypothesized to reduce tinnitus severity by modifying cortical activity in brain regions associated with the perception of tinnitus. However, individual response to tDCS has proven to be variable. We investigated the feasibility of using random forest classification to predict the response to high-definition (HD) tDCS for tinnitus relief.

Design:

A retrospective analysis was performed on a dataset consisting of 99 patients with subjective tinnitus receiving six consecutive sessions of HD-tDCS at the Antwerp University Hospital. A baseline assessment consisted of pure-tone audiometry and a set of questionnaires including the Tinnitus Functional Index (TFI), Hospital Anxiety and Depression Scale, and Edinburgh Handedness Inventory. Random forest classification was applied to predict, based on baseline questionnaire scores and hearing levels, whether each individual responded positively to the treatment (defined as a decrease of at least 13 points on the TFI). Further testing of the model was performed on an independent cohort of 32 patients obtained from the tinnitus center at the University of Regensburg.

Results:

Twenty-four participants responded positively to the HD-tDCS treatment. The random forest classifier predicted treatment response with an accuracy of 85.71% (100% sensitivity, 81.48% specificity), significantly outperforming a more traditional logistic regression approach. Performance of the classifier on an independent cohort was slightly but not significantly above chance level (71.88% accuracy, 66.67% sensitivity, 73.08% specificity). Feature importance analyses revealed that baseline tinnitus severity, co-occurrence of depressive symptoms and handedness were the most important predictors of treatment response. Baseline TFI scores were significantly higher in responders than in nonresponders.

Conclusions:

The proposed random forest classifier predicted treatment response with a high accuracy, significantly outperforming a more traditional statistical approach. Machine learning methods to predict treatment response might ultimately be used in a clinical setting to guide targeted treatment recommendations for individual tinnitus patients.

Keywords: Machine learning, Random forest classification, Tinnitus, Transcranial direct current stimulation

INTRODUCTION

In normal hearing subjects, the systematized and intricate trajectory of auditory stimuli from the cochlea to the cerebral cortex effortlessly results in the perception of sound. When auditory input is reduced, for instance after cochlear damage, neuroauditory responses may drastically alter cortical circuitry and function. This is the case in chronic subjective tinnitus, where dysfunctional activation of neuronal plasticity results in the generation of a sound that can only be perceived by the patient (Langguth et al. 2013; Van de Heyning et al. 2015). These maladaptive responses can include sensory deafferentiation and release from lateral inhibition, allowing irregular spontaneous hyperactivity within neuronal networks associated with sound processing (Eggermont & Roberts 2012; Shore et al. 2016). Indeed, aberrant patterns of brain activity in tinnitus patients have been found along the auditory pathway including the auditory cortex, as well as in nonauditory areas such as the prefrontal cortex and anterior cingulate cortex (Vanneste et al. 2010a; De Ridder et al. 2011; Elgoyhen et al. 2015). Thus, the tinnitus percept may be interpreted as an emergent property resulting from activity in multiple, partially overlapping but separable networks encompassing both auditory and nonauditory areas (De Ridder et al. 2014).

Neuromodulation is the act of modifying the nervous system and bears potential as a treatment modality. It is hypothesized that by inducing neuroplastic changes, neuromodulation can interrupt abnormal cortical activity and alter or reduce the tinnitus percept (Hoare et al. 2015). For instance, repetitive transcranial magnetic stimulation (rTMS) was shown to be beneficial in the treatment of tinnitus, although effect sizes are small and duration of the treatment effect often remains limited (Soleimani et al. 2016; Liang et al. 2020). Transcranial direct current stimulation (tDCS) might be considered a viable alternative approach based on its easy, painless, and noninvasive application (Song et al. 2012; Shekhawat et al. 2015; Rabau et al. 2017; Jacquemin et al. 2018, 2019). TDCS delivers direct currents at low intensities via scalp electrodes to the cerebral cortex, where it can modulate cortical excitability in a polarity-dependent manner. Interestingly, this technique may be particularly powerful on a longer term (Paulus 2003). For instance, through the induction of synaptic plasticity, anodal tDCS can have various effects on the cell level, including the release of neurotransmitters and neurotrophic factors and the growth of dendritic spines (Pelletier & Cicchetti 2014). Furthermore, tDCS can have extensive effects on cortical connectivity, impacting oscillatory activity, and functional coupling between spatially separated brain areas (Roche et al. 2015).

In existing tDCS trials for treating tinnitus, electrodes have often been placed over either the left temporal area (LTA) or the right dorsolateral prefrontal cortex (rDLPFC). Stimulation of the LTA targets aberrant activity in the primary auditory cortex, while anodal tDCS of the rDLPFC has been proposed to strengthen deficient inhibitory top-down mechanisms and interfere with the emotional processing of tinnitus (Vanneste et al. 2010b). Collectively, outcomes of tDCS in tinnitus treatment show considerable variability, both between and within individual studies. A meta-analysis published in 2012 concluded that overall, 40% of patients respond positively to active tDCS, resulting in a mean reduction of 13.5% in tinnitus intensity (Song et al. 2012). More recently published placebo-controlled studies have led to conflicting results, with only some authors reporting significant efficacy of tDCS treatment for tinnitus (Shekhawat et al. 2013; Teismann et al. 2014; Pal et al. 2015; Forogh et al. 2016; Hyvärinen et al. 2016). Varying protocols and limited sample sizes make definitive conclusions about the efficacy of tDCS for tinnitus rather elusive (Lefaucheur et al. 2017; Cardon et al. 2019). Despite the large degree of uncertainty regarding effective outcomes of this technique, many centers currently offer some form of tDCS as an experimental tinnitus treatment. The variability in treatment response is not only disheartening for the large group of patients who do not experience any benefit, but also inefficient in a clinical setting where cost-effectiveness is often crucial.

Predictive modeling might be applied in healthcare to provide targeted treatment to individual patients and reduce unnecessary costs. It has been suggested that, compared with more traditional inferential linear statistics, machine learning methods are more suitable for predictive purposes, and superior in handling datasets containing complicated nonlinear interactions (Bzdok & Ioannidis 2019). Within the tinnitus research field, machine learning methods have recently been applied toward outcome prediction of different tinnitus treatment modalities, including cognitive behavioral therapy and physiotherapy (Niemann et al. 2020; Rodrigo et al. 2021). However, an investigation of the feasibility of machine learning models to predict outcomes from neuromodulation treatment for tinnitus is currently lacking in the literature. The current paper uses the technique of random forest classification, consisting of a large number of randomly sampled decision trees operating as an ensemble. Crucially, this method can handle complex relationships between input data, whereas maintaining a high level of interpretability (Bzdok et al. 2018). In the related field of cochlear implantation, the technique has recently been used to successfully predict treatment outcome (Kim et al. 2018). Here, we apply a random forest classifier to predict response to high-definition (HD) tDCS treatment of tinnitus.

MATERIALS AND METHODS

Data and Code Availability

Patient data are uploaded to the Zenodo data repository with restricted access and can be made available upon motivated request (DOI: 10.5281/zenodo.5011428). The code generated during this study will be made available at GitHub.

Participants

All procedures were approved by the Ethical Committee of the University of Antwerp and the Antwerp University Hospital (file number: 16/41/415). Subjects gave written informed consent at the start of the study. An overview of patient characteristics is provided in Table 1.

TABLE 1.

Baseline characteristics of all participants

General Characteristics
 Gender: m/f (n) 81/18
 Age: mean (SD) 52 (12)
 Handedness (EHI): left-handed/ambidextrous/right-handed (n) 11/4/84
 PTAlow: mean (SD) 13 (12)
 PTAhigh: mean (SD) 20 (13)
Tinnitus Characteristics
 Duration in years: mean (SD) 6.15 (7.42)
 Side: bilateral/central/left/right (n) 60/15/10/14
 Type: pure tone/noise/polyphonic (n) 68/20/11
 Etiology: otologic/spontaneous/psychological/non-otologic/unknown 46/24/4/4/21
Questionnaire Scores
 TFI: mean (SD) 46.23 (19.84)
 VAS for mean tinnitus loudness: mean (SD) 59.26 (23.70)
 VAS for maximum tinnitus loudness: mean (SD) 71.99 (20.55)
 VAS for tinnitus awareness: mean (SD) 62.38 (29.46)
 HQ: mean (SD) 18.62 (8.20)
 HADS anxiety subscale: mean (SD) 7.60 (3.99)
 HADS depression subscale: mean (SD) 6.79 (4.44)
Open in a new tab

EHI indicates Edinburgh Handedness Inventory; HADS, Hospital Anxiety and Depression Scale.; HQ, Hyperacusis Questionnaire; PTAhigh, pure-tone average for 1, 2, and 4 kHz; PTAlow, pure-tone average for 0.5, 1 and 2 kHz; TFI, Tinnitus Functional Index; VAS, Visual Analog Scale, ranging from 0 to 100.

published online ahead of print June 7, 2022.

Procedure

Study Design

This study presents a retrospective secondary analysis of a previous prospective trial, the full details of which can be found in (Jacquemin et al. 2021). In short, participants received 6 sessions of high-definition transcranial direct current stimulation (HD-tDCS) performed according to the optimal parameters concerning electrode placement, intensity, and duration of the stimulation (Shekhawat et al. 2016). At the start of the therapy and at a follow-up time point of 7 weeks after the last HD-tDCS session, tinnitus severity was evaluated using a set of self-report questionnaires.

Baseline Evaluation

Before the start of the therapy, tinnitus patients were thoroughly evaluated at our outpatient ENT clinic. Tinnitus characteristics, including duration, etiology, type (i.e., pure tone, noise, or polyphonic) and laterality, were queried and hearing levels from 125 Hz to 8 kHz were assessed using standard pure-tone audiometry. Hemispheric asymmetry has recently been suggested to influence the effects of tDCS (Brookshire & Casasanto 2018). As a proxy for hemispheric dominance, handedness of the participants was assessed using the Edinburgh Handedness Inventory (Oldfield 1971). Tinnitus severity, ranging from 0 to 100, was assessed using the Tinnitus Functional Index (TFI) (Meikle et al. 2012) and a visual analog scale (VAS) from 0 to 100 was employed to explore mean and maximum tinnitus loudness. Finally, accompanying symptoms such as hyperacusis, anxiety, and depression were assessed via the Hospital Anxiety and Depression Scale (HADS) and the Hyperacusis Questionnaire (Zigmond & Snaith 1983; Khalfa et al. 2002).

HD-tDCS

Participants received 6 sessions of anodal HD-tDCS of the right dorsolateral prefrontal cortex (rDLPFC) biweekly during 3 consecutive weeks, with a minimum interval of 1 day between subsequent sessions. Positions of the silver/silver chloride (Ag/AgCl) ring electrodes were in accordance with the 10/20 international EEG system, with the central anode at F4 and the surrounding cathodes at AF4, FC4, F6, and F2. To ensure their optimal and reliable reuse, electrodes were rotated so that each of the used 5 electrodes functioned as the central electrode an equal number of times (Hampstead et al. 2020). A constant current of 2 mA was applied for 20 minutes, with a fade-in and fade-out time of 20 seconds. Current was delivered by a 1 × 1 tDCS low-intensity stimulator and 4 × 1 multichannel stimulation adaptor (Soterix Medical Inc., New York, NY).

Outcome measures

The TFI was chosen as the primary outcome measure. A binary division between responders and nonresponders was made to reflect whether or not patients experienced a meaningful improvement in tinnitus severity (Fackrell et al. 2016). It has been reported previously that the minimal clinically relevant difference, that is, the smallest change in TFI score that an individual patient would identify as important, is 13 points (Meikle et al. 2012). Therefore, responders to the therapy were defined as participants whose TFI scores decreased by at least 13 points from baseline to follow-up.

Quantification and Statistical Analysis

Data Preparation

Only participants who completed the follow-up assessment were included in the final dataset. Observations containing missing data were removed from the dataset. Categorical variables with more than two levels (i.e., tinnitus type, etiology, and laterality) were one-hot encoded before data analysis, meaning that these variables themselves were removed and one new binary variable was added for each unique integer value in the variable.

Random Forest Classifier Construction

A random forest classifier was trained to predict whether or not a participant responded positively to the treatment, based on variables available at the baseline assessment. This machine learning method was selected as it is able to capture complex relationships between input data, can be interpreted fairly easily, and is able to handle challenges arising from relatively small sample sizes (Qi 2012; Bzdok and Ioannidis 2019). The classifier was trained using the randomForest package in R (version 3.6.2, 2019 The R Foundation for Statistical Computing) on a random subset of 64 observations, while the remaining data (n = 35) was kept apart as a test set. Hyperparameters of the classifier were optimized during the training phase, with the number of trees set at 1000 and a minimal terminal node size of 1. The number of variables randomly sampled as candidates at each split, suggested to approximate the square root of the total number of included variables (Hastie et al. 2008), was 4 for the initial model with 16 variables and 3 for the final model with the 6 most important features.

Feature Selection

Feature selection was performed during the training phase based on variable importance. As a measure for feature importance, the mean decrease in accuracy when permuting out-of-bag (OOB) data was calculated. For each tree, the error rate on the OOB portion of the data was recorded. Then, the same was done after permuting each feature. The differences between these two error rates were then averaged across all trees and normalized by the standard deviation of the differences. Features with consistently high-importance values overall validation folds were selected for the final model.

Validation of the Model

Five-fold cross-validation was performed to validate the classifier model. For each fold, a randomly sampled subset consisting of ca. 20% of the training dataset was withheld from the training phase. After the training phase, the model was tested on this validation dataset.

Cost-sensitivity

The current classification model is intended to predict response to an experimental, but noninvasive treatment. As such, the cost of false negatives was deemed to be higher than the cost of false positives; depriving potential responders from the therapy would be more detrimental than subjecting patients to a noneffective, but ultimately nonharmful, treatment. Therefore, an ensemble approach was used based on the outcomes of the five-fold cross-validation. A standard thresholding procedure was applied by modifying the cutoff values for classification. This cutoff value was determined by a stepwise procedure designed to minimize the classification error for both classes. The final cutoff value was placed at 0.36, that is, a subject was classified as a responder if the predicted positive response probability exceeded 0.36.

Testing of the Model

After validation, performance of the final model was tested on the testing dataset (n = 35). Furthermore, the model was also tested on a cohort of tinnitus patients tested at the tinnitus center of the University of Regensburg. Data were provided from 32 patients before and after traditional tDCS of the rDLPFC. Results from this trial have been published previously (E. Frank et al. 2012). Where necessary, these external data were first modified to correspond to the dataset that was used to construct the classifier. Tinnitus severity in this study was enquired by the Tinnitus Handicap Inventory (THI), a self-report questionnaire of which total scores range from 0 to 100 (Newman et al. 1996). Responders were defined as subjects whose score decreased with at least the clinically relevant difference on the THI, that is, 7 points (Zeman et al. 2011). Severity of depressive symptoms in this cohort was examined using the Beck Depression Inventory (BDI). Although these questionnaires were developed for slightly different purposes, a literature review found strong correlations between their outcomes (Bjelland et al. 2002). Therefore, scores on the BDI were rescaled to match scores on the depression subscale of the HADS. Categorial handedness data were converted to continuous data to correspond to scores on the EHI. Missing data were handled by median imputation. Statistical significance of the model performance on this dataset was assessed by the Mann-Whitney U statistic, which can be seen as equivalent to the AUC of the receiver operating characteristic (ROC) curve (Bamber 1975).

Multiple Logistic Regression

To facilitate evaluation of the random forest classifier performance, a multiple logistic regression model was designed based on the variables included in the final random forest model. Similarly to the random forest model, a thresholding procedure was applied by modifying the cutoff values for classification. The final cutoff value was placed at 0.26, that is, a subject was classified as a responder if the predicted positive response probability exceeded 0.26. A five-fold cross-validation was applied on the training dataset. Feature selection was based on Wald Chi-square statistics, which were used as a measure of feature importance, as the dataset contained both continuous and categorical data. Performance of the logistic regression model and the random forest model on the test dataset were compared using a McNemar’s Chi-squared test (Dietterich 1998).

Post Hoc Analyses

Post hoc statistical tests were performed to compare the responder and the nonresponder group concerning the topmost important features identified in the random forest classifier model. For TFI scores at baseline, two-sided t tests were used. HADS depression scores, VAS maximum loudness scores, EHI scores, tinnitus duration, and pure-tone averages were not normally distributed (as evidenced by a Shapiro-Wilk test) and for these variables, nonparametric Wilcoxon tests were used. A Bonferroni correction for multiple comparisons was applied so that differences were considered significant at α = 0.0083.

Additional Resources

The study protocol of the clinical trial, which garnered the data discussed in this paper was registered at Clinicaltrials.gov (protocol number: NCT04565132).

RESULTS

Twenty-four of 99 Participants Responded Positively HD-tDCS

A total of 99 patients completed all assessments and were included in the analysis. An overview of demographic details, tinnitus-related characteristics, and questionnaire scores at baseline is provided in Table 1.

Overall, treatment outcome was variable, with a considerable number of participants showing no improvement after treatment. On average, Tinnitus Functional Index (TFI) scores dropped from 46.23 ± 19.84 at baseline to 42.24 ± 19.83 at the follow-up time point. The difference in TFI scores between baseline and follow-up was significant (paired t test: t = 2.395, P = 0.019). In 24 of 99 participants, TFI scores decreased with the minimal clinically relevant difference of 13 points or more. These participants were classified as responders. Overall, TFI scores of most nonresponders remained on a similar level from baseline to follow-up. TFI scores of 10 participants increased with 13 or more points.

The Random Forest Classifier Predicts Treatment Response With High Accuracy

An initial random forest model with five-fold cross-validation was developed on a training dataset (n = 64), based on all available parameters at baseline. These 16 parameters consisted of demographic variables, hearing level, tinnitus characteristics, and questionnaire scores. A complete overview of all used parameters is provided in Table 1. A measure of cost-sensitivity was added to the model, penalizing false negatives more strongly than false positives. Based on these initial analyses, all features were ranked according to feature importance across all folds and the top six parameters were selected for the final model. These parameters included TFI scores at baseline, VAS for maximum tinnitus loudness, HADS depression subscale scores, EHI handedness scores, tinnitus duration, and hearing level.

The final model, tested on an unseen dataset (n = 35), achieved an accuracy of 85.71%, corresponding to a sensitivity of 100% and specificity of 81.48%. Area under the curve (AUC) of the ROC curve was 0.815 (Fig. 1A), while AUC of the precision-recall curve was 0.548 (Fig. 1B). Feature importance computation showed that TFI scores at baseline and EHI scores, indicating handedness, were of the highest importance in the development of the model (Fig. 1C). The random forest model outperformed a predictive multiple logistic regression model based on the same six parameters included in the random forest classifier (68.57% accuracy, 87.50% sensitivity, 62.96% specificity). AUC of the ROC for this multiple logistic regression model was similar as for the random forest model (Fig. 1D), whereas that of the precision-recall curve was considerably lower (Fig. 1E). A McNemar’s Chi-squared test indicated that the difference between both models was statistically significant (P = 0.041).

Fig. 1.

Fig. 1.

Open in a new tab

Performance of the random forest classifier (above) is superior to a multiple logistic regression (below). A, ROC curve for the random forest classifier. B, Precision-recall curve for the random forest classifier. C, Feature importance, based on the permutation of out-of-bag data and normalized to the most important feature. D, ROC curve for the multiple logistic regression model. E, Precision-recall curve for the multiple logistic regression model. Red dotted lines in panels (A, B, D, and E) represent classifier models without skill. EHI indicates Edinburgh Handedness Inventory; HADS, Hospital Anxiety and Depression Scale; TFI, Tinnitus Functional Index; ROC, receiver operating characteristic; VAS, Visual Analog Scale.

Post hoc analyses were performed to aid interpretation of the classifier results. Compared to nonresponders, the group of HD-tDCS responders was characterized by significantly higher baseline TFI scores (t[97] = 4.62, P < 0.001) (Fig. 2). No significant differences between responders and nonresponders were found for any of the remaining features used in the final model. Results of these post hoc tests are provided in Table 2.

Fig. 2.

Fig. 2.

Open in a new tab

The random forest classifier performs above chance level on an unrelated dataset. A, ROC curve for the random forest classifier. B, Precision-recall curve for the random forest classifier. Red dotted lines represent classifier models without skill. ROC indicates receiver operating characteristic.

TABLE 2.

Results of post hoc tests comparing responders and nonresponders

Feature P
 TFI scores at baseline <0.001*
 VAS scores for maximum tinnitus loudness 0.31
 HADS depression subscale scores 0.14
 EHI handedness scores 0.50
 Tinnitus duration 0.22
 Hearing level 0.66
Open in a new tab

A two-sided t test was used to compare TFI scores at baseline between responders and nonresponders. Nonparametric Wilcoxon tests were used for the remaining features, as these data were not normally distributed. A Bonferroni correction for multiple comparisons was applied to the results of these post hoc t tests so that results were only significant if P < 0.0083.

*denotes a significant result.

EHI indicates Edinburgh Handedness Inventory; HADS, Hospital Anxiety and Depression Scale; TFI, Tinnitus Functional Index; VAS, Visual Analog Scale.

The Random Forest Classifier Does Not Perform Significantly Above Chance Level on an External Dataset

To further explore the generalizability of the random forest classifier, its performance was tested on a dataset composed of tinnitus patients who received traditional tDCS at the tinnitus center of the University of Regensburg (Frank et al. 2012). In this cohort of 32 patients, six subjects responded to the treatment, with response being defined as a reduction of at least 7 points in the THI. The random forest classifier was able to predict treatment response with an accuracy of 71.88%, corresponding to a sensitivity of 66.67%, and specificity of 73.08%. AUC of the ROC curve was 0.635 (Fig. 3). A Mann-Whitney U test was performed to examine whether this AUC was significantly different from 0.5. No significant difference was found (P = 0.071).

Fig. 3.

Fig. 3.

Open in a new tab

Responders are characterized by a significantly higher baseline tinnitus severity than nonresponders. TFI scores at baseline were significantly higher in responders (60.45 ± 16.85) than in nonresponders (41.69 ± 18.75). Responders are presented in green, while nonresponders are presented in gray. TFI indicates Tinnitus Functional Index.

DISCUSSION

We employed a machine learning approach to predict response to HD-tDCS treatment in a group of 99 patients with subjective tinnitus. The proposed random forest classifier predicted treatment response with a high accuracy of 86%, outperforming a more traditional statistical approach. To our knowledge, the current paper represents the first attempt to predict tinnitus treatment response using a machine learning method.

The final random forest classifier achieved high accuracy with balanced sensitivity and specificity. To increase the robustness of these results and reduce the risk of overfitting, we employed a five-fold cross-validation and tested our model on a separate dataset not used for training. Thus, the evaluation of model performance was solely based on how well it predicted treatment response in unseen data. A five-fold cross-validation was performed instead of a more commonly used 10-fold cross-validation to ensure that, given our relatively small sample size and imbalanced dataset, samples from both responder and nonresponder classes were present in all folds. As an additional test of the model’s generalizability, its performance was tested on a dataset containing the outcomes of a tDCS trial performed at a different tinnitus center with different stimulation settings and different outcome measures. Although the classifier performed with acceptable accuracy, the ROC indicated that the model did not perform significantly better than chance (P = 0.07). It must be noted that the sample size of this dataset was relatively small, and that the absolute number of responders (n = 6) was low. This might lead to either over- or underestimation of model precision, and as such, estimations of model performance on this test set should be interpreted with caution. Furthermore, although these data were relatively well-aligned to the dataset collected at the Antwerp University Hospital, some unavoidable differences between the two used datasets should be noted. Patients included in the Regensburg trial received traditional tDCS instead of HD-tDCS. Although an earlier study did not find significant differences in overall efficacy of tDCS and HD-tDCS of the rDLPFC for tinnitus treatment (Jacquemin et al. 2018), there remains a distinct possibility that the working mechanisms of diffuse tDCS differ from those of targeted HD-tDCS, and that positive responders of one technique would not necessarily respond well to the other. Furthermore, tinnitus severity was examined using the Tinnitus Handicap Inventory (THI) instead of the TFI, depressive symptoms were gauged using the BDI instead of the HADS, and handedness was assessed as a self-reported categorical variable but not examined by the EHI. The drop in performance accuracy might be at least partially explained by these differences in data structure. To facilitate the comparison of treatment response and the validation of both statistical and machine learning models, we strongly suggest the standardization of stimulation protocols and participant assessment in tDCS trials.

The random forest classifier significantly outperformed a multiple logistic regression model, which was based on the exact same features. Both specificity and sensitivity were notably lower for this model than for the random forest classifier, and this difference in performance was found to be statistically significant. This finding may be an indication of the presence of more complex, nonlinear interactions in the data that are not captured by a linear model. The superior performance of the random forest classifier on the test data is a clear indication of the possible value of ensemble learning techniques for clinical datasets, even when their sample size is relatively small.

A major benefit of the random forest technique is the high level of interpretability of its results, as the importance of each feature can be calculated reliably. We identified several important features necessary to predict HD-tDCS outcome. First, the importance of both Tinnitus Functional Index (TFI) scores and maximal subjective tinnitus loudness demonstrates the relevance of baseline tinnitus severity for treatment response. Responders were characterized by significantly higher TFI scores, suggesting that patients with a higher baseline tinnitus severity might be more susceptible toward HD-tDCS treatment. The predictive effect of baseline tinnitus severity on treatment response has been shown in previous neuromodulation trials (G. Frank et al. 2010; Lehner et al. 2012). Neuromodulation effects are known to be dependent on ongoing activity in the stimulated brain area, as has been demonstrated in animal experiments (Fritsch et al. 2010). Thus, if we assume that maladaptive brain activity is more pronounced in patients with higher tinnitus distress (Vanneste et al. 2010a), this could provide an explanation for the observed predictive effect of baseline tinnitus severity.

Scores on the depression subscale of the Hospital Anxiety and Depression Scale (HADS) represented an additional feature of importance. This is a finding of particular interest, as tDCS of the dorsolateral prefrontal cortex (DLPFC) has also been used in the experimental treatment of depression (Boggio et al. 2008; Fregni et al. 2006; Loo et al. 2012). The confounding effect of concurrent depressive symptoms on tinnitus treatment response should clearly be further explored. Ideally, future clinical trials into the efficacy of HD-tDCS for tinnitus treatment should control for the confounding effect of these co-occurring symptoms, for instance by excluding participants exhibiting clinical signs of depression. Moreover, scores on a handedness inventory were found to greatly influence the random forest classifier results, and a slightly higher proportion of left-handed participants was found in the responder group. This might be an indication that the neurophysiological effects of tDCS of the DLPFC depend on hemispheric dominance and handedness, as has been shown recently (Brookshire and Casasanto 2018). However, this claim is difficult to substantiate, as many tDCS studies have exclusively tested right-handed subjects. Moreover, it must be noted that although handedness has historically been used as an easy and reliably measured proxy for cerebral lateralization, the relationship between both factors is not straightforward (McManus 2019; Güntürkün et al. 2020). As such, handedness should merely be interpreted as an indirect surrogate of hemispheric dominance that is currently more accessible than more direct measures of the phenomenon. Clearly, further research is necessary to explore the putative role of hemispheric dominance in unilateral tDCS of the rDLPFC. Further research is necessary to confirm this putative role of hemispheric dominance and handedness in unilateral tDCS of the DLPFC.

An important limitation of this study remains the absence of a sham control group. This analysis was performed on a dataset obtained in a previous study that originated in a clinical setting (Jacquemin et al. 2021). A sham-controlled trial using this HD-tDCS method is currently ongoing (Cardon et al. 2019), and results from this trial will be used to further validate the model. In the absence of a sham arm, it is currently not possible to distinguish naturally occurring fluctuations of tinnitus severity from specific effects of the HD-tDCS. The predictive factors identified in this paper do not seem to play an important role in predicting spontaneous improvement of tinnitus, although the published research on this topic mainly focuses on acute rather than chronic tinnitus (Muhlmeier et al. 2016; Simoes et al. 2021). Moreover, high-baseline tinnitus severity has not been found to predict response to other treatments such as orofacial treatment (van der Wal et al. 2020), and thus may be a specific predictor for neuromodulation treatment response. To gain more insight into the role of baseline tinnitus severity, but also handedness and depressive symptoms, as predictive factors for treatment response, we strongly recommend the inclusion of a sham control group in future tDCS trials for tinnitus treatment. Overall, in order to ultimately achieve the implementation of predictive models in a clinical setting, we advocate the collection of large datasets within randomized controlled trials, ideally using a standardized set of baseline and outcome measurements. For instance, the development of a core outcome domains set that is specifically tailored toward neuromodulation treatment in tinnitus would greatly benefit the future implementation of these machine learning models in clinical practice (Hall et al. 2018).

In conclusion, we propose a machine learning classifier able to predict response to tDCS treatment for tinnitus with high accuracy. Input data for the model are easily obtainable, allowing this model to be readily implemented and evaluated in a clinical setting. The future development and validation of treatment outcome prediction models may ultimately aid caregivers to provide targeted treatment options for individual tinnitus patients.

ACKNOWLEDGMENTS

This work was supported by an Applied Biomedical Research grant of the University of Antwerp (FWO T001618N). The funder provided support in the form of salaries for authors E.C. and A.G. but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the article.

Abbreviations:

AUC
area under the curve
BDI
Beck Depression Inventory
EHI
Edinburgh Handedness Inventory
HADS
Hospital Anxiety and Depression Scale
HD-tDCS
high-definition transcranial direct current stimulation
HQ
Hyperacusis Questionnaire
LTA
left temporal area
OOB
out-of-bag
rDLPFC
right dorsolateral prefrontal cortex
ROC
receiver operating characteristic
rTMS
repetitive transcranial magnetic stimulation
tDCS
transcranial direct current stimulation
TFI
Tinnitus Functional Index
THI
Tinnitus Handicap Inventory
VAS
Visual Analog Scale.

E.C., V.V.R. and A.G. participated in conceptualization. E.C., L.J. and A.G. did methodology. L.J., M.S., and B.L. did investigation. E.C. did formal analysis. E.C. did writing—original draft. L.J., M.S., B.L., G.M., O.M.V., M.L., P.V.d.H., V.V.R., and A.G. did writing—review and editing. V.V.R. and A.G. did supervision.

The authors have no conflicts of interest to disclose.

The data that support the findings of this study are available upon request in Zenodo (doi:10.5281/zenodo.5011428).

REFERENCES

  1. Bamber D. (1975). The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Mathe Psychol, 12, 387–415. [Google Scholar]
  2. Bjelland I, Dahl AA, Haug TT, Neckelmann D. (2002). The validity of the Hospital Anxiety and Depression Scale. An updated literature review. J Psychosom Res, 52, 69–77. [DOI] [PubMed] [Google Scholar]
  3. Boggio PS, Rigonatti SP, Ribeiro RB, Myczkowski ML, Nitsche MA, Pascual-Leone A, Fregni F. (2008). A randomized, double-blind clinical trial on the efficacy of cortical direct current stimulation for the treatment of major depression. Int J Neuropsychopharmacol, 11, 249–254. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Brookshire G, Casasanto D. (2018). Approach motivation in human cerebral cortex. Philos Trans R Soc Lond B Biol Sci, 373, 20170141. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bzdok D, Altman N, Krzywinski M. (2018). Statistics versus machine learning. Nat Methods, 15, 233–234. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bzdok D & Ioannidis JPA. (2019). Exploration, inference, and prediction in neuroscience and biomedicine. Trends Neurosci, 42, 251–262. [DOI] [PubMed] [Google Scholar]
  7. Cardon E, Van Rompaey V, Jacquemin L, Mertens G, Vermeersch H, Joossen I, Beyers J, Vanderveken OM, Van de Heyning P, Topsakal V, Gilles A. (2019). Sequential dual-site high-definition transcranial direct current stimulation (HD-tDCS) treatment in chronic subjective tinnitus: study protocol of a double-blind, randomized, placebo-controlled trial. Trials, 20, 471. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. De Ridder D., Elgoyhen A. B., Romo R., Langguth B. (2011). Phantom percepts: tinnitus and pain as persisting aversive memory networks. Proc Natl Acad Sci U S A, 108, 8075–8080. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. De Ridder D, Vanneste S, Weisz N, Londero A, Schlee W, Elgoyhen AB, Langguth B. (2014). An integrative model of auditory phantom perception: tinnitus as a unified percept of interacting separable subnetworks. Neurosci Biobehav Rev, 44, 16–32. [DOI] [PubMed] [Google Scholar]
  10. Dietterich TG. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput, 10, 1895–1923. [DOI] [PubMed] [Google Scholar]
  11. Eggermont JJ & Roberts LE. (2012). The neuroscience of tinnitus: understanding abnormal and normal auditory perception. Front Syst Neurosci, 6, 53. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Elgoyhen AB, Langguth B, De Ridder D, Vanneste S. (2015). Tinnitus: perspectives from human neuroimaging. Nat Rev Neurosci, 16, 632–642. [DOI] [PubMed] [Google Scholar]
  13. Fackrell K, Hall DA, Barry JG, Hoare DJ. (2016). Psychometric properties of the Tinnitus Functional Index (TFI): Assessment in a UK research volunteer population. Hear Res, 335, 220–235. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Forogh B, Mirshaki Z, Raissi GR, Shirazi A, Mansoori K, Ahadi T. (2016). Repeated sessions of transcranial direct current stimulation for treatment of chronic subjective tinnitus: a pilot randomized controlled trial. Neurol Sci, 37, 253–259. [DOI] [PubMed] [Google Scholar]
  15. Frank E, Schecklmann M, Landgrebe M, Burger J, Kreuzer P, Poeppl TB, Kleinjung T, Hajak G, Langguth B. (2012). Treatment of chronic tinnitus with repeated sessions of prefrontal transcranial direct current stimulation: outcomes from an open-label pilot study. J Neurol, 259, 327–333. [DOI] [PubMed] [Google Scholar]
  16. Frank G, Kleinjung T, Landgrebe M, Vielsmeier V, Steffenhagen C, Burger J, Frank E, Vollberg G, Hajak G, Langguth B. (2010). Left temporal low-frequency rTMS for the treatment of tinnitus: clinical predictors of treatment outcome–a retrospective study. Eur J Neurol, 17, 951–956. [DOI] [PubMed] [Google Scholar]
  17. Fregni F, Boggio PS, Nitsche MA, Marcolin MA, Rigonatti SP, Pascual-Leone A. (2006). Treatment of major depression with transcranial direct current stimulation. Bipolar Disord, 8, 203–204. [DOI] [PubMed] [Google Scholar]
  18. Fritsch B, Reis J, Martinowich K, Schambra HM, Ji Y, Cohen LG, Lu B. (2010). Direct current stimulation promotes BDNF-dependent synaptic plasticity: potential implications for motor learning. Neuron, 66, 198–204. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Güntürkün O, Ströckens F, Ocklenburg S. (2020). Brain Lateralization: A Comparative Perspective. Physiol Rev, 100, 1019–1063. [DOI] [PubMed] [Google Scholar]
  20. Hall DA, Smith H, Hibbert A, Colley V, Haider HF, Horobin A, Londero A, Mazurek B, Thacker B, Fackrell K; Core Outcome Measures in Tinnitus (COMiT) initiative. (2018). The COMiT’ID Study: developing core outcome domains sets for clinical trials of sound-, psychology-, and pharmacology-based interventions for chronic subjective tinnitus in adults. Trends Hear, 22, 2331216518814384. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Hampstead BM, Ehmann M, Rahman-Filipiak A. (2020). Reliable use of silver chloride HD-tDCS electrodes. Brain Stimul, 13, 1005–1007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hastie T, Tibshirani R, Friedman J. (2008). The Elements of Statistical Learning (2nd ed.). New York, NY, USA: Springer New York Inc. [Google Scholar]
  23. Hoare DJ, Whitham D, Henry JA, et al. (2015). Neuromodulation (desynchronisation) for tinnitus in adults (Protocol). Cochrane Database Syst Rev, 6. [Google Scholar]
  24. Hyvärinen P, Mäkitie A, Aarnisalo AA. (2016). Self-administered domiciliary tDCS treatment for tinnitus: a double-blind sham-controlled study. PLoS One, 11, e0154286. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Jacquemin L, Mertens G, Shekhawat GS, Van de Heyning P, Vanderveken OM, Topsakal V, De Hertogh W, Michiels S, Beyers J, Moyaert J, Van Rompaey V, Gilles A. (2021). High definition transcranial direct current stimulation (HD-tDCS) for chronic tinnitus: outcomes from a prospective longitudinal large cohort study. Prog Brain Res, 263, 137–152. [DOI] [PubMed] [Google Scholar]
  26. Jacquemin L, Mertens G, Van de Heyning P, Vanderveken OM, Topsakal V, De Hertogh W, Michiels S, Beyers J, Moyaert J, Van Rompaey V, Gilles A. (2019). An exploratory study on the use of event-related potentials as an objective measure of auditory processing and therapy effect in patients with tinnitus: a transcranial direct current stimulation study. Otol Neurotol, 40, e868–e875. [DOI] [PubMed] [Google Scholar]
  27. Jacquemin L, Shekhawat GS, Van de Heyning P, Mertens G, Fransen E, Van Rompaey V, Topsakal V, Moyaert J, Beyers J, Gilles A. (2018). Effects of electrical stimulation in tinnitus patients: conventional versus high-definition tDCS. Neurorehabil Neural Repair, 32, 714–723. [DOI] [PubMed] [Google Scholar]
  28. Khalfa S, Dubal S, Veuillet E, Perez-Diaz F, Jouvent R, Collet L. (2002). Psychometric normalization of a hyperacusis questionnaire. ORL J Otorhinolaryngol Relat Spec, 64, 436–442. [DOI] [PubMed] [Google Scholar]
  29. Kim H, Kang WS, Park HJ, Lee JY, Park JW, Kim Y, Seo JW, Kwak MY, Kang BC, Yang CJ, Duffy BA, Cho YS, Lee SY, Suh MW, Moon IJ, Ahn JH, Cho YS, Oh SH, Chung JW. (2018). Cochlear implantation in postlingually deaf adults is time-sensitive towards positive outcome: prediction using advanced machine learning techniques. Sci Rep, 8, 18004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Langguth B, Kreuzer PM, Kleinjung T, De Ridder D. (2013). Tinnitus: causes and clinical management. Lancet Neurol, 12, 920–930. [DOI] [PubMed] [Google Scholar]
  31. Lefaucheur JP, Antal A, Ayache SS, Benninger DH, Brunelin J, Cogiamanian F, Cotelli M, De Ridder D, Ferrucci R, Langguth B, Marangolo P, Mylius V, Nitsche MA, Padberg F, Palm U, Poulet E, Priori A, Rossi S, Schecklmann M, Vanneste S, et al. (2017). Evidence-based guidelines on the therapeutic use of transcranial direct current stimulation (tDCS). Clin Neurophysiol, 128, 56–92. [DOI] [PubMed] [Google Scholar]
  32. Lehner A, Schecklmann M, Landgrebe M, Kreuzer PM, Poeppl TB, Frank E, Vielsmeier V, Kleinjung T, Rupprecht R, Langguth B. (2012). Predictors for rTMS response in chronic tinnitus. Front Syst Neurosci, 6, 11. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Liang Z, Yang H, Cheng G, Huang L, Zhang T, Jia H. (2020). Repetitive transcranial magnetic stimulation on chronic tinnitus: a systematic review and meta-analysis. BMC Psychiatry, 20, 547. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Loo CK, Alonzo A, Martin D, Mitchell PB, Galvez V, Sachdev P. (2012). Transcranial direct current stimulation for depression: 3-week, randomised, sham-controlled trial. Br J Psychiatry, 200, 52–59. [DOI] [PubMed] [Google Scholar]
  35. McManus C. (2019). Half a century of handedness research: Myths, truths; fictions, facts; backwards, but mostly forwards. Brain Neurosci Adv, 3, 2398212818820513. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Meikle MB, Henry JA, Griest SE, Stewart BJ, Abrams HB, McArdle R, Myers PJ, Newman CW, Sandridge S, Turk DC, Folmer RL, Frederick EJ, House JW, Jacobson GP, Kinney SE, Martin WH, Nagler SM, Reich GE, Searchfield G, Sweetow R, et al. (2012). The tinnitus functional index: development of a new clinical measure for chronic, intrusive tinnitus. Ear Hear, 33, 153–176. [DOI] [PubMed] [Google Scholar]
  37. Mühlmeier G, Baguley D, Cox T, Suckfüll M, Meyer T. (2016). Characteristics and spontaneous recovery of tinnitus related to idiopathic sudden sensorineural hearing loss. Otol Neurotol, 37, 634–641. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Newman CW, Jacobson GP, Spitzer JB. (1996). Development of the tinnitus handicap inventory. Arch Otolaryngol Head Neck Surg, 122, 143–148. [DOI] [PubMed] [Google Scholar]
  39. Niemann U, Boecking B, Brueggemann P, Mebus W, Mazurek B, Spiliopoulou M. (2020). Tinnitus-related distress after multimodal treatment can be characterized using a key subset of baseline variables. PLoS One, 15, e0228037. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Oldfield RC. (1971). The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia, 9, 97–113. [DOI] [PubMed] [Google Scholar]
  41. Pal N, Maire R, Stephan MA, Herrmann FR, Benninger DH. (2015). Transcranial direct current stimulation for the treatment of chronic tinnitus: a randomized controlled study. Brain Stimul, 8, 1101–1107. [DOI] [PubMed] [Google Scholar]
  42. Paulus W. (2003). Transcranial direct current stimulation (tDCS). Suppl Clin Neurophysiol, 56, 249–254. [DOI] [PubMed] [Google Scholar]
  43. Pelletier SJ, Cicchetti F. (2014). Cellular and molecular mechanisms of action of transcranial direct current stimulation: evidence from in vitro and in vivo models. Int J Neuropsychopharmacol, 18, pyu047. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Qi Y. (2012). Random forest for bioinformatics. In Ensemble Machine Learning (pp. 307–323). Berlin: Springer. [Google Scholar]
  45. Rabau S, Shekhawat GS, Aboseria M, Griepp D, Van Rompaey V, Bikson M, Van de Heyning P. (2017). Comparison of the long-term effect of positioning the cathode in tDCS in Tinnitus Patients. Front Aging Neurosci, 9, 217. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Roche N, Geiger M, Bussel B. (2015). Mechanisms underlying transcranial direct current stimulation in rehabilitation. Ann Phys Rehabil Med, 58, 214–219. [DOI] [PubMed] [Google Scholar]
  47. Rodrigo H, Beukes EW, Andersson G, Manchaiah V. (2021). Exploratory data mining techniques (decision tree models) for examining the impact of internet-based cognitive behavioral therapy for tinnitus: machine learning approach. J Med Internet Res, 23, e28999. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Shekhawat GS, Stinear CM, Searchfield GD. (2013). Transcranial direct current stimulation intensity and duration effects on tinnitus suppression. Neurorehabil Neural Repair, 27, 164–172. [DOI] [PubMed] [Google Scholar]
  49. Shekhawat GS, Stinear CM, Searchfield GD. (2015). Modulation of perception or emotion? A scoping review of tinnitus neuromodulation using transcranial direct current stimulation. Neurorehabil Neural Repair, 29, 837–846. [DOI] [PubMed] [Google Scholar]
  50. Shekhawat GS, Sundram F, Bikson M, Truong D, De Ridder D, Stinear CM, Welch D, Searchfield GD. (2016). Intensity, duration, and location of high-definition transcranial direct current stimulation for tinnitus relief. Neurorehabil Neural Repair, 30, 349–359. [DOI] [PubMed] [Google Scholar]
  51. Shore SE, Roberts LE, Langguth B. (2016). Maladaptive plasticity in tinnitus–triggers, mechanisms and treatment. Nat Rev Neurol, 12, 150–160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Simões JP, Neff PKA, Langguth B, Schlee W, Schecklmann M. (2021). The progression of chronic tinnitus over the years. Sci Rep, 11, 4162. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Soleimani R, Jalali MM, Hasandokht T. (2016). Therapeutic impact of repetitive transcranial magnetic stimulation (rTMS) on tinnitus: a systematic review and meta-analysis. Eur Arch Otorhinolaryngol, 273, 1663–1675. [DOI] [PubMed] [Google Scholar]
  54. Song JJ, Vanneste S, Van de Heyning P, De Ridder D. (2012). Transcranial direct current stimulation in tinnitus patients: a systemic review and meta-analysis. ScientificWorldJournal, 2012, 427941. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Teismann H, Wollbrink A, Okamoto H, Schlaug G, Rudack C, Pantev C. (2014). Combining transcranial direct current stimulation and tailor-made notched music training to decrease tinnitus-related distress–a pilot study. PLoS One, 9, e89904. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Van de Heyning P, Gilles A, Rabau S, Van Rompaey V. (2015). Subjective tinnitus assessment and treatment in clinical practice: the necessity of personalized medicine. Curr Opin Otolaryngol Head Neck Surg, 23, 369–375. [DOI] [PubMed] [Google Scholar]
  57. van der Wal A, Van de Heyning P, Gilles A, Jacquemin L, Topsakal V, Van Rompaey V, Braem M, Visscher CM, Truijen S, Michiels S, De Hertogh W. (2020). Prognostic indicators for positive treatment outcome after multidisciplinary orofacial treatment in patients with Somatosensory Tinnitus. Front Neurosci, 14, 561038. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Vanneste S, Plazier M, der Loo Ev, de Heyning PV, Congedo M, De Ridder D. (2010a). The neural correlates of tinnitus-related distress. Neuroimage, 52, 470–480. [DOI] [PubMed] [Google Scholar]
  59. Vanneste S, Plazier M, Ost J, van der Loo E, Van de Heyning P, De Ridder D. (2010b). Bilateral dorsolateral prefrontal cortex modulation for tinnitus by transcranial direct current stimulation: a preliminary clinical study. Exp Brain Res, 202, 779–785. [DOI] [PubMed] [Google Scholar]
  60. Zeman F, Koller M, Figueiredo R, Aazevedo A, Rates M, Coelho C, Kleinjung T, de Ridder D, Langguth B, Landgrebe M. (2011). Tinnitus handicap inventory for evaluating treatment effects: which changes are clinically relevant? Otolaryngol Head Neck Surg, 145, 282–287. [DOI] [PubMed] [Google Scholar]
  61. Zigmond AS & Snaith RP. (1983). The hospital anxiety and depression scale. Acta Psychiatr Scand, 67, 361–370. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

Patient data are uploaded to the Zenodo data repository with restricted access and can be made available upon motivated request (DOI: 10.5281/zenodo.5011428). The code generated during this study will be made available at GitHub.

Articles from Ear and Hearing are provided here courtesy of Wolters Kluwer Health

RESOURCES