A validation and potential clinical application of multivariate analyses of DPOAE data

Abstract

Objective

To test the generalizability of multivariate analyses of DPOAE data. Previously published multivariate solutions were applied to a new set of data to determine if test-performance improvements, evident in previous reports, are retained. An additional objective was to provide an alternative approach for making multivariate dichotomous decisions of hearing status in the clinic, based on DPOAE measurements.

Design

DPOAE level and noise were obtained in 345 ears of 187 subjects. Approximately 1/3 of the subjects had normal hearing, while the remainder had hearing loss, ranging from 25 to more than 120 dB HL. DPOAE data were collected at each of nine frequencies. Following data collection, clinical decision theory, in combination with univariate (DPOAE level and SNR) and multivariate (logistic regression) analyses, was used to construct ROC curves and to generate ROC curve areas. In addition, test performance was assessed by fixing the false-alarm rate and comparing different approaches to analyses in terms of their failure rates as a function of magnitude of hearing loss. The DPOAE test results were compared to either single-frequency or multi-frequency gold standards. The multivariate solutions were taken from previously published work (Dorn et al., 1999; Gorga, et al, 1999).

Results

DPOAE level and SNR resulted in roughly equivalent test performance (ROC curve areas and failure rates among ears with hearing loss), although DPOAE level performed better for frequencies above 1 kHz, and SNR performed better for frequencies at 0.75 and 1 kHz. Multivariate analyses resulted in better test performance for nearly all conditions, compared to the univariate approaches that used either DPOAE level or SNR. The improvements in test performance were greatest for the frequencies at which the univariate analyses performed poorest (0.75 kHz, 1 kHz, and 8 kHz). Less difference was observed between univariate and multivariate approaches when multi-frequency gold standards were used; however, even for the multi-frequency cases, multivariate analyses generally resulted in better performance. An approach that might facilitate the interpretation of multi-frequency DPOAE measurements in the clinic is described.

Conclusions

Previously described multivariate analyses were robust in that they improved test performance when applied to an entirely new set of DPOAE data. This, in turn, suggests that the previously described multivariate solutions may have clinical utility in that they are expected to improve test performance at no additional cost in terms of data-acquisition or data-analysis time. In addition to demonstrating that these solutions generalized to new data, an alternative approach to interpreting multi-frequency DPOAE measurements is provided that includes the advantages of using multivariate analyses. This new metric may be useful when DPOAEs are used for screening purposes.

I. Introduction

Distortion-product otoacoustic emissions (DPOAEs) are in widespread clinical use in both the identification and diagnosis of hearing loss. These measures are valuable because they can be performed relatively quickly and can be used when the behavioral audiogram is not easily obtained, such as when testing newborns or other difficult-to-test patients. In fact, one of their primary uses is in universal newborn hearing screening programs, where time efficiency and the need for objective measures are particularly important. DPOAEs are byproducts of normal, nonlinear cochlear-response properties thought to be linked to the status of the outer hair cells (OHC) (Brownell, 1990), one of two families of sensory cells in the cochlea. DPOAE responses occur when the ear is stimulated by two pure tones called primaries. In clinical use, these primary tones (f₁ and f₂, with f₂ being higher in frequency than f₁) are typically spaced such that their frequency ratio (f₂/f₁) approximates 1.22 (e.g., Brown et al., 1994; Harris et al., 1989). The levels of the two primaries may be equal (L₁ = L₂), or L₁ might be presented at a level that exceeds L₂, at least for the case when L₂ < 65 dB SPL (Martin et al., 1990; Stover et al., 1996; Kummer et al., 1998, 2000; Neely et al., 2005). Even though several different approaches have been described for selecting the relative level of the two primaries, it is generally the case that moderate levels are used in the clinic because such levels (e.g., 50-65 dB SPL) represent a good compromise between sensitivity and specificity (Whitehead et al., 1995; Stover et al., 1996).

The presentation of two primaries elicits many distortion products, but the largest emissions in humans occur at a frequency equal to 2f₁-f₂. For this reason, evoked responses are typically measured in the clinic at the frequency corresponding to 2f₁-f₂. As DPOAEs are byproducts of normal, nonlinear behaviors, these measures can be used to describe the pre-neural status of the auditory system (presumably the integrity of the OHC system). OHCs are particularly vulnerable to ototoxic insult; damage to them occurs prior to when damage occurs to the other sensory cells, the inner hair cells (IHC). OHC damage invariably results in hearing loss, with complete OHC damage resulting in about a 50-60 dB loss so long as the IHCs remain normal and intact. This relationship between OHC damage and hearing loss underlies the clinical utility of DPOAE measurements. Consistent with this view is the observation that normally hearing ears typically generate DPOAEs while ears with hearing loss typically do not produce a response or produce responses of reduced level (e.g., Martin et al., 1990; Gorga et al., 1993, 1997, 2000; Stover et al., 1996; Kim et al., 1996).

Making clinical decisions regarding whether ears are normal or impaired has traditionally been based on assigning criterion values to single measurement variables, such as DPOAE level or signal-to-noise ratio (SNR). The specific criterion values presumably are selected so that they result in good test performance. That is, a specific criterion DPOAE level or SNR would be chosen because it correctly identifies the majority of normal ears as normal (high specificity) and impaired ears as impaired (high sensitivity). This is the same as keeping the false-positive and false-negative (miss) rates to a minimum. Previous work with both DPOAEs and transient-evoked otoacoustic emissions (TEOAEs) have shown that, while criteria can be selected that result in good test performance, performance is never perfect (e.g., Gorga et al., 1993, 1997; Prieve et al., 1993; Kim et al., 1996; Harrison and Norton, 1999; Hussain et al., 1998). DPOAE test performance has also been shown to vary with frequency and level, performing better for mid and high frequencies (compared to frequencies below 1.5 kHz) (Gorga et al., 1993, 1997; Kim et al., 1996), and performing better for moderate-level primaries, compared to either low- or high-level primaries (Whitehead et al., 1995; Stover et al., 1996). However, the responses from normal and impaired ears are not completely separated, especially for ears in the borderline normal and mild hearing-loss ranges. No stimulus condition or response criteria can be selected for which all normal ears produce a response and all impaired ears do not (Gorga et al., 1996; 1997). Even under the best of circumstances, diagnostic errors occur, which is true for virtually every clinical audiological test. The source(s) of these diagnostic errors remain undetermined, but might result from the fact that OAEs only reflect OHC function (providing no information about the integrity of the IHCs or auditory nervous system), OAEs may be affected by reverse energy transmission through the middle ear (something not likely to influence behavioral thresholds in the same way), the possibility that DPOAE generation occurs over broad basal cochlear regions (making the DPOAE a less place-specific response), and damaged cochlear regions, basal to the characteristic place for the primaries might influence reverse energy transmission within the cochlea.

In an effort to improve DPOAE test performance, Kimberley et al. (1994) described multivariate analyses that used a combination of DPOAE and other variables (such as age or gender) to predict hearing loss at specific frequencies. They reported no improvement in test performance using multivariate techniques, compared to the more traditional univariate approaches in which SNR was used. In contrast, Dorn et al. (1999) described improvements in test performance when multivariate analysis techniques were applied to DPOAE data that included measurements at several frequencies, but did not include other variables (such as age or gender) in the multivariate solutions. In that study, data were available on over 1200 normal-hearing and hearing-impaired ears. The data from these ears were divided into two groups, a training group on which the multivariate solutions optimizing test performance were developed and a validation group on which the generalizability of the training-group solutions were evaluated. The multivariate analyses based on training-group data resulted in improvements in test performance over what was achieved with either DPOAE level or SNR. The improvements in test performance were greatest for frequencies at which the univariate techniques performed the poorest (0.75, 1, 1.5, and 8 kHz). The multivariate solutions derived from the training-group data were then applied to the data from the validation group. These multivariate solutions resulted in nearly the same improvement in test performance when they were applied to the validation-group data, suggesting that the original multivariate analyses that were developed from the training data set generalized to other data.

Dorn et al. (1999) focused on determining the improvements in test performance when predicting audiometric status at a single frequency. In other applications, however, diagnosis of auditory status might be made on the basis of results for a group of frequencies and not on the basis of results for each frequency individually. In an effort to further evaluate test-performance improvements with multivariate analyses, Gorga et al. (1999) evaluated DPOAE test performance when the “gold standards” were based on audiometric thresholds for groups of frequencies. Each group of frequencies was selected because of their potential to be included in screening applications of the DPOAE test. Several multi-frequency gold-standard definitions were used, including pure-tone averages (simple arithmetic averages across several frequencies) and extrema thresholds (in which extra weight was given to frequencies for which thresholds were particularly elevated). Just as in the single-frequency case, test performance improved when multivariate analyses of DPOAE data were used to predict audiometric status based on multi-frequency gold standards. Interestingly, there was little difference in performance among pure-tone average, extrema, or a combination of average and extrema gold standards.

In the context of DPOAE test performance, the goal of multivariate analyses is to achieve the greatest separation between distributions of responses from normal and impaired ears. Several input variables (e.g., DPOAE signal and noise levels at several frequencies) are provided to the multivariate analyses (for example, a logistic regression), which uses these inputs to create a new dimensionless variable (the LF score). This dimensionless variable represents a linear combination of the input variables, each of which is multiplied by a coefficient. The specific variables and their associated coefficients are chosen so that, along the new dimensionless variable (LF score), maximum separation is achieved between the distributions of responses from normal and impaired ears. There is no constraint on multivariate procedures in terms of either the selection of input variables or the value of their associated coefficients. Thus, multivariate solutions potentially could be idiosynchratic to the data set from which they were derived. While Dorn et al. (1999) partially addressed this issue by dividing a large set of data into training and validation sets, these data were not entirely independent in that they included both ears of many subjects and were all collected at the same time. Training and validation data sets were not constructed in the study in which multi-frequency gold standards were used (Gorga et al., 1999).

The purpose of the present study was to further evaluate the multivariate solutions described in Dorn et al. (1999) and Gorga et al. (1999), using DPOAE data collected on an entirely new set of ears using equipment that differed in several ways from the equipment used by Dorn et al. and Gorga et al. If the previously described multivariate solutions did not improve test performance relative to the univariate case, then this would indicate that the previous multivariate solutions were idiosynchratic to the set of data from which they were derived, and there would be little value in using them to improve test performance clinically. On the other hand, if test performance did improve when the new DPOAE data are analyzed using the multivariate solutions described in Dorn et al. and Gorga et al., then this would suggest that the original solutions were robust, generalizing to an entirely new set of data, and, thus, would support their use clinically.

A further aim of this study was to develop guidelines for interpreting clinical DPOAE data that recognize the overlap in response distributions (which exists for both univariate and multivariate variables), that also takes into account any advantages that might occur as a result of using multivariate analytical techniques. Criteria that are in typical clinical use for interpreting DPOAEs are not without their problems. SNR is in widespread clinical use as a criterion for evaluating DPOAEs. While SNR has value when determining if a DPOAE was reliably measured, using SNR alone to determine auditory status is not always the optimal choice. For example, if the SNR criterion for normal is set to 6 dB, an ear producing a DPOAE level of 5 dB SPL and a noise level of -1 dB SPL would have an SNR of 6 dB, and, therefore, would meet the criterion for normal hearing. However, an ear producing a DPOAE level of -12 dB SPL with noise equal to -18 dB SPL would also meet the same SNR criterion for normal hearing. In the second instance, it is less likely that the ear is normal, given the low DPOAE level, and making a clinical decision that this ear is normal based on SNR alone may miss hearing loss. Furthermore, it is likely that DPOAE level is closely related to the integrity of the OHC system, and thus, might provide a better metric for predicting auditory status. In the above example, a DPOAE of -12 dB SPL would be viewed as consistent with hearing loss, even if the SNR was 10 dB. On the other hand, DPOAE level may lead to erroneous conclusions if both the measured DPOAE level and noise are high. In the above example, a DPOAE level of 5 dB SPL might be consistent with normal hearing. However, if the noise level also was 5 dB SPL, then the measured DPOAE “response” may be nothing more than noise. These considerations suggest that both DPOAE and noise level (or DPOAE level and SNR) should be considered when making diagnostic decisions.

In a previous effort to address these issues, we have described an interpretive approach in which SNR is used to determine the reliability of the DPOAE measurement, and then DPOAE level is used to determine auditory status (Gorga et al., 1997, 2002). In this approach, a diagnosis based on DPOAE level is made only when DPOAE level is reliably measured, meaning that some consideration is given to the SNR. This approach was developed using univariate analyses of DPOAE data. Interpretative guidelines for DPOAE data also have been developed based on new variables produced by multivariate analyses (Dorn et al. 1999), but the new dimensionless variables produced by these analyses may not have received widespread clinical acceptance because they do not directly relate to the measured response. If it can be shown that previously described multivariate solutions improve test performance for an entirely new set of data, then it follows that there would be a clinical advantage in using the new variable for diagnostic decisions. An alternative set of interpretative guidelines that take advantage of the multivariate improvements in test performance that also may be more easily interpreted will be described as well.

II. Methods

A. Subjects

Data were collected in 345 ears of 187 subjects, who were paid for their participation in the project. The data from two ears of the same subject are not independent; however, we view this as less of an issue for the purposes of this study because no claims are made regarding the statistical significance of the present results. Furthermore, the present data were only used to test the generalizability of multivariate solutions developed on an entirely different set of data. They were not used to develop these solutions. Subjects ranged in age from 2 to 86 years, with a median of 29.7 years and an interquartile range of 11.8 to 48.8 years. This study was conducted in accordance with an approved IRB protocol and HIPPA regulations related to human research subjects. Prior to participation, informed consent (and assent, depending on age) was obtained from the subject and/or a legal guardian. Audiometric thresholds were measured with 5-dB precision for octave frequencies ranging from 0.25 to 8 kHz using standard clinical procedures that were appropriate to the age and developmental level of the subject. If thresholds between octave frequencies differed by 15 dB or less, threshold at the inter-octave frequency was interpolated from the thresholds at the two adjacent octave frequencies. If thresholds differed by 20 dB or more between octave frequencies, thresholds were measured for inter-octave frequencies as well. Any ear showing evidence of a conductive hearing loss (based on an air-bone gap ≥ 10 dB at 0.5 kHz or higher) was not included in the study. Audiometric thresholds ranged from -10 dB HL to in excess of 120 dB HL (re: ANSI, S3.2, 1996). Prior to DPOAE measurements, tympanometry was performed (probe tone = 226 Hz). DPOAE tests were performed only if the tympanograms were judged to be normal (tympanometric peak pressure > -100 daPa, compliance ≥ 0.2 mmho).

B. Stimulus Paradigm and Instrumentation

DPOAE data were collected using the same paradigm as was used in our previous studies of test performance (Gorga et al., 1997, 1999; Dorn et al., 1999). Specifically, DPOAEs were measured for f₂ frequencies ranging from 0.75 to 8 kHz, in half-octave steps. Additionally, DPOAE data were collected with f₂ set to 5 kHz. In the analyses to follow, predictions of auditory status were made using DPOAE data at both 5 and 6 kHz. The primary-frequency ratio (f₂/f₁) was kept constant at 1.22. Primary levels were fixed, with L₁ set to 65 dB SPL, and L₂ set to 55 dB SPL, as measured at the plane of the probe. Measurement-based stopping rules were used, such that, at any frequency, averaging stopped when the noise floor was ≤ -30 dB SPL or after 32 seconds of artifact-free averaging, whichever occurred first. DPOAE level was estimated as the level in the 2f₁ - f₂ frequency bin. Noise level was estimated as the average level in the 2 closest frequency bins above and below 2f₁ - f₂. The present DPOAE data were collected with a commercially available system (Bio-logic, Scout 3.45) which uses a custom-designed acoustic probe. This system differs from the one used during data collection in the studies in which the multivariate solutions were developed. In those studies, a custom-designed system and an ER-10C probe microphone system were used. All measurements were made in a quiet room adjacent to the audiology clinic. The test room, however, was not a sound-treated booth.

System distortion was estimated in a standard cavity (B&K, 711 Coupler), as well as in several non-standard cavities (reduced volume, 10m tube, MZ1). Estimates of system distortion were -25 dB SPL or less for the frequencies used in the present study. System distortion also was measured in a subject with a cochlear implant (implant turned off) in order to provide an estimate from a biological system that would not be expected to produce distortion. To the extent that the noise levels could be reduced in this subject, estimates of system distortion were similar to those observed in hard-walled cavities.

III. Results

A. Test Performance for Single-Frequency Conditions

In the first series of analyses, DPOAE data were compared to single-frequency audiometric gold standards. That is, DPOAE data were used to dichotomously determine if hearing was normal or impaired at individual audiometric frequencies. This effort parallels the work described in Dorn et al. (1999). Clinical decision theory was used to construct relative operating characteristic (ROC) curves at each of eight octave and inter-octave frequencies from 0.75 to 8 kHz. In the univariate case, DPOAE data at one f₂ was used to predict audiometric status at the same frequency. In the multivariate case, the solutions described in Dorn et al. (1999) and reproduced here in Table A1 were used to make predictions at each frequency. For these analyses, pure-tone thresholds of 20 dB HL or better were defined as normal at each frequency. Thresholds exceeding this level were defined as hearing impaired.

Figure 1 summarizes the results from this analysis by plotting area under the ROC curve as a function of frequency for four univariate and two multivariate analyses. Two univariate measurements (DPOAE level and SNR) were used to predict audiometric status. However, we used DPOAE data at either 5 or 6 kHz to make these predictions when the audiometric frequency was 6 kHz, thus resulting in a total of four univariate measurements. Similarly, the logistic regressions were performed in two ways, one in which DPOAE data at 6 kHz were used and one in which DPOAE data at 5 kHz were substituted for the DPOAE data at 6 kHz. When using DPOAE data at 5 kHz, the previously described multivariate solutions for 6 kHz were used (Dorn et al., 1999).

Figure 1.

Area under the ROC curve as a function of frequency for single-frequency conditions. The parameter is criterion measurement. Open symbols - DPOAE level, shaded symbols - SNR, filled symbols - LF score. Squares represent results when DPOAE data for f₂ = 5 kHz was used to predict auditory status. Circles represent results when DPOAE data for f₂ = 6 kHz was used.

Considering only the univariate results (open and shaded symbols in Fig. 1), DPOAE level resulted in similar or larger ROC curve areas (compared to SNR) at all frequencies except for 0.75 kHz. Still, the differences in ROC curve areas between DPOAE level and SNR were not large, on the order of 3% or less. For both DPOAE level and SNR, ROC curve areas were less at 0.75, 1 and 8 kHz, compared to other frequencies. These results are consistent with previous observations regarding test performance as a function of frequency and as a function of the univariate criterion (Gorga et al., 1993, 1997; Kim et al., 1996). Interestingly, better test performance occurred when DPOAE data with f₂ = 5 kHz were used to predict auditory status at 6 kHz than when DPOAE data with f₂ = 6 kHz were used to make the same prediction (compare squares to circles at 6 kHz in Fig. 1).

The results obtained with the multivariate analyses are shown as completely filled symbols in Fig. 1. Table A1 provides the previously derived coefficients and constants of the logit functions (Dorn et al., 1999) that were used in the present study for each audiometric frequency. Multiplying the DPOAE and noise levels by these coefficients, and summing them with the constant provides the LF score. For example, the LF score at 6 kHz is derived from the DPOAE levels at 4, 6, and 8 kHz and the noise at 1.5, 2, 4 and 6 kHz (each multiplied by its coefficient) plus a constant. The resulting dimensionless LF score was used in exactly the same way as DPOAE level or SNR to generate ROC curves.

ROC curve areas were larger for the two multivariate solutions, compared to the univariate approaches. The only exceptions to this observation occurred at 4 and 6 kHz, frequencies for which the univariate ROC curve areas were 0.96 or greater. Thus, there was little room for improvement with the multivariate analyses. Furthermore, the differences in ROC curve areas between univariate and multivariate analyses at 4 and 6 kHz were small, on the order of 0.004 or 0.005. At all other frequencies, the multivariate analyses achieved better test performance, as measured by the area under the ROC curve. Although some of the improvements were small, the use of multivariate analyses resulted in areas under the ROC curve that were greater than 0.95 at all test frequencies, including the ones for which DPOAE level and SNR performed poorest (0.75, 1, and 8 kHz),. These results suggest that the previously described multivariate solutions were robust and generalized to a new set of data. Although less obvious than in the case of univariate analyses, inclusion of data at 5 kHz also resulted in slightly better performance in the multivariate cases, compared to multivariate analyses when DPOAE data at 6 kHz were used (compare filled squares to filled circles).

An alternate, but related, approach to evaluating test performance is to fix the specificity (or its complement, the false-alarm rate) and evaluate sensitivity. The false-alarm rate is the failure rate among ears with normal hearing, while sensitivity is the failure rate among ears with hearing loss. To further evaluate test performance in relation to audiometric threshold, five categories of hearing were formed: normal hearing defined as thresholds ≤ 20 dB HL; mild hearing loss defined as thresholds between 25 and 40 dB HL; moderate hearing loss defined as thresholds of 45 to 60 dB HL; severe hearing loss defined as thresholds of 65 to 80 dB HL; and profound hearing loss defined as thresholds ≥ 85 dB HL. In the univariate case, analyses were performed for both DPOAE level and SNR, even though they resulted in similar test performance. DPOAE data from both 5 and 6 kHz were used.

In contrast to clinical decision theory analyses, in which all criterion values are assessed but ears are classified dichotomously, it is necessary to select a single criterion value when sensitivity is evaluated as a function of the magnitude of hearing loss. On the other hand, hearing loss can be parceled into sub-categories, which is not typically the case when clinical decision theory is used to evaluate test performance. To select a single criterion value for univariate and multivariate approaches, we selected, as a starting point, an SNR of 6 dB because this value is frequently used in clinical applications of DPOAE measurements. This criterion was used for the normal and hearing-impaired categories. Next, the DPOAE level resulting in the same false-alarm rate (failure rate in ears with normal hearing) as that achieved with a 6-dB SNR was determined. This DPOAE level was then used as the criterion for the four hearing-impaired categories. Finally, in efforts to facilitate comparisons between univariate and multivariate analyses, the multivariate criterion (LF score from the logistic regression) was determined that produced the same failure rate in ears with normal hearing as the 6dB-SNR and DPOAE level criteria. That LF score was also used as the criterion for the four hearing-impaired categories, just as a 6-dB SNR was used for all categories with hearing loss. With the same failure rate for DPOAE level, SNR, and LF scores among ears with normal hearing, the univariate and multivariate approaches were compared in terms of their failure rates among ears with hearing loss. Stated another way, we used a fixed criterion SNR to hold the false-alarm rate constant for univariate and multivariate conditions, and evaluated test performance in terms of sensitivity. It may be important to note that a 6-dB SNR does results in different false-alarm rates among ears with normal hearing, depending on frequency. This can be seen in Fig. 2, where the failure rate among normal ears ranged from about 1% at 4 and 6 kHz up to about 20% at 0.75 kHz.

Figure 2.

Failure rate (%) as a function of hearing-loss category for single-frequency conditions. Data for a different frequency are shown in each panel. Symbol convention is the same as that used in Fig. 1. Hearing-loss categories were formed as described in the text.

The results of this analysis are shown in Fig. 2, which plots failure rate as a function of hearing-loss category for two DPOAE level, two SNR, and two LF conditions. There are two conditions in each case because DPOAE data were collected at 5 and 6 kHz, and both of these frequencies were used to predict hearing status at 6 kHz in the univariate analyses and were available for use in the multivariate analyses for any frequency. Each panel provides results for a different frequency. The symbol convention used in Fig. 1 is also used here. Within each panel, note that (by definition) the results for all six criteria overlap in the normal-hearing group, for the reasons stated above. Any differences in test performance, therefore, will be reflected in differences in failure rates among ears with hearing loss. Note that, with few exceptions (e.g., the mild hearing-loss group at 1.5 kHz, mild and moderate hearing loss group at 4 kHz), the failure rates were equal to or higher for the multivariate analyses in each of the four hearing-loss categories and at all 8 frequencies, compared to the univariate approaches. That is, the multivariate approaches achieved higher hit rates for equivalent false-alarm rates, compared to the case when either DPOAE level or SNR were used. In general, these results are expected, given the ROC curve areas that were described in Fig. 1.

B. Test Performance for Multi-frequency Gold Standards

In the next series of analyses, DPOAE data were compared to multi-frequency gold standards, in which pure-tone averages (PTA) were calculated for each of three different frequency combinations: 2, 3, and 4 kHz (PTA₃); 2, 3, 4, and 6 kHz (PTA₄); and 1.5, 2, 3, 4, and 6 kHz (PTA₅). These frequencies were chosen because they represent combinations that might be used during screening applications of DPOAE measurements. In addition, these are frequencies for which test performance was good using traditional criterion measures, such as DPOAE level or SNR (see Figs. 1 and 2). While the multivariate approach resulted in better test performance at these frequencies when single-frequency gold standards were used, the differences were not as great as they were for other frequencies for which univariate performance was poor (such as 0.75, 1, and 8 kHz). Thus, we would not expect large differences between univariate and multivariate test performance for the case of multi-frequency gold standards. For the purposes of these analyses, normal hearing was defined as a PTA of 20 dB HL or less; average thresholds exceeding 20 dB HL were defined as hearing impaired.

For the univariate conditions, both DPOAE level and SNR were used as criterion measures. In both cases, criterion values were incremented over the entire range of observed values to produce ROC curves. When using these univariate criteria, a result was considered consistent with normal hearing (i.e., a pass occurred) only if the DPOAE level or the SNR exceeded the criterion value at each of the frequencies included in the multi-frequency case. For example, if the SNR criterion equaled 6 dB, then the SNR needed to be ≥ 6 dB at each of the three frequencies of 2, 3, and 4 kHz in order for a DPOAE pass to occur when PTA₃ was used as the gold standard. If the SNR was < 6 dB at any frequency, the results were categorized as consistent with hearing loss (i.e., a “failure” occurred), even if the SNR was above 6 dB at the other two frequencies. For the multivariate analyses, the coefficients and variables previously described in Gorga et al. (1999) and reproduced here in Table A2 were applied to the data. The LF score was derived from these coefficients and constants in exactly the same way as they were derived in the single-frequency case.

Figure 3 plots ROC curve areas for DPOAE level, SNR and LF score as a function of the three multi-frequency gold standards (PTA₃, PTA₄, PTA₅). As in the single-frequency case, analyses were performed using DPOAE data at both 5 and 6 kHz for both univariate and multivariate analyses. DPOAE level and SNR resulted in similar ROC curve areas, although there was a slight tendency for DPOAE level to achieve higher areas. For both univariate and multivariate conditions, P(A) was slightly larger when DPOAE data at 5 kHz were used to make predictions, compared to the case when data at 6 kHz were used. Larger ROC curve areas were observed for multivariate analyses, although the differences between univariate and multivariate ROC curves are less than the differences observed for the single-frequency cases described previously. This is likely due to the fact that these analyses include frequencies for which the univariate analyses performed well. While the differences between these areas and those achieved with univariate analyses were small (on the order of 2-3%), they always favored the multivariate approach.

Figure 3.

Area under the ROC curve as a function of the multi-frequency gold standard. Symbol convention is the same as that used in Fig. 1.

Following the convention used in Fig. 2, failure rate as a function of hearing-loss category is plotted in Fig. 4 for multi-frequency gold standards. As before, a SNR of 6 dB was used for the normal-hearing and four hearing-impaired categories. The DPOAE level was selected that achieved the same failure rate in normal ears as was achieved with an SNR of 6 dB. This DPOAE level was then used as the criterion for the four hearing-impaired categories. In similar fashion, the LF score resulting in the same failure rate in normal ears as was observed for SNR and DPOAE level was determined. This score was used as the criterion value for the four hearing-impaired categories, thus facilitating comparisons between multivariate and univariate results in ears with hearing loss. Smaller differences were observed between univariate and multivariate failure rates among ears with hearing loss, compared to the single-frequency condition (Fig. 2), which, in part may be due to the fact that the frequencies included in these conditions are ones for which the univariate analyses performed well. This result also may be due to the possibility that the univariate criteria are effectively more stringent in the multi-frequency case. That is, an ear would fail PTA₄ if the SNR was < 6 dB at one frequency, even if large SNRs were observed at the other 3 frequencies. This would have the effect of driving the failure rate up among ears with hearing loss, potentially making the outcomes from univariate and multivariate analyses more similar.

Figure 4.

Failure rate (%) as a function of hearing-loss category for multi-frequency gold standards. Results for a different PTA are shown in each panel. Symbol convention is the same as that used in Fig. 3. Hearing-loss categories were formed as described in the text.

C. Issues Associated with Interpreting DPOAE Data in the Clinic

To this point, we have demonstrated that multivariate analyses provide an improvement in test performance, compared to the performance achieved when either DPOAE level or SNR are used. However, this does not provide information that could be used in the clinic when interpreting DPOAE measurements on individual patients. In order to use DPOAE measurements (or any measurement, for that matter) in the clinic, one must select criteria that will serve as guidelines for interpretation. Previously, we have described an approach in which cumulative distributions of response properties were constructed separately for normal-hearing and hearing-impaired ears (Gorga et al., 1996, 1997, 2002). Specific performance levels were chosen, and the criterion values (such as DPOAE level) associated with these percentiles were then determined. While this approach allows one to select the criterion value associated with any performance level (i.e., percentile), a more limited range of performance levels probably would be of interest clinically. Using the cumulative distributions from impaired ears, the criterion values associated with the 90^th and 95^th percentiles were determined. One would expect hit rates (sensitivities) of 90% or 95% when using these criteria in the clinic, because those percentages of ears with hearing loss produced responses of this size or smaller. In a similar manner, we used the cumulative distributions of responses from normal ears to determine the criterion values associated with the 5^th and 10^th percentiles. Using these values in the clinic, we would expect false-alarm rates (1 - specificity) of 5% and 10% because these percentages of normal ears produced such small responses. A form was developed in which all four percentiles were plotted as a function of frequency and currently is in use in our clinic (Gorga et al., 2002). These coordinates have been used to form a template which is also available in several commercially available systems.

A similar form was produced based on multivariate analyses, using cumulative distributions of LF scores from normal and impaired (Dorn et al., 1999). The interpretative approach based on multivariate analyses has not been embraced clinically. This is understandable because the LF score lacks the intuitive clinical value of DPOAE level or SNR. However, it is also unfortunate because we demonstrated previously that the LF scores derived from multivariate analyses provided better test performance than either DPOAE level or SNR (Dorn et al., 1999; Gorga et al.,1999). The present results confirm the earlier findings related to improvements in test performance and suggest that the previous multivariate solutions are robust, thus, supporting the view that interpretative guidelines, based on the new dimensionless variable, would have clinical value.

As an alternative, the LF scores (which is the log of a likelihood ratio) can be converted into a probability of normal (P(N)). Although P(N) may seem more satisfying clinically, assigning meaning to a specific P(N) has not been described. On the other hand, it has the same test-performance advantage that was achieved by the LF scores from which it is derived. The LF scores can be converted into a probability of normal (P(N)), using the following equation:

$$\mathrm{P}\left(\mathrm{N}\right)=1∕(1+{\mathrm{e}}^{-\mathrm{LF}}).$$ (1)

A P(N) of 50% represents the point for which it is equally likely that a response is coming from either normal or impaired distributions. In this sense, it would represent the point of maximum uncertainty. The midline (P(N) = 50%) also can be used to make a maximum-likelihood decision, independent of specificity or sensitivity. This capability does not exist for the representations of either DPOAE level or SNR, although it would apply for the LF score, in which case, an LF score = 0 would be used to make the maximum likelihood decision.

D. Case Studies

Figure 5 provides three cases, in which pure-tone audiograms are directly compared to DPOAE data, and then to data analyzed with multivariate techniques and recast as P(N). Data from a different subject are shown in each row, and, at the right of each row, P(N) is provided for each of the three PTA gold standards. The 95^th and 90^th percentiles from impaired distributions and the 10^th and 5^th percentiles from normal distributions are shown as lines (going from top to bottom) limiting areas on the forms shown in the middle and right column of panels. Table A3 provides the DPOAE levels and P(N) associated with the limiting lines in this figure.

Figure 5.

Case studies: Left column - pure-tone audiograms; middle column - DPOAE level as a function of frequency (see DPOAE levels in Appendix 3). Right column of panels - P(N) as a function of frequency (see P(N) values in Appendix 3). P(N) for each of three multi-frequency gold standards are given at the far right.

In the top row of Fig. 5, results are shown for an ear with normal hearing. While the DPOAE levels produced by this ear would be considered normal, the response at 0.75 kHz fell within the top hatched area, and the response at 1 kHz was in the region of uncertainty. However, when converted to P(N), the results for this subject were in the normal range at all frequencies. Thus, in this case, the application of multivariate analyses (a byproduct of which is P(N)) reduced diagnostic uncertainty. P(N) relative to PTA₃, PTA₄, and PTA₅ also indicated that hearing was more likely to be normal in this case. The middle row provides data for a subject with normal hearing, but with one response (0.75 kHz) that fell into the region of uncertainty and several responses that were not entirely above the 95^th percentile (1 and 3 kHz) when the absolute DPOAE level was plotted. When these data were re-analyzed with multivariate analyses, converted into P(N), and plotted on the new form, diagnostic uncertainty was reduced at these three frequencies. The comparisons of data to multi-frequency gold standards also supported the conclusion that hearing was normal. Finally, the bottom row shows data from a case with hearing loss at several frequencies, but, at some of those frequencies (0.75, 2, 3 kHz), the DPOAE levels were in the region of uncertainty. The multivariate analyses classified these responses as being consistent with hearing loss, which was, in fact, the case. On the other hand, at one frequency (4 kHz), the DPOAE level might have led to a diagnosis of normal hearing, which would have been the correct diagnosis, whereas the multivariate analysis was more consistent with mild hearing loss. This demonstrates that there are cases in which DPOAE level led to correct decisions when multivariate analyses did not, although overall, it was more common for the multivariate analyses to result in the correct diagnosis.

The shaded area in right column (plots of (P(N)) is not centered over 50%, as one might expect for a criterion point at which it is equally likely to be normal or impaired. There are two possible reasons why this occurred. First, the empirical distributions from which P(N) was calculated may not have been Gaussian. However, they are based on a large-sample study (approximately 400 ears with normal hearing and 800 ears with hearing loss). While the shape of the empirically measured distributions may have been skewed, they probably are good representations of the population distributions in normal and impaired ears. It is unlikely that the shapes would change if data were collected on additional ears. Second, the unequal N’s for normal and impaired distributions, which depended on frequency, would be expected to shift overlap regions upward at lower frequencies (where normal hearing was more common) and downward at higher frequencies (where normal hearing was less common).

IV. Discussion

The primary goal of the present study was to provide an independent test of the generalizability of previously described multivariate analyses of DPOAE data. A secondary goal was to develop interpretative guidelines based on the multivariate analyses. The major observations from this study are listed below:

1. Univariate analyses using either DPOAE level or SNR result in roughly equivalent test performance, based on estimates of area under the ROC curve and estimates of failure rates among ears with hearing loss when false-alarm rates were fixed. DPOAE level performed slightly better than SNR at mid and high frequencies, whereas SNR performed slightly better at 0.75 and 1 kHz.

2. Multivariate analyses resulted in better test performance than DPOAE level and SNR at essentially every test frequency. This advantage was especially evident at those frequencies for which DPOAE level and SNR performed poorly (0.75, 1, and 8 kHz).

3. Although the advantages for the multivariate analyses were smaller when multi-frequency gold standards were used, for the most part, they still outperformed either DPOAE level or SNR.

4. An alternative response measure, P(N), was derived from the LF scores. This measure has the same test-performance advantage as the LF score, it can be used in the single-frequency case, and it may be useful when multi-frequency gold standards are used.

These observations may have implications for the use of DPOAEs in the clinic in terms of both improving test performance and providing a convenient means for interpreting DPOAE test results.

The observation that test performance for DPOAE level and SNR are nearly equivalent is not new (Gorga et al., 1993, 1997; Stover et al., 1996). These results are not surprising, given the recording paradigm we typically use, which incorporates measurement-based stopping rules. In the present work and most of our previous work, the primary stopping rule has been noise floor, in combination with a test-time stopping rule; measurements at any frequency stop when the noise floor is below either -25 dB SPL or -30 dB SPL or after 32 seconds of artifact-free averaging, whichever occurred first. We have used the noise-floor value because it approximates the level at which system distortion becomes evident and, thus, marks the lower limit of reliable measurements. Averaging until this minimum noise level is achieved also provides measurements over the widest possible dynamic range. In the present context, reducing the noise to such a low level both minimizes the noise floor and roughly equates its influence on test performance across frequency and subject. Efforts to equate noise floor for all test conditions, however, were not always successful. Despite using measurement-based stopping rules with a noise-floor criterion, there were cases in which the noise-floor criterion was not met. To avoid potentially long averaging times for individual conditions, the test-time stopping rule was included. In most cases, the test stopped on the noise-floor criterion, not the test-time limit. This was true in the majority of high-frequency test conditions; however, there were cases, especially for lower frequencies, in which the noise-floor criterion was not met. Under these circumstances, the DPOAE level criteria used during assessments of test performance (ROC curve analyses) might have been met even though the levels were close to the noise floor (increasing the possibility of a false-negative error). In these circumstances, however, test-performance criteria based on SNR would be expected to make a false-negative error less frequently because any case in which the DPOAE level and noise were close would fail to meet passing criteria. The fact that DPOAE level does slightly better at mid and high frequencies and SNR performs slightly better at lower frequencies is consistent with this hypothesis.

While the poorer performance at 0.75 and 1 kHz might be attributed to the higher noise levels (and therefore, less reliable measurements) at these frequencies, this explanation cannot account for the poorer performance at 8 kHz, where noise levels are typically low. In the past, we have attributed the poorer performance at 8 kHz to higher levels of system distortion at this frequency (Gorga et al., 2002). That is, the hardware associated with these measurements tends to produce higher levels of system distortion at 8 kHz, compared to other frequencies. This would presumably result in an increase in the false-negative rate, thus driving test performance down. More recent data suggest that primary level also may have influenced the present results at this frequency (Neely et al., 2005). Primary-level differences may need to be adjusted in order to maximize response level, at least for subjects with normal hearing. However, it was deemed important to collect data in the present study under the same conditions as those used previously (Dorn et al., 1999; Gorga et al., 1999) in order to provide an appropriate test of the robustness of the multivariate solutions. In any case, both increased system distortion and non-optimal primary levels may have contributed to the poorer test performance at 8 kHz.

The reasons for better performance when using DPOAE data for an f₂ = 5 kHz are not obvious. One possible explanation relates to the same level issues that were discussed in relation to test performance at 8 kHz. The extent to which previously used primary-level differences negatively impact DPOAE level decreases as frequency decreases (Neely et al., 2005). Perhaps the primary-level differences used in the present study (which were non-optimal in that they were constant as a function of frequency) were more optimal when f₂ = 5 kHz, compared to when f₂ = 6 kHz. We also observed slightly higher SNRs in the DPOAE data at 5 kHz, which might have resulted in more reliable measurements at this frequency. Both effects would be expected to contribute to better performance for univariate and multivariate analyses.

Multivariate analyses are performed by providing several inputs to the analysis, which then combines these variables and weighting coefficients in such a way as to maximize the separation between two distributions (in the present case, normal and impaired DPOAE responses). There are no constraints on these analyses other than that they must select from the set of variables that are provided as inputs when calculating the LF score. We have previously demonstrated that the use of multivariate analyses improved DPOAE test performance (Dorn et al., 1999; Gorga et al., 1999), and, in one case, we validated the results by dividing a large sample of data into training and validation data sets (Dorn et al., 1999). However, both the training and validation data were collected at the same time. In addition, data were collected on both ears of many (but not all) of the subjects whose data went into the analyses, although the assignment to either training or validation set was random. The two ears of a subject, however, are not independent on any measurement, including DPOAE measurements. Thus, the approach we took previously was not an ideal test of the robustness of the multivariate solutions. The primary observation from the present study (that previously described multivariate solutions are robust) should not be affected by these concerns. The present data were collected five years after the original multivariate solutions were generated on an entirely different group of subjects using different hardware. While it is the case that both ears of several subjects were tested, none of the present data were used during the development of the multivariate solutions and no claims are made regarding the statistical significance of the results.

As a result, the observation from the present study that test performance improved when multivariate analyses were used supports the view that the multivariate solutions are robust and may be applied with other groups of patients. The dimensionless variable (LF score) that was derived previously resulted in greater separation of the response distributions from normal and impaired ears when applied to a new set of data. Thus, fewer diagnostic errors would be made when the multivariate analyses were used, compared to more traditional approaches in which DPOAE level or SNR was used. In essentially every case, the use of LF scores based on the multivariate analyses resulted in improvements in test performance. The improvements were greatest for those conditions in which univariate analyses, such as DPOAE level and SNR, performed poorest. For example, at 0.75 kHz, ROC curve area improved from 0.86 or 0.88 to 0.96. Smaller differences were observed at other frequencies, but they nearly always favored the multivariate approach. Because the multivariate analyses were performed post hoc, they required no additional data-collection time. The calculations needed to apply the multivariate solutions are simple and essentially could be performed contemporaneously. Thus, no additional data-collection or data-analysis time would be needed in order to take advantage of the multivariate solutions. Given these facts, they may have clinical advantages over current univariate applications. It is important to recognize, however, that the multivariate solutions used in the present study can only be applied to data collected with the same stimulus and recording conditions used in the studies in which these solutions were developed (Dorn et al., 1999; Gorga et al., 1999).

Although the multivariate analyses outperformed univariate approaches when multi-frequency gold standards were used, the advantages were much smaller. This observation may not be surprising for several reasons. In the multi-frequency case, we selected frequencies that might be included as part of a screening protocol. As a result, we eliminated from inclusion the three frequencies (0.75, 1, and 8 kHz) for which (in the single-frequency case) univariate test performance was the poorest. It is not surprising that smaller differences were observed between univariate and multivariate analyses under these more restricted frequency conditions. In addition, the criteria for the univariate case were implemented in such a way that a “pass” occurred only if the criterion value was met at all frequencies included in the grouping. A failure would occur when the DPOAE level or SNR criterion was not met at just one frequency, even if it was met at all other frequencies included in the group. Thus, it would be expected to be harder to achieve a “passing” diagnosis for multi-frequency gold standards under univariate analysis conditions.

It may seem more applicable to consider DPOAE level or SNR when reaching clinical decisions, compared to the output from the multivariate analyses. The LF score, although, interpretable, does not carry the same intrinsic meaning as more direct measurements, such as level or SNR. For this reason, another metric was described, P(N). The use of this variable allows one to take advantage of the better test performance achieved by multivariate analyses because it is calculated from the LF score, yet may be more acceptable clinically. It should be noted, however, that any decision variable (whether it is DPOAE level, SNR, LF score, or P(N)) can be used to generate interpretative guidelines that might be helpful in the clinic. On the other hand, P(N) may have additional advantage when screening paradigms are used and one wishes to classify an ear as either normal or impaired based on the compilation of data from several frequencies. In this case, P(N) provides a single number on which the decision can be made. Finally, both the LF score and P(N) can be used to make a maximum-likelihood decision, an option that does not exist for either DPOAE level or SNR.

It is important to note that, even with multivariate analyses, perfect DPOAE test performance is never achieved. Although ROC curve areas always exceeded 0.95 when the LF score was used, it never reached 1.0. This means that diagnostic errors are unavoidable. It is important to recognize this fact when applying DPOAE measurements in the clinic. On the other hand, a great deal is now known about DPOAE test performance, enabling the clinician to make informed decisions under most circumstances.

Appendix

Table A1.

Coefficients of the logit function (LF) for each audiometric frequency. This table is reproduced with permission from Dorn et al. (1999)

	LF750	LF1000	LF1500	LF2000	LF3000	LF4000	LF6000	LF8000
DPOAE f2 frequency
DP750	0.087	0.048
DP1000	0.063	0.088	0.025	0.024
DP1500	0.085	0.122	0.136	0.055		-0.050		0.051
DP2000	0.068	0.069	0.078	0.146	0.079	0.068
DP3000			0.030	0.058	0.103	0.038
DP4000	0.028			0.025	0.085	0.172	0.060	0.049
DP6000			0.038		0.108	0.100	0.224	0.121
DP8000		0.024		0.023			0.023	0.122
Noise f2 frequency
N750	0.029				0.046
N1000	-0.044	-0.035	0.034
N1500	-0.073	-0.068	-0.036			0.034	0.070	0.034
N2000	-0.071	-0.087	-0.076	-0.086	-0.049		-0.062	-0.064
N3000		-0.078		-0.019	-0.106	-0.087		-0.049
N4000					-0.106	-0.082	-0.063
N6000						-0.051	-0.018	-0.062
N8000								-0.058
Constant	-0.190	-1.096	-1.057	-1.958	-2.664	-2.545	-0.607	-1.505

Table A2.

Coefficients of the logit function (LF) for three different pure-tone average (PTA) gold standards. See text for more details. Reproduced with permission from Gorga et al. (1999)

f2 frequency	LF 3 freq. PTA	LF 4 freq. PTA	LF 5 freq. PTA
DPOAE level
DP1500	----	----
DP2000	0.077	0.075	0.101
DP3000	0.071	0.041	0.038
DP4000	0.170	0.125	0.091
DP6000	----	0.130	0.149
Noise level
N1500	----	----	0.051
N2000
N3000	-0.050		-0.088
N4000	-0.061	-0.097	-0.104
N6000	----
Constant	-0.989	-0.532	-1.515

---- indicates that this was not an input variable for the PTA series under investigation.

Table A3.

DPOAE levels (top) and probabilities of normal (bottom) at the 95^th and 90^th percentiles of the cumulative distributions of responses from ears with hearing loss, and the 10^th and 5^th percentiles from the cumulative distributions of responses from ears with normal hearing. These values are plotted as the limiting lines in the second and third columns of Fig. 5. Normal hearing was defined as thresholds ≤ 20 dB HL. Thresholds exceeding this value were defined as hearing impaired

DPOAE LEVEL
Percentile	0.75 kHz	1 kHz	1.5 kHz	2 kHz	3 kHz	4 kHz	6 kHz	8 kHz
95th impaired	5.95	7.76	3.84	-0.93	-2.50	0.18	-2.13	-9.96
90th impaired	2.41	4.40	0.43	-3.51	-5.55	-4.43	-6.87	-12.84
10th normal	-10.40	-8.12	-6.83	-9.82	-11.50	-5.93	-7.83	-20.00
5th normal	-13.60	-12.35	-9.80	-13.87	-16.25	-9.55	-11.05	-20.00

PROBABILITY OF NORMAL
Percentile	0.75 kHz	1 kHz	1.5 kHz	2 kHz	3 kHz	4 kHz	6 kHz	8 kHz
95th impaired	84.02	83.99	80.63	72.99	67.82	62.27	69.13	55.16
90th impaired	75.47	67.48	68.35	56.91	46.76	37.47	44.13	32.59
10th normal	40,85	39.77	40.83	36.12	36.12	35.12	41.39	29.28
5th normal	26.06	25.83	24.53	16.80	19.78	17.51	18.63	9.28