Evaluation of Audiometric Threshold Shift Criteria for Ototoxicity Monitoring

Abstract

Background

There is disagreement about ototoxicity monitoring methods. Controversy exists about what audiometric threshold shift criteria should be used, which frequencies should be tested, and with what step size. An evaluation of the test performance achieved using various criteria and methods for ototoxicity monitoring may help resolve these issues.

Purpose

(1) Evaluate test performance achieved using various significant threshold shift (STS) definitions for ototoxicity monitoring in a predominately veteran population; and (2) determine whether testing in ⅙- or ⅓-octave steps improves test performance compared to ½-octave steps.

Research Design

A prospective, observational study design was used in which STSs were evaluated at frequencies within an octave of each subject’s high-frequency hearing limit at two time points, an early monitoring test and the final monitoring test.

Study Sample

Data were analyzed from 78 ears of 41 patients receiving cisplatin and from 53 ears of 28 hospitalized patients receiving nonototoxic antibiotics. Cisplatin-treated subjects received a cumulative dosage ≥350 mg by the final monitoring test. Testing schedule, age, and pre-exposure hearing characteristics were similar between the subject groups.

Data Collection and Analysis

Threshold shifts relative to baseline were examined to determine whether they met criteria based on magnitudes of positive STS (shifts of ≥5, 10, 15, or 20 dB) and numbers of frequencies affected (shifts at ≥1, 2, or 3 adjacent frequencies) for data collected using approximately ⅙-, ⅓-, or ½-octave steps. Thresholds were confirmed during monitoring sessions in which shifts were identified. Test performance was evaluated with receiver operating characteristic (ROC) curves developed using a surrogate “gold standard”; true positive (TP) rates were derived from the cisplatin-exposed group and false positive (FP) rates from the nonexposed, control group. Best STS definitions were identified that achieved the greatest areas under ROC curves or resulted in the highest TP rates for a fixed FP rate near 5%, chosen to minimize the number of patients incorrectly diagnosed with ototoxic hearing loss.

Results

At the early monitoring test, average threshold shifts differed only slightly across groups. Test-frequency step size did not affect performance, and changes at one or more frequencies yielded the best test performance. At the final monitoring test, average threshold shifts were +10.5 dB for the cisplatin group, compared with −0.2 dB for the control group. Compared with the ½-octave step size used clinically, use of smaller frequency steps improved test performance for threshold shifts at ≥2 or ≥3 adjacent frequencies. Best overall test performance was achieved using a criterion cutoff of ≥10 dB threshold shift at ≥2 adjacent frequencies tested in ⅙-octave steps. Best test performance for the ½-octave step size was achieved for shifts ≥15 dB at one or more frequencies.

Conclusions

An ototoxicity monitoring protocol that uses an individualized, one-octave range of frequencies tested in ⅙-octave steps is quick to administer and has an acceptable FP rate. Similar test performance can be achieved using ⅓-octave test frequencies, which further reduces monitoring test time.

Cisplatin is a platinum-based drug that is widely used to treat cancers of various types and is considered unrivaled in its chemotherapeutic effect. However, cisplatin produces ototoxic hearing loss in a large number of those treated with the drug. Schweitzer (1993) calculated that the incidence of cisplatin-induced hearing loss averaged across a large number of studies was 62%, with a range from 11 to 97%. Hearing loss due to cisplatin is sensorineural, often bilateral, and usually permanent (van Zeijl et al, 1984; Brock and Bellman, 1991; Waters et al, 1991; Nagy et al, 1999). The loss can be progressive, with hearing changes sometimes observed well after treatment has ended (Bertolini et al, 2004).

Damage from cisplatin initially affects the outermost row of outer hair cells located near the cochlear base, which codes the high frequencies. Further damage affects the remaining rows of outer hairs cells, inner hair cells, and supporting structures. The pattern of destruction proceeds from the base toward more apical regions of the cochlea, impacting increasingly lower frequencies with continued exposure (Brummett, 1980; Komune et al, 1981; Nakai et al, 1982; Konishi et al, 1983; Schweitzer et al, 1984).

As early as 1994, the American Speech-Language-Hearing Association (ASHA) proposed “Guidelines for the Audiologic Management of Individuals Receiving Cochleotoxic Drug Therapy.” These guidelines state that patients treated with ototoxic drugs should be prospectively monitored for hearing changes primarily using serial audiometric testing. Relying on subjective hearing complaints is problematic because patients tend not to notice hearing loss until it has progressed to the speech frequencies, an indication that the loss is already substantial enough to warrant aural rehabilitation. Additionally, serum drug levels are poor indicators of ototoxicity because there is variability in the dose-toxicity relationship that has been linked to numerous factors, including individual susceptibility, mode of drug administration (Kopelman et al, 1988; Bokemeyer et al, 1998; Ekborn et al, 2000), exposure to other risk factors or ototoxic agents (Blakley et al, 1994; Bokemeyer et al, 1998), and comorbidity issues such as renal insufficiency (Schaefer et al, 1985; Bokemeyer et al, 1998).

In their guidelines, ASHA recommended particular significant threshold shift (STS) definitions. These are defined according to a set of decision variables (threshold shifts at one or a combination of frequencies) and cutoff criteria (positive threshold shifts in dB). They include ≥20 dB threshold shift at any one frequency, ≥10 dB shift at any two adjacent frequencies, or loss of response at any three adjacent frequencies. For these STS criteria, small threshold shifts are required to occur at two or more adjacent frequencies, and any shifts must be confirmed in order to be considered a true change. These stipulations improve test-retest reliability (Royster and Royster, 1986).

ASHA also recommended including extended high-frequency threshold testing from 9 to 20 kHz when possible in the monitoring protocol because it significantly improves test sensitivity (Fausti et al, 1984; Tange et al, 1985; van der Hulst et al, 1988; Dreschler et al, 1989; Fausti et al, 1992, 1993, 1994; Ress et al, 1999). However, combining conventional and extended high frequencies results in a lengthy test.

Testing the conventional audiometric frequencies from 0.25–8.0 kHz is a common method for ototoxicity monitoring. Presumably this is because such testing is economically cheap, quick to administer, and less fatiguing for the patient compared with full frequency testing (0.25–20 kHz). Other monitoring methods have been proposed that are also rapid and somewhat easy to administer. Simpson and colleagues tentatively recommended using a threshold shift ≥20 dB at 8 kHz for adults treated with cisplatin (Simpson et al, 1992). Pasic and Dobie (1991) recommended using a two frequency average increase of ≥15 dB at 6 and 8 kHz for children treated with cisplatin. While these fixed test frequency approaches are faster than the full frequency testing recommended by ASHA, they would presumably be less sensitive to early ototoxic damage above 8 kHz than extended high-frequency testing in patients with good pre-exposure hearing. Another problem with selecting specific high frequencies to monitor is that some patients lack hearing at high frequencies even before treatment with ototoxic medication.

More recently, Fausti and colleagues have asserted that an effective screening test for ototoxicity is one that monitors the highest frequencies heard by an individual patient. Their method involves operationally defining a high-frequency hearing limit, followed by pure-tone threshold testing (with ⅙-octave precision) within a one-octave range up to this limit. Thus, the frequency range targeted for monitoring varies across patients and could be within the conventional or extended high-frequency range depending on pre-exposure hearing ability. Studies have shown that in about 90% of ears, cisplatin, carboplatin, and ototoxic antibiotic-induced hearing changes presented first within this sensitive (frequency) range for ototoxicity or SRO (Fausti et al, 1999; Vaughan et al, 2002; Fausti et al, 2003). Overlap between the SRO and the speech frequencies signifies that ototoxic hearing changes could immediately and directly impact speech communication.

Fausti and colleagues (1999, 2003) have proposed that ASHA recommended STS criteria be used with the SRO protocol. This approach relies on false positive (FP) rates within the SRO being within a clinically acceptable range. Previous studies have shown that test-retest variability, which could lead to the generation of FPs in serial ototoxicity monitoring tests, is typically within 10 dB for audiometric frequencies from 9 to 20 kHz even for subjects tested on the hospital wards (Matthews et al, 1997; Frank, 2001; Schmuziger et al, 2004; reviewed in Gordon et al, 2005). In a study evaluating SRO threshold test-retest comparisons, Gordon et al (2005) found that FP rates were low (5% or less) using ASHA-recommended STS criteria. Subjects in that study were not receiving ototoxic drugs and were tested in a sound booth and on a hospital ward using insert earphones (ER-4B MicroPro) and modified KOSS Pro/4X Plus circumaural earphones. None of the subjects had threshold changes greater than 20 dB, regardless of the transducer or test setting. Although these results suggest that ASHA criteria are appropriate for extended high-frequency testing, the ASHA criteria have not yet been systematically evaluated in patients treated with ototoxic drugs.

To determine the test accuracy and optimal usage of serial pure-tone threshold testing for ototoxicity monitoring, test performance was evaluated with receiver operating characteristic (ROC) curves developed using a surrogate “gold standard” method. True positive (TP) rates were derived from a cisplatin-exposed group and FP rates from a nonexposed, control group. Hearing changes within both groups were evaluated at frequencies within the SRO because early ototoxic hearing changes take place primarily there. The ROC curves were constructed for varying decision criterion cutoffs (threshold shifts in dB). The area under the ROC curve (AUC) is a nonparametric estimate of test performance that is independent of the exact measurement on which the diagnosis is made. Thus it was possible to compare test performance using various numbers of affected frequencies and test frequency step sizes.

This approach differs from standard ROC curves rooted in clinical decision theory (CDT) methods in an important way. An underlying assumption of CDT is that the true category to which each subject belongs, diseased or not diseased, is known with a high degree of reliability. Thus, there typically is a “gold standard” measurement of the disease condition with which the outcome on the experimental test can be compared. This assumption was not met for the present study. However, surrogate methods have been developed and used previously to assess serial audiometric testing in the absence of a true gold standard (Dobie, 1983; Royster and Royster, 1986; Simpson et al, 1992; Daniell et al, 2003; also see review by Dobie, 2005). Each of these methods has drawbacks, but in the absence of a true gold standard, they are sufficient to provide at least some information about the relative utility of various testing procedures.

Only two previous reports have used CDT methods to assess test performance for ototoxicity monitoring. These studies also employed surrogate gold standard methods. In the first study, Pasic and Dobie (1991) assumed that threshold improvements in cisplatin-treated children were indications of retest variability and that only a worsening of thresholds greater than the magnitude of this shift were actual TPs. Simpson and colleagues (1992) compared threshold shift rates for groups of cisplatin-exposed and nonexposed adults with cancer. All ears in both groups contributed to the analysis for this method, called the “Comparison Method” by Dobie (2005). Ears with STS in the exposed group divided by the total number of ears in that group defined the TP rate. Similarly, ears with STS in the unexposed group divided by the total number of ears in that group defined the false positive (FP) rate.

The Comparison Method was used in the analytic strategy for our study because it is superior to other surrogate methods, based on analyses using computer simulations of gold standard and ototoxic threshold shift data. In particular, Dobie (2005) showed that the Comparison Method accurately estimates FP rates and rankings for competing STS decision rules. A disadvantage of this method is that TP rates underestimate test sensitivity because the exposed cisplatin group includes individuals with and without a true hearing change. TP rates estimated using the Comparison Method depend on the incidence of STS among the cisplatin-exposed patients. This limitation does not, however, detract from the current study in which the goal involves examining relative test performance (and not absolute test sensitivity) achieved using various STS test criteria.

The objectives of this study are to (1) Evaluate test performance achieved using various STS definitions for ototoxicity monitoring in a predominately veteran population; and (2) determine whether testing in ⅙-or ⅓-octave steps improves test performance when compared to the use of ½-octave steps.

METHODS

Subjects

The present report is a prospective, observational study describing data obtained from subjects recruited from VA Medical Centers in Portland, OR; Nashville, TN; and West Los Angeles, CA. The Nashville, TN, site also recruited subjects from Vanderbilt Medical Center. Pure-tone threshold (Fausti et al, 1999, 2003) and DPOAE (distortion product otoacoustic emissions) data (Reavis et al, 2008) have been reported previously for subsets of these subjects in studies that examined methods for ototoxicity monitoring. Results in this paper are reported in terms of the proportion of ears in each subject group that exhibited shifts in the STS criteria.

Cisplatin-treated subjects were included in the analysis if they (1) received an accumulated dosage of cisplatin greater than or equal to 350 mg and (2) had at least three audiograms, including a baseline hearing test within 48 hours of receiving their first dose of cisplatin and a monitoring test that took place between 28 and 63 day from the baseline test. An important aspect of this work was the selection of a control group considered unlikely to develop ototoxic hearing loss but otherwise similar to subjects undergoing medical treatment with ototoxic drugs in terms of general health, age, pre-exposure hearing loss, and test-retest reliability. Control subjects, hospitalized primarily at the participating VA Medical Centers, were included in the study if they (1) were receiving widely used nonototoxic antibiotics including ceftriaxone, ampicillin, clindamycin, or nafcillin; (2) had not received a potentially ototoxic drug within the previous 30 day; and (3) had at least three audiograms, including a baseline test within 72 hr of their initial dose and a monitoring test that took place between 28 and 63 day from the baseline test.

In addition to the above, all subjects met the following inclusion criteria: (1)no active or recent history of middle ear pathology, (2) no history of retrocochlear or Ménière’s disease, and (3) ability to respond reliably to behavioral pure-tone audiometry based on a comparison of baseline thresholds to thresholds obtained within 24 hr of the initial test (as described in the “Subject Testing” section below). All subjects were consented to participate in the study following the guidelines of each participating VA Medical Center’s Institutional Review Board, signed an IRB approved consent form, and were compensated for their time.

Equipment and Calibration

Pure-tone thresholds were measured from 0.5 to 20 kHz using a Virtual Corporation, Model 320 (V320) audiometer. TDH-50P earphones in MX-41/AR cushions were used for testing 0.5 and 1 kHz thresholds. Koss Pro/4X Plus earphones, modified to improve signal-to-noise ratio for high-frequency testing (Fausti et al, 1990), were used for testing frequencies from 2 to 20 kHz. Calibration of the audiometer was conducted at least twice each month according to the appropriate American National Standards Institute (ANSI) standards for each earphone type (ANSI S3.6-1989; ANSI, 1989). TDH-50P earphones were coupled to a Bruel & Kjaer (B&K) 4153 artificial ear, and the acoustic output measured using a B&K 4134 half-inch condenser microphone routed to a B&K 2231 sound level meter. KOSS Pro/4X Plus earphones were calibrated in a similar manner, except that the coupler used was a 6 cc flat-plate coupler, as described in Fausti et al (1979). A Grason-Stadler GSI 33 Middle-Ear Analyzer was used to obtain tympanometry measures. Calibration of the middle ear analyzer was performed annually by MSR Northwest, Inc.

Subject Testing

Testing Schedule

Baseline tests usually occurred within 24 hr of the first drug treatment for cisplatin-treated subjects and within 72 hr of the first treatment for control subjects. Ototoxicity monitoring tests were performed following each cisplatin treatment, usually within 24 hr of drug administration. Monitoring tests for the control group were performed every 2–3 day. This report presents baseline test data, data obtained on the first test within a period from 23 to 68 day after baseline (early monitoring test), and the last test obtained on each subject (final monitoring test), which took place from 1.5 to 6.7 mo after baseline for the subjects sampled.

Audiometric Tests

Otoscopy and tympanometry were performed at each test session in order to rule out changes in hearing that could be attributed to abnormal middle ear function and not ototoxicity. Normal middle ear function was determined based on the presence of a compliance peak in the range from 0.3 to 1.67 cm³ and a pressure peak of −150 to 100 daPa obtained using a 226 Hz probe tone.

Pure-tone air conduction thresholds were obtained with pulsed tones using a modified Hughson-Westlake procedure (Carhart and Jerger, 1959). Frequencies tested at baseline were standard audiometric frequencies from 0.5 to 8 kHz and the interoctave frequencies 3 and 6 kHz. Subjects also underwent routine extended high-frequency testing, which included testing in approximately ⅙-octave frequency steps from 9 through 20 kHz. Baseline testing also involved the determination of an individualized, sensitive range for ototoxicity (SRO) defined as the uppermost frequency at which threshold is 100 dB SPL or less and the six consecutive lower frequencies in ⅙-octave steps (Fausti et al, 1999). Depending on the patient’s pre-exposure hearing configuration, the SRO (and, therefore, testing in ⅙-octave increments) could extend as high as 20 kHz and as low as 2 kHz.

Table 1 illustrates the method for determining the SRO. The high-frequency hearing limit, F, was operationally defined as the uppermost frequency with a threshold ≤100 dB SPL (12.5 kHz for this example). Thresholds were tested in ⅙-octave increments for a span of about one octave below F (labeled “F-1,” “F-2,” etc.). The number of test frequencies used for determining whether or not an ototoxic shift occurred decreased as a function of increasing step size. Thus, testing in ½-, ⅓-, and ⅙-octave steps below F yielded sets of three, four, and seven frequencies, respectively, within the SRO that were evaluated for changes.

Table 1.

Determining One Subject’s Test Frequencies and High-Frequency Hearing Limit

Baseline Test Frequencies (kHz)	0.50	1.00	2.00	3.00	4.00	6.00	6.35	7.13	8.00	9.00	10.00	11.20	12.50	14.00	16.00	18.00	20.00
Baseline Hearing Thresholds	25	25	35	40	55	55	55	60	70	75	85	90	100	110	> 115	> 115	> 115
Comparison Test Frequencies ⅙ octave							F-6	F-5	F-4	F-3	F-3	F-1	F
Comparison Test Frequencies ⅓ octave							F-3		F-2		F-1		F
Comparison Test Frequencies ½ octave							F-2			F-1			F

Following baseline testing, a retest of baseline thresholds was performed usually within 24 hr to validate intersession reliability. Subjects were considered capable of providing reliable responses if threshold retest values were within 5 dB of baseline measures.

Threshold data collected during monitoring sessions after the baseline test were sometimes restricted to frequencies within the subject’s SRO and other times included the entire audible frequency range. Threshold shifts observed during monitoring tests were confirmed by retest during the monitoring session in which the shift was identified. When the retest results indicated that SRO frequencies showed a significant change in hearing using the ASHA (1994) criteria listed in Table 2, all of the frequencies tested at baseline were retested within 24 hr or (if that was not possible due to limitations in the patient’s schedule or health) at the next monitoring test session.

Table 2.

Published Criteria for Significant Threshold Shifts

Type of Hearing Loss	Criterion Number of Frequencies	Criterion Threshold Shift	Source
Ototoxic induced	any one frequency, confirmed by retest	20 dB decrease	ASHA (1994)
	two or more consecutive, confirmed by retest	10 dB decrease
	three or more consecutive, confirmed by retest	loss of response
Cisplatin induced (adults)	threshold loss at 8 kHz	20 dB	Simpson et al (1992)
Cisplatin induced (children)	two-frequency average at 6 and 8 kHz	15 dB	Pasic and Dobie (1991)
Noise induced	three-frequency average at 2, 3, 4 kHz	10 dB	Occupational Safety and Health Administration (1983)
Noise induced	any one frequency, on two separate tests	15 dB	National Institute for Occupational Safety and Health (1998)

Data Analyses

Data for this study are based on the number of ears for the subjects entered. In some subjects, data were taken from a single ear; in others, both ears were included and underwent separate testing. The “gold standard” determination of ototoxic-induced hearing loss for each subject depended on whether they received cisplatin, in which case they were classified as having an ototoxic-induced hearing change, or did not receive cisplatin, in which case they were classified as having stable hearing. This procedure is described as the “Comparison Method” (Dobie, 2005).

TP rates were defined as positive threshold shift rates determined for the group of subjects exposed to cisplatin; FP rates were positive threshold shifts determined for the group of nonexposed control subjects. Receiver operating characteristic (ROC) curves were constructed by plotting the TP rate (sensitivity) as a function of the corresponding FP rate (1 − specificity) for four cutoff criteria that were based on magnitudes of the threshold shifts ≥5, 10, 15, or 20 dB. ROC curves were generated for each of three decision variables (changes at ≥1, 2, or 3 adjacent frequencies), in combination with the three frequency step sizes used in testing (½, ⅓, and ⅙ octave).

Differences between ROC curves were analyzed by comparing the AUCs, determined using the trapezoidal rule, a numerical integration method. Each ROC consists of five trapezoids defined by the six pairs of TP and FP points (from the four threshold shifts 5, 10, 15 and 20 dB, plus the two extremities TP = FP = 0.0 and TP = FP = 1.0). Summing the areas for these trapezoids provides the total AUC under the ROC. The AUC value is a nonparametric estimate that corresponds to the probability that a subject is correctly categorized as belonging to the disease group or the nondisease group (Hanley and McNeil, 1982). This probability ranges from chance (0.5) to perfect test performance (1.0). Pairwise comparisons of the AUCs for the three frequency step sizes in combination with the three affected frequencies were constructed by dividing the difference in the AUCs by the standard error (SE) of the difference (see Appendix for formula), adjusting for the correlation of the tests that were performed within the same group of subjects (Hanley and McNeil, 1983).

RESULTS

Subject Characteristics

A total of 69 subjects were selected for analyses. Forty-one subjects received cisplatin chemotherapy; 78 ears from this group were evaluated. Twenty-eight subjects in the control group received no ototoxic medication; 53 ears from this group were evaluated. Subject characteristics are given in Table 3. The two groups had a similar mean age and treatment duration at the early monitoring test. At the final monitoring test, the duration of treatment was slightly shorter for the cisplatin-treated subjects compared to controls. At the final monitoring test, all cisplatin subjects had an accumulated dose greater than 350 mg, a risk factor for cisplatin ototoxicity. Rates of ototoxicity increase dramatically when the cumulative dose approaches 400 mg (Schaefer et al, 1985) or less if thresholds at extended high frequencies are examined (Kopelman et al, 1988).

Table 3.

Summary of Age and Testing Schedule

Control Group (n = 28)
	Age (years)	Treatment Duration (days)
		Early Monitoring Test	Final Monitoring Test
Mean	56.0	44.0	205.9
SD	10.5	10.4	54.4
Range	37–80	29–61	115–297
Cisplatin Group (n = 41)
Mean	59.4	44.4	194.6
SD	10.2	9.1	90.8
Range	27–77	28–63	57–413

Cisplatin-induced threshold shifts usually occurred first near each subject’s high-frequency hearing limit. Figure 1 shows serial pure-tone threshold data in dB SPL plotted as a function of test frequency in kHz. Data are from a single subject, a 54-yr-old male with laryngeal cancer. Pure-tone air-conduction thresholds at conventional and extended high audiometric frequencies were tested at each evaluation for this subject. The threshold data are plotted by test date, which is indicated in the legend. Baseline (pre-exposure) hearing is represented by diamonds. The SRO for this subject is 7.13 to 14 kHz. This subject received two doses of cisplatin, three weeks apart, resulting in a cumulative cisplatin dose of 460 mg. Repeat evaluations (not shown) were completed confirming baseline and postchange thresholds.

Figure 1.

Example of hearing thresholds measured over time for a subject receiving cisplatin.

Hearing changes associated with the timing of drug treatments are evident in Figure 1. This subject had a 15 dB change from baseline to 6 mo postcisplatin at 3 kHz, a 40 dB change at 4 kHz, and a 65 dB change at 8 kHz. However, the greatest hearing changes observed were above the range of conventional audiometric frequencies, >8 kHz, including a complete loss of hearing at 10 kHz and above. Hearing changes were reported to the treating physician and were used to counsel the subject about strategies for communication, other aural rehabilitation options, and the necessity of follow-up with an audiologist. Notably, hearing changes within the conventional frequency range occurred primarily after drug treatment ended. Use of extended high-frequency threshold testing combined with follow-up testing after drug treatment provided strong evidence that hearing changes in this subject were attributable to ototoxicity.

Threshold Comparisons across Subject Groups, Time Points, and Test Frequencies

Since a degradation of threshold response is expected in the cisplatin group over time, we examined thresholds across baseline, early monitoring, and final monitoring tests for the seven normalized SRO test frequencies. Figure 2 shows control group (left) and cisplatin group (right) thresholds. Data for SRO frequencies are normalized to each subject’s high-frequency limit, F, defined during the baseline test. Error bars represent standard error.

Figure 2.

Comparison of pure-tone thresholds measured on three separate visits in control (left) and cisplatin-treated subjects (right). Data are for SRO frequencies normalized to each subject’s highest audible frequency, F, defined during the baseline test. F-1 is ⅙ octave below F, F-2 is ⅙ octave below F-1, etc. Mean and standard error for thresholds at SRO frequencies are given.

On average, control subjects had slightly better hearing at baseline compared with cisplatin-treated subjects. A two-by-seven repeated measures analysis of variance (ANOVA) was performed on baseline thresholds obtained in the two subject groups for the seven SRO frequencies. The main effect of subject group was not statistically significant (p = 0.07), and there was no group by frequency interaction (p > 0.05). As expected, there was a main effect of test frequency (p < 0.001) because thresholds in SPL tend to worsen near the high-frequency hearing limit.

Separate repeated measures ANOVAs were performed for control and cisplatin-treated subjects to evaluate the main effects of test session (baseline, early monitoring test, and final monitoring test), the SRO test frequency, and the interactions between them. For control subjects, the SRO frequency effect was significant (F_{6, 276} = 105.54; p < 0.01). Mean thresholds obtained in control subjects differed by less than 2 dB across test sessions. These differences were not significant (F_{2, 92} = 2.10; p = 0.13), and the test session by SRO test frequency interaction was not significant (F_{12, 552} = 0.33; p ≥ 0.99). In contrast, for the cisplatin-treated subjects, the SRO test frequency effect was significantly different (F_6,414 = 137.12; p < 0.0001), test sessions were significantly different (F_{2, 1380} = 32.89; p < 0.0001), and the test session by test frequency interaction was significant (F_{12, 828} = 3.40; p < 0.0001). Threshold changes in the cisplatin-treated group were only about 10 dB on average by the final monitoring test, which reflects the fact that mean data underestimate effects in individual subjects because some subjects have hearing loss at one or more test frequencies but others have stable hearing. However, the fact that the controls did not differ across the three test sessions examined while the cisplatin subjects did, appears to support the premise that the SRO method is effective for monitoring ototoxicity. Additionally, the lack of group changes among individuals receiving non-ototoxic medications suggests few if any had hearing changes, which substantiates the selection of the control group for determining FP rates.

Prediction of Ototoxicity by Threshold Shifts

Figure 3 shows ROC curves for data obtained at the early monitoring test. Panels represent the three decision variables, threshold shifts at one or more (left), two or more (middle), or three or more (right) adjacent test frequencies. Each data point represents the mean TP rate and the mean FP rate averaged across the three frequency step sizes examined (⅙, ⅓, and ½ octave; means were plotted because the TP and FP rates were very close across the three step sizes). As indicated in the left panel of Figure 3, the point on the ROC curve closest to the origin corresponds to a cutoff criterion threshold shift of 20 dB or greater. The next point on the curve corresponds to a shift of 15 dB or greater, and so on, with the point farthest to the right side of the curve representing a shift of 5 dB or greater. Thus, each ROC curve illustrates the trade-off between TP and FP rates for values of the cutoff criterion that would be considered stringent (20 dBshifts) orlax (5 dB shifts) in the context of normal test-retest variability for pure-tone audiometry.

Figure 3.

ROC curves comparing test performance for a variety of threshold shift criteria. Data were obtained by comparing baseline audiograms to the early monitor test. TPs are calculated from a mixture of affected and unaffected individuals and, thus, underestimate the sensitivity of the test. FPs are calculated from individuals not exposed to ototoxic drugs and thus provide an estimate of test-retest variability. The AUCs are indicated in Table 4.

At the early monitoring test, the probability of correctly designating a subject as exposed or non-exposed based on audiometric test criteria varied slightly from 70 to 75%, and the greatest AUC was achieved for the single frequency decision variable (Fig. 3, left). FP rates for a cutoff criterion of 5 dB were typically unacceptably high even for the three-frequency decision variable (Fig. 3, right). Octave step size was not significant early in treatment when hearing changes were slight and the spread of damage minimal. Therefore, separate plots for octave steps are not shown. However, across time points examined, AUCs consistently increased as octave step size decreased, as discussed below.

Figure 4 compares ROC curves constructed for data collected during the final monitoring test when threshold shifts for the cisplatin group were greater than 10 dB across most of the SRO range (see Fig. 2). The format of Figure 4 is consistent with Figure 3, except that separate curves in each panel represent the three octave steps examined. ROC curves for the final monitoring tests are summarized by Table 4 for the three frequency decision variables and three octave step sizes. The diagonal cells give the AUCs for the three octave steps (within a frequency decision variable) while the off-diagonal cells compare the AUC for the three pair-wise combinations of octave steps. For two or more (Fig. 4, middle) and three or more adjacent frequencies (Fig. 4, right), ⅙-and ⅓-octave steps provide significantly better AUCs than the ½-octave step; that is, for the ½-octave step, test performance becomes poorer when changes were required to occur at more than one frequency. Observed differences between ⅙- and ⅓-octave steps were not significant. Compared to the ⅙-octave step size, the ⅓-octave step size achieved similar test performance with reduced monitoring test time.

Figure 4.

ROC curves comparing test performance for a variety of threshold shift criteria. Data were obtained by comparing baseline audiograms to the final monitoring test on each subject.

Table 4.

Comparison of Final Monitoring Tests AUCs for Thresholds at One, Two, Three or More Adjacent Frequencies Tested Using ⅙-, ⅓-, or ½-Octave Steps

		OCTAVE STEPS
		One-sixth	One-third	One-half
One Frequency
OCTAVE STEPS	One-sixth	AUC = 0.779
		SE(AUC) = 0.042
			Diff(AUC) = 0.011	Diff(AUC) = 0.006
			SE(Diff) = 0.021	SE(Diff) = 0.026
			p = 0.60	p = 0.82
	One-third		AUC = 0.768
			SE(AUC) = 0.043
				Diff(AUC) = −0.005
				SE(Diff) = 0.021
				p = 0.80
	One-half			AUC = 0.773
				SE(AUC) = 0.042
Two Frequencies
OCTAVE STEPS	One-sixth	AUC = 0.809
		SE(AUC) = 0.039
			Diff(AUC) = 0.044	Diff(AUC) = 0.119
			SE(Diff) = 0.024	SE(Diff) = 0.034
			p = 0.072	p = 0.0004
	One-third		AUC = 0.765
			SE(AUC) = 0.043
				Diff(AUC) = 0.075
				SE(Diff) = 0.031
				p = 0.014
	One-half			AUC = 0.690
				SE(AUC) = 0.048
Three Frequencies
OCTAVE STEPS	One-sixth	AUC = 0.782
		SE(AUC) = 0.042
			Diff(AUC) = 0.031	Diff(AUC) = 0.119
			SE(Diff) = 0.036	SE(Diff) = 0.037
			p = 0.38	p = 0.001
	One-third		AUC = 0.751
			SE(AUC) = 0.044
				Diff(AUC) = 0.089
				SE(Diff) = 0.030
				p = 0.0034
	One-half			AUC = 0.662
				SE(AUC) = 0.050

Note: SE(AUC) = standard error of the AUC; Diff(AUC) = difference between AUCs; SE(Diff) = standard error of the Diff(AUC).

In addition to ranking decision variables based on AUCs, we determined the particular STS definitions that yield the highest TP rates combined with a clinically acceptable FP rate. In order to illustrate what was done, the ROC data presented in Figure 4 are replotted as bar graphs in Figure 5. The three columns in Figure 5 indicate the decision variables, and the three rows indicate the octave step sizes. STS definitions that fall to the right of the thick vertical lines meet an arbitrary “acceptable FP rate” near 5%. The 5% rule was chosen as a conservative FP rate to limit the number of patients incorrectly diagnosed with ototoxic hearing changes, with the understanding that it could result in a reduced TP rate.

Figure 5.

Mean TP rates (sensitivity) and FP rates (1 – specificity) plotted versus magnitude of threshold shifts. Columns: number of frequencies at which threshold shifts were required to occur. Rows: Test frequency step sizes examined. Data shown are for the final monitoring test date. Vertical lines indicate STS definitions that achieved FP rates near 5%.

For one or more affected frequencies (Fig. 5, left column), a criterion cutoff of 20 dB or greater yields no FPs regardless of the octave step size used. Criterion cutoff values less than 20 dB fail to meet the conservative 5% rule for the ⅙- and ⅓-octave step sizes. However, a criterion cutoff of 15 dB or greater results in an acceptable (near 5%) FP rate for the ½-octave step size.

Decision variables that incorporate ≥2 or 3 adjacent frequencies (Fig. 5, middle and right columns) yield FP rates near 5% for criterion cutoffs of 10 dB or greater, regardless of the octave step size. Also note that 5 dB shift criteria never resulted in an acceptable FP rate, even for the decision variable incorporating three or more adjacent frequencies. For these multiple frequency decision variables, smaller frequency steps consistently produce greater TP rates, with the greatest overall TP rate (61%) achieved using the ⅙-octave step size for a criterion cutoff of 10 dB or more at two or more frequencies.

The data in Figure 5 can be used to rank STS definitions based on fixed FP rate rules other than 5%. The use of a less conservative FP rate rule, 10%, yields slightly different results. However, the STS definition that results in the greatest overall TP rate is the same (i.e., ≥10 dB threshold shift at two or more frequencies within a ⅙-octave SRO) whether the FP rate is fixed near 5 or 10%.

DISCUSSION

Summary

This study sought to provide an evaluation of test performance for ototoxicity monitoring methods that might help determine best practice procedures. Various STS definitions were examined, including decision variables (shifts at one frequency or a combination of adjacent frequencies) and criterion cutoffs (positive threshold shifts ≥5, 10, 15, or 20 dB). These STS were evaluated for frequencies tested using approximately ½-, ⅓-, or ⅙-octave frequency step sizes. STS definitions included those recommended by ASHA (1994). All measurements were for frequencies within an octave of each subject’s high-frequency hearing limit, and this limit was often greater than 8 kHz. Cisplatin-exposed subjects as well as nonexposed controls were patients at participating medical centers, primarily within the Veterans Health Administration System of Care.

The analyses in this report include the construction of a series of ROC curves taken at two time points, an early monitoring test and the final monitoring test from the same sample of subjects. The Comparison Method, a surrogate method, was used to provide the “gold standard” determination of ototoxicity for the ROC analyses, with implications described in the introduction. Statistical methods (Hanley and McNeil, 1983) were employed in order to determine whether observed differences in test performance were significant. In addition to ROC curve analyses, STS definitions were identified that yielded the greatest TP rates for a fixed FP rate near 5%. Results provide much needed data to illustrate how certain protocol choices are likely to affect test performance for an adult population of patients treated with cisplatin. Results have implications for the design of ototoxicity monitoring programs, and the interpretation of test results obtained in such programs.

Ranking Test Performance

Ranking Decision Variables

Although test-retest differences were similar for controls at the early monitoring and final monitoring test, ROC curve analyses at these two time points yielded slightly different results. This was at least in part because ototoxic damage experienced by exposed subjects in our study increased over time as indicated by increases in the magnitudes of threshold losses and in the ranges of affected frequencies (as shown in Figs. 1 and 2). Observed AUCs were similar across decision variables at the early monitoring test. At the final monitoring test, the highest ranked ROC curve was for changes at two or more adjacent frequencies (i.e., a multi-frequency decision variable), consistent with a greater spread of damage at the final monitoring test compared with the early monitoring test. One interpretation of these results is that a combination of decision variables should be used for monitoring ototoxic-induced threshold shifts because the best single decision variable depends on the extent of the damage present at the time of testing.

Ranking Test Frequency Step Sizes

The ⅙-octave step size provided the highest AUCs for threshold shifts at one or more, two or more, and three or more adjacent frequencies for the final monitoring test (compare AUCs on the diagonals in Table 4 for the three decision variables) and the early monitoring test (not shown). This may relate to the test frequencies decreasing slightly as the octave step size increased because test frequencies were assigned relative to the high-frequency hearing limit, F. For the example given in Table 1, F-1 is 11.2 kHz for the ⅙-octave step size compared with 10.0 kHz for the ⅓-octave step size and 9.0 kHz for the ½-octave step size. However, octave steps do not differ significantly at the early monitoring test, or at the final monitoring test for the one or more frequencies decision variable (Fig. 4, left), suggesting that proximity to F is not the only important difference among octave step sizes. These results indicate that the range of frequencies with threshold shifts was sometimes narrow relative to the ½-octave step size. Because statistical differences between the ⅙-octave and ½-octave step sizes were present at the final monitoring test, when threshold losses were maximal (excluding any further posttreatment progression of the losses), these differences are deemed important. Compared to the ⅙-octave step size, the ⅓-octave step size achieved similar test performance with reduced monitoring test time.

Ranking Significant Threshold Shift (STS) Definitions

Within an ROC curve, best TP rates were found for STS conditions that consisted of small magnitude changes and narrow ranges of affected frequencies, conditions for which FP rates were the performance-limiting factor. An alternative to ranking decision variables based on AUCs is to identify the particular STS definitions that yield the highest TP rates combined with a clinically acceptable FP rate chosen to minimize the number of patients incorrectly diagnosed with ototoxic hearing changes. At the final monitoring test, the highest overall TP rate observed while holding the FP constant at 5% was about 60% and was associated with threshold shifts ≥10 dB at two or more adjacent frequencies using ⅙-octave steps (see Fig. 5, middle column). The same STS definition produced the best TP rate (~50%) for the ⅓-octave step size at the final monitoring test, and ≥10 dB at two or more adjacent frequencies performed as well as any of the other STS values at the early monitoring test (for ROC curves averaged across step size shown in Fig. 3). However, the best TP rate for the ½-octave step size at the final monitoring test (~50%) was for a change ≥15 dB at one or more frequencies.

Generally, increasing the number of affected frequencies reduced the FP rate, but also the TP rate. If the frequency extent of the lesion is narrow relative to the test frequency step size, which might be the case for early ototoxic hearing changes, this strategy may not produce an overall benefit in test performance, as was presumably the case for the ½-octave step size at the final monitoring test.

It is important to understand that the denominator for the TP rate in the Comparison Method may contain ears that do not have an actual hearing change, resulting in an underestimation of the TP rate. If we know from a “gold standard” diagnosis that all of the ears in the denominator have a hearing change, then we would expect a higher TP rate that would correctly reflect the testing of subjects with “true” hearing loss. If a true gold standard had been used, then a sensitivity rate near 60% would be considered an indication that the test was a poor test. However, in the context of the Comparison Method for generating ROC curves, a 60% sensitivity rate should be interpreted as the incidence of STS among the exposed group, which contains individual ears with and without hearing change. The utility of the comparison in this paper is not in estimating the absolute sensitivity of the various STS definitions; rather, it is to provide information concerning the relative test characteristics of various STS definitions.

Practical Implications of Monitoring

Hearing change resulting from the administration of ototoxic medication is well suited for a monitoring program. Effective monitoring programs target serious diseases or impairments and are associated with an intervention or treatment that, given prior to the onset of symptoms, reduces associated morbidity. Exposure to ototoxic medications can create or exacerbate a hearing impairment, which can have severe social and occupational consequences, in turn generating high rehabilitative costs. Prospective monitoring of hearing while receiving ototoxic medications, and appropriate intervention can reduce the severity of ototoxic hearing loss or the impact of that loss. This can in turn improve post-treatment quality of life and lessen the rehabilitative burden. Specifically, early detection of ototoxic hearing loss provides the prescribing physician the opportunity to reduce the drug dose or change to a less toxic medication, limiting the severity of the hearing loss. If drug regimen changes are not possible, early detection still provides an opportunity for aural rehabilitative treatment including counseling, coping strategies for communication challenges, hearing aids, and assistive listening devices. Thus, an important aspect of an active ototoxicity monitoring program is that it will ensure that ototoxic hearing loss does not go unrecognized and untreated.

Cisplatin ototoxicity can occur after a single dose (Kopelman et al, 1988) or be cumulative-dose dependent (Schaefer et al, 1985; Bertolini et al, 2004). Furthermore, it can occur beyond the end of treatment as shown in Figure 1 and, as previously noted, among children receiving cisplatin (Bertolini et al, 2004). The cisplatin–ototoxicity relationship can be further complicated by other patient factors. The varying temporal onsets of cisplatin ototoxicity coupled with the individual variability necessitates the early and ongoing audiometric monitoring of patients prescribed cisplatin, until some point beyond the end of treatment. Implicit in an ototoxicity monitoring strategy is that if the monitoring test reveals significant shifts in thresholds from baseline values, a definitive diagnostic audiometic evaluation should be conducted to validate that the hearing change is present, rule out any middle ear involvement, determine the full extent of the hearing loss, and assess its impact on speech understanding. Thus, patient management decisions would be based on results of a definitive diagnostic test rather than a screening test.

Comparison to Other Recommended Criteria

The specific STS recommendations arising from this study of ototoxic-induced threshold changes are similar to those proposed by previous reports. The closest comparisons can be made between the ROC analyses of SRO data in the present study and ROC analyses of conventional frequency data in two previous studies. One of these previous studies involved analyses of serial audiograms obtained in adult patients treated with cisplatin (Simpson et al, 1992). On the basis of these analyses, Simpson and colleagues recommended using a threshold shift ≥20 dB at a single frequency, 8 kHz, which yielded a TP rate of 48% and FP rate of 8%. In our study, threshold shift of ≥20 dB at one or more frequencies within a ½-octave increment SRO resulted in a somewhat lower TP rate of about 37% and an FP rate of 0%. Pasic and Dobie (1991) recommended using a two frequency average increase of ≥15 dB at 6 and 8 kHz (½-octave step size) for children treated with cisplatin. The TP rate and FP rate following this protocol was 50 and 0%, respectively. In our report, implementing a ≥15 dB shift at two or more frequencies tested in ½-octave steps within the SRO yielded a TP rate of just over 25% and an FP rate of 0%. However, the children in the Pasic and Dobie report experienced threshold shifts of 35 to 40 dB whereas our subjects experienced thresholds shifts on average of 10 dB at approximately equivalent dosages. These magnitude threshold shift discrepancies are likely secondary to population and or drug administration differences; presumably with continued exposure increasing dosages, the adult subjects in this report would experience higher rates of ototoxicity. However, the threshold shift magnitudes and the TP rate differences underscore the need to monitor with increasing frequency resolution to detect early onset threshold shifts.

Results from previous ROC studies of ototoxicity monitoring data suggest that certain high frequencies could be selectively tested to potentially shorten an ototoxicity monitoring protocol (Pasic and Dobie, 1991; Simpson et al, 1992). We believe this objective is better addressed using the SRO method as described by Fausti (1999), which targets an individually tailored set of frequencies near each patient’s pre-exposure high-frequency hearing limit for monitoring. For this method, decision variables are not tied to particular test frequencies because normal pre-exposure hearing is not assumed.

Results of the present study can be used to assess the three ASHA-recommended combinations of audiometric STS criteria for ototoxicity monitoring. Threshold shifts ≥20 dB at one or more adjacent frequencies and ≥10 dB at two or more adjacent frequencies resulted in low FP rates. The latter was the best overall STS definition for our study using both ⅙-and ⅓-octave step sizes and the second best STS definition for the ½-octave step size. Threshold shifts for three or more adjacent frequencies had to be ≥10 dB to produce an acceptable FP rate for each of the octave frequency steps examined. Shifts of 5 dB, even at three or more adjacent frequencies, generated FP rates greater than 5%.

Summary and Conclusions

Using a comparative analysis, we determined that an ototoxicity monitoring protocol that employs an individualized, one-octave range of frequencies tested in ⅙-octave steps is quick to administer and has an acceptable FP rate. Similar test performance can be achieved using ⅓-octave test frequencies, which further reduces monitoring test time. Use of ½-octave test frequencies decreased test performance for threshold shifts at ≥2 or ≥3 adjacent frequencies. Best overall test performance was achieved using a criterion cutoff of ≥10 dB threshold shift at ≥2 adjacent frequencies tested in ⅙-octave steps. Best test performance for the ½-octave step size was achieved for shifts ≥15 dB at one or more adjacent frequencies.

Abbreviations

AUC

area under the ROC curve

ASHA

American Speech-Language-Hearing Association

CDT

clinical decision theory

FP

false positive

ROC

receiver operating characteristic

SRO

sensitive range for ototoxicity

STS

significant threshold shift

TP

true positive

Appendix

Formulas for the comparison of two ROC curves designated as A1 and A2 (Hanley and McNeil, 1982, 1983): The SE of the AUC (A1 or A2) for a ROC curve is given by

$$\mathrm{SE}(\mathrm{AUC})={\{[\mathrm{A}(1-\mathrm{A})+({\mathrm{n}}_1-1)(\mathrm{Q}1-{\mathrm{A}}^2)+({\mathrm{n}}_2-1)(\mathrm{Q}2-{\mathrm{A}}^2)]/{\mathrm{n}}_1{\mathrm{n}}_2\}}^{1/2},$$

where Q1 = A/(2 − A) and Q2 = 2A²/(1 + A).

If A1 and A2 are the AUCs for two tests applied to the same set of cases, then:

$$\mathrm{SE}(\mathrm{Diff})=\mathrm{SE}(\mathrm{A}1-\mathrm{A}2)={\{\mathrm{S}{\mathrm{E}}^2(\mathrm{A}1)+\mathrm{S}{\mathrm{E}}^2(\mathrm{A}2)-\mathrm{2rSE}(\mathrm{A}1)\mathrm{SE}(\mathrm{A}2)\}}^{1/2},$$

where r represents the correlation between the two areas derived from the set of cases.

The quotient Z = Diff/SE(Diff) = (A1 – A2)/SE(A1 – A2) is normally distributed, for which p-values are readily available. See http://www.anaesthetist.com/mnm/stats/roc/index.htm for an explanation of ROC curves and applicable formulas.