Combined high-speed holographic shape and full-field displacement measurements of tympanic membrane

Abstract.

The conical shape of the tympanic membrane (TM or eardrum) plays an important role in its function, such that variations in shape alter the acoustically induced motions of the TM. We present a method that precisely determines both shape and acoustically induced transient response of the entire TM using the same optics and maintaining the same coordinate system, where the TM transient displacements due to a broadband acoustic click excitation ($50\text{-}\mu \mathrm{s}$ impulse) and the shape are consecutively measured within $<200\mathrm{ms}$. Interferograms gathered with continuous high-speed ($>2\mathrm{kHz}$) optical phase sampling during a single 100-ms wavelength tuning ramp allow precise and rapid reconstructions of the TM shape at varied resolutions (50 to $200\mu \mathrm{m}$). This rapid acquisition of full-field displacements and shape is immune to slow disturbances introduced by breathing or heartbeat of live subjects. Knowledge of TM shape and displacements enables the estimation of surface normal displacements regardless of the orientation of the TM within the measurement system. The proposed method helps better define TM mechanics and provides TM structure and function information useful for the diagnosis of ear disease.

1. Introduction

Sound-induced tympanic membrane (TM, eardrum) vibration is the first stage in the complex multiphysics energy transformation performed by the middle ear to convert sound energy into mechanical vibrations of the ossicles and vibrational energy within the fluids of the inner ear.^1,2 Despite decades of research on the TM, the transformation of sound energy into ossicular mechanical vibrations through the TM is not yet fully understood. Accurate description of the TM response to sound can help better determine its mechanical properties, diagnose middle-ear diseases (e.g., otitis media), evaluate the outcome of middle-ear rehabilitation surgeries (e.g., tympanoplasty), and quantify the performance of hearing devices (i.e., hearing aids and hearing protection)

The dominant factors governing the TM acoustomechanical energy transformation are its shape and spatially varied mechanical properties (due to complex fibrous structure and variations in thickness), as well as the prestress and the load of the ossicular chain.^3–15 Previous research has shown that its complex orthotropic fiber arrangement and frequency-dependent viscoelastic behavior^4,5,10,16 make the TM response to sound even more complex. For example, one of the complexities of TM response is that different regions of the TM (e.g., the regions tied to the manubrium of the malleus, the four quadrants of the pars tensa, or the pars flaccida) respond differently to acoustic excitation.^15,17 Therefore, full-field-of-view measurements (i.e., simultaneous measurements over the entire surface of TM) are needed to quantify the response of the entire TM and describe how different regions of the TM interact with each other to conduct sound to the ossicular chain.

Over the past decade, various holographic methodologies were developed to quantify TM functional parameters such as excitation induced displacements, shape, and thickness.^11,18–28 One of the main challenges in holographic measurements of TM acoustically induced nanometer-scale motions is the high sensitivity to physiological motions (micrometer to submillimeter) in live ears (motions due to respiration, heartbeat, muscle tremor, etc.). To solve this problem, high-speed digital holographic (HDH) methods were developed to measure the TM motions produced by brief acoustic transients (click response with duration of $<5\mathrm{ms}$).^17,26,29,30 The TM’s responses to such brief broadband excitations (0.2 to 22 kHz) are rapid enough that they are little affected by slow in vivo physiological motions ($<20\mathrm{Hz}$),^17,26,30,31 and recording at high sampling rates ($>40\mathrm{kHz}$) provides broadband information about the TM’s responses. Preliminary results on live animals show the applicability of the HDH as a quantitative full-field vibrometry tool to study TM mechanics in live ears for hearing research and the diagnosis of middle-ear disorders.³¹

Previous models and measurements of the three-dimensional displacements of the TM support the approximation of the TM by a Kirchhoff–Love thin-shell in which the dominant motions occur along the direction normal to the TM surface.^22,32–35 To accurately quantify the surface-normal displacements of the TM using a one-dimensional (1-D) HDH displacement measurement system (i.e., 1-D displacement sensitivity vector using single illumination and observation vector), knowledge of the TM shape and orientation are required.^32,35,36 Furthermore, with measurement systems placed relatively close to the TM inside the confined volume of the ear canal, we expect there to be variations in the holographic-sensitivity vector across the TM surface, and precise knowledge of these sensitivity vectors is required for accurate characterization of TM surface motions. This paper describes a high-speed holographic shape measurement method that is incorporated into our 1-D HDH displacement measurement system to enable near instantaneous ($<200\mathrm{ms}$) measurements of shape and displacements. Knowledge of TM shape and orientation allows us to quantify geometrically defined variations in the sensitivity of the holographic displacement measurements in an endoscopic optical configuration through the computation of a separate holographic sensitivity vector for each point on the TM. The coupling of shape and sensitivity vector magnitude and direction allows precise determination of the surface-normal displacements from the 1-D interference phase measurements.

2. Methods

2.1. Multiple Wavelength Holographic Interferometry Shape Measurement Method

2.1.1. Design constraints

The shape measurement system was developed with the aim of satisfying the following functional requirements:

• •Utilize the imaging optics of the current HDH displacement measurement system.
• •Measure shape of the TM within the small triangulation volume of the ear canal.
• •Utilize a series of shape measurements of varied resolution to account for local variations in TM shape and orientation.
• •Provide sufficient axial and lateral shape resolutions for accurate surface normal calculations.
• •Keep the time intervals between displacement and shape measurements short ($<100\mathrm{ms}$ with a total measurement time $<200\mathrm{ms}$) to preserve the same coordinate system despite slow motions of the sample.
• •Measure with high speed to reduce the effects of in vivo physiological motions.

2.1.2. Principals of multiple wavelength holographic interferometry shape measurement

The multiple wavelength holographic interferometry (MWHI) shape measurement is based on the principle of variations in wavelength of illumination with a constant optical path length (OPL) (i.e., a stationary sample).^37–42 As shown in Fig. 1, the interference phase produced by differences in two illuminating wavelengths generates contours of constant depth with shapes determined by the geometry of the object and measurement system. The relation between the interference phase $\mathrm{\Delta }\gamma$, the constant OPL, and wavelengths at two hologram exposures [i.e., $\lambda$ and ${\lambda }^{\prime }=(\lambda +{\lambda }_p\lambda )$, where ${\lambda }_p$ is the percent change in the wavelength between the initial and second exposure] is described as

$$\mathrm{\Delta }\gamma =\frac{2\pi }{\lambda }\mathrm{OPL}-\frac{2\pi }{{\lambda }^{\prime }}\mathrm{OPL}=\frac{2\pi }{\mathrm{\Lambda }}\mathrm{OPL},$$ (1)

where $\mathrm{\Lambda }$ is the synthetic or equivalent wavelength determined as

$$\mathrm{\Lambda }=\frac{\lambda {\lambda }^{\prime }}{|\lambda -{\lambda }^{\prime }|}=\frac{\lambda (\lambda +{\lambda }_p\lambda )}{|\lambda -(\lambda +{\lambda }_p\lambda )|}=\frac{\lambda (1+{\lambda }_p)}{{\lambda }_p}.$$ (2)

Determination of shape from the interference phase requires additional calibration efforts as the OPL represents a combination of the shape and orientation of the object within the optical and holographic configuration, where that configuration may impose spatial distortions (e.g., the curved contours in Fig. 1). The calibration transforms the interference phase $\mathrm{\Delta }\gamma$ onto a coordinate system where variations in $\mathrm{\Delta }\gamma$ define the shape of the sample along the camera axis ($z$ in Fig. 1).^41,43 The calibration can be done by several different methods.^41,43–46 We used a flat surface calibration method that includes subtraction of the measured interference phase of a flat surface (i.e., ${\mathrm{\Delta }\gamma }_{\text{flat}}$) from the interference phase $\mathrm{\Delta }\gamma$ to determine the calibrated interference phase (i.e., ${\mathrm{\Delta }\gamma }_{\mathrm{c}}=\mathrm{\Delta }\gamma -{\mathrm{\Delta }\gamma }_{\mathrm{flat}}$). To avoid accumulation of noise, the measured ${\mathrm{\Delta }\gamma }_{\text{flat}}$ is smoothed and filtered. The calibrated interference phase ${\mathrm{\Delta }\gamma }_{\mathrm{c}}$ is directly related to the surface $z$ component of point $P$ corresponding to the shape of the sample as

$$z_p(x,y)=\frac{\mathrm{\Lambda }}{4\pi }{\mathrm{\Delta }\gamma }_{\mathrm{c}}(x,y).$$ (3)

Fig. 1.

Schematic representation of contours of constant phase intersecting the sample in the MWHI shape measurements method. $h$ or “contour depth” is the distance corresponding to a $2\pi$ interference phase directly related to the synthetic wavelength.

2.1.3. High-speed multiresolution multiple wavelength holographic interferometry

As described later, the MWHI method has been designed to sweep the synthetic wavelength [$\mathrm{\Lambda }$ in Eq. (2)] allowing multiple measurements of shape in a short time. Alterations of $\mathrm{\Lambda }$ allow quantification of the shape with multiple resolutions: the smaller the synthetic wavelength, the higher the measuring resolution of the interferograms with higher sensitivity to fine surface details. Figure 2(a) shows how wavelength differences smaller than 0.3% of the base wavelength provides synthetic wavelengths that are orders of magnitude larger than the base wavelength $\lambda$ [the blue axis in Fig. 2(a)]. For example, for $\lambda =780\mathrm{nm}$ (the base wavelength used in this paper), ${\lambda }_p$ values of ($\lambda +0.2\mathrm{nm}$) to ($\lambda +2.3\mathrm{nm}$) produce a range of synthetic wavelengths from 3 to 0.3 mm [the orange axis in Fig. 2(a)].

Fig. 2.

The variations of $\mathrm{\Lambda }$ with respect to base wavelength $\lambda$ and wavelength difference percentile $\lambda p$ [Eq. (2)]. The left axis (in blue color) shows the normalized $\mathrm{\Lambda }$ by the base wavelength $\lambda$. The right axis (in orange color) is the synthetic wavelength conversion of the left axis for $\lambda =780\mathrm{nm}$.

The synthetic wavelength is the primary determinant of the shape measuring resolution in MWHI. However, there are two technical factors that constrain the usable range of the synthetic wavelength, and therefore limit the shape measuring resolution. These factors are: (i) the shape gradient (determined by curvature and depth of shape) and (ii) the spatial resolution of the imaging camera. The higher the shape gradient, the more interference fringes are accumulated over short distances and the higher the fringe density in the interferometric images captured by the camera. If the fringe density is higher than the spatial resolution of the camera, the fringes are no longer distinguishable, and no longer provide accurate estimates of the interferometric phase. An increase in the synthetic wavelength will reduce the number of accumulated wavelengths and reduce the fringe density. On the other hand, a too large synthetic wavelength will lead to extremely low fringe densities and interferometric phase differences that are too small to be distinguished.

The adjustability of the synthetic wavelength is particularly important in measurements of the shape of the TM with its angled orientation inside the ear canal. Figure 3(a) shows a typical “tent”-shaped TM and its normal angled orientation with respect to the long axis of the ear canal. The dimensions of the TM together with its shape and orientation require a sensor depth of field of $\sim 4$ to 6 mm and a field of view of about 5- to 7-mm diameter. Furthermore, as shown in Fig. 3(b), the shape gradient is not uniformly distributed across the field of view and there is usually a peak in the gradient near the center of the TM (close to the tip of the mallear attachment, in between the two blue dotted lines). Therefore, higher shape measuring resolutions are required to better describe the spatial gradients near that peak. Our use of a rapid series of shape measurements with regularly varied synthetic wavelengths allows us to choose, which wavelengths best describe the shape in different regions of the TM within the timing of a single measurement.

Fig. 3.

(a) Typical human TM shape and orientation dimensions (the TM has a tent-like conical shape 8 to 10 mm in diameter and 2 to 3 mm in height) and (b) the relation between the shape gradient [governed by angle $\theta$ shown in (a)] with respect to spatial resolution of the imaging sensor and the contour depth (governed by $\mathrm{\Lambda }$).

Using a new temporal phase sampling technique based on high-speed correlation interferometry, the interference phase was quantified multiple times during slow wavelength variations. Instead of performing instantaneous reference phase steps with fixed phase shifting value, where the actual phase shift is affected by the varying wavelength and motions of the sample, we imposed a series of rapid continuous $2\pi$ phase shifts in the reference arm. During a single 100-ms duration wavelength tune from ${\lambda }_1$ to ${\lambda }_2$ [Fig. 4(a)], 10 or more of these high-speed phase samplings (each of them of $500\mu \mathrm{s}$ duration) are performed. Two such samplings are used to compute the shape; the shorter the interval between the phase sampling, the more similar the wavelengths during the two samplings and the finer the synthetic wavelength increments.

Fig. 4.

Schematic representation of high-speed multiresolution MWHI shape measurement method. (a) An illustration of the temporal variation in the wavelength of illumination as it varies regularly between 1 and 2 over 100 ms. Each of the circled stem lines marks the initiation of a $500\text{-}\mu \mathrm{s}$ period of continuous phase shift and high-speed holographic measurement described in (b); (b) Illustrates one of the high-speed phase shifts with nearly stable wavelength at $\lambda i$. The dashed red line shows a 0 to $2\pi$ phase shift introduced into the reference path, and the fainter red rectangles illustrate the timing of images captured by the high-speed camera; (c) As in (b), the continuous red line shows the total real phase shift that results from the controlled wavelength variation, sample motions, and continuous phase shift applied to the reference path (dashed red line). The colored boxes show the camera frames that the later described correlation algorithm chose to define frames of desired total phase shift.

Figure 4(b) shows such a sampling driven by a continuous high-speed phase ramp applied to the reference arm ($>1\mathrm{kHz}$) that changes faster than the phase shifts induced by the wavelength tuning sweep ($<80\mathrm{Hz}$). However, as schematically shown in Fig. 4(c), while the effects of uncontrolled sample motions are smaller within the rapid phase sampling periods, they still make significant contributions to the total change in interference phase that can vary significantly from the shift imposed by the shifts in reference phase (i.e., red solid line versus the red dashed line). We have defined an automated phase correlation calibration algorithm, applied in postprocessing,⁴⁴ that specifies the sampled image frames with the desired change in interference phase (e.g., 0, $\pi /2$, $\pi$, and $3\pi /2$) during each high-speed phase ramp. Figure 4(c) schematically shows how the algorithm picked the desired phase shifted frames from the noisy phase-shifted frames.

This method significantly increased the speed of shape acquisition by removing the delays required for laser wavelength stability and eliminated the need for multiple lasers for high-speed shape measurement. Moreover, as the technique was applied throughout the slow (100 ms) wavelength tune, it defined the change in interference phase produced at a wide range of illumination wavelengths, which enabled precise shape measurements with a wide range of resolutions. As noted above, measurements at multiple resolutions allow definition of shape over a wide range of shape gradients.

2.2. Improved high-speed digital holographic Displacement Measurement Method—High-Speed Continuous Phase Sampling

We have previously developed and implemented HDH methods based on correlation interferometry that could “instantaneously” measure the full-field-of-view ($>\mathrm{200,000}$ points at 67,200 camera frame rate) transient displacements of the TM in response to impulsive acoustical and mechanical excitations.^17,26,31 This method, called $2+N$ high-speed correlation interferometry ($2+N$ HCI), is based on a hybrid spatiotemporal local correlation phase sampling approach, which allows quantification of the TM transient deformations by utilizing two reference frames, $I_{\mathrm{ref}}$, $I_{\mathrm{ref}+\pi /2}$, and $N$ consecutive deformed frames, ${(I_{\mathrm{def}})}_{i,i\in 1,2\dots ,N}$, recorded before and throughout the evolution of a high-speed event as shown in Fig. 5. The interference phase $\mathrm{\Delta }\varphi$ (i.e., due to sample displacements) at each kernel is determined as

$$\mathrm{\Delta }\varphi (x,y,t)={\tan }^{-1}\left[\frac{\rho [I_{\mathrm{ref}}(x,y),I_{\mathrm{def}}(x,y,t)]}{\rho [I_{\mathrm{ref}+\pi /2}(x,y),I_{\mathrm{def}}(x,y,t)]}\right],$$ (4)

where $\rho$ is the Pearson’s correlation coefficient for finite discrete sets having the recoded intensities as arguments and computed based on a spatial square kernel (e.g., $3\times 3\text{\hspace{0.17em}\hspace{0.17em}pixels}$) centered around each measurement point, $(x,y)$, of a pair of reference and deformed frames.²⁶ Note that zero-order terms of all correlated kernels (the summation of the intensity of reference and object beams) must be removed to accurately determine the interference phase $\mathrm{\Delta }\varphi$ using Eq. (4). In this paper, to distinguish between the phase in shape and displacement measurements, $\mathrm{\Delta }\varphi$ denotes the interference phase due to displacements, whereas $\mathrm{\Delta }\gamma$ represents the interference phase due to the shape.

Fig. 5.

Timeline of events during a set of displacement measurements. The blue line shows the phase shifter position. For phase quantification purposes, a continuous phase shift is performed prior to sample excitation with a stable phase during the sample response. The high-speed camera continuously captures reference and deformed frames for the desired duration. Using the automated correlation algorithm, we choose the reference frame with $\pi /2$ phase shift relative to the stable value. Note that the phase-shifted references have a phase that is negative compared with the baseline reference that results in change of the sign in Eq. (4), $\mathrm{\Delta }\varphi \to -\mathrm{\Delta }\varphi$.

The accuracy and precision of the $2+N$ method depend heavily on the accuracy of the $\pi /2$ phase shift in the reference frame. Therefore, to maintain resolution and precision of the phase shifts in the presence of in vivo physiological motions ($>100\mu \mathrm{m}$, $<20\mathrm{Hz}$), a single high-speed continuous phase sampling method [e.g., Figs. 4(b) and 4(c)] with a slower ramp (with 3-ms period) was used while the phase shifter stays at the $\pi /2$ position. As the reference in this case is negatively phase shifted, the sign of interference phase is changed ($\mathrm{\Delta }\varphi \to -\mathrm{\Delta }\varphi$). Figure 5 shows the sequence of events in a set of high-speed acoustically induced transient displacement measurement that starts by capturing a set of reference holograms during a continuous piezocontrolled phase shift followed by continuously capturing sample response holograms (i.e., $I_{\mathrm{def}}$) induced by a broadband acoustic click. More details of this method can be found in our publications.^17,26,31,44

2.3. Hardware Realization

In this study, shape and displacement measurements were made with separate lasers with different properties: A $780\pm 10\text{-}\mathrm{nm}$ tunable laser for shape measurements and a highly stable (linewidth 100 MHz) 532-nm laser for displacement measurements. As shown in Fig. 6, using two variable beam splitters (VBS), each laser beam is divided into reference and illumination beams with the same intensity ratio. To maintain the same observation vector for both measurements, the lasers are guided into the same illumination and reference axis using Dichroic mirrors (DM). The beam reflected from the sample (object beam) is combined with the reference using a wideband wedge to form an interference pattern on the sensor of the high-speed camera. Each laser is equipped with a mechanical shutter (MS) to switch rapidly between the sample illuminations.

Fig. 6.

(a) Schematic representation of the shape and displacements measurements setup and (b) photograph of the setup. VBSs are to balance the intensity ratio between the reference and object at different wavelengths and polarizers in the reference arm correct for the polarization mismatch caused by the VBS and polarizing wedge ($W$) beam combiner. The high-speed camera is Photron Fastcam SA-Z with a pixel peach of $20\mu \mathrm{m}$ and operates at 67.2 kHz ($512\times 512\text{\hspace{0.17em}\hspace{0.17em}pixels}$). MS, mechanical shutters; DM, dichroic mirror; PL, polarizers; BE-S, BEam-shaper (Anamorphic Prism); M, mirror; EL, expansion lens; IL, imaging lens.

2.4. Timeline of Combined High-Speed Displacement and Shape Measurements

Figure 7 shows the timeline of combined displacement and shape measurements with a total duration of about 150 ms. The transient displacement measurement (Fig. 5) is performed within 25 ms during illumination by the fixed-wavelength laser. Immediately after the displacement measurements, the piezo moves back to its original position, and the shutters are set to switch to the tunable laser (see Sec. 2.3 above for details of the two lasers) with a programmed wavelength variation suitable for multiresolution MWHI shape measurement that begins after another 25 ms. During the slow variation in the laser wavelength (which typically takes about another 100 ms, Fig. 7), a series of phase sampling triggers [Fig. 4(a)] are synchronized with a continuous ramp of the piezo positioner [Fig. 4(b)] to allow for quantitative phase sampling while the tunable laser slowly varies the laser wavelength. Both shape and displacement measurements are performed at 67,200 frames per second temporal resolution ($512\times 512\text{\hspace{0.17em}\hspace{0.17em}pixels}$ with exposure time of $3.9\mu \mathrm{s}$). The tuning range and the period between the phase samplings can be adjusted to account for different ranges of shape resolution and sample depths.

Fig. 7.

Flowchart and timeline of a set of combined HDH shape and displacement measurements.

2.5. Experimental Setup and Measurements on the Tympanic Membrane

A fresh (nonfixed) cadaveric human ear was used for these measurements (Fig. 8). The cartilaginous and boney parts of the ear canal were removed to provide a larger view of the TM. The middle-ear cavity was opened to check for ossicular normality and then closed again using silicone sealant. A small vent was created through the sealant to avoid the buildup of the static pressure within the middle ear cavity during the measurements. The TM surface was sprayed with a thin layer of white face paint to improve reflectivity. To ensure that the paint layer was sufficiently thin and uniform, a modified airbrush was set to generate a very fine mist while the sample was placed $>10\mathrm{cm}$ away from the airbrush. After painting, the manubrium (outlined) attached to the medial surface of the TM was still visible under ambient light [Fig. 8(c)]. The sample was excited by a broadband click (a $50\text{-}\mu \mathrm{s}$ square pulse sent to a loud speaker) and the sound pressure was measured by a calibrated high-frequency (HF) microphone positioned near the TM lateral surface [Fig. 8(a)].

Fig. 8.

Measurement methods for cadaveric human ears: (a) top view, (b) back view, and (c) photograph of the human cadaveric TM. The black contour shows the outline of the manubrium (the handle of the malleus embedded in the TM) and the umbo (the manubrial tip labeled with a U).⁴⁷

2.6. Determination of Surface-Normal Displacements along the Tympanic Membrane Surface

The displacements at each point of the moving surface in a holographic measurement of deformation give rise to the optical path length differences essential for interferometric measurements.⁴⁸ A complete description of displacement requires either measurements of motion in three directions (from which a calculation of magnitude and direction can be made) or knowledge of the direction of displacement. As the TM has a thin-shell structure with the dominant displacement component along the surface normal,³² we can eliminate the need for observations from different directions and determine the displacement direction using the shape information. As shown in Fig. 9, using knowledge of the shape and orientation, the surface normal ($\mathbf{n}$) and sensitivity vectors ($\mathbf{K}$) are computed at each point of the TM surface with respect to the displacement measurement coordinate system. This information is sufficient to determine the magnitude of the surface normal displacement as well as the surface normal displacement vector ($\mathbf{L}$)^44,47

$$|\mathbf{L}|=\frac{\mathrm{\Delta }\varphi }{\mathbf{K}·\mathbf{n}}=\frac{\mathrm{\Delta }\varphi }{|\mathbf{K}|\cos (\alpha )},\mathbf{K}={\mathbf{k}}_2-{\mathbf{k}}_1,\mathbf{L}=|\mathbf{L}|·\mathbf{n},$$ (5)

where $\mathrm{\Delta }\varphi$ is the motion-induced interference phase, $\mathbf{n}$ is the surface normal vector, $\mathbf{K}$ is the sensitivity vector, and the angle $\alpha$ is the angle between the two vectors, which varies at different locations on the surface of the structure. The cosine of angle $\alpha$ is directly related to the sensitivity of the holographic system, the smaller this angle the higher the sensitivity of the system; therefore, those positions on the TM with $\alpha =0$ have maximum sensitivity.

Fig. 9.

Schematics of optical path length difference of a thin-shell having dominant out-of-plane displacements. $\mathrm{\Delta }\varphi$ is the interference phase due to displacements, $\mathbf{K}={\mathbf{k}}_2-{\mathbf{k}}_1$ is the sensitivity vector, $\mathbf{L}$ is the displacements vector, and $\mathbf{n}$ is the surface normal vector at the point $zp$.⁴⁷

3. Results

3.1. Accuracy and Precision of the Shape Measurements

The accuracy and precision of the shape measurement were tested on a NIST traceable ball and a custom-made sample [Fig. 11(a)], respectively. To determine the shape accuracy, the sphere surface of the NIST ball is measured by the multiresolution MWHI method and fitted by least square optimization methods. The output of the least square algorithm is the center and radius of the fitted sphere.⁴⁹ The measurement accuracy is defined by the root mean squared error (RMSE) of the radii of all measured points with respect to the center of the fit compared with the nominal radius obtained from the NIST ball. Figure 10 shows the measured shape of the NIST traceable ball (radius $5\pm 0.025\mathrm{mm}$). The color map illustrates the spatial distribution of the errors on top of the measured shape compared with a sphere fit. The higher the accuracy of the measured shape the closer the dimensions to the actual shape and the lower the errors compared with the NIST nominal values. The error presented in Fig. 10 is the RMSE of the fitted sphere versus the radius of the ball from the NIST; the color map illustrates the spatial distribution of the differences between the fitted and actual sphere. The small RMSE value (0.045 mm) indicates a high accuracy of our shape measurements. Note that due to off-axis illumination (Fig. 6), one side of the NIST sphere has a shadow and the camera observes a nonsymmetric view of the sample compared with the fitted sphere with its center nearly on the camera axis.

Fig. 11.

The results of the shape measurements repeatability test. (a) a custom-made conical-shaped latex membrane to study the repeatability of the shape measurements, (b) calibrated wrapped modulo $2\pi$ phase corresponding to five consecutive measurements of the shape of the custom sample, (c) shape map of the sample from one of the measurements in (b), and (d) shape profiles along the dashed black line in (c) for all five iterations; represented by different colors.

Fig. 10.

Spatial distribution of errors between the measured radius and the NIST traceable gauge nominal radius.

The custom sample [Fig. 11(a)] was a deformed latex membrane that resembled the tent-like conical shape of the TM, and five consecutive shape measurements were made with fixed resolution to study repeatability. Figure 11 show the results of five consecutive shape measurements made within less than a minute in five consecutive fixed tunings of the laser. The mean of the standard deviation of the shape readings from all points across the field of view shows repeatability within $<17\mu \mathrm{m}$.

3.2. Human Tympanic Membrane Shape and Displacement Measurements

The shape and acoustically induced transient displacements of the human postmortem TM were measured by the combined multiresolution high-speed MWHI method and the high-speed $2+N$ HCI displacement measurements.

3.2.1. Multiresolution multiple wavelength holographic interferometry tympanic membrane shape measurements

Figure 12 shows the results of the multiresolution MWHI shape measurements performed on the human TM sample. Using the automated phase detection algorithm for each discrete set of phase samplings at different wavelengths [Fig. 12(a)], the frames corresponding to the phase shift of interest are chosen and the interference phase between the sets is calculated and saved in a database. Figure 12(b) shows how different combinations of the interference phase across the single wavelength tune can generate phase maps with a range of resolutions. To simplify the synthetic wavelength identification, a custom numbering was used in Fig. 12(b). The wavelength at each set is labeled by their timing increment (i.e., ${\lambda }_1$ is the wavelength for the set at $t_1$) and the synthetic wavelengths between any two sets are identified by the wavelengths used in its computation i.e., ${\mathrm{\Lambda }}_{\mathrm{1,10}}={\lambda }_1{\lambda }_{10}/|{\lambda }_1-{\lambda }_{10}|$. To show the multiresolution capability, a few phase maps with various synthetic wavelengths are also shown in Fig. 12(b). There are phase maps with excessive fringe density (e.g., ${\mathrm{\Lambda }}_{\mathrm{1,10}}$ where the innermost and next innermost fringes tend to merge) resulting in unwrapping errors, as well as maps where the phase change across the entire field of view is too small to produce a fringe (e.g., ${\mathrm{\Lambda }}_{\mathrm{5,6}}$).

Fig. 12.

Representative phase maps highlighting shape measurements at multiple resolutions: (a) timing of the triggers relative to the wavelength variation during the tune. The red line shows the theoretical change in the variable wavelength λ during a wavelength tune. The green, magenta, and blue lines show repeated measurements of $\lambda$ using a wavelength meter. The jumps observed in the wavelength readings are due to the low temporal resolution of the meter ($<80\mathrm{Hz}$); there are no jumps in the actual wavelength tuning. The black vertical lines show positions where a brief phase ramp was applied to measure wavelength and interference phase [Fig. 4(b)]. The measurements are numbered by their temporal sequence during the tuning ramp; (b) wrapped and unwrapped phase maps taken with selected arbitrary combinations showing various number of fringes across the sample; (c) the filtered wrapped phase map measured with optimum resolution; (d) wrapped phase map of panel (c) after calibration; and (e) measured 3-D shape of TM.

Using an optimization approach based on phase quality, an optimum resolution phase-map (with a maximum phase quality and maximum resolvable fringe density) was chosen from tens of shape measurements at varied resolutions (in this case, the phase map of ${\mathrm{\Lambda }}_{\mathrm{1,8}}$ in Fig. 12(b) was chosen). Figures 12(c) and 12(d) show the wrapped phase before and after the calibration procedure, as described in Sec. 2.1.2 using the flat fielding method. Figure 12(e) shows the shape after spatial filtering and masking to remove any surface discontinuities. The color bar in Fig. 12(e) quantifies the depth of the sample along the $z$-axis perpendicular to the plane of the flat surface used for the calibration.

3.2.2. Determination of surface normal and sensitivity vectors

Knowing the shape and orientation of the TM together with the coordinates of the observation/illumination points, the surface normal and sensitivity vectors are computed at each location on the TM surface. Figure 13(a) shows the spatial distribution of surface normals across the surface of TM. Figure 13(b) shows the variations of the surface normals with respect to the computed sensitivity vector across the red line shown in Fig. 13(a), illustrating spatial variations in angle $\alpha$ [Eq. (5)]. Using $\alpha$ and Eq. (5), a sensitivity multiplier map is computed for $\lambda =532\mathrm{nm}$ from the surface-normal and sensitivity vectors at each point, where the sensitivity multiplier converts the measured interference phase to the corresponding surface-normal displacement in micrometers. Figure 13(c) shows a two-dimensional map of the multiplier, which varies in space; such maps depend on the shape and orientation of the sample as well as the illumination/observation coordinates. Note that regions around the center of the TM show the largest differences in the orientation of the sensitivity and surface normal vectors (13B) and require the largest multipliers to convert the optical phase measured along the sensitivity vector to the magnitude of motion along the surface-normal direction.

Fig. 13.

(a) spatial distribution of surface normal vectors on a TM, (b) spatial distribution of surface normals (red arrows) as well as sensitivity vectors (black arrows) along the red line of panel (a) showing spatial variations of angle $\alpha$ (defined in Fig. 9) across the red line in (a), and (c) computed sensitivity multiplier map to determine surface-normal displacements from interference phase information.

3.2.3. Shape and acoustically induced transient surface-normal displacements of the tympanic membrane

With the sensitivity map, the acoustically induced TM transient surface-normal displacements are determined from the interference phase measured by the $2+N$ HCI method.³¹ The still images of Fig. 14(a) show the shape and surface-normal displacements at nine selected temporal instants from 1700 consecutive measurements (over 25 ms, @ 67200 FPS) during the TM transient response to a broadband acoustic click.

Fig. 14.

3-D shape and 1000 times exaggerated TM transient response surface normal displacements due to an acoustic click. This video consists of five sections: (a) the still image here shows nine temporal instances while the video shows up to 2.5 ms of the response. The manubrium of the malleus is shown with a black line and U represents the umbo location; (b and c) show the shape and displacements along the red and blue lines shown in (a); (d) the timeline of displacements at the intersection of the red and blue lines (umbo); and (e) modulo $2\pi$ phase map ($\mathrm{\Delta }\varphi$) of each frame (Video 1, MPEG-4, 10.9 Mb [URL: https://doi.org/10.1117/1.JBO.24.3.031008.1]).

The first four consecutive frames of the measurements in still image of Fig. 14(a) show the initial response of the membrane as the sound sets the entire surface of the TM into motion with essentially the same motion phase. We call this the “initiation” stage of the response, which can be used to determine TM stiffness, overall volume change, etc. After initiation, surface waves are seen to reflect from both the manubrium and the rigid TM annulus ring and form a complex superposition of traveling and standing waves across the TM (“surface wave superposition stage”). The TM response at this stage is temporally separated from the excitation, making it suitable for methods such as experimental modal analysis to determine mechanical properties, damping, etc. The last stage of the response includes the TM vibration at one of the primary mode shapes until the motions are completely damped out (the “dissipation stage”).

Figure 15 shows the time waveforms of three consecutive click excitations at six regions of the TM with respect to the microphone signal (shown as thick black line on region-1 panel of Fig. 15). The results illustrate how the shape can be used for decomposition of the response at different regions of the TM, such that the location of manubrium and umbo (region 1), pars-tensa (regions 2 to 5), the rim of the TM, and pars-flaccida (region 6) is clearly distinguishable. This decomposition is also visible as each region has different amplitudes of response and a different time constant or damping ratio. Figure 15 also shows repeatability of the displacement measurements as the three consecutive measurements are nearly on top of each other.

Fig. 15.

Time waveforms of six selected regions as shown in the top left panel. The sound pressure signal is shown only on region-1 panel. Note that the range of displacement axis is different at each region.

4. Discussion

4.1. Summary

In this study, a nondestructive noncontact high-speed holographic full-field measuring instrument was developed to enable nearly instantaneous shape and displacements measurements of the TM in $<200\mathrm{ms}$. This development can be used to obtain unprecedented details about the shape and motions of the TM in response to various types of stimulations, which can help better understand the function of the TM in the hearing processes. The shape measurement capabilities can account for spatial variations of the sensitivity vector along the TM surface due to its conical shape and angled orientation inside the ear canal. Meeting the design constraints (Sec. 2.1.1) of the shape measurements of the TM in one apparatus was not trivial. Many of the methods that have been previously developed for TM shape measurements are inappropriate. For example, methods such as structured light projection^35,50 require projection of fringes with sufficient triangulation volume inside the ear canal while others such as light field imaging⁵¹ require specialized optics and high-resolution sensors ($>10$ megapixels) incompatible with the current HDH imaging system. The shape measuring capacity was incorporated into the holographic displacements measurements without sacrificing any of its functionalities. Using the same illumination and observation optics and maintaining the same optical path for both measurements allowed the measured shape to be directly applied to surface-normal displacements computations without additional registration requirements.

4.2. Shape Measurements Advantages

The combination of shape and displacements measurements enables the determination of true surface normal displacements directly from interference phase data and makes the results independent of direction of observation. The knowledge of the shape of the TM advances the interpretation of the transient measurements, by enabling accurate measurements of speed and trajectory of the traveling waves, as well as TM shape parameters, such as curvature and area. The shape of the TM has many clinical applications in the diagnosis of middle-ear diseases, including evaluations of tympanoplasty and the manufacture of personalized grafts adapted with the shape of the TM.^52–54 Precise and accurate shape and displacement measurements enable the generation of a TM database to normalize different TM transient responses based on their shape parameters.

4.3. Significance of Determination of Surface-Normal Displacements

To demonstrate the significance of the conversion to surface-normal displacements and correction for the spatially varying sensitivities of the holographic system, the corrected surface-normal displacements are compared with the displacements estimated by assumption of maximum uniform sensitivity without shape compensation: i.e., when $\alpha$ is zero in Eq. (5).

Figure 16 shows the spatial distribution of difference in the magnitude of the absolute displacement in nanometers at the $382\text{-}\mu \mathrm{s}$ instant between the two cases. At regions around the umbo, where the angle $\alpha$ is the largest, the sensitivity of the holographic system is minimum and the differences between the uncorrected and surface-normal motion magnitudes are large ($>100\mathrm{nm}$ or $>40\%$ of the displacements). Such large differences at the areas critical to understanding the TM response show the importance of the application of shape for interpretation of the displacements. These errors can be greatly exaggerated as gradient and variations of angle $\alpha$ are larger in an endoscopic condition, where the source is close to the TM and the sample is oriented with larger angles with respect to the camera axis.

Fig. 16.

A map of the absolute displacement difference between the corrected surface-normal displacements and the displacements estimated by the assumption of uniform sensitivity without shape compensation for the set of human postmortem TM measurements. The spatial distribution of the differences along the red and blue lines is shown in the bottom and right panels, respectively, showing $>100\text{-}\mathrm{nm}$ differences. The manubrium of the malleus is shown with a black line and U represents the umbo location.

5. Conclusions and Future Work

A high-speed multiresolution holographic shape measurement method is developed to measure TM shape parameters (i.e., curvature and dimensions) as well as to compensate for geometrically induced spatial variations of the sensitivity vector. These improvements allow measurements of in situ and near instantaneous ($<200\mathrm{ms}$) TM displacements and shape. The use of a fixed (i.e., $\lambda =532\mathrm{nm}$) stable laser for displacement measurements allows accurate measurements over the 1 nm to near $1\mu \mathrm{m}$ range and the shape measurements (using the tunable laser) provide synthetic wavelengths that vary between 0.3 and 3 mm [Fig. 2(a)] with shape measuring sensitivity from $50\mu \mathrm{m}$ up to about a millimeter range. The new system enables the rapid sequential activation of a high-speed multiresolution MWHI shape measurement and high-speed holographic displacement measurements, enabling shape and displacement measurements within a fraction of a second to help reduce the effects of in vivo physiological motions. Furthermore, the knowledge of shape and orientation allows us to compute the motion normal to the TM surface and makes the displacements measurements independent of direction of observation. Such independence facilitates the comparison of measurements before and after the specimen relocation. The results also help better interpret the TM transient response either by accurate measurements of speed and trajectory of the traveling waves or by use of finite element⁵⁵ or experimental modal analysis.⁵⁶ Finally, the TM shape parameters, such as curvature and area, have direct clinical diagnostic values that can be used for rehabilitation surgeries as well as development of personalized grafts and artificial TMs.^35,36,52 Future work includes the analysis of the data gathered from more human TM samples and development of miniaturized optical head to enable in vivo measurements inside the ear canal.

Biographies

Payam Razavi is a postdoctoral fellow at Worcester Polytechnic Institute. He received his MSc degree in mechatronic engineering from Sharif University of Technology, Iran, in 2010, and his PhD in mechanical (manufacturing) engineering from WPI in 2018. His current research interests include high-speed holography, optical metrology, and hearing mechanics.

Haimi Tang is a PhD candidate in mechanical engineering at Worcester Polytechnic Institute. He received his BS and MS degrees in mechatronic engineering from Beijing Institute of Technology and the University of Southampton in 2010 and 2012, respectively. His current research interests include high-speed holography and digital image correlation.

John J. Rosowski is the Eliasen professor of Otolaryngology at the Harvard Medical School. His principal research interest includes sound transmission from the environment to the inner ear.

Cosme Furlong is an associate professor of mechanical engineering, electrical and computer engineering, robotics engineering, and physics, working on mechanics, design, optical metrology, and miniaturization engineering.

Jeffrey T. Cheng is an assistant professor of otolaryngology at the Harvard Medical School. His research interests are the relationships between structure and function in normal and pathological middle ears, as well as processes that damage, and techniques to repair the middle ear.

Disclosures

The authors have no relevant financial interests in this article and no potential conflicts of interest to disclose.