Spatial Segmentation for Laryngeal High-Speed Videoendoscopy in Connected Speech

Abstract

Summary: Objective.

This study proposes a new computational framework for automated spatial segmentation of the vocal fold edges in high-speed videoendoscopy (HSV) data during connected speech. This spatio-temporal analytic representation of the vocal folds enables the HSV-based measurement of the glottal area waveform and other vibratory characteristics in the context of running speech.

Methods.

HSV data were obtained from a vocally normal adult during production of the “Rainbow Passage.” An algorithm based on an active contour modeling approach was developed for the analysis of HSV data. The algorithm was applied on a series of HSV kymograms at different intersections of the vocal folds to detect the edges of the vibrating vocal folds across the frames. This edge detection method follows a set of deformation rules for the active contours to capture the edges of the vocal folds through an energy optimization procedure. The detected edges in the kymograms were then registered back to the HSV frames. Subsequently, the glottal area waveform was calculated based on the area of the glottis enclosed by the vocal fold edges in each frame.

Results.

The developed algorithm successfully captured the edges of the vocal folds in the HSV kymograms. This method led to an automated measurement of the glottal area waveform from the HSV frames during vocalizations in connected speech.

Conclusion.

The proposed algorithm serves as an automated method for spatial segmentation of the vocal folds in HSV data in connected speech. This study is one of the initial steps toward developing HSV-based measures to study vocal fold vibratory characteristics and voice production mechanisms in norm and disorder in the context of connected speech.

INTRODUCTION

Laryngeal imaging is an essential tool in clinical voice assessment in order to observe and evaluate the laryngeal structure and function during voice production.^1–4 The current laryngeal imaging technique that is widely used in clinics is videostroboscopy.^5–7 Using an endoscope coupled with a stroboscopic light in videostroboscopy, video recordings of the laryngeal structures can be obtained. Although videostroboscopy is used during connected speech, it can only capture gross laryngeal adjustments while the functional assessment of vocal fold vibration using videostroboscopy is limited to sustained vocalizations only.^8–11 In addition to this constraint, videostroboscopy has limitations in assessment of aperiodic vibrations of the vocal folds, which commonly are present in voice disorders.^12,13 The recent advancement that has come with coupling flexible fiberoptic endoscopes with laryngeal high-speed videoendoscopy (HSV) serves to overcome these limitations by capturing the true vibrations of the vocal folds since it does not rely on periodicity of the acoustic signals.^12–14 Thus, HSV is a powerful tool that enables the study of vocal function and its underlying physiological and physical mechanisms with high temporal resolution in ways not possible with videostroboscopy.^12,13,15,16

HSV provides a unique possibility to measure vocal fold vibrations with high temporal resolution during connected speech.^17–21 Vocal abnormalities often reveal themselves, not in sustained phonation, but in running speech.^22–27 Utilizing HSV in connected speech is therefore likely to serve as an invaluable tool, providing detailed information about the intra-cycle vibratory characteristics of the vocal folds, as well as nonstationary events, such as voicing onsets, offsets, and breaks.^{17,20,21,28–30} However, HSV recordings of connected speech result in big datasets, where navigating through this huge amount of information is not feasible in clinical settings. Thus, there is a need to develop automated techniques and algorithms for such big-data analysis and extraction of clinically relevant information from the vocal fold vibrations.

The analytic extraction of the edges of the vibrating vocal folds in connected speech would facilitates the development of HSV-based measures of vocal fold vibration. Previously, we presented a study on using HSV in connected speech, which developed a fully automated temporal segmentation method to extract the timestamps for onsets and offsets of vocalizations and epiglottic obstructions of the vocal folds.¹⁷ Additionally, in that study, we estimated a temporal correlate of the glottal area waveform (GAW).¹⁷ The need for developing a dedicated spatial segmentation method is driven by the need to segment analytically and register over time the edges of the vocal folds for more accurate measurement of glottal attack time, glottal offset time, GAW, vocal fold contact, phase asymmetry and other essential spatio-temporal characteristics of vocal fold behavior. Several approaches have been developed and applied for spatial segmentation of the glottal area in HSV during sustained phonation.^31–35 The main methods for extracting the glottal area from the HSV data are the region growing,^32,36,37 histogram thresholding,^33,38 level sets,^39,51 and active contour modeling approach.^34,35,50 These methods were used for HSV analysis in isolated sustained vowels. Most of the developed methods are not fully automated and require visual inspection of the data and some manual analysis.^{34,36,39–41} Region growing and histogram thresholding are both vulnerable to image noise and homogeneity.³¹ The level set method can accurately estimate the glottal cycle only if the vocal folds fully close during adduction and it is also prone to noise.³⁹ The active contour modeling approach, however, is less sensitive to noise and intensity inhomogeneity in images, can be initialized anywhere, even across boundaries, and efficiently preserves global line shapes.^42,43 Hence, this approach is a promising technique for spatial segmentation of the glottal area.⁴⁴

The active contour modeling method can be used to dynamically locate the contour of the desired image features, such as the edges of the glottis, and therefore, can be used for the GAW measurement. The active contour (aka snake) is a spline, which deforms based on certain energy minimization rules to capture the glottal edge. The active contour modeling approach has been employed to detect the glottal edges (1) spatially in each HSV frame: this method is based on using closed loop snakes for each individual HSV frame^32,34,50; (2) temporally in HSV kymograms: this method uses two open curve snakes for detection of the right and left vocal folds^35,45; and (3) spatio-temporally: the open curve snakes are used for glottal edge detection in the HSV kymograms across different cross sections of the vocal folds and the extracted edges are then registered back to each HSV frame.³⁵ The existing studies for spatial segmentation of glottal edges were performed on HSV data obtained during sustained vowels and not on connected speech.

The goal of this study is to present a spatial segmentation method based on the active contour modeling approach to provide an analytic representation of the edges of the vocal folds and perform a GAW measurement in HSV data during connected speech. The proposed computational algorithm in this work is fully automated and performs a spatio-temporal analysis of the HSV data to detect the glottal edges during vocal fold vibrations.

METHODS

Data acquisition

The HSV data were recorded from a vocally normal female (38 years of age) who did not have any history of voice disorder, while reading the “Rainbow Passage.” The examination was done at the Center for Pediatric Voice Disorders, Cincinnati Children’s Hospital Medical Center and was approved by the Institutional Review Board. A custom-built color HSV system with 4,000 frames per second (fps) and 249 μs integration time was used for the data acquisition. The recording length was 29.14 s (116,543 frames in total) with HSV image resolution of 256 × 256 pixels. The recording system included a FASTCAM SA-Z color high-speed camera (Photron Inc., San Diego, CA) coupled with a 3.6-mm Olympus ENF-GP Fiber Rhinolaryngoscope (Olympus Corporation, Tokyo, Japan), and a 300-W xenon light source, model 7152A (PENTAX Medical Company, Montvale, NJ). The camera had a 12-bit color image sensor with sensitivity of ISO 20,000 and 64 GB of cache memory divided into two 32-GB partitions. The selected zoom lens adapter had a focal distance of 45 mm in order to provide the optimal pixel representation and dynamic range. The distance of the endoscope to the vocal folds was selected to ensure that despite the active maneuvers of the larynx during connected speech, the vocal folds always fall within the field of view of the endoscope during the recording. The recorded HSV sequence was saved as an uncompressed 24-bit RGB AVI file and then analyzed.

The camera used for the data collection had a native resolution of 1,024 × 1,024 pixels at 20,000 fps. However, for the purposes of the study, the resolution was set to 256 × 256 pixels, which at the chosen speed of 4,000 fps provided for up to 30 seconds per partition to record the reading of the Rainbow Passage. The selected frame rate was shown to be clinically acceptable for voice assessment.⁴⁶ Moreover, using 256 × 256 pixels provided the optimal balance between the image resolution, camera frame rate, duration of the recording, and the light sensitivity necessary for this data collection.

Data analysis

Prior to developing the automated spatial segmentation method, several image processing steps were implemented. The first step was to apply a temporal segmentation method¹⁷ to extract the vocalizations in the HSV recording. The second step was to apply an algorithm for motion compensation to align the vibrating vocal folds across the frames during each vocalization.^20,47 The HSV kymograms were extracted next at all different cross sections of the vocal folds. The spatial segmentation was then performed to analytically describe the vocal fold edges in the kymograms using an active contour modeling approach. Subsequently, the segmented vocal fold edges were registered back to the HSV frames and the GAW was computed across the frames for each vocalization. Each of the aforementioned processing steps is elaborated in the following subsections.

Temporal segmentation

A temporal segmentation algorithm we developed previously¹⁷ was applied to the HSV data to automatically extract timestamps of the vocalized segments (vibratory onsets/offsets) and epiglottic obstruction of the glottis. The unobstructed vocalized segments of the HSV recordings were selected for further processing.

Motion compensation

A previously developed denoising algorithm along with a motion compensation technique,²⁰ were applied to each vocalized segment of the HSV recording. The goal of this step was to capture the spatial location of the vibrating vocal folds in a motion window across the frames. The motion compensation was mainly performed to overcome the problem with spatio-temporal misalignment of the vocal folds due to laryngeal maneuvers during running speech and the movements of the endoscopic tip relative to the laryngeal tissues over time. After detecting the location of the vibrating vocal folds across the frames, the HSV frames were cropped based on the location of the center of the motion window, designed in an ellipse shape (see²⁰ for complete description of the motion window). The size of the cropped frames was based on the horizontal and vertical second moments of inertia after applying the motion window to each HSV frame and removing the irrelevant noise and tissues. This step was essential for extracting the kymograms inside a rectangular window that enclosed the vibrating vocal folds in each frame.

Kymogram extraction

The HSV kymograms were extracted for the frames between 25 ms before the onset of vocalization and 25 ms frames after the vocalization offset for each vocalization to ensure the full pre-, post-, and periphonatory phases are included. For each vocalized segment, the first kymogram was extracted at the mid-line passing through the vibrating vocal folds inside the motion window. The spatial segmentation was then applied to the kymogram to detect the glottal edges. The kymograms were extracted at different cross sections of the vocal folds between the anterior and posterior commissure and the spatial segmentation algorithm, explained below, was applied to each kymograms.

Spatial segmentation

The spatial segmentation algorithm was based on an active contour modeling approach. The goal of the active contour technique is to extract a specific feature in the image. In this work, the feature of interest was the glottic edge in the kymogram image. To do so, a line or an active contour (also called a snake) should be drawn close to the glottal edge. The snake should be close enough to the edges of interest, otherwise, it might be attracted to other edges present in the kymograms. In each kymogram, a pair of open-curve snakes (left and right active contours) were used to capture the left and right edges of the glottis. The initialization of the snakes’ locations is an important step toward an accurate edge detection. Therefore, an algorithm for an automated snake initialization was developed prior to applying the active contour modeling approach.

Automated snake initialization.

The initial snake should pass through the center of the glottis. The snake initialization was performed for each kymogram n_i using the first moment of inertia, denoted by M₁(y, n_i) and calculated as follows⁴⁸:

$$M_1\left(y,n_i\right)=\frac{{\sum }_{x=1}^{K_w}{\sum }_{y=1}^{K_h}I\left(x,y,n_i\right)y}{{\sum }_{x=1}^{K_w}{\sum }_{y=1}^{K_h}I\left(x,y,n_i\right)},$$ (1)

where x and y correspond to the spatial coordinate of a pixel, I(x, y, n_i) is the kymogram image, K_w is the kymogram image width, which is the number of frames in the kymogram, and K_h is the kymogram image height in pixels. The green channel is less prone to noise and therefore, this channel was used for the calculation of the first moment. Since the first moment of inertia determines the center of brightness, the kymogram was inverted to find the center of darkness (ie, the centroid of the glottis). The first moment of inertia was then adjusted to find a more accurate line passing through the center of the glottis. This adjustment step was done because the first moment of inertia was vulnerable to the high level of noise in the present HSV data and resulted in a not fully accurate line. The adjustment was performed using the gradient information of the kymogram image around the moment line. The gradient was computed in each kymogram vertically (along the left-right dimension of the image) with a step size of 10 pixels. The negative gradient can represent the glottis since the pixels located inside the glottis area are darker than the neighboring pixels outside the glottis. Hence, the summation of the negative gradient values was calculated horizontally (across the time) at seven different lines just above and below the moment line. The pixel with the largest summation value of the negative gradient for each frame was considered the center point for that frame. The center point for different frames formed the snake initialization line across the frames, which was used for initialization of the right and left snakes.

Active contour modeling.

The active contour modeling was applied to the kymograms to detect the left and right edges of the vocal folds. The snake deformed spatially in the kymogram image according to certain energy optimization rules. The optimization process aimed to minimize an energy function, which was a summation of an external and internal energy function. The external energy was derived from the spatial gradient of the image to highlight the desired feature (ie, glottal edges). The internal energy controlled the shape and deformation of the snake.⁴⁴ The snake spline can be parametrized as a vector v(s) = [x(s), y(s)], where s ∈ [0,1]. The objective energy function that needed to be minimized was defined as:^44,49

$$E=\underset{0}{\overset{1}{\int }}\left[E_{int}(v(s))+E_{image}(v(s))\right]ds,$$ (2)

where E_int is the internal energy function and E_image is the external image function. The internal energy function (E_int) of the spline, acting as a soft constraint to make the shape of the snake smooth, was computed by:

$$E_{int}(v(s))=\frac{1}{2}\left[\alpha (s){\left|v^{\prime }(s)\right|}^2+\beta (s){\left|v^{\prime \prime }(s)\right|}^2\right],$$ (3)

where v′(s) and v″(s) are the snake first and second derivatives, respectively; α and β are two weights included to adjust the elasticity and rigidity of the snake, respectively, which control the snake shape. The two weights α and β were considered constant and were set to 0.1 and 0.03, respectively.

The image energy function (E_image) counterbalances the internal energy and was given by:

$$E_{image}(v(s))=-|\nabla I(x,y){|}^2,$$ (4)

where ∇ I (x, y) is the spatial gradient of the kymogram image.

The solution for Eq. (2) was based on discretization of the energy function. The finite difference method was used to approximate the first and second derivatives in the internal energy function. The internal energy function, given in Eq. (3), can be rewritten as follows, after the discretization:

$$E_{int}\left(v_i\right)=\frac{1}{2}\left[\alpha {\left|v_i-v_{i-1}\right|}^2+\beta {\left|v_{i+1}-2v_i+v_{i-1}\right|}^2\right],$$ (5)

where v_i refers to the ith snaxel; the snaxels are the vertices that make up the snake spline. The discretization of the image energy function, given in Eq. (4), yielded:

$$E_{image}\left(v_i\right)=-{\left|\nabla I\left(v_i\right)\right|}^2.$$ (6)

By discretizing the total energy function, given in Eq. (2), the following equation was derived, which can be considered as a dynamic-programming problem:⁴⁹

$$E_{total}=\sum _{i=1}^n\left[E_{int}\left(v_i\right)+E_{image}\left(v_i\right)\right],$$ (7)

where n is the total number of snaxels. The total number of snaxels refers to the total number of frames included in the kymogram. The solution to the above dynamic programming problem generated a sequence of functions ${\left\{S_i\right\}}_{i=1}^{n-1}$ with one unknown variable v_i, where S_i is the optimum value function. The S_i functions for several snaxels are shown in Eq. (8). At each S_i, a minimization was conducted over the v_i variable, where v_i is a state variable and can be assigned m possible values. The value of m refers to the number of pixels in the neighborhood of the snaxel that the algorithm searches to find the optimal v_i. In this study, the value of m was set to 5. Note that in Eq. 8, E_int(v_n) = E_int(v_(n–1), v_n, v_(n+1)).

$$\begin{gathered} S_1\left(v_1\right)=\underset{v_1}{min}\left[E_{int}\left(v_0,v_1,v_2\right)+E_{image}\left(v_1\right)\right], \\ {S}_2\left(v_2\right)=\underset{v_2}{min}\left[S_1\left(v_1\right)+E_{int}\left(v_1,v_2,v_3\right)+E_{image}\left(v_2\right)\right], \\ {S}_3\left(v_3\right)=\underset{v_3}{min}\left[S_2\left(v_2\right)+E_{int}\left(v_2,v_3,v_4\right)+E_{image}\left(v_3\right)\right], \\ ⋮ \\ {S}_n\left(v_n\right)=\underset{v_n}{min}\left[S_{(n-1)}\left(v_{(n-1)}\right)+E_{int}\left(v_n\right)+E_{image}\left(v_n\right)\right]. \end{gathered}$$ (8)

And in the general case,

$$\begin{gathered} S_i\left(v_i\right)=\underset{v_i}{min}[\sum _{i=1}^{i-2}\left(E_{int}\left(v_i\right)\right)+\sum _{i=1}^{i-1}\left(E_{image}\left(v_i\right)\right)+E_{int}\left(v_{i-1}\right) \\ +E_{int}\left(v_i\right)+E_{image}\left(v_i\right)]. \end{gathered}$$ (9)

The image gradient ∇ I was calculated in the vertical direction along the left-right dimension of the kymogram image in each frame with the step size of 10 pixels. The gradient was computed for each kymogram to signify the glottal edges, where the intensity changed rapidly. Accordingly, the movement of each snaxel for the left and right active contours was limited to the vertical direction. The positive and negative gradients were computed next. The positive gradient of the kymogram was obtained by assigning the negative gradient pixels value of zero. The negative gradient of the kymogram was obtained by assigning the positive gradient pixels zero. The positive gradient was used in the image energy function when searching for the left vocal fold edges while the negative gradient was used to search for the right vocal fold.

To find the snakes that captured the left and right vocal fold edges, the discretized energy function in Eq. (9) was solved for i = 1, 2, 3, …, n. The minimization of the S_i functions was done recursively, and the snake was updated during each loop until the position the snake remained almost unchanged through the minimization procedure. As such, during each loop and at each snaxel, the value of the S_i function was numerically calculated for the column-wise snaxel’s five neighboring pixels. The neighboring pixel that led to a minimum total energy was considered as the updated value of v_i. After updating all the v_i values, the next loop started until the algorithm converged and the optimum snake spline was found. The convergence of the algorithm depended on an error function. This error function was defined as the sum of squared of the difference between the calculated snake in the current loop and the previous loop. The algorithm converged when the error became smaller than or equal to 1.

The vocal fold edges were analytically described at each vocalized segment of the “Rainbow Passage”. The anterior and posterior commissure were detected automatically. That is, if no glottal area was present in the extracted kymogram, the active contour method returned a snake similar to the snake initialization line, at which the location of the posterior/anterior commissure was specified.

GAW

The detected edges in each frame of the kymograms were registered back to the HSV frames. Subsequently, the glottal area was measured for each frame. The glottal area was defined as the number of pixels in the enclosed area between the detected right and left edges of the vocal folds. The GAW was measured as the glottal area values across the frames for each vocalization.

RESULTS

Using the temporal segmentation algorithm, the timestamps for all the vocalized segments of the HSV connected speech recording were extracted (except for the segments with epiglottic obstruction). Subsequently, the motion compensation was applied to each vocalized segment of the “Rainbow Passage” to capture the location of the vibrating vocal folds across the frames. The result of applying the motion window to three individual frames during five different vocalizations are depicted in Figure 1a–e. Each row in Figure 1 shows three frames for a different vocalization between the following frame numbers: 40,505–41,255 (Figure 1a), 42,975–43,815 (Figure 1b), 84,281–84,891 (Figure 1c), 103,942–104,577 (Figure 1d), and 109,548–110,363 (Figure 1e). The individual frame numbers are shown on top of the HSV frames for each figure panel. The figure shows that the implemented motion window captures both the location and size of the vibrating vocal folds in different frames.

FIGURE 1.

HSV frames along with the applied motion windows for three different frames at five different vocalized segments (panels (a–e)).

The kymograms were extracted for each vocalization of the “Rainbow Passage” after aligning the vocal folds across the frames using the motion compensation method. Examples of the extracted kymograms at the medial section of the vocal folds are shown in Figure 2. The kymograms for five different vocalizations are shown in panel a–e between the same frame numbers as in Figure 1. You can see the onset and offset of phonation in each kymogram and that the darker glottal area is almost on a straight line across the frames for each kymogram.

FIGURE 2.

HSV kymograms at the medial section of the vocal folds for five different vocalized segments (panels (a–e)). The L and R on the y-axis refer to the left and right vocal folds, respectively.

A cropped frame is shown in Figure 3a, and the four dashed lines show the location of the intersections for which the kymograms are plotted in panel b–e. The first moment of inertias in solid magenta lines and the adjusted moment lines in dashed cyan lines, which correspond to the center of the glottis, are shown for each kymogram. All the kymograms are extracted from the same vocalized segment during a sustained phonation over 200 frames (frame #40,768–40,967). As seen, the adjusted lines span through the glottis center more accurately in comparison with the moment lines. The adjustment of the moment line becomes more critical when the image becomes blurrier as in Figure 3d and e.

FIGURE 3.

Cropped HSV frame (panel a) and four kymograms at the four intersections of the vocal folds (panel b–e). The first moment of inertia (solid magenta line) and the adjusted snake initialization line (dashed cyan line) are shown for the four kymograms between frames 40,768 and 40,967. The L and R on the y-axis refer to the left and right vocal folds, respectively. For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.

After the snake initialization, the active contour algorithm was applied to the kymograms. The upper and lower snakes (active contours) corresponding to the left and right vocal folds for the four kymograms in Figure 3 are shown in Figure 4. The number of frames shown in the figure is 546, between frame 40,585 and 41,165, including voicing onset, vocal folds vibration, and voicing offset. Two zoomed-in image segments are included in Figure 4 to better visualize the performance of the algorithm. As seen, the active contour approach detects both the left and right vocal fold edges (solid green line and yellow line, respectively) at different cross sections – providing an analytical representation of the glottal edges. Moreover, the algorithm is able to capture the edges before the phonation starts and after the phonation ends.

FIGURE 4.

Kymograms between frames 40,585 and 41,165 at four different cross sections of the vocal folds (panel a–d) along with the upper and lower active contours (solid green line and solid yellow line, respectively) corresponding to the left and right vocal folds. Two zoomed-in image segments are included to better visualize the performance of the algorithm. The L and R on the y-axis refer to the left and right vocal folds, respectively. For interpretation of the references to color in this figure legend, the reader is referred to the Web version of this article.

After detecting the edges of the vocal folds in the kymograms at different cross sections of the vocal folds across the frames, the edges were registered back to the HSV frames. Video S1 demonstrates the detected edges of the vocal folds during the first vocalization of the “Rainbow Passage” (692 frames presented during the voicing onset, sustained phonation, and voicing offset). The top figure in the video shows the sequence of HSV between frames 43,030 and 43,721. Both of the left and right edges of the vocal folds, extracted using the active contour modeling approach, are plotted on each HSV frame in green and yellow color, respectively. The corresponding GAW is shown in the bottom figure of the video. The red star moving along the GAW corresponds to the glottal area in the HSV frame in the top figure of the video. As can be seen in the top figure, the registered edges of both the left and right vocal folds synchronously track the true spatial location of the vibrating vocal folds across the HSV frames.

The computed GAW for five different vocalizations of the “Rainbow Passage” between frame numbers 40,505–41,255, 42,975–43,815, 84,281–84,891, 103,942–104,577, and 109,548–110,363 is shown in Figure 5a–e. The change in the glottal area can be seen during the onset and offset of phonation and the sustained phonation in each panel. The moments with GAW values of zero are associated with the vocal folds contact.

FIGURE 5.

The computed GAW at five different vocalized segments of the “Rainbow Passage.”

DISCUSSION

The temporal segmentation was successfully utilized to determine the vocalized segments of the “Rainbow Passage.” Subsequently, the motion compensation precisely located the vibrating vocal folds across the frames during the extracted vocalizations. After applying the motion compensation and determining the location of the vibrating vocal folds, digital kymograms were successfully extracted at various intersections of the vocal folds. The vibrating vocal folds always appeared on an almost straight line in the extracted kymograms, which was necessary for a better performance of the spatial segmentation algorithm.

The automated snake initialization tool was successfully developed and accurately located a line that spanned through the glottis center in the extracted kymograms. The adjusted moment line was introduced because the results revealed how vulnerable the first moment of inertia line was toward the noise in the kymogram image. Based on the results, the proposed adjusted moment line demonstrated a better estimation than using only the first moment of inertia line for finding the center of the glottis. Obtaining an accurate snake initialization line was a necessary step toward a better performance of the active contour modeling algorithm and its convergence.

The active contour technique was successfully implemented for the kymograms of the vocalized segments of the “Rainbow Passage.” The application of active contour method allowed the analytic representation of the vocal fold edges at different cross sections of vocal folds from the anterior to the posterior commissure. The performance of the algorithm exceeded the challenging quality of the HSV images. From 76 vocalizations of the “Rainbow Passage”, the visual observation of the detected edges and the HSV kymograms showed that the algorithm’s error was not more than one pixel for 67 vocalizations (88%) deeming it successful for precise detection of the glottis boundaries. Due to dim lighting in some of the frames in the kymograms of the other 9 vocalizations, the active contour modeling was not able to find the glottal edges. The visual observation also could not determine the glottal edges due to the lighting issue for these kymograms.

This study showed the feasibility of automatic edge detection using the proposed method in challenging data obtained using a color high-speed camera. Color images are preferred over monochrome images by clinical specialists since color images allow them to evaluate the health of the tissues while observing and evaluating the vibrations of the vocal folds. Therefore, this study used a color high-speed camera to demonstrate that the proposed algorithm can be applied to color images. Moreover, the goal of this study was to develop an algorithm that works for the most challenging conditions (ie, color images). Despite the edge uncertainties on the color images, the paired active contour was not attracted to erroneous edges and it maintained optimal rigidity. Since this work shows the robustness of the spatial segmentation method in the most challenging conditions due to color images, this method applied to HSV data recorded using a monochrome high-speed camera provides significantly better segmentation results. This is due to the inherently higher dynamic range (image quality) of monochrome images and to the significantly more accurate representation of the gradients of the edges in the image. The application of this method on monochrome HSV recordings, and the evaluation of its accuracy and reliability on a larger sample is part of an ongoing study, which will be presented in future.

After registering back the segmented edges in the kymograms to the HSV frames, based on the visual inspection of the results, the implemented active contour modeling successfully detected the edges of the vibrating vocal folds across the frames during each vocalized segment. The algorithm had some difficulty to find the edges before the start of the oscillation and after the oscillation of the vocal folds stopped and the registered vocal fold edges infrequently deviated from the actual location near the anterior commissure. This could be due to the deviation of the motion window from the location of the vocal folds prior to the beginning of vocal fold oscillations, also after the oscillations stopped. Due to this deviation, the vibrating vocal folds were not exactly in the center of the motion window. However, this was not an issue in this study since the main goal was to capture the vocal fold edges during vocalization. In future, we will aim to develop a hybrid edge detection model using machine learning to also detect the edges of the vocal folds while there is no oscillation. This would be useful in studying laryngeal tissue maneuvers during pre- and postphonation in connected speech.

The GAW was computed successfully using the segmented vocal fold edges for each vocalization of the “Rainbow Passage” across the frames. The calculated GAW was able to capture not only the fluctuations in the glottal area during vocal folds vibrations, but also the exact area fluctuations in the onset and offset of the oscillations. Even though the development of spatial segmentation of the vocal fold edges during connected speech is important mainly to enable the measurement of glottal attack and offset times, as well as various measures related to vocal fold contact and asymmetry during spontaneous speech production, the GAW is a common measure used in sustained phonations that is also made available through the proposed segmentation method, thus provided in this article for comparative purposes.

CONCLUSION

The application of HSV to study the vocal folds function during running speech is a breakthrough, which can lead to improved understanding of phonatory processes in connected speech and to the advancement of clinical voice assessment protocols. Developing automated image processing techniques for the analysis and interpretation of the HSV recordings is crucial. That is, automatic spatial segmentation is an important step to enter a new era of precision laryngeal imaging measurements. This work is necessary for automatic extraction and measurements of the relevant vocal fold vibratory features. The temporal segmentation and motion compensation were able to determine the vocalized segments and locate the vibrating vocal folds for the entire “Rainbow Passage.” The developed automated algorithm for spatial segmentation successfully captured the vocal fold edges across frames for each vocalization, which allowed for accurate computation of GAW at each frame.

In this study, the promising performance of the developed algorithm was shown for a vocally normal participant. This warrants future research using the developed method for the analysis of HSV data for a larger number of vocally normal participants and patients with voice disorders, which will help generalize the developed algorithm in norm and disorder, and address inter- and intra-subject reliability. Furthermore, HSV-based measures will be developed from the measured GAW for norm and disorder.