Isolating fast and slow flows in three-dimensional fluid dynamics

Abstract

In complex flows where motions at distinctly different speeds co-exist on the same plane, current optical flow (OF) methods preferentially track the slower component due to the small-motion assumption in OF algorithms. We overcome this limitation by balancing light intensity and applying per-pixel time-based high-pass filtering. We demonstrate this method using schlieren video of air motion generated by speech, where fast speech air flow co-exists with slower buoyancy flows from body heat–discrimination impossible with conventional OF. Our method extracted air flow patterns several centimetres from the mouth as a function of spoken sounds, with peak velocities in the English “pa” sound agreeing with CFD simulations. The method allows analysis of spatial and temporal variation in air velocity at distance from the speaker’s lips, made by different sounds (phones). To do this, kymographs (space-time velocity plots) were generated and analysed using Generalized Additive Mixed-effect Models (GAMM). At 30 cm from the lips, statistical models showed higher predictive power (: 2.64%→7.57%), reduced complexity, and improved fit (fREML: 6.8E7→7.3E6, 89.3% residual reduction) compared to uncorrected data. The method is generalizable to any 3D optically accessible flow with motion predominantly in the image plane containing co-located high and low speed motions.

Introduction

Many 3D flows feature components of distinct velocities, with distributions of velocity which may not overlap each other, and which lie in the same plane. In this case, 2D imaging methods cannot separate the fast and slow flows by choice of image plane^1–3.

Particle Tracking Velocimetry (PTV) and Particle Image Velocimetry (PIV)⁴ require careful choice of correlation window and image time separation (delta t) to select one flow or the other in such a case. PIV and PTV require trackable particles to be present. In some cases tracer particles cannot be added.

Where there are sufficient variations in refractive index of the flow, schlieren imaging allows visualization of refractive index features. Schlieren (“streifen”, German for “streaks”)⁵, are regions where fluid density varies, bending light, and so under the right conditions these schlieren make invisible flows visible⁶. In most cases the variation in refractive index variations are caused by variation in temperature or composition, and thus the refractive index variations can be calculated from the Gladstone-Dale relationship (Eq. 1).

1

 where n is the index of refraction, ρ is the mass density of the optical medium, and K is the Gladstone-Dale constant, which depends on wavelength and the composition of the medium.

The most common method for extracting flow velocities from schlieren image sequences is optical flow field analysis^7,8. However, optical flow field analysis relies on brightness being consistent from frame to frame, the motion being smooth, and the displacement during the time interval between frames being small. Inconsistent brightness causes tracking to go off course, and violation of the small displacement assumption causes high-speed elements in the velocity field to be ignored as the optical flow field analysis preferentially tracks the slower features.

Here we present a combination of two approaches, light intensity balancing and per-pixel time-based high-pass filtering, that jointly allow the discrimination between fast and slow flows which lie in the same plane, and without the need for seeding. Our method is expected to work with any 2D image sequence of flows where the flows to be discriminated are co-planar, and have velocity magnitude distributions which do not overlap. (In our example, speech-driven air flow features are much faster than buoyancy-driven air flow features caused by body heat, and both lie in the sagittal plane.)

Unlike tracer particles, schlieren change shape and size. Tracking schlieren in a sequence of images requires that the tracking algorithm is tolerant to changes in size and shape, and the frame rate is sufficient that these changes are small from frame to frame.

Here we perform our methods on an example composed of high-speed video recordings of schlieren caused by warm air expelled from the mouth during speech, co-existent with slower buoyant flow due to the speaker’s skin temperature. The schlieren were illuminated with a pinhole source of green light reflected from a parabolic mirror. Image contrast was controlled with a knife-edge. Our example met none of the optical flow analysis assumption criteria. As a result, each assumption violation needed to be mitigated to allow proper conversion of speech air flow schlieren into accurate velocity information via optical flow field analysis. We do this with the processing techniques described below. To show that the technique gives valid velocities, we compare the measurements of the velocity caused by speaking the English “pa” sound to CFD simulations of the same sound. Comparison is done over a range of distances from the mouth. With the confidence this gave us in the results, we go on to express the air motions as a function of time for various sounds as kymographs. We then analyse these statistically using Generalized Additive Mixed-effect Models (GAMM) to demonstrate that the GAMM-decomposition on data processed with our method is more predictive across our set of participants than using uncorrected data. The example provides proof-of-concept for velocity-based separation of components of any transparent three-dimensional flow.

Consistent brightness by light intensity balancing

The classic optical flow algorithm assumes that the illumination of the flow field is constant over time and over the entire field of view. In mirror-based schlieren, this requires that the mirror’s surface has uniform reflectivity, and the light source and mirror must stay perfectly still. Neither condition was met by our apparatus, due to blemishes in the 70 year old mirror, and vibrations in the lab floor. To resolve these assumption violations, we normalized the image intensity by: converting all images to greyscale; inverting the first image (making a negative); and averaging each of the subsequent images (which remain non-inverted i.e. positive) with the initial image (Eq. 2).

2

 

where is the intensity-balanced form of frame i, is the raw frame i, is the raw first frame, and A is a frame consisting of all pixels at maximum value.

This corrected for underlying imperfections in the mirror’s reflectivity, and for potential imbalances in light intensity related to tiny shifts in the camera position as the floor moved. This technique balances any aberrations but reduces recording headroom (the amount by which a signal can exceed the average or normal operating level without undergoing clipping or distortion).

This did not limit image quality because changes in brightness due to schlieren did not come close to exceeding the remaining headroom. The technique also addresses any potential reflectance variations (Lambertian surface abnormalities) as it eliminated the effects of differences in regional reflectance in our mirror surface.

We then used the OpenOpticalFlow toolset to apply whole-frame and Gaussian filter illumination correction to our light intensity balanced data⁹. The OpenOpticalFlow toolset then used a Horn-Schunck estimator⁹ for an initial estimate in order to resolve the aperture problem (where estimates of motion direction are unlikely to be accurate unless the edges of the moving object are captured).

Smooth motion

This coarse motion estimate obtained with Horn-Schunck was then refined with a Liu-Shen estimator for an improved motion estimate⁸. The Liu-Shen estimator allows velocity estimation of a fluid object that changes shape over time. The combination of both estimators allowed estimations of velocity for both large and small flow eddies. This Liu-Shen estimator, combined with frame rate of 250 frames per second, allowed our data to meet the smooth motion (temporal coherence) assumption. The frame rate was fast enough, at 4 milliseconds per frame, that changes to flow eddies were small enough that the Liu-Shen estimator could work around the remaining shape changes to track velocity accurately.

We resolved violations of the spatial coherence assumption with our previous light intensity balancing, along with OpenOpticalFlow’s Gaussian smoothing filter to remove noise in the visual data⁹, and lastly with a pyramid approach to analyze motion at different scales, capturing both large and small movements⁹. This multi-scale analysis simultaneously allows resolution of the small motion constraint. This resolution is especially important because we needed to remove slow-moving buoyancy-driven body heat air flow from our data to allow optical flow field analysis to capture only the faster moving structures produced from the speech air flow itself.

Per-pixel time-based High-pass filtering

However, the output of optical flow field analysis would still be dominated by the slow-moving buoyancy-based body-heat flow features as these naturally meet the small motion constraint. To remove these buoyancy-based body-heat flows, we applied per-pixel time-based high-pass filtering after the light intensity balancing and before optical flow field analysis. We applied a Butterworth high-pass filter to the time series of intensity of each pixel. A 4th order filter with 8 Hz cut-off was chosen, being the minimum necessary to remove clearly visually identifiable buoyancy based motion. This motion is easy to see as it appears across the whole surface of the mirror, including in locations where there cannot be speech air flow. Butterworth filtering was selected due to its simplicity, speed, flat response, and ability to add padding before and after speech motions. As each pixel covers 0.5 mm of the object plane, then when including the Nyquist limit, the filtering removes real or artifact-based motions at 0.008 m/s or less. The slowest speech flows were observed to be 1.2 m/s, giving at least 2 orders of magnitude separation.

Materials and methods

Our new method for improving optical flow field analysis is here used to analyse schlieren speech recordings to separate air flow features generated by speech from body heat-driven motion. The procedures for obtaining the schlieren video were initially devised for a study comparing audio, visual, and speech air flow with and without face masks¹⁰. The complete data processing methods were also presented at the 21st International Symposium on Flow Visualization (ISFV21)¹¹.

Participants

The University of Canterbury’s Human Research Ethics Committee (HREC) approved the study on October 27, 2022 and the method complied with H(R)EC 2020/43 - Amendment 1. Recruitment began on October 28, 2022, our first participant was recorded on November 15, 2022, and both recruitment and participant recordings ended on December 16, 2022.

We recorded audio and schlieren from 18 participants, 13 of whom provided data for the English data used in this study (4 female, 8 male, 1 gender-diverse, ages ranged from 24-60).

We asked participants three questions about their hearing based on Noble’s 3-question assessment¹²: 1) “Do you have difficulty with your hearing? (yes or no)”. 2) “Do you have difficulty following TV programs at a volume others find acceptable? (no, slight, moderate, or great)”. 3) “Do you have difficulty having a conversation with several people in a group or in a noisy environment? (no, slight, moderate, or great)”. None answered yes to question 1, but 5 of the English and 3 of the te reo Māori speakers reported difficulty in understanding speech in noisy environments, and 1 speaker (English and te reo Māori) reported slight difficulty understanding TV at normal volume.

All 13 spoke English natively, from childhood: 12 spoke New Zealand English (NZE), 1 Canadian English (CE). Seven (7) were mono-dialectal English speakers, (2) more spoke two or more dialects of English, and the remaining four (4) were multilingual.

All participants provided written informed consent and agreed to publication of their images. No participants were minors.

Setup and materials

The single-mirror schlieren system consisted of a green LED light source with a pinhole (Edmund Optics Inc., Barrington, NJ, USA), and a 400 mm diameter, 3.5 m focal length parabolic mirror, a knife edge on a linear stage with a micrometer adjustment, an 18-400mm f/3.5-6.3 zoom lens (Tamron HB028, Saitama City, Japan) and a Photron SA5 camera (Photron, Tokyo, Japan, RGB, 1024x1024 px, 250 fps), as seen in Fig 1. The resulting spatial resolution was 450-500 microns (μm) per pixel, depending on slight adjustments between recording sessions. The field of view of the schlieren images was 400 mm and included the complete outline of participant’s face. Both the mirror and the Photron camera were levelled and positioned on optical benches with tripod mounts. The Photron’s bench had a linear positioning stage for lateral adjustment (horizontal plane, motion perpendicular to the line of sight). The schlieren line of sight was parallel to the speaker’s coronal plane, their face was inside the field of view on the right, and they faced the left edge of the field of view.

Fig. 1.

Schematic of schlieren and video setup, to scale in a 0.1 m grid. Image obtained from Fig. 1, Derrick et al.¹⁰, used by permission following STM permissions guidelines.

Simultaneously, a Redlake Motion Pro X3 high speed camera faced the speaker and captured their facial motions.

A condenser microphone headset (Beyerdynamic TG H55c) attached to a USB PRE 2 microphone phantom power pre-amplifier captured audio recording.

The audio and video systems were connected to the same Windows machine (Windows 10, 16 GB memory, 1 GB SSD). Audio was recorded using Audacity (Audacity team, 3.0.2). Schlieren video was recorded using the Photron FASTCAM viewer software (PFV 4). Facial video was recorded using the Redlake MotionPRO X viewer software. Audio presentation was controlled using custom software written in LabVIEW (NI, Austin, Texas, USA), activation of cameras was controlled manually through their respective capture software tools.

The English stimuli were: “1: The beige hues on the waters of the loch impressed all 2: Including the French queen 3: before she heard the symphony again 4: Just as young Arthur wanted”. The phrase was divided into four parts, as indicated by the numbering. This sentence is a pangram, containing most or all of the individual phones of English, depending on dialect. The sentence therefore provides examples of the full range of turbulent, laminar, fast and slow air flow features used in English. The analysis presented in this paper focuses on the first phrase only for brevity’s sake. Setup and materials were first described in Derrick et al.¹⁰.

Procedure

Each participant was seated at a table and given a pair of small form-factor welding goggles (AS/NZS 1338.1 certified shade 5 filter goggles) to wear. The goggles protected against the two bright red lights used to illuminate the face of the speaker for simultaneous video recording (not used in the analysis reported here). The participant wore a head-mounted microphone.

We instructed the participants to focus on the forward camera to stabilize their head and allow video recording of their facial motion during speech. LabVIEW software was used to control the flow of data collection such that the participant would hear a recording of the phrase part that they were to repeat. The experiment controller then started recording with both cameras and the microphone, and then gave a verbal instruction to the participant to speak. The participant then spoke. This process was repeated for each of the four phrase parts, constituting one block of recording. Each English participant recorded 3 blocks of 4 phrases, taking approximately 1.5 hours. These procedures were first described in Derrick et al.¹⁰.

Audio data processing

The audio recordings were clipped to provide 500 ms of padding at the beginning and end. The audio recordings were then labelled and transcribed in PRAAT¹³ with narrow phonetic transcription applicable to each audio recording. The audio recordings were also labelled and transcribed in PRAAT¹³ with wide phonemic transcription, applicable to all of the audio recordings regardless of dialect or idiolect. In the wide transcription, consonants all have shared phonetic labels, and the vowels were transcribed with Wells lexical sets¹⁴. The durations of each segment, and each recorded phrase, were averaged across all the speakers to produce Procrustean fitted Textgrids to assist later analysis. The audio files and schlieren images were visually aligned within 1 frame using alignment of auditory release bursts and air flow at the mouth. The audio and visual files were aligned using lip motion patterns.

Schlieren data processing

Using a custom-made zsh shell script running ffmpeg¹⁵ and imagemagick¹⁶, all mirror images were initially converted to greyscale (See top row, Fig. 2).

Fig. 2.

Staged schematic illustrating methods for applying grayscale, light intensity balancing, and high-pass filters.

For light intensity balancing, the first mirror image was converted into a negative image. For each subsequent frame, the initial negative and relevant positive frames were averaged together by intensity such that the new image was composed of 50% of the intensity of the initial negative image, and 50% of the current positive image. The image was then cropped to include only the circular mirror.

Pixels outside the mirror area were converted to 0% alpha-blend to make them transparent, and the figures were saved as portable network graphic (PNG) files (see middle row, Fig. 2). As schlieren recording is inherently analogue with slight errors from recording to recording, the exact position of the mirror in the source image was carefully assessed by hand for each recording, and the relevant information on position and pixel-to-millimeter conversion was stored in a CSV (comma-separated value) spreadsheet.

For high-pass filtering, the light intensity balanced images for each frame were loaded into a three-dimensional matrix in R¹⁷. Each pixel position was filtered with 4th order Butterworth high-pass filtered using the filtfilt command in signal¹⁸ with 8 Hz cutoff (bottom row, Fig. 2).

To obtain velocity information, the high-pass filtered images were processed using Open Optical Flow⁹. This tool processes optical flow, frame-to-adjacent-frame, which was then integrated to produce velocity information and then saved as greyscale PNG files (See bottom row, right-hand-side, Fig. 2).

Validation

Peak velocities, computed by our schlieren optical flow processing method, were compared to smoke visualization experiments and CFD measures describing of English ‘pa’ from Derrick et al.’s 2009 article¹⁹, along with new, refined CFD simulation (See Appendix 1).

All methods agree at 30 cm from the mouth. At nearer distances, the data measured for this paper shows higher velocities than the 2009 smoke flow visualisation measurements. The grey band in Fig. 3 represents a 95% confidence interval based on the standard error of the peak velocity at each distance. At 30 cm, the confidence interval is comparable in magnitude to the confidence intervals shown in the GAMM results (Figure 8), though the latter are adjusted for adjacency information inherent in the GAMM fitting process. This is likely because smoke visibility, even under bright lights, is limited in the high velocity regions as fluid shear and turbulence disperses the smoke, making it harder to detect. The present results agree well with the 2009 CFD and present CFD at all distances. Those CFD simulations use initial conditions based on sensible estimates of intra-oral pressure. The CFD velocities come from tracking the pressure peak, which is expected to move with the temperature feature which causes the schlieren artefact tracked in the present results.

Fig. 3.

Validation showing comparison of the peak velocity of the ’pa’ sound from the present optical flow processing method applied to the present schlieren data (red); 2024 LES CFD simulations showing a particle run (orange); 2009 RANS CFD simulations (yellow)¹⁹; and 2009 experimental data with velocities measured from smoke flow visualization at 500 fps (green) and 2000 fps (blue)¹⁹. Grey shading around the red optical flow curve indicates the 95% confidence interval, calculated as ±1.96 × standard error of peak velocity measurements across participants, using ggplot2’s default geom_smooth confidence band (²⁰).

Kymographs of the spatio-temporal variation in air velocity for different speech sounds

Listeners predominantly use hearing to sense speech, but at very close ranges skin perception of speech air flow has been shown to influence speech perception^1,21. Schlieren is an effective way to analyse the air velocity in speech as a function of both time and distance from the speaker. Distance is important as different speech air flows (sometimes called pseudo-sounds) travel at different speeds, so air motion from fast, late sounds can pass slow, early sounds. Kymographs are an effective way of presenting both the temporal and spatial changes in air velocity. A kymograph is essentially a 3D plot of spatio-temporal changes in a parameter. Time is on the horizontal axis, one spatial coordinate (here, radial distance from the speaker’s lips) is on the vertical axis, and the dependent variable (here, air velocity magnitude) is plotted as an intensity on these axes.

Statistical analysis can then be performed on these kymographs for the spoken phrases, to determine whether the spatio-temporal relationships are consistent across all speakers tested (statistical significance). Only robust spatio-temporal patterns might actually alter speech perception during continuous speech, contra previous failed attempts that used both an incomplete understanding and incomplete reproduction of speech air flow outside the mouth²².

To construct the kymographs, first, for each utterance, the position of lip opening at the beginning of speech is identified, then all the points at a given set distance from the lips are identified (see Fig. 4), and each frame of video is placed in linear order from 500 milliseconds before the start to 500 milliseconds after the end of the phrase. These form a graph of time on the x-axis, the arc position of each equidistant point on the y-axis, (see the equidistant arcs in Fig. 4) and an example resulting graph in Fig. 5)

Fig. 4.

Close-up example of arcs of set distances from the lips overlaying an example of schlieren data that has been light intensity balanced, high-pass filtered, and contrasted using an orange-blue diverging colour lookup table.

Fig. 5.

Image showing kymograph of participant 6, first repetition of “The beige hue on the waters of the loch impressed all”, at 30 cm from the lips. Top row = optical flow of greyscaled images. Middle row = optical flow of light intensity balanced images. Bottom row = optical flow of light intensity balanced and high-pass filtered images.

Kymographs were then extracted by taking velocity values along arcs set at 5 cm intervals from 5 to 30 cm from the lips (as visually identified at the point of separation of the midline vermilion borders, as seen in Fig. 4. The kymograph arcs were adjusted based on the shape of the mirror to only include portions of the arc for which schlieren recordings were made. Kymographs were saved as PNG files.

The greyscale kymographs were then loaded into R, given a y-axis arc-position using the information from mirror- and lip-position recorded earlier. The time axes of the kymographs were normalized using their recorded phrase duration, fitted to the average phrase and phone duration information obtained from TextGrid data. These data were analyzed statistically using Generalized Additive Mixed-effects Models (GAMM). GAMMs are non-linear statistics that allow us to examine curved surfaces.

The GAMM formula used for each phrase is shown in Eq. 3:

3

 

where BAM refers to the fast version of the GAM function provided as part of MGCV²³, used for very large datasets. Here value refers to the air flow velocity stored as a greyscale value in the PNG files. The x value refers to non-dimensional time as a percentage of duration of the given phrase. The y value refers to the arc-position (altitude) from the lips. The subject refers to the participant code. The k refers to the number of knots used in the statistical analysis. This number is much higher than the default 10, but was chosen to make sure any changes of velocity can potentially show variance across each recorded phone. The formula te(x, y, k=30) is a tensor field that includes main and interaction effects - this is the result shown in each topographical figure, and is used primarily to show the pattern of air flow velocity as opposed to showing well-defined regions of significant difference. The formula s(x, y, subject, bs = “re”) is the random-effects formula designed to account for differences between participants. While more complex random-effects formulas are available, this one reduces the impact of dialect differences amongst our diverse participant population while still being computationally tractable.

Rho estimates, or the best estimated autocorrelation coefficient at a specific lag, were then obtained from the fast GAMM results, and the autoregression start was defined as 500 milliseconds before speech onset time (the first line of each kymograph). The results of the GAMMS were output using GGPLOT2, and presented using an orange-blue diverging gradient to highlight high vs. low velocity information²⁴. Our GAMM plots provide three-dimensional information on speech air flow in space (x and y axis) and intensity (orange-blue diverging gradient). Our GAMM plots encode statistical significance roughly at each black topographical line such that each section separated by one black line faster (blue-shifted) or slower (orange-shifted) air flow are significantly different from each other with an alpha at or better than 0.05.

Note that complete processing times for data analysis take about 1 day using a 2024 Macbook Pro M2 Max (32 GB memory) laptop. The techniques used can all be multithreaded for fast computation on clusters.

Generalized data processing procedure

This specific method can be applied to nearly any case requiring separation of flow components by velocity. Simply record the relevant flow image sequences, converting to greyscale if recorded in colour, apply the light intensity balancing technique, apply the desired filter, perform optical flow on the output, and optionally validate against alternative filters as needed. Validation along the lines described here would include running otherwise identical GAMM analyses on each filtered set of data and comparing the outputs both visually (qualitatively) to ensure reasonable observations have not been erroneously excluded, and quantitatively using the compareML function²⁵.

Results

To illustrate visually our techniques, we present a reverse ablation of one token of the optical flow field analysis of our schlieren images. The top row of Fig. 5 shows a contrast-enhanced kymograph of the first recording of participant 6 saying “The beige hue on the waters of the loch impressed all.” at 30 cm from the lips following the method shown in Fig. 4. The middle row shows a similar kymograph of the same recording that has been light intensity balanced. The bottom row shows a similar kymograph of the same recording that has been both light intensity balanced and high-pass filtered. Note that the top image is hard to read, with only two regions of slightly higher air flow velocity visible at the “t” in “waters” and the “p” in impressed. The second row shows improved clarity, and the bottom row shows clear and easily identifiable separation of regions of fast air flow in multiple positions.

Statistical analysis of optical flow data

To illustrate the power of our techniques, we present a reverse ablation of optical flow field analysis of schlieren images, first using raw data in a state-of-the-art tool OpenOpticalFlow⁹. We follow this with the use of OpenOpticalFlow’s optical flow field analysis of the same data after it has been light intensity balanced as described above. Lastly, we use OpenOpticalFlow’s optical flow field analysis on the same data after both light intensity balanced and high-pass filtered. The results are visually and statistically compared to identify whether each method improves the quality of optical flow field analysis.

Example: OpenOpticalFlow-only optical flow field analysis

We begin with optical flow field analysis of schlieren data processed from our raw recordings (converted to greyscale). The results can be visually seen in Fig. 6. The top-left of the image shows a greyscale image of a frame of a schlieren recording for subject 6, repetition 1, frame 1405 (S6, R1, F1405). The top center-left image shows optical flow field analysis of the same frame, made easier to see in the image by maximizing contrast and minimizing brightness using Photoshop. The top center-right image shows arcs at 5-35 cm distant from the lip opening in 5 cm increments. The top-right image shows a kymograph (2D graph of time on the x-axis and air flow at 30 cm from the lips along the arc in the top center-right of Fig. 6) of speech air flow starting from 0.5 seconds before speech onset to 0.5 seconds after, with a scale showing % through speech, in this case, the phrase “the beige hue on the waters of the loch impressed all”. These figures clearly show considerable noise in the data, particularly at the edges of the mirror (Fig. 6 top center-right), which obscures any unique air flow information that could be gleaned from the kymograph (Fig. 6 top right).

Fig. 6.

Image showing optical flow field analysis of raw schlieren data using OpenOpticalFlow⁹. Top left: greyscale of schlieren recording of subject 6, repetition 1, frame 1405 (S6, R1, F1405), 400x400 millimeters (mm). Top center-left: Optical flow field analysis of the same frame. Top center-right: arcs indicating 5-35 cm from lips in 5 cm increments of the same frame. Top right: kymograph of air flow velocity over time, indicating the start and end of the phrase. Bottom: GAMM of optical flow field analysis of schlieren recorded from an arc 30 cm from the lips of all the speakers saying ‘the beige hue on the waters of the loch impressed all’.

The bottom portion of Fig. 6 shows a generalized additive mixed-effects (GAMM) model of all participant recordings of the same phrase, presented as a topographical map of air flow velocities. The x-axis is time as a percentage of the average of all the phrase duration audio signals (3.296 seconds), with word labels and wide phonemic transcription for each speech sound. The y-axis is the arc position of the point measured in relation to the center of lip opening. The graph should be thought of as a topographical map, with darkest blue representing nearly 0 meters/second (m/s) velocity, and the darkest red representing almost 9 m/s velocity.

The peak air flow velocities in this chart are as high as 8+ meters/second (m/s), appear in multiple locations in the chart, and are far higher than the 1.2 m/s (see Fig. 12¹⁹) to 2.5 m/s (see Fig. 15¹⁹) from previous direct recordings and simulations of speech air flow velocity at 30 cm from the lips. These numbers are higher than expected because of the apparent but spurious velocity resulting from: Imperfections on the 70-year-old mirror used in this research; the room lighting changing due to the 50 Hz electric frequency in the room; and the minor vibrations in the floor of the second-floor location in the building in which the schlieren measurements were made.

The GAMM 3D-smooth term (which is used to produce the 3D graph surface at the bottom of Fig. 6) (Ref.df = 848.1, F = 891.9, p < 2e-16, fREML score = 6.817E7) has an adjusted R-square of 0.0264, accounting for 2.64% of the data. These observations combine to indicate that the air flow velocities are highly inaccurate, produced mostly because of violations of optical flow field assumptions.

Example: Light intensity balanced optical flow field analysis

In contrast, the optical flow field analysis of the schlieren data processed from light intensity balancing our recordings are much cleaner. There is still noise in the data, as seen in the air flow example (Fig. 7 top left), and in the optical flow field analysis example (Fig. 7 top center-left). Looking at the kymograph, it is easy to see rising bands of buoyancy-related air flow throughout the kymograph (Fig. 7 top right).

Fig. 7.

Image showing optical flow field analysis of light intensity balanced schlieren data using OpenOpticalFlow⁹. Top left: greyscale of light intensity balanced schlieren recording of subject 6, repetition 1, frame 1405 (S6, R1, F1405), 400x400 mm, shown in high contrast. Top center-left: Optical flow field analysis of the same frame shown in high contrast. Top center-right: arcs indicating 5-35 cm from lips in 5 cm increments of the same frame. Top right: kymograph of air flow velocity over time, indicating the start and end of the phrase. Bottom: GAMM of optical flow field analysis of schlieren recorded from an arc 30 cm from the lips of all the speakers saying ‘the beige hue on the waters of the loch impressed all’.

Nevertheless, the data are visibly cleaner, as seen in Fig. 7, the maximum velocity is about 1.4 m/s, which is within the range of previous research¹⁹ and no longer contains completely spurious data from changes in the lighting due to voltage changes or building vibration. The GAMM 3D-smooth term (Ref.df = 754.4, F = 844.3, p < 2e-16, fREML score = 2.47E7) has an adjusted R-square of 0.0496, accounting for 4.96% of the data. This result is a statistically significant improvement compared to the adjusted R-square of 0.0264 and the fREML of 6.817E7 from the OpenOpticalFlow-only⁹ optical flow field analysis, as shown using the compareML tests (Akaike Information Criterion (AIC) difference 8.69E07)²⁵. Lastly, a pattern of faster air flow close to the ‘t’ from ‘waters’ and the ‘p’ from ‘impressed’ begins to emerge. Visually, Fig. 6 shows an interfering combination of buoyancy-driven air flow features and speech-driven air flow features.

Example: Light intensity balanced and high-pass filtered optical flow field analysis

Lastly, the optical flow field analysis of the schlieren data processed from light intensity balancing and high-pass filtering are the cleanest of the images. This is easy to see in the flow example (Fig. 8 top left), and in the optical flow field analysis example (Fig. 8 top center-left). The kymograph clearly shows no more evidence of buoyancy-driven air flow (Fig. 8 top right). Instead, the kymograph only shows the unique features of two points of high-velocity air flow (from the ‘t’ from ‘waters’ and the ‘p’ from ‘impressed’).

Fig. 8.

Image showing optical flow field analysis of light intensity balanced and high-pass filtered schlieren data using OpenOpticalFlow⁹. Top left: light-intensity balanced and filtered schlieren recording of subject 6, repetition 1, frame 1405 (S6, R1, F1405), 400x400 mm, shown in high contrast. Top center-left: Optical flow field analysis of the same frame, shown in high contrast. Top center-right: arcs indicating 5-35 cm from lips in 5 cm increments of the same frame. Top right: kymograph of air flow velocity over time, indicating the start and end of the phrase. Bottom: GAMM of optical flow field analysis of schlieren recorded from an arc 30 cm from the lips of all the speakers saying ‘the beige hue on the waters of the loch impressed all’.

The maximum velocity of 1.3 m/s (Fig. 8, bottom) is well within the range of previous research¹⁹. As we would expect, air flow velocity before speech onset is nearly 0 m/s. Two very clear regions of high velocity from the ‘t’ from ‘waters’ and the ‘p’ from ‘impressed’, as well as a smaller region of higher-velocity air flow from the ‘h’ in ‘hue’, emerge. The GAMM 3D-smooth term (Ref.df = 396.5, F = 483.8, p < 2e-16, fREML score = 7.31E6) has an adjusted R-square of 0.0757, accounting for 7.57% of the data. This result is a statistically significant improvement compared to the adjusted R-square of 0.0496 and the fREML of 2.47E7 from the light intensity balanced optical flow field analysis as shown using the compareML tests (AIC difference 3.49E7)²⁵. This result represents the highest r-squared value of the three datasets, and a fREML score of 10.7% that of the one from the OpenOpticalFlow-only analysis, showing significantly decreased residuals (AIC difference 1.22E8 between the OpenOpticalFlow only vs light intensity balanced and high-pass filtered optical flow field analysis). Taken together, Fig. 8 shows speech air flow velocity and only speech-driven air flow velocity. Visually, Fig. 8 shows a near-complete removal of buoyancy-driven air flow velocity, leaving only speech air flow velocity.

Discussion

The comparison of Fig 8 to Fig 7 and Fig 6, shows that the combination of light intensity balancing to reduce the effects of mirror imperfections and lighting issues, along with per-pixel time-based high-pass filtering to filter out buoyancy-based air flow velocity information, allowed us to focus on and isolate high-speed speech air flow information from our schlieren video. This type of noise reduction and image enhancement is in line with the bulk of signal processing techniques used in closely related low-light image enhancement²⁶.

The quantitative statistical evidence supports our visual assessment through evidence of increased explanatory power (R²: 2.64%→ 7.57%), vastly reduced residuals (fREML: 6.8E7→7.3E6), and improved model fit (AIC reduction 1.22E8). The use of reduction of information criterion (such as AIC) to assess the effectiveness of light intensity balancing and high-pass filters to isolate fast speech air flow is computationally sound and well-grounded in modern signal processing theory^27,28.

Considering the non-dimensional parameters used to classify flows, the typical Grashof number of the buoyancy-driven flow in our example can be calculated from Eq. (4).

4

 

where g is the acceleration due to gravity of 9.81 , β is the coefficient of volume expansion taken to be 1/T where T is the absolute temperature of the air (293 K), is the skin temperature of 35 C, is the ambient air temperature of 20 C, is the lengthscale, taken to be the approximate diameter of the head at 0.25 m, and ν is the kinematic viscosity of air, taken to be . This gives a Grashof number of approximately , suggesting a laminar free convective flow in the absence of other disturbance.

The typical Reynolds numbers can be calculated from Eq. (5).

5

 

where U is the typical velocity of a flow feature caused by speech, taken as 20 m/s for the fastest flows, and 0.5 as a typical value for slow flows. is the lengthscale, here taken to be a typical opening between the lips of 0.02 m, and ν is again the kinematic viscosity with the same value as above. This gives typical Reynolds numbers ranging from 700 to 30000, suggesting laminar flow at the lower speeds, and potentially turbulent at the higher.

The Grashof and Reynolds numbers can be used to calculate the Richardson number Ri using Eq. (6)

6

 

with values between 0.05 and 80, suggesting our problem includes regimes where free convection and forced flow are both relevant to heat and mass transfer (Ri ∼ 1 and in which free convection has little effect (Ri ≫ 1), although here the convective flow still obscures details of the speech-induced air motions, as is seen above.

Future directions

We chose our example of schlieren images of speech air flow as part of an effort to follow up on previous research demonstrating speech air flow can affect speech perception in untrained and unaware perceivers¹. While prior work has inferred air flow-perception links from synthetic stimuli^1–3 or point measurements^19,29, our two approaches provide the first direct evidence that natural speech generates complex, far-field air flow patterns during continuous speech that have the potential to alter speech perception. For instance, our isolation of the /t/ burst in ’waters’ and the ‘p’ in impressed, propagating to 30 cm (Fig. 8) confirms that aero-tactile cues exist under ecologically valid conditions. Our schlieren-derived air flow maps provide the first template for synthesizing ecologically valid aero-tactile stimuli that preserve the spatiotemporal statistics of continuous speech. No prior study has correlated natural, continuous speech air flow with perception; our schlieren-derived templates will enable this for the first time. Additionally, differences in the penetration rate of speech air flow based on the type of speech sound can now be used to test the effects of variably asynchronous presentation of audiovisual and aero-tactile speech on brain activity, as measured by event-related potentials, and on behavioural responses during speech perception. These two approaches are therefore an essential bridge between fluid dynamics and neuroscience.

Our method should be considered as one option in a wider spectrum of tools used to identify modal behaviour in flows, including Proper Orthogonal Decomposition and its variants^30,31, and³². The relative advantages in terms of computational cost, sensitivity to noise, time and space resolution requirements, and bias to high energy structures are worthy of investigation. Our method does not produce the rich set of modes that POD does, but may be more robust in flows with two main components.

Generalizability

We demonstrated our framework using speech— where isolating far-field air flow reveals cues that may directly modulate perception. However, the two approaches described here are also easy to implement and highly generalizable to any unseeded transparent air flow: They involve two computational techniques that have been known to engineering for decades, allowing reuse of old equipment and systems to capture transparent fluid flows, or reanalysis of old recordings, which is of use in this era of budget constraints. While we focused on high-pass filtering in order to discriminate between easier-to-track slow-motion buoyancy and harder-to-track faster speech air flow, any high, low, or band-pass filter can be used depending on the research or engineering use-case. In our extended methods, we present a simple staged schematic to illustrate relevant techniques for ablative testing of multiple competing filters.

Also, our two approaches of post-processing can be used anywhere a transparent fluid flow recording can be made, and where a need to differentiate high and low velocity data arises, and where the recording can capture flow along the longitudinal (sagittal) plane. Such transparent flows not only include schlieren imaging, but also direct video of transparent flows, magnetic resonance velocimetry and acoustic tomography, where images are recorded fast enough to see body or fluid motion. A possible use-case is biomedical fluid dynamics, where isolating fast blood flow (arterial pulsation) from slow venous return or lymphatic drainage within in-vivo imaging is currently accomplished using contrast magnetic Resonance Imaging (MRI) or Positron Emission Tomography (PET)³³, which is low resolution, near-infrared spectroscopy with active tracer chemicals³⁴, and still cannot discriminate between flow sources of different velocity, or Doppler³⁵, which is a one-dimensional technique. Another possible use-case is plasma physics, where resolving fast magnetic reconnection events from slow background plasma drift in tokamaks (or other fusion reactors) is currently accomplished through Thomson scattering³⁶, which are point measurements, and laser-induced fluorescence, which alters plasma and is unsuitable in many experiments.

Our techniques are particularly suited for non-quasi-static problems where some part of the object is moving quickly, and thus has both a fast flow (at freestream speed) and a slow flow (due to the movement of the object e.g., turning of a vehicle, movement of an aircraft control surface, opening of a window etc.). These examples highlight the method’s potential in unseeded, high-speed, and multi-scale flow environments where traditional velocimetry struggles, or where cost savings is extremely important.

Limitations

The key limitations of our two approaches are:

1) They cannot distinguish slow flow from faster flows tangential to the imaged plane (in our case, the longitudinal plane), and so are best suited for situations where the desired flows have an overall direction lying in the imaged plane. This is common to all planar techniques which measure only in-plane velocity components.

2) The method is expected to work when the characteristic temporal frequencies of the two flow components are separated by at least one order of magnitude. As noted above, we removed the signal due to buoyancy-based motion at or below 0.008 m/s, and the peak air flow velocity at 30 cm from the lips was about 1.2 m/s. This represents a minimum of two orders of magnitude difference.

3) The method cannot work unless the flows are transparent as any one opaque flow will obscure the rest, making the technique invalid, and

4) The process assumes that the imaged features (in our case light refraction) are travelling at the fluid velocity, which is an assumption that does not always apply to schlieren imaging³⁷, but whose validity is improved by the techniques described here in that they remove conflicting information that can interfere with velocity analysis.

Further validation against other velocity measurement techniques is of course required.

Conclusion

The method of combining intensity balancing and high-pass filtering has been shown efficacious to successfully remove that part of the signal which is due to slow flows from the data, leaving only the signal from faster flows. This then allows an optical flow algorithm to track those fast flows more accurately than it would if slow flows, and lighting artifacts, interfered with its analysis. The method has been successfully demonstrated on schlieren videos of speech-generated air flows contaminated with slow body-heat-driven flow. The velocity fields have been checked against CFD simulations of the flow generated by the English ’pa’ sound. The velocity fields were successfully decomposed into the speech generated by different English speech sounds by generating kymographs and analysing these with GAMM statistical models. The data processed with our algorithm generates results of much higher statistical significance, and much lower residual information, than conventionally processed data. The method is generalizable to any dataset containing fast and slow motions.

Appendix 1 Simulation of air flow impinging on a face

We performed a new simulation recreating many of the features of the Derrick et al. simulations¹⁹, but with the Large Eddy Simulation (LES) turbulence model. Air flow impacted a simulated face at 30 cm from the speaker. The coupled governing equations for the air puff modelling are: 1) The transient incompressible Navier-Stokes equations with Boussinesq approximation, which assumes the air properties (e.g., density) are constant, except when the buoyancy is calculated, where the density is primarily dominant by temperature; 2) The heat balance equation. The governing equations were numerically solved using the Finite Element Method (FEM) with a P1+P1 discretization scheme in COMSOL Multiphysics 6.1 with a relative tolerance of 0.001, utilizing the generalized alpha time-stepping method^39,40.

Turbulence was modelled using the Large Eddy Simulations (LES) with a residual-based variational multiscale (RBVM) model⁴¹. The heat transfer due to turbulence was modelled using a user-defined turbulent Prandtl number of 0.85.

The simulation was initialized with an ambient temperature of 20°C and zero velocity everywhere. The boundary conditions for the simulation are established as follows:

The inlet (lip aperture) was modeled as an ellipse with semi-axes of 15 mm and 2 mm. A normal inflow velocity (m/s) is defined by Eq. 7

7

where t is time in ms, with a constant temperature maintained at 37C, representing the exhaled air during speech. This equation is matched both to the Derrick et al.¹⁹ simulation, which itself begins with a starting oral air pressure of 686.45 Pascals released as a puff of air (as opposed to a jet of air).

The speaker’s face’s external boundaries were modeled as no-slip walls, including a flat plane on which nose and mouth are sited with a constant surface temperature of 33C. This mimics the thermal interaction between the facial skin and the adjacent air.

The impingement surface, representing areas where expelled air may contact surfaces (e.g., another person), is characterized as no-slip and thermally insulated walls to prevent both air flow and heat transfer across this boundary.

The face of the cubic domain adjacent to the speaker’s face is treated as a symmetry plane to reduce computational effort.

The remaining boundaries of the cubic domain are set as outlets with a gauge pressure of 0 Pa, allowing for free exit of the air without resistance.

The mesh was initially auto-generated based on the physics modelled and the boundary conditions, then underwent adaptive refinement by minimizing the estimated error, resulting in a final volume of 0.1226 m³ comprising 2,242,827 elements. The mesh quality metrics show a minimum element quality of 0.039, an average of 0.59. The mesh independence study was conducted using the Richardson Extrapolation method⁴². Three meshes of increasing density were evaluated: a coarse mesh with 681,040 elements, a moderately refined mesh with 1,204,541 elements, and a highly refined mesh with 2,242,827 elements. The simulations recorded the velocity magnitude at 5 cm away from the inlet (i.e., the modelled lip aperture) across these meshes. The percentage errors associated with these meshes were estimated at approximately 20.0%, 5.8%, and 1.70%, respectively. This progression demonstrates a clear trend of improving accuracy with each level of mesh refinement. However, given the diminishing returns in accuracy improvements relative to the significant increase in computational cost, the results obtained from the finest mesh were deemed sufficient and thus accepted for further analysis.

Finally, output was recorded from 0 to 100 milliseconds with time intervals of 0.5 millisecond.

The entire structure of the simulation can be seen in Fig. 9. The simulation included a boundary plate used to estimate and recalibrate the relationship between pressure and force necessary for the best possible estimate of skin mechanoreceptor response in future work.

Fig. 9.

Speech air flow simulation boundaries.

Author contributions

Conceptualization: D.D.; Experimental design: D.D., M.J.; analysis and/or visualization: D.D., M.J.; administration, supervision and funding acquisition: D.D., M.J., writing and editing: D.D., M.J., J.C.

Funding

This research was funded by two grants: The first was a New Zealand Ministry of Business, Innovation, and Employment for providing us with a Covid Innovation Acceleration Fund grant to Mark Jermy (PI), Donald Derrick (AI), and Guy Colson (AI) of NIWA, Auckland, for our research on “Infection risk model of airborne transmission to facilitate decisions about PPE, ventilation and isolation in shared indoor spaces” (CIAF-160 UOCX2004). The second was a New Zealand Marsden Fund grant “Multi-sensory speech perception and syllable structure” to Donald Derrick, Mark Jermy, and Bryan Gick (21-UOC-046).

Data availability

Data availability Extended data has been deposited with the Open Science Foundation at: https://osf.io/yt3h8/[38]. The dataset includes all the code needed to replicate the anal- ysis shown in this paper for the example phrase in this paper. These include schlieren density and velocity videos, as well as audio files associated with each of these videos, including PRAAT[13] TextGrids containing labelled and segmented annotations all of the words and segments recorded for this paper. They also include a folder with extra images: the GAMMS for all phrases from 5, 10, 15, 20, 25, 30 (and portions of 35 cm) from the lips as available from the schlieren system. Finally, the supplementary materials contain a README text file describing contents and explaining how to run the provided code.

Declarations

Competing interests

The authors declare no competing interests.

Ethical approval

The University of Canterbury’s Human Ethics Committee (HEC) approved ethics for this study (HEC 2020/43 Amendment 1). The experiment was performed in accordance with the relevant named guidelines, regulations, and agreed-upon procedures listed in the HEC 2020/43 - Amendment 1 document. All participants provided written informed consent to participate in this research publication. This study did not include minors. In the consent form, each participant agreed to the publication of their images in journal articles and in open-access databases. All participants consented to allow audio and video recordings to be used in this research publication.