Detection of Breathing Movements of Preterm Neonates by Recording Their Abdominal Movements with a Time-of-Flight Camera

Abstract

In order to deliver an aerosolized drug in a breath-triggered manner, the initiation of the patient’s inspiration needs to be detected. The best-known systems monitoring breathing patterns are based on flow sensors. However, due to their large dead space volume, flow sensors are not advisable for monitoring the breathing of (preterm) neonates. Newly-developed respiratory sensors, especially when contact-based (invasive), can be tested on (preterm) neonates only with great effort due to clinical and ethical hurdles. Therefore, a physiological model is highly desirable to validate these sensors. For developing such a system, abdominal movement data of (preterm) neonates are required. We recorded time sequences of five preterm neonates’ abdominal movements with a time-of-flight camera and successfully extracted various breathing patterns and respiratory parameters. Several characteristic breathing patterns, such as forced breathing, sighing, apnea and crying, were identified from the movement data. Respiratory parameters, such as duration of inspiration and expiration, as well as respiratory rate and breathing movement over time, were also extracted. This work demonstrated that respiratory parameters of preterm neonates can be determined without contact. Therefore, such a system can be used for breathing detection to provide a trigger signal for breath-triggered drug release systems. Furthermore, based on the recorded data, a physiological abdominal movement model of preterm neonates can now be developed.

1. Introduction

Preterm neonates’ respiratory parameters differ greatly from those of adults. In contrast to adults, who have a tidal volume of approximately 500 mL, a respiratory rate of 15 breaths per minute and an inhalation-exhalation ratio (I:E) of 1:1, preterm neonates have a significantly lower tidal volume of 4–8 mL/kg, a higher respiratory rate of up to 60/min and an I:E of up to 1:3, depending on gestational age [1,2,3]. Therefore, the determination of respiratory parameters in preterm neonates is very challenging, and not every monitoring system is suitable for this task. However, the respiratory cycle must be known with high precision for the safe and effective operation of breath-triggered drug release systems for neonates. One of the primary drawbacks of inhalation therapy using a continuous drug delivery system is the substance loss during exhalation [4,5]. This loss mainly depends on the patient’s I:E. In addition, pharmaceutical aerosols can only reach the alveoli during the first half of the inspiratory phase [6], which may result in an increased aerosol loss of up to 90% in preterm neonates without breath-triggered administration. This high loss is in accordance with previous reports of low deposition in infants [7,8,9,10], in some cases of less than 1% of the nominal dose [11]. Therefore, breath-triggered drug release is highly desirable. It allows for the patient-specific delivery of pharmaceutical aerosols and thus has the following advantages over continuous drug release:

1. The previously mentioned loss during exhalation is theoretically reduced to zero, as the pharmaceutical aerosol is only released during inhalation. This leads to higher drug utilization and substantial cost savings [12,13,14].

2. Different lung regions can be targeted, as the pharmaceutical aerosol can be released as a bolus at different, pre-defined instants during the inhalation phase. A release at the beginning of inhalation mainly targets peripheral lung regions, while a release towards the end of inhalation mainly targets central lung regions. This also results in a more time-efficient treatment and a reduced distribution of the drug in the body, which reduces side effects [15,16,17,18,19,20].

Currently, there are several approaches to breath-triggered drug release:

For example, a breath-actuated pMDI (pressurized metered-dose inhaler), such as the Autohaler (3M Drug Delivery Systems, St. Paul, MN, USA) or the Tempo inhaler (formerly MAP Pharmaceuticals, Mountain View, CA, USA, now Allergan, Irvine, CA, USA), can be integrated into the ventilation circuit in combination with a spacer. In this case, however, the drug is only released as a single dose via active actuation of the pMDI, by the patient himself or by a third party [21]. Aradigm’s AERx Pulmonary Drug Delivery System (Hayward, CA, USA) follows a similar strategy. This system also releases a single dose, but only if the patient inhales at a flow rate within a pre-defined range [22,23].

Alternatively, an aerosol generator can be coupled to the ventilation circuit by means of an adapter, for example a T-connector [24]. Triggering is achieved via the detection of a pressure change in the ventilation tube, whereupon aerosol production is activated or deactivated; this method is implemented, e.g., in the Aerogen device (Aerogen Ltd., Galway, Ireland) with an integrated control module (Synchro-Neb) [25].

A controlled drug release can also be implemented by means of a pressure measurement at the mouthpiece and an associated evaluation algorithm that considers an average value of the last three breath cycles. A commercial device using this approach is the I-Neb AAD (Philips Respironics, Murrysville, PA, USA). This system releases the pharmaceutical aerosol on the basis of a moving average, which means that the aerosol cannot be released optimally in the event of an aperiodic breathing behavior [26].

In addition, there are pre-calibrated systems, such as the AKITA JET (Vectura Group plc, Chippenham, UK). For such systems, pulmonary parameters must be determined beforehand. The identified parameters are programmed into the system in order to guarantee an optimal aerosol delivery. The system then provides a constant inhalation air flow, delivered by an integrated pump, and starts aerosol production when the patient begins to inhale. As soon as a certain aerosol inhalation volume has been achieved, aerosol production is stopped while the pump continues to provide an inhalation air flow until a defined gas volume is reached. This has the advantage that a particularly large amount of aerosol is deposited in the deeper airways [27].

Moreover, the Fraunhofer Institute for Toxicology and Experimental Medicine (Hannover, Germany) developed a system for the specific requirements of preterm neonates, which manages the breath-triggered drug delivery via a miniaturized valve [28]. This aerosol valve contains an elastomeric membrane that opens and closes symmetrically in less than 25 ms [29]. Due to its miniaturized size, the valve can be integrated directly into a patient interface and is controlled by the detected respiratory signals.

As previously mentioned, breath-triggered drug release devices require breath detection systems that are able to identify the onset of inhalation with high precision and provide appropriate trigger signals. Such systems for breathing detection can be divided into contact-based (either invasive or non-invasive) and non-contact-based (non-invasive) devices [30,31].

There are numerous contact-based methods for measuring respiratory parameters. According to the literature, the best-known contact-based systems are transthoracic impedance measurement, inductive plethysmography, the measurement of abdominal expansion using respiratory belts or the neurally adjusted ventilatory assist method (NAVA) [32,33,34,35]. Transthoracic impedance measurement is a method to derive the respiratory rate by measuring the impedance changes of the chest wall during respiration and is widely used for neonatal respiratory monitoring [36,37]. Respiratory inductive plethysmography, requiring two coils placed in the abdominal and the chest region, respectively, detects changes in self-inductance due to the cross-sectional change of the abdomen during breathing [38,39,40]. The change in inductance is proportional to the lung volume and is successfully used for the detection of respiratory movements in preterm neonates [34,41,42]. Respiratory belts, such as the Graseby capsule (Smiths Medical, Minneapolis, MN, USA) or strain gauges, register the respiratory movement of the abdominal wall by means of pressure sensors, impedance sensors or piezoelectric sensors, which are usually integrated in a single point of measurement on the belt [43,44].

In general, all these systems have a lack of operational reliability because the quality of the measurement strongly depends on the correct placement of the sensor. The applied sensor can cause irritation and damage of the skin or tissue, or, as with NAVA, the placement of the sensor is invasive [32,43,44,45,46]. However, sensor arrays on flexible [47] and even stretchable [48] foils were developed that can be attached to the skin, yielding a higher tolerance to positioning uncertainties. The reliability of these systems is increased by incorporating a high number of individual sensors. The changing shape of the foil can be reconstructed by sophisticated algorithms [49]. Using this stretchable foil, trigger signals were also generated in experiments with neonate models, demonstrating the future potential use to detect inspiration of preterm neonates [50].

Contact-less systems offer some advantages, especially for long-term application, as they do not have to be in permanent and direct contact with the patient. This avoids a stressful situation and pain, which may lead to an increase in the respiratory rate [31]. The best-known contact-less systems for respiratory parameter monitoring are based on flow or pressure sensors at the patient interface [36,51]. However, due to the large technical dead space volume and the susceptibility to aerosol, these sensors are not suitable for monitoring the breathing of (preterm) neonates. In addition, the sensors only function when the patient is intubated and invasively ventilated [52,53]. Therefore, these systems cannot be used for non-invasive ventilation.

Contact-less systems include optical (time-of-flight cameras [54,55], stereo triangulation [56,57] and structured light methods [58,59,60,61,62]), radar [63,64,65,66,67,68], microwave [69,70,71], thermal [60,72,73,74] and laser [75,76] technologies. However, these systems have hardly been clinically tested in preterm neonates. A more suitable respiration monitoring system for (preterm) neonates would be highly desirable. The development and regulatory approval of a breathing detection system for the vulnerable patient group of preterm neonates requires extensive functional tests of the system [77].

An essential challenge in the development of new respiratory sensors, especially contact-based, is that they can only be tested on (preterm) neonates with great effort (including the submission of ethics applications).

For example, a sensor patch for the contact-based registration of a neonate’s breathing cycle was recently developed by Koch et al. [47]. So far, this sensor array foil was tested for mechanical functionality by means of bending experiments. Investigations were also carried out on a ventilated, simple, non-physiological preterm neonate demonstrator in comparison with the measured ventilation flow. However, due to the aforementioned hurdles, no investigations have yet been carried out on preterm neonates in a clinical study. A physiological phantom of the neonatal abdomen enabling simulation of neonatal respiratory patterns would therefore be very useful for rapid, efficient testing and validation of surface-based respiratory sensors. Ethical hurdles would not have to be overcome, and costs could be saved.

To develop such a model, abdominal movement data of preterm neonates are required. Time-of-flight (ToF) cameras are capable of recording depth maps in real time by detecting the phase shift between illumination and reflection and converting it into a distance value [78,79,80]. Furthermore, ToF cameras offer a high spatial resolution and high frame rates and are readily available at a relatively low cost. Therefore, they are well suited for the purpose described above.

It is thus the purpose of the present contribution to introduce and clinically evaluate an optical, non-invasive measurement system enabling the spatial reconstruction of the abdominal movement of a preterm neonate over the course of the respiratory cycle. The obtained 3D recordings, provided as a data set together with this manuscript, represent a valuable starting point for the development of a physiological model of a neonate’s abdominal area which could then be used to circumvent the obstacles described above. This would be an important step towards a clinically available, breath-triggered aerosolization device for the treatment of preterm neonates.

In the following, we demonstrate the feasibility of the derivation of a preterm neonate’s respiratory parameters by recording their abdominal movement using a ToF camera.

2. Materials and Methods

According to the regulations for conducting a clinical study at Hannover Medical School, an ethics application was submitted, and the conduct of the study was approved (approval ID: 8584_BO_S_2019).

We used the CamBoard pico flexx (pmdtechnologies AG, Siegen, Germany) ToF camera to record the abdominal movement data. This ToF camera contains a vertical-cavity surface-emitting laser with a wavelength of 850 nm; it is classified as laser class 1 and is therefore eye-safe [81]. At an object distance of 0.1 m to 1 m and a frame rate of 45 fps, the depth resolution of the camera is ≤2% of the object distance [82]. Preliminary tests showed an optimum distance of 0.2 m between the target region and the camera. For the clinical study, the ToF camera was thus positioned at a distance of 0.2 m above the abdomen of the preterm neonate using a support arm system (Vario Lock, W. Krömker GmbH, Bückeburg, Germany) as shown in Figure 1. The recorded abdominal movement data were transferred to a data acquisition notebook.

Figure 1.

Measurement setup for clinical data recording, exemplified by a preterm neonate demonstrator (NENASim Preemie, Medical-X, Rotterdam, The Netherlands).

In the clinical study, five frame sequences at a duration of 90 s each were acquired with each of five preterm neonates (a total of 25 sequences) at a frame rate of 45 fps (see Table 1 for the clinical parameters of the included neonates).

Table 1.

Clinical parameters of neonates included in this study.

ID	Gender	Gestational Age at Birth [Weeks]	Postmenstrual Age at Recording [Weeks]	Weight at Birth [g]	Weight at Recording [g]	Therapy Form
1	Male	25	32 1/7	800	1580	High-flow nasal cannula
2	Male	32 1/7	33 1/7	755	720	Spontaneous breathing
3	Male	27 3/7	33 1/7	585	1245	High-flow nasal cannula
4	Female	27 3/7	33 1/7	930	1920	Spontaneous breathing
5	Male	27 3/7	33 1/7	860	1550	Spontaneous breathing

Python (version 3.8) and the main packages NumPy, SciPy, Pandas and Matplotlib were used for data pre-processing, analysis and evaluation. The initial pre-processing was done in two steps:

First, irrelevant image sections were removed by segmentation. In this process, the center of the ROI was manually selected, and irrelevant sections were removed by means of a cut-off threshold based on the distance to the camera.

Second, faulty pixels and artifacts were removed using a noise reduction method. This method employed a 3D median filter that included a $3\times 3$ pixel neighborhood in the image and five consecutive frames ($3\times 3\times 5$ filter mask). By comparing the standard deviation of all pixels in the ROI over all time points, outliers were automatically removed.

The results of this procedure are pre-processed depth image files that can then be further analyzed and evaluated.

3. Results and Discussion

Figure 2 shows the placement of the ToF camera above a preterm neonate and the resulting non-pre-processed grayscale and color-coded depth image. The selected ToF camera is particularly suitable for the clinical environment due to its small overall size compared to other systems with similar specifications, such as the Microsoft Kinect 2.0 [83,84] (Microsoft Corp., Redmond, WA, USA), the BlasterX Senz3D [85,86] (Creative Labs (Europe) Ltd., Dublin, Ireland) or the Argos3D-P100 [87] (BECOM Electronics GmbH, Hochstraß, Austria). While the Kinect has a better spatial resolution and a better signal-to-noise-ratio, the ToF camera we used offered a higher frame rate and a better depth resolution, especially at a distance of less than 0.3 m [88,89]. A high frame rate and a high depth resolution are necessary for the accurate detection of breathing movements of preterm neonates because of their high respiratory rate on the one hand and the low amplitude of their abdominal movement in the vertical direction on the other hand. Furthermore, the pico flexx device, at a price of approximately $390, belongs to the low-cost depth cameras, which allows setting up a measuring system for breathing detection economically [89].

Figure 2.

Representation of the measurement setup (left) as well as an exemplary 8-bit grayscale (center) and 24-bit color-coded (lowest distance to camera: red, largest distance: blue) depth image (right) recorded by the ToF system.

The grayscale image contains 256 intensity levels. Highly reflective areas are displayed whitish, and areas of low reflections are shown darker. The depth image was processed with a color map, so that the intensity spectrum contains the entire color scale of human vision from blue (largest distance to the camera) to red (lowest distance to the camera). The highly reflective zones, which are displayed with high intensity in the grayscale image, appear as black artifacts in the depth image, as no valid depth measurement could be performed in these areas. By observing consecutive depth images during the abdominal movement, characteristic distance changes over time, and thus respiratory patterns, can be visualized (see Figure 3).

Figure 3.

Respiration curve of a preterm neonate, recorded with a pico flexx ToF camera.

After filtering the raw data as outlined in Section 2, the temporal evolution of the abdominal movements can be detected very well. The displayed local maxima (maximum abdomen-camera distance) represent the transition from exhalation to inhalation, whereas the local minima (minimum abdomen-camera distance) represent the transition from inhalation to exhalation. The respiratory parameters derived from Figure 3 are 60 breaths/min and an I:E of 1:2.3, which correspond very well with values mentioned in the literature [2,90,91,92]. The average abdominal movement in the vertical direction between maximum inspiration and maximum expiration was (2.3 ± 0.2) mm. Different breathing patterns can be reconstructed from the abdominal movement data (see Figure 4).

Figure 4.

Characteristic breathing patterns, such as forced breathing (a), sighing (b), apnea (c) and crying (d), extracted from the clinically acquired ToF recordings.

After applying a median filter for noise reduction, however, a slight temporal offset could be detected. According to Haju et al., this temporal offset is caused by a phase shift of $\frac{\mathrm{N}-1}{2}=2$ frames [93] (N: window size of median filter).

The extracted breathing patterns shown in Figure 4 correspond very well with those described by te Pas et al., e.g., the decrease of the I:E ratio to 1:1 during forced breathing (see Figure 4a) [91]. Besides, spontaneously breathing newborns represent very irregular respiratory patterns such as periodic breathing [94], apneas [95] and sighs [96] in order to recruit lung volume. Figure 4b depicts a sigh, followed by an apneic pause and subsequently a slower respiratory rate, whereas Figure 4c shows an apneic phase of about 5 s. Crying is defined by a strong inhalation followed by an interrupted exhalation [97], which can be observed in Figure 4d.

Another common breathing pattern seen in preterm neonates is paradoxical breathing, in which the chest pulls inward on inhalation and outward on exhalation. This often occurs when the thorax is unstable. Because the bony thorax of preterm neonates is very elastic and gives way during forced breaths, this breathing pattern is commonly seen [98]. However, as (preterm) neonates are obligate abdominal breathers, the present study is focused on recording abdominal movements. We are confident that by focusing on abdominal movements, our system allows for the reliable detection of the inspiratory phase, even if a paradoxical thoracic movement should occur.

Further, we can confirm that the proposed method is not suitable for the detection of obstructive events, as it does not evaluate gas flow. However, we believe that this is of limited clinical relevance because apneic episodes (central, obstructive, as well as mixed) occur regularly in preterm neonates, leading to temporary gas flow interruption.

Using the recorded movement data, it is now possible to develop a physical abdominal model, for example on the basis of an actuated pin matrix [99,100]. These are also known as Tangible User Interfaces. The concept consists of a matrix of pins, each controlled individually or as a group by an actuator. Figure 5 (left) shows a servo motor driving a connecting rod that is attached to an individual pin. As a result, this setup requires slightly more space than the setup on the right in Figure 5, in which a continuous motor drives a threaded rod, thereby changing its height by ∆z.

Figure 5.

Example of shifting individual pins by ∆z by means of a servo motor (left) and by means of a continuous motor (right) to simulate abdominal movement [101].

By controlling the individual pins independently, three-dimensional shapes can be represented, and movements can also be reproduced in a desired form. Thus, using the abdominal movement data generated with the ToF camera, breathing patterns such as crying and sighing can be exactly simulated. This allows the initial testing of newly-developed (contact-based) sensors on such a model before being made available for clinical purposes. After validation, such sensor systems can be used for real-time breath detection, making it suitable for the application of breath-triggered drug release for (preterm) neonates [28,29,102].

Further, we believe that our depth-sensing approach may be advantageous in estimating the functional residual capacity (FRC), as it provides a three-dimensional representation of the neonatal abdomen. However, our data processing algorithm would have to adapted to obtain an estimation of the temporal evolution of the abdominal volume. The Time-of-Flight sequences of our study may serve as a data set to develop and evaluate future algorithms for FRC estimation.

4. Conclusions

We showed that recording respiratory parameters, as well as respiratory monitoring, is possible without any patient contact using a ToF camera in preterm neonates. Therefore, the proposed optical measuring system is deemed suitable for a wide range of neonatal applications, for example, general patient monitoring or, in particular, breathing detection. In principle, the contact-less recording by a ToF camera offers strong advantages over a contact-based method: The preterm neonate is neither affected by the ToF camera, nor does this measurement technique require patient interaction. Therefore, such a system is suitable as a source of a trigger signal for the application of breath-triggered drug release in (preterm) neonates.

Furthermore, the acquired abdominal movement data are also suitable for the development and control of an abdominal movement model. Sensors for the detection of abdominal movements could first be tested on such a model before being used in a clinical setting.

5. Outlook

The present work constitutes the first step towards the efficient and ethical evaluation of novel breath phase sensors for inspiration-triggered drug delivery systems using an actuated phantom of the patient’s abdominal area. The spatial movement data acquired in the present contribution could be mapped onto a pin matrix, as described in Section 3. The shape of the surface formed by the individual pins could be adapted to the surface topography of the abdomen, identified during the acquisition of the abdominal motion, to obtain an even higher accuracy of the simulated breathing activity.

In the future, a multi-body system, inspired by the anatomy of the neonatal abdomen, could be developed and used to validate the physiology of the phantom’s motions over a breathing cycle.

In conclusion, in the progress in the fields of high-precision breath cycle detection, controlled aerosol delivery and drug discovery will contribute to the establishment of more effective and targeted strategies for the treatment of preterm neonates by inhalation therapy.

Author Contributions

Conceptualization, F.C.W.; software, D.B.; writing—original draft preparation, F.C.W.; writing—review and editing, F.C.W., J.F.F., G.M., J.M., T.O., A.D., B.B. and G.P.; visualization, F.C.W. and D.B.; supervision, B.B. and G.P.; funding acquisition, F.C.W., T.D. and G.P. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by funding from the European Union (EU) within its Horizon 2020 programme, project MDOT (Medical Device Obligations Taskforce), Grant Agreement 814654, and from the German Federal Ministry of Education and Research (BMBF), Grant Agreement GS2SH016. The content is the sole responsibility of the authors and does not necessarily reflect the views of the aforementioned parties.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of Hannover Medical School (MHH) (approval ID: 8584_BO_S_2019).

Informed Consent Statement

Informed consent was obtained from all parents of the probands (preterm infants) involved in the study.

Data Availability Statement

The complete set of Time-of-Flight sequences which we recorded in our work will be uploaded and made available in the publicly available research data repository of the Fraunhofer Society (Fordatis) (https://fordatis.fraunhofer.de, accessed on 22 April 2021).

Conflicts of Interest

The authors declare no conflict of interest.