Infra-Red Imaging to Detect Respirator Leak in Healthcare Workers During Fit-Testing Clinic

Abstract

Objective: This study addressed the problem of objectively detecting leaks in P2 respirators at point of use, an essential component for healthcare workers' protection. To achieve this, we explored the use of infra-red (IR) imaging combined with machine learning algorithms on the thermal gradient across the respirator during inhalation. Results: The study achieved high accuracy in predicting pass or fail outcomes of quantitative fit tests for flat-fold P2 FFRs. The IR imaging methods surpassed the limitations of self fit-checking. Conclusions: The integration of machine learning and IR imaging on the respirator itself demonstrates promise as a more reliable alternative for ensuring the proper fit of P2 respirators. This innovative approach opens new avenues for technology application in occupational hygiene and emphasizes the need for further validation across diverse respirator styles. Significance Statement: Our novel approach leveraging infra-red imaging and machine learning to detect P2 respirator leaks represents a critical advancement in occupational safety and healthcare workers' protection.

I. Introduction

As the COVID-19 pandemic continues to persist into its fourth year, with what now appears to yet another new wave of infection in the U.K. and USA as of August of 2023, the protection of healthcare workers and the patients they care for remains a critical concern [1], [2], [3]. Central to this protection is the use of P2 filtering facepiece respirators (FFRs), which provide a crucial barrier against viral particles [4], [5], [6], [7]. However, ensuring a proper fit of these masks is often challenging, with leaks posing a significant risk to users [8], [9], [10].

To ensure that healthcare workers are appropriately trained and have selected the correct design/size of respirator to fit their specific facial topology without leaks, a respiratory protection program that includes fit-testing and training is mandatory [5], [6], [7], [11]. However, a formal fit-test is only taken annually, and in the year between sessions workers are required to self-assess the suitability of their respirator by performing a user fit-check to assesses for air leak each time a respirator is worn. Multiple studies have shown that user fit-checks are inaccurate and unreliable in up to 54% of users [12], [13], [14], [15].

In an ideal world, healthcare workers should know the efficacy of fit of their respirator in real time, via objective measures to inform them of their level of protection when working in high-risk environments. Past research has attempted to harness the power of infrared (IR) imaging and machine learning to detect respirator leaks, with mixed success [16], [17], [18]. These studies encountered issues such as overfitting and low accuracy, largely due to data scarcity and limitations in their methodologies. Nevertheless, the promise of these technologies to advance respirator fit-testing suggests further investigation is warranted.

In this context, our study aims to build on previous work by using IR imaging and a selection of machine learning models to predict the fit-test result of P2 respirators, removing the need for expert classification when detecting leak. We aim to investigate whether machine learning models can accurately predict a pass or fail result based on IR imaging alone when validated against gold standard quantitative fit-testing methods. Our objective was to train and validate multiple machine learning models, and test their accuracy on unseen data, to provide a robust and reliable method for predicting respirator fit-test results. By doing so, we aim to contribute to the broader efforts to enhance the safety and protection of healthcare workers in this challenging pandemic context.

Hypothesis: That a machine learning algorithm can be found that can be used to predict the pass or failure of a P2 FFR from thermal imaging better than that of self-fit-check.

II. Materials and Methods

The methodology for this study is outlined in the following sections. Part 1 describes how we collected infra-red imaging data and correlated these images with fit-test results. Part 2 describes the data preparation and processing steps that were required prior to applying the machine learning algorithms.

A. Part 1 – Data collection

1). Participants

The study population consisted of healthcare workers from a medium-sized (∼1500 employee) community care organisation in suburban Southern Adelaide.

The organisation was prepared for COVID-19 outbreaks following guidance of the Australian Government Infection Control Expert Group, including a respiratory protection program. In this respiratory protection program, workers attended a structured clinic where quantitative P2 FFR fit-testing was performed to select and evaluate the appropriate size and style respirator for the individual.

Participants were invited to join the study at the beginning of their visit to the clinic by the occupational hygienist/nurse.

Inclusion criteria included employees required to undertake quantitative fit testing as part of routine respiratory protection program. Exclusion criteria were those who wore beard greater than 2 mm.

The study was conducted in accordance with the principles of the Declaration of Helsinki and was approved by the SALHN Human Research Ethics Committee (SALHN HREC 2021/GEM00074). Participants provided informed consent before enrolment.

2). Selection of P2 filtering facepiece respirator

P2 FFR's used for this study were taken from the current supply chain within the institution with four different sizes available at the time of fit-testing clinic. All respirators were flat-fold design with ear-loops that comes with a ‘clip’ that is used to tether the loops together on the head for additional fixation support. The use of the ‘clip’ that is provided with these respirators is required by the manufacturer in accordance with their regulatory certification (Therapeutic Goods Administration, Australia). It must be noted that not all respirators with ear-loops are supplied with this ‘clip’, and this is a particular feature of the respirators used in this study.

To select the first respirator to test, the occupational hygienist made a visual assessment of the participants facial features (nason-menton length, and bi-zygomatic width) according with standard process for this role.

3). Fit testing

During the visit to the clinic, fit testing was conducted to assess the adequacy of P2 FFRs in preventing the ingress of microscopic particles like COVID19 virus by a trained and experienced occupational hygienist. The fit testing clinic is required under that Australian Standard to assess the suitability of a respirator for an individual [3].

Quantitative fit-testing was conducted using a PortaCount 8048 device in N95 mode (TSI Inc, Shoreview, USA). In N95 mode, the PortaCount measures the concentration of microscopic, aerosolized particles (40 nm–60 nm) in the ambient air and compares it to the concentration of particles that leak into the FFR through gaps between the face and the FFR while it is worn (rather than through the filter medium of the FFR). For each fit test conducted in this study, the “Modified Ambient Aerosol CNC Quantitative Fit Testing Protocol for Filtering Facepiece Respirators” was selected, whereby participants are required to: bend at the waist as if going to touch toes for 30 seconds, talk out lout slowly and loud enough so as to be heard by a test conductor for 30 seconds, stand in place and turn head side to side for 30 seconds and finally stand in place and move head up and down for 30 seconds.

The occupational hygienist then conducted the fit test and recorded the overall result (pass/fail) and the overall fit factor achieved. A fit factor of 100 was used as the pass/fail threshold for the study.

4). Infra-red imaging

Infra-red imaging was performed using commercially available camera built for iPhone/iPad (Flir One Gen 3 -iOS, FLIR Systems Inc, Oregon USA). This camera has a thermal resolution of 160 × 120 pixels, and thermal sensitivity of 60 mK. For this study, the IR camera was attached to a tablet (iPad (8th Generation, Apple Inc, Cupertino, USA) with the accompanying software app installed (FLIR ONE, FLIR Systems Inc, Oregon USA).

The thermal camera was turned on and allowed to self-calibrate prior to any images taken. Using a floor-mounted stand, the tablet and thermal camera was manipulated into position by the hygienist to position it in front of the participant's face at a distance between 40 cm–60 cm [Fig. 1]. Immediately prior to commencing a fit-test with an individual (ie after tubes connected, PortaCount system prepared and the participant had donned their chosen respirator according to instruction), the hygienist used the FLIR ONE app to record an image of the respirator on end-inspiration. Images were saved to the tablet's internal memory and given a unique identification number for cross-referencing fit-test results on analysis.

FIGURE 1.

Infra-red setup for data collection showing key components and positioning.

B. Part 2 – Data preparation and analysis using ML

1). Thermal Image Data Pre-processing

Native infra-red images were exported from the study tablet to a study PC with software packages ImageJ (Rasband, W.S., ImageJ, U. S. National Institutes of Health, Bethesda, Maryland, USA) with loaded function for processing FLIR images (Glenn J. Tattersall. (2019). ThermImageJ: Thermal Image Functions and Macros for ImageJ) and computational programming language (Matlab 2022a, Matlab Inc, USA). All FLIR images were imported to imageJ in native .jpg format and converted to thermal .tiff format for analysis in Matlab [Fig. 2]. In Matlab, thermal images were separated into separate folders for ‘pass’ and ‘fail’ categories according to the results of the corresponding quantitative fit-test.

FIGURE 2.

Processing of FLIR image to thermal using ImageJ.

2). Region of Interest

A suite of custom Matlab scripts where used to create semi-automated region of interests (ROI's) within the thermal images around the boundary of the respirator on the nose and cheeks (where majority of leaks occur in P2 FFR's) [17]. To achieve this, images were displayed on the computer screen and an experienced occupational hygienist was instructed to trace around the upper boundary of the respirator to the zygomatic roll-off location, and then close the ROI making an approximately 2cm wide strip across the cheeks and nose [Fig. 3, Panel A]. To control the creation of this ROI as far as possible, the ‘AssistedFreehand’ function was used in Matlab to automatically follows edges in the underlying image. As the morphology of each ROI is determined by underlying facial geometry and not standardised, we used a normalization process to map the upper edge of the ROI to a vertical line to normalise the morphology of the images as far as possible [Fig. 3, Panel C].

FIGURE 3.

Process of making the ROI for analysis.

3). Feature Extraction

For each normalized ROI, image feature extraction was completed using grey-level co-occurrence matrix analysis [18]. In short, a gray-level co-occurrence matrix (GLCM) is a statistical representation of the spatial relationships between pixel intensities in an image. It quantifies the occurrence of pairs of pixel values at specified distances and angles, providing information about texture, patterns, and relationships within an image. In the context of detecting air leak from the P2 FFR, the spatial relationships between pixel intensities (i.e., thermal gradient) permit the quantitative detection of distinct temperature changes along the respirator's boundary, which could signal a leak.

For each ROI, we extracted 8 GLCM features; Contrast, Energy (or Angular Second Moment), Homogeneity, Correlation, Entropy, Dissimilarity, Autocorrelation (ASM), based on their consistent use and proven efficacy in texture pattern variations in prior research. It is worth noting that the GLCM method can generate a larger set of features, yet we focussed on eight to reduce the issues related to issues relating to dimensionality, reducing potential overfitting and computational costs, and enhancing the interpretability of our model.

4). Machine Learning for Leak Detection Classification

Machine learning (ML) techniques were implemented to classify respirators as either ‘passing’ or ‘failing’ based on quantitative fit test result. A comprehensive ML pipeline was developed, encompassing data pre-processing, model selection, training, performance evaluation, and deployment, all within a graphical computing environment (Classification Learner, Matlab) (Fig. 4).

FIGURE 4.

Implementation of the ML model on mask fringe region of interest.

5). Preliminary Assessment for Model Selection

A wide variety of algorithms were deployed to identify candidates that provide high accuracy in classifying respirator IR image as either ‘pass’ or ‘fail’. The data matrix of 8 GLMC features were divided using an 80%/20% partition for training and testing. This stratified partitioning guarantees the model's performance evaluation would be conducted on a separate set of data not used during the model training, providing a more objective measure of its predictive capabilities. An array of 22 models were trained and tested including Decision Trees, Support Vector Machines (SVM), Ensemble Methods (such as Bagged Trees, Boosted Trees, Random Forests, and Gentle Boost), Discriminant Analysis techniques (Linear, Quadratic, Regularized), Nearest Neighbours (k-NN), Naive Bayes, Generalized Linear Models (GLM, including Logistic Regression, Poisson Regression, Gaussian Regression), Deep Learning (Neural Networks), Gaussian Process, Hidden Markov Model (HMM), K-Means Clustering, and Self-Organizing Map (SOM). Models were evaluated using cross-validation with k-folds on 80% of the dataset, with 20% of the dataset reserved as an unseen test set. For all 22 models, the validation accuracy and validation total cost was reported in table form.

6). Model Sub-selection

To further evaluate model performance and generalizability, we selected the best performing model from each of model type: Decision Tree, Discriminant, Logistic Regression, Naïve Bayes, Support Vector Machines, Nearest Neighbour, Kernel Approximation, Ensemble Classifiers and Neural Network Classifiers.

Learning curves were used to evaluate the performance of well performing models with the augmented combined with original dataset and divided into training and validation sets using k-fold cross-validation. The training error and validation error were calculated for each iteration, where the size of the training set increased incrementally. The learning curves were plotted to visualize the change in error with increasing training examples. Learning curves were inspected to assess average model error over the folds, providing insights into the model's ability to generalize as the training set size varies.

III. Results

A total of 48 participants (75% female) were recruited with thermal image taken during end inspiration while wearing their professionally selected P2 FFR. There 27 PortaCount failures and 21 passes. All IR images passed visual inspection for suitability (screening for out of focus images, objects obscuring (ie hair) the respirator). All participants wore the same style of flat-fold P2 FFR (D95, Detmold Medical, Adelaide, Australia).

A. Image processing

All 48 thermal images were converted to .tiff file format containing only per-pixel temperature values and exported using ImageJ for custom processing [Fig. 2].

Region of interest (ROI) windows were made and exported for all images, with all final ROI data flattened to a rectangular window of 60 × 138 pixels [Fig. 3]. Data augmentation was performed on all rectangular ROI images [Fig. 3, Panel C] by flipping along the short axis extending the dataset size to 96 images.

GLCM features were extracted for each dataset using the following parameters:

• •Offset of: 10 × 2 pixels
• •Number of Levels = 50

1). Preliminary Assessment

Twenty two machine learning models were investigated using the full augmented dataset of 96 cases each with 8 GLCM features and classified as ‘pass’ or ‘fail’ according to the quantitative fit test result obtained during fit testing.

With 100% of the dataset used for model validation with 5 layer k-fold cross-validation, the highest accuracy was found to be with the Ensemble (Bagged Trees) model, with an accuracy of 90.5% at a cost of 17 and the lowest accuracy was found with Ensemble (Boosted Trees) model, with accuracy of 56.8% at a cost of 32.

The 6 Top performing models were selected for extended analysis of generalizability and accuracy.

To investigate the generalisability of the models, training was performed again on these select 6 with data partitioned at 90% for training and 10% as un-seen test data (Table 1).

TABLE 1. Top 6 Performing Models All Demonstrated an Accuracy of Greater Than 80% With a Cost No Greater Than 13 When the Models Were Trained on 100% of the Dataset, and Evaluated With 5 Levels of K-Fold Cross Verification.

Model Type	Accuracy % (Validation)	Total Cost (Validation)
Ensemble (Bagged Trees)	90.5	7
Fine Tree	86.5	10
Neural Network (Bilaryered)	86.5	10
SVM (Fine Gaussian)	85.1	11
KNN (Weighted)	83.8	12
Ensemble (RUSBoosted Trees)	83.8	12

By testing the models on an unseen dataset reduces the likelihood of overfitting to the data (Table 2). Based on these results, four of the models returned 100% accuracy of predicting if a participant would fail a PortaCount fit test; SVM, KNN, Ensemble Bagged Trees, and Bilayered Neural Network. From these data, the Ensemble RUSBoosted Tree was overfitting in the validation data and resulted in only 55.5% accuracy on the unseen data. Similarly, the Fine Tree model returned only 77.8% accuracy on the unseen data, compared to 83% in the validation. Of note, the total validation cost of both the Ensemble RUSBoosted Tree and the Fine Tree were higher than the other models, at with a cost of 20 and 14 respectively (Figs. 5 and 6). Of note, the total cost of the test data was highest in these two models, at 4 and 2 respectively, whereas the cost of all other models was 0 (Table 2).

TABLE 2. Accuracy and Cost of Models Used.

Model Number	Model Type	Accuracy % (Validation)	Total Cost (Validation)	Accuracy % (Test)	Total Cost (Test)
1	Tree	83.13	14	77.78	2
2	SVM	87.95	10	100	0
3	KNN	84.33	13	100	0
4	Ensemble (Bagged Trees)	89.15	9	100	0
5	Ensemble (RUSBoosted)	75.90	20	55.56	4
6	Neural Network	89.16	9	100	0

FIGURE 5.

Confusion matrix for PPV for sub-selected ML models (accuracy > 80%).

FIGURE 6.

Receiver operator characteristic for 6 selected models showing strong predictive ability to determine fit-test pass and fail based on IR image ROI features.

IV. Discussion

We have conducted a thorough investigation in to the utility of using machine learning classification algorithms to predict the binary fit-test result of P2 FFR wearing healthcare workers.

Key findings of this study:

• 1)Machine learning algorithms are able predict whether a P2 respirator passes or fails a quantitative fit test when using the Detmold D95 Flat-fold respirator.
• 2)Multiple machine learning algorithms return very high accuracy with cross validation for predicting pass or fail quantitative fit test.
• 3)SVM, KNN, Neural Networks and Ensemble Bagged Trees were able to predict fit-test result with 100% accuracy when tested on unseen data.

A. Comparison to prior literature

The gold standard for point-of-care respirator assessment is the self fit-check where a user is required to evaluate if the respirator they are wearing leaks by a series of blowing out and feeling for air flow with their hands, and assessing pressure changes behind their respirator [19]. Fit-checking however, has been shown to be inaccurate [12], [13], [15], [20], [21], [22], [23]. Due to the number of studies reporting the accuracy of fit-check, we conducted a rapid literature review and meta analysis to determine the pooled accuracy of fit-checking compared to the gold-standard quantitative fit-testing. A total 34 results were returned using keyword searching in PubMed (((((Accuracy) OR (performance) OR (Sensitivity))) AND ((N95 respirator) OR ("N95 mask") OR (n95 masks) OR (N95 Respirator))) AND (((fit check) OR (fit-check)) OR ("fit check")). Full text was retrieved for 9 articles based on abstract data meeting inclusion criteria (reporting on fit-check and PortaCount testing). After review of full text, further 4 studies were excluded due to inadequacy of data presented (not presented as proportion of participants. 5 studies met the inclusion criteria of reporting the proportion of participants that passed user fit-check but went on to fail quantitative fit testing. In this meta-analysis results of 5 studies investigating the proportion of participants who believed that their N95 Filtering Facepiece Respirators (FFR) was not leaking when tested using a user seal check but subsequently failed a PortaCount fit test. The results show considerable heterogeneity across studies, suggesting that differences in study conditions or populations may influence the outcomes. The random effects model, which accounts for this variability, estimates the average proportion of those who pass the user seal check but fail the PortaCount fit test to be approximately 24.03% (Fig. 7). This suggests that roughly a quarter of users who believe their N95 FFR is properly fitted could be wearing a respirator with leak, emphasizing the limits of user self-checks for N95 FFR fit.

FIGURE 7.

Literature review and meta analysis of articles reporting accuracy of self-fit-check when measured with quantitative fit-testing.

Other groups have reported on infrared imaging to predict respirator leak, however with varied success. In 2011, Roberge et al reported the first use of infra-red imaging for leak detection on N95 FFR's [18]. In this comprehensive study, they authors were able to detect leaks using IR which correlated with quantitative fit-testing (p< 0.001), and they even reported on observed leaks with IR when the subject achieved a resounding pass (fit factor >200) on quantitative fit testings. These authors suggested that IR may be more sensitive to leaks than quantitative test, yet the authors had no methods to verify their findings, yet overall, the authors were not confident that their methods were substantially reproduceable en-masse to supplant condensation nuclei counters.

In 2015, Harber et al re-visited the potential for infra-red imaging to detect respirator leak [16]. Building on work by Roberge et al, these authors used video imaging of tidal respiration while holding the head fixed in a gig to allow for their image analysis techniques that were sensitive to motion. In this study, IR was again able to detect leak, however even wit these advanced methods, the ability to provide a continuous variable to match that of condensation nuclei counting methods was not reliable.

Then in 2022, during COVID pandemic, Siah et al reported on the use of modern machine learning algorithms to predict respirator leak from infra-red images [17]. Armed with a state of the art high resolution infra-red camera, multiple images were taken of N95 FFR's of participants who had failed fit testing (qualitative). Images were cropped, reshaped and augmented and processed by a convolution neural network (CNN). It was found that with their CNN, accuracy of between 20%–30% (dependent on camera angles) was achieved. However, the authors conclude that overfitting was a possibility due to the small size of their dataset.

Our study, using single-frame IR images with simultaneous quantitative fit test results was able to train multiple machine learning algorithms to predict quantitative fit test pass or failure with very high accuracy which cross validated and then tested on un-seen data. Our study builds on the work of these previous pioneers in the field if infra-red imaging for leak detection, yet with the fortunate timing of technology convergence of computer vision and machine learning to remove the barriers that existed for previous groups. Furthermore, in contrast to our peers, we observed the thermal gradient in the respirator itself and not the surrounding skin, which has been reported to be insensitive to small temperature changes like those where a small leak is present.

V. Conclusion

In this study, we successfully employed infra-red (IR) imaging and machine learning to detect leaks in P2 respirators, overcoming the limitations of traditional self fit-checking methods. Our technique, which focuses on the thermal gradients of the respirator, demonstrated high accuracy, suggesting a novel and more reliable method for leak detection compared to skin gradient measures used by others in the past.

These findings have substantial implications for enhancing healthcare worker safety, and has promise to provide an objective point-of-use leak detection system for high risk workers. While our study was specific to flat-fold P2 FFRs, these results lay groundwork for broader investigation across various respirator types and populations.

Funding Statement

This work was supported by the Australian Government Department of Health and Aged Care under Grant GA147146.