Modifying NIOSH’s Manikin Fit Evaluation Method to Match Fit Testing with Human Subjects

Abstract

A manikin fit test method developed by the Center for Disease Control and Prevention’s (CDC) National Institute of Occupational Safety and Health (NIOSH) has been proposed as an alternative to fit testing with human subjects. The advantages of a manikin fit test method over actual fit testing are that it does not require human subjects which can be resource intensive, and hence easier to implement. At the beginning of coronavirus 2019 (COVID-19) pandemic, although early studies showed that manikin fit can be maintained after several decontamination cycles, real world evidence obtained using human subjects revealed that the N95 respirators failed only after a few decontamination cycles. The goal of this article was to make modifications to the NIOSH’s manikin fit-test method so it can mimic real world performance of N95 respirators better. After making modifications to this method, we then investigated the effect of long-term wear after donning of the respirators, repeated donning and doffing, as well as decontamination methods (i.e. autoclaving and microwave generated steam) on the fit factor. Averaging the overall manikin fit factor across all scenarios, our modified method overpredicted overall fit factor by only 7 % and 14 % compared to adult human subjects using a breathing routine that included simulated heavy breathing rates of 85, and 70 Liters/minute, respectively. In addition, a constant flow produced similar results as cyclic flow using a breathing simulator. The modified test method also offered the following additional insights into reuse of respirators during future pandemics – when reused within a single work shift, more than 5 donnings should be avoided; and microwave generated steam may be a more viable option for decontaminating N95 respirators compared to autoclaving for a single decontamination cycle.

INTRODUCTION

As evidenced by the COVID-19 pandemic, N95 respirators are considered essential equipment for healthcare workers to protect them against biological particles (e.g., SARS-CoV-2). Although N95 respirators are intended for single use, a severe shortage of these respirators forced healthcare workers to either use them repeatedly or decontaminate before re-donning. It is generally unclear how these decontamination methods affect the overall performance of N95 respirators. Fit, filtration efficiency, and breathability are the most important factors for a tight-fitting respirator to provide protection to the wearer. Here we define breathability as the resistance or pressure drop offered by the N95 respirators. Filtration efficiency and breathability can be measured on the benchtop (D’Alessandro and Chichowicz, 2020). However, the standard method for fit testing utilizes human subjects.

In the United States, the Occupational Safety and Health Administration (OSHA) has established regulations that require non-agricultural businesses including hospitals to implement a respiratory protection program as per Code of Federal Regulations (CFR) 1910.134(c) (OSHA 1998a) that includes measurement of fit (Mendeloff et al. 2013). To adhere to the program, in addition to other factors, employers need to fit-test employees either qualitatively or quantitatively (OSHA 2021). Quantitatively, employees must achieve a fit factor of at least 100 to be able to use a specific model and size of a N95 respirator. The Centers for Disease Control and Prevention’s (CDC) National Institute for Occupational Safety and Health (NIOSH) developed a method of using adult manikins for fit-testing respirators to act as a surrogate for human test subjects (Bergman et al. 2013, Bergman et al. 2015). This method was used during COVID-19 by NIOSH to test the fit of several brands of respirators for multiple decontamination system manufacturers to assess respirator performance following decontamination (CDC 2020a). It was assumed that if the fit-factor assessed using the manikins were greater than 100 for a certain number of decontamination cycles, and the filtration efficiency and breathability were sufficiently similar post decontamination, then, it was likely that the respirators could be used by health care providers for that many number of cycles. For example, for vaporized hydrogen peroxide decontamination, 3M’s N95 models 1860, 8200, Gerson’s 1730 and 1740, Moldex’s 1512, Sperian’s N1125 and One-fit were predicted to last 20 decontamination cycles (CDC 2020a). However, subsequent studies performed on human subjects revealed that for most brands and models of respirators the fit appears to fail only after a few (typically less than 4 to 5) decontamination cycles (Fischer et al. 2020, Lieu et al. 2020).

Given the difference between manikin fit-test results obtained in the laboratory settings with real world evidence, the authors in this article set out to modify NIOSH’s manikin method to close this difference. To validate the modifications, the authors performed the comparison of NIOSH’s modified manikin fit-testing method with human subjects for the following scenarios: long term wear after donning (to mimic the total wear of multiple work shifts); repeated donning and doffings (to mimic reuse within the same work shift); impact of decontamination on fit for respirators treated with autoclaving and microwave-generated steam, all of the preceding with constant suction flow rate; and then for a limited number of cases using a breathing simulator. The authors anticipate this method, having better alignment with real world evidence, can be used during a future pandemic in lieu of testing with human subjects. Also, this can be implemented by the manufacturers when developing future respirator designs before assessing fit using human subjects as per CFR 1910.134.

MATERIALS AND METHODS

N95 Respirator Brands

N95 respirators from two Manufacturers (A and B) were selected, with one model from A, and two models from B. These three N95 respirator models, were chosen based on their popularity and availability and have been certified as N95s by the CDC. These models were also cleared by FDA through 510(k) clearance process for marketing within the U.S. and were authorized by the FDA for reuse after decontamination specifically during COVID-19 pandemic.

Experimental Set-up

To conduct the fit testing, an experimental chamber was set up, which included an aerosol generator, (TSI Inc. Model 8026), utilizing a sodium chloride solution with a concentration of 0.01 g/mL to generate the aerosol particles (Figure 1a). For evaluating the fit of the respirators, a TSI PortaCount (TSI Inc. Model # 8048) was employed in the N95 companion mode. In this mode, the PortaCount fit test measurements are truncated at 200. In our calculations this truncation was retained. The fit testing was performed on an additively manufactured medium-sized NIOSH head form that featured a 5 mm imitation skin layer (Smooth-On Ecoflex 00-20).

Figure 1.

a) Schematic of the experimental chamber for fit testing with upstream or downstream sampling of C_in, and C_out sampled in front of the respirator using the blue sampling line, using b) an elastomeric headform with a 5 mm skin thickness printed at the FDA.

The head form (Figure 1b) used in this study had previously been employed in a recent study to evaluate the sealing effectiveness and breathability of additively manufactured face masks (refer to the supplementary information at this link and Fogarasi et al. (2023) for additional fabrication information). Airflow sampling was controlled by using a mass flow controller (Alicat, Model # MCR-100SL/min-D) to maintain constant suction flow. To simulate human breathing and compare fit-testing results between constant suction flow and oscillatory flow, we used a QuickLung^® Breather breathing simulator (Ingmar Medical) that can provide instantaneous inspiratory and expiratory flow rates up to approximately 50 L/min as measured by an air flow meter (TSI Inc., Model # 5300-2). The long-term donning studies utilized three manikins that could each wear a respirator concurrently to parallelize efforts and reduce the experimental time.

Manikin Fit Factor Calculation

Aerosol concentration was measured outside (C_out) and inside (C_in) the respirator and the fit factor (FF) from both the normal and deep breathing experiments were calculated according to the normal and deep breathing exercises used in fit testing procedure by the OSHA (Equation 1) (OSHA 2021). An overall manikin fit factor was used to evaluate respirator fit (Equation 2) in which “n” represents the number of activities, “FF1” represents the fit factor of activity one, ”FF2” represents the fit factor of activity two, and so on. This definition is consistent with the approach described in the Occupational Safety and Health Administration (OSHA) Accepted Fit Test Protocol (OSHA 2021) and Bergman et al. (2012). In general, fit testing with human subjects includes 7 activities includes normal breathing, deep breathing, head side to side, head up and down, talking out loud, grimace, and bending over (OSHA 2021). In this study, for use with the manikin, we used n=2 activities in all cases, where the first activity simulated normal breathing and the second activity heavier breathing. Student t-test was used in Excel 365 (Microsoft Inc.) to evaluate significant difference in the overall Manikin FF with the FF results obtained on several human subjects across multiple studies that are referenced in the results and discussion section.

$$\mathit{Fit\; factor}(\mathit{FF})\text{=}{C}_{\mathit{out}}/C_{\mathit{in}}$$ (1)

$$\mathit{Overall\; FF}\frac{n}{\frac{1}{{\mathit{FF}}_1}+\frac{1}{{\mathit{FF}}_2}+\cdots }$$ (2)

In our studies various flow rates were used and they are detailed in the next section.

Differences incorporated into NIOSH manikin fit-test method

The following simplifications and or modifications were made to the NIOSH manikin for easier adaptation (Bergman et al. 2013, Bergman et al. 2015).

• In prior manikin fit-testing studies NIOSH used two flow rates for their breathing simulators – normal breathing (approximately 11.2 L/min) and deep breathing (approximately 20.4 L/min). However, since human subjects can easily breathe at higher flow rates particularly during moderate and heavy activities (Coyne et al. 2006), this study investigated the impact of various flow rates on the manikin fit-test for constant flow rates of 10, 20, 50, 70 and 85 L/min, and oscillatory flow rates of ±10 and ±50 L/min. In equation 2, manikin fit factor determined at 10 L/min was FF1, and FF2 would be one of the other higher flow rates.

• Currently, only few companies can fabricate the tissue mimicking variable thickness skin layer (DeGreef et al. 2006) that is used by NIOSH. To simplify, this study used an elastomeric head form with uniform 5 mm skin thickness. Fabrication details are available in supporting information.

• In fit-testing studies with human subjects sampling downstream to assess fit is not an option, and hence a probe is inserted to sample aerosols from within the respirator (here termed ‘upstream sampling’). The NIOSH manikin method mimics this sampling strategy by also sampling upstream (Figure 1). However, in addition to upstream sampling, we also investigated sampling from within the headform (‘downstream sampling’). For upstream sampling, a probe was inserted into the respirator (as described in the Supplementary information).

Decontamination of Respirators

We also performed fit-testing on decontaminated respirators to determine how well our method can predict real world evidence. Several approaches for decontamination are available (Greenawald et al. 2021). We investigated two approaches that can be readily implemented in hospitals without the need of complex and expensive decontamination systems –

• For Autoclave Decontamination the respirators were placed in individual autoclavable bags before being placed in the autoclave (Tuttnauer Steam Autoclave Model # L5596-1V) and decontaminated for 20 minutes at 121°C and 15 lb/in². Following the completion of the decontamination cycle, the respirators were removed from the autoclave and allowed to rest for 20 hours in the room air before being analyzed.

• For Microwave-Generated Steam Decontamination a 1000 Watt microwave (Panasonic NE-1054F) was used to heat an open glass container filled with 100 mL of distilled water. To ensure steam generation, the water was preheated in the microwave for 120 seconds. The respirators were placed in the microwave immediately after the water had been preheated. The respirators were then microwaved for 180 seconds before being removed from the microwave and rested for 20 hours in the room air before being analyzed.

RESULTS AND DISCUSSION

Long-Term Wear Cycles of the Respirator

FDA has not cleared a NIOSH approved N95 respirator for multiple donning/doffing cycles (MOU 2017). However, because of shortages, during COVID-19 public health emergency healthcare workers often needed to wear the respirators for several hours, across multiple work-shifts (Jacobs 2020, CDC 2020b). To mimic the impact of donning of respirators for multiple work shifts, we measured the FF first at 0 hours, and then left the respirator donned on the manikin before measuring FF sequentially at 6 hours, 24 hours, and 72 hours to mimic a typical work shift, two-three work shifts, or multiday donning. Table 1 shows the overall fit-factor for three respirator models. The following trends were observed with the data:

Table 1.

Overall FF of the respirators after long-term wear on the manikin; upstream and downstream sampling. The p-values represent the results of a student t-test conducted between the upstream and downstream FF’s. Overall FF was determined using equations 1 and 2, where FF1 was measured at 10 L/min, and FF2 at 70 L/min. NA = not applicable.

Time (h) of measurement	Brand A-Model A			Brand B-Model A			Brand A-Model B
	Upstream	Downstream	p value	Upstream	Downstream	p value	Upstream	Downstream	p value
0	200	200	NA	200	200	NA	200	200	NA
6	200	199 ± 1	0.42	200	200	NA	200	200	NA
24	200	192 ± 17	0.18	194 ± 8	186 ± 20	0.68	200	199 ± 2	0.36
72	198 ± 4	189 ± 15	0.20	174 ± 8	147 ± 27	0.03	200	195 ± 11	0.36

• Despite the changes to the manikin fit test method, we were able to achieve an overall FF of 200 for three of the NIOSH approved N95 respirators models both with upstream and downstream sampling. In this regard, note that a minimum overall fit factor of 100 is needed to pass a fit-test (OSHA 2021).

• Although not always statistically significant, the average overall FF for all models were lower when sampled downstream. Specifically, sampling from upstream resulted in an overall FF of less than 200 for 9% of the respirators. However, downstream sampling yielded an overall FF of less than 200 for 26% of the respirators, suggesting that downstream sampling was slightly more sensitive than upstream sampling. This trend is independent of the brand and model of respirator used.

• Some respirator models may be more prone to overall fit degradation with long term donning compared to other models.

Consecutive Donnings

There are multiple ways to optimize use of N95 respirators during shortage including repeated donning and doffing over several hours (in a single work shift) or over days (multiple shifts) without decontamination. We also tested for this situation. After measuring the overall fit factor after the first donning, we proceeded to doff the respirator and don it again for a total of 20 consecutive donning/doffing cycles while measuring the overall FF after every 5 donnings. For each respirator model, the measurements were repeated in triplicates.

In line with studies conducted on human subjects (Bergman et al. 2012, Vuma et al. 2019), the average overall manikin fit factor showed a gradual decrease as the number of donning/doffing cycles increased. This trend is illustrated in Figure 2, at flow rates of 10 and 70 (10-70) L/min and 10 and 85 L/min (10-85). Note that the human fit factor testing included the 7 activities that are required as part of CFR 1910.134, so some differences were expected. Curiously, overall fit factor at flow rates of 10 and 85 L/min yielded an average FF that was 8% greater than the FF observed in human subjects. On the other hand, fit testing at flow rates of 10 and 70 L/min resulted in an average FF that was 24% greater than the FF in human subjects implying that fit testing at the higher flow rate for FF2 of 85 L/min provided a better qualitative match with data obtained from human subjects.

Figure 2.

Overall manikin fit factor with number of donnings and doffings, respirator Brand B - Model A. The PortaCount maximum reported FF for N95 respirators is also shown. Overall FF was determined using equation 1, where FF1 was measured at 10 L/min, and FF2 at 70 L/min or 85 L/min.

A secondary validation of our method was obtained by recasting our results as percentage of fit tests with overall FF of greater than 100 (i.e. percentage of occurrences of passing the fit test) and comparing it with a similar study done by NIOSH on human subjects (Bergman et al. 2012) (Figure 3). It is important to note that our study averaged three different models of respirators, while the NIOSH data averaged six different brands. As seen in Figure 2, the 10-85 L/min overall FF at each donning and doffing more closely matches the human subject data than the overall FF at 10-70 L/min. Quantitatively, obtaining the percentage difference between each of our lab data points with the human subject data, and then averaging across all donning and doffing, the 10-85 L/min test data predicts an average mean percentage difference that is only 8 % from human subject data, compared to 10-70 L/min data which is 18 % higher than the human subject data. This suggests that overall manikin fit testing at higher flow rate results in a closer match with the overall fit factor obtained with human subjects. In addition, the fit-testing results with human subjects, and our manikin fit-test results, suggest that repeated donning and doffing does result in significant decline in fit, and hence should be exercised with caution.

Figure 3.

Percentage of the fit tests with an overall FF > 100 vs number of donnings and doffings.

Fit testing after Single Microwave Decontamination

We conducted overall manikin FF testing on the respirators after microwave decontamination. In this regard microwave decontamination had no visually perceptible impact on the morphology of the respirators. The results at 10-85 L/min, like the previous section, demonstrated a closer match with data from human subjects (19) compared to the 10-20 and 10-70 L/min data. The difference in average values was less than 1% for two models across different brands, and 13% for the third model (Figure 4) (Viscusi et al. 2011) from human subject data. Overall, the FF values for all tested models after decontamination were acceptable, with averaged FF values ranging from 164 to 200. These findings, combined with other studies that show filtration and breathability remain unchanged (Viscusi et al. 2007), demonstrate that decontamination with microwave generated steam is a viable procedure for the reuse of N95 respirators when dealing with N95 shortages.

Figure 4.

Overall fit factor after microwave decontamination for three different models (n = 5). Overall FF was determined using equation 1, where FF1 = 10 L/min, and FF2 was 20 L/min, 70 L/min or 85 L/min.

Fit testing after Single Autoclave Decontamination

Previous studies have reported physical deformation of the respirator following autoclave decontamination (Chen et al. 2020, Meisenhelde et al. 2020, Yuen et al. 2022). Consistent with these studies, and unlike the microwave method, autoclave decontamination visibly changed the morphology of the respirator (Figure 5). Autoclaved respirators exhibited physical damage, affecting the overall structure and shape of the respirators, and consequently, all tested respirator types failed quantitative manikin fit testing after a single cycle of decontamination, with average FF values below 100 (Figure 6). These findings align with previous reports on the degradation of most N95 respirator types following autoclaving (Kumar et al. 2020, Fischer 2020, Grinshpun et al. 2020). A study on physical changes in filter media after autoclaving (Yuen et al. 2022) did not show notable changes in micro-scale filter fiber conformation, suggesting that changes in respirator fit characteristics were likely due to changes in the straps or macroscale changes in respirator shape that occurred during autoclaving. Thus, autoclaving is not a consistently appropriate procedure for the decontamination of N95 respirators for reuse. A paired two sample for means t-test was conducted, revealing p-values of 0.23 and 0.07 for two models across both brands, indicating comparable results with those of human subjects. However, fit-testing at 10-20 L/min yielded significantly different results from human subjects (p-value of 0.0007) implying that when using Manikin fit test method, using low flow rates may lead to FF > 100 results that do not match with human testing.

Figure 5.

Morphology of the respirators; a) before, and b) after autoclave decontamination (arrows show noticeable areas of deformation).

Figure 6.

Overall manikin FF after autoclave decontamination for two different models (n = 5). Overall FF was determined using equations 1 and 2, where FF1 was measured at 10 L/min, and FF2 at 20 L/min, 70 L/min or 85 L/min.

The overall average FF for human subjects during consecutive donning and doffing, microwave, and autoclave decontamination methods, was 113, and the corresponding overall FF based on our modified methods were 140 when using FF1=10 L/min and FF2=70 L/min, a 14 % difference. This difference was reduced to an overall manikin fit factor of 121, only a 7 % difference when using FF1=10 L/min and FF2 = 85 L/min.

Fit testing using Breathing Simulator

NIOSH manikin fit-testing uses a breathing simulator. One of our goals was to evaluate if our assessments vary with the type of flow patterns used. To simulate realistic human breathing and compare fit-testing results between constant suction flow and oscillatory flow, we performed a limited study and utilized one popular respirator model (Brand B - Model A). In Figure 7, we present the overall fit factor values compared for oscillatory and constant flow rates. The fit factor values for oscillatory and constant flow rates were similar based on Student’s t-test (p> 0.05). While this may imply constant and oscillatory flows are comparable in producing similar overall FF, given the limited scope of this study, the results should be interpreted with caution.

Figure 7.

Overall fit factor of Brand B - Model A respirators, fit-tested using oscillatory and constant suction flow, for fresh respirator (control), after microwave and autoclave decontamination. Overall FF was determined using equation 1, where FF1 = 10 L/min, and FF2 = 50 L/min for constant suction flow rate, and FF1 = ±10 L/min, and FF2 = ±50 L/min for oscillatory flow rate.

Limitations

In this study, we focused on calculating the overall FF based on simulated normal (10 L/min) and deep breathing exercises (20, 70, and 85 L/min) as representative measures of respirator fit. While these exercises provide valuable information about the fit performance, it is important to acknowledge that real fit testing on humans typically includes additional exercises to assess the respirator’s fit in various scenarios (OSHA 1998a) (e.g. turning the head side to side, moving the head up and down, talking, etc.).

While the medium headform used in this study is designed to represent the average characteristics of the US workforce (Zhuang et al. 2010), it is important to acknowledge that human facial features can vary significantly among individuals. Different factors such as ethnicity, age, gender, and facial dimensions can impact the fit and seal of a respirator on different individuals. The use of a medium headform provides a standardized approach for evaluating respirator fit and allows for comparisons across different models and brands. However, we recognize that there can be variations in facial shapes and sizes leading to results that may not be fully captured by a single headform.

For reducing complexity in headform construction, we did not implement variable skin thickness. However, given that the results obtained despite these simplifications were in good agreement with results obtained from real world evidence such as human subjects testing, it appears that variable skin thickness may not play an important role in overall fit-testing measurements.

Although we did not perform extensive testing of overall FF across multiple models and cases with the oscillatory breathing simulator, our preliminary results suggest that implementing constant suction flow rate does not result in any loss of accuracy for measuring manikin FF. Future research over wider number of respirator brands and models may be necessary.

In case of shortage, another approach to meet the shortage in respirator supply would be to extend the use of new respirators through repeated donning and doffing, and then eventually decontamination, following which they would be reused. This aspect was out of scope of this study.

We simulated long-term wear up to 72 hours, but in real world situations it is unlikely that anyone can wear respirators for such an extended period of time. They are likely to don and doff it in between several times which may manifest somewhat differently than the consecutive donning scenario we studied. In addition, such long-term wear may result in soiling of the respirator and may affect its performance as well. This aspect of long-term wear with doffing and re-donning in between, and potential soiling was not investigated in this study.

Biocompatibility, filtration and breathability of the N95 respirators were outside the scope of this study.

CONCLUSIONS

Despite being simple, and having limitations, the modifications made to the NIOSH’s manikin fit-test method have resulted in a more sensitive method than the original for evaluating fit for N95 respirators. Overall, the modified method employed in this study offered a practical and cost-effective approach for manikin fit testing. The increased sensitivity observed in this method may be attributed to downstream sampling at high flow rates. With long-term wear we did not find a decline in fit even with 72 hour wear. However, these results need to be interpreted with caution as in a real-world scenario a long-term wear is likely to be limited to one or two work shifts (i.e. 8-16 hours). Regarding consecutive donnings, the study found that the FF of > 100 can be achieved at least up to five consecutive donnings. The study also demonstrated that microwave may be a viable option for single decontamination of N95 respirators, unlike autoclaving, which resulted in morphological changes and compromised fit.

Our modified method can be adopted by manufacturers for assessing the design of respirators or during future pandemics as a potential alternative to testing with human subjects. Expensive breathing simulators may also be avoided, since our results showed comparable fit factors using constant suction flow and oscillatory flow. Additionally, the successful validation of the manikin-based method in this study paves the way for future research and development of children’s manikins for assessing technical characteristics of pediatric facemasks.