Sensitivity and specificity of the user-seal-check in determining the fit of N95 respirators

Summary

N95 respirators are recommended by the World Health Organization (WHO) and the Centers for Disease Control and Prevention (CDC) to prevent the inhalation of droplets which may transmit respiratory pathogens. The reliability of N95 respirators in preventing transmission depends on their fit to the wearer. Quantitative fit testing (QNFT) is the gold standard used to determine this fit objectively. The manufacturers of the respirators also recommend performing a self-reported user-seal-check to detect for leakage. This study aims to investigate the capability of the user-seal-check in determining the fit of N95 respirators by investigating the sensitivity and specificity of the user-seal-check compared with QNFT. A prospective and cross-sectional research design was used. A total of 204 local Chinese undergraduate nursing students were recruited to test two commonly used respirator models (3M 1860S and 3M 1862). The results of the user-seal-check were compared with the results of the gold standard QNFT using the Condensation Nucleus Counter Fit Tester System. The sensitivity and specificity of the user-seal-check results obtained with the respirators were calculated. The results indicated low sensitivity, accuracy and predictive value of the user-seal-check in determining the fit of the N95 respirators. The user-seal-check was not found to be reliable as a substitute for QNFT. The results also suggested that the user-seal-check may be unreliable for detecting gross leakage. We recommend that QNFT is used to determine the fit of N95 respirators.

Introduction

Worldwide outbreaks of infectious respiratory diseases such as severe acute respiratory syndrome (SARS), avian influenza A, and the H1N1 influenza pandemic, have brought about greater awareness of respiratory protection for healthcare workers.1, 2, 3, 4

The World Health Organization (WHO) and the Centers for Disease Control and Prevention (CDC) have recommended the use of N95 respirators to prevent the spread of airborne infectious diseases.5, 6 However, the reliability of the N95 respirator to prevent such transmission depends on its fit to the wearer.⁷ In ill-fitting respirators, the average penetration by an ambient aerosol has been found to be 33% compared with 4% in well-fitting respirators.⁷ Such penetration may be caused by the gap between the respirator and the wearer’s face, which is termed leakage. This gap may allow leakage of airborne contaminants into the wearer’s breathing zone, leading to ineffective protection. Because of this potentially ineffective protection, the National Institute for Occupational Safety and Health (NIOSH) made fit-testing of N95 respirators compulsory for tuberculosis prevention before 2003. Both the CDC and the WHO recommended that fit testing should be carried out prior to the use of N95 respirators for SARS prevention.5, 6 Fit testing is now a mandatory measure for frontline staff working in infected areas in Hong Kong. During an epidemic of an airborne transmitted disease, there may be logistic difficulties in performing fit testing for all clinical staff.

According to the protocol from the United States Occupational Safety and Health Administration (OSHA), both quantitative fit testing (QNFT) and qualitative fit testing (QLFT) are the recognised methods to determine whether a respirator fits a wearer or not. Nowadays, QNFT rather than QLFT serves as the gold standard in determining the fit.6, 7, 8 QNFT is an assessment of the adequacy of fit by numerically measuring the amount of leakage into the respirator.⁸ Using an electronic device, the ratio of specific particles in the air inside and outside the breathing zone when wearing the respirator is measured and this ratio directly reflects the quantity of leakage. By contrast, the QLFT is a pass/fail fit test to assess the adequacy of fit that relies on the individual’s response to a test agent such as isoamyl acetate or irritant smoke near the wearer’s nose while wearing the respirator.⁸ QLFT only measures the presence of leakage or not. QLFT may include bias such as differences in the wearer’s sense of smell or uneven and immeasurable concentration of the test agents in the air.9, 10 QNFT is thought to be the more reliable method for fit testing, as reported in worldwide guidelines and research literature.11, 12, 13

N95 respirator manufacturers recommend that the wearer should carry out a user-seal-check. This is a standardised test performed by the wearer to detect gross leakage in a respirator. As suggested by some guidelines, no further QNFT is needed for a given respirator if leakage is detected by the user-seal-check.8, 14, 15 Furthermore, it has been suggested that a user-seal-check may be used as a substitute for QNFT if QNFT is not yet available. A previous retrospective study reported that the user-seal-check failed because its false-positive and false-negative rates in determining the fit of N95 respirators were too high.¹¹ There were limitations to this study including the retrospective design, differences in the number of each given models being tested, staff being able to select the respirator for testing and no standardised procedures for donning the respirators. Variations in the donning technique may have affected the reliability of the reported results because the technique of donning has been shown to correlate with leakage.16, 17 These limitations restrict the reliability of the findings. Therefore, this topic warrants further investigation with a prospective study design.

Methods

Design and participants

A descriptive, prospective, and cross-sectional research design was employed in this study. A sample of 204 local Chinese undergraduate nursing students was invited to participate. All the students were in year 2 with 18 weeks of clinical experience.

Setting

In order to minimise variation in the environment such as the concentration of suspended particles and dusts, all tests in this study were conducted in an assigned air-conditioned room with temperature set at 23 °C and humidity at about 75%.

Data collection

The demographic data of the participants (age and sex), the results of the user-seal-check, and the results of the QNFT with two given respirators were recorded in a data sheet. A standardised N95 respirator wearing protocol and guidelines for performing the user-seal-check (i.e. visual fit check, pressure–tightness test or negative/positive pressure check) were presented to the participants at the beginning of the study.7, 15

Conducting a user-seal-check

To conduct a user-seal-check, the wearer visually checks whether there is a gap between his/her face and the respirator. Then, the wearer forcefully inhales and exhales several times. The respirator should collapse slightly upon inhaling and expand upon exhaling. The wearer should not feel any air leakage between his/her face and the respirator. If any leakage is found, this is the sign of poor facial fit and a positive result of user-seal-check for the detection of leakage.

Conducting QNFT

The fit of the N95 respirator to the wearer was measured using a QNFT device. This was a PortaCount respirator fit tester system (Model Pro+ 8038, TSI Incorporated, St Paul, MN, USA). This system is based on a miniature, continuous-flow condensation nucleus counter (CNC) which is technology recognised by the OSHA to count air particles ranging from 0.02 to 1.00 μm in diameter.¹⁸ The use of the QNFT device followed the manufacturer’s guidelines and the protocol described by the OSHA.8, 19 Once the wearer has donned the respirator and QNFT system in accordance with the manufacturer’s guidelines, the wearer performs eight exercises as recommended by the system. Figure 1 shows the fit tester system, tubing connection and respirator. The eight exercises included normal breathing, deep breathing, side-to-side head movement, up-and-down head movement, talking (a standard set of passage was provided for reading), grimacing, bending over, and normal breathing. The grimace was to try and intentionally break the face seal to see if the respirator would reset itself afterward. Table I summarises the exercises and their duration. Each exercise was given a particular individual fit factor (FF; range: 0–200), except that of the grimace task. The overall FF was calculated using an equation in the protocol. This overall FF ranged from 0 to 200 and was the ratio of the concentration of a challenge agent outside the respirator to the concentration of the challenge agent that leaks inside the respirator (C_out/C_in). An FF of >100 indicates a ‘pass’ rating, which means a given respirator is a correct fit for the wearer.8, 19, 20 The PortaCount respirator fit tester system went through a daily check procedure in order to standardise the performance of machine and system.

Figure 1.

The fit tester system, tubing connection and respirator.

Table I.

Sequence of exercises used in quantitative fit testing

Exercises (duration)	Method
1. Normal breathing (60 s)	Remain still in a normal standing position.
	Breathe as usual.
2. Deep breathing (60 s)	Remain still in a normal standing position.
	Take long deep breaths as if working hard.
3. Side-to-side head movement (60 s)	Remain still in a normal standing position.
	Breathe normally while slowly turning the head from side to side between the extreme positions on each side.
	Take several seconds from left to right of each cycle.
	Pause momentarily at each side to take a breath.
4. Up-and-down head movement (60 s)	Remain still in a normal standing position.
	Breathe normally while slowly alternating between looking up at the ceiling and down at the floor.
	Take several seconds from up and down of each cycle.
5. Talking (60 s)	Remain still in a normal standing position.
	Read a prepared paragraph slowly and loud enough so as to be heard clearly by the research assistant.
6. Grimacing (15 s)	Grimace by smiling and/or frowning to try to create a leak in the respirator face seal.
7. Bending over (60 s)	Keep standing position.
	Bend at the waist as if touching the toes.
	Breathe normally.
8. Normal breathing (60 s)	Keep standing position.
	Breathe as usual.

Adapted from TSI Incorporated.¹⁸

The 3M 1860S and 3M 1862 N95 respirators were selected because they were widely used in local clinical settings. Also, a previous study found that they fitted the greatest number of intensive care unit (ICU) staff in Hong Kong.¹¹ All participants performed a standardised user-seal-check and QNFT with both 3M 1860S and 3M 1862 N95 respirators. It took 10 min to complete a single QNFT and 30 min to complete the entire process of data collection.

Ethical considerations

Ethical approval was obtained from the School of Science and Technology of The Open University of Hong Kong. The participants’ written consent was obtained prior to the data collection. The consent letter included the approval signatures, consent statement, the purposes of the study, right to confidentiality, right to withdrawal, and the duration of QNFT. The consent letters were distributed and explained to all participants by a trained research nurse.

Data analysis

Descriptive statistics were used to describe the participants’ demographic variables and the results of the user-seal-check and QNFT. The results of the user-seal-check compared with the gold standard QNFT were used to measure the sensitivity (the capability of the user-seal-check to correctly identify a case which failed QNFT) and specificity (the capability of the user-seal-check to correctly identify a case which passed QNFT). When considering establishing the user-seal-check as a substitute for QNFT, both high sensitivity and specificity are important.²¹ According to the evaluation of the performance characteristics of diagnostic tests in medical literature, it is generally agreed that the value ≥0.8 is considered to be ‘high’.22, 23, 24 χ²-Test was undertaken to test for the difference between participants in the two groups with regard to their demographic variables and QNFT results. The level of significance was set at P  < 0.05.

Results

A total of 204 nursing students participated in the study. They ranged from 18 to 23 years, and 21.6% of them were males (N  = 44).

The results of user-seal-check against that of the gold standard QNFT for each type of respirator (3M 1860S and 3M 1862) are presented in Table II . For the 3M 1860S respirator, 12.8% of the participants (N  = 26) found leakage on the user-seal-check. However, 38.7% (N  = 79) failed the QNFT. The sensitivity and specificity of the user-seal-check in determining the fitness of the 3M 1860S respirator were 15.2% and 88.8% respectively. For the 3M 1862 respirator, 15.7% of the participants (N  = 32) found positive results on the user-seal-check, but 42.7% (N  = 87) failed QNFT. The sensitivity and specificity of the user-seal-check in determining the fit of the 3M 1862 respirator were 23.0% and 89.7% respectively. Table III presents the detailed results.

Table II.

Summary of the results of user-seal-check compared with quantitative fit testing (QNFT; gold standard) (N = 204)

User-seal-check	Failed, ill-fitting respirator (by QNFT)	Passed, well-fitting respirator (by QNFT)	Totals
3M 1860S
Positive (detected leakage)	12	14	26
Negative (no leakage)	67	111	178
Totals	79	125	204
3M 1862
Positive (detected leakage)	20	12	32
Negative (no leakage)	67	105	172
Totals	87	117	204

Table III.

Results of the user-seal-check compared with quantitative fit testing showing sex (N = 204; male: 44; female: 160)

	3M 1860S			3M 1862
	Male	Female	Overall	Male	Female	Overall
Positive user-seal-check (%)	11.3	13.1	12.8	18.2	15.0	15.7
Fit-testing failure rate (%) (Prevalence of leakage)	27.3	41.9	38.7	27.3	46.9	42.7
True positive (TP)	2	10	12	3	17	20
False positive (FP)	3	11	14	5	7	12
False negative (FN)	10	57	67	9	58	67
True negative (TN)	29	82	111	27	78	105
Sensitivity or true positive rate (TPR) (%)	16.7	14.9	15.2	25.0	22.7	23.0
Specificity or true negative rate (TNR) (%)	90.6	88.2	88.8	84.4	91.8	89.7
False positive rate (FPR) (%)	9.4	11.8	11.2	15.6	8.2	10.3
False negative rate (FNR) (%)	83.3	85.1	84.8	75.0	77.3	77.0
Accuracy (ACC) (%)	70.5	57.5	60.3	68.2	59.4	61.3
Positive predictive value (PPV) (%)	40.0	47.6	46.2	37.5	70.8	62.5
Negative predictive value (NPV) (%)	74.4	59.0	62.4	75.0	57.4	61.1

Sensitivity or TPR = TP/(TP + FN); Specificity or TNR = TN/(FP + TN); FPR = FP/(FP+TN); FNR = FN/(TP + FN); ACC = (TP + TN)/(N); PPV = TP/(TP+ FP); NPV = TN/(TN + FN).

Of the participants, 37.3% (N  = 76) passed the QNFT for both respirators, and only 18.6% (N = 38) of the participants failed the QNFT for both respirators (data not shown). Therefore >80% of the participants passed QNFT with one or other respirator. There was no association between the passing QNFT for the 3M 1860S respirator and that of 3M 1862 respirator (χ²  = 1.57, P  = 0.134). The user-seal-check was no more accurate among females compared with males with regard to accuracy (ACC), i.e. 57.5% vs 70.5% for the 3M 1860S respirator, and 59.4% vs 68.2% for the 3M 1862S respirator.

There was no association between sex and QNFT results for the 3M 1860S respirators (χ²  = 3.10, P  = 0.055), whereas there was a difference between sexes for the 3M 1862 respirators (χ²  = 5.42, P  = 0.014). The male participants had higher pass rates than the female participants for both the 3M 1860S respirator (72.7% vs 58.1%) and 3M 1862 respirator (72.7% vs 53.1%).

Discussion

Data from CDC indicated that for any given N95 respirator, the failure rate of the QNFT ranged from 20% to 100% and recommended that QNFT should be carried out on each user to find a brand of respirator that achieves an adequate fit.⁷ Previous data indicated that a given N95 respirator may fit only 55–69% of ICU staff.¹¹ The pass rates of QNFT in this study were 57.4% for the 3M 1860S and 61.3% for the 3M 1862 and these results are comparable with previous reports. Performing the QNFT with two models of N95 respirators improved the overall pass rate of QNFT to 80%. However, the remaining 20% of the staff will need to be tested on further respirators. To conduct a QNFT on one N95 respirator using the PortaCount respirator QNFT system takes at least 10 min. Therefore, if three respirator brands needed to be tested for a user to find a respirator which fitted, 30 min would be needed.

During an epidemic, time constraints may make it difficult to perform the QNFT for all relevant healthcare personnel, so a more rapid, simple alternative test to QNFT would be useful.¹⁶ The user-seal-check is a simple, inexpensive, fast and self-manageable test that can be conducted anytime and anywhere.¹⁵ However, findings consistently indicate that this test is unable to serve as an effective alternative to QNFT because of its low sensitivity, accuracy, and predictive value.⁸ In some cases, users subjectively reported that the respirator fitted well, but the result of the fit factor was very low, and the overall QNFT subsequently failed. Such an observation suggests that leakage between the face and the respirator is not easily detected by the user.

The manufacturer recommends that users should perform a user-seal-check after donning the respirator in order to check for any gross leakage. In the current study the participants started QNFT immediately after the user-seal-check. A positive predictive value (PPV) estimates the likelihood that a person who finds a leakage through user-seal-check fails QNFT.²⁵ When applying the recommendation from Straus et al. in the current study, for a user-seal-check to be useful at ruling out a leakage it must have high sensitivity, and for it to be useful at confirming a leakage it must have high specificity.²⁶ The combination of high specificity and high PPV indicates a promising prediction for a user-seal-check to confirm a leakage.²⁵ However, the current findings of high specificity (89–90%) and low PPV (46–63%) of user-seal-check limit its capability to identify a case with leakage. This is consistent with the findings of Derrick et al. for half-face respirators and Delaney et al. for full-face respirators.11, 27 Thus the user-seal-check seems to be of limited value in detecting the leakage of facepiece respirators.

This study found that male participants obtained a higher pass rate (72.1% for both respirators) compared with females (58.1% and 53.1%) in the QNFT of N95 respirators. The accuracy of user-seal-check for males (68.2–70.5%) was higher than that for females (57.5–59.4%). These results are inconsistent with a previous study where the pass rate with 3M 1860S was 50% in males, and the accuracy of user-seal-check for males ranged from 50% to 66% in determining the fit of N95 respirators.¹¹ However, interpretation deserves attention because of small sample size in males (2 tested with 3M 1860S; 12 tested with 3M 1862).

One of the limitations of the current study is that only two models of respirators were used for QNFT. Although the aim was not to investigate the pass rate of different models of N95 respirators, it was possible that different results might have been obtained with different respirators. Nevertheless, we believe that the results support the low sensitivity, accuracy and predictive value of the user-seal-check in determining the fit of respirators.

In conclusion, the user-seal-check was not a reliable substitute for QNFT of a respirator and was inadequate for detecting gross leakage. We recommend that QNFT is used to determine the fit of N95 respirators to their wearers. We suggest that healthcare workers should perform QNFT routinely, prior to any outbreak as part of emergency planning.

Conflict of interest statement

None declared.

Funding sources

This study was supported by the School Research Fund (ST-09/10-1, R201-1) provided by the School of Science and Technology of The Open University of Hong Kong, Hong Kong Special Administrative Region.