Development of a Calibration Chamber to Evaluate the Performance of Low-cost Particulate Matter Sensors

Abstract

Low-cost particulate matter (PM) air quality sensors are becoming widely available and are being increasingly deployed in ambient and home/workplace environments due to their low cost, compactness, and ability to provide more highly resolved spatiotemporal PM concentrations. However, the PM data from these sensors are often of questionable quality, and the sensors need to be characterized individually for the environmental conditions under which they will be making measurements. In this study, we designed and assessed a cost-effective (~$700) calibration chamber capable of continuously providing a uniform PM concentration simultaneously to multiple low-cost PM sensors and robust calibration relationships that are independent of sensor position. The chamber was designed and evaluated with a Computational Fluid Dynamics (CFD) model and a rigorous experimental protocol. We then used this new chamber to calibrate 242 Plantower PMS 3003 sensors from two production lots (Batches I and II) with two aerosol types: ammonium nitrate (for Batches I and II) and alumina oxide (for Batch I). Our CFD models and experiments demonstrated that the chamber is capable of providing uniform PM concentration to 8 PM sensors at once within 6% error and with excellent reliability (intraclass correlation coefficient > 0.771). The study identified two malfunctioning sensors and showed that the remaining sensors had high linear correlations with a DustTrak monitor that was calibrated for each aerosol type (R² >0.978). Finally, the results revealed statistically significant differences between the responses of Batches I and II sensors to the same aerosol (P-value<0.001) and the Batch I sensors to the two different aerosol types (P-value<0.001). This chamber design and evaluation protocol can provide a useful tool for those interested in systematic laboratory characterization of low-cost PM sensors.

Capsule:

Through simulations and rigorous experiments, this study designed and evaluated a cylindrical chamber capable of providing uniform concentrations and robust calibration relationships for laboratory calibration of low-cost air quality sensors.

Graphical abstract

1. Introduction

Human exposure to particulate matter (PM) is a major public health issue, contributing to lung cancer, cardiovascular diseases and premature mortality (Brook et al., 2010; Jensen, 2006; Raaschou-Nielsen et al., 2013). In 2016, ambient exposure to PM_2.5 (PM with an aerodynamic diameter less than 2.5 μm) decreased average global life expectancy at birth by approximately one year with a greater reduction in life expectancy (approximately 1.2–1.9 years) in more polluted countries (Apte et al., 2018). PM_2.5 concentration has conventionally been measured by expensive regulatory monitors. Although these regulatory monitors are highly accurate, their high costs and maintenance requirements limit the number of monitors that can be deployed. Consequently, these regulatory monitors are sparsely distributed and cannot capture intra-urban PM_2.5 spatial variations resulting from local sources (Bell et al., 2010; Steinle et al., 2013).

The emergence of low-cost light-scattering PM sensors (<$500) has provided an opportunity to understand neighborhood-scale differences in air pollution. A large number of such sensors have been deployed in various community air quality networks, such as PurpleAir (2019), AirBox (Chen et al., 2017) and CAIRSENSE (Jiao et al., 2016). However, data quality is a key concern for these PM sensors (Rai et al., 2017). They are not as accurate or precise as regulatory monitors (Shapiro et al., 2014). These sensors may be affected by environmental factors, such as humidity (Wang et al., 2015) and temperature (Gao et al., 2015). They may also experience drift over time (Clements et al., 2017; Sayahi et al., 2019). Many studies have relied on manufacturers’ performance data rather than calibrating individual sensors under expected deployment conditions (Ali et al., 2015; Arvind et al., 2016; Du Plessis et al., 2016; Liu et al., 2015; Van den Bossche et al., 2015; Zhang et al., 2017). These studies assumed that their sensors’ default calibration factors are appropriate for the conditions of use and that their sensors do not exhibit intra-sensor variability. However, several studies have demonstrated significant intra-sensor variability both in the field and laboratory tests (Gao et al., 2015; Manikonda et al., 2016; Sayahi et al., 2019; Wang et al., 2015). Therefore, it is important to calibrate these inexpensive PM sensors before deployment because unreliable data may lead to unnecessary public alarm or complacency (Shapiro et al., 2014).

The performance of low-cost PM sensors is commonly characterized using laboratory calibration. This type of calibration typically exposes the sensors and a research-grade/reference monitor to PM inside a cubic chamber. Laboratory calibration allows the sensors to experience a wide range of concentrations and environmental conditions in a matter of hours (Sousan et al., 2016; Vercellino et al., 2018). Conversely, field calibration may require days or months, and even then, the sensors may not experience the target PM concentration range, PM source or environmental conditions. Moreover, it is logistically complicated to co-locate a large number of sensors with a reference monitor in the field for weeks or months.

Calibration chambers have been a crucial tool in assessing laboratory low-cost air quality sensors because of their ability to provide controlled conditions (Papapostolou et al., 2017). Numerous studies have evaluated PM sensor performance in laboratory chambers (Table S-1). Initial studies took place in cubic chambers using decay tests; in these tests, an elevated particle concentration is reached, and then particle generation ceases. Subsequently, PM concentration decreases over time (Austin et al., 2015; Wang et al., 2015). Although these chambers are relatively inexpensive to build, they have difficulty generating stable PM concentrations, making it challenging to evaluate whether a low-cost PM sensor is capable of precise measurements because the PM concentration in the chamber is constantly dropping in a decay test. The time delay between the responses of PM sensors and the reference monitor is another potential source of error when using a decay test (Liu et al., 2017). To address the stability issue, Papapostolou et al. (2017) developed a sophisticated environmental chamber (cubic shaped) that can maintain a stable concentration over a wide range of concentrations (0–300 μg/m³ with less than 3.1% standard deviation).

However, cubic chambers can have two disadvantages: particle loss at the edges and, more importantly, an uneven distribution of particles (or lack of understanding of this potentially uneven distribution). For example, Wang et al. (2015) and Hapidin et al. (2018) examined PM concentration uniformity by placing two low-cost sensors (same model) on different faces of their cubic chambers, and they suggested that PM concentration at these locations differed by less than 15% and 10%, respectively. However, their analysis relied on the assumption that their low-cost sensors’ responses exhibited no intra-sensor variability. Liu et al. (2017) developed a cylindrical chamber, in which the PM flows axially, and this type of chamber addresses the weaknesses of cubic chambers. They evaluated PM distribution in their chamber with seven tubes mounted at the bottom of the chamber where low-cost PM sensors are located. Sequential DustTrak measurements at those seven tube locations suggested that the spatial differences in concentration differed by less than 10%. However, this evaluation relied on the assumption that the PM concentration remained stable during the test. There is still a need for a cost-effective chamber that provides stable and uniform PM concentrations to the low-cost PM sensors as well as a strategy for evaluating spatial uniformity of particles within the chamber. Such a chamber will enable efficient evaluation and laboratory calibration of PM low-cost sensors.

This study presents the development of a cylindrical calibration chamber that allows for the calibration of 8 low-cost sensors at once using continuous testing. The distribution of PM concentration was examined experimentally and computationally, resulting in a robust strategy for assessing the spatial distribution PM concentration within the chamber. Finally, 242 Plantower PMS 3003 PM sensors from two different purchased batches were calibrated with this new chamber with a goal of identifying malfunctioning sensors and understanding intra-sensor variability.

2. Materials and methods

This study developed and evaluated a cost-effective calibration chamber that can provide multiple low-cost PM sensors with a uniform PM concentration, regardless of their position in the chamber. The calibration system is shown in Fig. 1. This study focused primarily on the chamber design and used commercially available particles to evaluate the low-cost sensors. Several publications have discussed the aerosol generation and dilution strategies (Castell et al., 2017; Li and Biswas, 2017; Liu et al., 2017; Papapostolou et al., 2017; Sousan et al., 2017). We used a Computational Fluid Dynamics (CFD) model to refine the chamber design before fabricating (described in Section 2.2). The CFD model and a rigorous experimental and statistical approach were applied to evaluate the spatial variability of PM concentrations within the chamber.

Fig. 1.

Schematic of the calibration system. Note the dimensions are not to scale.

2.1. Chamber design

The new chamber, with component costs of $692 (Table S-2), is capable of evaluating eight low-cost PM sensors at once and has an access port for a research-grade/reference monitor. It was designed with several key features to encourage uniform particle concentration at the inlets of the low-cost sensors (Fig. 1). These features included the cylindrical shape, the tangentially positioned inlet, the baffle, the outlet, and the anti-static design. The cylindrical shape limited the particle loss due to edge effects and provided a more uniform distribution of particles compared to conventional cubic chambers (Section 3.1). The inlet had turbulent flow (turbulence intensity of 4.51 %–5.16% which falls in the medium to high turbulence case, (ANSYS, 2019) with incoming and exit flow velocities of 1–1.5 m/s. The outlet of the chamber was placed in the center of the chamber to allow the aerosols that migrate to the edges of the chamber to flow over the sensors while exiting. The chamber, which was made of acrylic, also had an anti-static system, consisting of a helical pattern of grounded copper strips and wire (Fig. 1). An ohmmeter was used to verify all connections were sufficiently grounded. Generated aerosols can have high electrostatic charges, and the anti-static system limited particle deposits on the walls, therby reducing the need for cleaning (Tian et al., 2017).

2.2. Computational fluid dynamics evaluation

CFD simulations using ANSYS FLUENT software predicted the spatial distribution of aerosols within the cylindrical chamber and permitted the iterative refinement of the chamber design. Using the same conditions and chamber volume, this study also compared CFD simulations of aerosol concentrations for a cubic chamber. The CFD simulations employed a realizable K-ε turbulence model and Eulerian-based mixture model that treats each interpenetrating phase as a continuous fluid, where the individual particles are not tracked. This model provides accurate results when the concentration of one of the phases is low and can be assumed to be well mixed. The primary benefit of using this model was that it had a much lower computational overhead than traditional Eulerian-Lagrangian coupled simulations, where each particle must be individually tracked and all collisions are resolved (Tu et al., 2013). Table S-3 summarizes the simulation settings for the newly developed cylindrical chamber and a cubic chamber with the same volume.

2.3. Experimental evaluation

The experiments included three types of tests (stability, repeatability, and rotation, described in Section 2.3.3) to assess the spatial distribution of PM within the chamber. After the chamber evaluation, this new chamber was used to evaluate the performance of 242 low-cost sensors compared to a research-grade instrument (TSI DustTrak 8530) and to identify malfunctioning sensors as well as to understand inter-sensor variability before deploying them in the field.

2.3.1. Experimental system

The laboratory calibration system consisted of a particle generator, a diffusion drier (if needed), a dilution system, and the newly designed cylindrical calibration chamber with 8-low cost PM sensors and a research-grade monitor (Fig. 1).

Aerosol generation.

Two different types of particles, alumina oxide and ammonium nitrate, were utilized for calibrating the low-cost sensors. Alumina oxide represented a dry and non-cohesive dust, and this was generated by a Palas solid particle dispenser (RBG 1000-C). The alumina oxide particles have a density of 4000 kg/m³ (Duralum, Washington Mills, Niagara Falls, NY, USA) and a mean mobility diameter of 470 nm (Fig. S-1-A). Ammonium nitrate is a key component of PM_2.5 concentration in Salt Lake City where the sensors are being deployed. During winter in Salt Lake City, ammonium nitrate contributes more than 50% of PM_2.5 mass (Baasandorj et al., 2018; Kelly et al., 2013; Sayahi et al., 2019). The 95% pure ammonium nitrate has a density of 1700 kg/m³ (Alfa Aesar, ThermoFisher SCIENTIFIC, ACS) and a mean mobility diameter of 137 nm (Fig. S-1-B). The ammonium nitrate particles were generated with a single-jet atomizer (TSI model 9302) and clean dry air. A building compressor provided the clean, dry air, which was further cleaned using an Enmet air filtration panel (5 μm, Ann Arbor, Michigan) that included a water collector bowl, a coalescer filter, a charcoal absorber, and an adsorber filter. Before introducing the aerosol flow to the calibration chamber, a diffusion dryer (TSI 3062) filled with dry silica gel reduced the humidity of the aerosol stream. The range of relative humidity inside the chamber during the calibration tests was 7.3% to 9.9%, measured with an Omegaette temperature/humidity meter (Omega Engineering, model HH314). The temperature of the chamber was similar to that of the laboratory, which ranged from 21.7 °C to 25.1 °C (measured with HH314). Some studies have reported that low-cost PM sensors may be affected by environemtal factors such as humidity (Wang et al., 2015) and temperature (Gao et al., 2015), but several other studies found negligible correlation between the sensors’ responses and those factors (Han et al., 2017; Holstius et al., 2014; Kelly et al., 2017; Sayahi et al., 2019). The small differences in relative humidity and temperature during the calibration tests were unlikely to have affected the results although as discussed in Section 3.6 the addition of temperature and humidity control would be beneficial for systemically evaluating these environmental factors.

Research-grade monitor.

A TSI DustTrak 8530 (Rivas et al., 2017) was placed directly below the calibration chamber, and a representative sample from the chamber flowed directly into the DustTrak (3 L/min) via conductive tubing positioned at the height of the low-cost sensors’ inlets. The DustTrak was equipped with a PM_2.5 inlet, and its mass concentration estimate was corrected for each type of aerosol with a calibration factor, developed from co-located measurements with an AirMetrics MiniVol, which is a filter-based monitor. The MiniVol was operated with a PM_2.5 inlet and a flow rate of 5 L/min. The MiniVol’s filter was pre- and post-weighed, in triplicate, to obtain an average mass concentration, which is used to develop a correction factor for each aerosol type. The raw DustTrak readings were divided by an appropriate correction factor for each aerosol type (1.78 or 0.959 for ammonium nitrate and alumina oxide, respectively).

Low-Cost Sensors.

The Plantower PM sensors (PMS 3003) were evaluated in this study (Fig. S-2). In this light-scattering based sensor, a fan with a rate of 0.1 L/min routes air past a tightly focused laser beam. Each particle passes through two 90-degree turns before passing the laser. The laser wavelength is 640 ±10 nm, measured with a Lambda 35 spectrophotometer (PerkinElmer, Inc). The manufacturer reported that the sensor’s response time is less than 10 seconds and its mean time to failure is more than three years. They also reported that the sensor’s effective PM_2.5 concentration range is 0~500 μg/m³ and that the sensors have an accuracy of ±10 (μg/m³ (for concentrations between 0~100 (μg/m³) and ±10% (for concentrations between 100–500 (μg/m³).

The PMS sensor was integrated into an in-house sensing system called AirU. In addition to the PMS 3003 sensor, AirU comprises a custom-printed circuit board, a Texas Instruments HDC1080 temperature and humidity sensor, and a MiCS-4514 reduction/oxidation gas sensor. The PMS3003 sensors were connected directly to the AirU circuit board, which in turn was powered through an FCC-certified 120V AC input, 5V DC output power adapter with a 1 amp current rating. The PMS3003 draws 60mA, and the total current draw of the AirU (including the PMS3003) is 120mA. The PMS3003 sensor directly received this 5V source with minimal additional filtering supplied by decoupling capacitors, which were also mounted on the AirU circuit board. PM measurements were sent once a second from the PMS3003 to the AirU. These measurements were accumulated, and every 15 seconds the sum of the measurements was divided by the number of measurements for the time period. The result was sent to a private database over WiFi. No data filtering was applied prior to the data collection/storage. The PMS sensors were placed in a 56.1 mm diameter, circular sensor holder within the chamber. Each sensor was placed at 45-degree intervals along the circular holder (Fig. S-3), and the AirU boards were located outside the chamber (Fig. 1).

2.3.2. Calibration test protocol

For each sensor calibration test, the calibration chamber was allowed to reach a steady-state target concentration based on DustTrak readings, and measurements at each concentration were collected for 10 min. The calibration test for all 242 low-cost sensors included 5 targets ranging from a PM_2.5 concentration <5 μg/m³ (except the stability test, Section 2.3.3) to approximately 150 μg/m³ (except Section 3.5). Finally, the performance of eight sensors was also evaluated at 10 steady-state concentrations ranging from <5 μg/m³ to 1000 μg/m³ in a continuous test.

Both the DustTrak and the PMS sensors collected data at 15-seconds intervals. The data were first time-matched and then averaged over one minute. A linear model was individually fit to the minute-by-minute data of DustTrak and each sensor.

2.3.3. Robustness of experimental calibration

The following three tests provided an estimate of particle distribution inside the newly designed calibration chamber.

Stability test.

The stability test was performed once at 5 steady-state target concentrations (10 min at each concentration), using DustTrak measurements (Table 1). The stability of the chamber at each concentration step was assessed using the standard deviation (SD) of the DustTrak measurements and the relative percentage of standard deviation (%RSD):

$$\%RSD=\left(\frac{SD\left(Re{f}_{ss}\right)}{\overline{Re{f}_{ss}}}\right)\times 100$$ (1)

Table 1.

Stability test results for ammonium nitrate and alumina oxide aerosols using SD and %RSD (Eq. 1) parameters for five target concentrations.

	Ammonium nitrate			Alumina oxide
	Target concentration μg/m3	SD μg/m3	%RSD	Target concentration μg/m3	SD μg/m3	%RSD
State 1	8.69	0.506	5.82	12.5	0.654	5.57
State 2	39.4	1.95	2.43	57.0	3.58	6.29
State 3	80.2	2.15	5.45	96.2	7.73	8.03
State 4	101	3.63	2.33	140	8.74	6.26
State 5	156	4.08	4.06	175	9.04	5.17

States 1–5 are the 5 steady-state concentrations for each stability test.

where ${\overline{Ref}}_{ss}$ is the average of the DustTrak readings at each steady-state concentration.

Repeatability test.

To evaluate the ability of the chamber to generate reproducible linear calibration curves for each of the sensors, the 5-point calibration (described in Section 2.3.2) was repeated three times for eight low-cost sensors. We used the intraclass correlation coefficient (ICC), a widely used reliability index in test-retest analyses, to examine the repeatability (Cicchetti, 1994). More specifically, for each low-cost sensor, we calculated the proportion of residual variance after calibration, which was not caused by the repeated experiments. The calculated ICC ranges from 0 to 1. A high ICC index indicates a lower impact of repeated experiments on the calibration test and hence better repeatability in the calibration model. We used the cluster bootstrap approach to calculate the 95% confidence interval of ICC for each sensor. We then followed the guideline developed by Cicchetti (1994), and interpreted the ICC values less than 0.4, between 0.4 and 0.59, between 0.60 and 0.74, and greater than 0.75 as poor, fair, good and excellent, respectively.

Rotation test.

The rotation test complemented the repeatability test and examined the robustness of the calibration lines for each sensor at different positions in the chamber. In the rotation test, eight sensors were evaluated once at four different positions. In other words, the test was repeated four times, and in each test, the sensors were rotated 90-degrees in relation to the center of the chamber (Fig. S-3). In this test, instead of reporting the concentration differences of different sensors (of the same model) at various positions as an indication of the particle distribution (Hapidin et al., 2018; Wang et al., 2015), each sensor was compared to its own calibration data in the different positions. This method eliminated the errors associated with intra-sensor variability. We used the same ICC approach in the reliability test, discussed above, to examine the impact of sensor location/rotation.

2.3.4. Calibrating Plantower PMS 3003 sensors

This study calibrated 242 Plantower PMS 3003 sensors from two different production lots (Batches I and II). The sensors had consecutive numbers (PMS 1- PMS 242). Batches I and II included 154 (PMS 1-PMS 154) and 88 sensors (PMS 155-PMS 242), respectively. They were purchased from Beijing Plantower Co., Ltd, in June 2017 and August 2018, respectively. Batch I was evaluated using both ammonium nitrate and alumina oxide aerosols; however, Batch II was evaluated only using ammonium nitrate particles. This calibration provided the slope, coefficient of determination (R²), intercept, and bias for all 242 sensors. Moreover, this study assessed four low-cost sensors from each batch to investigate the precision at five steady-state concentrations. In addition, the pairwise Spearman’s correlation coefficient (SCC) was used to evaluate the strength and direction of the correlation between each pair of PMS sensors for each test of eight sensors for both Batch I and II. The SCC ranges from –1 to 1, where the closer the SCC is to ±1, the stonger the association. SCC values greater than 0.80 indicate a very strong association (Daniel, 1990).

It should be noted that the performance criteria for low-cost sensors do not yet exist, but other studies have used these same metrics (slope, intercept, R², bias) for their evaluations of low-cost PM sensors (Sousan et al., 2016, 2017). In addition, the Environmental Protection Agency (EPA, 2006, 2018) also uses these metrics as well as the coefficient of variation (CV) in their criteria for continuous PM_2.5 monitors and candidate equivalent methods, and we discuss these criteria for comparison purposes.

The PMS sensor bias was calculated as the mean of the ratio of each PMS sensor measurement at time t (PMS_t) to the corresponding reference reading at time t (Ref_t):

$$Bias=mean\left(\raisebox{1ex}{$PM{S}_t$}\!\left/ \!\raisebox{-1ex}{$Re{f}_t$}\right.-1\right)\times 100$$ (2)

Ref_t measurements less than 1 μg/m³ were excluded from calculating sensor bias.

The precision of the four low-cost sensors from each batch (a total of eight sensors) was assessed at four steady-state concentrations (PMS_ss) in comparison to a reference monitor using the CV:

$$CV=\left(\frac{SD\left(PM{S}_{\mathit{ss}}\right)}{\overline{R{f}_{ss}}}\right)\times 100$$ (3)

where${\overline{\mathrm{Ref}}}_{\text{ss}}$is the average of reference monitor at each steady-state concentration.

3. Result and discussion

3.1. CFD simulation

This analysis investigated the ammonium nitrate particle concentration distribution in the cylindrical chamber at a flow rate of 15 L/min. The boundary conditions and the settings for the CFD model can be found in Table S-3. This model was tested for particle sizes up to 10 μm with no appreciable change in results. This CFD analysis is valid for Knudsen numbers significantly less than one. Fig. S-4 shows the fraction of the volume occupied by the particles in the overall control volume (the chamber) as time progresses. It takes the chamber about 10 minutes to reach steady-state. Fig. 2 represents the volume fraction profiles of the particles at the center of the sensors’ inlets within a ± 10 mm radial margin from the centerline (called inlet location, Fig. S-5). The two smaller concentric circles in the figure represent the inlet of the DustTrack. The maximum concentration difference among all the sensor positions at the inlet location is 5.50% (Table S-4). This indicates that the cylindrical chamber is capable of providing a uniform concentration at the inlet location within 6% error. The difference in concentration on one side of the chamber is due to the inlet position and angle of the inlet. The volume fraction is lowest at the point where the DustTrak inlet is located. One potential reason is that the DustTrack inlet pulls a negative pressure while the rest of the chamber is at atmospheric pressure.

Fig. 2.

Volume fraction distribution at the inlet location, a ±10 mm radial margin from the centerline of the sensors’ inlets. The two smaller concentric circles in the figure represent the inlet of the DustTrack. The larger concentric circle indicates the chamber outlet.

This analysis also provided a comparison between a cubic chamber and a cylindrical one at the same conditions. The CFD simulation settings and boundary conditions for the cubic chamber are summarized in Table S-3. Fig. S-6 shows the volume fraction profiles at the six walls of the cubic chamber on which the sensors were assumed to be located. Comparing the concentrations at the center of each face of a cubic chamber showed that the smallest difference was 7.48% and the largest difference was 16.4% (Table S-4). Concentration differences could be even larger than 16.4% if one compared a center point on one face with a different location on another face (not the center). The results indicate that this cylindrical chamber can provide a more uniform distribution of particles than a cubic chamber.

3.2. Effect of sensor location on calibration

Stability, repeatability and rotation experiments were performed to evaluate the robustness of the sensor calibrations to position inside the newly designed chamber.

3.2.1. Stability test

Table 1, Figs. 3 and S-7 show the results of the chamber stability test for five different concentrations of ammonium nitrate and alumina oxide. The shaded area in the figure shows the transition between the steady-state concentrations. The SD and %RSD ranges for all target concentrations during the stability test of the new chamber were low (Table 1) and in the same range as those reported by Papapostolou et al. (2017) (SD=0.2–4.3, %RSD=1.4–3.1%). These low SD and %RSD indicate that the chamber system has very good stability over the tested range of concentrations. As shown in Table 1, higher PM_2.5 concentrations showed larger standard deviations and relative standard deviations. The same observation was reported by Papapostolou et al. (2017). It should be noted that %RSD is independent of concentration, making %RSD easier to compare with other studies (Eq. 1, Table 1).

Fig. 3.

Concentration time-series for the ammonium nitrate stability tests.

Papapostolou et al. (2017) discussed that the stability of PM concentrations in a chamber is mainly due to two factors: a) the ability of the particle generator to provide aerosol concentrations, and b) the uniform distribution of particles inside the chamber. In this study, another source of error in creating a consistent aerosol concentration could be the fluctuations in the clean air flow, which was provided by the building’s compressor. Our results indicated that the Palas RGB-C’s aerosol generation was less stable than the TSI single-jet atomizer, as evidenced by the larger variations in the alumina oxide tests. In order to reduce other sources of error and focus on the ability of the chamber to provide uniform PM concentrations, the repeatability and the rotation tests (Section 3.2.3) were only conducted with ammonium nitrate particles.

3.2.2. Repeatability and rotation tests

Table 2 summarizes the ICC index values calculated for eight sensors (i.e., PMS1-PMS8, Fig. S-2) compared to the DustTrak. The ICC values for repeatability and rotation tests ranged from 0.961 to 0.999 and from 0.771 to 0.851, respectively, which indicate excellent reliability of the calibration tests regardless of the repeat number or the position of the sensors inside the chamber. For each sensor, the ICC value for the repeatability test was consistently higher than that for the rotation test. The estimated uncertainty (i.e., the 95% CI) for repeatability was also consistently smaller than that for the rotation test. Although the impact of rotating the sensors inside the chamber on the linear calibration model was greater than the impact of repeating the experiment, the ICC index values (>0.75) of the sensors show that the calibration model results are robust to both the repeatability and rotation tests.

Table 2.

Results of repeatability and rotation tests.

	Repeatability		Rotation
	ICC	95% CI	ICC	95% CI
PMS1	0.987	(0.938, 1.000)	0.771	(0.528,0.964)
PMS2	0.965	(0.916, 0.999)	0.851	(0.647,0.997)
PMS3	0.999	(0.981, 1.000)	0.823	(0.555, 1.000)
PMS4	0.999	(0.978, 1.000)	0.809	(0.655, 0.968)
PMS5	0.999	(0.967, 1.000)	0.838	(0.649, 1.000)
PMS6	0.961	(0.841, 1.000)	0.833	(0.666, 0.998)
PMS7	0.985	(0.928, 1.000)	0.790	(0.597, 1.000)
PMS8	0.995	(0.978, 1.000)	0.843	(0.595, 0.977)

The ICC approach was also used to describe the repeatability of each sensor’s response compared to itself in three repeated tests (within-sensor variability) and the repeatability of one sensor compared to the other seven sensors during the same test (between-sensor variability, three tests). The results showed that the ICC for within-sensor variability reliability ranged from 0.955 to 0.980, and the ICC for between-sensor variability ranged from 0.969 to 0.991. These ranges of ICCs are indicative of excellent reliability (Cicchetti, 1994).

3.3. Effect of aerosol type

The evaluation of the PMS 3003 performance for two different particle types (alumina oxide and ammonium nitrate) began with a PM_2.5 time-series evaluation of the DustTrak and four PMS sensors at five steady-state concentrations (Fig. S-8). The figure illustrates that the PMS sensors track the DustTrak PM_2.5 concentrations. As shown in Fig. S-8-A, the PMS sensors from Batch I overestimated ammonium nitrate PM_2.5 concentrations compared to the DustTrak but underestimated alumina oxide concentrations compared to DustTrak (Fig. S-8-B). Kelly et al. (2017) also reported that the PMS 3003 underestimated alumina oxide PM_2.5 concentrations in a wind-tunnel test. Many studies suggested that the differences in the response of light-scattering based sensors to different aerosol types can be attributed to variations in aerosol optical properties (Kelly et al., 2017; Liu et al., 2017; Sousan et al., 2016; Wang et al., 2015).

A linear model was fit to each of the 242 sensors based on the 5-point calibration curve. Fig. 4 shows the distribution of the slopes, intercepts, and goodness of fit metrics (R²) of these fits as well as bias measurements. For both aerosol types, the PM_2.5 readings of Batch I PMS sensors were highly correlated with the reference instrument, R² of 0.995–0.999 for ammonium nitrate and R² of 0.978–0.998 for alumina oxide. Clements et al. (2017) suggested that low-cost sensors can complement existing monitoring networks and enhance the spatial coverage of PM data if they have R² >0.4 with a federal reference/equivalent method. As discussed in Section 2.3.4, performance standards for low-cost sensors do not yet exist; for comparison purposes, we discussed EPA criteria for continuous PM_2.5 monitors and candidate equivalent methods. The correlations between the PMS sensors and the DustTrak meet the EPA’s criteria for continuous PM_2.5 monitors (r>0.9 or R²>0.81; EPA, 2018) and for candidate equivalent methods (r>0.97 or R²>0.94; EPA, 2006). However, the slopes of the linear model were not generally within the EPA’s criteria for continuous PM_2.5 monitors (1 ± 0.1; EPA, 2018) and varied significantly for the two different types of aerosols (Student t-test, p-value<0.001) from 0.414–0.645 (underestimation for alumina oxide) to 0.988–1.36 (overestimation for ammonium nitrate). The average of intercepts of all Batch I sensors were −1.37 μg/m³ (range of −3.88–2.47 ng/m³) and − 1.94 μg/m³ (range of −5.845.05 Hg/m³) for ammonium nitrate and alumina oxide, respectively. The small intercepts for the Batch I sensors for both aerosol types were generally within the EPA (2006) acceptable range for candidate equivalent methods (± 5 μg/m³).

Fig. 4.

Frequency distribution of slopes, intercepts, coefficients of determination, and bias for 154 Batch I sensors for calibration with ammonium nitrate (A) and alumina oxide (B) aerosols.

Note that the scales in the x-axes are different.

The average bias values for all the Batch I sensors for ammonium nitrate and alumina oxide were 9.53% and 49.6%, respectively. The small positive bias for Batch I sensors with ammonium nitrate shows that the difference between the sensor and the reference instrument is small and within the EPA (2018) criteria for continuous PM_2.5 monitors (± 10%). However, the large negative bias for alumina oxide exceeds EPA (2018) criteria. It is also an indication of the underestimation of PM_2.5 concentrations. Sousan et al. (2017) also reported different bias values for the same sensor (Footbot) in response to different aerosol types (−12% for Arizona air dust and < −46% for salt and welding fume).

The precisions of the four representative PMS sensors were between 1.96% and 10.2% (average of 5.06%) for ammonium nitrate and between 1.17% to 5.91% (average of 3.65%) for alumina oxide. The low precision values reflect the ability of the sensors in replicating the concentration measurements (NIOSH, 2012). These CVs are within EPA’s acceptable measurement uncertainty for continuous PM_2.5 monitors (CV < 10%, Table S-5; EPA, 2018).

3.4. Comparison of Batches I and IE sensors

Comparing the response of Batches I and II sensors revealed that Batch I slightly overestimated ammonium nitrate PM_2.5 concentrations (mean slope = 1.16 and bias = 9.53 %), but Batch II slightly underestimated ammonium nitrate PM_2.5 concentrations (mean slope = 0.919 and bias = − 4.52 %, Fig. S-9). As with Batch I, the Batch II sensor responses were highly correlated with the DustTrak (R² of 0.968–0.999). These slopes and correlations meet EPA (2018) criteria for continuous PM_2.5 monitors. All of these results in the following sections excluded the two malfunctioning sensors, which are further discussed below. As shown in Fig. S-10, the slopes of Batch I sensors differed significantly from those of Batch I (Student t-test, p-value<0.001). This result indicates that sensors purchased at different times could have different responses. However, the pairwise SCCs showed very strong positive correlation for each pair of eight sensors in the same test (>0.972 for Batch I and >0.978 for Batch II). Table S-6 shows the pairwise SCCs for one representative test from each batch.

Eight sensors from Batch II have much lower slopes than the mean slopes (0.5 to 0.7) and biases (−50 % to −20 %) although they still have high R² values (0.995–0.997). This means that these sensors underestimate the PM_2.5 levels more than the other sensors. However, they could still be deployed in the sensor network using proper correction factors. These results provide indications of intra-sensor variability and the importance of calibration before field deployment of the sensors. Possible explanations for the different responses of the batches include manufacturer modification of their software that converts the signal to PM concentration or manufacturing differences in the photodetector.

One of the goals of laboratory evaluation was to identify defective sensors and avoid deploying them in the field. The malfunctioning sensors were identified by a poor coefficient of determination (R² < 0.3, Moore et al., 2013). Two out of 242 evaluated sensors were classified as malfunctioning (PMS 163 and PMS 242). Neither sensor tracked the PM_2.5 concentration measurements of DustTrak. Sensor PMS 163 readings ranged from 0–24.3 μg/m³ with R² of 0.018 (Fig. S-11) while sensor PMS 242 constantly read zero with R² of NaN (not a number), even though both sensors were exposed to PM_2.5 concentrations up to 150 μg/m³. Both sensors were dismantled to find possible reasons for their behavior. Nothing suspicious was found in sensor PMS 163; its low readings could be due to laser misalignment or a manufacturing defect in the sensor’s laser, photodetector or electronics. Sensor PMS 242’s laser chamber was filled with small pieces of white paper (Fig. S-12) preventing the laser light from reaching the particles or the photodetector.

3.5. Sensor behavior at higher concentrations

Exposing eight PMS sensors to ammonium nitrate PM_2.5 concentrations up to 1000 (μg/m³ showed that the sensors maintained a linear response up to a concentration of 150 μg/m³ (R² of 0.995–0.999, Fig. S-13). However, the linearity deteriorated at higher concentrations. The exact physical reason behind the non-linear relationship of PMS 3003 sensors and the reference instrument is unclear. A second-order polynomial fit all the data (R² of 0.998–0.999, range: 5 – 1000 μg/m³) better than a linear model (Fig S-14). This non-linear behavior of low-cost PM sensors above a certain concentration was also seen in other laboratory studies (Austin et al., 2015; Hapidin et al., 2018; Sousan et al., 2017; Wang et al., 2015).

3.6. Limitations

The focus of this paper was to develop, evaluate, and demonstrate a cost-effective cylindrical chamber for calibration of low-cost PM sensors. A number of refinements would enhance the usability of this system. For example, using this chamber to calibrate hundreds of PM sensors can be time-consuming. An improved experimental design could streamline the calibration procedure and/or potentially reduce the number of data points needed for sensor calibration. Adding a neutralizer after the aerosol generation system could prevent particle deposits in the lines prior to the chamber; however, this would significantly add to the cost. Using copper mesh instead of copper strip in the anti-static system is another potential improvement. It would also be beneficial to integrate temperature and humidity controls into the system for future studies because some investigations have shown that temperature and relative humidity can affect the responses of some low-cost sensors (Gao et al., 2015; Wang et al., 2015).

A laboratory evaluation system is well suited for calibration of sensors prior to field deployment, and it can complement, but not replace, field calibration. Aerosols in the real word are heterogenous, and their properties vary over time. A one-time lab calibration cannot represent the range of real-world conditions that a sensor may experience, and it cannot ensure that measurements are accurate in the real world. However, it does allow one to evaluate low-cost sensor performance (i.e., intra-sensor variability, response over relevant concentration ranges), and aids with quality assurance and quality contorl concerns, such as identifying malfunctioning sensors. A mixture of lab and field calibrations along with advanced statistical techniques is a likely path forward (Johnson et al., 2018; Becnel et al., 2019.; Zimmerman et al., 2018).

4. Conclusion

This study developed and evaluated a cost-effective calibration chamber that provided spatially uniform concentrations to multiple PM sensors and calibration relationships that were robust to sensor position in the chamber. The design and evaluation relied on a CFD model and a rigorous experimental evaluation. The computational model showed that the chamber is capable of providing a uniform PM concentration to calibrate eight sensors at one time within 6% error, and the experimental results demonstrated the robustness of the calibration model to the sensor position within the chamber with excellent reliability (ICC>0.771). This new chamber was then used to evaluate the performance of 242 Plantower PMS 3003 sensors from two production lots using two different particle types: ammonium nitrate (for Batches I and II) and alumina oxide (for Batch I). The results identified two malfunctioning sensors and demonstrated that all the sensors (except the two malfunctioning sensors) were highly correlated with the DustTrak reference monitor (R²>0.978). The study also identified significant response differences between Batches I and II. As in several other studies, the PMS sensors exhibited a statistically significant difference in their responses to the two different aerosol types. This calibration chamber was able to identify manufacturer differences in PM response and malfunctioning sensors. This chamber can also be used to complement field observations and help to eventually determine whether good laboratory performance of a low-cost PM sensor indicates its good performance under real-world conditions.

Highlights:

• A calibration chamber was developed and evaluated through CFD simulations and experiments.

• This chamber provides uniform PM concentrations to 8 sensors at once.

• This chamber produces repeatable calibration curves, regardless of sensor position.

• 242 Plantower PMS 3003 sensors were evaluated using this chamber.

• This chamber can identify malfunctioning sensors and intra-sensor variability.