Sources of Variability in Real-time Monitoring Data for Fine Particulate Matter: Comparability of Three Wearable Monitors in an Urban Setting

Abstract

The increasing availability of portable air pollution monitoring devices has greatly enhanced the ability to measure personal exposures in real time. However, these devices vary considerably in their cost and specifications, and questions remain as to their reliability and practicality for use in epidemiological investigations. In this field study, three personal PM_2.5 exposure monitors (two nephelometers, one optical particle counter) were compared in an urban setting to assess their feasibility for use in future studies. In total, 3963 1-min measurements were collected over 12 days from locations of several types (e.g., above and below-ground subway stations, sidewalks next to urban traffic, outdoor construction sites) in the Washington, D.C. metropolitan area. Overall, we observed moderate-to-high agreement in pairwise comparisons of PM_2.5 concentrations between devices (R² range: 0.37 to 0.75). Bland-Altman plots showed that differences in device agreement varied over the range of mean concentrations. In linear mixed models adjusting for temperature and relative humidity, we saw significant interaction between device and location (p<0.05), suggesting that the relationship between devices was not constant in all locations. Our finding of heterogeneity in instrument comparability by location may have important implications in epidemiologic studies incorporating personal PM_2.5 measurements.

Graphical Abstract

Introduction

Particulate matter (PM) is a complex aerosol mixture arising from both anthropogenic and natural sources and is a major cause of morbidity and mortality worldwide^1,2. Fine PM (≤2.5 μm in aerodynamic diameter, PM_2.5) is one of six criteria air pollutants regulated by the U.S. Environmental Protection Agency (EPA) with daily (35 μg/m³) and annual (12 μg/m³) standards³. Studies that have used outdoor concentrations of ambient fine PM as a proxy for personal exposure have consistently found associations with cardiovascular and respiratory morbidity and mortality, and with lung cancer^4–6.

The gold standard for the characterization of personal PM exposure has traditionally been a gravimetric measurement collected and summarized for a relevant time period of interest using a filter-based sampler. However, instruments based on light scattering technologies, such as nephelometers, which measure light scattered from multiple angles on a group of particles, or optical particle counters, which count individual particles by measuring scattered light produced as each particle passes by a photodetector, can provide real-time personal exposure monitoring without the extra time needed for gravimetric analysis. Nephelometers, such as the MicroPEM v3.2A (MicroPEM; RTI International, NC) and the MIE pDR-1500 (pDR-1500; Thermo Scientific, MA) have been used previously for personal PM exposure monitoring of households, occupational settings, and public transportation systems^7–11. The pDR-1500, specifically, has also been used as a reference device for other personal sensors and has compared well to gravimetric methods¹².

Technological advances have enabled many inexpensive and smaller-sized personal PM monitoring devices (generally, optical particle counters) to come to market in recent years^13–15, making the addition of real-time personal monitoring a feasible component of ambient exposure assessments¹⁶. Increasingly, studies have been evaluating the performance of these devices in comparison to well-validated reference devices, both in controlled laboratory conditions^12,17,18, and ambient field sites^13,19,20. However, because certain sensors may be sensitive to changes in environmental conditions, such as temperature and relative humidity^19,21, evaluations of personal monitors as they move through ambient microenvironments are needed to gauge their real-world performance. Additionally, studies in settings that mirror those of a personal exposure assessment (e.g., wearable device, mobile measurements) offer the benefit of evaluating device performance under realistic operating conditions. This also provides the opportunity to examine practicality and feasibility of sensor operation before deployment in larger investigations.

In this field study, we evaluated three real-time PM_2.5 personal monitoring devices: an inexpensive Bluetooth-enabled prototype optical particle counter available to the research community, and two light-scattering nephelometers used in previous real-time exposure assessments, the MicroPEM and pDR-1500. Measurements were collected along public transit routes in an urban area. The objectives of this study were to examine the overall comparability in concentrations measured between devices and to evaluate sources of variability.

Materials and Methods

We simultaneously measured PM_2.5 concentrations using three real-time personal particle monitoring devices, a prototype sensor from Clarity Movement Co. (Clarity P1; Clarity Movement Co., CA), the MicroPEM v3.2A, and the MIE pDR-1500 personal particulate monitor. Technical specifications of the devices are presented in Table 1, and additional information on the devices can be found in the supporting information supplement.

Table 1.

Clarity P1, MicroPEM and pDR1500 specifications

Instrument	Detection Method	Sampling frequency interval	PM2.5 Air flowrate; method	PM size range; and size selective inlet type	Flowrate calibration method	Approximate cost
Clarity P1	Optical Particle Counter	2.5 sec	NA; fan	0.35 to 2.5 μm; nonea	Factory calibrated	$50–100 US
MicroPEM	Light Scattering Nephelometer	30 sec	0.5 L/min; pump	0.1 to 2.5 μm; 2-stage impactor inlet	User calibrated	$2000 US
pDR-1500	Light Scattering Nephelometer	30 sec	1.52 L/min; pump	0.1 to 2.5 μm; cyclone inlet	User calibrated	$5000 US

^aThe Clarity device uses used an algorithm to convert the particle count to mass concentration

The field measurements were collected on 12 weekdays from June 30 to August 18, 2016. Air sampling took place along common commuting modes in Washington, DC. All sampling instruments were housed side-by-side in a custom-designed personal backpack. Instruments inlets were attached to the pack shoulder straps and situated within the technician’s breathing zone. To avoid systematic bias from relative positioning, the position of instrument intakes was rotated each sampling day. The sampled locations involved combinations of walking along suburban and urban pedestrian ways, near idling vehicles and construction sites; and during activities associated with Metro train use, including standing at stations and riding in passenger compartments of Metro trains, both above and below ground. Detailed location information (e.g., active construction site) and possible ancillary determinants of exposure (e.g., visible dust present) were noted by a technician at the time of sampling. After sampling was complete, sampling locations were categorized into one of six location types for analysis: 1) Metro train above ground; 2) Metro train underground; 3) in a Metro station; 4) near an active construction site; 5) proximal to urban traffic 6) other outside locations. Temperature, relative humidity, and atmospheric pressure were collected in real-time by the pDR-1500.

Concentrations and meteorological measurements were temporally aligned and averaged to 1-minute arithmetic means and log-transformed. One-minute observations were removed if they were missing concentrations (pDR, 3.9%; MicroPEM, 9.3%; Clarity, 18.4%), temperature (0.7%), or relative humidity (0.6%). Data were visualized through a combination of time-series plots, scatterplots, and Bland-Altman plots with three pairwise comparisons of concentrations (Clarity vs. pDR, Clarity vs. MicroPEM, and pDR vs. MicroPEM). Bland-Altman plots were constructed with log-scale mean differences (MD) and limits of agreement (MD ± 1.96 SD of differences) to visualize differences between device measurements over the range of PM concentrations. The relationship between devices was further evaluated by fitting three separate linear mixed models with PM_2.5 concentration of one instrument as the dependent variable and the corresponding measurement from another instrument being the independent variable (treated as a fixed effect). In addition, fixed effect terms for temperature, relative humidity, and multiplicative interaction term between location category and the PM_2.5 concentration of the comparison instrument were included. The linear mixed model included a random effect term corresponding to day and a continuous-time exponential correlation structure to account for serial correlation in the measurements. For ease of interpretation, rather than present the parameter estimates for the exponential correlation structure, we show the estimated correlation for measurements one-minute and one-day apart. Additional models incorporating atmospheric pressure as a covariate and including relative humidity- or temperature-concentration interaction terms as fixed effects were examined in sensitivity analyses.

Results and Discussion

In total, 3,963 1-min averages were collected over 12 days. The geometric mean of PM_2.5 concentrations during the sampling period was 31.6 μg/m³ (interquartile range: 10.1 to 69.2 μg/m³) as estimated by the pDR-1500. Overall, temperatures and relative humidity ranged from 19.1–44.2°C and 24.1–77.0%, respectively, but varied by location (Table S1). There was high agreement on the linear scale between the two nephelometers (R²_min=0.75). However, the Clarity prototype showed lower agreement with both the MicroPEM (R²_min=0.37) and pDR-1500 (R²_min=0.39). A time-series plot of 1-min averages showed periods of both agreement and discord between device concentrations (Figure S1).

Pairwise Bland-Altman plots showed the differences between the two device’s PM_2.5 concentrations by mean concentration for all collocated 1-min averages (Figure 1). PM_2.5 concentrations (log μg/m³) from the Clarity device were lower on average than both the MicroPEM (MD: −0.22; 95% CI: −0.24, −0.20) and pDR (MD: −0.59; 95% CI: −0.57, −0.54). MicroPEM concentrations were also lower in comparison to pDR concentrations (MD: −0.34; 95% CI: −0.36, −0.33); however, the standard deviation of differences was lower for this pairwise comparison (0.48) than for either the Clarity-MicroPem (0.56) or Clarity-pDR comparisons (0.56). In both Clarity comparisons, more pairwise differences fell outside the upper agreement boundary indicating the Clarity device generally had higher concentrations when agreement was poor. The level of agreement also appeared to change over the range of mean concentrations for the Clarity-pDR comparison: as mean concentrations increased, the magnitude of the pairwise differences also increased.

Figure 1.

Bland-Altman plots for pairwise comparisons^a of log-transformed PM_2.5 concentrations between personal monitoring devices for all collocated 1-minute averages

^aDifference in A) calculated as Clarity minus MicroPEM; difference in B) calculated as MicroPEM minus pDR; difference in C) calculated as Clarity minus pDR. In each panel, the x axis reflects the mean of the two log-transformed pairwise measurements and the y axes reflect the difference of these two log-transformed values.

Pairwise scatterplots suggested differences in device agreement by location (Figure S2). These differences were statistically significant in all 3 pairwise linear models that adjusted for meteorological covariates and included interaction terms between the comparison device concentration and location (Table 2). In models predicting pDR concentrations from Clarity measurements, the slope of the linear agreement in concentrations, as calculated by the addition of the main effect and location-specific interaction terms, ranged from 0.094 (urban traffic; 0.483–0.389) to 0.623 (Metro underground; 0.483+0.140). For models predicting MicroPEM concentrations from Clarity measurements, slopes were highest for Metro station and urban traffic locations (0.844, 0.849, respectively) and lowest for the construction site location (0.311). Likewise, in the models comparing concentrations between MicroPEM and pDR, predictions also varied significantly by location; slopes were highest for the Metro station locations (0.955) and lowest for the construction site and urban traffic locations (−0.040, 0.092, respectively). Serial correlations were similar between all three models with the correlations across measurements being only sizable when they are within a few minutes. In sensitivity analyses, inclusion of atmospheric pressure in models did not substantially change the results. When we added interaction terms between device-specific concentrations and temperature or relative humidity, location interaction terms remained statistically significant (Tables S2, S3).

Table 2.

Pairwise linear mixed models^a with interactions by location^b

Model term	Device 1: Clarity Device 2: pDR		Device 1: Clarity Device 2: MicroPEM		Device 1: pDR Device 2: MicroPEM
	Est (SD)	p-value	Est (SD)	p-value	Est (SD)	p-value
Intercept	1.215 (0.403)	0.003	1.806 (0.405)	<0.001	0.634 (0.420)	0.131
Term Device 1	0.483 (0.082)	<0.001	0.627 (0.075)	<0.001	0.494 (0.071)	<0.001
Temperature	0.017 (0.009)	0.069	−0.020 (0.009)	0.024	−0.006 (0.009)	0.490
Rel. Humidity	0.002 (0.003)	0.548	0.000 (0.003)	0.970	0.015 (0.003)	<0.001
Location 1 (Metro train above ground)	Referent		Referent		Referent
Location 2 (Metro underground)	0.140 (0.202)	0.488	0.109 (0.183)	0.552	−1.219 (0.257)	<0.001
Location 3 (Metro station)	0.148 (0.362)	0.683	−0.330 (0.350)	0.346	−0.271 (0.440)	0.537
Location 4 (Construction site)	0.290 (0.288)	0.314	0.572 (0.261)	0.028	1.622 (0.445)	<0.001
Location 5 (Traffic)	1.497 (0.216)	<0.001	−0.573 (0.191)	0.003	1.918 (0.235)	<0.001
Location 6 (Other)	−0.213 (0.212)	0.315	−0.253 (0.190)	0.183	0.153 (0.236)	0.516
Device1:Location 1	Referent		Referent		Referent
Device1:Location 2	0.140 (0.087)	0.106	0.151 (0.080)	0.057	0.461 (0.082)	<0.001
Device1:Location 3	0.139 (0.113)	0.218	0.217 (0.110)	0.048	0.235 (0.113)	0.038
Device1:Location 4	−0.163 (0.101)	0.106	−0.316 (0.092)	0.001	−0.534 (0.134)	<0.001
Device1:Location 5	−0.389 (0.086)	<0.001	0.222 (0.079)	0.005	−0.402 (0.075)	<0.001
Device1:Location 6	0.020 (0.098)	0.836	−0.026 (0.090)	0.776	−0.152 (0.085)	0.075
StdDayc	0.159		0.268		0.222
StdResc	0.482		0.386		0.464
CorrelationDayc	0.002		0.002		0.002
CorrelationMinc	0.749		0.662		0.674

^aLinear mixed models constructed with PM_2.5 concentration of device 1 as a fixed effect and part of the multiplicative interaction with location. PM_2.5 concentration of device 2 was used as the dependent variable

^bLocation categories were defined as follows: location 1 - Metro train above ground (used as referent), location 2 - Metro train underground, location 3 - in a Metro station, location 4 - near an active construction site, location 5 - proximal to urban traffic, location 6 - other outside locations

^cStdDay represents the standard deviation in the day-specific random effect. StdRes represents the standard deviation in the measurement-specific error distribution. CorrelationDay and CorrelationMin are the correlation between measurements one day and one-minute apart, respectively.

In this study, we evaluated three portable real-time PM_2.5 monitoring devices side-by-side as they were moved through a variety of real-world urban microenvironments. We found that the devices showed moderate-to-high agreement in pairwise comparisons on the log-scale, and the level of agreement between devices varied in a location-dependent manner.

We found both systematic and location-dependent differences in PM_2.5 measurements between devices. Some systematic differences, specifically the lower mean concentrations of the Clarity device, were expected due to differing detection methods. Our findings of changes in device agreement by location are plausible given that sensor performance can change by particle composition¹⁸ and meteorological conditions^21,22. Agreement, as measured by uniformity of the slope, was highest for the Metro station location in all three models, perhaps due to device, meteorological, and PM source stability. Conversely, agreement was poor in locations where particle size and composition and ambient conditions may have been more dynamic, such as at urban traffic or active construction sites. Optical sensors are known to show positive bias under conditions of high humidity²³, and prior studies have suggested that device agreement may vary with fluctuations in temperature and humidity, particularly among sensors without established calibration criteria²². Hojaiji et al. (2017) showed that agreement across multiple real-time sensors varied by short-term changes in temperature and increasing humidity under controlled settings²². Likewise, Han et al. (2017) demonstrated discords in agreement for two optical particle counters by levels of relative humidity²¹. Our findings of significant interaction effects by location persisted in models that included interaction terms for temperature and humidity, suggesting that differences in particle size and composition or other factors that varied between locations may be the primary driver of these findings.

Other studies have generally found low-cost fine particle sensors to perform well in both laboratory and field conditions as assessed by hourly and daily correlations on the linear scale (R²>0.75)^13,19,21. Some studies have found that measurements from these sensors are temperature dependent¹⁹, correlate less well on shorter time scales²¹, perform better in laboratory conditions¹³ and correlate non-linearly above certain thresholds^13,24. Direct comparisons to our study findings are difficult, as these evaluations have generally used longer averaging times and were conducted under controlled laboratory conditions or in mostly uniform environments, such as field tests located at stationary sites. Our finding of relatively low correlations (0.37, 0.39) between the Clarity sensor and the two nephelometers should be compared in this context.

Our evaluation provides information that may be useful for personal PM_2.5 exposure monitoring in epidemiologic studies. Our finding of a small systematic difference in measurements between the prototype Clarity device and two nephelometers suggests that any new instrument should be evaluated and calibrated against previously validated reference devices. The finding that location (and the PM profile (i.e., size fraction and chemical composition) it represents) influences the performance of some personal exposure monitors may represent a source of measurement error. Therefore, others may want to prioritize gathering contextual information about each location during sampling, especially as they relate to the PM or meteorological stability. Our choice of using 1-minute averages to finely capture changes while moving within urban microenvironments is also relevant in this context. In an epidemiologic study, such a sampling schematic could enable the generation of finely resolved activity- and location-based exposure estimates that could help extend estimates to other participants where study-wide personal monitoring is not feasible. Another factor relevant to epidemiologic exposure assessments in large populations is the tradeoff between scalability and accuracy in measurement. In this field study, we compared two well-referenced but more expensive nephelometers to a low-cost optical particle counter. While agreement between these devices varied, each captured contrasts in concentrations over the sampling period and across locations.

Our realistic evaluation scenario incorporating wearable monitors housed in a backpack may be more informative to future personal exposure assessments than a controlled, stationary validation. In addition, another strength of this study was the classification of urban microenvironments into multiple location categories. This classification explained a large proportion of variation in measurement between the devices. However, additional studies that evaluate differences in PM profiles by location would be informative. Additionally, sampling over multiple seasons or for longer duration would extend the generalizability of our results.

In this evaluation of three PM_2.5 sensors as they moved through multiple urban microenvironments, we found both small systematic and location-dependent differences in PM_2.5 measurements between devices. These findings suggest that any new real-time air monitor should be evaluated and calibrated against previously validated reference devices, ideally in real-world and mobile exposure scenarios. These findings may have implications for future epidemiologic studies incorporating personal PM_2.5 measurement data and comparability between studies using different monitoring devices.