Investigating Measurement Variation of Modified Low-Cost Particle Sensors

Abstract

Particulate matter (PM) has demonstrably increased rates of cardiovascular and respiratory related disease; thus, a low-cost sensor that accurately measures PM is desirable including for smaller and more private environments such as residential homes. The low-cost Dylos and the Utah Modified Dylos Sensor (UMDS) have been shown to be highly correlated with references instruments for measuring particle counts and aerosol concentrations, which makes them useful tools for air quality studies. An analytical calibration equation (calibration) is used to describe the linear relationship between the UMDS and a reference instrument, providing the best estimate of PM concentrations when the UMDS is operated. In this study, an investigation of measurement variation of a UMDS was performed using a low-cost calibration technique to determine differences between the brand new UMDS pre-calibration equation (Pre_c), a contaminated UMDS post-calibration equation (Post_c), and a cleaned UMDS clean calibration equation (CC). The UMDS were calibrated against a high-grade aerosol spectrometer (Grimm model 1.109) as a reference instrument. Calibrations took place in a home or office environment. Counts per volume units from the UMDS were matched to the Grimm’s for comparison. The investigation of the UMDS for measurement variation was performed for the approximate estimates of PM_2.5 by using the small bin (i.e. ≥0.50μm) subtracted from the large bin (i.e. ≥2.5μm), and for total particulates by using the large bin. Linear regressions were performed between the UMDS and the Grimm per calibration event, which produced R² values and slopes that were indicative of measurement variation. Data exceeding the upper limit of quantification (ULOQ) of 106,000 particles/liter and the lower limit of quantification (LLOQ) of 4 particles/liter were excluded from statistical comparison. R² values greater or equal to 0.70 were used to assess measurement variation as a quality control standard for valid comparisons. A rank sum statistical test between calibration comparisons was performed. Pre_c/Post_c and Pre_c/CC had significant differences indicating measurement variation. Post_c/CC did not have any significant differences; cleaning the UMDS had no effect and did not demonstrate measurement variation. Reasons for measurement variation may include instrument contamination (dust/dirt), hardware degradation, altered fan flow rates, and potentially inadequate cleaning of the UMDS. Future work may investigate the rate of measurement variation in order to develop a recommended re-calibration schedule in order to maintain the most accurate estimates of PM for UMDS in long-term operation.

Introduction

Air quality, which plays a vital role in human health [1], is effected by particulate matter (PM) that is produced by a variety of sources such as industry (e.g., soot, smoke, mechanically generated dusts, bio-aerosols, combustible fumes, industrial mist sprays), home owners (e.g., dusts from linens, gas powered appliances, cooking, smoking, wood burning), and even nature (e.g., dusts, pollens, molds) [2–4]. High concentrations of PM in the ambient air are associated with mortality [5], and populations within developing as well as developed countries are at risk from short- and long-term exposures to PM [6, 7]. The World Health Organization (WHO) attributed more than 3 million premature deaths in 2012 to air pollution and also found that poor air quality effected 92% of the world’s population [8]. Of the premature deaths, 72% were associated with cardiovascular disease, heart attacks, and strokes; 14% were associated with lung infections and chronic obstructive pulmonary disease; another 14% were associated with lung cancer [8]. As recently as 2016, 80% of those that were living in metropolitan areas were exposed to air pollution levels that were above the WHO recommended limits [9]. PM is classified as a Group 1 carcinogen by the WHO’s specialized cancer agency, the International Agency for Research on Cancer (IARC), and has been attributed to increased rates of lung cancer in North America [1]. Strong associations of mortality and PM small enough to be inhaled (i.e., PM₁₀ and PM_2.5) have exhibited severe adverse effects on public mortality rates, cardiovascular and respiratory disease, decreased lung functions, coughing and difficulty breathing [2, 5, 10, 11]. Air pollutants have increased outpatient visits for asthmatic attacks up to 6% in sensitive populations in Anchorage, Alaska and the eastern portion of Washington State [5, 12]. It is estimated asthma has affected nearly 26 million American adults [13] and is the third highest cause of hospitalizations in children [13]. Of the 6.2 million American children with asthma, 3.1 million experienced an asthmatic attack in 2015 triggered by exercise, temperature change, and a plethora of sources that emit PM [13, 14].

A few widely-used methods for quantifying PM exposure includes gravimetric aerosol samplers, and low-cost sensors [4, 12, 15, 16]. A common way to quantify an individual’s exposure is by using aerosol samplers that can accurately collect PM in the air that is representative of the amount of particles entering a person’s mouth and nose through respiration, which is commonly known as the inhalable aerosol fraction [17]. However, aerosol samplers have drawbacks such as: some amount of PM can be lost when sampling medium is removed from the sampling environment, the need to control for moisture in analysis of the sample, and PM that becomes embedded into the sampler’s wall and is not read during analysis (i.e., only the filter is analyzed) [17]. Moreover, common personal samplers are generally limited to a short duration of sampling (often measured in hours or a single day sampling period), and as they are often gravimetric measures, yield only a single accumulated PM exposure. This type of measure fails to capture information about temporal events and/or activities associated with high- or low-PM exposures that may be an important component to the overall exposure profile; for example, those associated with asthma exacerbation.

In contrast to gravimetrically-based aerosol samplers, air monitoring with low-cost particulate sensors can be advantageous as they do not require laboratory analysis, they do not require the user to wear the sampler, they provide real-time data, and due to the fact they are stationary they allow for sampling inside home environments [18]. In addition, low-cost air-monitoring sensors have a display that end users can check to ascertain when particulate concentrations are too high in either indoor or outdoor environments [18]. However, low-cost sensors also have their limitations such as varying detection efficiencies depending on the aerosol’s refractive index, different data logging time frames, and few counting bins for characterizing particle sizes [19]. Previous research has shown low-cost sensors to accurately characterize PM in indoor and outdoor environments, as well as personal exposures [18–22]. Low-cost sensors have the capability of sensing a variety of contaminants and have specific methods to quantify a desired contaminant [23]. The Dylos Air Quality Monitor™ is a low-cost particle sensor that can characterize indoor suspended dusts when converting particle counts to mass concentrations [21]. The Dylos has a strong correlation with reference instruments’ (e.g., aerosol photometer pDR-1200, Tapered Element Oscillating Microbalance, SidePak photometer, Grimm, DustTrack) mass concentrations [19–21] when calibrated prior to sampling. In laboratory experiments, the DC1700 Dylos sensor has provided precise PM measurements [20], but aerosol type can effect this accuracy [20]. To control for the effect of aerosol type on accuracy, any calibrations must be performed in the monitoring environment with a comparison reference instrument [20]. Although calibration methods for the low-cost Dylos particulate sensors have strong correlations with other more sophisticated reference instruments, these samplers are more expensive because of their advanced ability to differentiate and count aerosols across a wider range of particle diameters [22–25]. However, previous research with Dylos particle counters have demonstrated up to 80% accuracy for various types of particulates without using a system of modification for corrections [21]. Prior research with another low-cost particulate sensor, the Plantower PMS 1003/3003 (PMS), in targeted ambient air conditions showed that the PMS overestimated particle concentrations in comparison to the reference instruments and a correction factor had to be applied [24].

Monitoring ambient air quality in urban areas exclusively with low-cost particulate sensors has been used to characterize personal exposures to PM with calibration correction steps [22]. Previous research on sensor measurement variation (i.e., the sensor’s move from accurate measures to measures with errors) has provided an alternative method of correction for measurement variation in linear models by using a correction factor [22]. In one study on low-cost and portable particular matter sensors, it was necessary for the sensor to be calibrated in the same atmosphere as the reference instrument for accurate calibration [23]. This same study and others of low-cost particle sensors also found that a constant concentration of suspended airborne particulates of different compositions and size distributions during calibration in a controlled environment displayed linear relationships at higher concentrations [23, 26]. Another calibration technique used by Budde et al. included the use of a low-cost sensor inside of a box using high concentrations of chalk dust. The data showed that the sensor measurement variation was linear with time [22].

A low-cost particulate sensor currently being used in studies by the University of Utah is the Utah Modified Dylos Sensor (UMDS). The UMDS is a commercially available Dylos DC1100 Pro (Dylos Corporation, Riverside, California) that has been modified by the University of Utah Department of Electrical and Computer Engineering to provide more information (i.e., temperature, humidity) and features useful to investigators and research participants. The modifications to the commercially available Dylos include a Beagle Bone Black wifi enabled microcomputer, a temperature and relative humidity sensor, and an upgraded color display (exchanged for the monochrome display)—as an integrated package, the new unit is known as the UMDS [16]. The aforementioned components fit in the standard Dylos instrument housing without moving or otherwise modifying the existing Dylos components with the exception of the replacement (swap) of the monochrome disply for a color-capable unit (see Figure 1). Figure 1 shows a UMDS with the front cover removed—exposing the internals of the instrument including the modifications made to a standard Dylos. Visibly, because the changes to the commercial Dylos that yields a UMDS are all internal, the completed UMDS maintains the same diminutive form factor and quiet operation of the commercial Dylos and is suitable for placement in homes, on tables or other furniture. Additionally, the secure wifi connectivity of the UMDS works in conjunction with a secure database, internet-based management platform, mobile application interactive software platform, as well as internet of things capability such as real-time voice interaction via Google Home and similar devices.

Figure 1:

Dylos DC100 Pro with modifications resulting in Utah Modified Dylos Sensor (added microcomputer (BBB), LCD display (RGB-LCD) and temperature/humidity sensor (SHT21) to the interior of the standard Dylos).

The UMDS is less costly than most particle counting devices because it does not have the focusing optics, lenses or mirrors found in more expensive particle sensors. The hardware costs for the UMDS are approximately $500. The UMDS has been successfully used by University of Utah researchers in the Pediatric Research using Integrated Sensor Monitoring Systems (PRISMS) study [16], which is funded by the National Institutes of Health (NIH) to test the integrated ecosystem comprised of low-cost environmental measurement, health experience and informatics technology among a cohort of pediatric asthmatics and their family’s [16]. The expectation is that by providing real-time exposure information asthmatics can more accurately tie their health experiences to PM exposures, thereby quantifying the relationship between PM exposure and asthma [16].

A principal component to the UMDS is the Dylos DC1100 Pro, which is designed to quantify PM concentrations by count [21]. As such, it uses a small computer fan to circulate air through the sensor while a photo-diode counts particles by emitting a light that strikes the particles, which is then refracted onto an internal sensor detecting the amount of light refracted at 650μm [21]. The Dylos interprets the light refracted and separates particles based on size into two bins: large bin for ≥2.5μm particles, and a small bin for ≥0.5 μm particles. For research purposes, in post-processing, PM_2.5 can be estimated by subtracting the large bin from the small (all particles) bin. With its low cost, enhanced data collection (i.e., humidity and temperature), and real-time data transmission, the UMDS is a promising particulate counter for characterizing PM exposures. With interest in utilizing the UMDS for long-term characterization of indoor air quality, there is a need to assess sensor drift, that is, examining the ability of the UMDS to maintain an accurate estimate of PM during extended operation. Industrial hygienists and environmental scientists are used to high tech instruments that require regular calibration, cleaning and/or factory servicing to maintain accurate measurement. The Dylos, and therefore also the UMDS, have no published recommendations or schedules to promote long-term measurement accuracy. As the UMDS has been utilized for continuous PM measurements in the homes of research participants engaged in a pilot project for extended periods (some going on several months of use), there is concern that measurement variation may occur as a result of extended operation (hardware/sensor degradation) and/or particle contamination. In keeping with the low-cost, low-resource nature of the UMDS—desirable qualities in large scale epidemiologic research whereby a large sample size of exposure estimates and their corresponding events or even health outcomes can be catalogued—it was deemed appropriate to use a similar approach as other researchers by devising a calibration procedure [16]. The first goal of this study is to investigate sensor measurement variation of a UMDS by using an analytical calibration equation (calibration) that describes the linear relationship between the UMDS and a reference instrument to provide the best estimate of PM concentrations. These linear relationships compare the pre-calibrated equation (Pre_c) of a new UMDS to the post-calibrated equation (Post_c) of the deployed UMDS. The second goal of this study assumes that particle contamination resulting from use is the cause of any measurement variation and we therefore investigate if a simple cleaning procedure can return the UMDS to the Pre_c status.

METHODS

Twenty-five UMDS used in the NIH PRISMS study were investigated for the study discussed herein. In order to determine if the UMDS experiences measurement variation resulting from contamination due to continuous operation, the calibration procedure used in the PRISMS study was replicated for this research.

Prior to deployment in participants’ homes, new, unused UMDS underwent a low-cost, non-lab (field based) calibration procedure. The procedure consisted of multiple UMDS being evaluated against a reference instrument, the Grimm model 1.109 (Grimm Aerosol Technik, Ainring-Berchtesgadener Land, Germany) portable laser aerosol spectrometer to determine the analytical calibration equation, Pre_c. That is, a linear equation describing the relationship between the reference instrument and the UMDS was determined by collocating the instruments in close proximity to each other (all within 2 m²) in a single room and exposing them to varying concentrations of a challenge aerosol assumed to be uniformly distributed throughout the room air. The challenge aerosol was generated by removing detritus from the bag of a commercially available vacuum cleaner (Eureka Boss Cannister Vacuum Model 3648D) and manually spreading this detritus in an approximately 1 m² surface of the floor within the room containing the instruments and the vacuum and then re-vacuuming the detritous from the floor. A diagram of the general field calibration set-up is provided in Figure 2. The high efficiency particulate air filter (HEPA) was removed from the vacuum during the challenge event in order to increase the small particle dispersion in the room. The vacuuming of the detritous was done using the vacuum’s flexible hose and wand attachment (hand held nozzle) and conducted in a slow, methodical manner whereby the wand was manually manipulated across the detritous laden floor in sequential parallel movements commencing from one side of the 1 m² to the other. When vacuuming of the 1 m² floor surface with the wand in one direction was complete, the vacuuming process continued with the wand directed across the floor surface in successive parallel movements, but this time in a perpendicular direction to the original wand motion. A similar procedure, using the same vacuum, was followed for the post-deployment calibration event (Post_c) and the clean calibration event (CC) after the UMDS had been deployed in PRISMS research participants’ homes for varying periods of continuous operation. All calibrations events were performed in one room of a single carpeted home or one room of an office environment (carpeted). After approximately 2 to 8 months of operation inside a home, each UMDS was collected separately and Post_c and CC procedures conducted. Upon removal from the home, the UMDS promptly underwent the post deployment calibration procedures. The deployed UMDS operated continuously for an average of 168 days with a minimum of 57 days and a maximum of 237 days of operation prior to any post-calibration procedures.

Figure 2:

Field Calibration set-up diagram (room schematic with existing furniture; not to scale).

In the Pre_c, 15 UMDS were calibrated with 1 Grimm for a 57-minute calibration period with the vacuum’s exhaust end facing all UMDSs and Grimm. At the start of the Pre_c, all sensors were turned on for 15 minutes with the vacuum off. Then, vacuuming detritous for approximately 10 minutes with the HEPA filter removed commenced in an effort to aerosolize varying particulate concentrations. The vacuum was turned off again for 10 minutes, followed by turning the vacuum on for another 10 minutes. The vacuum was turned off again for 12 minutes and the Pre_c ended. The Post_c and clean-calibration (CC) calibrations were done in the similar manner. Three UMDS and one Grimm were placed side by side on the carpet approximately 7 feet away from the end (i.e., non-suction end) of the portable vacuum per calibration event. The UMDS and the Grimm were then turned on, and vacuuming detritous commenced for 5 to 7 minutes. The vacuum remained on for approximately 10 minutes in order to provide air movement. After running for 10 minutes, the vacuum was turned off, but the UMDS and the Grimm remained on for another 5 to 7 minutes in order to record aersol measurements of presumably dwindling aerosol concentrations in the room in an effort to obtain multiple data points needed do describe the relationship between reference and UMDS sensor measurements. The total operation time of the instruments was 20 to 25 minutes, 10 of which the vacuum was also running. The instruments logged data in 1-minute intervals, providing roughly 20 calibration data points per calibration event. The assumptions made were that a limited aerosol concentration range would still yield an analytical calibration equation indicative of one derived from the data across the whole operational range of 4 to 106,000 particles per liter, and using 20 or less data points between the upper limit of quantification (ULOQ) and upper limit of detection (ULOD) was sufficient and consistent enough to be representative of a calibration.

At the time the Post_c procedure was conducted, all the UMDS sensors had visible signs of contamination (dust/debris accumulation at air inlets and outlets) and as such, if sensor use and/or contamination from field deployments resulted in measurement variation, our investigation could detect this. After the Post_c was complete, each UMDS was disassembled, cleaned with canned air, and then re-assembled prior to use for the CC. After cleaning, the UMDS were calibrated a second time following the same procedure as described above. Again, the instruments were disassembled and taken apart to be cleaned once again. The UMDS were post-calibrated for a third time for comparison to the previous cleaning.

Comparisons between the UMDS particle counts and the Grimm particle counts were performed. However, a direct comparison could not be performed without some data conversion. The Grimm uses 31 bins that are organized by particle counts per liter and range from 0.25μm to 32.0μm in particle diameter [27, 28]. The UMDS data are listed in particle counts per 0.01 cubic foot per minute, which was adjusted to the same units of the Grimm’s particle counts per liter per minute. This was done by dividing the UMDSs particle count per 0.01 cubic foot by a conversion factor of 0.2832 liters. The bins chosen from the Grimm were in the range of 0.50μm to 32.0μm for a direct comparison to the UMDS.

According to the manufacturer, the ULOQ of the Dylos DC 1100 pro is 582 particles per 0.01 cubic foot or 106,000 particles per liter, after which there is a loss in accurate particle counting. Furthermore, when PM concentrations exceed 65,536 particles per 0.01 cubic foot, the DC 1100 pro sensor’s 16-bit memory and data logging capabilities are exceeded and the instrument rolls over to zero [29]. When concentrations of PM exceed these levels, the data are unreliable. Conversely, the manufacturer has also stated the lower limit of detection (LLOD) of the DC 1100 pro is 1 particle per 0.01 cubic foot, or 3.5 particles per liter, which is the same as the lower limit of quantification (LLOQ). When particle concentrations are lower than the LLOD, the DC 1100 pro unreliably counts particle data. In this study, low-cost methods were used to calibrate the UMDS with the underlying assumption of staying between the ULOD and ULOQ. Exceeding theses limits with the UMDS were unreliable for comparison and excluded based on Grimm data.

There were twenty-five UMDS used in this study with one Grimm as the reference instrument, and there were twenty-three calibration events. Total run time for each UMDS depended on the calibration event, which lasted anywhere between 20 to 57 minutes. Thus, with all calibration events combined, there was a total run time for all functioning UMDS of 1,807 minutes. Of those 1,807 minutes, 1,160 were within the operational range of the UMDS, which were between the ULOQ and LLOQ. Of the 1,160 minutes within the operational range of the UMDS, 482 minutes were used for statistical analysis by using the last 20 minutes of each calibration event. If any data exceeded the ULOQ, then these data were omitted and the 20 minutes just prior to the last 20 were used for statistical analysis. The Grimm total run time for all calibration events within the ULOQ and LLOQ combined was 482 minutes. Every minute in the analysis was equal to one data point measured. Thus, there were 1,645 data points used in this study. The overall goal for investigating and identifying sensor measurement variation is met by using quality data provided within these limits.

All data points from each UMDS calibration were matched to the Grimm’s corresponding data points. Linear regression was conducted using Microsoft Excel_© (Microsoft Corporation, Redmond, WA) for each individual calibration to obtain a line of best fit using the equation:

$$\text{y}=\text{mx}+\text{b}$$ (1)

where y is a 1-minute UMDS data point in particle counts/liter, m is the slope of the line in linear regression, x is a 1-minute Grimm data point in particle counts/liter, and b is the y-intercept of the line in linear regression. Equation 1 provided R² and slopes values for a line of best fit. R² values and slopes are indicative of measurement variation for the approximation of PM_2.5 and total particulates linear regressions. For each calibration, the slopes given from the linear regressions were statistically compared using p-values from a ranked sum Wilcoxon-Mann-Whitney test. Wilcoxon-Mann-Whitney is a non-parametric (distribution free) equivalent to the Paired t-test, and was used to analyze the data due to small sample size. The repeated measures design (comparison of pre-calibration, post-calibration, and clean-calibration) allows for greater power due to the reduction of intersensor variability by making the comparison between the same sensor at the different calibration times as opposed to independent groups of sensors. For this study, we will utilize the Wilcoxon Signed Rank test. Research has shown it provides better performance than paired t-test in smaller samples that are not normally distributed [31].

Analyses of the R² and slopes included the following four comparisons: 1) Pre_c and Post_c; 2) Post_c, and CC; 3) Pre_c and CC. Cleaning was performed in order to identify if there was a difference in sensor data between all four comparisons that could be attributed to measurement variation. The difference in the R² and slopes of the UMDS helps identify possible causes of measurement variation, such as a clean UMDS calibration event (Pre_c, CC) versus a contaminated UMDS calibration event (Post_c), or hardware degradation.

RESULTS

UMDS Field Deployment Times

Table 1 is a summary of the days a UMDS spent operating in a home prior to post-deployment calibration procedures. All the days in the table include when the UMDS was first deployed in a home and when a UMDS was removed from a home.

Table 1.

UMDS Days of Operation Deployed in Homes

UMDS ID Number	Days UMDS Deployed In Homes
B007	57
B008	57
B009	57
B010	193
B011	193
B012	220
B013	220
B014	220
B015	222
B016	222
B017	222
B018	237
B019	237
B020	237
B021	222
B022	68
B023	68
B024	68

Calibration Data Collected

During calibration, many data points collected by the UMDS exceeded the ULOQ of 106,000 particles per liter. To standardize and adjust for data exceeding the ULOQ, any data recorded by the Grimm that exceeded 106,000 particles per liter were eliminated from analysis. An R² value of 0.70 was chosen as a quality control standard for a good calibration to assess measurement variation in slopes. Computations that resulted in R² below 0.70 were not included for analyses, as an investigation into the source data indicated much or all of the calibration procedure resulted in challenge aerosol concentrations outside of the operational range for the UMDS. Some data were excluded from this study because they were inaccessible or missing. Software errors in the data downloading process were the primary source of missing data that were excluded. Data used from a calibration event was considered valid if it remained between the ULOQ and LLOQ; thus, a total of eight UMDS provided data for this study. Of the eight UMDS used in this study, data with R² values <0.70 were excluded for comparison analysis. Pre_c was done in a carpeted home or office setting different than Post_c, CC, and calibrations, where performed in a private residence. There was only one Pre_c done as a single calibration event. All UMDSs were calibrated at the same time, in the same conditions under a 57-minute calibration.

UMDS Linear Regressions

In this study, the UMDS was investigated for measurement variation by applying linear regressions on all UMDS data collected for each calibration event using data collected from the Grimm as a reference. These regressions produced an R² value for a line of best fit and a slope of the line from one UMDS corresponding with the Grimm. An inter-comparison of linear regression results among all four calibration classifications are shown in Figures 2–5. The data illustrated in Figures 2–3 were applied to a Wilcoxon-Mann-Whitney test to provide the R² values for approximating PM_2.5 and for total particulates.

Figure 5.

Inter-Comparison of Linear Regression Slope Values in Approximating PM_2.5

Figure 3.

Inter-Comparison of Linear Regression R² Values in Approximating PM_2.5

Pre_c for PM_2.5 and total particulates were collectively near the value of 0.75. Post_c yielded more R² values that were higher than the Pre_c results. Post_c R² values yielded slightly lower results than CC, which may indicate the presence of drift.

The Pre_c’s and Post_c’s slopes in Figure 5 were different by a factor of eight. In Figure 6, the Pre_c is lower than the Post_c by a factor of five. In Figures 3–4, sensors B012 and B013 had good R² values corresponding with the Grimm, however, in Figures 5–6, these two sensors had slopes for CC that were decreased by a factor of five. From the Pre_c through CC, the slope values tend to increase as the next calibration event occurs for every individual UMDS (Figures 5–6). The data illustrated in Figures 5–6 were applied to a Wilcoxon-Mann-Whitney test to provide the slope values for approximating PM_2.5 and for total particulates.

Figure 6.

Inter-Comparison of Linear Regression Slope Values for Total Particulates

Figure 4.

Inter-Comparison of Linear Regression R² Values for Total Particulates

Forty-three R² and slopes were used in a Wilcoxon-Mann-Whitney rank sum statistical test for non-parametric data. Tables 2–5 show an inter-comparison of statistical outputs of R² and slope value outputs produced by the Wilcoxon-Mann-Whitney statistical test. To investigate and identify drift, R² and slope values were compared between Pre_c/Post_c, Post_c/CC, and Pre_c/CC.

Table 2.

R² Approximating PM_2.5 Wilcoxon-Mann-Whitney

Test	Observations	Rank Sum	Variance	P-value
Prec/Postc	8/7	44/76	75	0.0206
Postc/CC	7/8	42/78	75	0.1052
Prec/CC	8/8	36/100	91	0.0008

Table 5.

Slopes Total Particulates Wilcoxon-Mann-Whitney

Test	Observations	Rank Sum	Variance	P-value
Prec/Postc	8/7	36/84	75	0.0012
Postc/CC	7/5	56/49	38	0.5698
Prec/CC	8/5	36/84	47	0.0034

In Tables 2–5, the Pre_c/Post_c and Pre_c/CC comparisons were statistically significant with p-values <0.05. These data demonstrate that measurement variation was observed when testing both the R² and slope values of the resultant calibration equations. The Post_c/CC comparisons were not significant with p-values >0.05, for either the R² or slope comparisons, which indicates our cleaning procedure of the UMDS had no effect on the R² or slopes of the calibration equation. When approximating PM_2.5, the p-values are smaller than the total particulates p-values. Though all R² met the criteria of ≥0.70 prior to use in the Wilcoxon-Mann-Whitney test, there were statistical differences in R² for Pre_c/Post_c and Pre_c/CC comparisons. The results noted in Tables 2–5 indicate that measurement variation indeed took place for UMDS deployed in participant homes when examining their pre-calibration relationship to a reference instrument versus their post-calibration relationship to the reference instrument. The data also indicate that our cleaning procedure did not significantly alter the calibration equation of a deployed UMDS when compared to the calibration equation attained post deployment. Lastly, it follows and these results confirm that our cleaning procedure fails to return the used UMDS to a calibration relationship of a new UMDS.

DISCUSSION

The range of aerosol concentrations used to develop equations for the different calibration events varied (Figures 4–5). Though the Post_c, and CC underwent an identical calibration procedure in an identical home environment in rapid succession (limited time between events), a review of the data showed that particle concentrations for calibration events varied greatly. Moreover, some of the particle concentration exceeded the ULOQ and were therefore not applicable for generating the analytical calibration equation. The goal of any calibration event is to generate varying aerosol concentrations across the operational measurement range of the instrument, and analysis of our calibration data indicated this was much harder to achieve than we anticipated using our procedure. Therefore, relevant data from the reference instrument should be monitored in real-time to verify useful aerosol concentrations are being attained via the calibration procedure. All data collected in this study by the UMDS between the ULOQ and LLOQ were considered legitimate data points and used for statistical analysis.

There were no significant differences in Post_c/CC R² values or slopes (Tables 2–3). After cleaning the UMDs with pressurized canned air, there were no differences in the performance of a clean versus contaminated UMDS. In the Pre_c/Post_c and Pre_c/CC comparisons, all p-values were below the level of significance (p=0.05). This suggests measurement variation occurred in these two calibration comparisons that could be attributed to UMDS hardware or software rather than a UMDS contaminated with recirculating particles. This also suggests that the canned air cleaning, although visibly ridding the unit of accumulated household dust and debris, may have failed to clean sensitive electronic components critical to particle measurement. Internal dust accumulation from regular operation and design improvements to limit dust accumulation that may affect measurement has been documented by some low-cost sensor manufacturers in their analysis and testing [30]. For the Pre_c, the R² values were within the range of 0.70–0.80 (Figures 2–3). However, the Post_c and CC yielded R² values that were higher than the Pre_c established baseline (Figures 2–3). This is indicative of change in the UMDS over time from Pre_c to CC (Figures 2–3). The similarity in the lines of best fit for the Post_c/CC calibrations suggest that cleaning the UMDS did not eliminate measurement variation caused by recirculating particles. However, the p-values are less than 0.01 for the R² values and slopes of the Pre_c/CC are also significant (Table 2–3). This indicates that measurement variation occurred over time between Pre_c and CC with a clean UMDS when calibrations were performed at varying concentrations In reviewing the graphical information put forth in Figures 2–5, there appears to be no observable trends in the relationship between a UMDS’ Pre_c,Post_c, CC slope or R². As investigators with experience conducting a laboratory calibration of Dylos [29], we had some expectation that the pre-calibration protocol would yield the most highly correlated data. This was a premise we arrived to based on the fact that new instruments would have fewer factors impacting their measurement response—that is, new instruments are not subject to the effects that an extended home deployment imparts, such as instrument contamination and component degredation. The investigators suspect the observed phenomena is a reflection of the low-cost, low-resource calibration protocol and relatively small sample size of instruments investigated.

Table 3.

R² Total Particulates Wilcoxon-Mann-Whitney

Test	Observations	Rank Sum	Variance	P-value
Prec/Postc	8/7	43/77	75	0.0151
Postc/CC	7/5	39/39	38	0.2912
Prec/CC	8/5	36/55	47	0.0034

Another anticipated outcome was that the cleaning procedure may return UMDS’ measurement relationship to the reference instrument to a status similar to the new, or pre-calibration status. Again, the results do not support this, indicating that other variables may be influencing outcomes, and that perhaps, upon extensive operation in the field, the mechanical cleaning procedure alone (using canned air to removed dust/debris) may not be sufficient to overcome contamination and return the UMDS to its pre-calibration status. Other variables that the mechanical cleaning procedure may not rectify is contamination from airborne chemical pollutants (e.g. hairspray, cooking oils, ETS etc) on sensitive hardware components such as the laser, photo detector, and fan that can have a direct impact on aerosol measurement. Reasons for measurement variation in the Pre_c/Post_c and Pre_c/CC calibrations may indicate that the hardware may malfunctioning or degrade with extended use. However, the source for hardware error is uncertain and may be the result of potential degradation of the laser light to scatter and refract light, or perhaps in the sensor from continual operation, or even the UMDS fan flow rate. Calibrations derived from a set of data originating from a small portion of the UMDS operational range may not be representative of the calibration equation derived from data spanning the majority of the UMDS operational range. It is possible that concentration levels differed, resulting in slopes that were different in the Pre_c. The volumetric flow rate of the computer fan in the Dylos sensor in the UMDS is unknown. As it is a low-cost instrument, the fan’s volumetric flow rate is not measured or adjusted during operation. Therefore, carbon brush degradation in the fan, voltage fluctuations and/or contamination may impart changes to the volumetric flow rate to the instrument during extended use. As a result, the fans rpm may have changed from initial use to end use and that could influence R² and slopes. Further research opportunities may arise from characterizing the flow rate of the UMDS. Specific to contamination and hardware degredation, it is important to understand the expected impact of these factors on aerosol concentration measurement by the UMDS. Gross particle and debris contamination may result in additional particles being counted by the UMDS resulting in an overestimate of true exposures. Conversely, any degredation or contamination to the laser diode, photo detecter or fan is likely to result in underestimation of the true aerosol exposure. It follows that for the best estimate of aerosol exposure, in the absence of a in-situ calibration, whereby the relationship between reference and UMDS instruments are calculated in the actual measurement environment, a field calibration at regular intervals would at least facilitate the good estimate of aerosol exposure at any given time. As this project was undertaken as a pilot project to investigate participant acceptability as well as challenge the hardware, software and informatics infrastructure in a longitdual cohort study of pediatric asthmatics, it is of use to recognize the value of the data to clinicians. When reminded of the reality that the exposure estimates of deployed UMDS were not precise and influenced by many factors, the clinicians were not put off. Their interest, with respect to asthma exacerbation and airborne triggers identification, was fulfilled by the temporal changes in exposure that the UMDS did capture with some effictivness (e.g. indoor and outdoor pollution events were captured and documented by the UMDS and the informatics infrastructure).

The calibration procedure used in this study has a number of strengths and weaknesses. In keeping with the low-cost nature of the UMDS and a desire to have a low-cost, low-resource calibration procedure, we chose to use an inexpensive, commercially available vacuum cleaner as an aerosol generator. Moreover, we utilized a representative dust, detritus from carpet (not aluminum oxide/Arizona road dust etc.), and conducted the calibration procedure in a readily available carpeted home/office setting similar to a research participant’s home (non-lab, no purpose built chamber). However, a number of sensor calibration data are missing (see Figures 2 and 3). Some calibration data were unrecoverable due to a software error in uploading information from the UMDS to the secure database—this has subsequently been remedied. In addition, the vacuum cleaner-generated aerosol often times resulted in a room concentration above the ULOQ of the UMDS. As a consequence, several calibration events yielded aerosol concentrations entirely above the ULOQ of the UMDS and therefore were excluded from analyses. Additionally, several calibration events were excluded due to low R² values, likely due to the majority of the aerosol concentration values above the ULOQ leaving a limited number of aerosol concentration data points within the operational range of the UMDS. In hindsight, this low-cost, low-resource procedure could be utilized for calibration providing that 1) the operator monitored the reference instrument and 2) manipulated the vacuum cleaner-generated aerosol to ensure sufficient aerosol concentrations across the operational range of the UMDS are achieved.

Strengths of this study are that it is one of the first investigations of measurement variation in this variety of sensor. In keeping with the low-cost nature of the devices, a low-cost, non-lab method of calibration utilized aerosol generated from human occupation and activity (carpet dust). This aerosol more likely represents the dust that these sensors will measure in actual operation than a surrogate aerosol in a lab (Arizona road dust, aluminum oxide, etc.). In addition, each of the homes were suburban residential homes located in central Utah with a family comprised of a pediatric asthmatic, at least one adult guardian, and generally, a few other family members. A visual inspection indicated accumulation of dust and debris in the UMDS after use in homes for all UMDS. Thus, the setting in which this study was conducted is one in which low-cost sensors can and will be utilized in the future. Lastly, with extended continuous operation, the UMDS continued to exhibit a strong linear correlation with a reference instrument.

CONCLUSION

The purpose of this study was to investigate sensor measurement variation of the UMDS while using a low-cost calibration method. This study also investigated whether or not our cleaning procedure of the UMDS would return the sensors to their baseline performance. Our findings demonstrated the presence of measurement variation resulting from the deployment and use of UMDS in homes, but the measurement variation could not be directly attributed to a buildup of dust. Measurement variation was identified between Pre_c/Post_c and Pre_c/CC for both R² values and slopes. Post_c/CC comparisons did not exhibit any differences between R² and slopes for the UMDS when it was cleaned (Tables 2–5). Possible reasons for measurement variation are degradation of the laser, degradation of the sensor, instrument contamination, and/or altered fan flow rate of the UMDS. As our cleaning method did not change the calibration status of the used UMDS, this indicates either a deficiency in our cleaning methodology (canned air alone may be insufficient) or other hardware changes that took place as a result of continuous operation. Ongoing research with the UMDS would seem to include regular calibration in order to provide the best estimate of aerosol exposure but that the existing units did do a good job of documenting temporal changes of aerosol concentrations that were of value to clinicians investigating asthma exacerbation and their potential airborn triggers. Future research in this arena would include improvements to the low-cost calibration methodology that would ensure aerosol concentrations across the operational range of the instrument in order to yield improvements to calibration data (more data and improved correlation coefficient). In addition, an experimental design that discerns whether instrument contamination or hardware degradation is the primary driver of measurement variation would be valuable.

Table 4.

Slopes Approximating PM_2.5 Wilcoxon-Mann-Whitney

Test	Observations	Rank Sum	Variance	P-value
Prec/Postc	8/7	36/84	75	0.0012
Postc/CC	7/8	42/78	75	0.1052
Prec/CC	8/8	36/100	91	0.0008

Highlights.

Core Findings of the Article:

• The Utah modified dylos sensor (UMDS) demonstrated measurement variation in calibration comparisons of new UMDS vs. contaminated UMDS, and new UMDS vs. cleaned UMDS

• Reasons for measurement variation include inadequate cleaning by the procedure used in this study, degradation of the laser, degradation of the sensor or altered fan flow rates

• A reference instrument should be closely monitored during aerosol calibration measurements in order to ensure particulate concentrations across the operational range of a UMDS

VITAE

Dr. Leon Pahler

Leon Pahler, PhD, MPH, CAIH is an Associate Professor in the Department of Family and Preventive Medicine at the University of Utah with industrial hygiene teaching and research responsibilities. He has a master’s degree in Organic chemistry, a Ph.D. in Organic and Heterocyclic Chemistry, and a master’s degree in Public Health with an emphasis in industrial hygiene. Dr. Pahler is the Director of the Hazardous Substance Academic Training (HSAT) Program at the Rocky Mountain Center for Occupational and Environmental Health (RMCOEH) Education Research Center (ERC).

Dr. Bob Wong

Currently, I am the Co-Director of Research with UCEER (Utah Center of Excellence for ELSI Research). My main interest is the application of sound research practice across a variety of fields. Some of the skills and knowledge I apply include; study design, data collection systems, and quantitative statistical analysis. In 2009, I helped introduce REDCap (Research Electronic Data Capture) a software toolset and workflow methodology for electronic collection and management of research and clinical trial data to the University of Utah. I participate regularly with other REDCap consortium members (over 2000 international sites) to further develop and enhance the software.

Dr. Darrah Sleeth

Dr. Darrah Sleeth, PhD, MPH, CIH is an Associate Professor at the Rocky Mountain Center for Occupational & Environmental Health. She earned a PhD in Industrial Health and an MPH in Industrial Hygiene & Hazardous Substances from the University of Michigan, as well as a BA in Integrative Biology from the University of California, Berkeley. She has also qualified as a Certified Industrial Hygienist (CIH). Her research interests include exposure assessment for airborne respiratory hazards, particle size selective sampling methods and indoor air quality.

Dr. Rodney Handy

Rod Handy, MBA, PhD, CIH joined the RMCOEH faculty in the Department of Family & Preventive Medicine at the University of Utah on July 1, 2015, bringing with him over twenty-five years of experience in environmental and occupational health and safety. Dr. Handy is currently a Professor in the Department of Family and Preventive Medicine and the Director of Industrial Hygiene and Occupational and Environmental Health Programs, teaching courses in industrial hygiene and environmental and occupational health and safety.

Dr. Scott Collingwood

Dr. Scott Collingwood joined the Department of Pediatrics in 2009 in the capacity of Director, Environmental Monitoring for the University’s National Children’s Study (NCS) sites and as Assistant Professor. Today, Dr. Collingwood manages operational aspects, research and data collection associated with the NCS at both Salt Lake County and Cache County study sites. In the past two years, Dr. Collingwood has secured additional funding amounting to $2.15M from NIH for formative research aimed at informing the NCS. Dr. Collingwood received his undergraduate degree in Industrial Engineering from the University of Iowa and earned a PhD in Occupational & Environmental Health.