Operational evaluation of the RLINE dispersion model for studies of traffic-related air pollutants

Abstract

Exposure to traffic-related air pollutants (TRAP) remains a key public health issue, and improved exposure measures are needed to support health impact and epidemiologic studies and inform regulatory responses. The recently developed Research LINE source model (RLINE), a Gaussian line source dispersion model, has been used in several epidemiologic studies of TRAP exposure, but evaluations of RLINE's performance in such applications have been limited. This study provides an operational evaluation of RLINE in which predictions of NO_x, CO and PM_2.5 are compared to observations at air quality monitoring stations located near high traffic roads in Detroit, MI. For CO and NO_x, model performance was best at sites close to major roads, during downwind conditions, during weekdays, and during certain seasons. For PM_2.5, the ability to discern local and particularly the traffic-related portion was limited, a result of high background levels, the sparseness of the monitoring network, and large uncertainties for certain processes (e.g., formation of secondary aerosols) and non-mobile sources (e.g., area, fugitive). Overall, RLINE's performance in near-road environments suggests its usefulness for estimating spatially- and temporally-resolved exposures. The study highlights considerations relevant to health impact and epidemiologic applications, including the importance of selecting appropriate pollutants, using appropriate monitoring approaches, considering prevailing wind directions during study design, and accounting for uncertainty.

1. Introduction

While controls on vehicle emissions have helped to moderate effects of increasing traffic and urbanization, exposure to traffic-related air pollutants (TRAP) remains a public health concern due to the many adverse health outcomes associated with exposure (Anderson et al., 2011; Fang et al., 2015; IARC Working Group on the Evaluation of Carcinogenic Risks to Humans, 2014; Health Effects Institute (HEI), 2010), and because many people live and work near major roads, e.g., 4% of the US population (11.3 million persons) live within 150 m of a major highway, and up to 40% in cities (Health Effects Institute (HEI), 2010; Boehmer et al., 2013). The association between TRAP exposure and adverse health outcomes revealed by health impact and epidemiologic studies plays a critical role in developing air quality policies and standards. However, exposure assessment remains a recognized weakness of these studies (Batterman et al., 2014). The most accurate approach for determining exposures, personal measurements, is rarely feasible or cost-effective given the number of subjects required and the cost, burden and other limitations of the sampling equipment. Ambient air quality monitoring can be used, particularly in time series studies, however, conventional monitoring networks are spatially too sparse to capture small-scale variation or spatial gradients, e.g., the elevated concentrations found near large roadways (Zhang and Batterman, 2013). Surrogate measures, such as the proximity to roads and traffic intensity, only indirectly indicate concentrations and have other limitations (Batterman et al., 2014). Spatially- and temporally-resolved exposures are especially needed for urban-scale cohort and panel studies (Dionisio et al., 2015).

Combined modeled frameworks, which can include pollutant emission and physically-based dispersion models can provide predictions of near-road exposures at high spatial and temporal resolution, and new components of these combined frameworks can be applied where appropriate to provide potential enhancements to modeled estimates. The recently-developed Research LINE source dispersion model (RLINE) (Snyder et al., 2013), designed specifically for near-road applications, has been used to estimate TRAP exposure in several recent epidemiologic studies (Batterman et al., 2015a; Zhai et al., 2016; Pachón et al., 2016). However, applications of dispersion models require extensive input data, and prediction accuracy and uncertainty in urban settings are not well characterized (Jerrett et al., 2005). Performance evaluations of the RLINE model (Snyder et al., 2013; Venkatram et al., 2013; Heist et al., 2013; Chang et al., 2015) show generally comparable results as other line source models that simulate dispersion from on-road traffic emissions (Rao et al., 1980; Oettl et al., 2001; Levitin et al., 2005; Ganguly and Broderick, 2008), however, these evaluations have limitations with respect to epidemiologic and other applications. For example, they often lack evaluations of daily (and sometimes annual) average concentrations of TRAP, and they rarely are performed at the urban scale needed for population-level observations of health outcomes. Instead, most evaluations have examined hourly average concentrations, used experimental tracer gases that do not undergo chemical and physical transformations, and examined small (< 1 km²) and simplified domains that contain few sources (Heist et al., 2013; Chang et al., 2015; Isakov et al., 2014). (Datasets used in earlier evaluation studies are shown in Table S1.) While providing valuable diagnostic information that can help improve models, these evaluations do not represent the complexity and scale of urban settings, which can span large and diverse areas with many emission sources. The studies that have compared RLINE predictions to observations of TRAPs have other limitations, e.g., the use of short monitoring periods, single pollutants (Snyder et al., 2013; Patton et al., 2017), examination of only annual average concentrations (Zhai et al., 2016), and limited discussions of model performance and study methodology (Pachón et al., 2016). Further, performance has not been evaluated with respect to exposure-relevant factors (e.g., meterological and emission variability seen by day-of-week and season) that could alter results and lead to exposure measurement errors and misclassification.

This study performs an operational evaluation of a combined modeling system using the RLINE and AERMOD (Cimorelli et al., 2005) dispersion models. The evaluation focuses on daily exposure measures and the traffic-related portion modeled by RLINE, in an application relevant to many epidemiologic and health impact studies. We utilize routine observations of pollutant concentrations, emissions, meteorology and other variables with the goal of characterizing prediction uncertainties and limitations of models for particular applications, and include statistical and graphical analyses to determine whether model estimates agree with observations in an overall sense (Dennis et al., 2010). Here, daily average concentrations of nitrogen oxides (NO_x), carbon monoxide (CO), and fine particulate matter (PM_2.5) measured at sites across Detroit, MI for the 2011 to 2014 period are compared to predictions from RLINE and AERMOD dispersion models, for line and point sources respectively. Performance is evaluated by pollutant, site, wind direction, meteorological condition, averaging time and other factors. We discuss implications regarding the use of RLINE in epidemiologic studies.

2. Methods

2.1. Monitoring data

The study domain, the urban and industrial Detroit area in southeast Michigan, contains five Air Quality System (AQS) monitoring stations located near high traffic roads (Fig. 1; Table 1). The “suburban” or Allen Park site (AQS ID 261630001) is 190 m southeast of Interstate 75 (I-75), which has an annual average daily traffic (AADT) volume of 89,800 (Michigan Department of Transportation (MDOT), 2014). This site is shielded on one side by a row of trees, and a power substation and a truck park border the site. The surrounding area is mostly residential with single family homes. The “industrial” or Dearborn site (AQS ID 261631008) is northeast of the Marathon Petroleum refinery in southwest Detroit and 150 m northwest of I-75 (AADT = 105,800). The “schools” or East 7 Mile site (AQS ID 261630019) is located in a small park shared by three schools, 390 m east of MI-97 (AADT = 9500) and 2000 m south of MI-102 (West 8 Mile Road). Lastly, the “near-road” and “urban” Eliza Howell sites (AQS IDs 261630093 and 261630094, respectively) are 10 and 100 m north of I-96 (AADT = 152,000) with minimal obstructions.

Fig. 1.

The modeling domain, including Michigan Department of Environmental Quality (MDEQ) monitoring stations, National Weather Service (NWS) meteorological stations, a subset of Michigan State Trunkline Highway System (i.e., ‘major’) and non-Trunkline (‘minor’) roads, all modeled roads, and large point sources of NO_x in 2012 in Wayne County. Areas around the Urban, Near-road, Industrial, and Schools sites are shown (the Suburban site is below the modeled domain).

Table 1.

Starting date (month and year) and percent above detection limit (% > DL) of hourly CO, NO, NO₂ and NO_x data at Detroit area monitoring sites.

AQS ID	Site name	Poll	Methoda	Start	End	N	DL (ppb)	% > DL (%)
261630001	suburban	CO	IGFC	1/11	12/14	32,841	500	7
		NO	TECO-42S	1/11	12/14	27,962	0.05	98
261630019	school	NO	ICHEM	1/11	12/14	33,820	10	9
		NO2	ICHEM	1/11	12/11	8,633	5	77
		NO2	ICHEM	1/12	12/14	25,187	1	100
		NOx	ICHEM	1/11	12/14	33,820	10	51
261630093	near-road	CO	EC9830T	10/11	12/11	2,076	20	100
		CO	INDiI	1/12	12/14	24,838	500	51
		NO	IGpCHEM	10/11	12/13	18,186	10	68
		NO	ICHEM	1/14	12/14	8,584	10	51
		NO2	IGpCHEM	10/11	12/13	18,186	5	93
		NO2	ICHEM	1/14	12/14	8,584	1	100
		NOx	IGpCHEM	10/11	12/13	18,186	10	90
		NOx	ICHEM	1/14	12/14	8,584	10	87
261630094	urban	CO	INDiI	10/11	12/14	27,288	500	26
		NO	IGpCHEM	10/11	12/13	19,304	10	19
		NO	ICHEM	1/14	12/14	8,583	10	16
		NO2	IGpCHEM	10/11	12/13	19,304	5	80
		NO2	ICHEM	1/14	12/14	8,583	1	100
		NOx	IGpCHEM	10/11	12/13	19,304	10	63
		NOx	ICHEM	1/14	12/14	8,583	10	61
261631008	industrial	CO	IGFC	1/12	12/14	25,876	500	10

^aMethods: ICHEM = Instrumental Chemiluminescence; IGpCHEM = Instrumental Gas-Phase Chemiluminescence; TECO-42S = Low Level NO_x Instrumental-Teco 42s Chemiluminescence; IGFC = Instrumental Gas Filter Correlation Analyzer; INDiI = Instrumental Non-dispersive Infrared. EC980T = Instrumental Gas Filter Correlation Ecotech EC9830T.

Air quality data for 2011 to 2014 were obtained from the US EPA AQS Datamart (US Environmental Protection Agency (US EPA), 2016a). Over the study period, several types of monitoring methods/instruments were used that differed in sensitivity and possibly other characteristics although all used federal reference methods (FRM) or equivalent (US Environmental Protection Agency (US EPA), 2017). Hourly concentrations of NO_x were measured at three sites, CO at four, and PM_2.5 at three. NO_x at the near-road and urban sites was monitored using gas-phase chemiluminescence and Ecotech 9814B monitors (“IGpCHEM”) from October 2011 through December 2013, and using Thermo Environmental Instruments Model 42C instrumental chemiluminescence (“ICHEM”) in 2014. NO_x at the schools site was measured using a Thermo Environmental Instruments Model 42C and by ICHEM. CO was monitored at the near-road site by instrumental gas filter correlation using an Ecotech 9830 monitor (“EC9830T”) from October through December of 2011, and a Thermo Model 48C monitor using instrumental non-dispersive infrared (“INDiI”) through 2014. CO at the urban site was measured using a Thermo Environmental Instruments Model 48C and by INDiI, and at the suburban site by an instrumental gas filter correlation analyzer (“IGFC”). CO at the industrial site was measured using a Teledyne API T300 using IGFC. PM_2.5 at the schools and suburban sites was monitored as 24-h averages using the FRM and as 1-hr averages at the suburban site using a tapered element oscillating microbalance (TEOM). PM_2.5 sites and methods are shown in Supplemental Information (SI) Table S2.

Data processing and quality checks included the following: NO and NO₂ measurements in ppb were converted to NO_x concentrations using the average conversion rate (1 μg m⁻³ NO_x = 0.5495 ppb NO_x). Only the suburban and schools sites reported PM_2.5 blanks, thus blank corrections were not used. Negative observations were set to zero. Measurements below detection limits (DLs) were omitted in most analyses, or set to ½ DL in a sensitivity analysis. Daily averages were calculated from hourly NO_x, CO and PM_2.5 measurements.

2.2. Meteorological data

For each AQS site, a unique meteorological dataset was created that combined on-site measurements with other parameters measured at nearby weather stations. Each AQS site recorded wind speed, wind direction, temperature and pressure. Additional meteorological parameters needed for dispersion modeling (surface friction velocity, convective velocity scale and surface roughness) were calculated from data collected at the Detroit City Airport (DET) National Weather Service station (National Weather Service (NWS), 2016) (see Fig. S1 for the DET wind rose) and the Pontiac, MI radiosonde site (National Oceanic and Atmospheric Administration (NOAA), 2016). Quality-checked site-specific hourly meteorological input files for 2011 through 2014 was produced using AERMET (Cimorelli et al., 2005). RLINE utilizes a subset of the variables produced: sensible heat flux, surface friction velocity, convective velocity, convective and stable planetary boundary layer heights, Monin-Obukhov length, surface roughness (back-calculated from AERMET files provided by MDEQ), wind speed, and wind direction. Given RLINE's limited error checking, hours missing any of these parameters, with the exception of convective velocity (limited by a lower bound of zero), were excluded. Excluded hours represented 6–15% of hours, depending on site.

2.3. On-road mobile source modeling

Concentrations from on-road mobile sources were predicted using a spatially- and temporally-resolved link-based emission inventory and the RLINE model. A road network consisting of 9701 links and AADT volumes for 2010 (Snyder et al., 2014; Michigan Department of Transportation (MDOT), 2014) was updated using current AADT and commercial AADT (CAADT) volumes reported in the Michigan Trunkline Highway System (which includes interstates, US and state highways) (Michigan Department of Transportation (MDOT), 2014) and a custom mapping/linking algorithm (see SI). Percentage changes in AADT and the CAADT fraction were applied to matched links' 2010 AADT, and the estimated CAADT volumes were subtracted from AADT to derive updated non-commercial volumes by link and year. For unmatched links, 2010 volumes were used, which should not significantly affect results since vehicle miles traveled (VMT) on these roads was modest (below half of the Trunkline roads). The fleet mix on each link was derived using AADT and CAADT estimates, short-term counts (usually 2–3 days of data, excluding ramps and loop measurements), and permanent traffic recorders (PTRs) in the Traffic Monitoring Information System (TMIS; Table S3) (Michigan Department of Transportation (MDOT), 2016). Because count data were sparse, especially on minor roads, fleet mix was estimated by the road's National Function Class (NFC). NFC 12 and 19 links (without traffic count data) were assigned to NFC 14 and 17, respectively (Snyder et al., 2014). Hourly data using the 13 Federal Highway Administration (FHWA) classes were averaged across days, road direction and stations, and mapped to the 8 Highway Performance Monitoring System (HMPS) classes (Decker et al., 1996). The average HMPS-by-NFC volume fractions were allocated to commercial and non-commercial traffic (Table S4), normalized and weighted by average commercial traffic fractions by NFC from the final dataset (Table S5). Hourly commercial and non-commercial volumes for each link were estimated using hour-of-day, day-of-week and monthly temporal allocation factors (TAFs) derived for Detroit area roads (Batterman et al., 2015b). Hourly commercial and non-commercial emission factors for each NFC and speed bin (speeds were assigned to morning and evening rush hours, afternoon and evening periods) were calculated for each pollutant. Finally, link emissions were calculated as the product of link-specific volume with the speed-, month-, temperature- and vehicle type-specific emission factor (described next).

Emission factors (g vehicle⁻¹ mile⁻¹) were generated using the Motor Vehicle Emission Simulator (MOVES) version 2014a (US Environmental Protection Agency (US EPA), 2015) and 2015 inputs for the Wayne, Macomb and Oakland Counties (the most populated local areas) provided by the Southeast Michigan Council of Governments. Other MOVES inputs included monthly average local temperatures in 11 bins (0–100 °F in 10° increments) (Snyder et al., 2014)) and the default barometric pressure, which was similar to local conditions (Southeast Michigan Council of Governments (SEMCOG), 2011). Following previous work (Snyder et al., 2014), emission factors for running exhaust and running evaporative modes were calculated for CO, NO_x, PM_2.5 and PM_2.5 precursors (evaporative hydrocarbon emissions), and for PM_2.5 tire-wear and brake-wear emissions. Crankcase and other emissions were omitted to reduce computational time; these emissions are small compared to exhaust emissions. Again following previous work (Snyder et al., 2014), emission factors were consolidated within a pollutant type (e.g., tire and brake wear for PM_2.5), vehicle types (MOVES sourceTypeIDs) were mapped to the HPMS vehicle classes (Table S6), and averages were calculated weighted by vehicle type counts and VMT fraction on major roads (AADT > 10,000, called “urban restricted” in MOVES) and NFC 11 and 12 in the link network and minor roads (called “urban unrestricted” and NFCs 14, 16, 17 and 19), and the number of weekday and weekend days (5 and 2, respectively). CO, NO_x and PM_2.5 emission factors were calculated by vehicle type, speed and ambient temperature.

A modified version of RLINE v1.2 was implemented. Recent updates to this model include minor changes to the horizontal and vertical dispersion formulae, and major changes to the numerical integration algorithm. We used RLINE's numerical integration method, an iteration limit of 1000, and an error limit of 0.001. The beta modules for roadside barrier and depressed roadway algorithms were not used. Modifications taken to reduce run times and facilitate the large number of hours, links and receptors simulated included omitting calculations for receptor-link distances exceeding 4000 m (these concentrations were very small), using internal loops for multi-hour runs, pre-computing emission rates, and a more flexible and efficient input and output scheme.

2.4. Point source modeling

A point source inventory of CO, NO_x and PM emissions in southeast Michigan (including Lenawee, Livingston, Macomb, Monroe, Oakland, Washtenaw and Wayne counties) was created for the years 2011–2014. We consolidated stack-level data in the National Emission Inventory (NEI) (US Environmental Protection Agency (US EPA), 2014) with facility and stack-level data in the Michigan Air Emission Reporting System (MAERS) (Michigan Department of Environmental Quality (MDEQ), 2014); emission data was available for 564 facilities. Stacks were aggregated to the facility level by assigning emissions to the main stack. A subset of 179 facilities were selected based on the 100 highest emitting facilities for each pollutant). Of these, 58 mostly smaller sources had incomplete information and were excluded. Extensive quality checks, including comparisons between MAERS and the 2011 NEI data, showed good agreement for facility-level emissions for CO and NO_x (e.g., inventories agreed mostly within 5%). PM_2.5 data showed larger discrepancies, e.g., differences between MAERS and NEI at 99 of 121 sources, and MAERS filterable emissions exceeded primary emissions (i.e., the sum of filterable and condensable PM_2.5) at 23 facilities. These discrepancies were resolved following a 3-step procedure (Dorn et al., 2013): quality checking available data; trivial gap filling using available data; and then ranked “best-guess” estimates using, in sequence, data in an NEI year, primary emissions data converted directly using facility-specific SCC conversion factors, the median PM_2.5 emission estimate generated indirectly, and lastly, the PM_2.5 estimate created by trivial gap-filling of converted values. The final point source inventory contained 121 sources and represented over 90% of regional point source emissions (Table S7).

Pollutant concentrations from point sources were predicted using the inventory, the AERMOD dispersion model (View v8.1.0; AERMOD.exe v12345) (Cimorelli et al., 2005), and the preprocessed meteorological data described earlier. Sources in Detroit were classified as “urban” (Fig. S2) with a reference population of 10⁶ and the default surface roughness (Michigan Department of Environmental Quality (MDEQ), 2015).

2.5. Background concentrations

The performance evaluation requires “background” concentrations, defined in this case as contributions from both regional sources (outside the modeled area) and local but unmodeled area and mobile sources. The background sources are not explicitly modeled because they are distant, too numerous or too difficult to simulate (Arunachalam et al., 2014), or because data are incomplete. Therefore, background at each monitor was estimated by taking the monthly geometric mean of hourly differences between observations and predictions made during hours when each monitor was upwind of the nearest largest line source; similar methods have been used in recent work (Malby et al., 2013). Missing months were imputed by linear interpolation, and then leave-one-out nearest neighbor linear regressions were performed to obtain a smoothed sequence of monthly background estimates at each monitor.

2.6. Evaluation approach and metrics

The operational evaluation, which was guided by previous RLINE evaluations (Snyder et al., 2013; Venkatram et al., 2013; Heist et al., 2013) and the literature (Chang and Hanna, 2004; Hanna and Chang, 2012), compared observed and predicted concentrations using 24-h averages, an averaging period frequently used in epidemiologic and health impact studies. This period also is supported by previous evaluations suggesting that meteorological variability makes comparisons at the hourly level “almost fruitless” (Chang and Hanna, 2004). Comparisons were made between observed values and the sum of background and predicted values. (We performed several diagnostic tests to ensure that our results were similar to those obtained when comparing the sum of predicted values to observed minus background values.) Analyses were conducted by pollutant, wind direction, monitoring site, season and day-of-week. Wind directions were defined for wind speeds exceeding 1ms⁻¹, and monitoring sites were considered to be “downwind” for directions within ± 30° of perpendicular of the largest nearby road, and “parallel” for directions within ± 15° of parallel (Venkatram et al., 2013). Daily average downwind or parallel concentrations were calculated for those hours of each (calendar) day that met these conditions if at least 6 h of valid model-observation pairs were available. Periods with fewer than 5 valid days were not considered. Seasons were defined as “winter” (Dec., Jan., Feb.), “spring” (March, April, May), “summer” (June, July, Aug.), and “fall” (Sept., Oct., Nov.).

The statistical evaluation emphasized four metrics recommended in air quality model evaluation guidelines (Chang and Hanna, 2004; Hanna and Chang, 2012). (Formulas for the metrics are listed in Table S8.) The F2 statistic, the percentage of modeled values within a factor of 2 of observed values, shows over- and under-predictions and provides a measure of overall model performance. The Spearman correlation coefficient (R_SP) assesses the similarity between ranked observations and predictions, and may be particularly appropriate for epidemiologic studies since it can indicate whether exposures are correctly ordered. The fractional bias (FB) shows the tendency to over- or under-predict, i.e., the likelihood of false positives (FB_FP) or false negatives (FB_FN). (Equal weight is given to each.) Lastly, the geometric variance V_G indicates the irreducible (“systematic”) and reducible (“random”) errors. This metric can help identify conditions where performance potentially could be improved, i.e., the percentage of errors that are reducible (% reducible) is the ratio between the natural logarithm of the reducible component of V_G and the total V_G (the product of the systematic and random components). Suggested minimum performance criteria for air quality modeling are F2 ≥ 50%, mean bias ≤30%, and V_G ≤ 1.6 (Chang and Hanna, 2004; Hanna and Chang, 2012).

3. Results

3.1. Background and unmodeled contribution

For NO_x, most hourly measurements exceeded DLs (51–100%, depending on site), and background estimates generated fell into a narrow range (15–18 ppb; Table 2). For CO, observations frequently fell below the DL for the less sensitive instruments (IGFC and INDiI), which yielded relatively high background estimates (averaging 519–671 ppb); background levels were lower (128 ppb) for the more sensitive instrument (EC9830T). Because the background estimates reflected the instrument's DL, datasets were not pooled across sites or instruments. For PM_2.5, background estimates averaged 8.8 μg m⁻³ at the schools and suburban sites, equal to 88–92% of observed levels (9.5 and 10 μg m⁻³, respectively; Table S9), and day-to-day variability was significant. Predicted contributions from point and on-road mobile sources were small (averaging from 0.1 to 0.8 μg m⁻³), and including these sources in daily background estimates did not increase the correlation between observed and estimated background levels. Thus, the performance evaluation for PM_2.5 was not considered informative, a function of the dominance of regional sources and the small signal remaining from local sources, the gaps and uncertainties of the PM_2.5 emission inventory, the absence of chemical transformations in RLINE, and the paucity of near-road PM_2.5 monitoring data.

Table 2.

Model performance for daily average NO_x and CO.

Poll	Site	Method	Days	Means (ppb)						F2	RSP	FB		VG
				Obs	Back	Model	Ncom	Com	Point			FP	FN	Irr	Red
NOx
	school	ICHEM	918	23	17	3	1.2	0	1	95	0.32	0.07	0.22	1.01	1.12
	near-road	ICHEM	334	37	16	21	18.6	2	1	92	0.58	0.17	0.17	1.01	1.18
		IGpCHEM	705	48	15	23	18.5	4	1	95	0.74	0.05	0.28	1.03	1.11
	urban	ICHEM	238	25	18	11	8.5	1	1	93	0.57	0.22	0.09	1.03	1.12
		IGpCHEM	565	26	16	12	8.5	2	1	97	0.58	0.15	0.09	1.01	1.09
CO
	suburban	IGFC	40	673	671	27	19	3	5	100	0.21	0.11	0.07	1.00	1.04
	near-road	EC9830T	82	479	128	192	180	9	4	94	0.89	0.00	0.40	1.14	1.05
		INDiI	655	667	519	291	277	9	5	99	0.45	0.21	0.01	1.04	1.03
	urban	INDiI	284	639	545	126	115	5	6	99	0.17	0.12	0.07	1.00	1.05
	industrial	IGFC	63	585	535	115	100	10	5	100	0.00	0.14	0.03	1.01	1.03

Abbreviations (listed alphabetically): Back = Modeled background contribution; Com. = Modeled contribution from commercial traffic; F2 = % of model + background within a factor of 2 of observed; FB = Fractional bias; FP = false positive; FN = false negative; ICHEM = Instrumental Chemiluminescence; IGpCHEM = Instrumental Gas-Phase Chemiluminescence; Irr = Irreducible or systematic component of VG; Model = Modeled contribution from commercial, non-commercial and point sources; Ncom. = Modeled contribution from non-commercial traffic; Obs. = Observed concentrations; Point = Modeled contribution from point sources; R_SP = Spearman's correlation coefficient; Red = reducible or random component of V_G; V_G = geometric variance.

3.2. Performance by site

For NO_x, daily mean predictions (20–38 ppb) were similar to observations (23–48 ppb; Table 2). Performance tended to decrease with distance from the roadway, e.g., R_SP was from 0.58 to 0.74 at the near-road site (10 m from I-96), 0.57 to 0.58 at the urban site (100 m from I-96), and 0.32 at the schools site (350 m from MI-97). The near-road site using the IGpCHEM monitor had the highest R_SP (Fig. 2), the lowest % reducible V_G, and the highest mean model-to-background ratio. However, this case had the highest FB, mainly because the IGpCHEM measurements (average of 48 ppb) exceeded the ICHEM measurements (37 ppb), while predictions during these periods were similar (38 and 37 ppb, respectively). Performance at other sites varied: the schools site was under-predicted; the suburban, urban and industrial sites were over-predicted; and reducible errors at all four sites exceeded systematic errors, suggesting that improvements in model inputs or parameterization could improve model performance (Additional results are shown in Tables S10-11. These tables are shown graphically in Fig. S3 A-D.).

Fig. 2.

Observed versus modeled NO_x and CO at the near-road site using the EC9830T and IGpCHEM monitors. Figures show 1:1 and factor of 2 lines. For NO_x and CO respectively, day-of-week and prevailing wind-direction comparisons are differentiated by point color and shape.

For CO, daily predictions (180–320 ppb) generally fell below observed levels (479–673 ppb). As seen for NO_x, performance tended to decrease with distance from the roadway, e.g., R_SP was 0.45–0.89 at the near-road site, 0.17 at the urban site, and 0.21 at the suburban site. Despite its proximity to I-75 (150 m), the industrial site had R_SP near zero, possibly a result of that monitor's high DL that falsely elevated the background estimates. (The estimated background averaged 91% of measurements.) This site was also adjacent to active rail lines and large industrial emission sources. Ranks of mean predictions followed observations except for the suburban and near-road EC9830T samplers; at the suburban site, predictions fell below observations, probably because this site was far from known CO sources, and lower observations were recorded at the near-road EC9830T sampler (reflecting the lower DL of the EC9830T instrument), which influenced background estimates at this site. As for NO_x, the near-road site (with the EC9830T instrument) had the highest R_SP (Fig. 2) and again, this case had the lowest ratio of reducible to overall V_G, the highest mean model-to-background ratio, but the highest FB. Patterns at the other sites were similar to those seen for NO_x: daily averages at the schools site were under-predicted; suburban, urban and industrial sites were over-predicted; and reducible errors exceeded systematic errors.

3.3. Performance by wind direction

For NO_x, downwind conditions gave higher F2 (except for one case) and higher R_SP (0.30–0.64) than parallel conditions (Table 3). The exception was the near-road site using the ICHEM monitor, but both downwind and parallel winds had high F2 (≥ 90%) and large and reducible errors (V_G ≥ 1.16, % reducible ≥ 99%), indicating the potential to improve model parameterization. Other performance metrics gave mixed results, e.g., at the urban site during downwind periods, FB was slightly lower, V_G was unchanged, and the % reducible error was lower (mainly with the ICHEM monitor). Despite some inconsistencies, the F2 and R_SP metrics results indicated better performance during downwind as compared to parallel wind conditions.

Table 3.

Model performance for daily average NO_x and CO by wind direction.

Poll	Site	Method	Wind Dir	Days	Means (ppb)						F2	RSP	FB		VG
					Obs	Back	Model	Ncom	Com	Point			FP	FN	Irr	Red
NOx
	schools	ICHEM	Downwind	134	22	17	3	1	1	1	96	0.30	0.09	0.21	1.00	1.13
			Parallel	138	28	17	2	1	0	0	80	0.16	0.04	0.44	1.08	1.25
	near-road	ICHEM	Downwind	76	44	16	25	21	2	2	91	0.37	0.16	0.23	1.00	1.22
			Parallel	71	35	16	18	16	2	1	90	0.52	0.16	0.17	1.00	1.19
		IGpCHEM	Downwind	186	61	15	28	22	4	2	90	0.60	0.02	0.36	1.10	1.09
			Parallel	150	40	15	19	15	3	0	95	0.51	0.08	0.25	1.02	1.14
	urban	ICHEM	Downwind	51	25	19	12	9	1	3	96	0.64	0.23	0.04	1.05	1.08
			Parallel	39	25	22	8	6	1	1	92	0.23	0.26	0.10	1.05	1.15
		IGpCHEM	Downwind	170	29	16	14	10	2	3	97	0.57	0.15	0.09	1.01	1.10
			Parallel	74	23	17	9	7	2	1	92	0.31	0.20	0.07	1.02	1.11
CO
	suburban	IGFC	Downwind	1	–	–	–	–	–	–	–	–	–	–	–	–
			Parallel	4	–	–	–	–	–	–	–	–	–	–	–	–
	near-road	EC9830T	Downwind	26	557	128	205	192	9	5	92	0.83	0.00	0.50	1.28	1.04
			Parallel	11	326	128	146	136	7	2	100	0.60	0.04	0.21	1.02	1.05
		INDiI	Downwind	182	685	519	297	280	9	7	100	0.44	0.18	0.01	1.03	1.03
			Parallel	53	623	518	271	260	9	2	96	0.15	0.24	0.01	1.06	1.04
	urban	INDiI	Downwind	62	651	552	138	125	5	8	98	0.29	0.13	0.07	1.01	1.05
			Parallel	19	615	561	61	56	3	2	100	−0.07	0.08	0.07	1.00	1.03
	industrial	IGFC	Downwind	1	–	–	–	–	–	–	–	–	–	–	–	–
			Parallel	2	–	–	–	–	–	–	–	–	–	–	–	–

Abbreviations (listed alphabetically): Back = Modeled background contribution; Com. = Modeled contribution from commercial traffic; F2 = % of model + background within a factor of 2 of observed; FB = Fractional bias; FP = false positive; FN = false negative; ICHEM = Instrumental Chemiluminescence; IGpCHEM = Instrumental Gas-Phase Chemiluminescence; Irr = Irreducible or systematic component of V_G; Model = Modeled contribution from commercial, non-commercial and point sources; Ncom. = Modeled contribution from non-commercial traffic; Obs. = Observed concentrations; Point = Modeled contribution from point sources; R_SP = Spearman's correlation coefficient; Red = reducible or random component of V_G; V_G = geometric variance.

Performance for CO also was generally better during downwind periods, albeit less conclusively than for NO_x. F2 exceeded 92% at all sites. The near-road and urban sites had higher R_SP (0.29–0.83) during downwind periods compared to parallel winds (−0.07 to 0.60). (Other sites had insufficient data for robust evaluations.) At the near-road site with the EC9830T monitor, which had the highest R_SP, downwind conditions increased FB and decreased F2, but the fraction of reducible to overall errors was higher. Similar results were seen at the urban site with the INDiI monitor. While limited by high DLs, the CO dataset again indicates better performance during downwind conditions.

3.4. Performance by day-of-week

For NO_x, performance on weekdays generally was better than on Saturdays and Sundays (Table 4): weekdays gave higher F2 in all but one case (near-road site with the IGpCHEM monitor), although F2 exceeded 95%, and weekdays also had higher R_SP (although the urban site with the ICHEM monitor had comparable R_SP = 0.59 on both weekends and Saturdays, though still higher than on Sunday when R_SP = 0.46). At the near-road site with the IGpCHEM monitor, R_SP was high and comparable on weekdays, Saturdays and Sundays (0.75, 0.73 and 0.72, respectively), and weekdays had more under-predictions. Given that the reducible V_G on weekdays was low at this site, however, the overall conclusion of better performance on weekdays is unchanged.

Table 4.

Model performance for daily average NO_x and CO by day type.

Poll	Site	Method	Wind Dir	Days	Means (ppb)						F2	RSP	FB		VG
					Obs	Back	Model	Ncom	Com	Point			FP	FN	Irr	Red
NOx
	schools	ICHEM	Weekday	701	23	17	3	1	0	1	96	0.38	0.07	0.22	1.01	1.11
			Saturday	120	23	17	3	1	0	1	94	0.17	0.07	0.22	1.01	1.12
			Sunday	97	22	17	3	1	0	2	91	0.02	0.12	0.22	1.00	1.17
	near-road	ICHEM	Weekday	247	40	16	22	19	2	1	96	0.65	0.13	0.19	1.00	1.16
			Saturday	43	30	16	19	17	1	1	81	0.30	0.25	0.12	1.02	1.20
			Sunday	44	25	16	19	17	1	1	82	0.33	0.37	0.04	1.14	1.14
		IGpCHEM	Weekday	506	54	15	25	19	4	1	95	0.75	0.03	0.33	1.07	1.09
			Saturday	99	39	15	22	19	2	1	99	0.73	0.09	0.14	1.00	1.09
			Sunday	100	31	15	17	15	2	1	95	0.72	0.14	0.10	1.00	1.09
	urban	ICHEM	Weekday	183	26	18	11	8	1	1	96	0.65	0.18	0.10	1.02	1.10
			Saturday	32	22	17	10	8	1	1	91	0.59	0.31	0.08	1.06	1.13
			Sunday	23	21	19	12	10	1	2	74	−0.15	0.42	0.01	1.21	1.12
		IGpCHEM	Weekday	422	28	16	12	9	2	1	98	0.59	0.12	0.10	1.00	1.08
			Saturday	77	24	16	11	9	1	2	91	0.59	0.23	0.07	1.05	1.09
			Sunday	66	21	17	9	7	1	2	97	0.46	0.27	0.06	1.06	1.10
CO
	suburban	IGFC	Weekday	27	675	671	30	21	3	5	100	0.26	0.12	0.08	1.00	1.05
			Saturday	7	668	672	22	16	2	5	100	0.57	0.09	0.06	1.00	1.03
			Sunday	6	669	672	16	14	1	1	100	−0.32	0.08	0.05	1.00	1.02
	near-road	EC9830T	Weekday	58	495	128	199	185	10	4	93	0.91	0.00	0.41	1.16	1.04
			Saturday	12	492	128	225	211	7	7	100	0.85	0.00	0.33	1.11	1.03
			Sunday	12	386	128	129	122	4	3	92	0.90	0.02	0.42	1.09	1.13
		INDiI	Weekday	496	680	519	299	283	11	5	99	0.47	0.20	0.01	1.04	1.03
			Saturday	88	646	516	278	267	6	5	100	0.36	0.23	0.02	1.04	1.04
			Sunday	71	602	515	254	244	6	4	99	0.33	0.25	0.00	1.06	1.03
	urban	INDiI	Weekday	223	639	543	132	120	5	6	99	0.17	0.12	0.06	1.00	1.04
			Saturday	36	625	551	109	98	3	8	100	0.23	0.13	0.07	1.01	1.05
			Sunday	25	665	549	106	97	3	6	96	0.01	0.09	0.11	1.00	1.07
	industrial	IGFC	Weekday	51	582	536	113	98	10	5	100	−0.06	0.14	0.03	1.01	1.03
			Saturday	4	–	–	–	–	–	–	–	–	–	–	–	–
			Sunday	8	596	540	141	128	8	5	100	−0.47	0.17	0.04	1.02	1.04

Abbreviations (listed alphabetically): Back = Modeled background contribution; Com. = Modeled contribution from commercial traffic; F2 = % of model + background within a factor of 2 of observed; FB = Fractional bias; FP = false positive; FN = false negative; ICHEM = Instrumental Chemiluminescence; IGpCHEM = Instrumental Gas-Phase Chemiluminescence; Irr = Irreducible or systematic component of V_G; Model = Modeled contribution from commercial, non-commercial and point sources; Ncom. = Modeled contribution from non-commercial traffic; Obs. = Observed concentrations; Point = Modeled contribution from point sources; R_SP = Spearman's correlation coefficient; Red = reducible or random component of V_G; V_G = geometric variance.

For CO, the evaluation by day-type was hampered by data limitations, but weekday performance appeared better. F2 exceeded 92% at all sites. The near-road site had the highest R_SP on weekends (0.47 and 0.91 for INDiI and EC9830T samplers, respectively. The suburban site had higher R_SP for Saturdays than weekdays, but the sample size was small (weekend n = 7). At the urban site, weekdays and Saturdays had higher R_SP (0.17 and 0.23) than Sundays (0.01), but all correlations were low. The other performance metrics gave mixed results.

3.5. Performance by season

For NO_x, seasonal performance trends varied by site and method, however, slightly better performance was suggested during winter (Table 5). For example, the near-road site in winter had the highest R_SP (both instruments), the highest F2 (ICHEM instrument, and nearly so with the IGpCHEM instrument), and the lowest relative reducible error. The urban site had the highest R_SP (IGpCHEM) in winter. However, trends differed at other sites, e.g., R_SP was highest in summer at the schools site and highest in spring at the urban site (ICHEM monitor), and V_G was not lowest in winter at any site.

Table 5.

Model performance for daily average NO_x and CO by season.

Poll	Site	Method	Wind Dir	Days	Means (ppb)						F2	RSP	FB		VG
					Obs	Back	Model	Ncom	Com	Point			FP	FN	Irr	Red
NOx
	schools	ICHEM	Winter	311	25	17	3	1	0	1	94	0.29	0.05	0.28	1.03	1.13
			Spring	195	23	16	2	1	0	1	91	0.21	0.06	0.25	1.02	1.13
			Summer	165	19	16	3	1	0	2	99	0.44	0.10	0.08	1.00	1.05
			Fall	247	23	17	3	1	0	2	98	0.33	0.09	0.20	1.00	1.12
	near-road	ICHEM	Winter	84	51	20	21	19	2	1	99	0.79	0.05	0.27	1.03	1.12
			Spring	85	34	15	17	15	1	1	88	0.52	0.17	0.20	1.00	1.19
			Summer	87	22	12	22	19	2	1	85	0.59	0.43	0.00	1.21	1.06
			Fall	78	41	15	25	22	2	1	96	0.58	0.13	0.14	1.00	1.12
		IGpCHEM	Winter	184	55	16	21	16	3	1	91	0.84	0.01	0.41	1.14	1.08
			Spring	168	42	15	19	15	3	1	98	0.79	0.03	0.25	1.04	1.07
			Summer	142	39	14	25	20	4	1	97	0.69	0.11	0.12	1.00	1.10
			Fall	211	53	16	28	22	4	1	96	0.71	0.06	0.27	1.03	1.12
	urban	ICHEM	Winter	69	32	27	10	8	1	1	84	0.47	0.29	0.13	1.08	1.19
			Spring	59	25	17	9	7	1	1	98	0.60	0.12	0.08	1.01	1.07
			Summer	49	17	10	11	9	1	2	94	−0.02	0.24	0.05	1.03	1.09
			Fall	61	25	16	13	10	1	2	97	0.51	0.19	0.05	1.03	1.07
		IGpCHEM	Winter	168	31	19	11	8	2	1	97	0.63	0.10	0.14	1.00	1.09
			Spring	130	24	16	10	7	1	1	100	0.56	0.13	0.07	1.01	1.06
			Summer	97	19	13	12	9	2	2	93	0.40	0.29	0.01	1.08	1.05
			Fall	170	28	16	14	10	2	2	97	0.59	0.15	0.09	1.01	1.09
CO
	suburban	IGFC	Winter	24	651	666	31	24	3	3	100	0.51	0.12	0.05	1.01	1.04
			Spring	2	–	–	–	–	–	–	–	–	–	–	–	–
			Summer	3	–	–	–	–	–	–	–	–	–	–	–	–
			Fall	11	738	684	19	10	1	7	100	−0.18	0.07	0.12	1.00	1.04
	near-road	EC9830T	Winter	25	484	128	196	183	9	4	100	0.91	0.00	0.40	1.15	1.02
			Spring	–	–	–	–	–	–	–	–	–	–	–	–	–
			Summer	–	–	–	–	–	–	–	–	–	–	–	–	–
			Fall	57	476	128	191	178	8	4	91	0.88	0.00	0.40	1.14	1.07
		INDiI	Winter	133	677	540	262	247	10	5	100	0.57	0.19	0.02	1.03	1.02
			Spring	152	651	523	229	216	8	5	99	0.34	0.17	0.02	1.02	1.03
			Summer	188	650	502	338	324	10	5	100	0.39	0.26	0.00	1.07	1.03
			Fall	182	690	517	315	300	10	5	100	0.54	0.20	0.01	1.04	1.03
	urban	INDiI	Winter	97	665	564	101	92	4	5	98	−0.05	0.09	0.09	1.00	1.05
			Spring	61	677	570	98	87	4	7	100	0.45	0.07	0.08	1.00	1.03
			Summer	55	549	506	155	143	5	7	100	0.20	0.19	0.01	1.03	1.02
			Fall	71	643	526	164	150	5	8	99	0.21	0.14	0.07	1.01	1.06
	industrial	IGFC	Winter	13	565	514	114	97	12	5	100	0.15	0.14	0.03	1.01	1.03
			Spring	15	602	536	77	64	7	6	100	−0.57	0.07	0.06	1.00	1.02
			Summer	24	583	544	121	106	9	6	100	0.26	0.15	0.02	1.02	1.02
			Fall	11	587	541	157	142	12	3	100	0.01	0.20	0.02	1.03	1.04

Abbreviations (listed alphabetically): Back = Modeled background contribution; Com. = Modeled contribution from commercial traffic; F2 = % of model + background within a factor of 2 of observed; FB = Fractional bias; FP = false positive; FN = false negative; ICHEM = Instrumental Chemiluminescence; IGpCHEM = Instrumental Gas-Phase Chemiluminescence; Irr = Irreducible or systematic component of V_G; Model = Modeled contribution from commercial, non-commercial and point sources; Ncom. = Modeled contribution from non-commercial traffic; Obs. = Observed concentrations; Point = Modeled contribution from point sources; R_SP = Spearman's correlation coefficient; Red = reducible or random component of V_G; V_G = geometric variance.

Seasonal trends for CO were inconsistent, although some measures showed better performance in winter. R_SP was highest during winter at the near-road (both monitors) and industrial sites, however, R_SP was highest in spring at the urban site and negative during winter. F2 was uniformly high (≥91% and most values approached 100%). Data limitations restrict the reliability of the CO trends.

4. Discussion

The operational evaluation characterized dispersion modeling performance for daily average concentrations of NO_x and CO at multiple sites in Detroit over a four-year period. The performance metrics often, but not always, gave consistent information, and generally met criteria laid out in evaluation guidelines (Chang and Hanna, 2004; Hanna and Chang, 2012). Some interpretations can be complex, e.g., if R_SP is low, then comparisons of FB and V_G across sites may provide little information. Most downwind NO_x and CO predictions were within a factor of two of observations (F2 > 90%), and correlation coefficients were moderate to high for NO_x (0.32–0.74), but variable for CO (0–0.89). Agreement between observed and predicted concentrations improved when monitors were downwind of major roads, as shown by high R_SP, low FB (−0.19 to 0.34 for NO_x; −0.17 to 0.50 for CO), and somewhat consistent and positive FB at the best-performing sites. We found over-prediction and increased scatter with low NO_x observations and parallel winds, high contributions from on-road sources to CO levels at the near-road monitors, and uniform background levels of NO_x (15–18 ppb) across Detroit.

Dispersion models like RLINE are expected to perform best at unobstructed sites that are close to roads since the modeled on-road sources will contribute a larger fraction of observed concentrations and since these models do not explicitly model flows around buildings and other features. (RLINE simulates near-source dispersion using a general surface roughness parameter and dispersion parameters.) For NO_x and CO, two pollutants emitted primarily from traffic-related sources in urban areas, performance improved with proximity to major roads, and the best performance in Detroit was attained at the Eliza Howell near-road site located very close to the busy I-96 freeway.

Performance was generally better during downwind as compared to parallel wind conditions. Both observed and predicted concentrations tended to be higher under downwind conditions, thus, the increased agreement may reflect the greater signal from local (on-road) emission sources. (Plume models can produce the highest concentrations at near-road receptors with winds that are parallel or near-parallel to the road, although this was never observed in the daily averages in Detroit.)

Performance was better on weekdays as compared to weekends, possibly because the more regular traffic volume and fleet mix patterns on weekdays are better represented by temporal allocation factors (Batterman et al., 2015b). In contrast, traffic patterns on weekends, especially on Sundays, are more variable. The higher traffic volumes and stop-and-go congestion on weekdays might increase emissions, and the lower speeds and greater vehicle density might affect near-road turbulence and dispersion, thus increasing concentrations. The underprediction on weekdays might result from these factors, and possibly is due to a higher diesel fraction in the fleet mix than predicted. Such speculations might be examined using diagnostic (rather than operational) evaluations that focus on rush hour periods.

Model performance appeared slightly better in winter although results varied by site and method. Potentially important seasonal changes in Detroit include: shifts in prevailing wind directions, which alter the likelihood that a monitoring site will be downwind; changes in the frequency of stability regimes; large temperature swings, which alter MOVES emission factors (impacts on NO_x are complex, Table S9) (Chan et al., 2013); changes in temperature and the atmospheric composition (especially OH⁻) that can alter pollutant lifetime and fate; and changes in regional pollutants (particularly for PM_2.5). Only some of these processes are captured in dispersion models.

While of significant interest, no evaluation for PM_2.5 is presented as results were not informative. This largely results from the limited ability to discern PM_2.5 from local sources given the strength of background and regional sources of PM_2.5, and the lack of spatially- and temporally-resolved emissions data for area and non-road mobile emissions. Area and non-road emissions of PM_2.5 can be substantial, e.g., modeled on-road mobile sources constituted 48% of NO_x and 54% of CO emissions, but only 21% of PM_2.5 emissions (Table 6). Other studies have noted very high background concentrations of PM_2.5 (> 70%) in Sacramento and London (Chen et al., 2009). Diagnostic evaluations at near-road sites measuring PM-related pollutants that are more specific to TRAPs, e.g., black carbon and ultrafine PM for combustion products, and other markers for tire, road, and brake wear, might help indicate some of the factors affecting model performance.

Table 6.

Summary of 2011 Wayne County CO, NO_x and PM_2.5 emissions from the National Emission Inventory (US Environmental Protection Agency (US EPA), 2014) in short tons (rounded to the nearest ton), percent of total emissions (bolded), and of each category (not bolded).

Emission category	CO	%	NOx	%	PM2.5	%
Non-point	7,316	3	6,307	10	1,930	38
Industrial processes	194	3	4	0	489	25
Miscellaneous area sources	< 1	0	7	0	27	1
Mobile sourcesa	107	1	872	14	689	36
Natural sources	642	9	167	3	–	–
Stationary source fuel combustion	6,347	87	5,087	81	725	38
Waste disposal, treatment and recovery	27	0	170	3	–	–
Non-road mobile sources	65,491	27	6,847	11	493	10
On-road mobile sources	129,647	54	29,767	48	1,098	21
Highway - Compressed Natural Gas	54	0	42	0	0	0
Highway - Diesel	6,260	5	15,740	53	748	68
Highway - Gasoline	123,332	95	13,985	47	349	32
Point	36,335	15	19,489	31	1,610	31
External combustion	67	0	211	1	18	1
External combustion boilers	7,422	20	10,516	54	246	15
Industrial processes	20,230	56	3,082	16	904	56
Internal combustion engines	3,193	9	1,363	7	260	16
Mobile sources*	4,702	13	2,326	12	85	5
Petroleum and solvent evaporation	13	0	20	0	52	3
Waste disposal	708	2	1,972	10	46	3
Grand Total	238,788		62,411		5,131

^aRailroad equipment and marine vessels

^*Aircraft and airport support vehicles.

4.1. Comparison to literature

Many of our findings using RLINE are consistent with prior applications (e.g., in Detroit), and diagnostic evaluations using tracer gases (e.g., SF₆). For Detroit (all-direction) hourly NO_x at the schools site, an earlier study found a mean bias of 30% and F2 was 62% (Isakov et al., 2014); and for Detroit downwind near-road NO_x and CO, F2 was 100% (Chang et al., 2015). For downwind hourly near-road NO data, F2 was 93% and the geometric mean (M_G) was 1.12 (Snyder et al., 2013). Also similar to previous work, we found positive FB at the near road site, and over-prediction and increased scatter at low NO_x concentrations (Snyder et al., 2013; Venkatram et al., 2013; Heist et al., 2013). Our estimate of the ratio of the average on-road to background CO levels at the near-road site (1.46 at the more sensitive monitor) is similar to an earlier value for Detroit (Isakov et al., 2014). Finally, similar background concentrations of NO_x across Detroit have been reported (Isakov et al., 2014). Compared to studies using tracer gases, results are also comparable. For example, downwind 3-h averages of SF₆ at near-road sites in Sacramento, California showed F2 > 80% and M_G was 1.18 (Snyder et al., 2013); using this same dataset, another study obtained F2 > 78% (Heist et al., 2013). For downwind and hourly SF₆ gas data collected in rural Idaho, F2 was 75–100% (Venkatram et al., 2013; Heist et al., 2013). Using near-road and downwind SF₆ measurements, FB was 0.05 and NMSE was 0.34 (Heist et al., 2013).

In contrast to earlier work, we did not show significant over-prediction with parallel winds (Snyder et al., 2013) or downwind peaks (Venkatram et al., 2013), and our normalized mean square error estimates were smaller than those in a recent RLINE evaluation (Heist et al., 2013). We estimated that background sources were responsible for 70–90% of NO_x at the schools site, compared to approximately 50% estimated using hourly data (Isakov et al., 2014). These differences likely arose from our estimation of background and point sources and the use of daily averages.

Operational evaluations should be distinguished from diagnostic, dynamic and probabilistic evaluations. Comparisons to the previous RLINE evaluations, which were mostly diagnostic in nature, are limited by several factors. First, we examined daily concentrations, which are relevant to many health-related applications. Second, we did not evaluate performance as a function of meteorological conditions. Lower performance and over-prediction has been reported during stable periods (Snyder et al., 2013; Venkatram et al., 2013; Heist et al., 2013). Third, performance during upwind periods was not evaluated (measurements during these periods were used to estimate background); prior work shows over-prediction and increased scatter at upwind receptors (Snyder et al., 2013; Heist et al., 2013). Fourth, our large scale and multiyear urban application used data from a sparse (though typical) air quality monitoring network, and the ability to assess spatial performance was limited. In comparison, most other studies used tracer gases, a higher density of monitoring sites, few sources, a small study domain (< 1 km²), and short study periods.

4.2. Implications of varying performance

Dispersion models can be useful in developing exposure estimates of TRAP in health-related studies owing to their ability (given requisite data) to provide estimates with high spatial and temporal resolution. However, it is important to account for model performance and exposure measurement errors, that is, differences between the measured (or predicted) exposure compared to the underlying true exposure, or exposure misclassification, the analogous term for a categorical exposure variable. These errors may vary spatially or temporally, and they may differentially affect different groups of study participants. Exposure measurement error can lead to incorrect inferences in health impact and epidemiologic studies, specifically, biased and/or imprecisely estimated effect coefficients that may be serious enough to invalidate inferences regarding the effect of pollution on health (Sheppard et al., 2012).

The operational evaluation suggested that model performance is best at near-road sites (e.g., within 10–100 m from the road) and that uncertainty increases with distance from roadways. RLINE represented much of the day-to-day variation observed in daily average concentrations, suggesting that dispersion modeling can provide near-road (and potentially on-road exposures) predictions with good fidelity: this is important since many people live or work near roads where TRAP concentrations are highest (Health Effects Institute (HEI), 2010). While these results may be driven by the ability to discern contributions from local emission sources, dispersion model performance is likely to degrade with distance in urban settings for several reasons (Jerrett et al., 2005), e.g., shifts in wind fields, the presence of unknown or unmodeled sources (including other local roads), atmospheric transformation, and other unmodeled processes. Thus, at farther distances, daily fluctuations in concentrations may be less accurately estimated. This may increase the likelihood of errors from dispersion model-based exposure estimates if study participants are exposed over a range of distances from major roads. Such studies might benefit from weighting exposure estimates by their uncertainties.

A second concern is the effect of wind direction relative to the orientation of (major) roads and locations of study participants. Dispersion models perform best at downwind receptors, i.e., when winds are approximately perpendicular to the road's orientation. Correlation between the prevailing wind direction(s), road alignment(s) and study participant locations might yield differential errors. For example, in Detroit, prevailing winds come from the west and southwest (Fig. S1 shows wind roses at two local airports). Thus, model performance will be best for roads with north-south and northwest-southeast alignments with study participants on the downwind side, and poorer for roads that are aligned with the prevailing wind directions or with participants in upwind locations. These errors were investigated in Detroit by identifying the nearest (within 150 m) major road (AADT > 10,000) for a random sample of residences (n = 4000). Most roads are aligned on a north-south or east-west axis, thus directions from a residence to the nearest major road are mainly north and south (Fig. S4). Based on prevailing winds and the largest roads, individuals living downwind are east of north-south roads (e.g., M-10, M-39, I-75), “upwind” individuals live on west of the same roads, and individuals living south or north of east-west roads (e.g., I-96, I-94) will often experience parallel winds. Even if all individuals in a study lived at similar distances and/or had similar TRAP exposure, upwind and parallel groups have an increased likelihood of exposure measurement errors. In general, population patterns and the importance of directional effects will depend on many factors, e.g., demographic clustering (e.g., of residences, schools, workplaces) (Fessenden and Roberts, 2011; Cable, 2013), geographic boundaries (mountains, coastlines), economic (real estate) and administrative (municipal boundaries) factors. Some concerns might be addressed by selecting appropriate areas or, again, by using weights to account for prediction uncertainty.

Other implications for health or epidemiologic studies arise from the day-of-week variation in model performance and the reliability of time-activity data needed to assign exposures. Consider a statistical model associating health outcomes with the prior day's exposure, e.g., outcomes on Sundays and Mondays require exposure estimates for Saturdays and Sundays. Many models use 3- to 5-day lags. With a 3-day lag, Sunday's through Wednesday's outcomes require weekend exposure data. Given lower performance of the dispersion model and greater uncertainty (as well as variability) of weekend time-activity information, exposure measurement errors may increase from Saturday through Wednesday. Thus, a study incorporating 3-day exposure lags might emphasize, weight or separately test the health data for Thursdays, Fridays and possibly Saturdays when exposure uncertainty is smaller to control for these effects. A related concern is RLINE's tendency to under-predict on weekdays, which could bias concentration-outcome relationships if the (estimated) exposure variability is compressed, increase uncertainty since health models typically include both weekday and weekend periods, and falsely attribute variation to day-of-week or weekend/weekend covariates, if used. Such effects are hypothetical. Calibrating the dispersion model (i.e., mobile source inventory, TAFs) and the exposure assumptions might help to resolve this issue.

Lastly, seasonal variation in dispersion model performance, while less consistent than the day-of-week effects, raises additional concerns in epidemiologic applications. This variation can be coupled to seasonal time-activity information that affects exposure, e.g., the summer school holiday period for children, which can increase uncertainty since the home-school-home pattern is absent or less consistent and because of increased time spent outdoors. In addition, July, August and sometimes early September traffic patterns can have greater variability, a result of summer vacation, holiday travel and decreased commuting.

4.3. Uncertainty and limitations

Comparisons between observed and predicted pollutant concentrations are affected by many factors. Our results show the importance of selecting pollutants, sites and instrumentation that together produce concentration trends that are markedly influenced by local traffic-related emissions. The ability to discern traffic-related contributions of PM_2.5 was limited, a result of high background concentrations, the lack of spatial and temporal detail for area, non-road and fugitive emissions, the omission of pollutant transformations in RLINE, and the sparseness of the monitoring network. The use of monitoring parameters more specific to TRAP, e.g., black carbon or ultrafine PM, would be valuable.

Modeling results can be affected by many factors. While detailed, the mobile source inventory used estimates of traffic volumes, time allocation factors derived from mostly larger roads, and MOVES emission factors for the greater Detroit area that may not have fully reflected local traffic volume, vehicle mix and emissions. Point sources were aggregated to the facility level, used average emission rates, and temporal variability was not modeled. Background estimates only partly accounted for regional sources and may not have fully represented short-term fluctuations and gradients. (Other studies have used complex regional chemical models to estimate background (Arunachalam et al., 2014).) The classification of downwind and parallel periods refers to only the nearest major road. We assumed that the meteorological datasets driving the model were representative and appropriate. Hours when measured concentrations were low (< DL) were omitted from the evaluation, which may artificially increase correlations by limiting analyses to those observations when local source impacts are seen. This was tested by setting values below the DL to ½ DL and repeating all analyses. This dampened some trends, e.g., the wind direction analysis of NO_x, and R_SP and other metrics changed noticeably. However, removing low values has the advantage of largely eliminating (meaningless) comparisons between modeled and measured background, which can be important if roadway impacts are small or if monitoring methods have low detection frequencies. Finally, the relatively few observations available on weekends may have influenced results.

Overall, results highlight the sensitivity of evaluation results to monitor placement, instrument sensitivity (e.g., DL), and the ability to observe contributions from local sources. Results for NO_x appear most meaningful given the NO_x instrumentation's greater sensitivity and ability to detect traffic-related emissions. In contrast, the CO evaluation was limited by low detection frequencies at some sites, which resulted in a small number of valid observations, especially when analyses were stratified by wind direction, day-of-week and season.

5. Conclusions

An operation evaluation of dispersion model performance characterized the agreement between daily average predictions and observations of traffic-related air pollutants (TRAP) in an urban scale application in Detroit, Michigan that used a detailed link-based mobile source inventory and the RLINE model. Model performance was best for locations downwind of major roads, for winds perpendicular to roads, for sites near major roads, on weekdays, and during winter and spring seasons. Model performance was best for NO_x and CO; the evaluation was not informative for PM_2.5 mainly due to the scarcity of monitors near major roads and the presence of high background levels. These findings were consistent across most sites and for the two pollutants. Performance evaluations should test a wide range of environments, utilize sampling methods that are sufficiently sensitive and ideally selective for TRAP, and use an ensemble of evaluations to provide robust and representative results. Our results are consistent with the literature, and they demonstrate factors that affect model performance for the 24-h averages commonly used in epidemiologic studies.

RLINE's performance in near-road environments suggests its usefulness for estimating spatially- and temporally-resolved exposure estimates. However, the use of dispersion models in epidemiologic studies should address factors that can influence model performance and result in exposure measurement errors, including distance and direction from the road, day-of-week and seasonal effects. Appropriate study designs and analytical techniques can help avoid exposure measurement errors and improve the exposure estimates used in health and epidemiologic studies.