Pollutant emissions of conventional energy generators with increased renewable energy sources: a 2030 New York case study

Abstract

This work analyzes 2019 hourly data from the New York Independent System Operator (NYISO) energy generators to forecast pollutant emissions for 2030. Leveraging methods and insights from the Advanced Research Projects Agency–Energy (ARPA-E) Performance-based Energy Resource Feedback, Optimization, and Risk Management (PERFORM) project’s optimization tools and the Environmental Protection Agency (EPA) historical data, we produce a method to determine future emissions scenarios within the NYISO region to be used as part of operational studies. Integrating historical data with forecasting software, this study evaluates the impact of renewable energy variability on pollutant emissions, informing policy decisions for a sustainable energy future.

Introduction

The ongoing transformations of electricity systems, driven by technological change, and the scientific consensus over the effects of emissions from human activities on climate, present challenges for companies and territories^1–3. For different jurisdictions, one of the main difficulties is understanding the consequences of evolving generation portfolios for global and local pollutants derived from electricity production decisions. Additionally, risk management mechanisms for balancing supply from Variable Renewable Energy Sources (VRES), conventional resources, and an increasingly active demand side require better information systems assessing the emissions from these resources and determining how these stakeholders interact in alternative counterfactual scenarios. In this work, we present a methodology to estimate future emissions of pollutants from electricity generation, illustrated using 2030 future scenarios for New York State. We complement the information from sensors and devices (e.g., continuous emissions monitoring systems) with statistical and simulation models leveraging insights from analyzing the operation of the 2019 NYISO electricity generators. Our optimization module, energy load forecast, and renewable energy generation forecast for 2030 are developed with funding from the ARPA-E PERFORM project^4–11.These optimization and simulation tools are designed to address and solve problems used in system operations, including the Security-Constrained Unit Commitment (SCUC) and Security-Constrained Economic Dispatch (SCED). The SCUC and SCED optimization modules used in this framework follow the formulation introduced in our earlier work¹², to which the reader is referred for a detailed mathematical description and implementation details. The information about generators and the transmission system accounts for the evolving characteristics of the grid, including generator retirements and additions, as well as the interconnection queue. By explicitly linking renewable energy variability to unit commitment, dispatch decisions, and emissions outcomes, the framework enables the evaluation of how increasing renewable generation production changes the operations and timing of conventional resources, and the associated pollutant emissions. Our study forecasts potential pollutant emissions scenarios for selected weeks in 2030 within the NYISO region, focusing on periods that reflect high-demand operating conditions observed in 2019, and integrating the capabilities of an innovative environmental module. These forecasts are relevant for policy decisions related to emissions regulation, resource adequacy, renewable integration targets, and the design of operational and environmental constraints faced by system operators.

Our analysis offers insights into the changing dynamics of energy generation, enabling a more comprehensive understanding of how emissions levels, timing, and location respond to operational and portfolio changes, and informing policy and planning decisions aimed at balancing tradeoffs on grid reliability, energy affordability, and sustainability of evolving power systems.Guided by these considerations, we study the following research questions. First, how do changes in generation portfolios and renewable energy variability affect the level, timing and spatial distribution of pollutant emissions from conventional generators? Second, how can historical production level operational and emissions data be combined with.

forward-looking simulations to produce internally consistent emissions forecasts under future grid configurations? And, third, what insights do these forecasts provide for system planning and regulatory analysis with increased use of VRES?

The objective of this study is to use historical data on pollutant emissions from energy generators to develop a forecasting framework that characterizes emissions outcomes for 2030 with the proposed grid characteristics and producer-level operational features. we illustrate in the NYISO system. The 2019 data were obtained from the EPA^13–15, and include hourly operational and pollutant emissions reporting to the Continuous Emission Monitoring Systems (CEMS). Data for 2030 are generated through forward-looking simulations conducted. These produce counterfactual operational conditions for a stylized 2030 grid. Several modules forecast input features, optimize the operational dispatches for future 2030 scenarios, and extract the producer-level operational information from the SCUC and SCED modules.

We present a comprehensive machine learning pipeline for forecasting pollutant emissions of electric generators by combining models trained with 2019 data and simulated 2030 operating conditions. By integrating outputs from SCUC and SCED modules, the framework accounts for operational constraints and the variability introduced by renewable energy sources. We develop a feature engineering process that incorporates temporal dependencies, geographic characteristics, and meteorological conditions, with explicit attention to seasonal and spatial interactions. The resulting forecasts provide granular emissions estimates capturing both spatial and temporal variations critical for policy evaluation and system planning. Extensive validation supports the robustness of our approach. Although the empirical application illustrates using New York State information, the methodology is adaptable and scalable to other regions, time horizons, and system arrangements with analogous operational and emissions information, supporting broader efforts to evaluate emissions impacts of power system transitions.

The article is organized as follows Sect. 2 presents our literature review on methodologies used to forecast emissions of energy generators within a system. Section 3 Explores the data sources, methodology, and statistical method used to construct the 2030 emissions forecast. Section 4 Presents the model fit and results for the trained methodologies and comprehensively compares different methods. We present the total pollution and confidence interval for the three pollutants studied. Finally, Sect. 5 presents the discussion and conclusion of the work.

Literature review

Mitigating pollutant emissions is a critical priority in addressing the escalating impacts of climate change. Electric generators, particularly those reliant on fossil fuels, are significant contributors to global emissions. These generators encounter additional ramping constraints that result in increased emissions, as they must compensate for the variability introduced by the growing penetration of renewable energy sources and the rise of Peer-to-peer (P2P) markets that prioritize these renewable energies¹⁶. Consequently, fossil fuel generators often need to adjust their output more frequently to balance supply and demand. Accurate emissions forecasting is crucial for policymakers, industry stakeholders, and researchers to develop effective strategies for minimizing environmental impacts¹⁷. These efforts are supported by Independent System Operator (ISO) and Regional Transmission Organization (RTO), which are equipped with detailed operational data and are well-positioned to provide precise, location-based Emission Factor (EF). This capability enhances forecasting models for estimating grid-related pollutant emissions¹⁸.

Building on the importance of accurate pollutant emissions forecasting, this section reviews the literature in three key areas:

(1) Forecasting methods, including both traditional and machine learning approaches; (2) Economic modeling, which accounts for non-convexity in costs, emissions, and regulatory factors; and (3) The integration of emissions into the scheduling process across various sectors, particularly electricity generation. These approaches all aim to enhance the accuracy of emissions forecasting and accounting, supporting better planning and policymaking for effective emissions reduction strategies.

Forecasting approaches: traditional and machine learning

Emission forecasting plays a crucial role in energy generation planning and ensuring compliance with environmental regulations at both national and regional levels. Traditional statistical methods, such as time series analysis, trend analysis, and regression models, have been widely used for emission forecasting. For example¹⁹, employed trend analysis to predict CO₂ emissions for the top 25 emitting countries, relying on historical data from 1971 to 2007. While these models are relatively simple and interpretable, they often face challenges in capturing the complex, non-linear relationships inherent in emission processes, especially in the context of renewable energy sources.

In recent years, Machine Learning (ML) and Artificial Intelligence (AI) models have emerged as promising tools for improving the accuracy of emission forecasts²⁰. applied Artificial Neural Network (ANN), including the Group Method of Data Handling (GMDH) and Multi-Layer Perceptron (MLP), to model CO₂ emissions in European countries. Other works, such as^21,22, explored deep learning models for renewable energy forecasting, leveraging historical emission data, operational parameters, and weather conditions to produce more reliable forecasts. Despite the growing reliance on ML, traditional methods remain relevant for certain applications, especially where interpretability is a priority.

Studies like²³used evolutionary algorithms to train neural networks, demonstrating the applicability of ML to forecast power generation, demand, and carbon intensity in Ireland. Moreover²⁴, employed Monte Carlo simulations to forecast emissions, accounting for system dispatch, reserve capacity, and carbon emissions across different scenarios. These studies highlight the diversity of approaches used in emission forecasting and the challenges posed by renewable energy sources.

Uncertainty remains a significant challenge in emission forecasting, driven by factors such as data fluctuations, model assumptions, and external economic or policy influences. The literature discusses probabilistic forecasting and ensemble modeling as viable approaches to address uncertainty and improve the robustness of predictions.

More recent synthesis studies indicate that no single forecasting paradigm can be considered universally optimal for emissions prediction, particularly under conditions of increased system variability driven by high renewable energy penetration. As discussed², emissions forecasting methods span statistical, machine learning, deep learning, and hybrid approaches, with model performance being highly context dependent. Their review emphasizes that ensemble and hybrid frameworks, including boosting-based methods and optimization-enhanced machine learning models, are among the most effective strategies for capturing nonlinearities and variability while maintaining robust generalization performance.

In line with these findings, hybrid forecasting approaches that integrate machine learning or ensemble models with complementary statistical or optimization components have been increasingly recommended. Such frameworks enable the simultaneous representation of temporal dynamics and complex feature interactions, addressing key limitations of standalone models³. This study therefore focuses on ensemble and boosting-based regression models within a hybrid framework, as they are well suited to capture the increased variability introduced by renewable energy sources while preserving a level of interpretability that remains essential for emissions analysis and policy-relevant decision making.

Economic modeling in electric markets: non-convexity, costs, and regulations

Economic modeling for emission reductions in electric markets faces challenges related to non-convexity in both costs and emissions, alongside the complexity introduced by regulatory frameworks. Fossil fuel power plants, including coal, natural gas, and oil-based generators, continue to dominate the energy mix in many regions, contributing substantially to global pollutant emissions. Studies such as²⁵ focus on integrating non-linear emissions curves, which are piecewise linearized, into the SCUC optimization process, ensuring that emissions are accounted for in operational decisions.

The increased integration of renewable energy sources such as wind and solar further complicates economic modeling efforts. These sources introduce variability in both costs and emissions due to their intermittent nature. For example²⁶, explores the use of negative pricing for wind generators in unit commitment as a mechanism to ensure dispatch priority for wind, thus reducing emissions while managing grid stability.

Regulatory aspects also play a crucial role in shaping emission reduction strategies²⁷. examined the impact of both direct (e.g., tax credits) and indirect (e.g., carbon tax) subsidies on renewable energy investment and emissions. The author found that while both subsidies contribute to lower emissions, indirect subsidies may inadvertently lead to lower investments in renewable energy. Emissions trading systems (ETS), such as the European Union’s ETS, heavily rely on accurate emission forecasts to set appropriate caps and allocate allowances. Similarly, in the U.S., emission forecasting is an essential component of state implementation plans under the Clean Air Act²⁸.

Incorporating emissions into scheduling: energy and other sectors

Incorporating emissions into scheduling processes is a growing trend, particularly in energy markets but also in sectors like manufacturing and transportation. The internalization of environmental externalities into scheduling decisions is increasingly recognized as a key approach to reducing emissions²⁹. explored the integration of emissions trading directly into the SCUC optimization problem, employing Particle Swarm Optimization (PSO) to address emissions reduction in a smart grid environment. Another study by²⁵integrated pollutant emissions directly into the unit commitment optimization problem, converting emissions into monetary costs. Moreover³⁰, presents a model examining how oligopolistic firms producing partially differentiated products can integrate emissions into their energy scheduling through two regulatory approaches: Cap-and-Trade and Taxes. It demonstrates that well-designed regulations not only reduce pollution but can also enhance firms’ profits by enabling them to coordinate production cuts.

Beyond the energy sector³¹, analyzed emissions trading in production planning, creating a dynamic model that allowed a manufacturer to choose between green and conventional production technologies. The model accounted for stochastic market prices for emissions allowances and demonstrated the financial benefits of adopting green technologies.

Machine learning methods have also been applied to integrate emissions forecasting into energy scheduling³². leveraged deep learning techniques to forecast renewable energy output, helping grid operators balance demand and minimize emissions. The literature consistently emphasizes the importance of incorporating emissions considerations into scheduling decisions to meet environmental goals. The reviewed literature underscores the critical role of emission forecasting in mitigating environmental impacts. Traditional statistical methods remain relevant but are increasingly complemented by ML and AI techniques, particularly in capturing the non-linear dynamics of renewable energy sources. Economic modeling for emission.

reductions faces unique challenges in electric markets, especially concerning the non-convexity of costs and emissions. Additionally, the integration of emissions into scheduling processes is essential for reducing environmental footprints in both energy and other sectors. Across these domains, the goal of improving forecast accuracy and incorporating emissions into operational decisions remains paramount.

Methodology

Our methodology is aimed at modeling the stochastic nature of pollutant emissions from energy generators, incorporating their generation profiles and characteristics. The structure built seeks to understand the relationship between the independent features and the individual pollutants studied. This framework allowed us to forecast emissions levels in 2030 using the output of the SCUC and SCED optimization module for NYISO as well as the forecasted renewable energy sources and load profiles that account for the grid topology and demand requirements.

Methodology overview

We initiated our forecasting process by obtaining projections for renewable energy sources and load for 2030 through collaboration with the ARPA-E PERFORM project. These projections consist of ensembles of internally consistent scenarios for demand, wind, and solar generation, capturing uncertainty and correlation across renewable resources and load. These forecasts provided crucial inputs regarding expected renewable energy outputs and electricity demand. Utilizing the 2030 grid topology and the forecasted renewable energy sources and load from the previous step, we conducted SCUC and SCED optimization runs. SCUC ensured optimal unit commitment considering transmission capacities and operational constraints, while SCED refined generation dispatch to economically meet forecasted electricity demand. Running the optimization across multiple scenarios allows the resulting emissions forecasts to reflect variability in renewable availability and demand conditions, rather than a single deterministic outcome. Following optimization outputs, we engaged in data preparation and feature engineering. This initial stage involved cleaning datasets, normalizing variables, handling missing values, and addressing outliers. Additionally, new influential features were engineered to capture non-linear relationships with pollutant emissions. The subsequent stage focused on training machine learning models using historical data from 2019. We partitioned datasets into training and testing sets using a temporal split to preserve data sequence integrity.

Model training included fine-tuning hyper-parameters and leveraging advanced methodologies to enhance predictive accuracy. Evaluation of model performance was crucial as we aimed to strike a balance between accuracy and interpretability. This was achieved by using appropriate metrics to validate the accuracy of pollutant-level forecasts. Our approach considered a comprehensive set of features, detailed in Table 1, and incorporated additional complexities to enhance forecasting precision. Finally, we introduced lagged variables and feature interactions to capture temporal dependencies in pollutant levels. Lagged variables provided insights into historical trends, while interactions among variables revealed non-linear relationships, enriching our understanding of emissions dynamics.

Table 1.

Independent variables descriptive statistics.

Variable	Mean	Min	25%	Median	75%	Max	Std. dev
SO2_RATE (lbs/mmBtu)	0.00	0.00	0.00	0.00	0.00	0.48	0.01
NOX _RATE (lbs/mmBtu)	0.01	0.00	0.00	0.00	0.00	1.50	0.06
CO2_RATE (tons/mmBtu)	0.01	0.00	0.00	0.00	0.00	0.11	0.02
HEAT_INPUT (mmBtu)	129.06	0.00	0.00	0.00	0.00	9,298.10	467.98
GLOAD (MW)	11.39	0.00	0.00	0.00	0.00	990.00	45.65
SLOAD (1000lb/hr)	9.25	0.00	0.00	0.00	0.00	1,458.00	79.19
PG	56.15	0.00	0.00	19.87	46.70	1,027.00	123.56
QG	8.82	-50.68	0.00	0.00	9.58	247.58	37.12
QMAX	68.35	9.00	14.00	29.30	39.70	718.90	117.51
QMIN	-54.53	-564.90	-31.20	-23.00	-11.00	-7.10	93.65
VG	1.02	1.00	1.02	1.02	1.02	1.03	0.01
PMAX	99.12	12.90	20.00	41.90	56.70	1,027.00	170.26
PMIN	29.10	1.00	11.00	16.00	37.00	150.00	31.80
NameplateCapacity	99.12	12.90	20.00	41.90	56.70	1,027.00	170.26
NameplatePowerFactor	0.92	0.80	0.85	0.90	1.00	1.00	0.07
SummerCapacity	92.17	8.70	17.50	34.15	54.33	988.50	163.29
WinterCapacity	97.43	10.10	22.75	41.20	65.10	977.80	161.20

Recognizing the influence of temporal and seasonal patterns on pollutant levels, we incorporated Fourier series features. These features capture periodic variations, such as daily, monthly, and seasonal cycles, contributing to a more accurate representation of the underlying data. In total, our model leveraged over 90 features, combining the variables listed in Table 2 with lagged variables, interactions, Principal Component Analysis (PCA) derived factors, and Fourier series features to comprehensively capture the complexity of the pollutant level dynamics. across a range of renewable generation and load.

Table 2.

List of independent variables.

Variable	Description
GLOAD (MW)	Gross load in megawatts.
PG	Real power generated in MW.
QG	Reactive power generated in MW.
QMAX	Maximum reactive power limit.
QMIN	Minimum reactive power limit.
VG	Voltage magnitude.
PMAX	Maximum real power limit.
PMIN	Minimum real power limit.
NameplateCapacity	Nameplate capacity.
NameplatePowerFactor	Power factor at nameplate capacity.
SummerCapacity	Generator capacity summer months.
WinterCapacity	Generator capacity winter months.
MinimumLoad	Minimum generator load.
Gencoordlat	Latitude of the generator.
Gencoordlon	Longitude of the generator.
GenType	Type of generator.
FuelType	Type of fuel used.
County	County where is located.

scenarios. While this study does not explicitly model alternative policy-driven energy mix scenarios, the use of scenario-based renewable and load projections, together with the 2030 NYISO system design represented in the SCUC and SCED optimization modules, allows the analysis to reflect a broad range of plausible operating conditions and associated emissions outcomes.

The analyzed week was selected from historical NYISO data as the period in 2019 with the highest energy consumption and was used to generate the projected 2030 demand and renewable generation profiles as well as optimization runs for the SCUC and SCED Modules. The 2nd of August was subsequently chosen at random from this week for detailed presentation of the results, in order to avoid selection bias.

Figure 1 presents an overview of the proposed methodological framework, which integrates renewable generation and load forecasting, SCUC and SCED dispatch optimization, data engineering, and ML based emissions forecasting. The figure illustrates the flow of information from scenario generation through model training, validation, and results analysis. Outputs from the SCUC and SCED optimization models are subsequently processed through a data engineering and feature engineering pipeline that includes data cleaning, consistency checks, outlier detection, normalization, and feature construction, following established practices in energy systems and data-driven modeling^36–38.

Fig. 1.

Methodological framework.

Methodological pipeline and reproducibility

Beyond the high-level workflow described in Sect. 3.3, the applied methodology follows a structured and transferable machine learning pipeline aligned with established power system analytics frameworks¹. Although the models are trained and validated using data from the NYISO, the modeling approach is designed to be applicable to other electricity markets, subject to appropriate adaptation to system-specific market rules, generation portfolios, and reserve policies.

Machine learning pipeline, hyperparameter tuning and validation strategy

Model development is conducted using a consistent and reproducible pipeline that includes feature preprocessing, model training, hyperparameter tuning, and performance evaluation. In addition to regularized linear models (Lasso, Ridge, and Elastic Net), non-linear ensemble methods such as eXtreme Gradient Boosting (XGBoost) are evaluated to capture complex relationships between generator operation and pollutant emissions. Hyperparameters refer to model configuration parameters that are specified prior to training and are not directly learned from the data. These parameters govern model complexity and learning behavior, such as regularization strength in linear models or tree depth and learning rate in ensemble methods. Proper selection of hyperparameters is essential to balance model flexibility and generalization performance. Hyperparameter optimization is performed systematically for all candidate models using grid-based search procedures applied exclusively to the training data. Candidate configurations are evaluated through cross-validation to ensure robust model selection and to prevent overfitting. For regularized regression models, penalty strengths and mixing parameters are explored over predefined ranges, while for ensemble methods, parameters such as maximum tree depth, learning rate, and number of estimators are tuned. Model performance during tuning and final evaluation is assessed using regression-appropriate metrics (e.g. Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE), etc.). Final performance is reported on a held-out test set, providing an unbiased estimate of out-of-sample predictive capability. Additional robustness checks include comparisons across pollutants and generator categories, as well as stability assessments of model coefficients and feature importance rankings.

Model selection and interpretability

Although non-linear ensemble models often yield higher predictive accuracy, model selection explicitly considers the trade-off between performance and interpretability. While this study prioritizes accurate emissions forecasting rather than decision-based risk classification, interpretability remains important for understanding the drivers of pollutant variability. To this end, feature importance measures and SHapley Additive exPlanations (SHAP) based explanations are used to analyze the contribution and directional impact of key predictors across pollutants.

Reproducibility and data accessibility

The datasets used in this study, including renewable generation, load demand, emissions data, and grid topology inputs, are described in detail in Sect. 3.1. That section also documents data provenance, validation procedures, and access restrictions associated with each source. All modeling and machine learning experiments were implemented using a Python-based framework built on standard open-source libraries for data processing, optimization, and statistical learning. To ensure reproducibility, all experiments were executed with fixed random seeds, and preprocessing, model training, and evaluation steps were applied consistently across training and testing datasets. While certain emissions datasets obtained from the EPA Clean Air Markets Division cannot be publicly redistributed due to licensing constraints, the full modeling workflow, feature engineering procedures, and validation strategies are documented to facilitate replication using equivalent or authorized data sources. The codebase supporting this analysis is available from the authors upon reasonable request.

Propensity score matching methodology

Our research project employed Propensity Score Matching (PSM) to assign emission profiles to electric generators with missing data. The aim was to match these generators with those for which we had complete information, leveraging their known characteristics to estimate the missing emissions profiles for CO₂, SO₂, and NO_x. Propensity score matching is a statistical technique that reduces selection bias by balancing covariates between treated and control groups. Following³⁹, the propensity score summarizes the information contained in the covariates X_i such that, when potential outcomes are independent of treatment conditional on X_i, they are also independent of treatment conditional on the propensity score.

The propensity score represents the conditional probability of assignment to a treatment given a set of observed covariates. We used a logistic regression model to estimate the propensity scores due to its effectiveness in binary classification tasks and its capability to handle the probabilistic nature of matching procedures. The logistic regression model is defined as:

where e_i denotes the propensity score for generator i, x_ij are the covariates used in the model, and β_j are the regression coefficients.

The covariates x_ij used in the propensity score model were consistent with those available for the emissions model, as detailed in Table 2. To ensure the model’s generalization and robustness, we employed cross-validation techniques. The dataset was split into training and testing subsets, and the performance of the propensity score model was evaluated on the testing set to minimize discrepancies between actual and predicted emissions profiles.

Following validation of the propensity score model, it was applied to predict emissions for generators with missing data. Before matching, we conducted industry-standard preprocessing and standardization procedures to enhance the accuracy and reliability of the propensity scores. These procedures included handling missing data, normalizing variables, and addressing outliers.

While our primary validation method was cross-validation, which provides insight into model performance on unseen data, additional sensitivity analyses and validation techniques could further strengthen the robustness of the matching process. The selection of appropriate accuracy metrics during model training ensured a solid foundation for the propensity score matching, contributing to the reliability of the estimated emissions profiles.

Machine learning methods

In this section, we explore the application of advanced machine learning techniques, including Lasso, Ridge, and Elastic Net regression, as well as ensemble methods such as XGBoost, to construct predictive models for the pollutants CO₂, NO_x, and SO₂. Our primary objective was to evaluate the accuracy of these models, with interpretation taking a secondary role in the purpose of this study. These models are developed utilizing a combination of independent variables derived from the hourly operation of electric generators. This approach allows us to capture the intricate relationships between pollutant levels and the interactions within the energy grid participants.

Lasso regression

Lasso regression, or L1 regularization, introduces a penalty term to the linear regression objective function. The objective function for Lasso regression is given by:

Here, y_i represents the response variable, β₀ is the intercept, β_j are the coefficients for the independent variables x_ij, n is the number of observations, p is the number of independent variables, and λ controls the strength of the penalty.

The advantage of Lasso regression is its ability to perform feature selection by driving some coefficients to exactly zero. This helps in identifying the most influential variables in predicting pollutant levels. However, Lasso tends to select only one variable among highly correlated variables, which may lead to instability in variable selection.

Ridge regression

Ridge regression, or L2 regularization, adds a penalty term based on the square of the magnitude of coefficients to the linear regression objective function. The objective function for Ridge regression is given by:

Similar to Lasso, λ controls the strength of the penalty.

Ridge regression addresses multicollinearity by shrinking the coefficients of correlated variables towards each other. It provides a stable solution even when the number of predictors (p) is larger than the number of observations (n). However, it does not perform variable selection as rigorously as Lasso.

Elastic net regression

Elastic Net regression combines both L1 and L2 regularization terms in the objective function, offering a compromise between Lasso and Ridge. The objective function for Elastic Net is given by:

Here, λ₁ and λ₂ control the strengths of the L1 and L2 penalties, respectively.

Elastic Net addresses the limitations of Lasso by encouraging grouping and variable selection while still benefiting from the stabilizing effects of Ridge regression⁴⁰. It is particularly useful when there are groups of correlated variables. In our pollution modeling, we will apply Lasso, Ridge, and Elastic Net regression to assess their performance in predicting CO₂, NOx, and SO₂ levels based on relevant independent variables.

Boosting and ensemble methods

We evaluated ensemble methods, such as XGBoost, in our research as an alternative to forecasting the electric generators’ pollutants. XGBoost is a gradient-boosting algorithm known for its high accuracy and scalability, and it has been successfully applied in various domains, including pollutants forecasting from different sources⁴¹.

1

where  is the predicted value for the i-th instance, K is the number of trees in the ensemble, fk is the k-th decision tree, and xi is the feature vector for the i-th instance.

During our experimentation with XGBoost, we found that it indeed demonstrated superior accuracy compared to Lasso and Ridge regression or the elastic net. The algorithm’s ability to handle non-linear relationships and its ensemble nature provided a promising avenue for improving predictive accuracy.

One of the primary concerns when evaluating this methodology was its increased complexity and the difficulty in interpreting its results compared to other tested methodologies. XGBoost operates as a black-box model, meaning that the relationship between input variables and the final prediction is not as transparent or easily explainable. Nonetheless, for this study, the main goal is to accurately represent the stochastic variability of pollutant emissions.

The hyper-parameter tuning of XGBoost hyper-parameters, such as the learning rate, maximum tree depth, and sub-sample ratios, can be a challenging task that requires extensive experimentation and computational resources. This was considered when selecting the final methodology to use for this study. In contrast, linear regression models with regularization coefficients limit the flexibility of the machine learning model; however, make it easier to fine-tune and interpret, saving valuable time and effort.

Considering the trade-off between predictive accuracy, computational time, and interpretability, we found that the benefits gained from XGBoost’s higher accuracy were sufficient to outweigh the advantages of the regularization techniques. Although the ensemble method has a black box characteristic, feature importance can be obtained and we used SHAP values to analyze the effect and correlation of the independent features with each one of the pollutants. Therefore, we chose to prioritize the XGBoost ensemble methodology as it offered a better balance between predictive performance and the ability to provide understandable and actionable insights, aligning well with the requirements of our research objectives.

Results

Accuracy metrics

Tables 3 and 4 summarize the accuracy metrics of the models evaluated in this study. Linear models, including Lasso, Ridge, and ElasticNet, underperform in comparison to the ensemble approach, as indicated by their R² Adjusted, AIC, and BIC values. For example, when predicting CO₂ emissions, ElasticNet achieves an Adjusted R² of 0.75, with AIC and BIC values of−187.664 and−186.761, respectively. Lasso performs slightly worse, with an R² Adjusted of 0.71 and higher AIC and BIC values of−170.300 and−169.398. Ridge yields results similar to ElasticNet.

Table 3.

Summary of mean absolute percentage error (MAPE) and RMSE fit metrics for the ML model by pollutant.

Pollutant	Model	MAPE (train)	MAPE (test)	RMSE (train)	RMSE (test)
CO2	Elastic net	0.360	0.359	0.233	0.247
	Lasso	0.372	0.371	0.268	0.282
	Ridge	0.361	0.360	0.234	0.248
	XGBoost	0.012	0.005	0.013	0.002
NOx	Elastic net	2.412	2.420	0.499	0.674
	Lasso	1.789	1.777	0.605	0.839
	Ridge	2.427	2.434	0.499	0.675
	XGBoost	0.008	0.004	0.020	0.008
SO2	Elastic net	0.360	0.360	0.234	0.674
	Lasso	0.372	0.371	0.268	0.839
	Ridge	0.361	0.360	0.234	0.675
	XGBoost	0.008	0.005	0.020	0.008

Table 4.

Summary of akaike information criterion (AIC), bayesian information criterion (BIC), and R-squared statistics for the ML model by pollutant.

Pollutant	Model	AIC	BIC	R2 adjusted
CO2	Elastic net	-187664.10	-186761.63	0.752
	Lasso	-170300.92	-169398.45	0.718
	Ridge	-187440.47	-186538.00	0.751
	XGBoost	-825131.48	-824229.01	0.998
NOx	Elastic net	-52912.12	-51999.85	0.368
	Lasso	-23476.83	-22564.56	0.214
	Ridge	-52719.97	-51807.69	0.367
	XGBoost	-655496.78	-654584.50	0.993
SO2	Elastic net	-52912.12	-51999.85	0.368
	Lasso	-23476.83	-22564.56	0.214
	Ridge	-52719.97	-51807.69	0.367
	XGBoost	-655496.78	-654584.50	0.993

In contrast, the XGBoost model demonstrates better accuracy across all pollutants. The CO₂ model, achieves an adjusted R² of 0.99, with much lower AIC and BIC values of−825.131 and−824.229. A similar trend is observed for NOx and SO₂, where the ensemble method outperforms the linear models in capturing the intricate relationships between features and pollutant emissions. When comparing the other accuracy metrics between training and testing datasets highlights the XGBoost model’s effectiveness in delivering accurate predictions and its ability to generalize well across datasets.

In addition to differences across modeling approaches, predictive accuracy also varies by pollutant. For all linear models, CO₂ consistently exhibits higher accuracy than NOx and SO₂, as reflected in lower MAPE, lower RMSE, and higher adjusted R² values. This behavior reflects the more direct and stable relationship between CO₂ emissions, and the operation of the generators. In contrast, NOx and SO₂ emissions are likely influenced by more complex and non-linear operational factors, resulting in reduced performance for linear specifications. Notably, these pollutant-specific differences largely disappear for the XGBoost model, which achieves similarly high predictive accuracy across CO₂, NOx, and SO₂. This result highlights the ability of ensemble tree-based methods to capture non-linear emission dynamics and supports the selection of XGBoost as the preferred modeling approach for emissions forecasting.

Training and testing using 2019 data

Figure 2a–c show the comparison of real and forecasted values for the pollutants in the test data set. The x-axis presents the hour of the month for each day on the available hourly data for August 2019. The y-axis presents the pollutant. The data presents hourly and daily seasonalities. These figures illustrate the effectiveness of the emission forecasting models for CO₂, NOX, and SO₂, by comparing historical data from 2019 with forecasted and actual test data across hourly intervals within a month. The selected forecasting methodology accurately represents the underlying stochastic process of the analyzed variables. The CO₂ emissions exhibit a regular, cyclical pattern that the model captures well, suggesting stable and predictable operational processes. In contrast, NOX and SO₂ emissions display higher variability, with notable spikes that challenge the model’s predictive capacity. Although the forecasted data aligns closely with the real test data, the slight deviations during these spikes suggest potential areas for improvement in modeling extreme events or fluctuations.

Fig. 2.

Comparison of emissions forecasts.

Analysis of 2030 results

We use the models calibrated in Sect. 4.2 to generate the emissions forecast for August 2030. We add the estimate from the model and the output information of the SCUC and SCED optimization modules, we use the models to forecast the emissions of all the electric generators proposed to be available in NYISO for 2030. Figures 3 and 4 shows the forecasted total absolute emissions for all generators on August 2nd, 2030. For the illustrative day of the analysis, CO₂, SO₂, and NOx have high variability. All pollutants shown in the chart such as Coal, Wood, Distillate Fuel Oil (DFO) and Natural Gas (NG), occur around 16:00 for CO₂ and SO₂, and between 16:00 and 18:00 for NOX. These distributions reflect emissions outcomes across multiple renewable generation and load demand scenarios, as well as across multiple generators operating within the same hour.

Fig. 3.

Comparison of emissions forecasts distribution for August 2nd, 2030 with historical 2019 emissions confidence interval.

Fig. 4.

Comparison of emissions forecasts box-plots by fuel type for August 2nd, 2030.

Figure 3 compares the hourly emission rate distributions for a representative day in 2030 with the historical 2019 hourly profiles used for model training. Compared with 2019, the 2030 projections exhibit systematic departures from the historical mean diurnal patterns across all pollutants, affecting both median emission levels and variability. While the overall daily profile remains broadly consistent, several hours show 2030 median values and interquartile ranges that partially or fully exceed the 2019 confidence intervals, indicating structural changes in.

system operation rather than random variability. In particular, CO₂ and NOX emissions display increased variability during daytime peak periods, whereas SO₂ emissions present a more concentrated peak that is temporally shifted relative to 2019. These patterns reflect changes in dispatch decisions, unit commitment, and load-following behavior projected for 2030, driven by the increased penetration of VRES and the modeled system infrastructure. From an operational perspective, the altered timing and heightened variability of emission peaks highlight the importance of hour-by-hour emissions analysis, as reliance on daily average metrics would obscure these critical intra-day dynamics.

Understanding the temporal dynamics of emissions is crucial for assessing the environmental impact of the energy grid. The aggregated and individual emission profiles for 2030 serve as valuable inputs for quantifying the emissions associated with the planned 2030 grid. This quantification is facilitated by integrating the output derived from the SCUC and SCUC optimization modules with the forecasted load and renewable energy sources generation profiles. This approach ensures a nuanced understanding of emissions patterns and aids in strategic planning for a sustainable energy future.

Figure 5a–f provide a geographic depiction of CO₂, NOx, and SO₂ emission rates from energy generators in the northeastern United States on August 2, 2030, at two specific times: 07:00 and 17:00. NOX and SO₂ rates are measured in pounds per million British thermal units (lbs/mmBtu), whereas CO₂ rates are measured in tons per million British thermal units (tons/mmBtu).

Fig. 5.

Geographic representation of energy generators for key pollutants.

The maps reveal significant spatial variations in emission rates for the different pollutants. Higher emission rates are represented by colors transitioning from blue to yellow, as indicated by the color bars on the right side of each figure.

Figure 6 illustrates the spatial variability of CO₂, NOx, and SO₂ emission rates across NYISO generators, expressed as the range (maximum minus minimum) observed at each generator location. A logarithmic color scale is used to improve the visibility and interpretation of variability across generators with different magnitudes. The results show clear spatial patterns in emission-rate variability across the NYISO region. Upstate locations generally exhibit higher absolute ranges, reflected by lighter colors, indicating larger overall variability in emission rates with relatively consistent patterns across generators. In contrast, downstate regions, particularly in and around New York City and Long Island, show lower absolute variability but greater differences between neighboring generators. This pattern is consistent with a dense and diverse generation mix and more frequent changes in dispatch.

Fig. 6.

Geographic representation of energy generators for key pollutants - ranges.

Differences across pollutants are also evident. NOx shows the widest spatial spread and the highest variability across the system, with elevated ranges observed in both upstate and downstate areas. CO₂ variability is more spatially uniform, reflecting its strong dependence on fuel type and load-following behavior. SO₂ shows the lowest overall variability, with higher ranges limited to a smaller number of generators. Overall, Fig. 6 highlights that emission-rate variability depends on both pollutant type and location, reinforcing the importance of spatially resolved emissions analysis.

Emission rates show notable temporal variations between the morning (07:00) and late afternoon (17:00) periods. This temporal distinction captures the influence of daily operational cycles of energy generators and varying energy demands. The times of 07:00 and 17:00 were chosen to capture the geographical tendencies of pollutant emissions during peak activity periods.

The analysis of CO₂, NOx, and SO₂ emission rates reveals distinct patterns throughout the day. Morning CO₂ emissions are moderately spread across the region, with concentrations in urban areas. In contrast, evening CO₂ emissions show higher concentrations, particularly around major cities, indicating increased energy consumption during peak hours. Morning NOx emissions are relatively low, with pockets of higher concentrations. By the evening, NOx emissions are significantly higher, especially in northern New York State. Similarly, morning SO₂ emissions are generally low, with a few hotspots that show increased pollution in the evening.

Feature importance analysis for static and hourly predictors

Feature importance is a metric that quantifies each independent predictor contribution in an ensemble model’s prediction task, such as a random forest or gradient-boosting model. The methodology used, does not generate a coefficient that can be interpreted and whose significance can be estimated. Feature importance provides insights into the relevance of different features in influencing model output, enabling a better understanding of the underlying data relationships and aiding in feature selection or model interpretation. This subsection analyzes the feature importance values for static and hourly predictors for the pollutants analyzed: CO₂, SO₂, and NOx.

Table 5 presents the feature importance of static predictors for each pollutant. For CO₂, the most influential feature is Lagged Pollutant Levels, with an importance score of 0.788. This dominance suggests that historical levels of CO₂ are a critical determinant of current levels, likely due to persistence in emissions trends. The second most significant feature for CO₂ is Fuel Type: Wood (0.151), indicating its substantial role in emissions relative to other fuels. Conversely, geographical and capacity-related variables exhibit minimal importance, as seen from low scores for Latitude (0.006) and Longitude (0.002). This can also be observed in the spatial characteristics of CO₂ emissions.

Table 5.

Feature importance for CO₂, SO₂, and NO_x from models trained on 2019 data (static predictors).

Feature name	CO2 importance	SO2 importance	NOx importance
Power generation (MW)	0.002	0.020	0.007
Maximum generation Capacity (MW)	0.005	0.002	0.108
Minimum generation Capacity (MW)	0.002	0.004	0.009
Fuel type: distillate fuel oil	0.002	0.000	0.006
Fuel type: natural gas	0.001	0.000	0.000
Fuel type: wood	0.151	0.000	0.000
Lagged pollutant levels	0.788	0.320	0.416
Latitude	0.006	0.043	0.008
Longitude	0.002	0.097	0.014
Latitude-longitude interaction	0.001	0.000	0.038
Temperature (°F)	0.002	0.050	0.023
Wind speed (m/s)	0.001	0.019	0.014

For SO₂, Lagged Pollutant Levels again holds the highest importance (0.320), but geographical factors, such as Longitude (0.097) and Latitude (0.043), play a more pronounced role compared to CO₂. This finding highlights the spatial variability of SO₂ emissions. The influence of meteorological conditions, exemplified by Temperature (0.050) and Wind Speed (0.019), further underscores the dispersion mechanisms associated with this pollutant.

For NOx, Lagged Pollutant Levels (0.416) remains the dominant predictor, followed by Maximum Generation Capacity (0.108). This pattern reflects the strong dependence of NOx emissions on power generation activities. Additionally, geographical and meteorological factors contribute modestly to NOx predictions, as indicated by Longitude (0.014) and Temperature (0.023). Table 6 examines the feature importance of hourly predictors on pollutant levels. The importance of time-of-day variables is generally low for CO₂, with the highest score observed for Hour 19 (6–7 PM, 0.005), likely reflecting peak energy consumption during the evening. For SO₂, hourly importance is more pronounced, especially during Hour 15 (2–3 PM, 0.205), indicating heightened emissions during mid-afternoon hours. This trend may correspond to industrial activity patterns or diurnal temperature variations that affect SO₂ dispersion. This hypothesis, however, requires further analysis to understand the underlying causality. Additionally, Hour 16 (3–4 PM, 0.057) and Hour 22 (9–10 PM, 0.058) emerge as notable periods for SO₂ emissions. In contrast, NOx displays a relatively even distribution of importance across hours, with slightly elevated scores for Hour 21 (8–9 PM, 0.034) and Hour 10 (9–10 AM, 0.029). These findings suggest consistent emissions throughout the day. The analysis demonstrates the pivotal role of lagged pollutant levels in predicting emissions across all three pollutants, emphasizing the autoregressive nature of the data. For SO₂ and NOx, the interaction between emissions, geographical features, and hourly trends highlights the spatial and temporal dynamics of these pollutants.

Table 6.

Feature importance for CO₂, SO₂, and NO_x from models trained on 2019 data (hourly predictors).

Hour of the day	CO2 importance	SO2 importance	NOx importance
Hour 1 (midnight-1 AM)	0.000	0.000	0.021
Hour 2 (1–2 AM)	0.000	0.000	0.024
Hour 3 (2–3 AM)	0.000	0.001	0.020
Hour 4 (3–4 AM)	0.001	0.005	0.000
Hour 5 (4–5 AM)	0.001	0.001	0.000
Hour 6 (5–6 AM)	0.000	0.002	0.003
Hour 7 (6–7 AM)	0.002	0.000	0.004
Hour 8 (7–8 AM)	0.000	0.036	0.024
Hour 9 (8–9 AM)	0.002	0.000	0.019
Hour 10 (9–10 AM)	0.002	0.000	0.029
Hour 11 (10–11 AM)	0.002	0.000	0.019
Hour 12 (11 AM-Noon)	0.002	0.000	0.012
Hour 13 (Noon-1 PM)	0.001	0.000	0.016
Hour 14 (1–2 PM)	0.003	0.000	0.018
Hour 15 (2–3 PM)	0.001	0.205	0.008
Hour 16 (3–4 PM)	0.002	0.057	0.015
Hour 17 (4–5 PM)	0.004	0.003	0.020
Hour 18 (5–6 PM)	0.001	0.044	0.013
Hour 19 (6–7 PM)	0.005	0.000	0.010
Hour 20 (7–8 PM)	0.002	0.004	0.013
Hour 21 (8–9 PM)	0.003	0.018	0.034
Hour 22 (9–10 PM)	0.002	0.058	0.019
Hour 23 (10–11 PM)	0.003	0.012	0.014

To extend the ensemble-based feature importance analysis, model interpretability was examined using both global and local explanation methods. As shown in Fig. 7, SHAP results confirm the dominant influence of lagged pollutant levels across all pollutants, reflecting strong temporal persistence in emissions, with secondary contributions from fuel type, operational parameters, and spatial characteristics. Additionally, Fig. 8 presents LIME explanations, highlighting how short-term operating conditions and hour-of-day effects influence individual predictions. Although hourly and meteorological variables have limited global importance, their localized impacts capture transient dynamics not evident in aggregate rankings. Overall, the authors conclude that the consistency between the model importance, SHAP values and the LIME results supports the robustness and interpretability of the modeling framework.

Fig. 7.

Global feature importance (SHAP) for emission-rate models (top 10 features).

Fig. 8.

Local feature contributions (local interpretable model-agnostic explanations (LIME)) for representative predictions (top 10 contributing features).

Discussion

The results we present here reproduce the emission profile of energy generators for the proposed New York State power system in 2030 using historical data from 2019. Due to data limitations, our forecasting horizon was limited from August 2nd, 2030 to August 9th, 2030. Worldwide economies are committed to the integration of renewable energy sources into their energy grids. This integration is important for the environment and long-term growth of the energy system. However, maintaining the grid stability affected by the inherent variability of renewable energy sources poses a challenge for system operators. This complexity might generate negative loops in pollution as system operators seek to maintain grid stability. This study helps to narrow that bridge of information and allows participants to account for pollution profiles in their decision-making.

We analyzed and studied pollution data from 2019 leveraging EPA and the ARPA-EPERFORM project data and optimization modules. We constructed an ad-hoc ML forecasting pipeline to generate high-accuracy emissions estimates for energy generators. These estimates are used to represent the hourly emissions profiles of generators in the NYISO system for the 2030 scenario.

Due to data availability constraints and uncertainty in grid topology projections for 2030, this research focuses on a representative week of energy generation. This week was selected due to the heightened stress on the energy grid that happened in August 2019 as a consequence of increased demand and generation constraints. Areas of future research include integrating this emissions forecasting framework with assessments generator reliability by examining how operational stresses associated with renewable variability, including increased ramping, cycling and reserve provision, affect both emissions outcomes and reliability costs and performance. Particularly under high renewable energy scenarios, conventional generators and system operators need to work together in their energy contracts to guarantee grid stability, as policies to control emissions and deviations from energy contracts are likely to take place.

The primary aim of our study is forecasting emissions for the NYISO during the simulated 2030 period. The complexity introduced by advanced machine learning techniques, including Lasso, Ridge, Elastic Net regression, and ensemble methods like XGBoost, may pose challenges in the straightforward interpretation of model outcomes. However, it’s important to note that although the model lacks interpretability, its primary accomplishment lies in effectively modeling the complex underlying stochastic process to transform the energy generation profiles of non-renewable energy sources into their expected emissions.

The performance of the proposed model is consistent with results reported in the emissions forecasting literature³. employs a hybrid ARIMA-SVR framework for urban-level carbon emission forecasting and reports a MAPE of 0.096 with an R² of 97.5%. Although a direct comparison is not possible due to differences in methodology and scope, our results show comparable predictive performance. In particular, the tree-based XGBoost model informed by SCUC and SCED outputs achieves low test-set MAPE values for CO₂, NOx, and SO₂, indicating strong generalization and alignment with state-of-the-art emissions forecasting approaches.

Moreover, while our results for emissions heavily rely on 2019 data, it is imperative to note that the grid topology, energy generation profiles, and other critical factors were meticulously modeled based on the current plan and structure of the grid for 2030. Despite inherent spatial and regional limitations, we assert that our study’s adaptability allows for potential expansion to other regions, accounting for their unique characteristics.

In summary, this study develops an integrated framework to estimate generator-level pollutant emissions for a representative week in 2030 by leveraging historical data from 2019 within a hybrid operational setting that links optimization dispatch outcomes with ML emissions forecasting models. By combining SCUC and SCED outputs with generator-level operational characteristics, the approach produces hourly emissions profiles for CO₂, SO₂, and NOx at a level of granularity that is difficult to obtain using system-average emission factors alone. The empirical application to the NYISO system is intended to demonstrate the performance and forecasting accuracy of the proposed hybrid optimization and machine learning framework under realistic operational conditions, rather than to provide a direct comparison across years or to derive policy conclusions. The case study illustrates how renewable variability and operational constraints influence not only the magnitude of emissions, but also their temporal and spatial patterns.

These results highlight the value of forecasting tools that jointly represent operational decisions and emissions outcomes. While the primary contribution of this work is methodological, the proposed framework provides a foundation that could support emissions informed planning and exploratory policy analysis by enabling consistent evaluations across alternative portfolio and operational assumptions. Future research may extend this approach by considering longer simulation horizons, incorporating additional sources of uncertainty in operational inputs, and further integrating emissions estimates into operational and market design studies under increasing VRES penetration.

Abbreviations

AI

Artificial intelligence

AIC

Akaike information criterion

ANN

Artificial neural network

ARPA-E

Advanced research projects agency–energy

BIC

Bayesian information criterion

CEMS

Continuous emission monitoring systems

DFO

Distillate fuel oil

EF

Emission factor

EPA

Environmental protection agency

ETS

Emissions trading systems

GMDH

Group method of data handling

ISO

Independent system operator

LIME

Local interpretable model-agnostic explanations

MAE

Mean absolute error

MAPE

Mean absolute percentage error

ML

Machine learning

MLP

Multi-layer perceptron

NG

Natural gas

NREL

National renewable energy laboratory

NYISO

New York independent system operator

P2P

Peer-to-peer

PCA

Principal component analysis

PERFORM

Performance-based energy resource feedback, optimization, and risk management

PSM

Propensity score matching

PSO

Particle swarm optimization

RMSE

Root mean squared error

RTO

Regional transmission organization

SCED

Security-constrained economic dispatch

SCUC

Security-constrained unit commitment

SHAP

SHapley additive exPlanations

VRES

Variable renewable energy sources

XGBoost

eXtreme gradient boosting

Author contributions

A.F.R.A. and A.J.L.L. conceived the study and methodology. A.F.R.A. performed data processing, modeling, and analysis. A.J.L.L. contributed to optimization modeling and interpretation. Both authors wrote and reviewed the manuscript.

Funding

This research was supported by the Advanced Research Projects Agency–Energy (ARPA-E) of the U.S. Department of Energy through the PERFORM grant #DE-AR0001277. The opinions expressed are solely those of the authors.

Data availability

The datasets for renewable energy sources, load demand and emissions used in this study were obtained from the ARPA-E PERFORM³³ and from the EPA Clean Air Markets Division. The reliability and validity of the data was verified by ARPA-E PERFORM funded organizations, including the National Renewable Energy Laboratory (NREL) and Princeton University’s PGscen project. The grid topology for the SCED and SCUC optimization modules were implemented using an extension of the MATPOWER Optimal Scheduling Tool³⁴, along with a Python-based Machine Learning framework developed for scalability and computational efficiency. The PERFORM datasets are publicly available in the ARPA-E PERFORM repository³³. Data from the EPA Clean Air Markets Division is subject to access restrictions and is used under license; this data is therefore not publicly available but may be obtained from the authors upon reasonable request and with permission from the U.S. Environmental Protection Agency.

Code availability

The custom code used to generate the results in this study is archived in a Zenodo repository³⁵. The Zenodo record is publicly visible, but access to the files is restricted due to licensing and data-use constraints associated with third-party input datasets, including emissions data from the EPA and scenario inputs developed under the ARPA-E PERFORM. The code is available from the corresponding author upon reasonable request.

Declarations

Competing interests

The authors declare no competing interests.