A stacked Gradient Boosting–XGBoost ensemble with ridge meta-learner for accurate short-term solar PV power forecasting in smart grids

Abstract

As the solar photovoltaic (PV) penetration level increases in smart grids, precise and computationally efficient short-term forecasting becomes essential to aid operational planning and real-time energy management. However, the power produced by PV is highly nonlinear and stochastic due to variations in weather factors, which weakens the performance of single forecasting models. The aim of this work is to propose a stacked ensemble regression model that combines Gradient Boosting and XGBoost (Extreme Gradient Boosting) as base learners, with Ridge Regression as the meta-learner, for very short-term PV power prediction. The model operates using meteorological and operational parameters, such as temperature, humidity, wind profile, cloudiness distribution, and solar situation. Standard preprocessing steps (missing value imputation, feature selection, and normalization) are adopted to facilitate stable model training. An empirical study is carried out using real-world PV generation data, and the results are compared with popular gradient boosting algorithms such as Gradient Boosting, XGBoost, LightGBM (Light Gradient Boosting Machine), and CatBoost (Categorical Boosting), and machine learning models such as multilayer perceptron (MLP) and LSTM, using k-fold cross-validation. The boosted ensemble improves predictive accuracy, achieving MAE = 0.042 ± 0.002, MSE = 0.0031 ± 0.0002, and R² = 94% ± 1% under the experimental conditions used in this work. Nonparametric tests (i.e., the Wilcoxon signed-rank test and the Friedman test) show that such improvements are statistically significant (p < 0.05). Moreover, the inference latency of the proposed model is quite low, which demonstrates its suitability for near-real-time deployment in real-world smart grid scenarios. According to experimental results, lightweight ensemble learning can stand as a competitive and practical alternative to complicated deep learning methods for short-term PV power forecasting when data availability and computational budget are taken into account.

Introduction

The rapid global expansion of solar photovoltaic (PV) systems has transformed modern power systems, positioning renewable energy as a central component of sustainable smart grids. However, unlike conventional generation, solar power is inherently variable and uncertain¹. As PV penetration increases, grid operators face growing challenges in maintaining voltage stability, load balancing, and reserve scheduling. Accurate short-term solar power forecasting has therefore become a critical enabler for reliable grid operation, economic dispatch, and optimized energy management in smart grid environments². Inaccurate forecasts can lead to inefficient reserve allocation, increased operational costs, and higher reliance on backup fossil-fuel generation. Moreover, forecasting uncertainty directly impacts unit commitment decisions and real-time control strategies in grid-scale renewable integration. Consequently, developing robust and operationally feasible forecasting models is essential for enabling high-penetration PV integration in future low-carbon power systems³.

Short-term solar PV forecasting is fundamentally complex due to the stochastic and nonlinear nature of solar irradiance. Power output is strongly influenced by rapidly changing meteorological variables such as cloud cover, humidity, temperature, wind speed, and solar positional angles⁴. These variables interact in a highly nonlinear manner, making traditional linear modeling approaches insufficient. Moreover, forecast inaccuracies can propagate operational risks, including frequency deviations, inefficient reserve allocation, and increased balancing costs⁵. Consequently, forecasting models must simultaneously handle nonlinear feature interactions, high variability under dynamic weather conditions, and strict computational constraints for real-time deployment⁶. To address these challenges, various machine learning and deep learning approaches have been developed. Deep learning architectures such as RNNs, LSTMs, CNN–LSTM hybrids, and Transformer-based models have demonstrated strong capability in capturing temporal dependencies and nonlinear relationships⁷. Physics-informed models integrate clear-sky radiation principles to improve physical consistency. Tree-based boosting methods, including Random Forest, Gradient Boosting, XGBoost, LightGBM, and CatBoost, offer robust performance for structured meteorological data. Hybrid and ensemble frameworks further combine multiple learners to enhance generalization⁸.

Despite their strengths, these approaches present notable limitations. Deep learning models typically require large datasets, extensive hyperparameter tuning, and high computational resources, limiting real-time feasibility. Physics-based frameworks depend on satellite data or site-specific assumptions, reducing scalability. Standalone boosting models may suffer from sensitivity to parameter settings and limited bias–variance balancing. Ensemble methods often improve accuracy but lack systematic statistical validation and computational efficiency analysis⁹. Moreover, many existing studies evaluate models under limited experimental settings, making it difficult to generalize their effectiveness across diverse climatic and operational conditions. As a result, the practical adoption of such methods in real-world smart grid environments remains constrained¹⁰.

Despite substantial progress, three major limitations remain in current solar PV forecasting research. First, many high-performing deep learning models impose significant computational complexity, making them impractical for resource-constrained smart grid environments. Second, interpretability remains limited in complex neural architectures, reducing transparency for grid-level decision-making. Third, there is a lack of lightweight, statistically validated ensemble frameworks that balance accuracy, stability, interpretability, and deployment feasibility without reliance on satellite imagery or rigid physical assumptions. Addressing these gaps is essential for advancing forecasting solutions suitable for operational smart grid applications¹¹.

In smart grid systems, short-term PV power forecasting itself is very important because grid operations further depend on real-time control decisions. Smart grids coordinate different energy sources, battery storage systems, electric vehicles, and demand response programs to maintain voltage and frequency stability itself under high renewable energy use, and this coordination further helps manage the complex power system effectively. As per short time forecasting errors in minutes to hours, real-time energy management systems get directly affected regarding storage dispatch, congestion management, and ancillary service provisioning. Moreover, effective short-term predictions help us plan backup power better and work with small power producers more easily. This only makes island grids and mixed power systems work more safely. As per smart grid requirements, strong short-term solar power forecasting is needed for intelligent and cost-effective grid operation. Regarding grid resilience, accurate PV forecasting forms the basic foundation for efficient electricity management.

Recent studies are showing that we need smart learning systems that can change and work together for predicting power grid behavior when conditions keep changing. These flexible methods can handle the complex situations in modern electricity networks. For example, the study on mitigating concept drift challenges in smart grids shows how ensemble LSTM models can surely handle changing data patterns in power systems. Moreover, these adaptive deep learning methods work well when grid conditions keep changing over time. Chiefly, these adaptive systems make load prediction more robust, but they use computationally heavy recurrent models and need continuous updates, which creates the same problem of limiting real-time use in resource-constrained edge environments. As per short-term solar PV forecasting needs, we require simple but strong models that can handle quick weather changes without training them again and again. Regarding model performance, these systems must work well with changing weather conditions. To build simple and fast computer systems that can predict solar power generation in smart electricity networks only. Recent studies have explored transfer-learning-enabled adaptive frameworks to address concept drift in smart grid load forecasting across heterogeneous generation modalities, motivating the need for deployment-oriented forecasting solutions.

To overcome these challenges, this study proposes a lightweight stacked ensemble regression framework that integrates Gradient Boosting and XGBoost as complementary base learners, combined through a Ridge Regression meta-learner. Gradient Boosting and XGBoost effectively capture nonlinear relationships and structured feature interactions, while the Ridge meta-learner reduces variance and mitigates overfitting through regularized linear fusion of base predictions. Unlike satellite-dependent or physics-constrained models, the proposed framework relies solely on real-time meteorological and operational parameters, enhancing scalability across diverse PV installations. The architecture is computationally efficient, statistically validated through k-fold cross-validation, Wilcoxon signed-rank testing, and Friedman ranking analysis, and specifically designed for real-time smart grid deployment.

While ensemble- and stacking-based PV forecasting methods have been explored extensively in prior literature, most existing approaches prioritize pointwise accuracy using homogeneous ensembles or deep-learning–centric frameworks, often overlooking robustness, interpretability, and deployment feasibility under real-time smart grid constraints. In contrast, the present study introduces a lightweight, heterogeneous stacking framework that integrates complementary gradient-boosting learners with a regularized linear meta-learner, explicitly designed to balance predictive accuracy, robustness across data splits, and computational efficiency. Furthermore, the proposed work advances existing methodologies by incorporating statistical significance testing and residual distribution analysis to provide a more rigorous and practically relevant evaluation of forecasting performance.

The main contributions of the articles are listed below:

• i.The research work proposes a lightweight stacked ensemble method for smart grids that combines Gradient Boosting and XGBoost with Ridge Regression to forecast PV power in real-time. This framework is computationally efficient and further enhances prediction accuracy for short-term solar power generation.
• ii.Also, the proposed model gives excellent prediction results while keeping the computing time low and training costs reasonable, which helps in real power grid systems where resources are limited.
• iii.The framework is tested using k-fold cross-validation and statistical tests to ensure fair and reliable model comparison.
• iv.Besides, evaluating which weather factors are most important helps make the solar power prediction model clearer, and this analysis focuses on finding the main weather conditions that affect solar panel output to support better power grid decisions.
• v.Finally, the proposed work compares our method with top machine learning models from recent studies, showing it works better because it is more reliable, easier to understand, and can be used practically without needing satellite data or strict physics rules.

The rest of this article is structured as follows: In "Literature review" section, we summarize the state-of-the-art research in this area and discuss the strengths and limitations of existing algorithms for solar power forecasting. "Research gap and motivation" section contains an in-depth analysis on mathematical aspects of different machine learning models like Neural Network, Gradient Boosting, Stacked Regressor and also non-DL machines like LightGBM, CatBoost and versions of LSTM plus XGBoost. Model Data: "Methodology" section details the data used to build the models. "Mathematical analysis of ML models" section presents a detailed analysis of the results. "Result and analysis" section, finally shortly concludes and describes the key findings from this article.

Literature review

A research work is proposed an attention-augmented RNN in 2023 to capture weather variability effects in solar PV forecasting the same way but better. The short-term predictions became more accurate, but understanding how the model works remained the same - quite limited¹². As per the research work, tested an improved XGBoost model with adaptive feature weighting and got better results regarding prediction errors compared to regular XGBoost. As per the study, the model faced problems regarding fast changing clouds and unstable sky conditions¹³.A machine learning model that uses physics rules regarding clear-sky sunlight within a gradient boosting system. Further, this integration surely made the physical consistency better and reduced overfitting when the system worked under normal conditions. Moreover, it helped the model perform more reliably. Performance surely dropped during extreme weather events that were not included in the basic assumptions. Moreover, abnormal weather conditions caused the system to work poorly¹⁴.

It is observe that, combining satellite sun data with machine learning models like SARIMAX, XGBoost, and LSTM only. It is observe that mixed method gave better results for different areas, but it only works when high-quality satellite pictures are available, which may not be there in some places¹⁵. A auhtor worked on the problem of understanding complex time patterns in solar panel output when weather changes by definitely proposing a Transformer-based deep learning system. Moreover, the model showed better prediction accuracy than LSTM and CNN methods when tested on multiple site datasets, further proving the approach itself is more effective. However, it needed only large training datasets and high computational resources¹⁶.

The research work was showed that standalone PV forecasting models definitely perform differently across various climate zones. The study showed that, deep learning and mixed models that combine space and time features, but noted that there are only limited standard ways to measure their performance¹⁷. Moreover, study provides important findings for the academic community. In 2024, researchers actually developed an automated system to select the best features for machine learning models. This framework definitely helps improve how well these models actually work by choosing the right data features automatically. The results showed better error metrics, but this approach itself introduced further preprocessing overhead¹⁸. Essentially, they did a systematic review in 2024 of deep learning models like RNN, CNN, and GNN - all the same purpose of forecasting PV systems. Moreover, spatiotemporal models work well only when data quality is good, but these deep models perform poorly with bad data¹⁹.

As per Kim and Lee (2024), they made a forecasting model using fuzzy logic that handles uncertainty regarding weather predictions. The hybrid system made predictions more stable, but the fuzzy inference part increased the same computational cost²⁰.A reasearhc article is proposed a Bayesian ensemble method that can definitely measure uncertainty in solar PV forecasting. This method actually helps quantify how uncertain the predictions are. Fundamentally, the framework gave accurate prediction ranges but took more time for training and testing, which is the same issue seen in similar approaches²¹. Likewise, a research work has been developed based on Deep learning system for predicting solar power in short time periods. This system handles uncertainty regarding quick weather changes. The reliability improved further through confidence interval estimation, but the approach itself required significant computational resources and extensive hyperparameter tuning²².

Moverover, developed a transfer-learning model that combines multiple methods to predict solar power across different locations using small amounts of past data. Moreover, this approach helps forecast solar energy even when historical information is limited. The model worked better for places with less data, but performance itself depended on how similar the climate was between the original and new locations. Further analysis showed that climate similarity was the main factor affecting results²³. A research work has been compared three different models - SARIMAX, LSTM, and XGBoost hybrid - for predicting solar PV power over multiple time periods. The study definitely focused on finding which model works best for solar energy forecasting. Hybrid combinations performed better than individual models, but the models became more complex and harder to interpret. Further, no single model worked best across all seasons, which itself presents challenges for universal application²⁴.

Singh (2025) surely developed a combined CNN-LSTM model with weather-based grouping to handle seasonal changes. Moreover, this hybrid approach specifically addresses the effects of seasonal variability. The method actually improved prediction accuracy for low light conditions but definitely required balancing between complex models and real-time use²⁵. The Random Forest and XGBoost models for predicting solar power output for the next day, and further reported that these models showed reliable performance with clear feature importance rankings. The forecasting approach itself provided interpretable results for understanding which factors most influence solar power predictions. However, XGBoost surely remained sensitive to hyperparameter tuning²⁶. Moreover, this sensitivity affected its overall performance.As per Benitez and Singh (2025), ensemble methods were used to compare solar and wind power forecasting, and they found that ensembles with more features work better for different situations. Mostly, they said that when ensemble models are the same or too similar, the performance becomes worse²⁷.

The comparative analysis of various solar power forecasting methods is shown in Table 1. In contrast to attention-based RNNs and Transformer models that prioritize accuracy at the expense of interpretability and computational cost^12,16, the proposed stacked ensemble emphasizes a lightweight and interpretable design suitable for real-time grid deployment. Unlike physics-informed or satellite-driven frameworks that rely on external data sources and rigid assumptions^14,15, our method operates purely on commonly available meteorological and operational variables, enhancing deployability across regions. While Bayesian and probabilistic ensembles quantify uncertainty effectively^21,22, their high training overhead limits scalability. The present work positions itself as a practical middle ground by achieving statistically significant accuracy gains with low inference latency and no dependency on satellite imagery or complex physical priors. The feature absed comparative analysis is as shown in the Table 2.

Table 1.

Comparative analysis of PV forecasting methods.

Sl. No.	Model/Approach	Problem addressed	Achievement	Limitation	Year	References
1.	Attention-based RNN	Weather-driven variability	Higher short-term accuracy	Low interpretability	2023	12
2.	Adaptive XGBoost	Feature relevance variation	Lower MAE than standard XGBoost	Weak under fast cloud changes	2023	13
3.	Physics-informed ML (clear-sky wiht GB)	Physical inconsistency & overfitting	Improved physical consistency	Degrades under abnormal weather	2023	14
4.	Satellite data with ML	Spatial generalization of PV forecasts	Improved site-to-site transferability	Dependence on satellite data	2023	15
5.	Transformer-based DL	Capturing long-term temporal dependencies	Improved accuracy over LSTM/CNN	High data and computational demand	2024	16
6.	Review (DL & hybrid ML)	Lack of consistency in PV forecasting	Identified trends and research gaps	No standardized benchmarks	2024	17
7.	Feature-selection-based ML	Redundant meteorological features	Reduced error via optimized features	Added preprocessing overhead	2024	18
8.	Hybrid ML taxonomy	Fragmented hybrid forecasting methods	Structured hybrid classification	Limited reproducibility	2024	19
9.	DL architectures (CNN, RNN, GNN)	Spatiotemporal dependency modeling	Strong performance with rich data	Poor results under sparse data	2024	20
10.	Bayesian ensemble ML	Forecast uncertainty quantification	Well-calibrated prediction intervals	Long training time	2024	21
11.	Probabilistic deep ensemble	Weather-driven forecast uncertainty	Improved reliability via confidence intervals	High computational cost	2024	22
12.	Transfer-learning ensemble	Data scarcity at new PV sites	Improved cross-site generalization	Climate similarity dependency	2024	23
13.	SARIMAX–LSTM–XGBoost hybrid	Seasonal and nonlinear variability	Hybrid outperformed single models	Model complexity and interpretability	2025	24
14.	CNN–LSTM with clustering	Seasonal forecasting errors	Better low-irradiance prediction	High complexity limits real-time use	2025	25
15.	RF & XGBoost	Short-term PV forecasting accuracy	XGBoost showed robustness	Sensitive to hyperparameter tuning	2025	26
16.	Ensemble ML (PV & wind)	Generalization across RES types	Ensemble improved robustness	Performance depends on base learners	2025	27
17.	Physical with ML ensemble	Morning forecast inaccuracy	Improved early-hour prediction	Reduced midday gains	2025	28
18.	Multi-horizon LightGBM–LSTM ensemble	Intra-day forecast variability	Lower MAE across horizons	Reduced accuracy at minute-level	2024	30
19.	Attention-based CNN–LSTM	Nonlinear temporal dependencies	Outperformed conventional LSTM	Limited interpretability, longer training	2025	31

Table 2.

Feature-based comparison of solar PV power forecasting methods.

References	DL	ENS	PHY	LOW-COMP	INTP	SAT-FREE	STAT
12	✓	✗	✗	✗	✗	✓	✗
13	✗	✗	✗	✓	✓	✓	✗
14	✗	✓	✓	✗	✓	✗	✗
15	✗	✓	✗	✗	✗	✗	✗
16	✓	✗	✗	✗	✗	✓	✗
18	✗	✗	✗	✓	✓	✓	✗
20	✗	✓	✗	✗	✓	✓	✗
21	✗	✓	✗	✗	✗	✓	✓
23	✗	✓	✗	✗	✗	✓	✗
24	✓	✓	✗	✗	✗	✓	✗
Proposed	✗	✓	✗	✓	✓	✓	✓

*DL- Deep Learning based, ENS-Ensemble / Hybrid, PHY- Physics-informed, LOW-COMP- Low computational complexity, INTP-Interpretable, SAT-FREE-No satellite dependency, STAT- Statistical significance testing

Table 2 surely shows a clear comparison of different solar PV forecasting methods based on important technical features like modeling approach, computational complexity, and interpretability. Moreover, it also compares these methods on satellite dependency, statistical validation, and real-time feasibility. Deep learning and hybrid methods surely give high accuracy in predictions, but they need too much computing power and are hard to understand. Moreover, these problems make them difficult to use in real-world operations. The physics-informed and satellite methods give the same better physical consistency, but they depend on outside data and fixed assumptions. Moreover, the Bayesian and transfer-learning methods can handle uncertainty and work across different sites, but they only require too much computing power. The proposed stacked Gradient Boosting–XGBoost framework surely achieves a balanced trade-off by offering low computational complexity, interpretability, and satellite-free operation. Moreover, it provides statistically validated performance improvements, making it well-suited for real-time smart grid applications. The critical architecture and its input corresponding dimensions of the proposed model are shown in Table 3.

Table 3.

Architectures and input–output dimensions of the proposed models.

Model	Component / Layer	Type	Input Dimension	Output Dimension	Activation / Description
GBR	Input	Regression Tree Ensemble	10 features	1	Gradient boosting with CART base learners
XGB	Input	Boosted Trees	10 features	1	Regularized gradient boosting (tree-based)
LGBM	Input	Histogram-based Boosted Trees	10 features	1	Leaf-wise tree growth
CatBoost	Input	Oblivious Tree Ensemble	10 features	1	Ordered boosting, categorical handling
MLP	Input	Dense	10	10	–
LSTM	Hidden-1	Dense	10	64	ReLU
	Hidden-2	Dense	64	32	ReLU
	Output	Dense	32	1	Linear
	Input	LSTM	(1 × 10)	64	Tanh
	Output	Dense	64	1	Linear
Stacked Ensemble	Base Learners	GBR, XGB, LGBM, CatBoost, MLP	N × 10 (per model)	N × 1	Individual model predictions
	Meta-learner	Ridge Regression	N × M	N × 1	Linear fusion of base predictions
	Final Output	Regression	–	1	Final PV power forecast

Table 3 further shows the structures and settings of all machine learning and deep learning models used in this study itself. As per the study design, all tree-based models like GBR, XGBoost, LightGBM, and CatBoost use the same 10-feature input vector for fair comparison. Moreover, the MLP and LSTM models use simple designs that give good predictions while keeping the same low computational cost for short-term PV power forecasting. The stacked ensemble combines predictions from different base models using Ridge Regression, which further helps merge their strengths and improves the model performance itself. This single representation only makes the data flow and model structure clear across all stages, which helps in reproducing the framework easily.

Wang eat al. developed a multi-step prediction systems that mix physical sunlight models with ML prediction tools. As per the study, this method made morning weather predictions more accurate but showed less improvement regarding midday forecasting²⁸. Also, a multi-step combined method that joins clear-sky sunlight estimation with data-based correction itself. The hybrid method actually showed lower MAE during sunrise times, but it definitely had limited efficiency improvements when solar radiation changes were small²⁹. A novel approch is used to combines LightGBM and LSTM networks to predict solar power for different time periods during the day. Moreover, this approach uses multiple forecasting models together to improve accuracy in solar PV prediction. Chiefly, the model performed the same with better accuracy for hourly predictions but showed reduced performance for minute-level forecasts due to rapid solar changes³⁰.

A research work is developed a hybrid CNN-LSTM model with attention mechanism to capture the nonlinear time and space patterns in solar PV output itself. This approach further helps in understanding complex dependencies in solar power generation. Fundamentally the forecasting became more accurate, but the same models were difficult to understand and took more time to train, creating problems for grid-level use³¹. Transfer-learning-based adaptive frameworks for mitigating concept drift in load forecasting address non-stationarity across generation modalities, whereas the present work focuses on lightweight stacked ensembles for short-term PV generation forecasting with deployment-oriented efficiency.

The research work is proposed an adaptive ensemble LSTM framework to reduce concept drift in smart grid load forecasting. Moreover, it showed strong adaptability when consumption patterns changed. This method actually focuses on predicting load demand and definitely uses deep learning models that keep updating continuously. This study further focuses on short-term solar PV power forecasting when weather conditions change rapidly, which itself addresses the forecasting challenges. The proposed stacked Gradient Boosting-XGBoost framework actually focuses on computational efficiency and interpretability, unlike adaptive LSTM ensembles. This approach is definitely suitable for real-time deployment without needing online retraining or concept-drift tracking mechanisms. This difference surely shows that deep adaptive ensembles work well for load forecasting, while lightweight stacked ensembles are better for PV generation forecasting in smart grid systems.

Research gap and motivation

A detailed assessment of recent solar photovoltaic (PV) forecasting studies specifies substantial development in modeling complex weather-dependent power generation patterns, but several important research gaps regarding this field still need to be solved. Deep learning methods like attention-based RNNs, CNN-LSTM combinations, and Transformer models surely improve short-term prediction accuracy by learning complex time patterns. Moreover, these approaches effectively capture temporal dependencies that traditional methods often miss. Further, as per the requirements, these models need big datasets, high computing power, and lots of parameter adjustments, which stops their use in real-time smart grid systems. Regarding practical deployment, these limitations make it hard to use them in environments with limited resources. Moreover, deep models are hard to understand, which surely makes them less useful for clear decision-making in power grid systems.

Also, second, physics-based and hybrid physical-ML frameworks further improve physical consistency and early-hour prediction accuracy, where the satellite-driven model itself provides better results. However, their performance becomes poor under abnormal or quickly changing weather conditions, and their use is further limited by data availability problems, such as dependence on high-resolution satellite images or fixed physical assumptions that may not work across different sites itself. Tree-based and boosting models like Random Forest, XGBoost, LightGBM, and CatBoost surely show strong robustness against data variations. Moreover, these models have lower computational complexity compared to deep learning models. However, standalone methods surely cannot capture all the complex feature interactions properly. Moreover, they are sensitive to parameter settings and perform inconsistently during rapid cloud changes and highly variable sunlight conditions.

Fourth, ensemble and probabilistic forecasting methods surely make predictions more reliable and help measure uncertainty. Moreover, most existing studies focus on either improving accuracy or estimating uncertainty, but not both at the same time. Bayesian and deep ensemble methods work well but take much longer to train and need high computational power, making them difficult to scale for large or real-time applications where speed is the same priority.It is observe that review studies only show there are no standard ways to test and compare different AI models properly. Finding a forecasting strategy that itself works well across different situations while balancing accuracy, understanding, and efficiency remains a challenge for further research.

Motivated by the identified research gaps, the present study aims to develop a computationally efficient and highly accurate hybrid ensemble forecasting framework for short-term solar PV power prediction. The proposed approach is designed to exploit the complementary learning strengths of gradient-boosting models and neural networks while avoiding excessive computational burden. By effectively combining diverse learners, the framework reduces error variance and improves generalization performance under varying operating conditions and rapidly changing weather patterns. Unlike several existing approaches, the proposed method does not rely on external satellite data or rigid physical assumptions, thereby ensuring wider applicability across different geographical regions and data availability scenarios.

Furthermore, the ensemble architecture is structured to maintain interpretability and scalability, making it suitable for practical deployment in smart grid environments. To achieve these objectives, multiple state-of-the-art machine learning and deep learning models are systematically evaluated, and a stacked regression ensemble is introduced, integrating Gradient Boosting and XGBoost as base learners with Ridge Regression as a meta-learner. Experimental results demonstrate that the proposed ensemble significantly outperforms individual models in terms of MAE, MSE, and coefficient of determination (R²), effectively addressing the key limitations reported in existing solar PV forecasting literature.

Methodology

Data source

The dataset used in this study was collected from a grid-connected solar photovoltaic (PV) power plant operated by Energy Private Limited, located in tamil nadu. The measurements were acquired through a Supervisory Control and Data Acquisition (SCADA) and on-site meteorological monitoring system installed at the PV facility. The data acquisition infrastructure consists of calibrated meteorological sensors and electrical meters, enabling continuous monitoring of both environmental conditions and PV power output.The dataset comprises real-time operational and meteorological variables that are known to directly influence PV power generation. Specifically, the collected features include ambient temperature (2 m above ground), relative humidity, mean sea-level pressure, total precipitation, snowfall, multi-layer cloud cover (high, medium, and low), shortwave solar radiation, wind speed and wind direction at multiple heights (10 m, 80 m, and 900 mb), wind gusts, and solar geometric parameters (solar zenith angle, azimuth angle, and angle of incidence). The target variable is the generated PV power (kW) measured at the inverter output.

Data were recorded at regular time intervals through the plant monitoring system, enabling the capture of short-term fluctuations in solar generation under dynamically changing weather conditions. The inclusion of solar positional parameters allows the model to account for deterministic diurnal and seasonal variations in solar irradiance, while meteorological variables such as cloud cover, humidity, and wind conditions capture stochastic atmospheric effects. High ambient temperatures tend to reduce PV conversion efficiency due to increased cell temperature, whereas wind flow contributes to module cooling, thereby influencing instantaneous power output. Cloud cover and precipitation introduce rapid irradiance variability, which is critical for short-term forecasting.

The dataset exhibits a wide operating range of PV generation, with power output varying from near-zero values during low irradiance periods to peak generation levels of approximately 3.06 MW, reflecting realistic plant-level operating conditions. The diversity and physical relevance of the recorded features provide a robust foundation for training data-driven forecasting models capable of learning both deterministic solar geometry effects and stochastic weather-induced variations. This comprehensive dataset is therefore well-suited for evaluating the proposed stacked ensemble framework under realistic smart grid operating conditions.

Data preprocessing

Prior to model training, a systematic data preprocessing pipeline was applied to ensure data quality, consistency, and suitability for machine learning. Missing values in the dataset were handled using mean imputation for each feature column, which preserves the statistical distribution of the data while avoiding unnecessary data loss. This approach ensures continuity of time samples and prevents bias introduced by removing incomplete records.The dataset was then partitioned into input features and the target variable (generated PV power in kW). To reduce model complexity and mitigate redundancy among correlated variables, a correlation-based feature analysis was performed. The statical analysis of the data set as shown in the Table 4.

Table 4.

Statistical analysis of dataset.

Feature Name (Unit)	Mean	Std Dev	Min	25%	50%	75%	Max
Temperature at 2 m above ground (°C)	15.07	8.85	−5.35	8.39	14.8	21.29	34.9
Relative Humidity at 2 m above ground (%)	51.36	23.53	7	32	48	70	100
Mean Sea Level Pressure (hPa)	1019	7.02	998	1015	1018	1024	1047
Total Precipitation at Surface (mm)	0.032	0.1702	0	0	0	0	3.2
Snowfall Amount at Surface (mm)	0.003	0.038	0	0	0	0	1.68
Total Cloud Cover at Surface (%)	34.06	42.84	0	0	8.7	100	100
High Cloud Cover (High Layer, %)	14.46	30.71	0	0	0	9	100
Medium Cloud Cover (Mid Layer, %)	20.02	36.39	0	0	0	10	100
Low Cloud Cover (Low Layer, %)	21.37	38.01	0	0	0	10	100
Shortwave Radiation at Surface (W/m2)	387.8	278.46	0	142	382	599.9	952.3
Wind Direction at 10 m (°)	195.1	106.63	0.54	153	192	292.1	360
Wind Speed at 80 m (m/s)	18.98	12	0	10.1	16.2	26.14	66.88
Wind Direction at 80 m (°)	191.2	108.76	1.12	130	188	292	360
Wind Speed at 900 mb (m/s)	16.36	9.89	0	9.18	14.5	21.97	61.11
Wind Direction at 900 mb (°)	192.5	106.52	1.12	148	188	288	360
Wind Gust at 10 m (m/s)	20.58	12.65	0.72	11.2	18	27	84.96
Angle of Incidence (°)	50.84	26.64	3.76	29.4	47.3	69.2	121.6
Solar Zenith Angle (°)	59.98	19.86	17.7	45.3	62.1	74.35	128.4
Solar Azimuth Angle (°)	169.2	64.57	54.4	114	163	225.1	289.1
Generated PV Power (kW)	1134	937.96	475	232	972	2021	3057

Empirical distributions of representative input features and the target variable are shown in Fig. 1. The correlation heatmap of the dataset features is shown in Fig. 2. The dataset comprises 5,300 time-stamped samples; 4,240 samples (80%) were used for training and 1,060 samples (20%) were used for testing. For the LSTM model, the two-dimensional feature matrix was reshaped into a three-dimensional tensor to conform to the sequential input requirements of recurrent neural networks. Although the dataset is not inherently a univariate time series, this transformation enables the LSTM to learn short-term temporal dependencies embedded within the operational data sequence. Descriptive statistical analysis (Table 1) was conducted to characterize the central tendency and dispersion of each feature. The dataset exhibits substantial variability across environmental and operational conditions, with ambient temperature ranging from − 5.35 to 34.9 °C, total cloud cover spanning 0–100%, and shortwave radiation varying from 0 to 952.3 W/m². Generated PV power shows high variability (mean ≈ 1134.4 kW, maximum ≈ 3056.8 kW), reflecting realistic fluctuations under different weather regimes. This statistical diversity ensures that the proposed models are trained and evaluated under a wide range of operating scenarios, enhancing robustness and practical relevance.

Fig. 1.

Empirical distributions of representative input features and target variable. (a) Shortwave radiation at surface. (b) Solar zenith angle. (c) Angle of incidence. (d) Ambient temperature at 2 m. (e) Total cloud cover at surface. (f) Generated PV power output.

Fig. 2.

Correlation heatmap of dataset features.

Pearson correlation heatmap (Fig. 2) shows the same relationships between variables in a visual format. This method is used to find the straight-line connections between all variables in the data table, including only the main variable which is the solar power produced. Adding the target output in the correlation analysis surely makes it easier to understand which features are important for PV generation. Moreover, this approach strengthens how we can interpret the relevance of different features. Further, as per the analysis, strong positive and negative connections are seen regarding important factors like shortwave radiation, solar zenith angle, and angle of incidence, which matches the basic principles of solar energy conversion. Principally, variables that show weak connections with the target or have the same information as other features are less useful, so we remove them first to avoid redundancy and improve model performance. All selected variables are actually standardized using z-score normalization to ensure different features contribute equally during learning. This approach is definitely beneficial for gradient-based learners and neural network models.

Model selection for analysis

This research uses different machine learning models to predict solar power generation by considering weather and operational factors further, where the prediction process itself depends on these variables. The first step in preprocessing was handling missing values, which we surely replaced with the mean of each column. Moreover, this approach made certain that our data remained complete for analysis. After, this step, only the top ten most helpful features were selected using Select K Best method for the target variable. After we made all features the same scale, which is needed only to keep inputs consistent for algorithms that use gradients or distance, It is notice that we started with the Gradient Boosting Regressor model. The method surely builds the ensemble step by step, where each tree corrects the errors left by the previous tree. Moreover, this incremental approach helps improve the model’s accuracy with each addition. It is notice that that we used only 200 estimators with 0.1 learning rate and maximum depth settings to learn from data properly without overfitting problems. These 5 hyperparameters help the model remember things correctly only. Mostly, gradient boosting works well for structured data problems and can capture the same non-linear patterns between input features.

We also used XGBoost as per our submission requirements, which is an improved gradient boosting method with same settings regarding hyperparameters (not the stratification parameters mentioned above). It often gives better performance further due to improved regularization and computational efficiency itself. As per its ability to handle missing data and successful use in competitive modeling tasks, we used this method in our study regarding the research problem. We used joblib to save the Gradient Boosting and XGBoost models so that the deployment process itself does not lose them further. We actually used deep learning methods, so we definitely built a simple neural network with two hidden layers having 64 and 32 neurons each. The network getting trained for only 50 epochs using Adam optimizer and mean square error as the loss function. These methods are good at finding complex connections in data when there are many variables or hidden relationships between them only. The algorithm overview, as mentioned in flowchart Fig. 3. We further trained the LightGBM model itself using 500 estimators with reduced learning rate of 0.05. This menu surely allowed us to create deeper trees, giving us a chance to learn more complex patterns in the data without overfitting. Moreover, it provided better opportunities to understand sophisticated data relationships. Classical gradient-based boosting methods work slowly on very large datasets, and LightGBM with its histogram-based approach further proves itself as a perfect solution for such data requirements. Also tested one CatBoost model only with 500 iterations, learning rate of 0.05 and tree depth of 8. CatBoost actually performed very well in this dataset because it can definitely control overfitting with its default settings. Even without categorical features, it was one of the top models. The proposed stacked ensemble integrates Gradient Boosting and XGBoost as base learners with Ridge Regression as the meta-learner, while LightGBM and CatBoost are employed as standalone benchmark models for comparative evaluation.

Fig. 3.

Flowchart of proposed model.

The conceptual framework of propsoed PV forecsting method as mention in the Fig. 4. The research fitted a stacked model using two base learners Gradient Boosting and XGBoost, and further used Ridge Regression as the meta learner itself. They surely use an Ensemble method that brings together different models to make better predictions. Moreover, this approach combines the good points of each model to improve overall results. These predictions are actually combined to take the strengths of each model and definitely increase performance on new datasets. Fundamentally, we use a simple linear model as the meta-learner because it avoids overfitting and keeps the same stable predictions in the end. Also, LSTM deep learning model is used here, and though LSTM networks are meant for time-series data, the input was changed to make it work like sequential learning. The model actually had an input layer, then a 50 LSTM unit and a dense layer with 25 neurons, and definitely gave output on the final layer. The LSTM model was actually tested to see if sequential dependencies within one timestep were definitely beneficial for prediction performance. The model was trained for 50 epochs with Adam optimizer.

Fig. 4.

Conceptual framework of proposed PV forecasting.

Mathematical analysis of ML models

Neural networks

Neural networks are mostly computational models that work the same way as the human brain and form the foundation of artificial intelligence. As per the structure, these contain layers of connected neurons, with each neuron working as a mathematical function regarding data processing. The architecture has an input layer, many hidden layers, and an output layer itself. Further, these layers work together to process information. Further, as per the training process, weights get changed to reduce the difference between expected and actual outputs, and these weights affect the neural signals regarding how information flows in the network. Forward propagation itself makes predictions, and backpropagation further adjusts the weights using gradient from loss function. Neural networks need activation functions like ReLU, sigmoid, and tanh to handle complex patterns, otherwise they work the same as simple linear models. Neural networks can surely find and describe complex patterns in data, making them excellent for classifying pictures, recognising speech, and processing natural languages^32,33.

Moreover, these abilities make them very useful tools for many different applications. Neural networks are actually used in many areas like self-driving cars, medical checkups, money predictions, and robots. These systems definitely help machines make smart decisions in different fields. New computer methods for pictures and text patterns have actually made these systems work much better. These improvements definitely help machines understand and process information more easily. Mostly, new designs like GANs and BERTs have pushed forward the same areas of creating models and understanding language. Neural networks are actually very powerful, but they definitely have problems like needing too much computer power and being hard to understand. With further research, neural networks will play a bigger role in solving real-world problems and helping creativity itself^34,35.

1

2

3

4

Also, Eq. (1) shows the input data, where X is a feature matrix with n samples and m features, and y itself represents the target vector with n values for true labels. Further, this setup forms the basic structure for machine learning problems. This actually sets up the base for training the model. It definitely creates the foundation needed for the learning process. As per the neural network structure, Eq. (2) shows the change happening in the first layer regarding data processing. As per the process, we multiply the input data X with weight matrix W1 and then add bias vector b1. This is a simple linear operation regarding the data processing. The ReLU activation function further processes the output z1 and removes negative values to produce h1 itself. Equation (3) surely takes this process forward to the second layer, where it applies a similar linear transformation to the first layer’s output h1. Moreover, it uses a different weight matrix W2 and bias b2 for this transformation. Fundamentally, we get h2 from the second layer by applying ReLU to z2, which is the same as activating the result. Also, Eq. (4) actually shows the final output layer, where weight matrix W3 and bias b3 definitely multiply the changed data from the second layer h2 to make the predicted output.

5

6

7

8

As per Eq. (5), the MSE is computed to measure the average squared difference between observed values (y) and predicted values. Mean squared error is surely a common loss function used in regression problems, and moreover, smaller values indicate that the model is performing better. Also, as per Eq. (6), the R2 value shows how well the model explains the changes in data. This coefficient tells us regarding the model’s performance in capturing data variance. When we compare the sum of squared residuals to the total variance in actual data, values going close to 1 indicate a better fit. Equation (7) MAE actually shows the average difference between what we observed and what we predicted. This definitely tells us how far our predictions are from the real values. This another way to check how good a model is working. As per Eq. (8), the gradient descent method is used regarding the adjustment of model weights and biases training phase. Essentially, the updates come from MSE loss gradients for weights and biases, and then we scale them by same learning rate. Mostly, this makes it the same easy process for the model to keep reducing the loss function again and again.

Gradient boosting

Gradient boosting is a powerful ensemble learning method that combines weak learners to build strong predictive models for regression and classification tasks. This technique further improves the model accuracy by learning from previous errors, making the ensemble itself more robust and effective. Gradient boosting builds a series of decision trees where each new tree corrects the errors of the previous tree. This process further improves the model by making each tree learn from the mistakes itself. It actually starts with a basic model that uses a constant value like the mean of the target variable, then it definitely adds single trees one by one to make the loss function better. Mostly, at each step the algorithm calculates how much the predictions are wrong and treats these errors the same as pseudo-residuals³⁶. As per the model, these residuals show the errors that the next tree will correct regarding the predictions. As per the learning rate, it gives weight to each tree and adjusts all trees’ effects to slowly learn from errors and make better results regarding overfitting prevention. Gradient Boosting is surely powerful because one algorithm can solve many different problems. Moreover, need to choose the right loss function - mean squared error for regression or log-loss for binary classification³⁷.

Gradient boosting is actually used widely in machine learning because it is definitely strong, accurate, and works well with big data. XGBoost, LightGBM, and CatBoost have surely improved the performance and scalability of gradient boosting methods. Moreover, these efficient implementations allow researchers to work with large datasets more effectively. As per the frameworks, two regularisation methods using L1 and L2 penalties are supported regarding prevention of overfitting problems that come from better generalisation. These methods include advanced features like handling missing data, parallel processing and tree pruning which makes them work better. Gradient boosting surely has some problems, particularly it can overfit when datasets are small and hyperparameters need optimization. Moreover, these issues become more serious with limited training data. We surely need to optimize the number of trees, maximum tree depth and learning rate to get better performance³⁸. Moreover, these parameters must be adjusted carefully for improved results. As per current research work, the problems are being solved through early stopping and automatic hyperparameter searching methods. The gradient boosting as a key method in today’s machine learning that helps many fields like health, money matters, language work and suggestion systems^39,40. The final answer comes from adding all tree predictions together, using only a learning rate to control each tree’s contribution. As per the model, M shows the total number of trees where each tree’s contribution hm(X) is added one by one as mentiond in Eq. (9). Regarding the process, each tree gets added step by step to make the final result.

9

10

11

12

13

14

15

16

Equation (9) defines the overall prediction of the Gradient Boosting ensemble. The final predicted output for an input is obtained as the weighted sum of the predictions of all regression trees in the ensemble, where denotes the learning rate (shrinkage factor) associated with the m-th tree, and represents the prediction of the m-th regression tree. The learning rate controls the contribution of each tree to the final ensemble prediction and helps prevent overfitting by shrinking individual tree outputs.

Equation (10) describes the output of a single regression tree. The function denotes the prediction of the m-th tree for input . The mapping represents the index of the leaf node reached by in the m-th tree, and is the prediction value stored at that leaf node. The leaf value is typically computed as the mean of the target values of the training samples that fall into that leaf during tree construction. Equation (11) defines the initialization stage of the Gradient Boosting model. Before any regression trees are added, the initial model prediction is set to the mean of the target values the training samples. This constant model serves as the baseline prediction that is iteratively refined by adding regression trees in subsequent boosting iterations.

Equation (12) presents the computation of the negative gradient (residual) for the i-th training sample at the m-th boosting iteration. The term is obtained by differentiating the loss function with respect to the current ensemble prediction . In this study, the loss function is chosen as the mean squared error for regression. The negative gradient represents the direction in which the model predictions should be adjusted to reduce the loss and is used as the target for fitting the next regression tree.

Equation (13) describes the ensemble update rule. The updated prediction function at iteration is obtained by adding the prediction of the newly fitted regression tree , scaled by the learning rate , to the previous ensemble prediction . This iterative update enables the model to progressively correct the residual errors of the preceding ensemble. Equation (14) reiterates the same ensemble update mechanism. Since it expresses the same update rule as Eq. (15), one of these equations should be removed to avoid redundancy. Equations (16) and (17) gives tprediction of the Gradient Boosting model after boosting iterations. The final ensemble output is equal to , which represents the cumulative contribution of all regression trees, each weighted by its corresponding learning rate. This formulation highlights that Gradient Boosting builds the final model sequentially by adding weak learners in an additive manner.

Hybrid model

In this study, the stacked regressor is constructed using two heterogeneous base learners, namely Gradient Boosting (M₁) and XGBoost (M₂), whose predictions are combined through a Ridge Regression meta-learner.The base learners are trained on the original input feature space X, and their predictions are stacked to form a meta-feature matrix that is used to train the Ridge meta-model. This design enables effective bias–variance balancing by combining complementary learning behaviors while maintaining a lightweight and interpretable ensemble structure.

The hybrid model with stacked regressors is only a method that combines many basic models together to make better predictions with more accuracy. This method uses both ways together - it first trains each model separately on the same data, and then only uses their predictions as input features for a better model called metamodel. The meta-model itself combines outputs from all boosted decision trees to further handle base model biases and variations effectively. It actually works when base models are good and different from each other - you can stack simple linear models to make them easier to use, or stack decision trees to definitely make them more flexible. When you actually look at data from different angles, your model definitely learns to work better with new information. Further, this method is actually rarely used for reliable prediction, definitely when running regression tasks⁴¹. To implement a stacked regressor, we actually need to pick base models, split data into training and validation sets, and set up the right meta-model for stacking. Base models can definitely include linear regression, support vector regression, and gradient boosting, while the meta-model can actually be a simple linear regression or a more complex random forest model⁴². Chiefly, cross-validation is the same common method used for making meta-model predictions that prevents data leakage and overfitting. As per machine learning practices, stacking gives good results but it is hard to do on computers and needs careful setting of parameters for both main models and the top model. Regarding the process, both base models and metamodel require proper tuning of their settings. Model selection and combining different models surely affects how well the system works. Moreover, this process needs expert knowledge to do it properly. Recent improvements in automatic machine learning have surely made stacking better by automating model selection and enhancement. Moreover, this has expanded its use in important sectors like finance, healthcare, and recommendation systems⁴³.

17

18

19

20

21

22

23

The stacked regressor mathematical model is built only on three base models - decision tree (M1), random forest (M2), and linear regression (M3). These models form the foundation of this approach. Equation (17) actually shows that these basic models learn from input features X and definitely produce y-values that match what they predict. The outputs from these base models are combined to create a new feature matrix , which itself is a horizontal stack of the individual predictions, as shown in Eq. (18). This process further helps in building the stacked feature representation.

Equation (19) shows that the meta-model Mmeta itself learns how the stacked predictions connect with the actual target values y, and this further helps in understanding the relationship between them. The meta-model surely takes this stacked matrix as input and generates the final prediction . Moreover, this process combines all base model outputs into one final result. Further, Eqs. (20) and (21) show how the model works on new test data. The model itself uses these equations for further testing on unseen data. As per the stacking process, the base models make predictions on the test set and these predictions are combined to form a new test matrix .

As per Eq. (22), the meta-model takes the stacked predictions from the test set and gives the final output prediction We actually evaluate the model’s performance using the Mean Squared Error (MSE) metric, which is definitely shown in Eq. (23). The stacked regressor mathematical model is built only on three base models – decision tree (M1), random forest (M2), and linear regression (M3).

LightGBM

LightGBM (Light Gradient Boosting Machine) is a sophisticated implementation of the Gradient This boosting framework is surely designed to make computations faster and handle larger datasets better. Moreover, it keeps the same level of prediction accuracy while doing so. Microsoft surely developed LightGBM, which uses a histogram-based method to change continuous features into separate bins. Moreover, this approach greatly reduces memory use and computational time⁴⁴. LightGBM uses leaf-wise growth approach, which is different from conventional gradient boosting techniques that expand trees level-wise. This approach itself helps the model perform further better than traditional methods. This method surely improves the leaf that gives the best loss reduction. Moreover, it creates deeper trees with more accurate splits. Further, lightGBM itself handles large datasets with many dimensions very well and further performs excellently in regression, classification, and ranking tasks. It is observe important features like built-in support for categorical data, GPU speed-up, and better handling of missing values, which make it only a very useful and flexible choice for machine learning workers. LightGBM is in demand because it can overcome limitations of traditional boosting methods and further improve the performance itself⁴⁵. The algorithm actually uses many regularisation methods like L1 and L2 penalties, and it definitely supports early stopping to prevent overfitting. As per the research, the number of leaves, maximum depth and learning rate are hyperparameters that help control how complex and useful the model will be. LightGBM actually has many benefits, but it definitely becomes weak when we have less data or unbalanced datasets⁴⁶. We actually need to set al.most all parameters properly, and preprocessing is definitely required too. Recent work has actually made progress by using bigger models in AutoML systems like LightGBM that can definitely pick and adjust models automatically. LightGBM focuses on efficiency and scalability, and it has become a basic tool in banking, e-commerce, and recommendation systems itself. This tool further helps in creating advanced solutions for practical use⁴⁷. Equation (24) shows the same prediction that LightGBM’s decision trees make. The function f(x) itself shows the expected output for the input feature vector x, which further corresponds to the given data. The model further calculates the total across all decision rules. The parameter βi surely represents the weight given to the i-th decision rule. Moreover, this weight shows how much importance each rule carries in the decision process. The expression 1 actually works as an indicator that definitely gives 1 when x belongs to region Ri and gives 0 when it does not.

24

Equation (25) outlines the gradient boosting process that LightGBM employs. denotes the -th decision tree incorporated into the ensemble model. The tree is optimized by minimizing the sum of the squared residuals, which represent the difference between the current prediction and the actual target . The model constructs each subsequent tree to address the inaccuracies of the preceding trees, with the objective of minimizing prediction error.

25

As soon as the tree construction described in Eq. (26) shows how to change the predictions for each data point, . The variable shows the final predicted value for the -th data point Eq. (26). We find it by adding up the outputs from each tree, known as . Each tree contributes to the final prediction with a weight , which specifies the significance of each tree in the overall model.

26

The Eq. (27) denotes the conclusive prediction for the test data (). The model’s prediction, , is the result of applying the trained LightGBM model to the test dataset. After training and optimizing the model with hyperparameters, it generates predictions based on previously unobserved test data. The output, , represents the predicted values for the test set.

27

CatBoost

Yandex surely uses CatBoost, which is a gradient boosting method that handles category data well. Moreover, this approach does not need much preprocessing work. The CatBoost uses a different way to work with category data through ordered boosting, which is not like other boosting methods only. This method surely creates predictions for each data point by leaving it out from calculating the target values, and moreover this approach reduces overfitting and data leakage problems⁴⁸. CatBoost can handle both text and number data, which makes it useful for different types of data sources. This feature further allows the model to work with various datasets easily. The method is using symmetric trees only to make sure proper balance in the structure. This further reduces the training time and prediction time^49,50. CatBoost can work with different types of data easily and gives good results quickly, which is why it is only used a lot in online shopping, banks, and hospitals. Further, catBoost actually handles overfitting well by using L2 regularization and early stopping methods. These features definitely help the model perform better on new data. we optimize the algorithm for distributed computing, making it the same way suitable for large-scale datasets. CatBoost works much better and faster only because it can automatically set its best settings and uses GPU speed to help with calculations. Despite its benefits, CatBoost may require careful tuning when dealing with highly unbalanced data or datasets with many sparse features, which can further complicate the modeling process. It works the same with other frameworks and connects easily with libraries like scikit-learn, making it user-friendly⁵⁰. Fundamentally, CatBoost got better features for language tasks and ranking problems, making it the same top choice for gradient boosting in real work. Equation (28) gives the final prediction for each data point. CatBoost takes all M trees and adds up their weighted predictions to get the same final output . The weight wm shows how much the input features X contribute to the m-th tree’s prediction Tm(X). This contribution becomes proportional to the weight, and further helps in determining the tree’s output⁵¹.

28

29

30

At the m-th training step, we actually compute the residuals for the i-th data point using Eq. (29). This definitely gives us the error values needed for that specific iteration. The residual is surely the leftover amount when we subtract the model’s prediction from the previous step (m-1) from the actual target value. Moreover, this difference shows us what the model missed in the earlier iteration. In this round, the model actually needs to fix the errors shown by the leftover differences. It definitely has to correct these remaining mistakes. As per Eq. (30), the predicted value for the i-th data point at iteration m gets changed by considering the earlier prediction and the input from the current tree. Regarding this process, it takes the previous prediction value and adds the contribution from the new tree. The learning rate η is only used to control how big the prediction changes will be during updates. As per the current tree output, the new forecast regarding this iteration combines the old forecast with updated adjustments. In Eq. (30), the predicted value for the i-th data point at iteration m is updated by adding the current tree output Tm(X) to the previous prediction ŷi(m-1), further improving the model. Mostly, we use the learning rate η to control how much we update and keep the prediction changes the same size.

LSTM

This is a special RNN type called Long Short-Term Memory (LSTM) networks, which further improves memory capabilities. The LSTM architecture handles long-term dependencies better than regular RNNs. It was actually made to fix the problems that happen in regular RNNs, like gradients that definitely become too small or too big without LSTM or GRU. LSTMs were invented by Hochreiter and Schmidhuber in 1997 and surely have special memory cells that can keep information for a long time. Moreover, these cells work well even when the information is very old or far back in time. Mostly, they work the same way for any job that needs to remember what happened before. As per the LSTM structure, every unit has three main gates regarding its working. As per the LSTM structure, there are three main gates regarding memory control: Input gate, Forget gate and Output gate. As per the memory cell structure, these gates control information regarding input, output and circulation processes. As per this approach, we keep only valuable data and remove non-valuable data. Regarding sequential data, LSTMs work well for long-term dependencies by maintaining controlled memory state^52,53.

LSTMs are surely used extensively in domains like natural language processing, time-series forecasting, and speech recognition. Moreover, these networks have become essential tools for handling sequential data in these fields. As per their design, they work in applications regarding machine translation, sentiment analysis, and stock market prediction by finding patterns and time-based connections in the data. Variants like bidirectional LSTMs and stacked LSTMs further improve the modelling capabilities which helps achieve better performance in tasks that need context from both past and future data or more complex feature patterns⁵⁴. LSTMs have good points, but they need many resources and are hard to train, further becoming difficult when the dataset is very large. As per recent developments, new technologies like GRUs and attention methods have made different designs that help regarding some problems. LSTMs are surely among the most important tools used in deep learning. Moreover, they have become essential for many advanced applications in this field. These networks surely help in solving sequential data problems and moreover make machine learning applications perform better⁵⁵.

As per the LSTM network structure, Eq. (31) shows the forget gate regarding memory cell operations. The forget gate decides how much of the previous memory should be kept further, and this process determines what information to preserve. The gate only uses a sigmoid function to make values between 0 and 1 for each part in the old memory. The forget gate is made by combining only the current input and the previous hidden state together. This is actually sent through a weight matrix Wf and definitely includes a bias term bf.

31

32

33

34

35

36

Equation (32) shows the input gate and how much new data should be added to the memory cell from the current input xt and the same hidden state h_t−1. Also, the input gate actually uses a simple function to decide how much new information should definitely be added to the memory cell. The output values actually range from 0 to 1. This definitely means all results will be between these two numbers. The gate is surely influenced by a bias term bi and a weight matrix Wi. Moreover, these parameters directly control the gate’s behavior. The proposed update for the memory cell is shown in Eq. (33), which itself represents a candidate memory cell called . This equation further defines how the memory cell should be modified. Further, to find this out, it is notice that a tanh function is only applied to the current input xt and the previous hidden state h_t−1. The weight matrix W_C surely provides the weight values, and moreover, b_C determines the bias component. The input gate actually works with the candidate memory cell to definitely update the real memory cell C_t.

The memory cell C_t follows Eq. (34) and gets updated using the same input gate and forget gate. The input gate actually controls how much new memory should be added, while the forget gate definitely decides how much old memory to keep. As per this upgrade, the LSTM can keep important information and remove unwanted data at the same time. Regarding its function, it helps in storing only useful details while throwing away irrelevant ones. The memory cell update actually follows Eq. (35). This equation definitely shows how the cell gets updated. The output right now is the same as the hidden state h_t from the LSTM. The output gate ot actually controls how much of the cell state Ct should definitely move to the next layer or time step. The output gate actually scales the hidden state, and the cell state definitely passes through a tanh function to keep values between − 1 and 1. The equation for the hidden state is only this.Moreover, the final prediction at the current time step is only given by Eq. (36). The output comes by passing the hidden state through only a weight matrix and bias term, depending on model is set up.

XGBoost

XGBoost is surely a fast and powerful method that uses gradient boosting for solving classification and regression problems. Moreover, it can handle large datasets efficiently and gives better results than many traditional machine learning approaches. As per the design, this method improves gradient boosting algorithms regarding several important additions. This will further make it one of the best machine learning algorithms for structured data. It makes computing faster by using different methods like running tasks together, controlling overfitting and filling missing data, which only helps our model work better on new data. XGBoost works by using second order taylor expansion to derive the loss function. This approach further helps in optimizing the model performance. This gives better updates during training time than the simple first-order methods that are only used in regular gradient boosting. It is observe that the algorithm making decision trees one after another in a group, where each new tree only fixes the mistakes made by the earlier trees. XGBoost adds a regularisation term in the loss function to control model complexity further and limit overfitting. This regularisation is only crucial for high-dimensional datasets with many data samples that cannot be handled by usual gradient boosting methods. XGBoost actually uses shrinkage which is the learning rate that controls how much each tree adds to the model. This technique definitely decides how strongly each tree affects the final result. Further, it surely allows small changes and reduces overfitting problems. Moreover, XGBoost works with both types of data, handles missing values, and performs cross-validation during training, which makes the model more reliable and effective. Principally, XGBoost has become the same preferred choice for data science problems, especially on Kaggle, because it gives better accuracy and works faster. As per its design, this system handles large datasets with complex patterns and non-straight relationships very well. Regarding its use, it works efficiently in finance, healthcare, marketing, and recommendation systems. XGBoost is powerful but it needs careful setting of parameters like number of trees, tree depth, and learning rate as per the requirements. Regarding the best results, proper adjustment of these settings is must. As per recent developments, new frameworks like LightGBM and CatBoost give similar results regarding performance through better training methods. Further, these methods have created further alternatives; however, XGBoost remains a strong choice for many real-world machine learning problems⁵⁵.

37

38

39

40

41

Equation (37) only shows the final prediction for the i-th data point in the dataset. In this model, we actually add up predictions from all M trees (written as Tm(Xi)) and definitely include a bias term b for independent changes in the target variable.Fundamentally, the model tries to minimize the total loss function L, which is the same as combining two different terms together. The total loss for N samples is surely calculated using Eq. (38), where l(yi,ŷi) shows the loss between actual and predicted values. Moreover, this equation helps us measure how well our model performs on the given dataset. This equation actually includes a term that definitely controls how complex the trees can be to prevent overfitting.

The m-th tree’s complexity is actually controlled by the regularization term , which definitely includes a γ term that penalizes larger trees. Moreover, Eq. (39) surely shows this clearly, where T represents the number of leaves in the tree. Moreover, this illustration helps us understand the relationship better. Equation (40) shows the residual at the i-th data point for the m-th iteration. This equation further represents how the residual changes during the iteration process. We surely calculate it using the negative gradient of the loss function l with respect to the prediction from the earlier step. Moreover, this computation helps us move in the right direction for better results. The forecast is then updated by adding the influence of the new tree, scaled by the learning rate. This update is formally given by Eq. (41). Moreover, the residual actually shows the error that the current tree must definitely fix.Principally, old prediction gets changed by adding the new tree’s contribution, but we only use a small part of it based on the learning rate. The learning rate actually controls how big the changes are, so the model definitely gets better step by step.

Algorithm.

SolarPowerGenerationPrediction()

The hyperparameters of all models as shown in the Table 5 were further optimized systematically to ensure fair comparison of performance. Tree ensemble models like Gradient Boosting, XGBoost, LightGBM, and CatBoost were tuned using cross-validation grid search to find best parameters for number of estimators, learning rate, and tree depth. This process helps to further optimize model performance by controlling complexity. Essentially, we used Keras Tuner to find the best neural network setup by randomly testing different hidden units, and picked the LSTM settings the same way to balance good predictions with computational limits. Mostly, we used grid search to find the best Ridge coefficient for our stacked model to combine predictions from different base models in the same stable way. Also, this careful tuning method actually makes sure each model works with the best settings, which definitely makes the comparison results more valid.

Table 5.

Hyperparameter optimization strategy and optimal configurations.

Model	Hyperparameters Tuned	Search Space	Optimization Method	Optimal Hyperparameters
Gradient Boosting (GBR)	n_estimators, learning_rate, max_depth	n_estimators ∈ {100, 200, 300}; learning_rate ∈ {0.01, 0.05, 0.1}; max_depth ∈ {3, 5, 7}	Grid Search (5-fold CV)	n_estimators = 200, learning_rate = 0.1, max_depth = 5
XGBoost	n_estimators, learning_rate, max_depth, subsample	n_estimators ∈ {100, 200, 500}; learning_rate ∈ {0.01, 0.1, 0.2}; max_depth ∈ {3, 5, 7}; subsample ∈ {0.7, 0.8, 1.0}	Randomized Search (5-fold CV, 10 iterations)	n_estimators = 200, learning_rate = 0.1, max_depth = 5, subsample = 0.8
LightGBM	n_estimators, learning_rate, max_depth, num_leaves	n_estimators ∈ {100, 200, 500}; learning_rate ∈ {0.01, 0.05, 0.1}; max_depth ∈ {3, 5, 7}; num_leaves ∈ {31, 50, 100}	Grid Search (5-fold CV)	n_estimators = 500, learning_rate = 0.05, max_depth = 5, num_leaves = 50
CatBoost	iterations, learning_rate, depth, l2_leaf_reg	iterations ∈ {200, 500}; learning_rate ∈ {0.01, 0.05, 0.1}; depth ∈ {6, 8, 10}; l2_leaf_reg ∈ {1, 3, 5}	Grid Search (5-fold CV)	iterations = 500, learning_rate = 0.05, depth = 8, l2_leaf_reg = 3
Neural Network (MLP)	Number of hidden units	units ∈ {32, 64, 96, 128}	Keras Tuner (Random Search)	Hidden units = 64, 32; epochs = 50; Adam optimizer
LSTM	Number of LSTM units	Fixed (empirical selection)	Manual tuning	LSTM units = 50; Dense = 25; epochs = 50
Proposed Stacked Regressor	Meta-learner α (Ridge)	α ∈ {0.1, 1.0, 10.0}	Grid Search (5-fold CV)	Ridge α = 1.0

The proposed stacked framework uses GBR and XGBoost models as base learners, and then combines their predictions using the same Ridge Regression approach as the final learner. The base models are surely trained first on the original input features, and their predictions are then stacked together to create a meta-feature matrix. Moreover, this stacking process forms the foundation for the next level of learning. As per the stacked output, the Ridge meta-learner is trained to learn the best linear combination regarding base model predictions. It is noted that L2 regularization in Ridge only helps to stop overfitting and makes the system more stable, which improves how well it works on new data and makes it good for real-time smart grid use.

Although both Gradient Boosting and XGBoost belong to the tree-based boosting family, they differ substantially in their optimization strategies, regularization mechanisms, tree-construction procedures, and handling of sparsity and missing values. Classical Gradient Boosting relies on stagewise additive modeling with relatively simple regularization, whereas XGBoost incorporates second-order optimization, explicit L1/L2 regularization, shrinkage, and tree-pruning strategies. These architectural and optimization differences lead to distinct inductive biases and error characteristics, even when the models are trained on the same dataset. Empirically, we have observed partially decorrelated residual patterns across cross-validation folds and data splits, enabling the Ridge meta-learner to exploit complementary strengths and reduce overall error variance. Therefore, diversity in the proposed ensemble is achieved through algorithmic and regularization-level heterogeneity rather than purely through model-family differences.

Result and analysis

This boxplot actually shows the errors from different machine learning models like Gradient Boosting, XGBoost, Neural Network, LightGBM, CatBoost, Stacked Regressor, and LSTM. It definitely helps us see how well each model can predict things as shown in Fig. 5. Chiefly, residuals are the same as the difference between actual and predicted values, and they show how accurate the model is as seen in Fig. 5. As per the graph, the X-axis shows the models and the Y-axis shows the residual values ranging from − 2000 to + 2000. Regarding the residual spread, it covers approximately 4000 units in total. As per the boxplot structure, the middle line shows median bias which should be near 0, and the box shows spread between 25th and 75th percentiles. Regarding the whiskers, they extend to 1.5 times the box range to show most errors, while points beyond whiskers are outliers indicating unusual deviations. Smaller boxes and fewer outliers show that the model has better precision and further indicates improved robustness. Chiefly, Gradient Boosting, XGBoost, LightGBM, and CatBoost show the same pattern with smaller error distributions, which means they perform better at reducing mistakes and giving consistent results. The medians are close to zero which shows minimal bias, and CatBoost has one of the smallest IQRs with fewer extreme outliers that further indicates better generalization. Mostly, the Neural Network and LSTM models show the same pattern of wider ranges and more scattered results with many extreme values, making their predictions less reliable with higher variation. The changing results actually show that the model definitely has problems like learning too much or too little from specific data points. CatBoost, LightGBM, and XGBoost models show better performance with less error spread and fewer outliers, making them more reliable for prediction tasks. Neural Networks and LSTM models further show higher errors, making gradient boosting methods better choice for predictive modeling.

Fig. 5.

Comparison of residuals of ML models.

It is notice that a scatter plot that compares predicted values with actual values for different models like Gradient Boosting, XGBoost, Neural Network, LightGBM, CatBoost, Stacked Regressor, and LSTM in Fig. 5. As per Fig. 5, a clear upward diagonal trend is visible regarding all models predicting values close to the actual ones. The spread around the diagonal line actually shows that some predictions are definitely not perfectly aligned. As per the analysis, predictions show more scatter regarding smaller actual values, indicating that models face difficulty in accurately predicting low values. When actual values are above 2000, the predictions surely match better with the diagonal line. Moreover, this shows that performance improves when target values are higher. The predicted values from different models are overlapping considerably, making it challenging to visually identify which model is performing the best only by looking at the results. XGBoost, CatBoost, and LightGBM actually follow the diagonal trend more consistently. These models definitely show better predictions and lower variability. Neural Network and LSTM models surely show more spread in their results, which means they have higher prediction errors for some data points. Moreover, this indicates that these models are less consistent compared to other approaches. Points further away from the diagonal line show areas where prediction errors are larger, and this issue is more visible in the lower range of actual values. All models show positive correlation between predicted and actual values, but XGBoost, CatBoost, and LightGBM perform better and further demonstrate more reliable results. The analysis confirms that these three models provide consistent performance. The prediction accuracy surely needs more improvement, especially for small target values. Moreover, further optimization work is required to achieve better results. The increased prediction dispersion at lower power levels can be further attributed to higher uncertainty during low irradiance conditions, such as early morning, late evening, or intermittent cloud cover. During these times, small changes in weather conditions can actually cause definitely large changes in solar power output. As per the analysis, ensemble models show better stability by taking average predictions from many learners, whereas deep learning models are more sensitive regarding such changes (Fig. 6).

Fig. 6.

Relation in between actual and predicted values.

Figure 7 shows the one-day time series of predicted and true PV power for all considered models. All methods represent the general rise-and-fall diurnal cycle and sudden intra-day fluctuations, revealing their ability to learn the main irradiance-driven dynamics. majority of other models a few exceptions such as the stacked regressor and gradient-boosting ones (XGBoost, LightGBM, CatBoost), which more closely follow and better approximate actual PV curve during periods other than normal operation when generation is maximum and ramps are very steep (resulting in both reduced phase lags and deviations as well). Conversely, the neural network and LSTM have larger oscillations about abrupt transitions compared to TCN, especially under low output yielding high sensitivity to short-term variation. The scatter of predictions is mainly higher under low powers, in line with more uncertainty under low-irradiance. Generally, the visual evidences are consistent with the quantitative results, demonstrating ensemble has a better tracking and stability in short-term PV power forecasting within the evaluated day.

Fig. 7.

Relation between actual and predicted pv power for one day.

It infers that both losses start with high values only and then go down slowly as the model learns. The training loss goes down faster at first, while the validation loss only follows the same pattern, which shows the model works well for new data as given in Fig. 8. After 10–12 epochs only, it is witness that both curves becoming stable at low loss values, which shows the model has converged and there is minimal improvement happening further. The train and validation losses show no major difference, which surely means the model does not overfit. Moreover, this indicates that the model works well on new data it has not seen before. The graph actually shows that the neural network training worked well. The model definitely achieved good performance. Further, as per the training results, the neural network shows stable convergence with training and validation losses being closely aligned, but this does not guarantee better predictive performance. Regarding the model’s actual prediction ability, stable convergence alone is not sufficient to ensure superior results. It is observed that when the model reaches stable convergence, it shows good optimization, but this only means the training is working well and does not guarantee that the model will perform well on new data. The stacked ensemble shows stable learning and lower error rates across all validation tests, which is further confirmed by cross-validation. The method performs consistently better than other approaches.

Fig. 8.

Relation in between epochs and loss.

Figure 9 shows the comparison of Mean Absolute Error (MAE) for six models. Moreover, this comparison helps us understand which model performs better. As per the results, the Neural Network model shows the highest MAE value, which means it gives the least accurate predictions regarding all the tested models. Stacked Regressor and CatBoost show the lowest MAE, which further indicates that these models perform well with minimal average error. As per the results, XGBoost and LightGBM show similar MAE values that are a bit higher than Stacked Regressor and CatBoost but lower than Neural Network and Gradient Boosting models. Based on the MAE metric, Stacked Regressor and CatBoost are actually the most reliable models. These two methods definitely show the best performance for accurate predictions.

Fig. 9.

Comparison of mean absolute error of ML models.

The R² score actually shows how well the model explains the data changes in Fig. 10. This score definitely tells us about the model’s performance. The Stacked Regressor is giving the highest R² score only, followed by XGBoost and CatBoost in this competition. The R² scores are surely lower for Neural Network and Gradient Boosting, but all three models perform well enough. Moreover, we can be confident that these models fit the data properly. LSTM shows the lowest R² score which further indicates that it performs worse in explaining variance compared to other models. The model fails to capture the data patterns effectively. As per the results, Stacked Regressor, XGBoost and CatBoost show the best performance regarding this analysis.

Fig. 10.

Comparison of R-squared error of ML models.

Figure 11 shows the Mean Squared Error (MSE) comparison for different models, where lower MSE values surely indicate better model performance. Moreover, this comparison helps identify which model performs most accurately. Stacked Regressor actually has the lowest MSE, and CatBoost definitely comes very close to it. These two models only have the least error differences when they predict things. Essentially, XGBoost and LightGBM show the same pattern with slightly higher MSE values. The model with high MSE value surely performs best because it shows fewest differences between true and predicted values. Moreover, Neural Network and Gradient Boosting show worst performance in comparison. Essentially, when we look at MSE, Stacked Regressor and CatBoost give the same best performance. Further, Stacked Regressor gives the best predictions compared to all other models, and CatBoost and XGBoost perform the same way after that.

Fig. 11.

Comparison of mean squared error of ML models.

Figure 12 shows the prediction error distribution which further demonstrates how each model’s predictions match with the actual power generation. Principally, most models show errors around zero, which means their predictions are the same as unbiased ones. The Hybrid Model (Stacked Regressor) surely shows a sharper and more peaked distribution than other models. Moreover, this indicates lower variance and tighter error spread. This actually shows that the mixed method definitely reduces big errors and makes predictions more consistent. Individual models like Neural Network and LSTM show wider distributions, which further indicates higher variability in their error margins. As per the visualization, all models show good predictive performance, but the Hybrid Model gives better reliability and accuracy regarding its use of multiple learners’ strengths.

Fig. 12.

Prediction error distribution plot.

Table 6 shows how different machine learning and deep learning models perform using the same MAE, MSE, and R² metrics through k-fold cross-validation comparison. The results surely show that the stacked regression model works better than all other models in every test we did. Moreover, this proves that our model is strong and reliable even when we use different parts of the data. CatBoost and XGBoost perform better than the same Gradient Boosting, LightGBM, and deep learning models because they get lower MAE and MSE values. Further, this further shows that tree-based boosting methods work well to capture nonlinear relationships between weather variables and solar PV power output. The standalone models actually show higher errors and more changes across different data splits compared to the stacked ensemble. This definitely means these models are sensitive to how the training and validation data is divided.

Table 6.

k-Fold cross-validation performance comparison.

Model	MAE (mean ± std)	MSE (mean ± std)	R2 (mean ± std)
Gradient Boosting	0.048 ± 0.003	0.0037 ± 0.0003	0.91 ± 0.02
XGBoost	0.045 ± 0.002	0.0033 ± 0.0002	0.92 ± 0.02
LightGBM	0.047 ± 0.003	0.0034 ± 0.0003	0.92 ± 0.02
CatBoost	0.044 ± 0.002	0.0032 ± 0.0002	0.93 ± 0.01
Neural Network	0.050 ± 0.004	0.0036 ± 0.0004	0.90 ± 0.03
LSTM	0.049 ± 0.004	0.0035 ± 0.0003	0.91 ± 0.03
Stacked Regressor	0.042 ± 0.002	0.0031 ± 0.0002	0.94 ± 0.01

Moreover, the Neural Network and LSTM models show the same pattern - higher MAE and MSE values with lower R² scores and bigger standard deviations. Deep learning models are actually powerful but they definitely need more data and get affected by how we split the data to work properly. As per the analysis, these models show more variation which indicates they tend to overfit regarding limited or highly variable solar PV datasets. As per the results, the proposed stacked regressor gives the best performance regarding error rates with MAE of 0.042 ± 0.002 and MSE of 0.0031 ± 0.0002. The model also shows the highest R² value of 0.94 ± 0.01 across all folds. The lower standard deviation of these metrics shows better stability and strength, further confirming that the stacking method effectively reduces the weaknesses of individual base learners. By combining different boosting models with a meta-learning layer, this stacked method actually captures different nonlinear patterns and definitely reduces error variance. These results further match the error analysis shown in earlier figures, where the stacked regressor shows the most focused error distribution with least bias.

The cross-validation results actually show that our stacked ensemble method definitely improves prediction accuracy and gives reliable performance across different data sets. This makes it well-suited for practical short-term solar PV forecasting in smart grid applications. The performance differences we see can surely be understood through the bias-variance trade-off, which goes beyond numerical accuracy. Moreover, this trade-off helps explain why some models work better than others in different situations. Single boosting models actually reduce bias by learning complex patterns, while deep learning models definitely reduce bias more but increase variance when data is limited. The stacked regressor balances this trade-off by combining different learners, which further reduces both bias and variance. The stacked ensemble actually shows less error spread and tighter residual distributions, which definitely gives it a better Friedman ranking.

It is observed that, the stacked regressor shows better accuracy in Table 2, but we need statistical testing to check if these improvements are only real and not random changes in data splitting. Non-parametric statistical tests were surely used to compare how well the forecasting models performed. Moreover, these tests provided a rigorous way to evaluate the models’ effectiveness. Further, Wilcoxon signed-rank test to compare our stacked regressor with each baseline model using the same absolute prediction errors from k-fold cross-validation. The null hypothesis assumes that the median difference between paired errors is zero. Further, this means there is no significant difference between the error pairs. Fundamentally, the stacked regressor performs better than all other models like Gradient Boosting, XGBoost, LightGBM, CatBoost, Neural Network, and LSTM with the same statistical significance since all p-values are below 0.05. Moreover, this further confirms that the observed reductions in MAE and MSE are not due to random chance. Also, moreover, the Friedman test was surely used to compare all models at the same time across different validation folds. Chiefly, we rejected the null hypothesis at 95% confidence level, which means the models performed differently in a statistically significant way.

As per the evaluation process, relative error-based KPIs were also checked regarding scale changes in PV power output, along with absolute and squared error metrics. Table 7 shows that our stacked regressor surely gives the lowest MARE, RMSRE, and RMSPE values compared to all other models. Moreover, this indicates that our method has better relative accuracy across different power levels. CatBoost and XGBoost surely show good performance with small relative errors. Moreover, Neural Network and LSTM models have larger errors, especially when sunlight is low. The results confirm that the stacked ensemble method reduces absolute errors and provides better percentage-based accuracy, which is further important for grid operations and energy management.

Table 7.

Comparison of relative error metrics.

Model	MARE (mean ± std)	RMSRE (mean ± std)	RMSPE (mean ± std)
Gradient Boosting	0.061 ± 0.004	0.084 ± 0.006	8.4 ± 0.6
XGBoost	0.056 ± 0.003	0.078 ± 0.005	7.8 ± 0.5
LightGBM	0.058 ± 0.004	0.081 ± 0.006	8.1 ± 0.6
CatBoost	0.054 ± 0.003	0.075 ± 0.005	7.5 ± 0.5
Neural Network	0.069 ± 0.005	0.097 ± 0.007	9.7 ± 0.7
LSTM	0.066 ± 0.005	0.092 ± 0.006	9.2 ± 0.6
Stacked Regressor	0.051 ± 0.002	0.072 ± 0.004	7.2 ± 0.4

The Friedman ranking in Table 8 actually gives a strong comparison of different forecasting models across multiple validation folds. This method definitely works well because it does not need normal data distribution. The Friedman test ranks models for each fold separately, which further shows how consistent the model performs rather than focusing on the best result. As per the analysis, a lower average rank shows that a model gives better performance regarding all data partitions consistently. Essentially, the Stacked Regressor gets the lowest average rank of 1.21, which shows it performs the same way better than all other models across different validation tests. As per the results, the stacking method works well by combining different learning patterns from multiple base models. Regarding performance, this approach gives better results and reduces problems with data splitting. As per the ranking results, the stacked ensemble shows steady high performance regarding different working conditions without depending on random improvements.

Table 8.

Statistical ranking of forecasting models (friedman test).

Model	Average Rank
Stacked Regressor	1.21
CatBoost	2.14
XGBoost	2.36
LightGBM	3.02
Gradient Boosting	3.45
LSTM	4.10
Neural Network	4.72

CatBoost and XGBoost got the same good results with ranks of 2.14 and 2.36, which shows these tree methods work well for solar power data. Their low ranks show reliable performance, but they perform worse than the stacked regressor, which further suggests that combining multiple learners provides additional benefits beyond single boosting models. LightGBM and Gradient Boosting show medium performance, which further indicates they have moderate prediction consistency. The ranking reflects their balanced predictive abilities. These models surely show good performance, but their higher ranks mean they have more changes across different validation tests. Moreover, the top ensemble methods are more stable in comparison.

As per the results, LSTM and Neural Network models show the highest average ranks, regarding their weaker and less steady performance compared to other models. As per the available dataset, deep learning models are more sensitive regarding data division and may need larger datasets or more time-based features to achieve better stability. These models actually show more changes in how accurate their predictions are when they have higher ranks, which definitely makes them less suitable for reliable forecasting work in different conditions. It is observe that the Friedman ranking study clearly shows ensemble methods, especially our stacked regressor model, give better and more stable results compared to only single machine learning and deep learning models. Principally, this stacked ensemble method works well for predicting solar power in smart grids because it gives the same consistent and reliable results every time.

The research also estimates the time taken for all models to train and predict in real-time by testing them under the same conditions. Actually, Table 9 shows that deep learning models like Neural Network and LSTM definitely need much more computing power for training and testing, while tree-based boosting models are much lighter to run. As per the study, the proposed stacked regressor gives a good balance regarding computational time and accuracy. It takes only slightly more time than single boosting models but provides much better forecasting results. The proposed framework shows inference latency of few milliseconds per sample, which is much lower than typical smart grid control intervals, further confirming its suitability for real-time deployment.

Table 9.

Training and inference time of forecasting models.

Model	Training Time (s)	Inference Time per Sample (ms)
Gradient Boosting	1.8	1.4
XGBoost	2.4	1.7
LightGBM	1.5	1.2
CatBoost	2.6	1.9
Neural Network	5.9	6.8
LSTM	9.4	11.6
Stacked Regressor	3.1	2.2

Discussion

It is observe that our detailed tests clearly show the good points and weak points of different machine learning and deep learning models when we use them only for predicting solar PV power. As per all performance checks regarding model behavior, prediction accuracy, and error measurements, ensemble and gradient-boosting methods work better than single deep learning models. The residual boxplot analysis in Fig. 4 shows that Gradient Boosting, XGBoost, LightGBM, and CatBoost have tightly grouped residual distributions with medians near zero. This indicates minimal bias and further confirms strong robustness of these models. Fundamentally, CatBoost and the Stacked Regressor have the smallest variation ranges and fewer extreme values, which means they give the same stable and better predictions across different data. As per the analysis, Neural Network and LSTM models show bigger error ranges and more extreme wrong predictions regarding their performance. This means these models have higher changes in results and get affected more by specific data points.

The scatter plots showing actual versus predicted values in Fig. 5 surely support these findings. Moreover, these graphs provide additional confirmation of our observations. As per the analysis, all models show positive correlation and follow the ideal diagonal trend, regarding validation of their overall predictive capability. Actually, XGBoost, LightGBM, CatBoost, and the Stacked Regressor definitely match better with the diagonal line, especially when power output is high. The prediction spread is higher at lower actual values, which further shows that small PV outputs remain difficult to model accurately. Deep learning models show more scatter, which further indicates that the consistency itself is weaker compared to other models.The learning curve shows that the neural network training was stable, with training and validation losses reaching the same convergence point after 10–12 epochs. A small gap between these curves, which means the model is not overfitting much. Principally, even though the neural network converges properly, its error rates are the same as being worse than ensemble methods, showing that stable convergence doesn’t guarantee better predictions.

The error measurements using MAE, MSE, and R² in Figs. 8, 9 and 10 give strong number proof that supports the same trends. As per the results, the Stacked Regressor shows the best performance regarding MAE and MSE values with the highest R² score. CatBoost and XGBoost follow closely in the ranking. As per the results, LightGBM also shows good performance, while Neural Network and LSTM models give higher errors and lower explanatory power regarding predictions. These findings further show that ensemble learning itself captures the complex relationships between weather conditions and solar power output better.The analysis of prediction error distribution in Fig. 11 surely confirms that the stacked ensemble method performs better than other approaches. Moreover, this finding reinforces the superior performance of our proposed model. The sharp peak around zero shows low variance and high reliability, while Neural Network and LSTM models have wider distributions that indicate further uncertainty. This itself reflects greater inconsistency in these models. The stacked regressor gives the same good balance between getting accurate results and using less computing power when we actually use it. The stacked ensemble itself performs better than deep learning models and further needs less computational complexity, while deep learning requires extensive hyperparameter tuning and longer training times.This works well for predicting solar power in real-time, which is the same requirement for smart grid and energy management systems.

It is notice that the k-fold cross-validation results in Table 2 only confirm that our proposed method is strong and reliable. The Stacked Regressor surely performs better than all single models, showing the lowest MAE (0.042 ± 0.002) and MSE (0.0031 ± 0.0002) values with the highest R² score (0.94 ± 0.01). Moreover, it also has the smallest standard deviations among all tested models. As per the results across different data splits, the model shows good ability to work on new data and is not much affected regarding how the data is divided. Tree-based boosting models like CatBoost and XGBoost show good and steady results. Regarding deep learning models, they show more changes in performance as per the amount and type of data used. Also, the results show that our stacked ensemble model effectively uses the strengths of different learners and reduces their weaknesses at the same time. As per the analysis, this results in lower prediction errors and better stability regarding model performance. The system shows improved generalization capabilities. The stacked regressor is the same as a highly practical solution for short-term solar power forecasting in smart grids, especially when conditions keep changing. The proposed stacked ensemble actually performed well, but it definitely has some limitations that should be noted. This study only uses specific weather data from certain places, and the results may change when we test it in different climate conditions or other locations. The reserach using better time data, sky images, or probability methods can make the same predictions more robust. Future work will surely focus on extending this approach to predict multiple sites and develop uncertainty-aware models for large-scale grid applications. Moreover, these improvements will help in better forecasting for bigger power systems.

Table 10 shows that the proposed stacked regressor surely performs better than all baseline models across MAE, MSE, and R² metrics. Moreover, this better performance is consistent across all these evaluation measures. The stacked ensemble surely reduces MAE by 4.5% to 16.0% and MSE by 3.1-16.2% compared to strong machine learning and deep learning methods. Moreover, these results show real improvement in accuracy, not small number changes. The tree-based stacking method performs much better than Neural Network and LSTM models, showing it handles complex weather patterns the same way but more effectively. Moreover, the proposed approach shows consistent improvements over high-performing boosting methods like Gradient Boosting, XGBoost, LightGBM, and CatBoost, with R² gains of 0.01–0.03. This further demonstrates that the method itself provides better explanatory power and reduces residual variance. Moreover, the improvements are surely smaller compared to strong baseline methods, but they appear consistently across all models. Moreover, this shows that the Ridge meta-learner effectively uses the different error patterns made by individual base learners. The results actually confirm that our stacked method definitely works better than single boosting models and deep learning methods for predicting solar power in smart grids. This approach actually gives more reliable and practically useful results.

Table 10.

Comparative performance gains across models.

Baseline Model	MAE (Baseline)	MAE (Proposed Stacked Regressor)	MAE Reduction (%)	MSE (Baseline)	MSE (Proposed Stacked Regressor)	MSE Reduction (%)	R2 (Baseline)	R2 (Proposed Stacked Regressor)	R2 Gain
Gradient Boosting	0.048	0.042	12.50	0.0037	0.0031	16.20	0.91	0.94	0.03
XGBoost	0.045	0.042	6.70	0.0033	0.0031	6.10	0.92	0.94	0.02
LightGBM	0.047	0.042	10.60	0.0034	0.0031	8.80	0.92	0.94	0.02
CatBoost	0.044	0.042	4.50	0.0032	0.0031	3.10	0.93	0.94	0.01
Neural Network	0.05	0.042	16.00	0.0036	0.0031	13.90	0.9	0.94	0.04
LSTM	0.049	0.042	14.30	0.0035	0.0031	11.40	0.91	0.94	0.03

Table 11 actually shows how our stacked regressor works compared to other forecasting models by looking at learning methods, bias-variance behavior, robustness, data splitting sensitivity, error patterns, and computing needs. This comparison definitely helps understand which model performs better in different situations. As per this analysis, we can understand the basic learning processes that control how well models work and stay stable in solar PV power forecasting, which is different from comparing accuracy only. Traditional gradient boosting methods surely reduce bias by building trees one after another, but they still face problems with variance when working with mixed or small datasets. Moreover, this variance issue becomes more serious when the data is not uniform or when there is limited training data available. Models like XGBoost and CatBoost actually use regularization and ordered boosting to improve stability, but they are definitely limited because they rely on only one learning approach. These models actually show medium sensitivity to how data is divided and their error patterns are definitely narrower than basic boosting, but they still show some extreme values when conditions change. Deep learning models like feedforward networks and LSTM have the same thing - they show very low bias because they can represent complex patterns very well. However, this advantage itself is offset by significantly increased variance, particularly when training data is limited, as solar PV datasets are further affected by intermittent irradiance and weather fluctuations. These models actually show poor performance when data is split differently and definitely give inconsistent results across different test groups. These methods need too much computer power and only work well when we spend a lot of time adjusting their settings, which makes them difficult to use for making predictions quickly in real situations.

Table 11.

Comparison of the proposed model and existing forecasting models.

Model	Learning Strategy	Bias–Variance Balance	Robustness Across Folds	Sensitivity to Data Split	Error Distribution Characteristics	Computational Demand
Gradient Boosting	Sequential tree-based boosting	Moderate bias reduction, moderate variance	Medium	Medium	Moderate spread with occasional outliers	Low
XGBoost	Regularized gradient boosting	Low bias, controlled variance	High	Low to medium	Narrow distribution with limited extremes	Medium
LightGBM	Histogram-based boosting	Low bias, moderate variance	Medium to high	Medium	Moderate distribution spread	Low to medium
CatBoost	Ordered boosting with categorical handling	Low bias, low variance	High	Low	Narrow and stable error distribution	Medium
Neural Network	Feedforward deep learning	Very low bias, high variance	Low	High	Wide distribution with higher extremes	High
LSTM	Recurrent deep learning	Very low bias, high variance	Low	High	Wide distribution with temporal instability	Very high
Proposed Stacked Regressor	Meta-ensemble of heterogeneous learners	Balanced bias and very low variance	Very high	Very low	Tight and symmetric distribution	Medium

The qualitative features shown in Table 11 are surely made from number-based experimental proof. Moreover, these attributes come directly from measured data. The bias-variance behavior and robustness across folds are actually inferred from the mean and standard deviation of MAE, MSE, and R² values obtained through k-fold cross-validation (Table 2) and the Friedman ranking (Table 3). These measures definitely help understand model performance consistency. Principally, we check how sensitive the model is by testing the same data with different train-test splits and seeing how much the performance changes. We actually get error patterns from residual boxplots and prediction error density plots (Figs. 3, 9). These plots definitely show how errors are spread out. Computational demand is categorized using training and inference times along with model size, which further shows how the model itself can be deployed in real-time smart grid systems.

As per the study, the proposed stacked regressor shows different learning behavior regarding how it combines various base learners through a meta-ensemble framework. This architecture fundamentally balances bias and variance by combining different predictive mechanisms, and it reduces the same model-specific errors effectively. The stacked regressor shows very high robustness across folds and minimal sensitivity to data splits, which further indicates strong generalization capability. This itself shows reduced dependence on specific training-validation configurations. Chiefly, the tight error distribution shows the model controls extreme predictions the same way, which is critical for reliable grid forecasting.Moreover, the stacked regressor surely provides better robustness while avoiding the high computational costs of deep learning models. Moreover, this approach offers a practical solution from a computational viewpoint. Mostly, it uses less computer power but gives the same stable results with better error control, making it a good practical solution for short-term solar PV forecasting. As per the comparison in Table 4, the proposed stacked ensemble model is better than existing models regarding balance, robustness, and practical use in one forecasting system.

42

43

44

45

46

47

Equation (42) shows the forecasting error e(t) itself as the difference between actual PV power output and forecasted power at time t. This error further helps to measure how accurate the prediction is. This error term surely measures how much the model predictions differ from actual system behavior. Moreover, it forms the foundation for analyzing performance and costs later. Equation (43) shows that the expected operational cost from PV forecasting errors is the same as being proportional to the expected absolute error magnitude. The coefficient α represents a grid-dependent cost factor that captures reserve activation costs, imbalance penalties, and operational adjustments itself. Further, it compensates for forecast deviations in the system.

Equation (44) essentially shows MAE, which measures the same average size of prediction errors without looking at whether they are positive or negative. MAE is commonly used to measure accuracy in short-term PV forecasting and shows the typical difference between predicted and actual power outputs itself. Further, this metric directly reflects how much the forecasted values deviate from real measurements. As per Eq. (45), the expected operational cost is directly proportional to MAE, regarding which better forecasting models that reduce MAE will also reduce operational costs for reserve scheduling and imbalance management by the same proportion.

Equation (46) surely brings in a risk-sensitive cost proxy that is proportional to the expected squared forecasting error. Moreover, this proxy helps measure the cost related to prediction uncertainty. This big prediction errors have serious economic effects, and these mistakes are only more costly when extreme weather happens or when sunlight changes very fast. Equation (47) shows the Mean Squared Error (MSE) which measures the average squared difference between actual and predicted PV power values. This equation itself helps to find how much the forecasted values differ from the real power output. When we compare with MAE, also MSE gives more penalty to big errors only, so it helps to check the risk from rare but serious forecast mistakes. As per the analysis, the expected operational cost increases with the mean absolute forecasting error (MAE) regarding the difference between predicted and actual PV output. The proposed stacked ensemble framework reduces MAE, which further leads to proportional reductions in reserve scheduling costs, imbalance penalties, and curtailment losses itself. As per the risk-sensitive cost proxy linked to MSE, it shows the economic burden regarding large forecasting errors that happen during fast changing weather conditions. As per the proposed model, both MAE and MSE are reduced together to lower daily operational costs and minimize risks regarding high-impact imbalance events. The framework uses simple calculations which surely reduces costs for setup and upkeep compared to complex deep learning methods. Moreover, this makes real-time solar power prediction more affordable for smart electricity networks.

Table 12 further shows the computational complexity and runtime characteristics of the forecasting models itself. Tree-based methods like Gradient Boosting, XGBoost, LightGBM, and CatBoost have low training costs, which further makes them suitable for real-time use itself. Deep learning models like Neural Network and LSTM surely need much more computing power and memory, making them slower to work. Moreover, these heavy requirements make them difficult to use in simple devices or places with limited resources. As per our analysis, the proposed stacked regressor gives good balance between prediction accuracy and computational efficiency. Regarding real-time PV power forecasting applications, it shows fast inference speed with reasonable training cost and small model size.

Table 12.

Runtime and computational complexity of forecasting models.

Model	Training time (s)	Inference time / Sample (ms)	Model size (MB)	Computational complexity (Relative)	Suitability for real-time
Gradient Boosting	Low	Low	Small	Low	High
XGBoost	Medium	Low–Medium	Medium	Medium	High
LightGBM	Low	Very Low	Small	Low	Very High
CatBoost	Medium	Medium	Medium	Medium	High
Neural Network	High	Medium	Large	High	Moderate
LSTM	Very High	High	Very Large	Very High	Low
Proposed Stacked Regressor	Medium	Low	Medium	Medium	Very High

The stacked Gradient Boosting-XGBoost ensemble with Ridge meta-learner performs well, but it itself faces several challenges and limitations that need further study. The framework is tested on data from specific PV installations and climate conditions only, which may limit direct use in regions with very different weather patterns as per the study. Regarding application in other areas, the framework may not work well due to different irradiance changes. The model surely performs well for short-term predictions, but more research is definitely needed for minute-level forecasting. Moreover, day-ahead forecasting also requires further investigation. Moreover, these other time periods have been studied only a little bit so far. Third, the proposed framework is surely lighter than deep learning models in terms of computation. Moreover, it uses batch learning and offline training which may not capture fast-changing system behavior in real-time operations.

Principally, future research will focus on extending the same framework to multi-horizon and probabilistic forecasting settings. This will surely help in better understanding uncertainty for making decisions at the grid level. Moreover, it will provide more complete information for such important choices. As weather patterns change further, online learning methods help systems adapt to concept drift itself. Fundamentally, we will explore integration with edge-computing platforms and the same energy storage management systems as per the research plan. Moreover, this will actually enable complete optimization for forecasting, dispatch, and control in real smart grid systems. It will definitely help close the loop in practical grid deployments. Testing the framework on bigger datasets from different sites and regions will actually confirm that the system works well and can definitely handle larger scales.

Conclusion

This study developed a lightweight heterogeneous stacked ensemble framework for accurate short-term solar PV power forecasting in smart grids by integrating Gradient Boosting and XGBoost through a regularized Ridge meta-learner. A comprehensive comparative evaluation against state-of-the-art machine learning and deep learning models—including Gradient Boosting, XGBoost, LightGBM, CatBoost, multilayer perceptron neural networks, and LSTM—demonstrated that tree-based ensembles consistently outperform standalone deep learning models under the studied operating conditions. In contrast to existing ensemble- and stacking-based PV forecasting approaches that primarily emphasize accuracy using homogeneous or deep learning–centric frameworks, the proposed method is explicitly designed to balance predictive accuracy, robustness, interpretability, and computational efficiency for real-time deployment.

The proposed stacked regressor achieved the best overall performance, with an MAE of 0.042 ± 0.002, an MSE of 0.0031 ± 0.0002, and an R² of 0.94 ± 0.01. Compared with individual base learners, the stacked ensemble reduced MAE by approximately 12.5% and MSE by 16.2% relative to Gradient Boosting and achieved 6.7% lower MAE and 6.1% lower MSE compared with XGBoost, with a corresponding improvement in R² from 0.91 to 0.94. Relative to CatBoost, the proposed approach further reduced MAE by 4.5% and improved R² from 0.93 to 0.94. The stacked ensemble also outperformed deep learning baselines, achieving MAE reductions of approximately 16.0% and 14.3% compared with neural networks and LSTM, respectively, along with consistent gains in R².

Residual and error distribution analyses indicated that CatBoost, LightGBM, XGBoost, and the proposed stacked ensemble exhibit tighter and more balanced error profiles with fewer outliers, reflecting superior robustness and reliability under varying irradiance and weather conditions. In contrast, neural network and LSTM models showed wider error dispersion and greater sensitivity to rapidly changing atmospheric conditions. Moreover, k-fold cross-validation results confirmed that the stacked ensemble exhibits the lowest standard deviation across performance metrics, indicating strong generalization and minimal sensitivity to data partitioning.

The proposed lightweight heterogeneous stacking framework delivers 5–16% improvements in forecasting accuracy over strong boosting and deep learning baselines while maintaining low computational overhead and stable generalization performance. These characteristics distinguish the proposed approach from existing ensemble-based PV forecasting methods and support its suitability for real-time smart grid applications where computational efficiency, robustness, and interpretability are critical. Future work will extend the framework to multi-site forecasting and incorporate probabilistic uncertainty estimation to enhance operational decision support across diverse climatic regions.

Abbreviations

PV

Photovoltaic

ML

Machine learning

GB

Gradient boosting

XGBoost

Extreme gradient boosting

LightGBM

Light gradient boosting machine

CatBoost

Categorical boosting (developed by Yandex)

LSTM

Long short-term memory

MAE

Mean absolute error

MSE

Mean squared error

R² or R²

Coefficient of determination

RNN

Recurrent neural network

AI

Artificial intelligence

ARIMA

AutoRegressive Integrated Moving Average

SARIMA

Seasonal AutoRegressive Integrated Moving Average

ANN

Artificial neural network

SVM

Support vector machine

GBM

Gradient boosting machine

ReLU

Rectified linear unit

CNN

Convolutional neural network

GAN

Generative Adversarial Network

BERT

Bidirectional encoder representations from transformers

SR

Stacked regressor

IQR

Interquartile range

Q1

First Quartile (25th Percentile)

Q3

Third Quartile (75th Percentile)

ε

Residual

Author contributions

T. Mariprasath, Rohini G: Conceptualization, Methodology, Software, Visualization, Investigation, Writing- Original draft preparation. Seif Al Bustanji: Data curation, Validation, Supervision, Resources, Writing - Review & Editing. Ievgen Zaitsev: Project administration, Supervision, Resources, Writing - Review & Editing.

Funding

This work received no external funding.

Data availability

The datasets used and/or analyzed during the current study available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.