Potential climate predictability of renewable energy supply and demand for Texas given the ENSO hidden state

Abstract

Climate variability influences renewable electricity supply and demand and hence system reliability. Using the hidden states of the sea surface temperature of tropical Pacific Ocean that reflect El Niño–Southern Oscillation (ENSO) dynamics that is objectively identified by a nonhomogeneous hidden Markov model, we provide a first example of the potential predictability of monthly wind and solar energy and heating and cooling energy demand for 1 to 6 months ahead for Texas, United States, a region that has a high penetration of renewable electricity and is susceptible to disruption by climate-driven supply-demand imbalances. We find a statistically significant potential for oversupply or undersupply of energy and anomalous heating/cooling demand depending on the ENSO state and the calendar month. Implications for financial securitization and the potential application of forecasts are discussed.

Tropical Pacific sea surface temperatures predict monthly renewable energy supply and demand for Texas up to 6 months ahead.

INTRODUCTION

With over 37 GW of wind, fifth highest in the world if it were a country, and 16 GW of solar electric generation capacity, Texas is the leader in renewable energy in the United States (1). Texas plans to install another 36 GW of solar and 10 GW of wind capacity in the next 5 years (2). The share of solar and wind electricity is expected to pass natural gas-fueled electricity in 2023 (3). The average capacity factors for wind and solar in Texas are 44 and 25%, respectively. These are some of the highest in the United States. However, they can vary markedly spatially, seasonally, and across years (4). Battery storage or generation overcapacity and networking are the typical solutions to achieve “clean” power reliability. Texas has 2 GW of battery storage, with 8 GW planned to be installed by 2025.

Electricity demand in Texas has seen rapid increases driven by population growth, climate change, and the electrification of building heating. During the 2023 heat waves in Texas, the peak demand for electricity reached 85 GW at 2 p.m. on August 10, the 10th new record for the summer. This exceeded the total electric demand of California and New York at that time (5, 6). Solar energy met 15.5% of that demand and wind 11%. These sources, particularly solar, helped avert a breakdown of the electrical grid because the peak solar production coincided with the peak demand timing. The February 2021 cold snap led to a sharp increase in heating demand, which led to cascading failures of the electricity, gas production, transportation, and water systems and 246 deaths (7, 8). The United States experienced a severe wind drought in 2015, which was related to the persistent positive expression of the North Pacific Mode (9, 10), an atmospheric circulation pattern associated with anomalous sea surface temperature (SST) in the tropical Pacific Ocean.

Because wind and solar energy as well as heating and cooling demand are influenced by climate variations, their predictability at subseasonal-to-seasonal timescales is of interest. From energy dispatch considerations, much work has focused on short-term (hour to day) availability of solar and wind (11–14). However, from system reliability and from a financial management perspective, persistent energy droughts or surpluses the past several days or months are of greater interest because spot market prices could surge or collapse, respectively. Could one use ex ante information on global climate teleconnections to a region to provide a probabilistic prediction of the likelihood of such deficits or surpluses a month or more into the future? If yes, then the development of forecast-driven financial securitization products would be feasible. Here, we explore the question in the context of Texas, given the high penetration of wind and solar in the state, and because its electricity system, managed by the Electric Reliability Council of Texas (ERCOT) (15), is largely isolated from the rest of the United States. Hence, it is ideal for regional analysis.

Climatic teleconnections associated with ocean-atmosphere variations are well established as influencing regional hydroclimatic variability as well as energy supply and demand (16, 17). This has led to research on their use for seasonal-to-interannual renewable energy forecasting (18–20). At seasonal-to-interannual timescales, the El Niño–Southern Oscillation (ENSO) is the most well-identified signal in the global climate system (21–30). ENSO has traditionally been identified with two phases, namely, the El Niño (warm phase) and La Niña (cold phase), that are determined by the SST anomalies in eastern equatorial Pacific Ocean. The Niño 1.2 and Niño 3.4 regions are the most widely used ones for global correlative teleconnection analyses (31–35). However, additional expressions of ENSO events, notably in the Central Pacific, have been identified recently, leading to a number of new ENSO indices being proposed by averaging the SST over different equatorial regions. The global climate teleconnections to these indices are also different (31, 36).

As an alternative to the use of the typical ENSO indices, Rojo Hernández et al. (37) modeled the spatial and temporal dynamics of tropical Pacific SST using a nonhomogeneous hidden Markov model (NHMM). They objectively identify five SST hidden states. The spatial composites of the five hidden states are qualitatively similar to those of different ENSO flavors, i.e., classical La Niña, mild La Niña, neutral, Modoki or Central Pacific ENSO, and the classical El Niño pattern. Zhang et al. (31) identified the global predictability of monthly rainfall using these hidden Pacific states for 1 month to a year in the future. Here, we explore how tropical Pacific SST influences the predictability of energy supply and demand at month-to-season timescales in the context of the five hidden states that were identified. The conceptual probabilistic model is illustrated in Fig. 1. We use the historical chronology of identified hidden states of ENSO to explore how wind, solar, heating degree days (HDD), and cooling degree days (CDD) for Texas evolve in a spatially gridded sense for future months. We find that the potential predictability varies by variable, spatial location, and calendar month but is statistically significant under many SST conditions for several months into the future.

Fig. 1. Schematic diagram of the conceptual probabilistic model.

The NHMM simulates the dynamics of the monthly gridded SST_t (SST at time t) using hidden states S_t whose evolution is Markov but with probabilities changing over time using covariates X_t (such as annual cycle and long-term warming). The teleconnection to future τ-month-ahead Energy_t+τ of Texas is mapped through the hidden states. All the conditional relationships in the figure are modeled as conditional probabilities with parameters estimated to maximize the likelihood of the SST field.

RESULTS

Our main findings are:

1) We develop spatially aggregated forecasts of climate-influenced monthly energy demand and renewable supply as conditional means and conditional variances of these variables given the SST hidden state, for multiple months (from 1 to 6) in the future. We find that, typically, the conditional mean has a much higher correlation with the observed energy variable than if a forward correlation is made directly with the ENSO index. The conditional variance provides insight into the potential uncertainty in the forecast and the likely probabilistic coverage interval for a future energy observation.

2) The nonlinearity and heterogeneity in the teleconnection response are revealed by the variation in the forecast skill spatially across Texas, temporally across calendar months, and by the hidden state used for conditioning.

3) The correlations in the spatially distributed and aggregated energy demand and renewable energy supply are influenced by the SST teleconnections, implying that the potential risks of energy surplus or drought at the aggregate level require an understanding and modeling of these features.

Response of energy variables integrated over Texas

The NHMM identifies five hidden states based on the monthly spatiotemporal dynamics of the tropical Pacific SST field using the annual cycle as a covariate. The SST composites of these hidden states are loosely similar to the patterns usually associated with ENSO classifications (classical La Niña, mild La Niña, neutral, Central Pacific El Niño, and classical El Niño). The spatial composites of SST corresponding to each hidden state are shown in Fig. 2. The annual progression based on the dominant annual hidden state and the corresponding annualized energy variable distributions for each hidden state are provided in fig. S1 to provide a qualitative link to the ENSO literature. Over Texas, CDD, HDD, and wind and solar energy varied with ENSO states (fig. S1). Particularly, in years when El Niño–like hidden states are active, the anomalies are negative for energy supply (solar and wind). In addition, the energy supply surpasses the energy demand (CDD and HDD) during years with the La Niña–like hidden states, while during El Niño–like hidden states, there is a deficit in the energy supply relative to the energy demand (fig. S2). This suggests some promise for our goal of exploring the potential predictability of the energy variables for several months forward considering the current SST hidden state.

Fig. 2. Spatial pattern of SST for each hidden state identified by the NHMM.

Composite SST of each hidden state, labeled as (A) classical La Niña, (B) mild La Niña, (C) neutral, (D) central El Niño, and (E) classical El Niño, to correspond loosely to common ENSO nomenclature, identified using the NHMM for the tropical Pacific Ocean.

To set the stage for the results from our models, consider the correlations between the spatially averaged monthly (derived from hourly data) CDD, HDD, wind (W), and solar (S) energy for Texas with the NINO3.4 index that is commonly used to explore ENSO teleconnections. Correlations for concurrent (lag = 0) and NINO3.4 leading by 1, 3, and 6 months ahead are illustrated in fig. S3. Oddly, CDD-NINO3.4 has significant correlations in the winter months, while HDD is barely significant at some lead times in the fall through spring. Wind has significant correlations at 3 months—negative in February and positive in July. Solar has significant correlations in May, October, and November. Overall, the results are not impressive, and the question is if our proposed method can improve on them.

We develop an m-month-ahead forecast of each variable v_t+m = {W, S, HDD, CDD}_t+m for each calendar month i(t + m) by using the Pacific SST hidden state (see fig. S4 with the Viterbi sequence) identified for month t and then computing the conditional mean $\overline{v}$_t+m and SD sv_t+m. This is the simplest model that one could consider for this setting. The correlation of this forecast with the observed v_t+m is illustrated in Fig. 3. The figure conveys considerably more promise than the correlation with NINO3.4 suggested. If we consider the m = 1- and 6-month-ahead forecasts and compare the one based on hidden states with the one based on NINO3.4 using abs(correlation_hidden_states) − abs(correlation_NINO3.4), then we see in Fig. 4 that the performance of correlations using hidden states is better than that using NINO3.4, particularly for May to December for both lead times. More sophisticated models could no doubt be developed using either the hidden states or NINO3.4. Here, we just offer a comparison of their potential predictability.

Fig. 3. Skill of continuous forecasts from 1950 to 2020.

Month-by-month forecast skill for different lead times (1, 2, 3, 4, 5, and 6 months) and calendar month for Texas spatially averaged (A) CDD, (B) HDD, (C) solar, and (D) wind. The solid circle in each grid indicates the significance of that forecast skill at the specified level based on a bootstrap test of significance of the conditional mean. The forecast correlation greater than 0.4 is highlighted by black box.

Fig. 4. Correlation difference between the hidden states and NINO3.4 index.

The (A) lag_1 and (B) lag_6 correlation difference between forecasts using hidden states versus those using the NINO3.4 index, computed by abs(correlation_hidden_states) − abs(correlation_NINO3.4). If the difference is larger than 0 (located in the pink shaded area), then that means the performance of hidden states model is better. We see that this is predominantly the case for both lead times and for most calendar months. Note that, potentially, one could improve the forecasts for both the hidden states and the NINO3.4 index by using more sophisticated methods. The results here merely suggest differences in linear predictability.

Spatial variability in forecast skill by SST hidden state

Looking beyond the performance at the aggregate level, we consider the spatially gridded HDD, CDD, wind, and solar data and illustrate, as examples, the hidden state–based forecast made in November for February (i.e., a 3-month lead time) and made in January for July (i.e., a 6-month lead time) in Figs. 5 and 6, respectively, and summarized in Table 1. Unlike in the previous section where the overall predictability was assessed through the correlation between the predicted and observed variables, here, we stratify the results based on the SST hidden state for the month when the forecast was prepared. This allows us to explore when there is skill, and this information can then be presented to users based on the hidden state identified from the SST data for the current month. We anticipate and find that hidden state 3, corresponding to ENSO neutral conditions, typically leads to poorer predictability than one of the states that corresponds to an El Niño or La Niña condition. The strength of predictability can be classified into three levels: strong (significant at the 1 or 5% significant level), moderate (significant at the 5 to 10% significant level), and weak (significant at the 10 to 20% significant level). All the spatial forecast skill maps (1, 3, and 6 months ahead) for each calendar month can be found in the Supplementary Materials.

Fig. 5. The 3-month-ahead forecast results for February.

(A) Conditional mean values and (B) bootstrap significance test for the null hypotheses of no effect for HDD, CDD, solar, and wind in February over Texas, using the SST hidden states from the previous November.

Fig. 6. The 6-month-ahead forecast results for July.

(A) Mean values and (B) bootstrap significance test for HDD, CDD, solar, and wind in July over Texas, using the SST hidden states from the previous January.

Table 1. Summary of predictability using ENSO information.

Predictability of wind, solar, HDD, and CDD in February and July using the SST hidden states from the previous November and January, respectively. Strong, moderate, and weak indicate that the forecast results are significant at the 0.01 or 0.05, 0.05 to 0.1, and 0.1 to 0.2 significance level, respectively. Letters “C”, “N”, “S”, “W” and “E” stand for central, north, south, and west, respectively. “+” and “−” denote that the forecast value is high and low compared with multiyear average, respectively.

Months	States	Wind	Solar	HDD	CDD
February	1	Strong CSE+	Strong WCS+	No skill	Strong S and coastal+, moderate CW+
	2	Weak coastal−	No skill	Strong W−, moderate coastal−, and weak CS−	Strong coastal+, weak CW+
	3	Moderate S and coastal+	Weak W+	Weak SE+	No skill
	4	Strong CE−	Strong WC−	No skill	Strong W−, moderate CSE−
	5	Strong CE−, moderate N−	Weak coastal+	Strong W+, weak C and SW+	Strong E−, moderate WS−
July	1	Weak SE+	No skill	No skill	Strong S−
	2	Moderate far E	Moderate C+	Strong and weak NE−	No skill
	3	Moderate C and SE−	No skill	Strong and moderate E+	Moderate far S+
	4	Strong far W−	Moderate SW−	Strong NW+	Weak SW− and far E+
	5	Strong+	Weak NW+	Weak NE−	Strong EWS+

For the February hidden state 2 (“mild La Niña”), the predictability is strong for low HDD in west Texas, moderate for low HDD in the coastal area, and weak for low HDD in central and south Texas. Similarly, using hidden state 5 (“classical El Niño”) predictability is strong for high HDD in west Texas and weak for high HDD in central and southwest Texas in February, while the forecasted wind is expected to be low with strong or moderate predictability except for far west and far south Texas (Fig. 5). A July forecast based on hidden state 5 indicates high CDD in west, east, and far south Texas with strong predictability and, in northwest and south Texas, with moderate and weak predictability. Correspondingly, wind is forecasted to be extreme high across Texas with strong and moderate predictability except for west Texas (Fig. 6) for this hidden state. The results from Figs. 5 and 6 are summarized in Table 1. The figures with forecast results and summaries of forecast results (tables S1 to S3) for other lead times, calendar months (January to December), and each hidden state can be found in the Supplemental Materials. In summary, there is considerable variation in the predictive skill depending on the calendar month, SST hidden state at the time of forecast, and the variable of interest. The direction of the response of a variable to the SST hidden state is usually the same across Texas. However, the amplitude and the statistical significance of the response do vary spatially.

Cross-correlations across energy demand and supply

If solar and wind are positively correlated, then shortages/surpluses are likely to be amplified as the underlying climate state changes. Conversely, if they are negatively correlated, then a decrease in one is offset by an increase in the other. Similarly, a negative correlation between HDD or CDD and wind or solar suggests that, when the thermal energy demand increases/decreases, the renewable energy supply decreases/increases, and one could have an energy drought or surplus. However, when they are positively correlated, their changes are potentially offsetting, leading a reduced risk of drought/surplus.

For Texas, there is a high negative correlation between HDD and solar or wind in November to February, while for May to September, the correlation between CDD and solar or wind is high and positive (Fig. 7). This means that the renewable energy system could have trouble with reduced supply and higher energy demand in winter, but in summer, increases/decreases in thermal demand are matched by like changes in renewable energy supply. In addition, throughout the year, solar and wind are positively correlated with each other and are negatively correlated with precipitation. Thus, a wind-solar-hydro complementary power system could be beneficial for Texas.

Fig. 7. Correlations between HDD, CDD, wind, solar, and precipitation over Texas.

(A to L) Correlations between spatially averaged values from January to December for 1950 to 2020. The size of the circle indicates the magnitude of correlation, and the color indicates the direction of the correlation, that is, warm color for positive correlation and cold color for negative correlation. The time series of aggregated HDD, CDD, wind, solar, and precipitation (PRECI) are shown in fig. S5.

Revisiting Table 1 in this context, we note that the forecast for February indicates that if November has SST hidden state 5 (typical El Niño), then wind may be weaker in central, northern, and eastern Texas, while solar will be mildly higher in coastal Texas, and HDD (heating demand) could be higher in the western parts of Texas. For the July forecast, we see that there is an increase in CDD (cooling demand) if January is in state 5 but with a corresponding increase in the wind and solar available that would help offset that condition.

DISCUSSION

The utility and feasibility of subseasonal-to-seasonal prediction of hydroclimatic variables such as precipitation, temperature, and streamflow using long-range teleconnections with climate indices have been established by many diagnostic and modeling analyses over the past two to three decades. As the penetration of renewable energy into the electricity grid has increased, the concepts of energy drought and energy surpluses are entering into the energy system planning discussions (38–43). Financial instruments, such as energy derivatives and option/forward contracts, have historically been used by system operators as well as large energy users to hedge risks associated with climate-induced demand extremes (44–48). However, now, stochastic supply and demand and the spatial dependence structure of these factors also need to be considered (49). Where regional supply and demand respond to large-scale climate teleconnections and the interest is in the response of a spatially aggregated energy demand and renewable generation, as is the case for the ERCOT in Texas, then a quantification of the anticipated changes for a subregion in the next several months, the conditions under which those changes manifest and are predictable, and how they are correlated across space and variable provides a basis for integrating that information into specific generation and demand centers and for decision-making at the aggregate level. There are studies analyzing the impacts of ENSO on the energy variables on a seasonal basis and discussing the potential to predict energy variables a few months in advance based on ENSO information (25, 50).

Our paper provides the exploration of this structure. We find evidence that a model that uses SST hidden states can perform better for predicting these variables at 1 to 6 months in the future than a model that just uses correlations with an ENSO index. However, we also find that the predictability varies substantially by variable, by calendar month, and by the current SST hidden state, and its spatial projection is also quite variable over Texas. These kinds of variations are also seen in prior analyses of global precipitation fields (31). Thus, the practical question that arises is if climate-based forecasts can actually be useful for system operation, given that they have significant skill only when the extreme ENSO states are active. While the answer to this question will depend on the sophistication of and tools available to a decision-maker—a system operator or an energy purchaser, the primary utility may well be in anticipating potential extreme conditions with energy drought or surplus and designing and executing financial risk hedging strategies when needed (51). We are working on the development of appropriate financial instruments that can leverage such information and plan to show the utility of the proposed strategies, now that the potential conditions have been identified in the context of Texas. It is possible that climatic factors other than ENSO, including modes of secular climate change, are influencing the evolution of supply and demand of renewable energy (38, 52). Because ENSO is the best understood and identified global mechanism of seasonal-to-interannual climate variations (32, 53), our initial exploration was focused on its utility. Future work will focus on a more comprehensive modeling approach that can use additional predictors to characterize the conditional probabilities of impending energy drought/surplus and on strategies for investing in appropriate financial instruments that are informed by this information (54). A fully stochastic simulation that leverages the regimes of ENSO and its impacts may also be useful for a climate risk assessment of energy supply-demand imbalances, and our approach is directly useful as a building block for such an analysis because the NHMM can be used to simulate long stochastic sequences of SST regimes that condition the energy variables of interest.

Our current model aimed at assessing the predictability of spatially varying renewable energy supply and climate-influenced energy demand conditional on the SST regimes and did not use any of the target energy data to parameterize the model. Rather, we use the SST hidden states to consider the conditional mean and variance of each target variable and assess the corresponding skill for a full year of evolution and identify calendar months and subregions where there may be skill at different lead times, relative to the bootstrap distribution that prescribes a null model. A forecast model that uses the spatial distribution and intensity of the SST field to formally predict the energy field over the monthly lead times would be very useful, and our results here provide a benchmark for both the design and the testing of such a model. We note that the correlation skill of the conditional mean given the hidden state classification is typically higher than that of the NINO3.4 index, which provides a commonly used measure of ENSO intensity.

Unlike the practice in many statistical analyses of teleconnections where all time series are detrended prior to correlative analyses, we focused on the identification of the canonical patterns of SST spatiotemporal evolution using the NHMM and then associate the teleconnection of interest with the hidden state thus identified. Thus, we focus on how a particular SST forcing pattern leads to a teleconnected response, i.e., their physical linkage. The nonstationary occurrence of the SST pattern can be modeled separately by extending the NHMM to consider that the hidden state marginal and transition probabilities depend on time or on a smoothed version of the global temperature or CO₂ as a proxy for the changing climate (55). However, this representation may not alter the teleconnection to the hidden state but merely change its frequency of occurrence. We prefer this approach to detrending because it provides an explicit model for the process, including potential nonstationarities, whose parameters can be formally estimated.

We conducted additional analyses to check if the response of the energy variables to the hidden state of SST had changed over time. We used the Anderson-Darling test to check if the energy variables conditional on each hidden state have the same distribution across the first and the second half of the data. In addition, we used the Mann-Kendall test to check the stationarity of the energy variables (table S4) conditional for each hidden state. We find no trends at the 5% significance level for most of the energy variables’ responses to different hidden states, except for CDD, HDD, and solar under classical La Niña and CDD under neutral state. In addition, there is no significant difference between the distributions of most energy variables for 1950 to 1985 and 1986 to 2020, except for CDD, HDD, and solar under classical La Niña. The nonstationarity of the energy variables is partly due to the small sample size of the classical La Niña months after 1980 (fig. S6). We further used regression analysis to check the conditional dependence of the energy variables on global temperature and hidden states on energy variables (table S5). The global temperature is significant only for CDD and HDD. Thus, we see that it would be useful to also condition the CDD and HDD variables on the smoothed global temperature for the classical La Niña and neutral ENSO states, if we wanted to implement a fully nonstationary model.

MATERIALS AND METHODS

Observational datasets

Climate data

Hourly, gridded (0.5° × 0.5°) wind speed at 100-m altitude, downward surface solar radiation, and 2-m surface temperature for the period of 1950 to 2020 were taken from the ERA5 reanalysis datasets (https://ecmwf.int/en/forecasts/dataset/ecmwf-reanalysis-v5) for Texas. Monthly datasets were derived for these fields. The monthly standardized anomalies of wind speed and solar radiation at time t, denoted by va_t, were computed for each pixel, respectively, using

$${\mathit{va}}_t=\left[v_t-{\overline{v}}_{i\left(t\right)}\right]/{\mathit{sv}}_{i\left(t\right)}$$ (1)

where v_t is the observed monthly climate variable at time step t. ${\overline{v}}_{i\left(t\right)}$ and sv_i(t) are the mean and SD of monthly climate variable, respectively, for calendar month i(t) associated with month t. The ERA5 dataset is one of the most used global reanalysis datasets that provides gridded climate variables with high spatiotemporal resolution (56). Here, we used ERA5 because it can provide long-term (1950 to 2020) and spatially gridded data (0.5° × 0.5°). Although it is reported that the ERA5 dataset is the best reanalysis dataset in representing daily and extreme temperature across the United States (57), it should be noted that the reanalysis dataset could introduce biases and errors, especially in regions with sparse observational data and complex terrain.

Sea surface temperature

SST anomaly data are taken from the monthly Kaplan Extended SST V2 dataset for the period of 1856 to 2020 (58). The spatial resolution of this dataset is 5° × 5°.

Calculation of HDD and CDD

The thermal variation in residential energy demand is assessed using HDD and CDD as proxies. The ambient indoor temperature (AIT) is assumed to be 18.3°C (65°F). It should be noted that 18.3°C is recommended by the American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE) (59) and is commonly used in degree-days studies in the United States (60, 61) as well as in global analysis (62) to calculate the HDD and CDD. To be consistent with ASHRAE and ensure comparability with other studies, we used 18.3°C as the AIT. If the temperature is higher than the AIT, then residential cooling requirements emerge, and conversely heating requirements are assessed. Monthly HDD and CDD are computed based on the deviation between the AIT and temperature, which are

$$\{\begin{array}{c}{\mathit{HDD}}_t=\frac{1}{p}{\displaystyle \sum _{k=1}^p}\text{max}(18.3-T_{\mathit{tk}},0)\hfill \\ {\mathit{CDD}}_t=\frac{1}{p}{\displaystyle \sum _{k=1}^p}\text{max}(T_{\mathit{tk}}-18.3,0)\hfill \end{array}$$ (2)

where HDD_t and CDD_t are the monthly mean HDD (°C/hour) and CDD (°C/hour) for month t, respectively; T_tk is the temperature for hour k (1,…,p) in month t. These are computed at every grid cell.

Prediction methods of energy demand and supply

Ahmad et al. (14) provide a detailed review of the forecasting of renewable energy, i.e., wind, solar, geothermal energy, and electricity load demand requirement using different machine learning models at different forecast lead times, i.e., short (up to 72 hours), medium (week to year), and long-term prediction (multiple years). Most reported studies focused on short-term (hour to day) forecasts rather than a medium or long-term forecast (11–14). Short-term energy forecasting is crucial for power companies to ensure a consistent supply and reliability. However, medium and long-term predictions can provide valuable insights for investment and strategic planning from the current standpoint. Most just use historical solar, wind, or temperature data from the region of interest and focus on the shorter timescales where there is more data volume.

We briefly summarize the methods used in our past work to identify spatiotemporal patterns or hidden states of the tropical Pacific Ocean SST and then describe how we use those patterns to assess the predictability of renewable energy supply and demand for the succeeding months.

Identifying the SST hidden states: The NHMM

Rojo Hernández et al. (37) showed the advantages of the NHMM in simulating and predicting the spatiotemporal evolution of ENSO using SST fields. They highlighted the capability of the NHMM to capture asymmetry, long-term trends, persistence, and seasonal evolution of ENSO events, compared with the widely used dynamical and statistical ENSO prediction models. The NHMM (fig. S7) application to SST is now briefly sketched. R_t = (R_t¹, …, R^M_t), t = 1, …, N refers to the time series of the M gridded SST anomaly values. The R_t is assumed to depend on a hidden state S_t. The hidden state series S_1:N is assumed to be a nonstationary Markov process. Time-varying covariates X_t are used as predictors for the hidden state Markov transition probabilities. As a result, the hidden state probabilities change over time. The hidden state S_t takes values from the set {1,…,k}, in month t. It is determined by both the predictor vector X_t and the hidden state S_t−1 in the previous month t − 1. The model is defined as

$$\begin{array}{c}\text{Pr}\{{\mathbf{R}}_{1:N},{\mathbf{S}}_{1:N}\mid {\mathbf{X}}_{1:N}\}=\hfill \\ \text{Pr}\{S_1=h_1\mid {\mathbf{X}}_1\}{\displaystyle \prod _{t=2}^N}\text{Pr}\{{\mathbf{S}}_t=h_t\mid {\mathbf{S}}_{t-1}=h_{t-1},{\mathbf{X}}_t\}{\displaystyle \prod _{t=1}^N}\text{Pr}\{{\mathbf{R}}_t\mid {\mathbf{S}}_t\}\hfill \end{array}$$ (3)

Here, we allow the hidden state transition probabilities to depend on the annual cycle by defining

$${\mathbf{X}}_t\mathit{=}[\mathbf{\text{sin}}(\mathrm{\pi }t/6),\mathbf{\text{cos}}(\mathrm{\pi }t/6)] \text{for} t=1,\dots ,N$$ (4)

In a related work (59), we have also considered a smoothed global temperature as a covariate to model the potential aspects of climate change. The R package “NHMM” (63) that uses Bayesian inference was used to develop the NHMM for the SST data, and the details are provided in (31, 37). The Viterbi sequence identified by the NHMM represents the sequence of the “optimal” hidden states assigned to each historical month. The frequency of occurrence of each of these states changes over time. Five hidden states were identified as optimal in our analysis, and based on the spatial patterns of the SST averaged over each state, they are qualitatively identified as “classical La Niña, mild La Niña, neutral, Modoki or Central Pacific ENSO, and the classical El Niño.” Note that these labels may not match the definitions of the ENSO state using other criteria. The spatial composite of the SST field associated with each hidden state identified is presented in Fig. 2A. This hidden state sequence is then used to assess how the energy variables for Texas may respond for future months.

Monthly predictability given the hidden SST state information

Consider that we are in a specific calendar month (e.g., January) and “know” the hidden state assigned to the Pacific SST (e.g., classical La Niña) and are interested in what may happen to one of the Texas energy variables (e.g., solar radiation) in March (i.e., m = 2 months into the future). We address this by identifying all months for which the hidden SST state for that calendar months corresponds to the one of interest (e.g., Januaries, with classical La Niña) and then computing the mean of the energy variable of interest for all the corresponding m months later (e.g., March such that January is classical La Niña). This conditional mean of the energy variable given the hidden state for the specified calendar month is then the m-month-ahead forecast.

Formally, for the period of 1950 to 2020, we first identify the hidden state s(t) = k associated with each calendar month i(t); (ii) for each energy variable v, compute the mean values ${\overline{v}}_{i(t+m),s\left(t\right)=k}$ across all calendar months i(t + m), corresponding to the hidden state s(t) = k as the forecast vf_t+m for month t + m

$${\mathit{vf}}_{t+m}={\overline{v}}_{i(t+m),s\left(t\right)=k}$$ (5)

The lead time m was varied from 0 (i.e., concurrent) to 6 in the results presented here. This forecast procedure is applied to all pixels of HDD, CDD, solar, and wind. Correspondingly, we also compute the conditional SD sv_{i(t+m),s(t)=k}. The conditional mean and the conditional SD provide a measure of the conditional probability distribution of the energy variable.

Continuous monthly predictability assessment

After assessing the statistical significance of the conditional forecast given the SST hidden state, we consider a continuous application of the forecast. A continuous “forecast” is developed by sequentially applying the forecast vf_t+m, month by month over 1950 to 2020, and then spatially averaging the gridded values (over all of Texas) of HDD, CDD, wind, and solar for m months into the future. The Pearson correlation coefficient between the “forecasted” and observed variable is calculated for the selected lead time and month. This correlation is compared with that of NINO3.4_t with observed values of the variable for month (t + m).

Consider an example. Say the hidden state of January 1950 is state 2, the lead time m is 6 months, and the variable of interest is HDD. The conditional mean of HDD for July associated with the years whose January SST hidden state is state 2 is the forecast vf_t+6 for July 1950. This procedure is applied sequentially for each July from July 1950 to July 2020, using the SST state identified for January of that year. The forecast skill for a 6-month-ahead HDD forecast for July made in January is then the correlation between this vf_t+6 and v_t+m. This is compared with the correlation between NINO3.4_t and v_t+m.

Bootstrap significance test

The statistical significance of vf_t+m for each calendar month i(t), each hidden state s(t), and each lead time m (0, 1, 3, and 6) is tested against the null hypothesis of no dependence on the hidden state s(t) = k. We have 71 observations for each calendar month from 1950 to 2020, and the detailed procedure is summarized as follows:

1) Bootstrap a sample with 71 observations with replacement for each calendar month i(t);

2) Generate K subsamples associated with hidden states 1 to K, by randomly allocating the bootstrapped observations with the same sample sizes as in the historical record for each hidden state;

3) Compute vbf_t+m,j for each such bootstrap sample j using the same process as for the historical record, using the corresponding bootstrapped vb_t+m,j values. Note that here the assignment of the hidden state k is random and does not correspond to the NHMM applied to the historical data.

4) Repeat steps (i) to (iii) 1000 times and generate 1000 vbf_t+m,j for each hidden state and each calendar month; and

5) Identify the percentile of vf_t+m relative to the bootstrap sample of 1000 bootstrapped vbf_t+m,j for each calendar month i(t) and hidden state s(t) = k. If this percentile is below the 5th percentile or larger than the 95th percentile, then the forecast is considered to be significantly different from what would be expected from a null model with random assignment of hidden states.

Note that, because no data on the energy variables were used in selecting the SST hidden states, the hypothesis test using the bootstrap provides a direct comparison of the energy variable across two groups of sample sizes of equal length: one drawn randomly from the historical data and one drawn based on the conditioning on the SST hidden state. Thus, varying sample sizes associated with each hidden SST state are explicitly addressed.

Supplementary Materials

This PDF file includes:

Supplementary Text

Figs. S1 to S7

Tables S1 to S5