Residual delay maps unveil global patterns of atmospheric nonlinearity and produce improved local forecasts

Abstract

We use residual-delay maps of observational field data for barometric pressure to demonstrate the structure of latitudinal gradients in nonlinearity in the atmosphere. Nonlinearity is weak and largely lacking in tropical and subtropical sites and increases rapidly into the temperate regions where the time series also appear to be much noisier. The degree of nonlinearity closely follows the meridional variation of midlatitude storm track frequency. We extract the specific functional form of this nonlinearity, a V shape in the lagged residuals that appears to be a basic feature of midlatitude synoptic weather systems associated with frontal passages. We present evidence that this form arises from the relative time scales of high-pressure versus low-pressure events. Finally, we show that this nonlinear feature is weaker in a well regarded numerical forecast model (European Centre for Medium-Range Forecasts) because small-scale temporal and spatial variation is smoothed out in the grided inputs. This is significant, in that it allows us to demonstrate how application of statistical corrections based on the residual-delay map may provide marked increases in local forecast accuracy, especially for severe weather systems.

Over the past 30 years, there has been growing theoretical and modelling evidence of a marked change in atmospheric nonlinearity between tropical and extratropical regions. In the tropics, although the important driving mechanisms are highly nonlinear (convective in nature, occurring at the meso-scale), the large-scale response is primarily linear (1–5). There is little wave–wave interaction and this large-scale variability seems to be largely a response to anomalies in sea- and land-surface temperature and albedo, and soil moisture (6). By comparison, much of the large-scale variability in the extratropical atmosphere is highly nonlinear and arises from baroclinic instability associated with the development of cyclones, anticyclones, and fronts (6–11).

Most studies of atmospheric nonlinearity have concentrated on theoretical aspects (e.g., chaos in models) or have examined analyzed meteorological fields, either by normal mode decomposition or by computing variances and covariances. A few investigators have attempted to directly analyze time series of field observations to obtain estimates of the degree of deterministic chaos in the atmosphere (12–17). However, many of the methods used to find chaos in field data, and in particular, those involving the simple computation of the correlation dimension and its relatives (18), are not compelling, in that apparent low-dimensional chaos can easily arise as an artifact of some forms of random noise (19–21). This has cast into doubt many of the results from application of these techniques to observational data.

Here, we step back from the perhaps overly restrictive question “Is the atmosphere chaotic?” to address more general and pertinent issues regarding the presence and functional form of nonlinear structure in atmospheric time series. This is in the spirit of Crutchfield and McNamara, who were able to use observational data to deduce equations of motion for an experimental system (22). We introduce a new residual-delay technique for studying observational field data of necessarily limited length. This method is not subject to the difficulties mentioned above, and it provides a fresh approach to obtain direct evidence of the degree of atmospheric nonlinearity, its latitudinal gradient, and its variability within a particular region. Moreover, we show how the V shape of the residual-delay map (RDM), and in particular the response of this shape to the time scales used in map generation, provides fundamental information about the physical mechanisms responsible for the observed atmospheric nonlinearity.

We conclude by addressing the practical importance of this basic atmospheric property as it applies to weather forecasting. We apply the residual-delay method to a numerical weather prediction model from the European Centre for Medium-Range Forecasts (ECMWF) and demonstrate that model output does not capture the V shape present in real atmospheric data. By incorporating this feature as a statistical correction, however, the model’s spatially smoothed predictions can be adjusted to produce improved local forecasts. The corrected forecasts show most promise for the cases of greatest interest: extreme pressure events, including severe weather.

Methods

We present two demonstrations, based on an RDM and anisotropic variances, to show that the atmosphere in temperate regions is indeed nonlinear and that this nonlinearity diminishes toward the tropics. In comparison to previous studies outlined earlier, we are able to use observational data to identify the specific source and nature of the nonlinearity, and we demonstrate how this varies systematically across a latitudinal gradient and within a geographical region.

We apply these methods to daily time series of observed barometric pressures from 25 monitoring sites around Australia (Fig. 1), each of which spans 30–45 years (10,000–17,500 points each). The reason for focusing on station and mean sea-level pressure as variables is that they are measures of the integrated mass of an atmospheric column and key dynamic variables. In Australia, where the upper-level network is poor, especially over the surrounding oceans, and the land is relatively flat, the MSLP chart is by far the most widely used and disseminated of all forecast charts. We illustrate the use of the methods by choosing four representative locations: Darwin (tropical), Townsville (subtropical), Perth (temperate, west coast) and Sydney (temperate, east coast). Samples of the time series for these locations are shown in Fig. 2.

Figure 1.

Map showing locations of 24 monitoring sites for field measurements of barometric pressure. The other station used, Macquarie Island (54° 29′ S, 158° 56′ E), lies south of the map.

Figure 2.

Representative sections of the time series of daily barometric pressures (hPa at 0900 LST), for two tropical sites at Darwin and Townsville and two temperate sites in western (Perth) and eastern (Sydney) Australia. Station locations are in Fig. 1. The colors refer to the four seasons: magenta, winter; cyan, spring; red, summer; blue, fall. These colors should be cross-referenced with Fig. 3.

RDM.

The first method, which we call the RDM, involves constructing the maximum likelihood AR3 model (an optimal global linear autoregressive model). We found that in nearly all 25 cases, a model having three lagged coefficients produces the best, or nearly best, out-of-sample linear forecasts of the one-day-ahead mean sea-level pressure (explaining at least 80 percent of the variance). Thus, we consider daily linear forecast solutions having the simple linear form X̂_t+1 = a₁X_t + a₂X_t−1 + a₃X_t−2, where X̂_t+1 is the linear prediction for tomorrow’s pressure (t + 1), and X_t, X_t−1, and X_t−2 are the observed pressures for t, t − 1, and t − 2. We then examine the relationship between the residual to the prediction at t + 1 (R_t+1 = X_t+1 − X̂_t+1) and the observed raw barometric pressure at time t, X_t (hence, the name residual-delay map).**

To reduce the variance and clarify any systematic patterns that may exist, the raw values of barometric pressure are first sorted and binned into groups of 100. For each bin, the average residual from a linear forecast for the points in that bin is calculated and plotted against the mean barometric pressure of the bin. RDMs thus employ a two-way averaging procedure: each point in the RDM can be thought of as the expected residual (at time t + 1) where the expectation is calculated over 100 points, contained within a given bin of barometric pressure (at time t). The map (Fig. 3) effectively shows how the expected residual of the optimal linear forecast model (AR3) varies as a function of past barometric pressure.

Figure 3.

RDMs of barometric pressure (hPa) at time t, versus the binned residual for the AR3 prediction at time t + 1 for four representative time series shown in Fig. 2 [RDM(AR3)]. The colors correspond to those in Fig. 2 and indicate AR3s for each season. The white points use a single AR3 for the whole year.

If the underlying process responsible for generating the barometric time series is linear then there should be no structure to the residuals displayed; that is, they should fluctuate evenly around zero. Alternatively, if there is nonlinear structure of sufficiently low dimension to be detected with the data in hand, the functional form of the lagged nonlinearity—the fahshion in which the residuals depart from a random distribution centered around zero—will likely become apparent.

The plots in Fig. 3 show a lack of any clear nonlinearity in the tropics (Darwin and Townsville), but a very distinct nonlinear signature in the temperate zone (Perth and Sydney). Moreover, the nonlinearity that is displayed at the temperate sites is shown to be present in all seasons (cf. colors in Fig. 2). This information can be used in two ways: (i) to predict the next few steps in the time series and (ii) to statistically correct deterministic numerical weather prediction forecasts for local (subgrid-scale) conditions and model bias. In this paper we concentrate on the second application.

Although residuals are commonly used to judge the fit of models, our method is novel in that we plot the binned residuals against the time-lagged raw variable.‡‡ Structure emerging from an RDM identifies pressure-specific time-dependencies in the residuals that can be exploited to improve prediction. Moreover, we shall see it can provide insight into the physical meaning of the time scales of the dynamics. This method does not require intensive computation as do other nonlinear methods, and the data do not have to be continuous as in spectra or polyspectra.

Anisotropic Variances.

The second method of detecting nonlinearity in field data involves constructing a one-dimensional phase portrait and tests the assumption that variances in an AR model should be isotropic and bivariate Gaussian between lagged variables. That is, if the underlying process is linear and driven by Gaussian noise, the variability in the scatter between lagged variables should be isotropic. Thus, to implement this test, one simply needs to plot the pressure at time t, (P_t), versus the pressure at time t + 1, (P_t+1), as is done in Fig. 4, and measure the anisotropy of the variance in the scatter of points obtained to get an indication of the degree of non-Gaussianity (often associated with nonlinearity) potentially present in the data.

Figure 4.

One-dimensional phase portraits for the four time series in Fig. 2.

Although this approach is not novel, it is simple to implement and provides an independent test of the findings from the RDM. Anisotropic variances are not a conclusive test of nonlinearity, however, as non-Gaussianity does not necessarily imply nonlinearity. Further, they do not show the specific functional form of the nonlinearity or how to exploit it to improve forecastability. Thus we use this only as a corroborative test of the RDM, which has the advantage that it can show the specific form of the nonlinearity. To our knowledge, this is the first use of phase portraits to obtain a quantitative measure of the increased variance with low pressure (the anisotropy of the system), which occurs at temperate latitudes.

Results and Discussion

Figs. 3 and 4 indicate a consistent latitudinal trend with regard to the relative nonlinearity of the time series from the four sites in Fig. 2. Darwin (tropical) and Townsville (subtropical) appear to lack any nonlinear structure in the RDM (Fig. 3) and have essentially the same variability at times of high and low pressure (isotropic variances) (Fig. 4). Their dynamics, therefore, appear to be adequately modeled as linear processes.†† In sharp contrast, the two temperate sites, Perth (west coast) and Sydney (east coast), both of which have more variable time series that appear noisier to the naked eye (Fig. 2), also show distinct nonlinear signatures, both in the RDM and in the demonstrated anisotropic trend of higher variability with lower pressure.

Of particular note is the V-signature shown in the RDMs for the high-latitude stations. Clearly the residuals are not Gaussian, as would be expected from a linear system. The interesting nonlinear structure appears to a first approximation to be roughly quadratic or V-shaped in the first lag. Moreover, as shown by the color-coded seasons in Fig. 3, this nonlinear signature is an annually robust feature that is present in all seasons. It is present during, El Niño, La Niña, and non-event years, and with the seasonal mean removed. As will be discussed, this V-shape is the key to understanding the nature of the nonlinearity in daily observations of barometric pressure.

We next extend these methods to analyze the daily time series of barometric pressures from all of the 25 Australian monitoring sites shown in Fig. 1. An index of nonlinearity is defined by taking the mean of the squares of the binned residuals of an RDM, R, and normalizing this quantity by the product of the standard deviations of the binned and unbinned residuals, r, to correct for the larger residuals in midlatitudes compared to the tropics:

As a rough approximation, this index of nonlinearity is related to the correlation between predicted and actual residuals, where predictions are made using 100 nearest neighbors (21, 25–29).

Let r̂_t+1 denote the predicted residual from a 100 neighbor prediction, which is approximated by binning the residuals as in an RDM. Then the correlation between predicted and actual residual is given by

But on the right-hand side, we may approximate r̂_t+1 = r_t+1 + n_t+1, where n_t+1 is a noise term uncorrelated with r_t+1, so that the sum may be approximated as the sum of the squared terms of the RDM.

Fig. 5 demonstrates that the index of nonlinearity for RDMs is nearly constant equatorward of 25° latitude, increases rapidly towards southern Australia, and then perhaps decreases again at higher latitudes. The uniformly low indices in the tropics lie within roughly one equatorial Rossby radius of deformation from the equator. The sharp increase into midlatitudes and the evidence for a decrease at higher latitudes also correspond closely to the shape of the storm frequencies in this region (11).

Figure 5.

Index of nonlinearity for the barometric pressure series at each of the 25 locations in Fig. 1. The higher the index, the more nonlinear the series. Error bars indicate 2 standard deviations from the mean for the index. The schematic curve illustrates that tropical dynamics are independent of latitude and that extratropical dynamics have a maximum in nonlinearity which corresponds to the approximate latitude of the average storm track [cf. the covariance plots of Trenberth (11) as a function of latitude].

A similar result is found for the index of anisotropy of variances (Fig. 6). Here a model of the form |R_t+1| = aX_t + b is fitted to the barometric pressure, X_t, by least squares. Because the residuals, R_t+1, have more variability at low pressures, a negative value is obtained for a. We refer to the absolute value of a as the index of anisotropy of variances as it quantifies the results in Fig. 4 (the cone-shape, or increase in variance between X_t and X_t+1 at temperate latitudes).

Figure 6.

Summary of the degree of non-Gaussianity in the AR method for the barometric pressure series at each of the 25 locations in Fig. 1. The higher the index, the more non-Gaussian the series. The curve is as in Fig. 5.

Nonetheless, for reasons that will become clear, the particular form of nonlinearity revealed in the RDM is a feature that is not fully reproduced in the prevailing numerical weather prediction data (ECMWF for Perth and Sydney). More importantly, we shall show that the V shape in the RDM contains information about the nature of the time scales associated with different dynamical events.

Nature of the V-Shaped Nonlinearity.

A major feature of the global circulation is the Inter-Tropical Convergence Zone, a zone of low pressure that moves from the southern to the northern hemisphere from approximately May to November in the Australasia region. As a result, areas close to the equator will exhibit a clear seasonal change in pressure (e.g., Darwin and Townsville in Fig. 2). Indeed, the Inter-Tropical Convergence Zone movement will dominate the variability in these locations. For places far from the equator (e.g., Perth and Syndey), this periodic forcing will weaken and the series will appear noisier. However, this effect alone cannot account for the latitudinal gradient in atmospheric nonlinearity we have documented (see supplemental Fig. 10 on the PNAS web site, www.pnas.org).

We hypothesize that the quadratic feature found in the RDM may be due, in large part, to a difference in the time scales of high-pressure events versus low-pressure events (E. N. Lorenz, personal communication). Low-pressure systems tend to be more ephemeral and localized than do high-pressure systems, and have the higher amplitude and stronger gradients. The passage of alternating highs and lows past a station will create an asymmetry in the shape of the observation time series, which might generate this nonlinear feature.

This idea is demonstrated with the toy model illustrated in Fig. 7 Upper, which generates a sequence of gently rounded highs and spiky lows:

with u_i a uniform random variable in the range [0.5, 1.5].

Figure 7.

(Upper) Time variation of the toy model that simulates passage of sharp lows and broad highs past a station. (Lower) RDM of the time series generated by the toy model.

This simple geometry reproduces the V-shape in the corresponding RDM (Fig. 7 Lower). Moreover, that this broad feature which we have modelled in caricature here (persistent highs, and less persistent lows) is true can be verified empirically by computing the transition probability for a low event following a low P(L_t|L_t−1) and comparing it to that for a high event following a high P(Ht|H_t−1). In Table 1 we see that for Perth and Sydney these transitions are very asymmetrical with P(H_t|H_t−1) > P(L_t|L_t−1); however, this is not the case for the tropical sites of Darwin and Townsville where the highs are not more persistent than the lows.

Table 1.

Transition probabilities for highs and lows for field observations and the ECMWF model predictions

City name	Conditional probabilities
	Field observations			ECMWF model predictions
	P(Ht|Ht−1)	P(Lt|Lt−1)	Skewness	P(Ht|Ht−1)	P(Lt|Lt−1)	Skewness
Darwin	0.838	0.843	−0.37  ±  0.02	0.855	0.870	−0.36  ±  0.09
Townsville	0.827	0.836	−0.52  ±  0.02	0.861	0.835	−0.63  ±  0.09
Perth	0.638	0.547	−0.01  ±  0.02	0.680	0.624	−0.23  ±  0.09
Sydney	0.703	0.575	−0.12  ±  0.02	0.661	0.542	−0.31  ±  0.09

Here, we condition on high and low values that are more than 0.75 standard deviations from the mean. As a correction for the skewness of the distribution of the barometric pressure, the standard deviation is calculated separately for points on either side of the mean. The table shows that there is a strong asymmetry in the transition probabilities in the field observations for the temperate sites (Perth, CHI² = 68.32, P < 0.0001; Sydney, CHI² = 103.4, P < 0.0001) which is absent in the tropics (Darwin, CHI² = 0.3895; Townsville, CHI² = 0.8791, both not significant). Moreover, the asymmetry at the temperate sites is less marked in the ECMWF forecasts (Perth, CHI² = 0.906, not significant; Sydney, CHI² = 4.222, P < 0.05). 

A final and compelling piece of evidence in favor of this hypothesis is created by changing the observation time scale for the pressure series at Sydney (Fig. 8). If indeed the dominant feature responsible for the nonlinearity we have measured is the differing time scales of high- versus low-pressure events, then reducing the observation time step should result in the rounded and persistent highs becoming more nearly linear. At the same time, we would expect the spiky lows to continue to appear more strongly nonlinear. This would differentially shrink the right limb of the V as the highs become more linear with increasing temporal resolution, or sharpen the V as the time step increases. Both of these processes are clearly occurring in Fig. 8, lending confidence to our hypothesis and the results of the toy model simulation.

Figure 8.

RDM graphs for observed barometric pressures at Sydney sampled and forecast at four different time scales: (a) 36 hours, (b) 24 hours, (c) 12 hours, (d) 6 hours. These RDMs are calculated for a data set with 3-hour samples.

This implies that linear models based on daily pressure observations will underestimate both high and low pressure regimes. In effect, the quadratic structure may be thought of as a quantification of the basic feature of mid-latitude synoptic weather systems: persistent and large highs and ephemeral, small and intense lows. We believe that this is the dominant source of nonlinearity that can be detected in daily observations of barometric pressure.

Forecasting and Comparison with Numerical Models.

We are now interested in investigating whether the V shape in the lagged residuals is captured in output from the ECMWF model. To this end, RDMs were created for 2 years of forecast data (1 October 1989 to 30 September 1991), with Lagrangian cubic interpolation used to extract daily series of mean sea-level pressure at the locations of Darwin, Townsville, Perth, and Sydney. As will be explained below, only a weak V structure is found for Sydney and Perth. This concurs with the lower asymmetry seen in the transition probabilities for the ECMWF as compared with the observations in Table 1. The table shows that there is a strong asymmetry in the transition probabilities in the field observations for the temperate sites which is absent in the tropics; moreover, the asymmetry at the temperate sites is less marked in the ECMWF forecasts.

Why should the ECMWF model fail to capture this nonlinear feature of barometric pressure time series? The answer arises from the fact that forecast models of the atmosphere do not use the monitoring station data directly. Instead, they employ a data assimilation and analysis procedure that takes into account (i) the raw observations, their reliability (instrument error), and their representativeness; and (ii) the state of the atmosphere (all the atmospheric variables must be mutually compatible and certain balance conditions must be satisfied) (30). The input data are then analyzed onto a regular spatial grid at fixed times. Use is made of the model in this analysis step, so that the final input data, as well as the forecast, are limited by the model spatial resolution and its parameterization of physical processes that occur on scales too small to resolve or that are too complex to solve explicitly. This combined process results in a smoothing of small-scale spatial and temporal variations. Numerical forecast models also may have biases in particular geographic regions (31) that are corrected in operations by application of statistical corrections, such as the Model Output Statistics (MOS) (32–34).

Does spatial averaging play a role in the degradation of the nonlinear V signature? We demonstrate the possibility by considering a set of stations–Albany, Port Lincoln, Adelaide, Wagga Wagga, and Canberra–that are all located at approximately the same latitude (35S; Fig. 1). The procedure is first to create an average barometric pressure time series for each pair of stations, and then construct a residual delay map for these spatially averaged pressure series. The index of nonlinearity is calculated as before and plotted as a function of the spatial separation betweeen the pair of stations contributing to that averaged time series (Fig. 9). If spatial averaging is important in the loss of the nonlinear signal in the ECMWF, then we expect this procedure to lead to a smooth decline in the index as the station pairs become increasingly remote. This is precisely what we observe in Fig. 9, lending credence to this explanation for the failure of the ECMWF to capture the nonlinearity of observational barometric pressure data in temperate latitudes.

Figure 9.

Degradation of nonlinearity as a function of spatial averaging. The index of nonlinearity is calculated for average pressure series between pairs of stations located at ^∼35S latitude, and plotted as a function of distance between the pair. This test supports our interpretation that spatial averaging (as is typical of ECMWF analysis) leads to a degradation of the V shape and, correspondingly, the nonlinearity of the data.

Although the ECMWF model was not designed to forecast raw observations at sub-grid scales, we suggest that it is possible to adapt it to do so by using RDMs to provide site-specific corrections. This can be done empirically as an added quadratic term. These corrections are possible because the spatial smoothing of the ECMWF results in systematic errors when forecasts are made at finer scales. The systematic nature of the error, an underestimation of high-pressure and an overestimation of low-pressure values, is a consequence of the temporal assymetry of event durations observed in a particular location being mirrored by a spatially assymetric distribution of barometric pressure values. In other words, short-lived extreme low-pressure events also tend to be local events, whereas longer-lived times of high barometric pressure tend to be broader in space as well. This characteristic is obscured by the coarse spatial scale and smoothing of the ECMWF.

We test this using RDMs to provide site-specific forecasts for each of our four representative locations. For each point in the series from each site, corrections were made out-of-sample by keeping points in the same year and season from contributing to the construction of the RDM. So, for example, the calculation of the correction for forecasts made for the spring of 1991 utilized no information about other pressure conditions during the spring of that year. As expected, RDMs offer the most improvement to forecasts in times of extreme pressure events. At Sydney, for example, we see 28 and 20 percent improvement in average error at the two lowest-pressure bins, whereas for Perth the best results are for times of high pressure, with a 16% improvement in mean error for the highest-pressure bin. Also as expected, results in tropical latitudes are more erratic: RDMs give significantly improved forecasts at Darwin regardless of pressure, but not at Townsville. RDMs are even more effective under some conditions at removing systematic error from forecasts; for example, there is a 70% improvement at Darwin during times of lowest pressure. Detailed results are given as supplemental data on the PNAS web site (www.pnas.org).

A logical expectation is that an improved monitoring grid and a higher-resolution model will lead to more successful forecasts and an improved ability to capture the nonlinear V shape we have described. As a surrogate, it appears that RDMs may be able to correct the current model as is, tuning spatially smoothed predictions into better forecasts at single points. As in the above examples, such corrections may be expected to best improve site-specific prediction skill in the temperate region during extreme climatic events. This is of particular relevance to the public, as there are also times of greatest interest, including severe weather.

In conclusion, we have introduced the RDM as an effective and readily implemented tool for detecting nonlinear structure in observationally derived natural time series. The application of RDMs to records of barometric pressure has enabled us to unveil and quantify global patterns of atmospheric nonlinearity, and extract the specific functional form—the V shape—of this basic nonlinearity. We have provided a mechanistic explanation of the origin of this shape in terms of relative time scales of low- and high-pressure events. In the weather forecasting context, applied to ECMWF output, the RDM provides a new, powerful statistical method to correct forecasts for model bias and local variability, in particular for times of extreme conditions, making the forecast more useful to the public. Finally, while a detailed comparison of the RDM method with the widely used MOS technique for statistical correction to weather forecasting models will be reported at a later date, we stress that the RDM method is unique in that it incorporates specific information about relative time scales of atmospheric events into its correction. As such, the improved forecasts are not purely phenomenological corrections, but rather result from deeper insight into time scales of the system’s dynamics.

Abbreviations

RDM

residual-delay map

ECMWF

European Centre for Medium-Range Forecasts