Opportunities and challenges for machine learning in weather and climate modelling: hard, medium and soft AI

Abstract

In September 2019, a workshop was held to highlight the growing area of applying machine learning techniques to improve weather and climate prediction. In this introductory piece, we outline the motivations, opportunities and challenges ahead in this exciting avenue of research.

This article is part of the theme issue ‘Machine learning for weather and climate modelling’.

Ever since the development of the first fully programmable electronic digital computers at the end of World War 2, the cutting edge of weather prediction science has lain in the development of accurate numerical representations of the governing dynamical equations of meteorology. As begun in the 1950s [1], these numerical representations have been further developed to allow first-principle simulations of climate. Since then, numerical modelling of weather and climate have dominated meteorological forecasting, to the extent that only 10 years ago prediction using statistical empirical models seemed rather antiquated.

With the recent advances in deep neural networks and other related machine learning techniques, statistical empirical prediction is firmly back in vogue. This raises a number of questions. Will the methods of artificial intelligence replace numerical models? Or, failing that, can they be used to supplement or enhance numerical models?

These were the overriding themes of a workshop held at Corpus Christi College Oxford in September 2019, papers from which are published in this Special Issue.

In order to address the questions posed above, it is worth breaking weather and climate prediction into different timescales.

Now-casting describes the evaluation of the weather and in particular precipitation right now or with a forecast lead time of a couple of hours. Here, conventional predictions are mostly based on observations and advection models with only limited representation of physical processes such as cloud physics or turbulence. Prudden et al. [2] summarize the current state of the field, particularly in the context of machine learning.

Short-range weather prediction describes forecasts for a day or two ahead. On these timescales, one may be interested in the timing of a cold front coming in or how individual thunderstorm complexes propagate and develop. This is a challenging problem as such cloud systems require very high-resolution modelling using spatial grids that have horizontal spacings on the order of 1 km. Inevitably, given current computational power, this means short-range prediction models are regional and not global in their domain. The development of such convective cloud systems depends critically on the condensation of moisture, these frequently occur in updrafts too narrow to be properly resolved even with a 1 km grid [3]. As a result, modelling on these space and timescales remains challenging.

Medium-range prediction, on timescales of a couple of days out to a couple of weeks, has, perhaps more than any other timescale, seen the most dramatic increases in skill over past decades [4]. The archetypal phenomenon on the medium-range timescale is the baroclinic wave, whose characteristic scales are well resolved by contemporary global numerical weather prediction models with grid spacings on the order of 10 km. Accurate prediction on these timescales over two generations of baroclinic waves is often possible.

Subseasonal forecasting spans timescales from two weeks to one season. A key source of predictability on this timescale is the Madden Julian Oscillation [5,6]. This is an eastward-propagating equatorially trapped oscillation in tropical winds with a timescale of 30–80 days, associated with suppressed or enhanced organized convection. This timescale has seen much attention in recent years with the birth of the Subseasonal to Seasonal (S2S) Prediction Project [7]. Associated with this project are several large databases containing S2S forecasts from operational and research models [7,8], invaluable for potential ML applications.

Seasonal forecasting depends critically on the coupled dynamics of the atmosphere and ocean and an archetypal phenomenon is the El Nino event in the tropical Pacific Ocean [9,10]. Coupled ocean-atmosphere models show skill in prediction for El Nino on seasonal timescales [11]. However, at this range, models must use grid spacings of many tens of kilometres. At these resolutions, models show substantial biases in key fields such as precipitation and wind. These biases arise from the need to parametrize key physical processes, notably associated with deep convective cloud systems.

Finally, climate-change prediction asks the question of how the statistics of weather changes as the greenhouse gas concentration of the atmosphere increases, on timescales of decades and longer. Again, on these timescales, the numerical models (with grid spacings of 100 km) exhibit quite substantial biases whose magnitudes are comparable with the climate-change signal one wishes to predict [12].

With this as background, let us ask the question: could prediction on any or all of these different timescales simply be replaced by AI? One might refer to this as ‘Hard AI’.

Now-casting applications appear to be a good starting point for Hard AI. On these timescales, physical constraints, such as conservation laws, can be ignored as errors will not accumulate significantly over a couple of hours. Furthermore, the wealth of Internet-of-Things (IoT) data, for example, in the form of mobile phone data, is increasingly available and usable for weather predictions. Assimilating this data using conventional methods will be very difficult due to the high number of measurements and large errors. In promising recent work, machine learning methods are becoming competitive for short timescales (e.g. [13]). These approaches combine the data assimilation and forecasting problems, directly using observations (e.g. satellites) as inputs to the prediction system.

For short and medium range predictions, an approach using Hard AI becomes more challenging. Skilful forecasts on these timescales would implicitly require the representation of the equations of motion and the interactions between features of the system (such as clusters of deep-convection, gravity waves and mid-latitude jets). It is in principle possible to learn the equations of motion from data using machine learning methods [14–16], however, challenges with this approach include satisfying conservation principles and stability of the simulations. Forecast skill is found to decrease after a few days, such that it is currently difficult to envisage such Hard AI outperforming and hence potentially replacing numerical models on medium-range timescales. However, for short-range predictions, this approach can produce skilful forecasts [16] indicating potential for Hard AI to compete with numerical models. An alternative approach which has shown some promise on short timescales is to combine analogue forecasting (prediction using a database of past evolving weather patterns) with deep learning [17].

One of the biggest challenges for Hard AI on short and medium-range timescales is the lack of high-quality data that is available for training. To train machine learning systems from conventional models would potentially generate improvements in high-performance computing efficiency. This would, however, not improve prediction quality. If the model data could be generated at a higher spatial resolution than can be currently run within an operational forecasting window then there is scope for prediction improvements, however, the computation power required to generate this training dataset would be large. To train from observations requires consistent and high-quality data which is only available since satellite measurements have been made, beginning with Spunik 1 in 1959 [18]. This restricts the training datasets to $\mathcal{O}({10}^4)$ days of training data. There is perhaps scope to pre-train on existing model data before final training on some observational datasets, but it is unclear if model and observational datasets are close enough for pre-training to be a valuable step.

For seasonal predictions, machine learning methods are more likely to beat conventional prediction systems. Here, the systematic errors of models are sufficiently large that AI may soon be competitive [19,20]. However, the same challenges of training datasets size still exist for the seasonal prediction problem.

Adopting the Hard AI approach on the climate change timescale is particularly difficult. The problem is inherently one of extrapolation, predicting climates not yet observed, a major challenge for machine learning techniques. One key question for climate change is whether cloud feedbacks are positive or negative. There is no indication from existing data that global cloud cover is increasing or decreasing [21]. Hence a single black-box AI scheme would struggle in giving us meaningful projections of the future. Our only hope are schemes which fully embrace the laws of physics. This could incorporate AI but likely as a sum of many elements rather than a single entity.

The potential for Hard AI was discussed extensively at the workshop, and no consensus was reached as to whether it will be likely to outperform conventional tools for medium-range and climate predictions. In particular, participants with a focus on high-performance computing were optimistic regarding machine learning tools, since their computational power will jump by a large factor in the coming years as chips and computing systems are co-designed towards the need of deep learning. Furthermore, the speed of developments in machine learning techniques is breathtaking with O(100) papers published on artificial intelligence every day.

If at least some of the weather and climate applications cannot currently be replaced in total, how can machine learning methods help to still improve their predictions? Via the use of what we call ‘Medium AI’ and ‘Soft AI’. Here, the intention is not to replace the entire model. It is rather the aim to improve forecasts using tools that have been learned from observations—Medium AI—or via the emulation of existing kernels of weather forecasting models—Soft AI—to allow for improvements in computing efficiency. Several papers in this Special Issue discuss such an application of AI techniques.

Medium AI should have a sufficient wealth of independent data points for each application. One approach uses machine learning as a postprocessing tool. This includes downscaling grid scale output to some point-specific location within a gridbox, bias correction, improving the representation of forecast uncertainty, and automating model post-processing such as feature detection for early warning of extreme events. These applications are fairly uncontroversial as they have a long-standing history—not necessarily for deep learning but certainly for the wider toolbox of machine learning [22,23]. One promising technique is the use of Generative Adversarial Networks (GANs) [24] to increase spatial resolution of model or satellite data [25].

A second approach with Medium AI is to replace components of the models with new tools that have been learned from observations or other data sources (such as high-resolution models). This is an attractive proposal, which could transform the representation of poorly understood, complex physical processes in our models. However, this is hindered by the lack of knowledge of the coupling between grid-scale and sub-grid-scale processes. There is a view that parametrization—reducing sub-grid variability to some sort of bulk formula—is an inherently ill-posed concept, which would set an upper bound on the improvements possible from AI [26]. There are also difficulties associated with using observational data (for example from satellites) for training of parametrization schemes as data on all vertical levels of the model are required. This is typically only possible to achieve within data assimilation systems but the signal-to-noise ratio will be challenging when extracting information from observations. Nevertheless, there are some model components which are amenable to a Medium AI approach, particularly those which are based on empirical relationships. One example is the scaling function used within Monin–Obukhov similarity theory to determine interactions between the atmosphere and the surface layer [27].

The Soft AI approach was first used by Krasnopolsky [28], and Chevallier et al. [29] over 20 years ago. In particular, by training a neural network on some existing parametrization scheme (e.g. long-wave radiation) we can replace the parametrization with an AI scheme which will not be more accurate than the original scheme, but will arrive at the parametrized tendency with significantly reduced computational cost. The saved computational cost can then be reinvested to improve the resolution or complexity of the numerical model, which in principle should improve the accuracy of the model. This approach is, in our view, among the lowest hanging fruit in the applications of AI to weather and climate modelling. It should be noted that this approach still has challenges, for example Brenowitz & Bretherton [30] found that multi-timestep training was necessary to produce a stable simulation.

Putting all this together, we believe that the real way forward is through a careful assessment of how AI-based schemes and numerical models can be synthesized to allow the production of next-generation high-resolution global weather and climate models running on exascale supercomputers. It can be expected that such models will, as far as possible, be used across all timescales, from days to decades. With climate change affecting the nature and frequency of weather extremes, making many in the world even more vulnerable to natural hazards, the need for such enhanced prediction models has never been greater.

The weather and climate modelling community has a long history in the use of machine learning techniques, such as parameter estimations or the extraction of the leading dynamic modes. However, the recent boost in capabilities of machine learning, and in particular deep learning, and the recent advance in HPC resources that enable the use of machine learning on much more complicated problems, is yet to be fully exploited. Doing so requires the development of additional skill and infrastructure within the community.

1. Workshop overview

In addition to presentations and posters, our workshop also included breakout groups. These discussed:

• —How to strengthen the link between physical understanding and machine learning
• —How to improve physical parametrization schemes within models
• —How to improve HPC efficiency of weather and climate models using machine learning
• —How to improve the useability of model output using machine learning

It was evident that there is a need for the community to develop machine learning solutions that are customized to the needs of weather and climate modelling, rather than relying on machine learning solutions that are mainly developed for other application areas such as image recognition. Such customized solutions would need to be able to impose physical understanding within the solution architecture. This is necessary to build machine learning tools of minimal complexity to optimize efficiency during training and inference. Furthermore, machine learning solutions for weather and climate applications would need to cope with changes of dynamic regimes due to climate change and would therefore need to be able to be trained outside of the training regimes that are available in past weather. In an ideal but unlikely scenario, solutions would be able to accurately extrapolate in addition to interpolation, where standard deep learning thrives. Some promising machine learning approaches that were presented at the workshop focused on the correction of a signal rather than the prediction of the signal itself. This provides a natural connection between conventional tools (that are used to generate the signal) and machine learning tools (that are used to correct the signal) and makes solutions more resilient against errors as the machine learning signal becomes smaller. Another promising approach is the use of machine learning in uncertainty quantification. This includes both the use of approaches to represent uncertainty within models, for example via the use of multi-bin Markov chains, GANs [31], or the use of machine learning techniques to post-process the output of ensemble simulations [32,33].

To enable machine learning solutions to take up information from physical knowledge, or to at least obey conservation properties, a number of research projects were presented such as an adjustment of architecture designs, constraint terms in loss functions during training, genetic programming or regression techniques that learn coefficients for terms that were guided by physical understanding (equation discovery), or the more conventional approach to perform parameter optimization, for example via the use of approximate Bayesian computing.

A number of infrastructure requirements to support machine learning applications within weather and climate modelling have been identified during the workshop. These include

• —To design standard methods that enable researchers to easily link Python and Fortran programmes as most machine learning methods are trained and used within Python code (based on machine learning libraries such as TensorFlow [34] or Keras [35]) while weather and climate models are typically based on Fortran code. Such a link between Python and Fortran would in particular be important for machine learning approaches that try out online learning within a running model. Some progress on this task has been made by [36].
• —To develop benchmark training datasets to allow for a qualitative comparison between machine learning approaches that are developed within the different scientific groups (see also [37]). The benchmark datasets would need to be tiered in terms of complexity including small datasets for conceptual testing up to the terabyte range to allow for tests of high-performance computing efficiency during both training and inference. For most datasets that are available currently, the time frequency is not good enough to allow for training of machine learning solutions.
• —To make better use of heterogeneous hardware when running weather and climate models. Most current weather forecasting models run on CPU hardware only, whereas machine learning solutions are typically faster on GPU hardware. This may generate tension for weather and climate computing centres with their own supercomputing facilities. Better adoption of heterogeneous hardware for weather and climate models would have benefits outside of machine-learnt kernels as these models feature dense linear algebra which can be further accelerated with low numerical precision [38].
• —To prepare data centres for larger data requests by machine learning users and to provide computing infrastructure that is optimal for training and inference close to the data.
• —To develop vanilla solutions for machine learning in weather and climate applications that reduce the need for excessive screening of all architecture and training options within deep learning. It may even be possible to use and share pre-trained machine learning solutions between groups. In particular, machine learning solutions that are able to work on unstructured grids and with spherical geometry as well as the three-dimensional structure of atmosphere and ocean models with changing grid spacings in the vertical are required.

It is still unclear how the workflow of weather and climate modelling will change in the coming 5–10 years. However, it is very likely that machine learning will be used within many components within the workflow. This will not necessarily make the workflow—which is already very complicated—easier and it will require machine learning resources such as special hardware and staff with experience in the design of machine learning tools. However, the use of machine learning within the workflow will likely allow a number of improvements in results.

This special issue contains a mix of new research and review articles. Most of the articles directly attack the problem of weather forecasting, with many in the community believing that this is a good testing ground before considering the problem of climate prediction. The new research articles mostly focus on what we have dubbed Medium or Soft AI, looking to improve parameterization kernels of weather and climate models or optimize existing kernels. There are several papers concerning the problem of data assimilation, an important and expensive part of weather prediction. Contributions also take on the subject of probabilistic forecasting, a key approach to forecasting on all timescales, as well as the exciting topic of machine learning for post-processing (please see Haupt [39] in this issue for an overview on post-processing approaches).

Data accessibility

This article has no additional data.

Authors' contributions

Each author committed equally to the writing of this introductory paper.

Competing interests

We declare we have no competing interests.

Funding

We gratefully acknowledge generous funding from the sponsors of the Machine Learning for Weather and Climate Modelling workshop, namely the ESiWACE-2 Centre of Excellence, the Oxford Martin School, the Office of Naval Research, Amazon, Vulcan Inc., the Copernicus Atmosphere Monitoring Service (CAMS), the Copernicus Climate Change Service (C3S), and NVIDIA. The ESiWACE-2 project has received funding from the European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No 823988. Tim Palmer acknowledges funding under the ERC Advanced Grant 741112 ITHACA.