Adaptive data analysis: theory and applications

In many applications in science, engineering and mathematics, it is useful to understand functions depending on time and/or space from many different points of view. Accordingly, a wide range of transformations and analysis tools have been developed over time. Fourier series and the Fourier transform, first proposed almost 200 years ago, provided one of the first mechanisms to write a complex waveform as the linear combination of elementary wave functions; many more would follow. Nearly 100 years ago, it became clear that for some applications it is especially useful that the elementary ‘building block functions’, into which more complex signals are decomposed, have a limited spread in both time and frequency—transformations or representations that used such simultaneous time–frequency (or space/spatial frequency) localization have been important tools in micro-local arguments in mathematics, quantum mechanics and semi-classical approximations, and many types of signal and data analysis. Typically, the tools used to compute such transforms or representations are linear in the input—making them (fairly) easy to implement, and versatile instruments in the data analyst’s toolbox. Yet, in some cases, the very versatility of these linear tools makes them come up short, and to obtain a more detailed, precise analysis, it becomes necessary to adapt parameters and procedures to (often local) behaviour changes of the data or signal. Examples of this abound. With the advances of sensor technology, we are dealing with vast increases in not only the volume of data to be analysed, but also in their quality—leading to the ubiquitous discussions of what to do with all these ‘big data’. Big data provide not only a challenge, but also an opportunity, especially because computation and storage have likewise become much more powerful. In response to these needs and opportunities, adaptive data analysis methods are being developed and explored for many different scientific and engineering frameworks.

Such data-adaptive methods for signal or data analysis are the focus of this theme issue. The past few years have seen several workshops on this general topic, organized by the international applied mathematical community: the 2011 Hot Topic Conference ‘Instantaneous Frequency and Trend for Nonlinear and Nonstationary Data’ at the Institute of Mathematics and its Applications in Minnesota; a 2013 weeklong workshop ‘Adaptive Data Analysis and Sparsity’ at the Institute of Pure and Applied Mathematics, UCLA; and the 2015 International Conference on ‘Optimization, Sparsity and Adaptive Data Analysis’ at the Morningside Center of Mathematics, Chinese Academy of Sciences. None of these produced one convenient volume describing to a wide public the different approaches that were presented. This theme issue is meant to fill this gap, and to give visibility to both theoretical and application aspects of the important topic of adaptive data analysis. We hope it will also serve as a new starting point for future developments.

Some of the articles in this issue concern new, adaptive versions of earlier, non-adaptive methods in which settings or parameters are fine-tuned based on first observations, and for which this adaptivity leads to surprisingly strong results; others are completely new nonlinear approaches. Some of the articles concentrate on developing new theory; others are focused more on special concrete applications.

The authors for the articles in this theme issue stem from many different fields, including applied mathematics (with articles on new theoretical developments by Cicone et al. [1]; Daubechies et al. [2]; Hou & Shi [3]; Joliffe & Cadima [4]; and Mallat [5]), physics (Meignen et al. [6]), physiology (Yeh et al. [7]), general science and data analysis (Huang et al. [8]), meteorology and climate (Qiao et al. [9] and Wu et al. [10]) and biomedical research (Hemakom et al. [11]).

To make this issue useful to a wide audience, all the authors were encouraged to include a review of existing methods, and to illustrate their approach with many examples. When space restrictions made it impossible to explain their arguments as fully as they wished, they were encouraged to expand their arguments in the electronic supplementary material published online, concurrently with this issue.

We hope that readers will enjoy this issue and that it will encourage them to explore adaptive data analysis approaches for their own data challenges.