Operator compression with deep neural networks

Abstract

This paper studies the compression of partial differential operators using neural networks. We consider a family of operators, parameterized by a potentially high-dimensional space of coefficients that may vary on a large range of scales. Based on the existing methods that compress such a multiscale operator to a finite-dimensional sparse surrogate model on a given target scale, we propose to directly approximate the coefficient-to-surrogate map with a neural network. We emulate local assembly structures of the surrogates and thus only require a moderately sized network that can be trained efficiently in an offline phase. This enables large compression ratios and the online computation of a surrogate based on simple forward passes through the network is substantially accelerated compared to classical numerical upscaling approaches. We apply the abstract framework to a family of prototypical second-order elliptic heterogeneous diffusion operators as a demonstrating example.

Introduction

The remarkable success of machine learning technology, especially deep learning, in classical AI disciplines such as image recognition and natural language processing has led to an increased research interest in leveraging the power of these approaches in other science and engineering disciplines over the last years. In the field of numerical modeling and simulation, promising approaches are emerging that try to integrate machine learning algorithms and traditional physics-based approaches, combining the advantages of the data-driven regime with known physics and domain knowledge. In this spirit, many different approaches to approximating solutions of partial differential equations (PDEs) with neural networks have been proposed, for example so-called physics-informed neural networks (PINNs) [60], the deep Galerkin method [63], or the deep Ritz method [19]. It has become evident that the strategy of using neural networks as ansatz functions for the approximation of a PDE’s solution is especially advantageous for high-dimensional problems that are outside the reach of classical mesh-based methods [17, 18, 33]. For some classes of PDEs, e.g., Kolmogorov PDEs and semilinear heat equations, it has even been proven that neural networks break the curse of dimensionality [9, 39].

In this spirit, we strongly believe that the strength of neural networks lies in scenarios where one deals with a whole family of PDEs rather than one single equation, for example in the context of so-called parametric PDEs, i.e., settings where a family of partial differential operators parameterized by some coefficient is considered, see, e.g., [62]. This is particularly true for multiscale problems, where one is interested in computing coarse-scale surrogates for problems involving a range of scales that cannot be resolved in a direct numerical simulation.

In this paper, we study the problem of approximating a coefficient-to-surrogate map with a neural network in a very general setting of parameterized PDEs with arbitrarily rough coefficients that may vary on a microscopic scale. In other words, we are not trying to directly approximate the parameter-to-solution map, but rather compress the fine-scale information contained in the continuous operator to a finite-dimensional sparse object that is able to replicate the effective behavior of the solution on a macroscopic scale of interest even in the presence of unresolved oscillations of the underlying coefficient.

The output surrogate models are based on the idea of modern numerical homogenization techniques such as localized orthogonal decomposition [46, 49, 56], gamblets [52], rough polyharmonic splines [53], the multiscale finite element method [21, 38], or the generalized finite element method [7, 20]; see [5] and the references therein for a comprehensive overview. These methods have demonstrated high performance in many relevant applications such as porous media flow or wave scattering in heterogeneous media to mention only a few. In particular, they typically do not require explicit assumptions on the existence of lower-dimensional structures in the underlying family of PDE coefficients and yield sparse system matrices that ensure uniform approximation properties of the resulting surrogate. Moreover, the computation of the system matrices mimics the standard assembly procedure from finite element theory, consisting of the generation of local system matrices and their combination by local-to-global mappings, which is exploited to reduce the size of the network architecture and its complexity considerably.

The possibility of fast computation of the surrogates has high potential for multi-query problems, such as in uncertainty quantification, and time-dependent or inverse multiscale problems, which require the computation of surrogates for many different a priori unknown coefficients. Though the aforementioned numerical homogenization methods lead to accurate surrogates for the whole class of coefficients, their computation requires the resolution of all scales locally which marks a severe limitation when it has to be performed many times for the solution of a multi-query problem. There have been attempts to tackle this problem, but the results so far are only applicable to small perturbation regimes [35, 50] or settings where the parameterization fulfills additional smoothness requirements [3].

To overcome this problem, we propose to learn the whole nonlinear coefficient-to-surrogate map from a training set consisting of pairs of coefficients and their corresponding surrogates with a deep neural network. In other words, we are combining the domain knowledge from numerical homogenization with a data-driven deep learning approach by essentially learning a numerical homogenization method from data. To this end, we propose using an offline-online approach. In the offline phase, the neural network is trained based on data generated with existing numerical homogenization techniques. In the online phase, the compression of previously unseen operators can then be reduced to a simple forward pass of the neural network, which eliminates the computational bottleneck encountered in multi-query settings.

Our method is conceptually different from the existing approaches that try to integrate ideas from homogenization with neural networks. In [6] for example, the authors propose to learn a homogenized PDE from simulation data by linking deep learning with an equation-free multiscale approach. Other papers in the context of uncertainty quantification suggest training a neural network to identify suitable multiscale basis functions for the finite volume method given a porous random medium [13, 54]. In [31], the authors consider the problem of elasticity with history-dependent material properties, where a recurrent deep neural network connects microscopic and macroscopic material parameters. In deep multiscale model learning [65], learning techniques are used to predict the evolution from one time step to another within a given coarse multiscale space. The goal of this approach is to obtain a reasonable coarse operator for the successive approximation of a time-dependent PDE. Furthermore, several experimental and theoretical works on the approximation of the coefficient-to-solution map [11, 28, 30, 43] or other quantities of interest such as the ground state energy in Schrödinger equations [41] by deep neural networks have been published in the context of parametric PDEs.

This paper is structured as follows: in Sect. 2, we introduce and motivate the abstract framework for a very general class of linear differential operators. After that, we study the problem of elliptic homogenization as an example of how to apply the general methodology in practice. In Sect. 4, we conduct numerical experiments that show the feasibility of our ideas developed in the previous two sections. We conclude this work with an outlook on further research questions.

Abstract framework

In this section, we describe the general abstract problem of finding discrete compressed surrogates to a family of differential operators that allow us to satisfactorily approximate the original operators on a target scale of interest, given only the underlying coefficients but not a high resolution representation of the operators. We elaborate on how to speed up the online computation of those compressed representatives using deep neural networks after an initial offline training phase.

Setting

Let $D\subseteq {\mathbb{R}}^d$, $d\in \{1,2,3\}$ be a bounded Lipschitz domain and $H_0^1(D)$ be the Sobolev space of $L^2$-functions with weak first derivatives in $L^2(D)$ that vanish on the boundary of D. We write $H^{-1}(D)$ for the dual space of $H_0^1(D)$ and $\langle \cdot ,\cdot \rangle$ for the duality pairing between $H^{-1}(D)$ and $H_0^1(D)$. Consider a family of linear differential operators

$$\mathfrak{L}:=\{{\mathfrak{L}}_A:H_0^1(D)\to H^{-1}(D)|A\in \mathfrak{A}\}$$

that is parameterized by some class $\mathfrak{A}\subseteq L^{\mathrm{\infty }}(D)$ of admissible coefficients. We emphasize that we do not pose any assumptions on the structure of the coefficients $A\in \mathfrak{A}$ such as periodicity or scale separation and explicitly allow for arbitrarily rough coefficients that may vary on a continuum of scales up to some microscale $\epsilon \ll \text{diam}(D)$. We assume that for every $A\in \mathfrak{A}$ the associated operator ${\mathfrak{L}}_A$ is symmetric ($\langle {\mathfrak{L}}_{A}u,v\rangle =\langle u,{\mathfrak{L}}_{A}v\rangle$), local ($\langle {\mathfrak{L}}_{A}u,v\rangle =0$ if u and v have disjoint supports) and bijective.

Bijectivity implies that for any given $A\in \mathfrak{A}$ and $f\in H^{-1}(D)$ there exists a unique $u\in H_0^1(D)$ that solves the equation

$${\mathfrak{L}}_{A}u=f$$ 2.1

in a weak sense, i.e., the solution satisfies

$$\langle {\mathfrak{L}}_{A}u,v\rangle =\langle f,v\rangle \text{for all }v\in H_0^1(D).$$ 2.2

For given problem data A and f, we are interested in computing an approximation to u on some target scale in reasonable time.

Discretization

In order to be able to solve this problem computationally, we choose a finite-dimensional subspace $V_h\subseteq H_0^1(D)$ of dimension $m=\dim (V_h)$. As a standard example, one could take $V_h$ to be a classical finite element space based on some mesh ${\mathcal{T}}_h$ with characteristic mesh size h and approximate (2.2) with a Galerkin method. However, in the very general setting with A possibly having fine oscillations on a scale that is not resolved by the mesh size h, this approach leads to unreliable approximations of u. Then again, the resolution of these fine-scale features can be prohibitively expensive in terms of computational resources if ε is very small. Note that resolution here may mean that the actual mesh size is significantly smaller than ε, depending on the oscillations of the coefficient and its regularity [8, 58]. This means that more advanced discretization techniques are required to still obtain reasonable approximations in the unresolved setting. In practice, the challenge is therefore to compress the fine-scale information that is contained in the operator ${\mathfrak{L}}_A$ to a suitable surrogate ${\mathfrak{S}}_A$ on the target scale h, i.e., the surrogate ${\mathfrak{S}}_A$ must be chosen in such a way that it is still able to capture the characteristic behavior of the operator ${\mathfrak{L}}_A$ on the scale of interest. Moreover, we require ${\mathfrak{S}}_A$ to be a bijection that maps the space $V_h$ to itself. This ensures that for any $A\in \mathfrak{A}$ and $f\in H^{-1}(D)$ we can find unique $u_h\in V_h$ that weakly solves the discretized equation

$${\mathfrak{S}}_A{u}_h=f_h,$$ 2.3

with $f_h=\mathfrak{M}f$, where $\mathfrak{M}$ is a quadrature-type operator that maps a function in $H^{-1}$ to an appropriate approximation in $V_h$. Problem (2.3) needs to be understood as finding $u_h$ that satisfies

$$\langle {\mathfrak{S}}_A{u}_h,v_h\rangle =\langle f_h,v_h\rangle \text{for all }{v}_h\in V_h.$$

The choice of the surrogate is obviously highly dependent on the problem at hand, see for example Sect. 3.2 for possible choices in the case of second-order elliptic diffusion operators.

Characterization in terms of a system matrix

We restrict our discussion to choices of surrogates that can be represented by an $m\times m$ system matrix $S_A$ that is often called the effective system matrix in the following. We assume that $S_A\in {\mathbb{R}}^{m\times m}$ is of the form ${(S_A)}_{ij}=\langle {\mathfrak{S}}_A{\lambda }_j,{\lambda }_i\rangle$ for a basis ${\lambda }_1,\dots ,{\lambda }_m$ of $V_h$. Note that the basis should be chosen as localized as possible in order for the resulting system matrix to be sparse. The process of operator compression can then be formalized by a compression operator

$$\mathfrak{C}:\mathfrak{A}\to {\mathbb{R}}^{m\times m}$$

that maps a given coefficient A to the system matrix $S_A$ representing the compressed surrogate ${\mathfrak{S}}_A$ of the operator ${\mathfrak{L}}_A$. Once $\mathfrak{C}$ has been evaluated for given $A\in \mathfrak{A}$, the solution to (2.2) can then be approximated with a function $u_h\in V_h$ for any right-hand side $f\in H^{-1}(D)$ by solving the linear system $S_{A}U=F$, where $F\in {\mathbb{R}}^m$ is the vector with entries $F_i:=\langle \mathfrak{M}f,{\lambda }_i\rangle$ and $U\in {\mathbb{R}}^m$ contains the coefficients of the basis representation $u_h={\sum }_{i=1}^m{U}_i{\lambda }_i$.

Multi-query scenarios

For many classes of coefficients $\mathfrak{A}$ and based on the choice of the surrogate, evaluating $\mathfrak{C}$ requires solving local auxiliary problems, during which the finest scale ε has to be resolved at some point. While this is acceptable if one wants to compress only a few operators in an offline computation, it becomes a major problem once $\mathfrak{C}$ has to be evaluated for many different coefficients A in an online phase, as for example in certain inverse problems, uncertainty quantification, or the simulation of evolution equations with time-dependent coefficients. This motivates a data-driven offline–online approach, where the offline phase consists of training a neural network to approximate the compression operator $\mathfrak{C}$, such that in the subsequent online phase the evaluation of $\mathfrak{C}$ can be replaced with a simple forward pass through the network, thus eliminating the computational bottleneck.

System matrix decomposition

In principle, one could try to directly approximate the global operator $\mathfrak{C}$ with a neural network. If the coefficient involves oscillations on some fine scale ε, this would lead to a network architecture with an input layer of size $\mathcal{O}({\epsilon }^{-d})$, an output layer of size $\mathcal{O}(m)$, and possible hidden layers. Particularly for small ε, this leads to very large networks and, thus, requires a huge amount of free parameters and therefore extraordinary amounts of training data and storage space in order to preserve good generalization capabilities.

To reduce the necessary size of the network, one can exploit available information on the compression operator $\mathfrak{C}$ by means of a certain structure in the resulting effective matrices $S_A$. To this end, we think of $S_A$ as a matrix composed of multiple inflated sub-matrices, i.e.,

$$S_A=\sum _{j\in J}{\mathrm{\Phi }}_j(S_{A,j}),$$ 2.4

where J denotes some given index set and $S_{A,j}\in {\mathbb{R}}^{s\times t},s,t\ll m$, are (typically dense) local matrices of equal size. The functions

$${\mathrm{\Phi }}_j:{\mathbb{R}}^{s\times t}\to {\mathbb{R}}^{m\times m}$$ 2.5

represent local-to-global mappings inspired by classical finite element assembly processes as further explained below. More precisely, there exist index transformations ${\pi }_j$ and ${\phi }_j$, such that

$${\mathrm{\Phi }}_j(S)[{\pi }_j(k),{\phi }_j(l)]=S[k,l],1\le k\le s,1\le l\le t,$$

where $M[i_r,i_c]$ denotes the entry of a matrix M in the $i_r^{\mathrm{th}}$ row and the $i_c^{\mathrm{th}}$ column. Note that ${\pi }_j$ and ${\phi }_j$ can also map to zero, indicating that the corresponding entry should be disregarded. Let us also emphasize that the mappings ${\mathrm{\Phi }}_j$ (and, in turn, the index transformations ${\pi }_j$ and ${\phi }_j$) are completely independent of coefficients A and solely depend on the domain D as well as the geometry of an allotted discretization. The precise definitions of the maps ${\pi }_j$ and ${\phi }_j$ as well as the index set J are usually determined in a canonical way by the choice of the computational mesh and the compression operator $\mathfrak{C}$. In Sect. 3.4, we present an example of how such mappings may look like.

Depending on the compression operator $\mathfrak{C}$ and decomposition (2.4), we can expect that all the local matrices $S_{A,j}$ are created in a similar fashion and only depend on a local sub-sample of the coefficient. This can be understood as a generalization of the assembly process that underlies classical finite element system matrices: these matrices are composed of local system matrices that are computed on each element separately and only require knowledge about the coefficient on the respective element. In that context, the local sub-matrices all have a similar structure and the mapping by the functions ${\mathrm{\Phi }}_j$ leads to overlapping contributions on the global level. Going back to the abstract setting, we generalize these properties and assume the existence of a lower-dimensional reduced compression operator

$${\mathfrak{C}}_{\mathrm{red}}:{\mathbb{R}}^r\to {\mathbb{R}}^{s\times t},$$ 2.6

such that the contributions $S_{A,j}$ are of the form

$$S_{A,j}=({\mathfrak{C}}_{\mathrm{red}}\circ R_j)(A),$$ 2.7

where the operators

$$R_j:\mathfrak{A}\to {\mathbb{R}}^r$$ 2.8

extract r relevant features of a given global coefficient. In the context of, e.g., finite element matrices, the operators $R_j$ correspond to the restriction of a coefficient to an element-based piecewise constant approximation and ${\mathfrak{C}}_{\mathrm{red}}$ incorporates the computation of a local system matrix based on such a sub-sample of the coefficient. To achieve a uniform length r of the output for the operators $R_j$, these operators may include artificially introduced zeros depending on the respective geometric configurations (e.g., at the boundary). An example for a quadrilateral mesh in two dimensions is shown in Fig. 1.

Figure 1.

Illustration of the extended element neighborhood ${\mathtt{N}}^1(T)$ around a corner element $T\in {\mathcal{T}}_h$. An asterisk indicates that $A_{\epsilon }{|}_K\in [\alpha ,\beta ]$, a zero that $A_{\epsilon }{|}_K=0$ in the respective cell K of the refined mesh ${\mathcal{T}}_{\epsilon }$

The problem of evaluating $\mathfrak{C}$ can now be decomposed into multiple evaluations of the reduced operator ${\mathfrak{C}}_{\mathrm{red}}$ that takes the local information $R_j(A)$ of A and outputs a corresponding local matrix as described in (2.7). In our setting of a coefficient A that is potentially unresolved by the target scale h, evaluating ${\mathfrak{C}}_{\text{red}}$ is nontrivial and might become a bottleneck in multi-query scenarios as already indicated in Sect. 2.4. In such cases, we propose to approximate the operator ${\mathfrak{C}}_{\text{red}}$ with a deep neural network

$$\mathrm{\Psi }(\cdot ,\theta ):{\mathbb{R}}^r\to {\mathbb{R}}^{s\times t},$$

where $\theta \in {\mathbb{R}}^p$ is a set of p trainable parameters of moderate size such that, for given $A\in \mathfrak{A}$, the effective system matrix $\mathfrak{C}(A)=S_A$ can be efficiently approximated by

$${\hat{S}}_A:=\sum _{j\in J}{\mathrm{\Phi }}_j\left(\mathrm{\Psi }(R_j(A),\theta )\right),$$

which requires just a single forward pass of the minibatch ${(R_j(A))}_{j\in J}$ through the network. Note that the approximation ${\hat{S}}_A$ possesses the same sparsity structure as the matrix $S_A$, since the neural network yields only approximations to the local sub-matrices $S_{A,j}$, whereas the assembling process which determines the sparsity structure of the global matrix is determined by the mappings ${\mathrm{\Phi }}_j$, which are independent of the network Ψ.

We emphasize that a decomposition of $S_A$ as described in (2.4)–(2.8) does not necessarily require a uniform operator ${\mathfrak{C}}_{\mathrm{red}}$. If multiple reduced operators are required for such a decomposition, the idea of approximating them by one single neural network can still be applied. It is, however, necessary for the ability of the network to generalize well beyond data seen during training that the reduced operators at least involve certain similarities.

Network training

In practice, the neural network $\mathrm{\Psi }(\cdot ,\theta )$ has to be trained in an offline phase from a set of training examples before it can be used for approximating the mapping ${\mathfrak{C}}_{\mathrm{red}}$. We propose to draw N global coefficients ${(A^{(i)})}_{i=1}^N$ from $\mathfrak{A}$, extracting the relevant information $(A_j^{(i)}):=(R_j(A^{(i)}))$ from them and compressing it into the corresponding effective matrices $(S_{A,j}^{(i)})$ with ${\mathfrak{C}}_{\text{red}}$. This results in a total of $|J|\cdot N$ training samples available for the neural network to train on, namely $(A_j^{(i)},S_{A,j}^{(i)}),i=1,\dots N$, $j\in J$. In order to learn the parameters of the network, we then minimize the loss functional

$$\mathcal{J}(\theta )=\frac{1}{N\cdot |J|}\sum _{i=1}^N\sum _{j\in J}\frac{1}{2}\frac{{\parallel \mathrm{\Psi }(A_j^{(i)},\theta )-S_{A,j}^{(i)}\parallel }_{{\mathbb{R}}^{s\times t}}^2}{{\parallel S_{A,j}^{(i)}\parallel }_{{\mathbb{R}}^{s\times t}}^2}$$ 2.9

over the parameter space ${\mathbb{R}}^p$ using iterative gradient-based optimization on minibatches of the training data. This can be very efficiently implemented within modern deep learning frameworks such as TensorFlow [1], PyTorch [55], or Flux [40], which allow for the automatic differentiation of the loss functional with respect to the network parameters.

Full algorithm

After having established all the conceptual pieces, we now put them together and return to the abstract variational problem (2.2) from the beginning of the section. Suppose that we want to solve (2.2) for a large number of given coefficients $A^{(i)},i=1,\dots ,M$, and a given right-hand side $f\in H^{-1}(D)$. For ease of notation, we restrict ourselves to a single right-hand side, which is, however, not necessarily required for our approach. The proposed procedure is summarized in Algorithm 1, divided into the offline and online stages of the method.

Algorithm 1.

Operator compression with neural network

Application to elliptic homogenization

In this section, we specifically consider a family of prototypical elliptic diffusion operators as a demonstrating example of how to apply the abstract framework laid down in Sect. 2 in practice.

Setting

From now on let the domain D be polyhedral. We consider the family of linear second-order diffusion operators

$$\mathfrak{L}:=\{-div(A\mathrm{\nabla }\cdot ):H_0^1(D)\to H^{-1}(D)|A\in \mathfrak{A}\},$$

parameterized by the following set of admissible coefficients which possibly encode microstructures:

$$\mathfrak{A}:=\{A\in L^{\mathrm{\infty }}(D)|\mathrm{\exists }0<\alpha \le \beta <\mathrm{\infty }:\alpha \le A(x)\le \beta \text{for almost all }x\in D\}.$$ 3.1

For the sake of simplicity, we restrict ourselves to scalar coefficients here. Note, however, that also the consideration of matrix-valued coefficients is not an issue from a numerical homogenization viewpoint. We remark that the family of operators $\mathfrak{L}$ fulfills the assumptions of locality and symmetry from the abstract framework. In this setting, the abstract problem (2.1) amounts to solving the following linear elliptic PDE with homogeneous Dirichlet boundary condition:

$$\{\begin{array}{cc}-div(A\mathrm{\nabla }u)=f\hfill & \text{in }D,\hfill \\ u=0\hfill & \text{on }\partial D,\hfill \end{array}$$

which possesses a unique weak solution $u\in H_0^1(D)$ for every $f\in H^{-1}(D)$ and $A\in \mathfrak{A}$. The corresponding counterpart to the weak formulation (2.2) can be written as: find $u\in H_0^1(D)$ such that

$$a_A(u,v):={\int }_{D}A\mathrm{\nabla }u\cdot \mathrm{\nabla }v\mathrm{d}x=\langle f,v\rangle \text{for all }v\in H_0^1(D)$$ 3.2

by using integration by parts on the divergence term.

Discretization and compression

Let now ${\mathcal{T}}_h$ be a Cartesian mesh with characteristic mesh size h and denote with $Q^1({\mathcal{T}}_h)$ the corresponding space of piecewise bilinear functions. We consider the conforming finite element space $V_h:=Q^1({\mathcal{T}}_h)\cap H_0^1(D)$ of dimension $m:=\text{dim}(V_h)$. Generally, also other types of meshes and finite element spaces could be employed, but we restrict ourselves to the above choice for the moment. As we have already mentioned in Sect. 2.2, if the mesh ${\mathcal{T}}_h$ does not resolve the fine-scale oscillations of A, approximating u with a pure finite element ansatz of seeking $u_h\in V_h$ such that

$$a_A(u_h,v_h)=\langle f,v_h\rangle \text{for all }{v}_h\in V_h$$

will not yield satisfactory results. In a setting where resolving A with the mesh is computationally too demanding, we are therefore interested in suitable choices for a compression operator $\mathfrak{C}$. In particular, we want $\mathfrak{C}$ to produce effective system matrices on the target scale h that can be used to obtain appropriate approximations on this scale. In the following, we briefly comment on possible choices for this operator that are based on the finite element space $V_h$.

Compression by analytical homogenization

The idea of analytical homogenization is to replace an oscillating A with an appropriate homogenized coefficient $A_{\mathrm{hom}}\in L^{\mathrm{\infty }}(D,{\mathbb{R}}^{d\times d})$. The mathematical theory of homogenization can treat very general nonperiodic coefficients in the framework of G- or H-convergence [14, 51, 64]. However, apart from being nonconstructive in many cases, homogenization in the classical analytical sense considers a sequence of operators $-div(A_{\epsilon }\mathrm{\nabla }\cdot )$ indexed by $\epsilon >0$ and aims to characterize the limit as ε tends to zero. In many realistic applications, such a sequence of models can hardly be identified or may not be available in the first place. Assuming that the necessary requirements on the coefficient A are met, a homogenized coefficient $A_{\mathrm{hom}}$ exists and does not involve oscillations on a fine scale. The coefficient $A_{\mathrm{hom}}$ can then be used in combination with a classical finite element ansatz, since $A_{\mathrm{hom}}$ does no longer include troublesome fine-scale quantities. In practice, the homogenized coefficients cannot be computed easily and need to be approximated. This is, for instance, done with the heterogeneous multiscale method (HMM) [2, 15, 16], which in the end replaces A with a computable approximation $A_h\in L^{\mathrm{\infty }}(D,{\mathbb{R}}_{\text{sym}}^{d\times d})$ of $A_{\mathrm{hom}}$ with $(A_h){|}_T\in {\mathbb{R}}_{\text{sym}}^{d\times d}$ for all $T\in {\mathcal{T}}_h$. With this piecewise constant approximation of A, we obtain a possible compression operator $\mathfrak{C}$. Given an enumeration $1,\dots ,m$ of the inner nodes in ${\mathcal{T}}_h$ and writing ${\lambda }_1,\dots ,{\lambda }_m$ for the associated nodal basis of $V_h$, the compressed operator $\mathfrak{C}(A)$ can be defined as

$${(\mathfrak{C}(A))}_{i,j}={(S_A)}_{i,j}:=\sum _{T\in {\mathcal{T}}_h}{\int }_T((A_h){|}_T\mathrm{\nabla }{\lambda }_j)\cdot \mathrm{\nabla }{\lambda }_i\mathrm{d}x.$$ 3.3

That is, one takes the classical finite element stiffness matrix corresponding to the homogenized coefficient $A_{\mathrm{hom}}$ as an effective system matrix. In this case, decomposition (2.4) corresponds to a partition into element-wise stiffness matrices (with constant coefficient, respectively) that are merged with a simple finite element assembly routine.

We emphasize that approaches based on analytical homogenization – such as (3.3) – are able to provide reasonable approximations on the target scale h but are subject to structural assumptions, in particular scale separation and local periodicity. The goal to overcome these restrictions has led to a new class of numerical methods that are specifically tailored to treating general coefficients with minimal assumptions. These methods are known as numerical homogenization approaches and typically only require a boundedness condition as in (3.1).

Compression by numerical homogenization

The general idea of numerical homogenization methods is to replace the trial space $V_h$ with a suitable multiscale space ${\tilde{V}}_h$, see for instance the references [5, 7, 21, 38, 46, 52, 53]. One possible construction uses a one-to-one correspondence of ${\tilde{V}}_h$ to the space $V_h$, which implies that the two spaces possess the same number of degrees of freedom. Typically, the multiscale space is chosen in a problem-adapted way. We indicate this dependence by defining the new space ${\tilde{V}}_h:={\mathcal{P}}_A{V}_h$, where ${\mathcal{P}}_A:V_h\to H_0^1(D)$ particularly depends on A. Therefore, another possible choice of the operator $\mathfrak{C}$ leads to the effective matrix $\mathfrak{C}(A)$ given by

$${(\mathfrak{C}(A))}_{i,j}={(S_A)}_{i,j}:=a_A({\mathcal{P}}_A{\lambda }_j,{\lambda }_i).$$ 3.4

A prominent example for such an approach – and, thus, the operator $\mathfrak{C}$ – is the Petrov–Galerkin version of the localized orthogonal decomposition (LOD) method which explicitly constructs a suitable operator ${\mathcal{P}}_A$. The LOD was introduced in [46] and theoretically and practically works for very general coefficients. It has also been successfully applied to other problem classes, for instance, wave propagation problems in the context of Helmholtz and Maxwell equations [26, 27, 45, 57, 61] or the wave equation [4, 29, 44, 59], eigenvalue problems [47, 48], and in connection with time-dependent nonlinear Schrödinger equations [37]. However, it requires a slight deviation from locality. That is, while the classical finite element method and the HMM result in a system matrix that only includes neighbor-to-neighbor communication between the degrees of freedom, the multiscale approach (3.4) moderately increases this communication to effectively incorporate the fine-scale information in A for a broader range of coefficients, which is a common property of modern homogenization techniques. As indicated in [12], this slightly increased communication indeed seems to be necessary to handle very general coefficients.

Since we consider a class $\mathfrak{A}$ of arbitrarily rough coefficients, the compression operator (3.4) corresponding to the operator ${\mathcal{P}}_A$ constructed in the LOD method is a suitable choice for our discussion as well as for the numerical experiments of Sect. 4. In the following subsection, we therefore have a closer look into its construction and summarize some main results. Note that we restrict ourselves to an elliptic model problem with homogeneous Dirichlet boundary conditions, but the compression approach can generally be extended to more involved settings such as the ones mentioned above.

Localized orthogonal decomposition

The method is based on a projective quasi-interpolation operator ${\mathcal{I}}_h:H_0^1(D)\to V_h$ with the following approximation and stability properties: for an element $T\in {\mathcal{T}}_h$, we require that

$${\parallel h^{-1}(v-{\mathcal{I}}_{h}v)\parallel }_{L^2(T)}+{\parallel \mathrm{\nabla }{\mathcal{I}}_{h}v\parallel }_{L^2(T)}\le C{\parallel \mathrm{\nabla }v\parallel }_{L^2(\mathtt{N}(T))}$$

for all $v\in H_0^1(D)$, where the constant C is independent of h, and $\mathtt{N}(S):={\mathtt{N}}^1(S)$ is the neighborhood (of order 1) of $S\subseteq D$ defined by

$${\mathtt{N}}^1(S):=\bigcup \{\bar{K}\in {\mathcal{T}}_h|\bar{S}\cap \bar{K}\ne \mathrm{\varnothing }\}.$$

For a particular choice of ${\mathcal{I}}_h$, we refer to [24].

For given ${\mathcal{I}}_h$ with the above properties, we can define the so-called fine-scale space $\mathcal{W}$, which contains all functions that are not well captured by the finite element functions in $V_h$. It is defined as the kernel of ${\mathcal{I}}_h$ with respect to $H_0^1(D)$, i.e.,

$$\mathcal{W}:=ker{\mathcal{I}}_h{|}_{H_0^1(D)},$$

and its local version, for any $S\subseteq D$, is given by

$$\mathcal{W}(S):=\{w\in \mathcal{W}|\mathrm{supp}(w)\subseteq S\}.$$

In order to incorporate fine-scale information contained in the coefficient A, the idea is now to compute coefficient-dependent local corrections of functions $v_h\in V_h$. To this end, we define the neighborhood of order $\ell \in \mathbb{N}$ iteratively by ${\mathtt{N}}^{\ell }(S):=\mathtt{N}({\mathtt{N}}^{\ell -1}(S))$, $\ell \ge 2$. For any function $v_h\in V_h$, its element corrector ${\mathcal{Q}}_{A,T}^{\ell }{v}_h\in \mathcal{W}({\mathtt{N}}^{\ell }(T))$, $T\in {\mathcal{T}}_h$, is defined by

$$a_A({\mathcal{Q}}_{A,T}^{\ell }{v}_h,w)={\int }_{T}A\mathrm{\nabla }{v}_h\cdot \mathrm{\nabla }w\mathrm{d}x\text{for all }w\in \mathcal{W}({\mathtt{N}}^{\ell }(T)).$$ 3.5

Note that in an implementation, the element corrections ${\mathcal{Q}}_{A,T}^{\ell }$ have to be computed on a sufficiently fine mesh that resolves the oscillations of the coefficient A. Since the algebraic realization of the correctors and guidelines for an efficient implementation of the LOD method in general are not within the scope of the article, we refer to [23] for details. We emphasize that, by construction, the supports of the correctors ${\mathcal{Q}}_{A,T}^{\ell }{v}_h$ are limited to ${\mathtt{N}}^{\ell }(T)$. The global correction ${\mathcal{Q}}_A^{\ell }:V_h\to \mathcal{W}$ then consists of a summation of these local contributions and is given by

$${\mathcal{Q}}_A^{\ell }:=\sum _{T\in {\mathcal{T}}_h}{\mathcal{Q}}_{A,T}^{\ell }.$$

Note that the choice $\ell =\mathrm{\infty }$ corresponds to a computation of the element corrections on the entire domain D and leads to the orthogonality property

$$a_A((1-{\mathcal{Q}}_A^{\mathrm{\infty }})v_h,w)=0\text{for all }w\in \mathcal{W},$$ 3.6

that defines an $a_A$-orthogonal splitting $H_0^1(D)=(1-{\mathcal{Q}}_A^{\mathrm{\infty }})V_h\oplus \mathcal{W}$. This particularly explains the name orthogonal decomposition. The use of localized element corrections is motivated by the decay of the corrections ${\mathcal{Q}}_{A,T}^{\mathrm{\infty }}$ away from the element T. This is, for instance, shown in [36, 56] (based on [46]) and reads

$${\parallel \mathrm{\nabla }({\mathcal{Q}}_A^{\mathrm{\infty }}-{\mathcal{Q}}_A^{\ell })v_h\parallel }_{L^2(D)}\le C{e}^{-c_{\mathrm{dec}}\ell }{\parallel \mathrm{\nabla }{v}_h\parallel }_{L^2(D)}$$

with a constant $c_{\mathrm{dec}}$ which is independent of h and ℓ.

Motivated by decomposition (3.6) and the localized approximations in (3.5), we choose ${\mathcal{P}}_A:=1-{\mathcal{Q}}_A^{\ell }$ in (3.4). The space ${\tilde{V}}_h:={\mathcal{P}}_A{V}_h=(1-{\mathcal{Q}}_A^{\ell })V_h$, which has the same number of degrees of freedom as $V_h$, can then be used as an ansatz space for the discretization of (3.2). Note that the original LOD method introduced in [46] considers a discretization where ${\tilde{V}}_h$ is also used as test space. We, however, consider the Petrov–Galerkin variant of the method as analyzed in [22] that uses the classical finite element space $V_h$ as test space instead, i.e., we seek $u_h\in V_h$ such that

$$a_A((1-{\mathcal{Q}}_A^{\ell })u_h,v_h)=\langle f_h,v_h\rangle \text{for all }{v}_h\in V_h,$$ 3.7

where $f_h=\mathfrak{M}f\in V_h$ is again a suitable approximation of $f\in H^{-1}(D)$. This defines a compressed operator ${\mathfrak{S}}_A$ as in (2.3) that maps $u_h\in V_h$ to $f_h\in V_h$. As it turns out, the Petrov–Galerkin formulation has some computational advantages over the classical method, in particular in terms of memory requirement. For details, we again refer to [23]. The theory in [22] shows that the approximation $u_h$ defined in (3.7) is first-order accurate in $L^2(D)$ provided that $\ell \gtrsim |logh|$ and, additionally, $f\in L^2(D)$. More precisely, it holds

$$\begin{array}{rl}{\parallel {(-div(A\mathrm{\nabla }))}^{-1}-{\mathfrak{S}}_A^{-1}\parallel }_{L^2(D)\to L^2(D)}& =\underset{f\in L^2(D)}{sup}\frac{{\parallel {(-div(A\mathrm{\nabla }))}^{-1}(f)-{\mathfrak{S}}_A^{-1}(\mathfrak{M}f)\parallel }_{L^2(D)}}{{\parallel f\parallel }_{L^2(D)}}\\ & \lesssim h+e^{-c_{\mathrm{dec}}\ell },\end{array}$$

where $\mathfrak{M}f$ denotes the $L^2$-projection of f onto $V_h$. Note that the methodology can actually be applied to more general settings beyond the elliptic case, see for instance [49] for an overview.

System matrix surrogate

We now return to the discussion of the compression operator $\mathfrak{C}$ introduced in (3.4) that maps coefficients $A\in \mathfrak{A}$ to the effective system matrices

$${(S_A)}_{i,j}:=a_A((1-{\mathcal{Q}}_A^{\ell }){\lambda }_j,{\lambda }_i)$$

obtained from the Petrov–Galerkin LOD method. Once $S_A$ has been computed for a given A, an approximation $u_h={\sum }_{j=1}^m{U}_j{\lambda }_j$ can be computed by solving the following linear system for the coefficients $U={(U_1,\dots ,U_m)}^T$:

$$S_{A}U=F,$$

where $F:={(\langle \mathfrak{M}f,{\lambda }_1\rangle ,\dots ,\langle \mathfrak{M}f,{\lambda }_m\rangle )}^T$. Since $u_h$ is equivalently characterized by the solution of (3.7), it captures the effective behavior of the solution to the continuous problem (3.2) on the target scale h as discussed in the previous subsection.

The remainder of this section is dedicated to showing how the abstract decomposition (2.4) translates to the present LOD setting and how it can be implemented in practice. Writing $\mathcal{N}(S)$ for the set of inner mesh nodes on some subdomain $S\subseteq D$ and denoting $N_S=|\mathcal{N}(S)|$, the effective system matrix $S_A$ can be decomposed as

$$S_A=\sum _{T\in {\mathcal{T}}_h}{\mathrm{\Phi }}_T(S_{A,T}),$$ 3.8

where the matrices $S_{A,T}$ are local system matrices of the form

$${(S_{A,T})}_{i,j}={\int }_{{\mathtt{N}}^{\ell }(T)}A\mathrm{\nabla }(1-{\mathcal{Q}}_{A,T}^{\ell }){\lambda }_j\cdot \mathrm{\nabla }{\lambda }_i\mathrm{d}x,j\in \mathcal{N}(T),i\in \mathcal{N}({\mathtt{N}}^{\ell }(T)),$$ 3.9

i.e., they correspond to the interaction of the localized ansatz functions $(1-{\mathcal{Q}}_{A,T}^{\ell }){\lambda }_j$ associated with the nodes of the element T with the classical first order nodal basis functions whose supports overlap with the element neighborhood ${\mathtt{N}}^{\ell }(T)$. This means that $S_{A,T}$ is an $N_{{\mathtt{N}}^{\ell }(T)}\times N_T$ matrix. In practice, the coefficient A in (3.9) is often replaced with an element-wise constant approximation $A_{\epsilon }$ on a finer mesh ${\mathcal{T}}_{\epsilon }$ that resolves all the oscillations of A and that we assume to be a uniform refinement of ${\mathcal{T}}_h$.

As already explained in the abstract framework, the mappings ${\mathrm{\Phi }}_T$ in (3.8) are local-to-global mappings that assemble the contributions $S_{A,T}$ on an element neighborhood to a global matrix. In particular, given an enumeration $1,\dots ,N_{{\mathtt{N}}^{\ell }(T)}$ of the nodes in $\mathcal{N}({\mathtt{N}}^{\ell }(T))$, one considers a mapping $g_T(\cdot )$ that assigns to a given node index i in the element neighborhood ${\mathtt{N}}^{\ell }(T)$ its global node index $g_T(i)\in \{1,\dots ,m\}$. This mapping can be represented by an $m\times N_{{\mathtt{N}}^{\ell }(T)}$ sparse matrix with entries

$${\pi }_T[i,j]=\{\begin{array}{cc}1,\hfill & \text{if }i=g_T(j),\hfill \\ 0,\hfill & \text{otherwise.}\hfill \end{array}$$

Analogously, given an enumeration $1,\dots ,N_T$ of the nodes in $\mathcal{N}(T)$, there exists a mapping ${\tilde{g}}_T(\cdot )$ – represented by an $m\times N_T$-matrix – that assigns to a given node in $\mathcal{N}(T)$ with index i its global representative with index ${\tilde{g}}_T(i)$. The corresponding matrix is given by

$${\phi }_T[i,j]=\{\begin{array}{cc}1,\hfill & \text{if }i={\tilde{g}}_T(j),\hfill \\ 0,\hfill & \text{otherwise.}\hfill \end{array}$$

Using these matrices, decomposition (3.8) reads

$$S_A=\sum _{T\in {\mathcal{T}}_h}{\pi }_T{S}_{A,T}{\phi }_T^{\prime },$$ 3.10

where ${\phi }_T^{\prime }$ denotes the transpose of the matrix ${\phi }_T$.

From the definition of the local contributions $S_{A,T}$ introduced in (3.9), it directly follows that $S_{A,T}$ does only depend on the restriction of A, respectively $A_{\epsilon }$, to the element neighborhood ${\mathtt{N}}^{\ell }(T)$. Let now ${\mathcal{T}}_{\epsilon }({\mathtt{N}}^{\ell }(T))$ be the restriction of the mesh ${\mathcal{T}}_{\epsilon }$ to ${\mathtt{N}}^{\ell }(T)$, consisting of $r=|{\mathcal{T}}_{\epsilon }({\mathtt{N}}^{\ell }(T))|$ elements. Enumerating the elements then leads to the following operators that correspond to the abstract reduction operators in (2.8):

$$R_T:\mathfrak{A}\to {\mathbb{R}}^r$$ 3.11

that map a global coefficient A to a vector that contains the values of $A_{\epsilon }$ in the respective cells of ${\mathcal{T}}_{\epsilon }({\mathtt{N}}^{\ell }(T))$.

As already mentioned in the abstract section above, we aim for a uniform output size of the operators $R_T$, since the outputs of the operators $R_T$ will later on be fed into a neural network with a fixed number of input neurons. In order to achieve that, we artificially extend the domain D and the mesh ${\mathcal{T}}_h$ by ℓ layers of outer elements around the boundary elements of ${\mathcal{T}}_h$, thus ensuring that the element neighborhood ${\mathtt{N}}^{\ell }(T)$ always consists of the same number of elements regardless of the respective location of the central element $T\in {\mathcal{T}}_h$ relative to the boundary. Further, we extend the piecewise constant coefficient $A_{\epsilon }$ by zero on those outer elements. Figure 1 illustrates this procedure for the case $d=2$ and $\ell =1$ for an element T that lies in a corner of the computational domain. In this figure, ${\mathcal{T}}_h$ is a uniform quadrilateral mesh on the domain D and ${\mathcal{T}}_{\epsilon }$ is obtained from ${\mathcal{T}}_h$ by a single uniform refinement step. The asterisks indicate the coefficient $A_{\epsilon }$ taking a regular value in the interval $[\alpha ,\beta ]$, whereas in the cells outside of D, we set $A_{\epsilon }$ to zero.

Note that this enlargement of the mesh ${\mathcal{T}}_h$ to obtain equally sized element neighborhoods ${\mathtt{N}}^{\ell }(T)$ also introduces artificial mesh nodes that lie outside of D and that are all formally considered as inner nodes for the definition of $N_S=|\mathcal{N}(S)|$ with a subset S in the extended domain. This implies that the local system matrices $S_{A,T}$ of dimension $N_{{\mathtt{N}}^{\ell }(T)}\times N_T$ introduced in (3.9) are all of equal size as well and the rows of $S_{A,T}$ corresponding to test functions associated with nodes that are attached to outer elements contain only zeros. During the assembly process of the local contributions to a global matrix, these zero rows are disregarded (which is also consistent with our definition of the matrices ${\pi }_T$, ${\phi }_T$).

Finally, in order to unify the computation of local contributions, we use an abstract mapping ${\mathfrak{C}}_{\text{red}}$ with fixed input dimension r and fixed output dimension $N_{{\mathtt{N}}^{\ell }(T)}\times 2^d$ as proposed for the abstract framework in Sect. 2.5. The mapping takes the restriction of $A_{\epsilon }$ to an element neighborhood ${\mathtt{N}}^{\ell }(T)$ as input data and outputs the corresponding approximation of a local effective matrix $S_{A,T}$ that will be determined by an underlying neural network $\mathrm{\Psi }(\cdot ,\theta )$.

Numerical experiments

In this section, we demonstrate the feasibility of our proposed approach by performing numerical experiments in the setting of Sect. 3. For all experiments, we consider the two-dimensional computational domain $D={(0,1)}^2$, which we discretize with a uniform quadrilateral mesh ${\mathcal{T}}_h$ of characteristic mesh size $h=2^{-5}$. The coefficients are allowed to vary on the finer unresolved scale $\epsilon =2^{-8}$.

Coefficient family

In order to test our method’s ability to deal with coefficients that show oscillating behavior across multiple scales, we introduce a hierarchy of meshes ${\mathcal{T}}^k,k=0,1,\dots ,8$, where the initial mesh ${\mathcal{T}}^0$ consists only of a single element, and the subsequent meshes are obtained by uniform refinement, i.e., ${\mathcal{T}}^k$ is obtained from ${\mathcal{T}}^{k-1}$ by subdiving each element of ${\mathcal{T}}^{k-1}$ into four equally sized elements. This implies that the characteristic mesh size of ${\mathcal{T}}^k$ is given by $2^{-k}$. In the following, we refer to the parameter k as the mesh level. In particular, the computational mesh ${\mathcal{T}}_h={\mathcal{T}}^5$ corresponds to the mesh level 5, whereas the coefficients may vary on the mesh level 8 and are therefore only resolved by the finest mesh ${\mathcal{T}}^8$. We thus have a scenario where an information gap of 3 mesh levels has to be bridged. Based on the mesh hierarchy, we now define the coefficient family $\mathfrak{A}$ of interest. Let ${\mathfrak{A}}_k$ denote the set of element-wise constant coefficients on ${\mathcal{T}}^k$ whose values in the respective cells are iid uniformly distributed on the interval $[\alpha ,\beta ]:=[1,5]$, i.e.,

$${\mathfrak{A}}_k:=\{A\in Q^0\left({\mathcal{T}}^k\right)|A{|}_T\stackrel{\text{iid}}{\sim }U([1,5])\text{ for all }T\in {\mathcal{T}}^k\}.$$

Furthermore, let ${\mathfrak{A}}_{\text{ms}}$ denote the set of coefficients of the form

$$A=\frac{1}{9}\sum _{k=0}^8{A}_k,A_k\in {\mathfrak{A}}_k.$$

These multiscale coefficients are especially interesting, since they vary on all considered scales simultaneously and are therefore very hard to deal with using classical homogenization techniques due to their unstructured nature. The total set of interest $\mathfrak{A}$ is then defined as

$$\mathfrak{A}:=\bigcup _{k=0}^8{\mathfrak{A}}_k\cup {\mathfrak{A}}_{\text{ms}}.$$

In the following, we will frequently index coefficients sampled from $\mathfrak{A}$ by their corresponding level, i.e., write $A_k,k\in \{0,\dots ,8,\text{ms}\}$ instead of a plain A.

Data generation and preprocessing

In order to train the network, we sample 500 coefficients $A_k^{(i)},i=1,\dots ,500$, from each ${\mathfrak{A}}_k,k\in \{0,\dots ,8,\text{ms}\}$, where the individual samples $A_k^{(i)}$ on the coarser mesh levels $k=0,\dots ,7$ are prolongated to the finest mesh ${\mathcal{T}}^8$ in order to achieve a uniform dimension across all scales. The set of all sample coefficients is subsequently divided into a training, validation, and test set according to a 80–10–10 split. In order to achieve an identical distribution in all three sets, the splitting is performed separately on every level (including ms), i.e., for every $k\in \{0,\dots ,8,\text{ms}\}$, the first 400 coefficients $A_k^{(i)},i=1,\dots ,400$, get assigned to the training set ${\mathcal{D}}_{\text{train}}$, the samples with indices $401,\dots ,450$ to the validation set ${\mathcal{D}}_{\text{val}}$, and those with indices $451,\dots ,500$ to the test set ${\mathcal{D}}_{\text{test}}$. Then, we individually split each sample $A_k^{(i)}$, using the reduction operators $R_T$ introduced in (3.11), into sub-samples $A_{k,T}^{(i)}$ based on element neighborhoods ${\mathtt{N}}^{\ell }(T)$ for $\ell =2$ that are centered around the elements $T\in {\mathcal{T}}_h$, also taking into account the artificial extension of the element neighborhoods around the boundary of D. Since our target scale of interest is $h=2^{-5}$ and ${\mathcal{T}}_h$ is a uniform quadrilateral mesh, this yields 1024 sub-samples $A_{k,T}^{(i)}\in {\mathbb{R}}^{1600}$ per sample $A_k^{(i)}\in {\mathbb{R}}^{65\text{,}536}$. Note that the size of the sub-samples is obtained from the construction of the local neighborhoods ${\mathtt{N}}^{\ell }(T)$. Here, each neighborhood consists of ${(2\ell +1)}^2=25$ elements in ${\mathcal{T}}_h={\mathcal{T}}^5$, which corresponds to $64\cdot 25=1600$ elements in the mesh ${\mathcal{T}}_{\epsilon }={\mathcal{T}}^8$.

The corresponding “labels”, i.e., the local effective system matrices $S_{A,k,T}^{(i)}\in {\mathbb{R}}^{36\times 4}$, are then computed with the Petrov–Galerkin LOD according to (3.9) and flattened column-wise to vectors in ${\mathbb{R}}^{144}$. In total, we obtain $10\cdot 400\cdot 1024=4\text{,}096\text{,}000$ pairs $(A_{k,T}^{(i)},S_{A,k,T}^{(i)})\in {\mathcal{D}}_{\text{train}}$ to train our network with, and $512\text{,}000$ pairs in ${\mathcal{D}}_{\text{val}}$ and ${\mathcal{D}}_{\text{test}}$ each.

Network architecture and training

Given the above dataset, we now try to fit it with a suitable neural network Ψ. As network architecture, we consider a dense feedforward network with a total of eight layers including the input and output layer. As activation function, we choose the standard ReLU activation given by $\rho (x):=max(0,x)$ in the first seven layers and the identity function in the last layer. By convention, the activation function acts component-wise on vectors. The network output is thus of the form

$$\mathrm{\Psi }(x)=W^{(8)}\rho (W^{(7)}(\dots \rho (W^{(2)}\rho (W^{(1)}x+b^{(1)})+b^{(2)})\dots )+b^{(7)})+b^{(8)},$$ 4.1

where the weight matrices and bias vectors have the following dimensions:

$$\begin{array}{rl}& W^{(1)}\in {\mathbb{R}}^{1600\times 1600},W^{(2)}\in {\mathbb{R}}^{800\times 1600},W^{(3)}\in {\mathbb{R}}^{800\times 800},W^{(4)}\in {\mathbb{R}}^{400\times 800},\\ & W^{(5)}\in {\mathbb{R}}^{400\times 400},W^{(6)}\in {\mathbb{R}}^{144\times 400},W^{(7)}\in {\mathbb{R}}^{144\times 144},W^{(8)}\in {\mathbb{R}}^{144\times 144},\\ & b^{(1)}\in {\mathbb{R}}^{1600},b^{(2)},\in {\mathbb{R}}^{800},b^{(3)}\in {\mathbb{R}}^{800},b^{(4)}\in {\mathbb{R}}^{400},\\ & b^{(5)}\in {\mathbb{R}}^{400},b^{(6)}\in {\mathbb{R}}^{144},b^{(7)}\in {\mathbb{R}}^{144},b^{(8)}\in {\mathbb{R}}^{144},\end{array}$$

yielding a total of $5\text{,}063\text{,}504$ trainable parameters. The idea behind this architecture is that in the first six layers, information about the coefficient in the input element neighborhood is gathered and combined by allowing communication between all inputs in layers with odd indices, whereas in the layers with even indices, this information is repeatedly compressed. That is, every other layer is built in such a way that the input and output dimension are equal. If the neurons in that layer are understood as some sort of degrees of freedom in a mesh, this refers to having communication among all of these degrees of freedoms, while the layers in between reduce the number of degrees of freedom, which can be interpreted as transferring information to a coarser mesh. In the last two layers, this compressed information is taken and assembled to the local effective system matrix. Note that this logarithmic dependence of the number of layers on the number of scales that need to be bridged by the network (two layers per mesh level to be bridged plus two layers to assemble the local effective matrix) yielded the most reliable results in our experiments. Shallower networks had difficulties fitting the complex training set consisting of coefficients varying on different scales, whereas deeper networks were more prone to overfitting the training set. More involved architectures, for example the ones that include skip connections between layers like in the classic ResNet [34], are also conceivable; however, this seems not to be necessary to obtain good results. The key message here is that the coefficient-to-surrogate map can be satisfyingly approximated by a simple feedforward architecture, whose size does depend only on the scales ε and h, but not on any finer discretization scales. The implementation of the network as well as the training is performed using the library Flux [40] for the open-source scientific computing language Julia [10].

After initializing all parameters in the network according to a Glorot uniform distribution [32], network (4.1) is trained on minibatches of 1000 samples for a total of 20 epochs on ${\mathcal{D}}_{\text{train}}$, using the ADAM optimizer [42] with a step size of 10⁻⁴ for the first 5 epochs before reducing it to 10⁻⁵ for the subsequent 15 epochs. It could be observed that further training led to a stagnation of the validation error, whereas the error on the training set continued to decrease (very slowly but gradually), indicating overfitting of the network. The development of the loss functional $\mathcal{J}$ defined in (2.9) over the epochs is shown in Fig. 2. Note that training and validation loss stay very close to each other during the whole training process since ${\mathcal{D}}_{\text{train}}$ and ${\mathcal{D}}_{\text{val}}$ have the same sample distribution due to our chosen splitting procedure. The development of the loss during the training and an average loss of $7.78\cdot {10}^{-5}$ on the test set ${\mathcal{D}}_{\text{test}}$ indicates that the network has at least learned to approximate the local effective system matrices. In applications, however, we are mostly concerned about how well this translates to the global level, when computing solutions to problem (3.2) using a global system matrix assembled from network outputs. In order to investigate this question, the next three subsections are dedicated to evaluating the performance of the trained network at exactly this task for several coefficients unseen during training. For a given right-hand side f and coefficient A, we denote with $u_h$ the solution of (3.7), obtained with the Petrov–Galerkin LOD matrix $S_A$ defined in (3.10), and with ${\hat{u}}_h$ the solution obtained by using the neural network approximation of this matrix, i.e.,

$${\hat{S}}_A:=\sum _{T\in {\mathcal{T}}_h}{\pi }_T\mathrm{\Psi }(R_T(A)){\phi }_T^{\prime }.$$ 4.2

Figure 2.

Development of loss functional $\mathcal{J}$ during 20 epochs of the network training

The spectral norm difference ${\parallel S_A-{\hat{S}}_A\parallel }_2$, the $L^2$-error ${\parallel u_h-{\hat{u}}_h\parallel }_{L^2(D)}$ as well as the visual discrepancy between the two solutions are then considered as a measure of the network’s global performance. We emphasize once more that computing approximate surrogates via (4.2) is significantly faster compared to (3.8) and (3.9). This is due to the fact that no corrector problems of the form (3.5) have to be solved to obtain the surrogate model. As pointed out, the solution of these local problems requires inversion on a very fine discretization scale that is significantly smaller than the scale ε on which the coefficient varies. In order to compute the system matrix $S_A$, one has to solve $N_{{\mathcal{T}}_h}$ fine-scale linear systems, where $N_{{\mathcal{T}}_h}$ denotes the number of elements in ${\mathcal{T}}_h$. In contrast, the main computational effort of evaluating our trained network consists of $N_L$ matrix-matrix multiplications, where $N_L$ is the number of layers in the network (not taking into account bias vectors and the activation function).

Experiment 1

For our first experiment, we consider an unstructured multiscale coefficient sampled from ${\mathfrak{A}}_{\text{ms}}$ that was not a part of the training set and a constant right-hand side $f\equiv 1$. The coefficient (top left), the error $|u_h(x)-{\hat{u}}_h(x)|$ (top right) as well as representative cross-sections along $x_1=0.5$ (bottom left) and $x_2=0.5$ (bottom right) of the two solutions $u_h$ and ${\hat{u}}_h$ are shown in Fig. 3. The spectral norm difference ${\parallel S_A-{\hat{S}}_A\parallel }_2\approx 6.58\cdot {10}^{-2}$ and the $L^2$-error ${\parallel u_h-{\hat{u}}_h\parallel }_{L^2(D)}\approx 2.13\cdot {10}^{-4}$ confirm the visual impression – the network has successfully learned to produce a system matrix that is able to capture the behavior of the solution on the target scale well.

Figure 3.

Results of Experiment 1: Unstructured multiscale coefficient sampled from ${\mathfrak{A}}_{\text{ms}}$ (top left), $|u_h(x)-{\hat{u}}_h(x)|$ (top right), and comparison of $u_h$ vs ${\hat{u}}_h$ along the cross sections $x_1=0.5$ (bottom left) and $x_2=0.5$ (bottom right)

Experiment 2

Next, we test the network’s performance for smoother and more regular coefficients than the ones it has been trained with. As a demonstrating example, we consider the coefficient $A(x)=2+sin(2\pi x_1)sin(2\pi x_2)$. The network’s input is obtained by evaluating A on the midpoints of the mesh ${\mathcal{T}}_{\epsilon }$ on the fine unresolved scale ε. In this example, we choose the function $f(x)=x_1{\chi }_{\{x_1\ge 0.5\}}$ as a right-hand side, where ${\chi }_S$ denotes the characteristic function of the set $S\subseteq D$. The results are shown in Fig. 4. We obtain an even better $L^2$-error of ${\parallel u_h-{\hat{u}}_h\parallel }_{L^2(D)}\approx 7.56\cdot {10}^{-5}$ and a spectral norm difference of ${\parallel S_A-{\hat{S}}_A\parallel }_2\approx 3.91\cdot {10}^{-2}$. A comparison between the solutions at the cross-sections shows that there is almost no discernible visual difference between the LOD-solution and the approximation obtained using our trained network.

Figure 4.

Results of Experiment 2: Smooth coefficient (top left), $|u_h(x)-{\hat{u}}_h(x)|$ (top right), and comparison of $u_h$ vs ${\hat{u}}_h$ along the cross sections $x_1=0.5$ (bottom left) and $x_2=0.5$ (bottom right)

Experiment 3

As a third experiment, we choose another coefficient that possesses an unfamiliar structure not seen by the network during the training phase, this time a less regular one. The coefficient is shown in the top left of Fig. 5. It is composed of a background part (blue region), which is obtained by sampling uniformly and independently on each element of ${\mathcal{T}}_{\epsilon }$ from the interval $[1,2]$, and several “cracks” (yellow regions), in which the coefficient varies uniformly in $[4,5]$. The right-hand side here is $f(x)=cos(2\pi x_1)$. A computation of the $L^2$-error ${\parallel u_h-{\hat{u}}_h\parallel }_{L^2(D)}\approx 2.72\cdot {10}^{-4}$ shows that the overall error is still moderate; a closer visual inspection of the solutions along the cross-sections however reveals more prominent deviations of the neural network approximation to the ground truth. The spectral norm difference ${\parallel S_A-{\hat{S}}_A\parallel }_2\approx 2.81\cdot {10}^{-1}$ is also one order of magnitude larger than in the previous examples. Nevertheless, it might be possible that performing a few corrective training steps including samples of this nature would be sufficient to fix this issue. A thorough investigation of this hypothesis, along with the extension to other coefficient classes is subject of future work.

Figure 5.

Results of Experiment 3: Coefficient with cracks (top left), $|u_h(x)-{\hat{u}}_h(x)|$ (top right), and comparison of $u_h$ vs ${\hat{u}}_h$ along the cross sections $x_1=0.5$ (bottom left) and $x_2=0.5$ (bottom right)

Conclusion and outlook

We proposed an approach to the compression of linear differential operators – parameterized by PDE coefficients that may depend on microscopic quantities that are not resolved by a target discretization scale of interest – to lower-dimensional surrogates that are based on a combination of existing model reduction methods with a data-driven deep learning framework. Our method is motivated by the fact that the computation of the surrogates (represented by effective system matrices) via classical methods is nontrivial and requires significant computational resources in multi-query settings. To overcome this problem, we showed how the compression process can be approximated by a neural network based on given training data that can be generated using existing compression approaches. Importantly, we avoid a global approximation by a neural network and instead first decompose the compression map into local contributions, which can then be approximated by one single unified network. As an example, we studied a class of second-order elliptic diffusion operators. We showed how to approximate the compression map based on the Petrov–Galerkin formulation of the localized orthogonal decomposition method with a neural network. The proposed ansatz has been numerically validated for a large set of piecewise constant and highly oscillatory multiscale coefficients. Furthermore, it has been shown that the approach also generalizes well, in the sense that a well-trained network is able to produce reasonable results even for classes of coefficients that it has not been trained on.

For future research, many possible research questions building on the present work are conceivable. Straightforward extensions would be to consider stochastic settings with differential operators parameterized by random fields or settings with high contrast. Another question to investigate is to what degree the method can be made robust against changes in geometry, for example, by training the network not only on coefficients that are sampled on a fixed domain, but rather on reference patches with varying geometries. Mimicking a hierarchical discretization approach, one may also try to directly approximate the inverse operator which can be represented by a sparse matrix [25]. On a more theoretical level, the approximation properties of neural networks for various existing compression operators could be investigated, along with the question of the number of training samples required to faithfully approximate those for a given family of coefficients.

Authors’ contributions

The work emerged from a close collaboration between the authors. All authors read and approved the final manuscript.

Funding

The work of F. Kröpfl and D. Peterseim is part of the project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 865751 - Computational Random Multiscale Problems). R. Maier acknowledges support by the Göran Gustafsson Foundation for Research in Natural Sciences and Medicine. Open Access funding enabled and organized by Projekt DEAL.

Availability of data and materials

The dataset generated and used for the numerical experiments in this work are available from the corresponding author F. Kröpfl on reasonable request.

Declarations

Competing interests

The authors declare that they have no competing interests.