Artificial Intelligence in Meta-optics

Abstract

Recent years have witnessed promising artificial intelligence (AI) applications in many disciplines, including optics, engineering, medicine, economics, and education. In particular, the synergy of AI and meta-optics has greatly benefited both fields. Meta-optics are advanced flat optics with novel functions and light-manipulation abilities. The optical properties can be engineered with a unique design to meet various optical demands. This review offers comprehensive coverage of meta-optics and artificial intelligence in synergy. After providing an overview of AI and meta-optics, we categorize and discuss the recent developments integrated by these two topics, namely AI for meta-optics and meta-optics for AI. The former describes how to apply AI to the research of meta-optics for design, simulation, optical information analysis, and application. The latter reports the development of the optical Al system and computation via meta-optics. This review will also provide an in-depth discussion of the challenges of this interdisciplinary field and indicate future directions. We expect that this review will inspire researchers in these fields and benefit the next generation of intelligent optical device design.

1. Introduction

Since the invention of computers, an intriguing question to study has been how to let machines mimic human intelligence, which is broadly defined as artificial intelligence (AI). The ambitious goal of allowing machines to achieve the intellectual capability of humans, such as abstract thinking, decision making, adapting to new environments, creativity, and social skills, is usually called general AI. Although this goal has not yet been achieved, the application of AI to tackle specific tasks has achieved great success. Narrow AI-based solutions that target specific tasks can be found in our daily life, such as photo tagging, chatbot-based customer service, product recommendations, etc. More applications of AI in automatic and precise diagnosis, drug design, and early detection of cancer are on the way. The fast development of AI primarily benefits from the acceleration of computational power, including both storage size and processor speed. In particular, the computing power of the graphics processing unit (GPU) has been enhanced to emancipate the capabilities of AI in learning from big data. This hardware support greatly helps bridge the gap between AI theory and application.

It is an excellent time to make AI technology in synergy with interdisciplinary, especially optics-related, fields. Meta-optics is an optical technology that has emerged in recent years. Metasurfaces are advanced flat optical devices that can manipulate light or electromagnetic waves. The metasurface is composed of a specially arranged array of meta-atoms. The artificial meta-atoms are designed and computed to control the phase, amplitude, and polarization of the incident electromagnetic (EM) waves. The new wavefront can be reconstructed by the metasurface. The novel function of the meta-devices can be implemented through several steps such as material selection, meta-atom design, phase distribution design, layout configuration, fabrication, characterization, and application demonstration. These steps have been sorted out and introduced in detail in literature reviews. These reviews all described incisive views on meta-optics, which are fabrication methods^1,2 developed from principle to applications³⁻¹⁰ for quantum optics¹¹ and cutting-edge optical devices.^12,13

This review first offers an overview of meta-optics and AI cooperation, as shown in Figure 1. The recent development via these two topics in synergy is discussed. Section 2 introduces the fundamental concepts of meta-optics and AI technology. We describe two categories: AI works for meta-optics and meta-optics works for AI. Section 3, “AI for Meta-optics”, describes how to apply AI to the research of meta-optics from design to application. Section 4, “Meta-optics for AI”, describes how to use meta-optics to realize optical AI operations and AI systems. Section 5 concludes this review by providing an in-depth discussion of the challenges and future directions of the interdisciplinary field.

Figure 1.

Overview of artificial intelligence in meta-optics.

Figure 2 shows the development trend of AI and meta-optics. Coincidentally, both areas started to grow rapidly in about 2012. An orders-of-magnitude speed-up in deep learning algorithms enabled by GPUs significantly accelerated the development of AI.¹⁴ The historical event of AlexNet winning the ImageNet competition in 2012 marked a new era of AI.¹⁵ A little earlier, the innovative concept of metasurface design was proposed in October 2011.¹⁶ This is the starting point of using a 2D structured device to control light in meta-optics. Each meta-atom of the metasurface can be individually designed to provide a special optical modulation. The new wavefront can be engineered by phase profile tailoring. From this point in time, both communities witnessed a series of major breakthroughs in computational models or applications. In 2014, DeepFace achieved over 97% accuracy for facial recognition.¹⁷ Later, in February 2015, meta-holograms reached 80% efficiency.¹⁸ As a fundamental component in optics, imaging technology was greatly advanced by the development of a meta-lens for a single wavelength in 2016.¹⁹ Traditional optical imaging components are bulky and suffer from material limitations and complex configurations. A thin and flat meta-lens is far more preeminent than a traditional bulky optical lens. Further, a broadband achromatic meta-lens was proposed for practical applications in 2017.²⁰ Two years later, the meta-lens array demonstrated a fascinating light field application.²¹ Meanwhile, AlphaGo defeated the world Go champion in 2016, which attracted the world’s attention.²² This AI’s victory over mankind in specific tasks is of epoch-making significance in the history of AI. In 2017, AlphaGo Zero won 100–0 against AlphaGo through self-teaching.²³ Since then, reinforcement learning has also launched a renaissance.²⁴

Figure 2.

Development trend of AI and meta-optics. The number of publications per year was collected from a Web of Science search.

2. Meta-optics and Artificial Intelligence

2.1. Meta-optics

2.1.1. Fundamental Principle

In ideal classical optics, the propagation of light in two media is related to the speed of light in the media and the optical properties of the two media, such as light refraction and reflection. The emergence of metasurfaces has changed this optical behavior. The metasurface is an ultrathin and flat optical device so that the optical characteristics change when light passes through this interface. The generalized Snell’s law of refraction and reflection can be derived from Fermat’s principle,¹⁶ as shown below

1

2

where n_i is the refractive index of the input media; n_t is the refractive index of the medium when light is refracted; θ_i is the angle of incidence; θ_t and θ_r are the angles of refraction and reflection, respectively; λ₀ is the wavelength of the incident light; and dΦ/dx is the phase gradient. The metasurface contributes to providing a phase gradient to achieve a full 2π phase modulation. The method of designing the metasurface to realize light manipulation and wavefront control has been researched and developed. Specifically, the metasurface contains an array of nanostructures, also called meta-atoms, each of which is regarded as a secondary point light source. When the incident light meets this interface, the nanostructure changes the optical properties of the incident light and reradiates new electromagnetic waves. By effectively controlling the phase distribution of the metasurface, the wavefront of the incident light can be reconstructed with unique properties and new functions.

2.1.2. Design and Fabrication

Meta-optics has been demonstrated from a principle investigation to various optical function developments. These novel optical properties are realized through the careful design of the geometry and configuration of meta-atoms. The selection of the material is closely related to the method of the optical dispersion design. The permittivity (ε) and permeability (μ) of the materials are utilized to calculate the electromagnetic (EM) spectrum behavior of the meta-atoms. The transmittance, reflectance, absorption spectrum, polarization state, and phase delay of the interest structure are simulated and investigated by EM field solvers. The function of the metasurface is to manipulate the wavefront of incident light by tailoring the phase profile of the given working wavelength. The method of achieving complete full 2π phase modulation is the key for providing complete wavefront control. According to the materials used, metasurfaces can be divided into plasmonic (metallic) and dielectric metasurfaces. The plasmonic metasurfaces work due to the oscillation of the surface free electrons of the metal structure. Based on the resonance excitation and EM field enhancement, the 2π phase modulation approaches for designing the meta-atoms include multiresonance¹⁶ and gap-plasmon,²⁵ among others. The gap plasmon metasurface is comprised of a three-layer structure, including a metal antenna layer, a dielectric isolation layer, and a bottom metal reflective layer. This metal–insulator–metal (MIM) building block presents the strong mode confinement and the high-quality factor. The strong near-field coupling in the thin dielectric isolation layer is supported by the dipoles of the antenna layer and the mirror layer. The gap plasmon resonator can be described as the equation of the Fabry–Perot resonator²⁶

3

where w is the length of the antenna structure parallel to the direction of the electric field, λ is the wavelength of the incident light, n_G is the real part of the gap plasmon resonator’s effective refractive index, and φ is the phase shift at the end of the antenna. The 2π phase modulation is achieved by the standing wave resonance of the gap plasmon resonator along the propagating axis. The natural shortcomings of metallic metasurfaces are the high ohmic losses. Sun et al. demonstrated a gap plasmon metasurface with a high operation efficiency for anomalous reflection.²⁵ The conversion efficiency reached 80% due to the fully controlled intrinsic and radiative losses. However, it is usually used as a reflective meta-device due to the sandwich structure.

Dielectric metasurfaces have been promoted to satisfy the more practical need for flat lenses for optical imaging and optical property conversion devices because of the dielectric material’s low-loss and high refractive index in the visible light region. The dielectric meta-atoms support various local resonances of strong electric and magnetic scattering, which are described via the Mie theory.²⁷ When the frequency of the incident light is below or near the bandgap frequency of the dielectric material, the electric dipole and magnetic dipole resonances are excited. The dielectric Mie resonator contains various electric and magnetic components that allow the engineering of EM waves. Huygen’s metasurface is able to generate equal-strength and overlapping electric and magnetic dipole moments for 2π phase modulation using nanoparticles or nanodisks. The low intrinsic losses of dielectric meta-atoms enable the internal generation of circular displacement currents to produce a strong magnetic dipole resonance. The geometric parameters of the dielectric meta-atoms are tuned to maintain the high transmission and the 2π phase modulation simultaneously.

In addition, there is a phase modulation method that can be used universally for most metal and dielectric materials. The Pancharatnam–Berry (PB) phase,^28,29 also called the geometry phase, is a material-independent metasurface design method. This method can be applied to any material and is the simplest phase modulation method. The phase is tuned by varying the orientation angle of the same meta-atoms with the anisotropic geometry.³⁰ The Jones matrix can describe the concept of the PB phase. For the rotated meta-atom, the Jones matrix can be written as³¹

4

where θ is the rotation angle of the meta-atom. The rotation angle is set relative to the x-axis. R(θ) is the rotation matrix. M_xx is the reflection coefficient, and M_yy is the transmission coefficient. The output light can be written as

5

where E_i^± and E_i^∓ are the incident light with circular polarization (+, left-handed; −, right-handed). The output light is generated with two circular polarizations. One is the same as the incident light, which is the unmodulated light. The other is the PB phase-modulated light with the opposite helicity. In short, the rotation angle of the meta-atom is set from 0 to π, which is divided into several equal levels. The incident circular polarization light, the 2θ phase shift from 0 to 2π, is arranged for the light with the opposite handedness.

Fabricating meta-optics is the direct way to connect theoretical designs and practical applications. The database of meta-atoms with phase, polarization, and dispersion functions was built for layout generation. The phase profile of the target optical function is satisfied by the careful configuration of the meta-atom array. The fabrication technologies have been well-developed and selected for different purposes, such as subwavelength-scale, structure carving, large area, high aspect ratio, high throughput, etc. This section reviews and introduces the fabrication technologies of optical meta-devices. The most commonly used fabrication methods are photolithography, electron beam lithography (EBL), focused ion beam (FIB) lithography, nanoimprinting, laser direct writing, and 3D printing. Photolithography is an important fabrication technology for mass production in the semiconductor manufacturing industry. Photolithography uses a photomask and an exposure light to transfer the pattern to the photoresistor.³² The solubility of the exposed photoresistor layer changes to enable pattern generation after the developing process. The lift-off and etching processes can further sculpt the feature of the target material using the patterned photoresistor layer. EBL is a maskless lithography process. A focused electron beam is utilized to directly write the pattern on the photoresistor layer. This technology is widely used for meta-optics research because of its customizable design and quick verification. The medium used is a photoresist whether a photomask or electron beam is used to transfer the pattern. Combined with the choice of a positive or negative photoresist, etching and lift-off processes can transform positive and negative structure pattern generation. A high-aspect-ratio nanostructure can be fabricated with the hard mask etching process.^20,33 A FIB is an ion beam generated by the ion source and focused on the sample’s surface. FIB lithography is a straightforward fabrication method in which the patterning and etching processes happen simultaneously.³⁴ The accelerated ion beam strips the surface atoms of the sample. FIB with a scanning electron microscope becomes a dual-beam system, which can realize visualization during the ion-beam writing. Nanoimprinting is mechanical lithography method that uses a mold to imprint onto a UV- or thermal-curable resist. The nano-or microstructures are formed through mechanical deformation. The mold can be used repeatedly to make the same pattern with a low cost and a high throughput. Laser direct writing uses a focused laser beam to cause localized damage on the material’s surface.³⁵ The pattern can be directly written, similar to the FIB method. The damage is controlled by tuning the laser power. Developed in recent years,³⁶ 3D printing technology is a rapid prototyping manufacturing technology also known as additive manufacturing. It is a technology that builds structures on the basis of a digital model and uses bondable materials such as powdered metal or plastic to print layer by layer. 3D printing opens one more manufacturing freedom in the vertical axis.³⁷ Meta-atoms of different heights can be considered as design parameters for the design. Nanoimprinting is a promising method for the mass production of meta-devices. A mother stamp or mold can be used repeatedly. The soft photocured resist is patterned via the mold with mechanically imprinted micro- or nanostructures.^38,39 Through these advanced fabrication methods, meta-devices can be implemented for applications.

Figure 3.

Diagram of fabrication methods for metasurfaces.

Meta-devices with novel and special optical functions have been generated to meet the optical demand. The great advantages of meta-devices are their new properties, compact size, lighter weight, high efficiency, better performance, broadband operation, lower energy consumption, data volume reduction, and CMOS compatibility for mass production. Optical meta-devices have been well-developed for beam shaping,^40,41 anomalous deflection and reflection,^25,42,43 polarization control and analysis,⁴⁴⁻⁴⁶ holographic images,⁴⁷⁻⁵² nonlinear effects,^49,53−57 lasing,^58,59 tunability,⁶⁰⁻⁶² color display,⁶³⁻⁶⁵ light focusing,⁶⁶⁻⁶⁹ imaging,^{21,33,70−74} image processing,⁷⁵ beam shaping,⁷⁶ multiplex color routing,⁷⁷ sensing,^21,78−81 encryption,^82,83 spectrometry,⁸⁴ optical characterization,^85,86 communication,^87,88 augmented reality,^89,90 and quantum technology.^91,92

2.2. Artificial Intelligence

2.2.1. Brief History

Alan Turing discussed the mathematical possibility of realizing AI in his 1950 paper named “Computing machinery and intelligence”. However, the limited capacity and the high cost of computers prevented the implementation of the concept. Six years later, a program named “Logic Theorist” by Newell, Shaw, and Simon was presented at a workshop at Dartmouth College.⁹³ “Logic Theorist” was widely regarded as the first proof-of-concept implementation of an AI program that could mimic the problem-solving skills of humans. One of the workshop hosts, John McCarthy, coined the term artificial intelligence to distinguish it from cybernetics as an independent science.⁹⁴ Arthur Samuel coined a new word, machine learning, in 1959.⁹⁵ He defined machine learning as a scientific field that makes the computer itself capable of learning. In 1997, Tom Mitchell gave a more detailed explanation as a computer program with experience E, tasks T, and performance P.⁹⁶ Taking digit recognition as an example, the accuracy is the measurement criteria (P). Under the instruction of the accuracy rate (P), the computer performs better on the digit recognition task (T) as it sees more training data (E). From the 1960s to the beginning of the 21st century, the development of AI fluctuated.^93,97 With the investment and withdrawal of research funds, AI has repeatedly experienced renaissances and cold winters. At the beginning of the 21st century, AI gradually recovered from its disrepute thanks to promising solutions to specific problems.⁹³ The real turning point in history was 2012, a huge victory for deep learning. In October 2012, Alex Krizhevsky used the convolutional neural network (CNN) to defeat the shallow machine learning method with a significant advantage in the ImageNet competition.¹⁵ This network later became known as AlexNet. The victory of this competition marked the beginning of the deep learning revolution in the AI industry. Many deeper neural networks were proposed with increased capabilities.

2.2.2. Classification and Approaches

AI today has become an interdisciplinary field that draws on knowledge, methods, and models from multiple areas such as computer science, neuroscience, statistics, physics, and mathematics. The coverage of AI technology is very broad, including expert systems, machine learning, robotics, recommender systems, data mining, fussy logic, rough set, and so on. Generally speaking, machine learning is considered to have evolved from pattern recognition in artificial intelligence.⁹⁸ Pattern recognition was popular in the 1970s and 1980s and was dedicated to the automated recognition of patterns and regularities in data. Pattern recognition often categorizes similar but not identical things or phenomena according to a certain purpose. Pattern recognition is based on prior experience and the algorithm design of professionals. In the 1990s, people realized that data-driven algorithms could bring better performance and higher economic benefits than expert designs. The core of machine learning is to collect data and let the machines learn with experience (data). Machine learning gradually developed into a major research direction in AI from then on. It has achieved great success for problems that lack analytic solutions or domains for which humans only have a poor understanding.

There are different paradigms in machine learning. Machine learning algorithms can be broadly divided into supervised, unsupervised, semi-supervised, and reinforcement learning depending on whether the data has known labels. Supervised learning, a major paradigm in machine learning, trains classification or regression models using labeled training data. In classification problems, each training sample has a label, and supervised learning aims to achieve a high accuracy in the prediction of labels for test samples. It is one of the most widely studied and utilized learning paradigms. Many techniques have been developed for supervised learning, such as decision trees, the naïve Bayes classifier, the perceptron model, the support vector machine (SVM), k-nearest neighbor (kNN), the artificial neural network (ANN), deep learning, and ensemble models including random forest. Unlike supervised learning, unsupervised learning aims to identify the implicit structures of the input data. It does not require labels for the data. Clustering is a major application of unsupervised learning, with the goal of dividing the data samples into multiple groups so that the samples inside the same group are more similar than samples in different groups. Thus, clustering requires carefully designed distance functions for different types of data and corresponding clustering algorithms. Commonly used clustering algorithms include k-means, agglomerative clustering, DBSCAN, spectral clustering, the Gaussian mixture model, and other graph-based clustering algorithms. When there is a small set of labeled data and a large amount of unlabeled data, semi-supervised learning can be applied to take advantage of both types of data for model training. Reinforcement learning can be best applied when the input data contains the grade for the input and thus is commonly used to let computers learn to play a game.

Deep learning is a powerful machine learning technology. Deep learning generally refers to a neural network with multiple layers. With a large amount of data as the input, multiple layers in the neural network are expected to learn more abstract or implicit features that cannot be obtained using a shallow network. Because the deep learning model has more parameters to optimize, it is best used in domains where large amounts of data are available.

We can use supervised learning to explain the major components of learning. Given input training data D = (x,y), which are a set of training samples with x ∈ χ and their labels y ∈ Y, the goal of learning is to search for an optimal mapping formula f:χ → Y so that f(x) is consistent with the known labels of x as much as possible (i.e., fit the data). When Y only contains two labels, this is a binary classification problem. When Y has a finite set of labels, this is a multiclassification problem. When Y is a set of unbounded values or a probability, this is a regression problem. The formula’s space is usually called the hypothesis space, which defines the model’s complexity. For example, the hypothesis spaces of linear models contain lines in 2D space or hyperplanes in high-dimensional space. Once the hypothesis space is defined, machine learning needs an optimization algorithm to search for the best hypothesis to fit the given data (i.e., D). For example, the optimization algorithm for a linear model searches for a linear function to separate the observed data. If the observed data are not linearly separable, more flexible and expressive hypotheses and corresponding optimization algorithms are needed to fit the data better. If the training data D can represent χ and Y, the learning algorithm designed for the observed data D can be generalized to unseen data with controlled prediction error rates according to the theories of learning feasibility.

In the remaining part of this section, we will briefly describe some popular learning models, with a focus on supervised learning models because of their close relevance to current meta-optics research.

2.2.3. Basic Principle of Neural Networks

The development of neural networks originates from perceptron. Frank Rosenblatt originally proposed perceptron in 1958,⁹⁹ with the name inspired by neurons in the brain and nerve system. Neurons in the brain collect bioelectrical signals through numerous dendrites and transmit signals between the cells. When the processed signal reaches the threshold, the neuron is activated to generate an output signal. Perceptron builds an imitative structure of a neuron, as shown in Figure 4a. One perceptron has multiple inputs and one output. The processing procedure contains a linear operation and an activation function. The linear operation performed on the inputs is

6

where w is a vector of real-valued weights, x is a vector of inputs, w·x is a dot product ∑_i = 1^mw_ix_i where m is the number of the inputs, b is the bias, and z is the output of this linear function. The sign of z is used to assign the label (−1 or 1) for the input x, as shown below.

7

Figure 4.

Diagrams of common neural networks. (a) Perceptron. (b) Deep neural network (DNN). (c) Convolutional neural network (CNN). (d) Encoder–decoder framework. (e) Variational autoencoder (VAE). (f) Generative adversarial network (GAN) in an example of the generation of handwritten digits.

A single perceptron model applies to a binary classification problem where the number of labels is two, such as yes and no, cats and dogs, sunny and rainy, and so on. If the entire training data set χ(x) ∈ χ can be linearly separated, the perceptron learning algorithm (PLA) can be applied to find a hyperplane to fit χ. Perceptron cannot recognize multiple categories, nor can it learn nonlinearity.

Although perceptron has a limited learning capacity, it lays the foundation for more flexible learning models. Using perceptron as a building unit, an artificial neural network (ANN) simulates the human brain’s neural network. ANN is usually shortened as neural networks. Unlike perceptron, a neural network introduces one or more hidden layers, as shown in Figure 4b. Each hidden layer has multiple perceptrons. Among the hidden layers, the output from the previous layer will be used as the input of the next layer. The neural network is composed of many perceptrons with a network-like structure. Neural networks can be used for binary classification or multiclassification. Additionally, there are more choices of activation functions in the neural network than in perceeptron, such as hyperbolic tangent (tanh), sigmoid,¹⁰⁰ rectified linear unit (ReLU),¹⁵ and parametric rectified linear unit (PReLU).¹⁰¹ The activation functions enable the neural network to learn nonlinear functions, increasing its utility for more complex learning tasks. Without the activation function, no matter how many layers the neural network has, it is just matrix multiplication.

Deep neural networks (DNN) are the foundation of deep learning. A DNN has one input layer, one output layer, and multiple hidden layers. Figure 4b illustrates a typical DNN. A DNN is also called a fully connected neural network and sometimes a multi-layer perceptron (MLP).

Once the network structure of a neural network is designed, the weight matrix, which describes the weight vectors of each layer, needs to be learned using the training data. The same as a typical supervised learning problem, the goal of the training is to let the model fit the observed data. The most commonly used neural network training method is backpropagation. The backpropagation algorithm has two elements, the loss function calculated in the forward direction and the gradient descent derived in the reverse direction. There are some commonly used loss functions for specific learning tasks, such as absolute error, mean squared error (MSE), and KL-divergence. Training is used to minimize the defined loss function. The backpropagation algorithm updates the neural network’s weights using a gradient descent controlled by the learning rate.

The inputs to the DNN are numerical vectors containing human-extracted features or raw features from the inputs, such as each pixel of an image. When using raw features of high dimensions, the computation load of the DNN is huge. Compared with a DNN, a convolutional neural network (CNN, or ConvNet) demonstrated superiority in processing data with a grid-like topology, such as images and videos. The CNN showed dazzling performance in Computer Vision (CV) in recent years. As demonstrated in Figure 4c, the basic CNN structure includes the convolutional layer, the pooling layer, and the fully connected layer. Designers could arrange these functional layers using empirical experiment results and general guidelines.

The convolutional layer performs the convolution, the key operation of CNN, on inputs. In DNN, the output of each neuron is affected by all previous data. The neuron in CNN could only “see” the data of a certain area. This area, called the reception field, is usually square-shaped. A shared convolution kernel is adopted to execute the dot product with the data in the reception field. The convolution layer uses the same convolution kernel to traverse all inputs. The role of convolution is to acquire high-dimensional features. However, these features are redundant. As an efficient sampling method, the pooling layer is used to filter redundant information. The pooling layer has two common types, max and average. The feature map from the previous convolution is divided into disjointed regions in the pooling layer. Each region is a custom-sized square. These regions’ maximum (or average) features are used to represent the convolution feature after the pooling layer. Dimensionality reduction leads to a low computation complexity. The fully connected layer is the same as that in the DNN and is often arranged at the end of the CNN. The extracted feature map has been significantly reduced in size after the repetition of convolution and pooling. A fully connected layer could perform the final mapping from feature to label with minimum computation consumption.

Based on the CNN, many variations of CNN have been developed. For example, to transfer from an image-level classification to a pixel-level semantic segmentation, the encoder–decoder framework is the fundamental architecture. As shown in Figure 4d, an encoder and a decoder are sequentially arranged in the data flow. Typically, the encoder extracts the original data’s low-resolution or low-dimensional features of interest. The decoder is adapted to recover the dimensions of the data. As a general framework, encoder–decoder is widely used in various algorithms for communication, image processing, and computer vision.

For pixel-level applications, the dimensions of the output should be similar to those of input. In 2015, using the encoder–decoder framework, a fully convolutional network (FCN) introduced a deconvolution layer for upsampling and skip architecture and removed the fully connected layer.¹⁰² Similar to the FCN, U-Net adopted a U-shaped symmetrical architecture in the same year.¹⁰³ Additionally, U-Net correspondingly connects the feature between down-sampling layers and up-sampling layers. Later, SegNet was further proposed to pass the pooling indices from the encoder to the decoder to reduce redundancy.¹⁰⁴

In the development of deep networks, gradient vanishing (gradient becomes zero) and gradient exploding (gradient becomes too large) prohibit weight updates while training. For very deep networking, a residual neural network (ResNet) with a residual block was proposed.¹⁰⁵ The core idea of the residual block is to introduce an identity shortcut connection, which directly skips one or more layers. In the process of gradient backpropagation, the residual module realizes the cross-layer propagation of the gradient, which is simple but efficient. Residual blocks are frequently used in today’s CNNs to solve the problems of gradient vanishing and gradient exploding.

Two families are particularly famous in the large family of generative models, namely variational autoencoder (VAE) and the generative adversarial network (GAN). VAE¹⁰⁶ was developed from autoencoder (AE). The AE follows the encoder–decoder architecture, as shown in Figure 4d. The generative model can be regarded as a database that records data information. A low-dimensional key, normally a vector, can unlock high-dimensional data stored in the model. Such a reconstruction process can be realized by a decoder architecture. To improve the efficiency, the encoder is introduced to automatically encode vectors to represent high-dimensional data in a large quantity. In the AE, the encoder compresses the data like an image into a low-dimensional vector representation, which is regarded as the latent vector of variables. Then, the decoder decodes this latent vector to obtain reconstructed data. By minimizing the error function that compares the reconstructed image and the original image for training, the information on the training image is retained in the parameters of the network. However, AE was accused of simply memorizing data, with a poor ability to generate new data. For example, given a vector of the digit “1” image or the digit “7” image, AE can produce (decode) a corresponding digit “1” image or a digit “7” image. With a vector of median value between “1” and “7”, the output of a successful generator should resemble both “1” and “7”. However, AE fails to do this. The cognitive performance of AE heavily depends on the training data.

In VAE, the latent vector is one point in the latent space. The information that a point provides is limited. To completely leverage the latent space, VAE adds constraints to the encoder, as shown in Figure 4e, forcing the encoder to produce latent vectors that obey the Gaussian unit distribution. Therefore, we can randomly sample from the Gaussian distribution space of a latent vector as the input of the decoder to generate new data. The new data is similar to but different from the training data and may be data that has never been seen before.

GAN¹⁰⁷ uses the two modules in the framework, a generator and a discriminator, as shown in Figure 4f. The two modules learn from each other in a zero-sum game where one’s gain is the other’s loss. Given an image that may be either a real image from the training set or a fake image generated by the generator, the discriminator should determine whether the image is real or fake. On the contrary, the generator tries to generate a fake image that can fool the discriminator. In training, two modules all become stronger and stronger. Finally, the GAN can generate data that belong to the same category. Figure 4f shows an example of handwritten digit image generation. After training, the generator masters the distribution of real data so that the generator can generate an unseen image of the digit. Compared with the GAN, the VAE directly compares the difference between the reconstructed picture and the original picture, indicating the direction of optimization in training. However, since no confrontation network is used, the VAE tends to produce fuzzy results.

3. AI for Meta-optics

We will discuss how AI can empower the resolution of forward and reverse problems in meta-optics, data analysis for a metasurface-based system, and an intelligent programmable meta-device. In the forward problem, AI can be used as either a high-speed search engine or a surrogate physical computing model. Unlike traditional simulation software that solves Maxwell’s equations, neural networks establish a shortcut for mapping between the structure geometry and optical responses. Neural networks demonstrate faster computation speeds than traditional simulation tools. In the reverse problem, AI can reverse-design the parameters and configurations of the meta-device based on the performance goals. One way to view this is that AI finds an approximate solution for the equation that does not have an analytic solution or finds an optimal solution for the equation with countless solutions. Most importantly, these solutions normally cannot be found by humans due to the huge computational load. In optical-based detection or monitoring systems, EM data are usually accompanied by strong noise of unknown significance. Moreover, it is difficult for designers and users to directly establish a semantic understanding of the collected signals. AI, especially convolutional neural networks, has demonstrated powerful feature extraction and semantic recognition capabilities for two-dimensional (2D) or three-dimensional (3D) optical data. Additionally, with the support of high-computing-power platforms, AI can perform high-speed information processing. In programmable metasurface-based systems that have high requirements for real-time performance, AI can process continuous input information in a timely manner and predict the trend of target development and change. In this way, intelligent meta-devices empowered by AI can respond in real time to emergencies and provide countermeasures.

Before talking about AI applied in meta-optics, the concepts of the discriminative model and the generative model should be discussed. In models, the observable variable is X, and the target variable is Y. The discriminative model establishes the decision boundary for distinction using learning data. The generative model learns the statistical model of the joint probability distribution on X × Y and generates new data using the obtained probability model. The discriminative model and the generative model have inverse conditions. Given an observation x, the discriminative model is a conditional probability model P(Y|X = x). Given a target y, the generative model is a conditional probability model P(X|Y = y). The discriminative model is usually used to perform classification or regression tasks on given data. As a complementary method, generative models can generate new data similar to existing data.

There are many subsets of generative models and discriminative models. They are not classified according to the network structure used. For example, CNNs can be found in both the discriminative model and the generative model. Generally, however, in AI applications for meta-optics, discriminative models are widely used in surrogate modeling, inverse design, and data processing. VAE and GAN are two typical generative models that are common in the inverse design application of meta-optics.

3.1. Surrogate Modeling

3.1.1. Modeling of Optical Properties

Conventionally, the interactions between light and meta-atoms are derived by solving Maxwell’s equations.¹⁰⁸ Many analysis methods and commercial tools have been developed for computer-based simulations. The full-wave Maxwell solvers such as finite difference time domain technique (FDTD),¹⁰⁹ finite element method (FEM),¹¹⁰ and rigorous coupled-wave analysis (RCWA)¹¹¹ are all widely used. These numerical simulations tend to be time-consuming for complicated electromagnetic systems. High time complexity has always been an obstacle to electromagnetic design. AI, especially deep learning, provides a groundbreaking shortcut for optical simulations that is direct and time-saving. In recent years, surrogate modeling with AI demonstrated many competitive achievements.

In surrogate modeling, ANNs are adopted as approximate predictors for the optical responses of meta-atoms. The prediction may deviate from the correct answer to a certain extent or only be accurate within a certain range. Surrogate models cannot completely surpass traditional Maxwell’s solvers. ANNs for surrogate modeling are superior choices in specific design tasks.^112,113

Meta-optics is famous for its manipulation of light–matter interactions with subwavelength meta-atoms. The most common simulation in meta-optics is modeling the interaction of light and the meta-atom with a specific three-dimensional geometry. The meta-atom is the foundation of the metasurface, which is generally an array of meta-atoms. In wavefront-shaping applications, phase and amplitude are the most common optical responses. In 2019, An et al. proposed a DNN called Predicting NN to model amplitude and phase responses of all-dielectric meta-atoms over a wide range from 30 to 60 THz.¹¹⁴ As shown in Figure 5a, the inputs of the Predicting NN are the geometric parameters, and the outputs are the real or imaginary parts of the transmission coefficients. An et al. developed two DNNs for the separate prediction of the real part and the imaginary part,. The demanding amplitude and phase responses were further calculated using the transmission coefficients. This indirect operation is because the amplitude and the phase response of a typical meta-atom change abruptly around resonant frequencies. The prediction performance of the neural network will degrade greatly at the resonances because of the hard regression of sharp nonlinearity. Therefore, the authors innovatively used the varied continuous real and imaginary parts of transmission coefficients as the prediction targets. The prediction accuracy of cylinder-shaped and “H”-shaped meta-atoms reached over 99% at speed on a millisecond scale, which was 600× faster than a conventional simulation. In the following year, An et al. presented a new approach with a CNN to characterize the amplitude and the phase from a meta-atom in the same working bands.¹¹⁵ Differently, the modeling objects are freeform structures with varied material properties instead of simple structures, as shown in Figure 5b. The design freedom includes the 2D pattern image, the lattice size, the structure’s thickness, and the material’s refractive index. The head of the CNN is divided into two input branches. One processes the 2D pattern image, and the other deals with the vector of different properties. With down-sampling and up-sampling procedures, the two branches are recombined with feature maps of matching dimensions. The outputs still employ the format of real and imaginary parts of the transmission coefficient. Compared with previous work, this approach uses more training data but provides more efficacy for free-form design. Additionally, under the same hardware condition, the prediction speed is 9000× faster than that of a conventional simulation, which also greatly surpasses previous work.

Figure 5.

Overview of surrogate modeling for meta-atom characterization. (a) Amplitude and phase prediction for a cylinder-shaped meta-atom.¹¹⁴ Reprinted with permission from ref (114). Copyright 2019 American Chemical Society. (b) Amplitude and phase prediction for a free-form all-dielectric meta-atom.¹¹⁵ Reprinted with permission from ref (115). Copyright 2020 Optical Society of America. (c) Scattering cross-section prediction for alternate-material-shell nanoparticles.¹¹⁶ Reprinted with permission from ref (116). Copyright 2018 American Association for the Advancement of Science. (d) Prediction of diffraction efficiencies for a 16-side polygon-shaped meta-atom.¹¹⁷ Reprinted with permission from ref (117). Copyright 2018 American Institute of Physics. (e) Absorption spectra prediction for a free-form meta-atom via a DNN¹¹⁸. Reprinted with permission from ref (118). Copyright 2021 American Chemical Society. (f) Absorption spectra prediction for a free-form meta-atom via a CNN and a RNN.¹¹⁹ Reprinted with permission from ref (119). Copyright 2019 Springer Nature.

In 2018, Peurifoy et al. proposed a DNN to approximate the scattering cross-section of light from the nanoparticle.¹¹⁶ The modeled nanoparticle was composed of eight shells of alternating SiO₂ and TiO₂, as shown in Figure 5c. The inputs of the proposed DNN are the thicknesses of the eight shells, ranging from 30 to 70 nm. Outputs are the 200 sampling points for scattering cross-sections between 400 and 800 nm. The proposed DNN is a simple fully connected neural network with four layers. It was trained with only 50 000 examples. The performance in the unknown setting of this surrogate model coincides with the results of commercial simulation tools. For nanoparticles of different shell numbers, the prediction accuracies all reach above 98%. Essentially, the fewer layers a nanoparticle has, the higher the accuracy would be. This powerful performance originates from the discriminative model’s accurate grasp of the relationship between unknown data and known data. As a discriminative model, the DNN in this work established a probability distribution between the inputs and the outputs. The probability distribution gradually approaches the actual relationship in reality under a certain amount of training. That is why a simple DNN could predict the accurate response to unknown data. Without intermediate variables, the direct modeling between inputs and outputs occurs at times orders of magnitude faster than conventional solvers.

Inampudi et al. developed a DNN to predict the diffraction efficiency distribution from the meta-atom shape of a meta-grating as an accelerated simulation strategy in 2018.¹¹⁷ The modeled meta-atom was a 16-side polygon (see Figure 5d). In a polar coordinate system, the vertexes are uniformly distributed in polar angles, but the radii of each vertex vary freely. Given the 16 radii, the proposed DNN outputs diffraction efficiencies of 13 diffraction orders. Although this work has not reached the free form, the 16 structural parameters have provided sufficient complexity.

The design of meta-atoms is concerned with not only the diffraction efficiency but also the absorption, which is an important observation. Microwave-absorbing materials have attracted wide attention in recent decades. Metasurfaces have been reported to be able to broaden the absorption bandwidth when placed on traditional absorbers.¹²⁰ In the design of a meta-atom for such a metasurface, Zhang et al. found it could take months or years to traverse all possibilities via conventional simulation software.¹¹⁸ They proposed a DNN (Figure 5e) to predict the reflection loss from the pattern of a metal-material-covered meta-atom, which was about four orders of magnitude faster than the conventional simulator. The reflection loss was further calculated to derive the absorption performance. The DNN can only handle vectors. The free-form structure pattern should be converted to binary arrays before being sent to a DNN. Since the architecture of a DNN will grow rapidly with a giant computation load as the number of elements in the input vector increases, the pattern gridding of the DNN-based freeform design cannot be very fine. The pattern in this work is only discretized as 8 × 8. Sajedian used a more direct method to retrieve absorption responses, as shown in Figure 5f.¹¹⁹ The developed surrogate model is a combination of a CNN and a recurrent neural network (RNN). Specifically, this CNN is a ResNet with residual blocks for the training of a very deep network architecture. Compared with a DNN, a CNN is better at freeform pattern image processing. The pattern gridding in this work reached 100 × 100. The CNN extracts low-dimensional semantic features about the spatial information from the structure pattern image. Additionally, the RNN is responsible for mapping the relationship between the shape’s extracted features and its optical response features.

Single-layered metal nanostructures are common modeling objects due to their low modeling complexity. In 2018, Liu et al. also took the pattern of a metal material covering as the modeling object (see the top panel of Figure 6a).¹¹² Besides the finer discrete grid of the pattern, 2D images were used as the model input instead of binary vectors. The modeling input is the pattern of Au covering the meta-atom. The output is four transmission spectra for different polarization directions of incidence and detection. Additionally, a CNN was developed for this specific task. Figure 6a shows the CNN, which uses convolutional layers to extract pattern features from the 2D image. After the dimension reduction, the fully connected layer achieves the mapping between the geometry information and the transmission spectra. Covering a wide wavelength range from 500 to 1800 nm, this CNN can achieve an average absolute prediction error of less than 0.01. The transmission spectrum is an important index for design. Transmission spectrum modeling with a high accuracy is of vital importance.¹²⁴

Figure 6.

Overview of the surrogate modeling of various objects. (a) Transmission spectrum prediction from a single-layer meta-atom geometry with a CNN.¹¹² Reprinted with permission from ref (112). Copyright 2018 American Chemical Society. (b) Plasmonic color prediction from the geometry of a nanoparticle with a DNN.¹²¹ Reprinted with permission from ref (121). Copyright 2019 Springer Nature. (c) 3D electric field distribution inside the structure via a CNN.¹²² Reprinted with permission from ref (122). Copyright 2019 American Chemical Society. (d) Electric field distribution prediction from the investigation of nine meta-atoms. A DNN was adopted with SVD in this work.¹²³ Reprinted with permission from ref (123). Copyright 2021 American Chemical Society.

Extending from the common spectrum, plasmonic colors can also be treated as modellable responses. Plasmonic colors are generated from plasmonic resonances in metallic nanoparticles activated by a picosecond laser pulse. Baxter et al. developed a DNN with fully connected layers to directly predict colors from the geometric parameters of the Ag meta-atom.¹²¹ As demonstrated by the validation results in the right panel of Figure 6b, the output of the numerical simulation and the DNN prediction are highly consistent. The training data of the proposed DNN are from both a simulation and real experiments, with the aim of reaching a high prediction accuracy in real industrial applications.

In 2019, Wiecha and Muskens presented a three-dimensional CNN to provide nano-optical models for nanostructures of arbitrary shape.¹²² As illustrated in the left panel of Figure 6c, the design volume was discretized into voxels. A three-dimensional matrix containing geometry voxels was fed to the CNN as the input. The output is the electric field inside the nanostructure in a voxel-level correspondence manner. Given the electric field, various physical quantities could be calculated, such as polarization, amplitude, near-field response, far-field response, Poynting vector, and nonlinearity. The demonstrated 3D CNN follows the encoder–decoder framework to achieve the voxel-level output. Additionally, the whole network architecture was modified from U-Net and introduced residual blocks. The authors demonstrated attempts on both plasmonic and dielectric structures. The plasmonic one is comprised of planar gold nanostructures of random polygonal shapes, and the dielectric one is comprised of silicon pillar structures. The prediction of various physical quantities matches well with the ground truth in both plasmonic and dielectric experiments. In terms of the field intensity distribution, the median value of cross-correlation between the CNN prediction and the ground truth is above 0.9, with about less than 10% outliers. The rates of failed prediction for the scattering cross-section and the polarization were all kept below 10%. On the GPU test, the prediction speed reaches 3 and 6 ms for Au and Si structures, respectively. Even when compared fairly with conventional software on a CPU, the neural network is still 3–5 orders of magnitude faster.

In 2021, Zhelyeznyakov et al. also developed a DNN to perform the forward prediction of the electrical field of a meta-atom with the support of singular value decomposition (SVD).¹²³ Their work focused on the planar field distribution λ/2 from the scatters. Meta-atoms in this work are nanopillars of different radii. Given the radii of each nanopillar and its eight nearest neighbors, as shown in Figure 6d, the proposed surrogate model can provide the electrical field over the square area that contains the nine input nanopillars. SVD is an important dimensionality reduction algorithm in machine learning. The authors first use SVD to decompose the field matrix into the multiplication of low-dimensional matrices, a reduced left singular vector, and a weight matrix. The input of the DNN is nine radii, and the output is the weight matrix in separated real and imaginary parts. The prediction speed is four orders of magnitude faster than the numerical simulation. Besides the DNN model, the authors also developed a linear regression model via SVD, demonstrating that surrogate modeling is not restricted to neural networks. Most importantly, this model is not at the meta-atom level. Such non-atom-level works include the characterization of the supercell,¹²⁵ single metasurfaces,^126,127 and multiple metasurfaces.¹²⁸ This model takes the intercoupling between meta-atoms into consideration. However, the electrical field prediction of the nine-pillar cell is not very accurate because the adopted training data lack sufficient diversity. The train set of this work is very large, but data were generated from configuration maps of 10 lenses in which the field distributions were too specific. Although this model is not very accurate, it demonstrates another direction of a surrogate Maxwell’s solver.

3.1.2. Performance Evaluation

Neural networks are data-driven, and the internal logical function relationships in hidden layers are hard to interpret. Thus, the capabilities of surrogate models will naturally be questioned. Besides the unlearned interpolation data, researchers also used boundary cases such as the parameter limit¹²⁹ and a blank model¹¹³ to examine the surrogate models’ generalization ability, which measures whether the learning model can be effectively applied to unseen samples. These experiments demonstrated the feasibility of learning by the surrogate models. With this, the prediction of the model is not limitedly coupled in the given data, but the probability distribution relationship between the variables is really learned.

To evaluate the effectiveness of a surrogate model, its accuracy is often compared with a conventional simulation tool that solves Maxwell’s equation. Generally, most surrogate models exhibit high fidelities in various optical properties. Regardless of the free-form structure in Figure 7a or meta-atoms with predefined characteristics in Figure 7b, the corresponding optical responses from the surrogate model all nearly coincide with results from the commercial simulation tool. Extending from the spectrum to the 3D electric field distribution in Figure 7c, the prediction from ANN closely resembles the simulation result. As for the chirality characterization in Figure 7d, the prediction is almost consistent with simulation. To further test the practicality of the surrogate model, the measurements of actually fabricated designs are included in the comparison, as shown in Figure 7e. The transmission spectra from the prediction, the simulation, and the measurement show good qualitative agreement. In addition to the competent accuracy, the surrogate model is orders of magnitude faster than the conventional simulation (see Figure 7f).

Figure 7.

Validation of the surrogate model. (a) The absorption spectrum of the free-form structure.¹¹⁹ Reprinted with permission from ref (119). Copyright 2019 Springer Nature. (b) Amplitude and phase responses of an “H”-shaped meta-atom.¹¹⁴ Reprinted with permission from ref (114). Copyright 2019 American Chemical Society. (c) Forward and backward scattering from a nanorod under TE and TM modes, with a diagram of the internal electric field distribution (top).¹²² Reprinted with permission from ref (122). Copyright 2019 American Chemical Society. (d) Reflection spectra and corresponding CD spectra.¹²⁹ Reprinted with permission from ref (129). Copyright 2018 American Chemical Society. (e) Validation of the transmission spectrum with the measurement of the actually fabricated design.¹¹³ Reprinted with permission from ref (113). Copyright 2018 Springer Nature. (f) A computation time comparison between the numerical simulation and the deep learning-based surrogate model.¹¹⁹ Reprinted with permission from ref (119). Copyright 2019 Springer Nature.

To summarize surrogate modeling with neural networks, Table 1 lists the information worthy of attention for direct comparison and understanding. Modeling objects discussed in Table 1 cover various shapes, including multilayer nanoparticles, H-shaped structures, cylinder-shaped structured, symmetrical structures, twisted SRRs, polygons, and free-form structures. Each optical structure has a specific optical response. Therefore, most adopted algorithms are discriminative models used to solve the one-to-one mapping. The working band involved covers the near-ultraviolet, visible light, infrared, and microwave regions, wherein most attention is paid to the visible light and near mid-infrared regions. However, these working bands are discretized into sampling points. No matter how many, these sampling densities are fixed, making them less flexible than conventional numerical simulation tools. From plasmonic to dielectric meta-atoms, the materials listed in Table 1 cover common metals and dielectrics. Selected references in the table have different modeling responses, proving that the current surrogate model can learn almost all common optical properties from the meta-atom structure geometry.

Table 1. Summary of Surrogate Modeling.

modeling object	algorithm	working bands	lattice period	materials	inputs	outputs	accuracy	train set (samples)	test set (samples)	runtime	acceleration	hardware condition	ref
8-alternating-shell	DNN	400–800 nm (200 points)		SiO2 and TiO2	2–8 GPa	scattering cross section	98.5–99.7%	50 000	6250	0.01–1 s	∼2 orders of magnitude	2.9 GHz Intel Core i5 processor	(116)
“H”-shaped	DNN	600–1650 nm (43 points)		Au and ITO covered SiO2	8 GPa	transmission spectrum	MSE 0.16	>10 000					(113)
predifined symmetry	VAE and CNN	1200–1700 nm	1 μm	Si	100 × 100 SPb	3 BIC frequencies	MSE < 0.1	18 000	1000				(130)
twisted SRR	CNN and DNN	30–80 THz (201 points)	2.5 μm	Au and dielectric	5 GPa	reflection spectra and CD spectrum	MSE 0.0008	25 000	5000	1 s	∼5 orders of magnitude	i7 CPU personal computer	(129)
16-sided polygon	DNN	1.55 μm	3.33 μm	dielectric	16 GPa	diffraction efficiencies	MSE 0.00085	63 000	13 500		∼2 orders of magnitude	workstation with 24 cores	(117)
8 × 8 coding grid	DNN	0.1–18 GHz (180 points)	2 mm	Cu, Ag, Au, or Al	binary SPb vector and frequency	absorption at certain frequency	MSE 0.204	324 000	36 000	0.06 s	∼4 orders of magnitude		(118)
100 × 100 coding grid	CNN and RNN	800–1700 nm (1000 points)	500 nm	Ag and SiO2	SPb	absorption spectra	MSE < 0.0001	60 000	30 000	<1 s	∼2 orders of magnitude	GPU server with two GTX 1080ti graphics cards	(119)
Cylinder-shaped	DNN	30–60 THz (31 points)		all-dielectric	4 GPa	amplitude and phase	99.50%	35 000	15 000	34.4 ms	500 times	quad-core CPU with a 3.5 GHz clock speed and two Nvidia 1080Ti GPUs	(114)
“H”-shaped	DNN	30–60 THz (51 points)			6 GPa		amplitude of 99.4%	14 800		35.4 ms	600 times
							phase of 99.3%
28 × 28 coding grid	CNN	30–60 THz (51 points)			SPb			30 000
64 × 64 coding grid	CNN	170–600 THz (32 points)	340 nm	Au and SiO2	SPb	transmission spectrum	average absolute error <0.01	6500				single Quadro P5000 GPU	(112)
64 × 64 coding grid	CNN	30–60 THz (51 points)	trainable	all-dielectric	SPb and 3 DPc	amplitude and phase	amplitude of STDd 0.005	70 000	30 000	5.5 ms	9000 times	quad-core CPU with a 3.5 GHz clock speed and two Nvidia 1080Ti GPUs	(115)
							phase of STDd 0.78°
3D arbitrary shape	CNN	700 nm	900 nm	Au	45 × 45 × 10 geometry matrix	electric field inside the nanostructure	median value of CCe > 0.9	28 000	2000	53 ms	3–5 orders of magnitude	third-generation Intel i7 quad-core CPU (i7-3770)	(122)
				Si				28 000	2000	250 ms
9-cylinder	DNN and SVD	400 and 633 nm		Si3N4	9 radii	electric field	relative error 400 nm = 0.37 and 633 nm = 0.26	98 568	24 642	12 s	4 orders of magnitude	a common laptop	(123)

^aGP is the abbreviation for geometric parameters.

^bSP is the abbreviation for structure pattern.

^cDP is the abbreviation for design parameters.

^dSTD is the abbreviation for standard deviation.

^eCC is the abbreviation for cross correlation between the prediction and the ground truth.

The input of the surrogate model is the description of the modeling object. The modeling representations of the meta-atom geometry are generally divided into two kinds. One is of geometric parameters, and the other is of the pattern matrix. Free-form structures are represented by a pattern matrix. Meta-atoms described by geometric parameters have predefined shape limits. The image of a meta-atom pattern can usually bring a much higher design degree of freedom (DOF) than several geometric parameters. However, directly comparing the DOFs between the geometric parameter model and the geometry image model is unfair. The data in the pattern matrix is binary, while geometric parameters are usually continuous or vary in fine increments over a wide range. Therefore, the DOF of geometric parameters is not necessarily smaller than that of geometric images. We prefer the choice between geometric parameters and structural images to be the choice of two designs for specific shapes and free shapes rather than a measure of superiority. Both cases provide a high DOF, which is sufficient for a specific target. Besides these two mainstream modeling descriptions, the modeling object in ref (122) is described as a 3D matrix. Geometric parameters are 1D data. The structure pattern is a 2D data. 3D modeling additionally takes the height of the meta-atom as the variable of consideration, in contrast to the 2D pattern case, which has a fixed height or thickness. The multidimensional description of the modeling object will provide more practicability and flexibility. Surrogate modeling is not limited to the meta-atom level. In ref (123), the electrical field was predicted for a region containing nine meta-atoms.

Accuracy is the most commonly used metric for evaluating the prediction performance of the surrogate model. Most neural networks in Table 1 adopt mean-squared error (MSE) as the loss function to train the model. Therefore, we also use this quantity to indicate the accuracy for those works that did not clearly provide the accuracy value or alternative metrics. The author in ref (122) adopts normalized cross-correlation, which ranges from 0 to 1, as the evaluation metric. Cross-correlation represents the relevance between two sequences, such as the optical response over different wavelengths, electric field distributions, etc. A cross-correlation of 0 means the sequences are completely irrelevant. The greater the cross correlation, the greater the relevance. When cross correlation reaches 1, the two comparison objects are exactly the same. From the table, most surrogate models can promise at least 90% accuracy.

Normally, more complex models need more training data to increase the their generalization capability and robustness. The generalization capability determines whether the model can achieve accurate predictions for unseen structures. Robustness represents the anti-interference ability. Especially for free-form structure modeling, the diversity of the training data is very important. All discussed works used over 10 000 data points as the training set. We can use this quantity as a lower reference cutoff for the preparation load of a surrogate model. The prediction speed for meta-atom characterization is almost at the millisecond level. Compared with a conventional numerical simulation, the surrogate model is 2–5 orders of magnitude faster.

As an approximate Maxwell’s equation solver, surrogate modeling has three major shortcomings. The performance of the surrogate model is limited by the construction of the training data. Each model can only work at specific interests, such as transmittance, reflection, polarization, etc., at specific working wavelengths. The performances of some surrogate models will degrade at the resonance frequencies. Additionally, the process of generating the training data is a labor-intensive and tedious task. However, surrogate modeling based on neural networks has been proved many orders of magnitude faster than conventional simulation tools. With a more complex network architecture or special attention to outliers, the surrogate model can perform an accurate prediction even at resonate frequencies. If the designer team shares the proposed surrogate model, the computation and time consumption will be affordable. Apart from the fast speed, surrogate modeling has another attraction. In the inverse design of meta-optics, a real-time simulation response is required. Compared with current commercial software, surrogate models based on neural networks can be integrated into the inverse design scheme easily and have more design freedom. NN-based inverse design must propose a surrogate model simultaneously or integrate the surrogate model into the inverse design scheme. Only in this way can the data in the neural network for inverse design be iterated in training. Even for inverse design based on evolutionary computation, the hybrid strategy leveraging the NN-based surrogate model demonstrates better time efficiency.

3.2. Inverse Design

Artificial intelligence-assisted inverse design performs iterative optimization according to the difference between the target and the intermediate result, leading to an optimal final result that matches the design requirements to the greatest extent. Therefore, the resulting device is usually complex, random, and nonintuitive free-form. In inverse design, the requirements such as optical response are qualified as specific physical quantity values. The intermediate value generated in the iterative process of the algorithm and the target physical quantity value construct the objective loss function together. Additionally, the value difference indicates the optimization direction. Concrete device design usually requires the careful consideration or the simultaneous satisfaction of multispecification demands. Through the cleverly design of variables, weights, and penalty terms in the loss function, reverse design can easily handle multi-objective problems. Even the designer can give preference to certain observations and force the algorithm to satisfy certain conditions first in the optimization process.

This section discusses two major paths for optimization problems. One is gradient-based neural networks, and the other is gradient-free evolutionary computation. Although gradient-based neural networks efficiently find local optimal solutions for convex problems, they have difficulty dealing with multiple objectives and nondifferentiable functions. Gradient-free methods have fewer constraints on the problems and thus can be applied to these challenging cases. However, in the actual applications, evolutionary computing universally faces troubles such as a low solution speed and difficulty in convergence. Therefore, for problems such as linear programming and convex optimization, the gradient-based mathematical optimization method is more promising.¹³⁰ The inverse design of free-form structures can take conventional simulation software months or even years due to the great design freedom. For such cases, designers usually design an additional forward optical response prediction model (surrogate model) via deep learning to replace the numerical simulation. However, even if there is a deep learning-driven surrogate model for the optical response prediction, inverse design can be based on either deep learning^112,129,131 or evolutionary computation.^114,118

3.2.1. Gradient-Based Neural Networks (Deep Learning)

Given an arbitrary spectrum, designers always want to find the geometry of a single meta-atom that satisfies the target spectrum. The inference difficulty of the corresponding nanoscale geometry depends on the spectral complexity. Figure 8a shows a general understanding of the application positions of various approaches.¹¹³ For a simple target spectrum, the designer can guess the corresponding nanostructure directly based on experience and physical intuition with only a few attempts. As the complexity of the spectrum increases, designers must use numerical methods, such as the FEM or FDTD method. Stochastic algorithms like evolutionary algorithms can further handle more complex systems but cannot address the inverse problem efficiently. Deep-learning-assisted inverse design provides both efficient and nonintuitive solutions. A rough comparison between evolutionary algorithms and deep learning algorithms is shown in Figure 8b and c. From the perspective of the entire optimization process, the deep learning method is more computationally efficient. Deep learning has demonstrated much efficacy in accelerating meta-optic designs, such as an auxetic metamaterial¹³² and a transmissive meta-atom.¹³³

Figure 8.

Comparison of different design schemes for plasmonic meta-atoms. (a) General distribution of design feasibility versus spectral complexity for various computational approaches. (b) Working flow of general evolutionary computation methods. (c) Working flow of general deep learning methods.¹¹³ Reprinted with permission from ref (113). Copyright 2018 Springer Nature.

Inverse design assisted by deep learning could be divided into two parts according to the adopted model types. One is based on the discriminative model, and the other is based on the generative model. The inverse design method based on the discriminative model can be further divided into two classes. In the first class, design parameters are arranged at the input position, and target responses as the output influence the design parameters via backpropagation. Specifically, device parameters generated by random or educated guesses are evaluated by the proposed design scheme. Then, the error between the output response and the expected response is calculated through the loss function, and backpropagation is used to reduce it iteratively. This class of design schemes is simple but time-consuming as an iterative optimization method. The second class is more straightforward and thus the mainstream. Given the expectation, NN outputs the prediction. For example, the second class method sets the expected performance such as the spectrum as the input, and the corresponding device parameters such as the geometry can be retrieved directly. The second class is usually more efficient than the first class. Therefore, the adoption of the first class is relatively rare. Below we provide examples of both classes.

Peurifoy et al.¹¹⁶ demonstrated examples of spherical nanoparticles that have alternating shells of different materials, as shown in the left panel of Figure 5c. In their work, they developed a scattering model of nanoparticles based on a DNN. Fed with the thickness of each shell, the DNN outputs corresponding scattering cross sections overworking wavelengths. They used the same DNN to perform inverse design rather than developing another model, which is a typical example of the first class. With the weights in the hidden layers fixed, the DNN is retrained by taking the input as new trainable variables. Besides fitting a random spectrum, their inverse design also succeeded in satisfying specific spectra with scattering peaks at a single wavelength or over a broadband. The first class has demonstrated success in the multiple-metasurface system,¹³⁴ geometrical designs for high-efficiency meta-hologram,¹³⁵ and plasmonic colors.¹²¹

Qiu et al.¹³⁶ proposed a deep learning-based method, named REACTIVE, to achieve the automatic design of a metasurface structure for a triple-band absorber application. The computation was 200× faster than a conventional parameter sweeping optimization. Given target S parameters indicated in Figure 9a, REACTIVE outputs the demanded metasurface structure with a design process shown in Figure 9b. The direct access to structure geometry from S parameters conforms to the second class. The designed meta-atom is a fourfold MIM structure with Cu that is described as a pattern matrix, as shown in Figure 9c. In the design process, an autoencoder-based dimension reduction method was employed to extract features. After that, REACTIVE adopted a DNN (MLP) to map the metasurface structure with the extracted feature. REACTIVE provides an average accuracy of 76.5%, which far surpasses those of other machine learning methods. Such direct derivation of design parameters from demanded responses can also be found in the graphene-based metamaterial.¹³⁷

Figure 9.

Inverse design example of the second class. (a) Target optical properties of S-parameters and the absorption rate. (b) Workflow of the proposed design scheme. (c) The 3D diagram of the model under investigation, which can be described as a matrix.¹³⁶ Reprinted with permission from ref (136). Copyright 2019 Wiley-VCH.

For the second class, designers usually develop the design model with two modules, inverse design and forward prediction (surrogate modeling). Regarding the structure of the meta-atom and the target optical response as a one-to-one mapping as a regression problem is inconsistent with physical intuition. The optical response of a meta-atom is fixed, so the forward modeling problem only has a unique solution. However, a specific optical response may correspond to a variety of completely different meta-atom structures, and this inverse design process is a multisolution problem. Therefore, a direct inverse network is difficult to converge. To solve this problem, tandem strategies have been used to avoid unstable training losses generated from a mismatch and force the inverse network to converge to a possible solution. A tandem strategy refers to the cascade of an inverse network for inverse design and a forward network for forwarding modeling. The input of the inverse network is the output of the forward network and vice versa. The forward network that solves the one-to-one problem is easy to converge, so the forward network is trained independently in advance. The inverse network is trained with the trained forward network, whose weights are fixed. In this way, the whole tandem neural network can also achieve one-to-one convergence. Inverse design schemes based on tandem neural networks (TNNs) demonstrated fascinating outcomes in various tasks.

Being invisible has been a long-time dream for humans. In recent years, the breakthrough of the metasurface has provided a promising way to achieve this dream. In 2021, Zhen et al. reported a transparent invisibility cloak with a transmitted metasurface.¹²⁸ Compared with previous metasurface cloaks that operate in a reflective manner, the transmitted cloak has wide application scenarios without the limit of ambient reflection. As shown in Figure 10a, this transparent cloak is composed of two planar metasurfaces. A dielectric cat model was set as the hidden object in this case. From the simulated magnetic fields in Figure 10b, the field with a cloak (right panel) remains flat, which proves the invisibility. The performance of the cloak is characterized as the near-field distribution and the far-field radar cross-section (RCS). The design scheme provides a phase profile arrangement of the metasurface. One desired EM response could relate to many candidates of the phase arrangement. To solve the one-to-many problem, the authors adopted the TNN that contained an inverse neural network NN1 and a forward neural network NN2, as shown in Figure 10c. The output of NN1 is the input of NN2. The input and output of TNN are both EM field distributions. The predicted phase arrangement was obtained from the intermediate layer. NN2 was first trained independently, and NN1 was trained with fixed-weight NN2. Figure 10d demonstrates the consistency among the target, the prediction, and the simulation. Therefore, the proposed TNN is an accurate design scheme.

Figure 10.

Inverse designs based on tandem neural network (TNN). (a–d) TNN for the design of invisibility metasurface cloak.¹²⁸ Reprinted with permission from ref (128). Copyright 2021 Optical Society of America. (a) Schematic of the transmitted metasurface cloak. Layer 1 and layer 2 are two planar metasurfaces, and a cat is a hidden object. PEC indicates a perfect electric conductor. (b) The left panel and the right panel are the simulated magnetic fields without and with the cloak, respectively. (c) The architecture of the proposed TNN. (d) Validation results of the proposed TNN. (e–g) TNN for the design of the MIM metasurface absorber.¹²⁵ Reprinted with permission from ref (125). Copyright 2021 Walter de Gruyter. (e) Diagram of a multiplexed array of plasmonic resonators. (f) Test results for dual-band and triple-band absorber designs. (g) Data preparation and inverse design schematic. (h) TNN for the on-demand transmission spectrum design of a thin-film meta-atom.¹³⁸ Reprinted with permission from ref (138). Copyright 2018 American Chemical Society.

Periodic MIM metasurface absorbers naturally yield strong resonances with high Q factors, and the resonance frequency can be shifted by varying the geometry of the resonator.¹³⁹ Yeung et al. used a tandem neural network to design a MIM metasurface absorber with multiple high Q-factor resonances and broadband absorption under the mid- and long-wave infrared regimes.¹²⁵ This case focuses on the supercell design, which is fourfold symmetric. Figure 10e shows the paradigm of a MIM supercell with Au and Al₂O₃. Each cross-shaped resonator has fixed widths and varied lengths. Figure 10f demonstrates two examples of dual- and triple-resonance designs whose performances are in good agreement with the target. As shown in Figure 10g, the resonator lengths are the design parameters. The mapping between absorption spectra and design parameters is handled by the proposed TNN. It is worth noting that the number of sampling points of the spectrum is 800, which is much larger than the tens quantity of this method’s counterparts. Additionally, dense sampling also promises efficacy and accuracy.

Liu et al.¹³⁸ also employed a typical TNN to deal with the data inconsistency in training, as shown in Figure 10h. The investigation object in this work was a thin all-dielectric multilayer-film meta-atom composed of alternating SiO₂ and Si₃N₄, as shown in the left panel of Figure 10h. Given the target transmission spectrum (see the right panel of Figure 10h), designers can retrieve the corresponding design parameter vector of the layer thickness.

The inverse design demonstrated in Figure 10 covers three levels of the whole metasurface, supercell, and meta-atom. Among them, meta-atoms have attracted more attention because of their flexibility and basic status. Figure 11 demonstrates three inverse design examples of meta-atoms via neural networks under the tandem strategy. As talked about before, the tandem strategy brings the consistency of input–output. Additionally, there is a significant benefit to training one neural network that carries surrogate modeling and inverse design simultaneously rather than developing both separately.¹¹³ Therefore, many inverse design methods take advantage of the tandem strategy but do not strictly follow the completely symmetrical structure of a TNN.

Figure 11.

Inverse design samples based on bidirectional neural networks under the tandem strategy. (a) Schematic of a bidirectional DNN. This discriminative model designed the “H”-shaped freeform nanostructure (on the left panel) on the demand of spectrum. This DNN is composed of a geometry-predicting network (GPN) for inverse design and a spectrum-predicting network (SPN) for surrogate modeling. Given the transmission spectrum and a material’s property, the GPN predicts the geometry of a nanostructure that meets those needs. SPN is an approximate spectrum solver for the given nanostructure’s geometry.¹¹³ Reprinted with permission from ref (113). Copyright 2018 Springer Nature. (b and c) Inverse design for meta-filter.¹¹⁴ Reprinted with permission from ref (114). Copyright 2019 American Chemical Society. (b) The working flow of the cascaded model generator and the spectrum prediction PNN. (c) Spectra of single-band (top) and dual-band (bottom) target and design results. (d–f) Two assembled neural networks for both forward and inverse predictions between the structure and optical responses.¹²⁹ Reprinted with permission from ref (129). Copyright 2018 American Chemical Society. (d) Designed unit cell with five design parameters. (e) Proposed neural networks and their data flow. (f) The optical responses involved include reflection spectra (LCP-to-LCP, RCP-to-RCP, and cross-polarization reflection) and the CD spectrum.

In the broadband from 600 to 1600 nm, Malkiel et al. proved that the geometry of a single nanoparticle could be found for arbitrary multiple resonances in the transmission spectrum under dual polarization.¹¹³ A bidirectional DNN for an “H”-shaped nanostructure design is shown in Figure 11a. The outer edges of the “H”-shaped structure can either vary in length and angle or be omitted during the design. The bidirectional DNN consists of a geometry-predicting network (GPN) for inverse design and a spectrum-predicting network (SPN) for surrogate modeling. Cascading the GPN and the SPN in a tandem structure provides better performance. Taking the MSE between the target and the prediction as the standard, the MSE of the bidirectional DNN is 0.21 lower than the MSE (0.37) of the separately trained GPN and SPN. Given the parameters of the nanoparticle geometry, SPN outputs the corresponding transmission spectrum under dual polarization. To evaluate the performance, the authors demonstrated a nanometer-scale geometry design that could address various resonances for chemical sensing applications. The designed nanostructures were used to enhance the light–matter interactions with various chemicals and biomolecules. Additionally, the deep learning prediction was also verified by the fabrication of the inferred structure with gold on an indium tin oxide (ITO) layer cover glass. Measurements on the fabricated metasurface conform to the design requirements.

In addition to arbitrary transmission spectra, the on-demand meta-atom design can also be used for the specific distribution transmission spectra of meta-filter applications. Compared with the design of the “H”-shaped meta-atoms, An et al. used more simple cylinder-shaped nanopillars over a broader range from 5 to 10 μm.¹¹⁴ As demonstrated in Figure 11b, the authors first developed a DNN called PNN that predicted the transmission coefficient from geometric parameters to save time during the simulation. The authors cascaded another DNN, a meta-filter generator, before the fully trained PNN with fixed weights under the tandem strategy. The meta-filter generator employs the target spectrum as the input and outputs geometric parameters. The geometric parameters are further delivered to the PNN to retrieve the corresponding transmission spectrum. The loss function in training is the Euclidean distance between the input and output spectra. In this cascaded network, both the input and the output are transmission spectra, which guarantees convergence. This design scheme achieved nearly 90% accuracy for on-demand Gaussian-shaped single-band spectra. Addtionally, Figure 11c demonstrates two examples of a meta-filter for single-band and dual-band targets. The result is in agreement with the target’s spectrum to a certain extent.

Chirality is another attractive design topic. Three-dimensional chiral metamaterials can also be automatically modeled and optimized via AI.¹²⁹Figure 14d–f demonstrates a machine learning scheme that contains two bidirectional neural networks under the tandem strategy aiming to solve surrogate modeling and the inverse design of a chiral meta-mirror simultaneously. As shown in Figure 11d, the chiral meta-atoms are modeled as two twisted Au split-ring resonators (SRRs) separated by two dielectric layers, with an Au reflector at the bottom of the structure. The five geometric parameters denoted in Figure 11d determine the degree of freedom of structural variation. Each network in Figure 11e has both a forward path for surrogate modeling and an inverse path for inverse design. Two networks were assembled by two combiners. Three reflection spectra (LCP-to-LCP, RCP-to-RCP, and LCP-to-RCP) and a circular dichroism (CD) spectrum quantitatively specify the design performance (Figure 11f). In surrogate modeling, this machine learning model predicts reflection spectra and the CD spectrum from chiral structure parameters. Given the demanding reflection and CD responses, the machine learning model can also retrieve geometric parameters for inverse design. In the evaluation of an i7-CPU, the optical response prediction speed was about five orders of magnitude faster than the numerical simulation. Researchers normally cascade the inverse network and the forward network via the same variable of geometry. However, the tandem structure can also be achieved by cascading the two networks via the same spectrum variable.^140,141

Figure 14.

Examples of a VAE. (a–d) An inverse geometry design for multiple BIC frequencies.¹³¹ Reprinted with permission from ref (131). Copyright 2021 Optical Society of America. (a) Schematic diagram of a crossed neural network architecture. The green and blue components form a VAE for nanostructure generation. The vector b contains BIC frequencies. Latent vector z could be regarded as a low-dimensional representation of the geometry. CNN₁ reads z out from b, while CNN₂ does the opposite work. (b) The data flow during network training, where 1 is for VAE, 2 is for CNN₂, and 3 is for CNN₁. (c) Data flow for inverse design and surrogate modeling. (d) Demonstration of the generative ability. Variations in one component of the latent vector lead to slightly deformed geometries, like geometry 8 becomes geometry 12, with the sites of deformation marked by red arrows (top). Additionally, the shift of TE-like and TM-like bands at Γ-point varies as the latent vector continues (bottom). (e–g) VAE for the MIM meta-atom design.¹⁴⁵ Reprinted with permission from ref (145). Copyright 2019 Wiley-VCH. (e) Working flow of inverse generation and forward prediction between the geometry and reflection spectra. (f) Two validation graphs. (g) The architecture of the proposed VAE.

Similarly, concentrating on the reflection regime, So et al. used the tandem-structured DNN to design the MIM grating meta-atom.¹⁴² In this case, the working wavelengths focus on the visible range. As shown in Figure 12a, a five-layer grating structure of alternating Al and SiO₂ was under investigation in this work. The five design parameters, including the grating period, the width, and the thicknesses of the three layers, are also indicated in Figure 12a. Figure 12b demonstrates the architecture of the adopted neural network. The input contains a custom-defined reflection spectrum and specified resonant wavelengths, as shown in Figure 12c. To validate the efficacy of the proposed design scheme, the authors fabricated a designed meta-grating, as shown in Figure 12d. The reflection spectra of the target, the simulation, and the actual fabrication are all in good agreement. To further test the design accuracy, the authors designed a series of single-band absorbers with gradually changing target resonant wavelengths (see Figure 12f). We can see that the shift in resonance becomes greater closer to the edge of the wavelength range.

Figure 12.

Meta-atom design via tandem-structured neural networks. (a–f) MIM meta-grating design for the target reflection spectrum.¹⁴² Reprinted with permission from ref (142). Copyright 2021 Optical Society of America. (a) 3D Diagram of the concerned meta-grating with design parameters. (b) Adopted network architecture. (c) Example of the reflection spectrum. The yellow dots indicate the resonant wavelengths. (d) Scanning electron microscope (SEM) image of one design. (e) Verification result with the actual fabrication experiment. (f) Target spectra and corresponding designed responses for the single-resonant experiment. (g and h) Multilayer nanoparticle design for target extinction spectra.¹⁴³ Reprinted with permission from ref (143). Copyright 2019 American Chemical Society. (g) 3D diagram of concerned nanoparticles with design parameters. (h) Concerned optical responses and the proposed DNN architecture.

Besides the geometric parameters, NN-based inverse design can also take the choice of material into consideration. So et al. reported an inverse design scheme for three-layer nanoparticles.¹⁴³ The relevant design parameters include the material choices and the thickness configuration for each layer, as shown in Figure 12g. Additionally, the involved optical responses are extinction spectra of electric and magnetic dipoles. The design network also follows the tandem architecture (see Figure 12h). It is worth noting that the prediction of continuous geometric values is a regression problem, while the material configuration is a classification problem. The loss function calculates the regression and classification losses simultaneously to promise unbiased convergence.

The tandem strategy restricts design diversity to the data diversity of the forward network, which leads to limited generalization capabilities. However, this is also the characteristic of the discriminative model. To mitigate the challenges of low diversity, generative models are applied to learn the many-to-one mapping between the design space and target space. VAE and GAN are two representatives of the generative model. Their detailed structures and principle are discussed in section 2.2.3.

A VAE is composed of an encoder and a decoder. We briefly explain the utility of VAE in image generation. The encoder in a VAE encodes an image as a latent vector, which is a point in the latent space. After training, latent vectors with the same attributes are clustered together in the latent space. The training goal of a successful VAE is to derive a sampling interval in the latent space. A new image with the same attributes can be reconstructed from a latent vector by the decoder in a VAE, which is the generation principle of a VAE. For example, a data set is composed of a large number of nanostructure patterns.¹⁴⁴ These patterns can be divided into eight groups, as shown in Figure 13a. Figure 13b demonstrates the 2D latent space distribution of the data set in Figure 13a by reducing the dimensions from 20 to 2. Each color represents a shape, and each point is a geometry pattern image. From the visualization, we can conclude that points of the same shape are clustered together, and similar shapes are also closer in space. We can generate corresponding types of geometry graphics by sampling in the space corresponding to a certain color. The two distributions in Figure 13b are encoded by two different VAE models. It is clear that the right panel is better, with more considerable interclass distances. Therefore, the designers could generate a massive number of candidates for a specific target with a VAE.

Figure 13.

Generation principle of latent space sampling. (a) Examples for nanostructures data set of eight shapes. (b) Visualization of the latent space via dimension reduction. The two distributions of the data set in panel a were derived from two different encoders.¹⁴⁴ Reprinted with permission from ref (144). Copyright 2020 Science China Press.

The optical bound states in the continuum (BICs) refer to states that areis located in the continuous radiation spectrum but remain completely confined.¹⁴⁶ In applications, a weak disturbance can turn BICs with infinite Q factors into leakage resonances with finite but high Q factors. Based on BICs, applications that make extensive use of high Q factors have shown unprecedented potential, such as sensors,¹⁴⁷ lasers,^148,149 and nonlinear optics.¹⁵⁰ Conventional design based on BICs is the parameter sweeping of regular geometries, which is limited and time-consuming. Therefore, inverse design for arbitrary multiple BIC frequencies in free-form structures is much more fascinating. A recent study demonstrated that inverse design could support on-demand BIC design. As shown in Figure 14a, Lin et al. developed a composite-crossed neural network architecture for the on-demand design of BICs.¹³¹ This cross-architecture contains a VAE for nanostructure geometry generation and two CNNs for property readout. The latent vector z between the encoder and the decoder in the VAE is a low-dimensional representation for each geometric design. Components in b are BIC frequencies. CNN₁ reads z out from b, while CNN₂ does the opposite work. The training sequence and data flow are demonstrated in Figure 14b. Route 1 is for the VAE, route 2 is for CNN₂, and route 3 is for CNN₁. The mapping relationship from z to b is unique. The training of CNN₂ is easy to converge. However, the mappings from BIC frequencies to the possible geometry are not limited. In the training of CNN₁, CNN₁ and trained CNN₂ with fixed weights are bound together. After training, parts of different NNs are recombined to perform surrogate modeling and inverse design, as shown in Figure 14c. With the latent vector z as the intermediate variable, authors could freely predict a geometry with predefined symmetry from BICs and vice versa. Figure 14d proves the generative ability. Each z is a sampled point in a Gaussian distribution. Variations in the latent vector lead to slight geometric deformation, such as the difference between geometry 8 and geometry 12. The bottom diagrams in Figure 14d demonstrate obvious changes in band frequencies as the geometry slightly varies.

Out of the heuristic intuition of the optical structure design, similar geometries behave similarly in terms of their optical properties. The generation of such a predefined geometry conforms with the reproduction in the VAE. With a specially designed VAE, Ma et al. focused on meta-atom pattern design according to the requirements of the reflection spectra of different polarizations, including x-in-x-out, y-in-y-out, and the cross-polarized reflection.¹⁴⁵ The meta-atom under investigation is a sandwiched MIM structure, as shown in Figure 14e. The design result is discretized to a 64 × 64 2D binary image. Validation results in Figure 14f show good consistency with the ground truth, which is important for on-demand design. Figure 14g shows the architecture of the proposed VAE, which can be divided into three parts: a recognition model for geometry encoding, a prediction model for reflection spectra, and a generation model for geometry reconstruction. In training, with inputs of both pattern image features and reflection spectra, the recognition model learns the latent distribution of geometries conditioned on reflection responses. This deep generative model is trained in an end-to-end manner, which avoids nonconvergence and design diversity limitations.

GAN is another powerful generative method for on-demand inverse design. In the original GAN, the input of the generator was a random vector. For on-demand design, the researcher normally adopts the conditional generative adversarial network (cGAN), whose input has one more variable such as the demanding spectra. At the same time, the architectures of the generator and the discriminator are not limited to multilayer perceptron (MLP). For convenience, we use GAN for short in the following text. With sufficient generative capabilities, GAN holds promise to design challenging free-form meta-atoms with a high degree of freedom.¹⁵¹

Liu et al. proposed a GAN scheme of a meta-atom design for custom-defined optical spectra. When fed arbitrary transmission spectra across the range of 500 nm to 1.8 μm, the generative model provided possible structure patterns with a high matching degree and fidelity. As Figure 15a shows, the proposed GAN is composed of a generator and a critic (discriminator). As the name GAN suggests, the generator and the critic are adversaries. In training, the images generated from the generator are fed to the critic. The critic is responsible for judging whether the generated data and the real data are in the same distribution (same class). After constant adversarial training, the generator learned the inherent probability distribution model of the pattern of a natural meta-atom. Therefore, the generator can generate a demanding geometry rather than a random noise graph. The simulator adopts a CNN to retrieve the optical response of the specific geometry, as shown in Figure 15b. With backpropagation from the simulator and the critic, the generator can achieve the automatic design of the meta-atom pattern.

Figure 15.

Overview of the GAN-based inverse design scheme. (a and b) On-demand meta-atom design with custom-defined transmission spectra.¹¹² Reprinted with permission from ref (112). Copyright 2018 American Chemical Society. (a) Architecture of the proposed GAN with a simulator for surrogate modeling. The GAN is composed of a generator and a critic (discriminator). T is demanding transmission spectra, T̂ is the spredicted pectrum of the designed pattern generated from the generator, z is the noise, and l is the authenticity of the generated image, which is determined by the critic. (b) Example of the meta-atom under investigation and the concerned transmission spectra. (c–e) Design of a free-form meta-atom on the basis of the reflection performance.¹⁵² Reprinted with permission from ref (152). Copyright 2019 Walter de Gruyter. (c) Schematic of the proposed GAN architecture composed of the generator (left) and the discriminator (right). (d) The data prepared for deep learning with pair of a pattern image and a reflection spectrum. (e) Validation of the generation capability.

Subsequent works further demonstrated the design potential of the GAN in applications that required various optical properties. For example, So et al. also used the GAN to design meta-atom geometry patterns for custom-defined reflection spectra.¹⁵² The specific network architecture is demonstrated in Figure 15c. The GAN was built to design free-form meta-atoms, as shown in the bottom panel of Figure 15d. The nanostructure was predefined as an Ag antenna on a stacked MgF₂ spacer, a Ag reflector, and a Si substrate with a 500 nm lattice size. Given the noise and the target spectrum, the generator outputs the corresponding probability distribution of the structure pattern via a series of up-sampling processes. The discriminator tries to distinguish the generated images from the training set. The output of the probability distribution is a fast way to force the network to converge. Because the direct binary output is an extreme nonlinear process whose loss function has less restriction on the weights, the binary demanding output image can be easily obtained by an automatic thresholding algorithm. Most interestingly, the design performance shows great consistency with the target spectrum (see Figure 15e), while the generated structure is completely different from the known ground truth. This proves that the proposed design scheme has good generation diversity.

It is universally acknowledged that adjoint-based topology optimization¹⁵⁴ is computationally expensive. Jiang et al. reported a GAN accelerated free-form diffractive meta-grating design scheme.¹⁵³ The architecture of the proposed GAN is shown in Figure 16a. The inputs are the target outgoing angle λ ranging from 35 to 85 degrees and the working wavelength ranging from 500 to 1300 nm. The pattern of the designed structure is symmetric about the y-axis, as shown in Figure 16b. The authors used the GAN to generate meta-atom patterns quickly and massively. The 50 candidates with top deflection efficiencies were selected from all generated pattern images. These candidates were further refined by iterative optimization. Most interestingly, the initial training set of this GAN was only 600 samples, which was much smaller than its counterparts’ thousands or ten thousand samples. Fewer initial training sets mean a lower precomputation cost. The performance of the proposed GAN is enhanced via retraining, which is demonstrated in Figure 16c. Additionally, the training data for the second-generation GAN are the refined images from the GAN and additional normal samples. With a retraining loop, the efficacy of the GAN can be improved. The GAN-based generator can be regarded as a computationally efficient design tool. As shown in the left panel of Figure 16d, the proposed scheme behaves about 5000× faster than the iterative-only optimization method. Besides saving time, the GAN-based method also demonstrates an ability to generate designs with a higher deflection efficiency (see right panel of Figure 16d). Similar work that provides design candidates closer to the optimum via the generative network was applied to the design of meta-gratings.¹⁵⁵

Figure 16.

Generative networks for inverse design with a high degree of freedom (DOF). (a–d) Free-form pattern generation with a high efficiency via a GAN.¹⁵³ Reprinted with permission from ref (153). Copyright 2019 American Chemical Society. (a) Architecture of the proposed GAN. (b) The paradigm of the symmetrical meta-atom pattern. The dashed blue line indicates the symmetry line. (c) Working flow of the GAN retraining. (d) Validation results of the proposed GAN on the basis of the computational time and the efficiency distribution. (e and f) Phase map design of the whole metasurface with the VAE-GAN framework.¹²⁶ Reprinted with permission from ref (126). Copyright 2021 Optical Society of America. (e) Schematic of the proposed VAE-GAN framework. (f) The designed phase profile of the coding metasurface (left), the corresponding holographic imaging result (right-top), and the unit cell in the coding metasurface (right bottom).

Recently, a kind of metasurface raised attention for its easy programming, which is called a coding, programmable, or digital metasurface. With binary-coded phase responses, the meta-atom in such a metasurface can be controlled easily and in real time. Therefore, coding a metasurface is an excellent subject for AI technology. AI has already demonstrated intelligent control in coding metasurface applications, such as imager¹⁵⁶ and smart system.¹⁵⁷ A real-time reprogrammable metasurface controlled by a field-programmable gate array (FPGA) is promising for holographic imaging applications. However, for meta-hologram design, both the Gerchberg–Saxton (GS) algorithm¹⁵⁸ and the stochastic optimization algorithm are iterative approaches, which cannot support the demand of real-time response. Liu et al. proposed a new method composed of a VAE and a GAN.¹²⁶ As shown in Figure 16e, the proposed new method uses a VAE structure to realize image reconstruction in which a forward propagation process is integrated as the decoder. The propagation process calculated the electric field distribution from the current at the source point. The discriminator followed the structure of the GAN. Their meta-hologram under investigation was a 40 × 40 coding metasurface at 35 GHz. Each meta-atom radiates 0-phase or π-phase EM waves, which correspond to the 0 or 1 in the binary coding, respectively. With a demanding phase profile, this meta-hologram is expected to project the corresponding image at an imaging plane 30 cm away, as shown in Figure 16f. The loss function comprehensively considers both the MSE from the VAE part and the Wasserstein distance¹⁵⁹ from the discriminator. Compared with the conventional mainstream method, GS, the proposed VAE–GAN shows a lower MSE and a higher peak signal-to-noise ratio (PSNR). Such combined VAE and GAN network architectures also perform well in the design of the high-efficiency thermal emitter. Kudyshev et al. proposed a similar framework called adversarial autoencoders (AAE) to optimize the structure topology for high efficiency.¹⁶⁰

Because original GANs take random vectors as the input, the initial results are normally noisy for the inverse design. For example, the edges of the nanometer-scale geometric design are jagged, leading to challenging fabrication in practice. Designers need extra postprocessing such as smoothing to refine the structures.

The works talked about in Table 2 cover the inverse design of meta-atoms, supercells, and whole metasurfaces. For the meta-atom design, the design variables are either geometric parameters or structure patterns, similar to the description in surrogate modeling. The design of the supercell contains a geometry configuration for multiple unit cells. Additionally, the inverse design of a whole metasurface normally focuses on the phase profile. The shapes of discussed meta-atoms vary, as they can be free-form, have a specific structure, or have a symmetry limit. The involved algorithms can be divided into discriminative models and generative models (GAN and VAE, respectively). Additionally, the working bands mentioned in Table 2 penetrate visible light, infrared light, and microwave wavelengths. The inverse design scheme can work for both plasmonic and dielectric metasurfaces.

Table 2. Summary of Inverse Design.

algorithm	working bands	materials	lattice period	input	output	design specification	training set	validation set	test set	application	accuracy	runtime	hardware condition	ref
DNN	400–800 nm (200 points)	Ag and SiO2		5 GPa	scattering cross section	on-demand	50 000	6250	6250		>98%	0.01–1 s	Two 2.9-GHz Intel Core i5 processors	(116)
GA	30–60 THz (31 points)	all-dielectric		frequency and permittivity	4 GPa	maximize phase coverage				meta-atom		22 s	quad-core CPU with a 3.5 GHz clock speed and two Nvidia 1080Ti GPUs	(114)
DNN		all-dielectric	1.23 μm	transmission spectrum		on-demand	40 000		40 000	meta-fileter	single-band 89.76%	4.8 ms
											dual-band 67.95%
GA		GST	1.23 μm	operating frequency		maximize phase coverage				phase-coding metasurface		253 s
DNN (tandem)	400–700 nm (101 points)	Al and SiO2		reflection spectrum and resonant wavelengths	5 GPa	on-demand	9680	1210	1210	meta-absorber	MSE 0.023		Nvidia GTX 1080 Ti GPU	(142)
CNN and DNN	30–80 THz (201 points)	Au and dielectric	2.5 μm	reflection spectra and CD spectrum	5 GPa	on-demand	25 000		5000	chiral meta-mirror			i7 CPU personal computer	(129)
DNN	600–1650 nm (43 points)	Au and ITO-covered SiO2		transmission spectra under dual-polarizations and material property	8 GPa	on-demand	>10 000			chemical sensing	MSE 0.16			(113)
DNN (tandem)	300–750 THz (200 points)	SiO2 and Si3N4	500 nm	transmission spectrum	16 GPa	on-demand	500 000		50 000	transmission phase modulator				(138)
DNN	300–1000 nm (100 points)	free selection in Ag, Au, Al, Cu, TiO2, SiO2, and Si		extinction spectra of electric and magnetic dipoles	three material choices and 3 GPa	on-demand	14 400	3600	1800	dipole resonance engineering	MSE 0.00943	<1 S	Nvidia GTX 1080 Ti GPU	(143)
AE and DNN	6.1, 17.3, 19.1 GHz	Cu and dielectric	10 mm	S-parameters	4 × 4 SPb	on-demand	1995		105	triple-band absorber	76.50%	in seconds	Intel(R) Core i5-8250U CPU	(136)
GAN	170–600 THz (32 points)	Au and SiO2	340 nm	transmission spectrum	64 × 64 SPb	on-demand	6500				>90%		Quadro P5000 GPU	(112)
VAE	40–100 THz (31 points)	Au and dielectic	2 μm	reflection spectra	64 × 64 SPb	on-demand	20 000		1000	chiral meta-mirror				(145)
VAE	40–100 THz (61 points)	resonator and spacer	2 μm	reflection spectra	64 × 64 SPb	on-demand	16 000		8000	reflective metasurface	MSE 0.001			(144)
VAE and ES	170–600 THz (32 points)	Au	380 nm	transmission spectrum	64 × 64 SPb	on-demand	10 000		500		>95%	<5 s		(124)
GAN	2–18 GHz (128 points)	metal and F4B	10 mm	reflection spectra	32 × 32 SPb	on-demand	7500		2500	anisotropic metasurface	>80%	3.5 min	NVIDIA GeForce GTX 2080Ti GPU	(161)
GAN	250–500 THz (200 points)	Ag, MgF2, and Si	500 nm	reflection spectrum	64 × 64 SPb	on-demand	8104	2026	20		average loss 0.0056	<3 s	GTX 1080-Ti GPU	(152)
CNN and VAE	1200 - 1700 nm	Si	1 μm	three BIC frequencies	100 × 100 SPb	on-demand	18 000	1000	1000	BIC design	discrepancy around 2.5% of the shortest operation wavelength			(130)
GAN and iterative optimization	500–1300 nm (17 points)	Si and SiO2		operating wavelength and deflection angle	128 × 256 SPb	largest absolute efficiency	600 (+6000)		935 000	meta-grating		29.6 h	Tesla K80 GPU	(153)
DNN (tandem)	4–12 μm (800 points)	Au and Al2O3	∼10 μm	absorption spectra	25 GPa	maximize resonance efficiency	3000		300	meta-absorber	>90%	<1 min		(125)
DNN (tandem)	7.5–9 GHz		20 mm	near-field distribution and far-field RCS	8 × 2 phase arragement	on-demand	140 000	40 000	20 000	transparent invisibility cloak	near-field 92.4%			(128)
											far-field 93.2%
VAE and GAN	35 GHz	PIN diode and Roger RO 4003C		holographic image	40 × 40 binary phase map	on-demand	35 000	5000	10 000	meta-hologram	mean MSE 0.0382		Nvidia Quadro RTX 8000 GPU	(126)
											mean PSNR 14.33

^aGP is the abbreviation for geometric parameters.

^bSP is the abbreviation for structure pattern.

In free-form meta-atom structure design, some researchers use a fine discretization grid for a high DOF. While the several-pixel vibration of the structure pattern sometimes leads to great variation in optical properties, the fabrication deviations are common in practice. Therefore, this is likely due to the fact that the theory exists and the experiment is not feasible. Considering the current feasible manufacturing scale, it is difficult to manufacture binary structure pattern designs with pixel sizes less than 10 nm. Considering the lattice period and the discrete levels of patterns, some works are overfine, which is difficult to fabricate. Fine discretization brings a high degree of freedom to the design, but the manufacturing complexity should be considered in the actual configuration.

The majority of inverse design schemes are for automatic on-demand design, and a minor part is to solve the optimization problem. Most algorithms consume over ten thousand training data. Due to semi-supervised learning, methods based on generative models require less training data (usually thousands). The designers will normally demonstrate an application example for their specially constructed design scheme and that the validation results are in line with expectations.

NN-based inverse design requires less knowledge of optics. What the neural network provides is only an approximate solution of the system, which is not exactly the same as the target requirements. Most methods exhibit an accuracy of over 70% for on-demand design at a considerable speed. Inverse design of the conventional trial-and-error mode is time-consuming and cannot guarantee the solution. Despite the discrepancy, a quasi-solution is better than no solution.

3.2.2. Gradient-Free Evolutionary Computation

Evolutionary computing is a crucial subfield of artificial intelligence and is a family of metaheuristics, including genetic algorithm, evolutionary algorithm, ant colony algorithm, and particle swarm algorithm,. It is inspired by the natural selection mechanism of the survival of the fittest and the influence of the law of the transmission of genetic information. Evolutionary computation mimics the process of biological evolution. The race multiplication process is simulated through the iterative process of computer programs. Each generation introduces mutations as small random changes, and the unqualified solutions are discarded via choice. In this way, the optimal solution is finally obtained through evolution. Benefiting from guided random search in the solution space, evolutionary computation is designed to search for the global optimal solution. Thus, evolutionary computation is normally acknowledged as a collection of global optimization algorithms.

Genetic algorithm (GA, sometimes named evolutionary algorithm)¹⁶² is one of the most frequently used evolutionary computation strategies. Additionally, GA has greatly promoted the inverse design of metasurfaces in recent years, such as a metalens,¹⁶³ a terahertz quarter-wave plate,¹⁶⁴ programmable metamaterials,¹⁶⁵ and subwavelength lattice optics.¹⁶⁶

Figure 17a demonstrates a general working flow of GA. GA usually takes a set of randomly generated solutions or artificially set initial points as the first generation of the population. Every subsequent iteration of the algorithm is a new generation of the population. The objective function is built according to the requirements of the optimization problem, and the objective function is used to evaluate the fitness of the individuals in the population. Individuals with low fitness are abandoned, and individuals with high fitness are selected to generate a new generation of the population through crossover and mutation. Crossover refers to the replacement and recombination of the parental generation, while mutation is the random change of certain attributes of an individual. As a result, the fitness result of each new generation is better than that of the previous generation, and the population evolves in the iterative process. The final algorithm ends when it meets the fitness requirements or reaches the set number of iterations.

Figure 17.

Use of a GA and GA variants. (a) Flowchart of the general GA. (b) Flowchart of the GA for the generation of the coding metasurface. The coding metasurface works for absorption bandwidth broadening.¹¹⁸ Reprinted with permission from ref (118). Copyright 2021 American Chemical Society. (c) Design diagram of a genetic-type tree search (GTTS) that integrates Monte Carlo tree search (MCTS) and a GA.¹⁶⁷ Reprinted with permission from ref (167). Copyright 2021 American Chemical Society. (d) Application for a large-pixel metasurface design with a segmented hierarchical GA. The genome in the lower segmentation is composed of fewer pixels than that in the higher segmentation. Stage separation reduces the amount of computation. The cyan and yellow blocks represent pixel states denoted as values “1” and “0”, respectively.¹⁶⁸ Reprinted with permission from ref (168). Copyright 2019 American Chemical Society.

In the specific application design, GA is modified to obtain various variants to meet different requirements. In a phase profile design of a PB phase-controlled metasurface light sheet, the adopted modified GA removed the crossover operation for processing efficiency.¹⁶⁹

Microwave-absorbing materials have become prolific because of military radar stealth and commercial antenna telecom protection. For address the increasing demand for broadband absorption, strategies combining a metasurface and the substrate of the material raise researchers’ interests. The absorption bandwidth can be tremendously improved by loading a metasurface onto the substrate of the absorbing material. Zhang J. et al. proposed an efficient metasurface to broaden the absorption bandwidth of the substrate with the support of GA.¹¹⁸ To meshing and coding the meta-atom of the metasurface (see Figure 17b), they employed the absorbing performance as the fitness value. Under the indication of the fitness, the coding pattern is optimized in iterations. In the validation experiment, the authors demonstrated the absorption bandwidth increased from 4.5 to 13.5 GHz with the optimized metasurface. Additionally, this GA-based strategy was proven to be universally applicable to various absorbing materials and is a universal design method.

When the phase modulation range is less than 2π, or even less than π, the metasurface can still exhibit a strong beam -steering ability. As shown by the example in Figure 17c, Lin et al. proposed a method named genetic-type tree search (GTTS), which originated from Monte Carlo tree search and a genetic algorithm with an unsupervised k-means clustering algorithm, to inversely design the high-performance beam-steering metasurface.¹⁶⁷ They demonstrated three Al-based beam-steering metasurfaces for different wavelengths and a tunable beam-steering metasurface with Au–-graphene units. Given the target steering angle, GTTS can output the corresponding amplitude and phase profile, which are sampled as 30 points. The angular resolution and steering angular range of beam deflection design could reach 0.5° and ∼25°, respectively.

In previous works, the Gerchberg–Saxton (GS) algorithm was extensively applied for meta-hologram design. Compared with the GS algorithm, inverse design based on GA shows a preponderance in search space, computing speed, image quality, and image pixels. However, slow convergence caused by the large search space hinders the application of GA. Jin et al. introduced a segmented hierarchical evolutionary algorithm to decrease the computation load of complex and large-pixelated inverse meta-hologram design.¹⁶⁸ They divided the evolutionary process into two stages, as shown in Figure 17d. Taking the meta-hologram as a binary amplitude array, Jin Z called this array the genome in GA. The low-resolution genome surrogate was first evolved in the early stage, which had a low computational cost. In the later stage, the genome with the actual resolution was initialized using the evolutionary result of the low-resolution genome surrogate and further evolved. This segmented hierarchical evolutionary algorithm greatly reduces the size of the search space in the initial search stage and significantly improves the convergence speed.

Evolution strategy (ES) is also an optimization technique based on ideas of evolution. ES is very similar to GA. To eliminate the limitations of GA’s binary coding, ES uses real numbers to encode genotype sequences and introduces mutation strength. Unlike the direct conversion of 0 and 1 in GA, the mutation in ES adds a random vector to the current real-value DNA sequence. This random vector conforms to the normal distribution, with a standard deviation of the mutation strength. The coding chain of the mutation strength will also perform a crossover operation and be passed on together with the real-value genes of the individual from the parent to the offspring. The covariance matrix adaptation evolution strategy (CMA-ES) is an adaptive version of ES. CMA-ES will calculate and update the covariance matrix of the entire parameter space. By adaptively adjusting the genotype and the mutation strength, the size of the search space can be increased or decreased in the next-generation search. A simple metasurface design scheme of the CMA-ES is illustrated in Figure 19a. The CMA-ES part is responsible for the random global solution search, and the simulation tool is in charge of the specific optical response of the found mathematical solution.

Figure 19.

Optimization examples with the evolution strategy. (a) Flowchart of the PSO algorithm working together with the optical simulation software.¹⁷¹ Reprinted with permission from ref (171). Copyright 2017 The Japan Society of Applied Physics. (b) Schematic diagram of a homogeneous chromatic lens with chromatic aberrations (top left) and an apochromatic GRIN metasurface lens (bottom left) and the ratios of the lens parameters from the theoretical prediction and the CMA-ES optimization for the front radius of curvature R_f, the thickness T, and the metasurface power ϕ_m versus f/# (right side from top to bottom).¹⁷² Reprinted with permission from ref (172). Copyright 2018 Optical Society of America. (c) Illustration of a diatomic meta-molecule consisting of two meta-atoms (top) and the hybrid design scheme (bottom). The CPPN decoder is a neural network that generates the pattern of the meta-atom at a pixel level. Each species is composed of control parameters for the pattern shape.¹⁷³ Reprinted with permission from ref (173). Copyright 2020 Wiley-VCH. (d) Schematic diagram of a hexagonal metasurface-enabled A-SBFA.¹⁷⁴ Reprinted with permission from ref (174). Copyright 2019 Springer Nature. (e) Flowchart of the hybrid VAE-ES optimization loop. The optimization target is the latent vector v. Each v has a mutation strength m. The generator (G) is separated from a well-trained VAE to generate structure pattern S from the sampled latent vector V. The applied ES covers fitness evaluation, selection, reproduction, and mutation. q̂ is the simulation results, q is the desired optical property, and r is the fitness score.¹²⁴ Reprinted with permission from ref (124). Copyright 2020 IEEE Circuits and Systems Society.

Sieber and Werner compared the performance of CMA-ES with that of binary GA for metasurface design in 2014.¹⁷⁰ The computational efficiency and the quality of the solution found were contrasted between the Bézier metasurface optimized by CMA-ES and the pixelized metasurface optimized by binary GA for the application of infrared broadband quarter-wave and half-wave plates. As shown in Figure 18, CMA-ES far exceeds GA not only in the convergence speed but also in the final performance. CMA-ES showed a performance closer to the ideal, with a significant reduction in evolutionary generations.

Figure 18.

Performance comparison of the pixelized metasurface (GA optimizations) vs the Bézier metasurface (CMA-ES optimizations). The averages, means, and best fits over 20 seeds for the two algorithms are demonstrated. The use of the same seeds ensures the two algorithms have the same randomness, which means their initial random populations are the same. The fitness value reflects the error from the ideal performance. The smaller the fitness value, the better.¹⁷⁰ Reprinted with permission from ref (170). Copyright 2014 Optical Society of America.

CMA-ES was employed to compute the values of capacitors and resistors in reconfigurable metasurface absorbers.¹⁷⁵ In conjunction with the commercial optical simulation tools, CMA-ES can optimize the dimensions of each meta-atom. It was used for the nanostructure optimization of metasurface-enabled waveplates¹⁷⁶ and a gradient-index (GRIN) metasurface lens.¹⁷⁷ CMA-ES also demonstrated topology-optimization capabilities in thermal cloaks.¹⁷⁸ As a global optimizer, CMA-ES provides designers with the confidence to achieve optimal target performance. Nagar et al. employed CMA-ES to maximize the three-color (apochromatic) correction performance of the GRIN metasurface lens, as shown in Figure 19b.¹⁷² The optimized lens parameters demonstrate consistency with those predicted by paraxial analytical theory. Comparison ratios for the front radius of curvature, the thickness, and the metasurface power all vary slightly at the beginning and quickly approach unity.

Working with other optimization methods, the application of CMA-ES is more flexible and powerful. Elsawy et al. combined CMA-ES with statistical learning.¹⁷⁹ They achieved GaN phase gradient metasurfaces in the visible range with over 85% diffraction efficiency. Neural networks and evolutionary computation are not clearly separated in practice. Designers can use both algorithms simultaneously in inverse design. The cooperation between deep learning and the evolution strategy is a new direction for the inverse design of a metasurface. The combination alleviates the problems of each method, namely the low convergence speed in evolutionary computation and local optima in deep learning. As shown in Figure 19c, Liu et al. proposed a hybrid AI-based design framework composed of deep learning and an evolution strategy.¹⁷³ Following the hierarchical relationship in materials science, they named the working units on the metasurface that consisted of several different meta-atoms as meta-molecules. The specific procedure was realized on the meta-atom design. A generative network is responsible for generating the pattern of the meta-atoms on the basis of the shape-controlling parameter. The workflow of the cooperative evolutionary part follows the conventional evolution strategy. Neural network and evolution steps form a loop together. The pattern of the meta-atom evolves in multiple iterations until it reaches the target. Their hybrid design casts off the conventional laborious manual trial and error simulations and performs the metasurface design conveniently and systematically.

Liu et al.¹²⁴ also developed a hybrid design strategy based on modified ES and deep learning. Given the customer-defined input spectra, metasurfaces with continuous or discrete topological structures are generated automatically. An advanced short backfire antenna (A-SBFA) enhanced with metasurface demonstrated unprecedented performance on aperture efficiency across two working bands, as shown in Figure 19d.¹⁷⁴ In this work, the metasurface is used to manipulate the electric field. The key to the design of the corresponding metasurface is the arrangement of the impedance values. Inverse analysis of A-SBFA is difficult to achieve because of its functional complexity. CMA-ES is employed together with the electromagnetic simulation tool, HFSS, to search the optimal settings globally. To maximize the peak directivity in the two GPS working frequency bands L1 (1.575 GHz) and L2 (1.227 GHz), the optimization variables contain the dimensions of proposed A-SBFA such as geometry and metasurface response like reactance values at working bands. Authors did not place restrictions on the searching of metasurface reactance values. The search space covers all possible design parameters.

Liu et al. proposed a hybrid strategy that consolidated a VAE and a modified ES, as shown in Figure 19e.¹²⁴ The design objective was to optimize the structure topology in binary image format to fit the on-demand transmission spectra. The VAE encodes the input binary topology into a latent vector and decodes the sampled latent vector to retrieve a reconstructed pattern. With a generator separated from a well-trained VAE, a slight variation in the latent vector can produce a new structure topology. The ES algorithm was employed on the topology patterns generated from the latent vector to find an optimum with the fittest optical responses. As shown in Figure 19e, the latent vector v can be regarded as the genotype in the ES. Reproduction and mutation can be easily applied to the vector representation. In the optimization loop, a new structure pattern s is generated from the latent vector v. The pattern s is then sent to the simulator to obtain the corresponding transmission spectra. The fitness score is the difference between the simulation result q̂ and the target q.

Particle swarm optimization (PSO) simulates the foraging behavior of a flock of birds and is a random search algorithm based on group collaboration. Birds cooperate in searching for food. They know the distance from themselves to the food, but they do not know the specific location of the food. Therefore, the foraging strategy of a flock of birds is to get close to the bird closest to the food and then keep searching in the vicinity of this bird. PSO uses a particle, the candidate solution, to simulate each bird. Unlike GA, PSO does not have crossover and mutation, and the particle updates its speed and direction through both its own optimal solution (personal best) and the population optimal solution (global best).¹⁸⁶ PSO’s information from the global best is one-way, which is a search method that follows the update of the global optimal value.¹⁸⁷

PSO is commonly used as a global search method for nanostructure parameter optimization in the reverse design of meta-optics.^188−190 Optimization targets are used as the evaluation criteria for PSO. Common optimization targets the efficiency maximum and the phase-amplitude fitness. The specific fitness evaluation is executed by commercial electromagnetic simulation software. In conjunction with the simulation assessment, PSO could greatly improve the optical performance of the designed optics after numerous generations of evolution.^191,192 The corresponding procedure is shown in Figure 20a.¹⁸⁰

Figure 20.

Various metasurfaces were designed with PSO. (a) Flowchart of the PSO algorithm working together with the electromagnetic simulation software, CST Microwave Studio.¹⁸⁰ (b) The micro-coding unit cell of the metasurface.¹⁸⁰ Reprinted with permission from ref (180). Copyright 2017 Springer Nature. (c) Schematic view of a single-layer metasurface with controllable multiwavelength functions.¹⁸¹ Reprinted with permission from ref (181). Copyright 2018 American Chemical Society. (d) Comparison of imaging under white light illumination with and without the designed meta-corrector.¹⁸² Reprinted with permission from ref (182). Copyright 2018 American Chemical Society. (e) Schematics of a conventional diffractive spectrometer (left) and a folded metasurface spectrometer (right).⁸⁴ Reprinted with permission from ref (84). Copyright 2018 Springer Nature. (f) Scheme for a folded metasurface hyperspectral imager.¹⁸³ Reprinted with permission from ref (183). Copyright 2019 American Chemical Society. (g) Scheme for the inverse design of a tunable metasurface with the support of PSO and corresponding liquid crystals directors under voltage-on and voltage-off states.¹⁸⁴ Reprinted with permission from ref (184). Copyright 2020 American Chemical Society. (h) Schematic of the electrically tunable multifunctional polarization-dependent metasurface.¹⁸⁵ Reprinted with permission from ref (185). Copyright 2021 American Chemical Society.

The applications of PSO are generally divided into two kinds. One application is to search for the most appropriate nanostructure that possesses a demanding phase or amplitude. As shown in Figure 20b, Zhang et al. used PSO to arrange the micro-coding units of metasurface to achieve desirable phase compensation.¹⁸⁰ Each particle could represent one potential binary coding pattern. Shi et al. took the phase and amplitude difference between the goal solution and the current particle as the error function.¹⁸¹ The evolution direction of PSO is to narrow the error between the target phase-amplitude and the realized phase-amplitude. Metasurfaces with controllable multiwavelength functions such as achromatic meta-lenses and wavelength-controlled beam generators were demonstrated in this way (see Figure 20c). Chen et al. also adopted PSO to design a meta-corrector to correct spherical and chromatic aberrations, as shown in Figure 20d. PSO was used to select the most suitable elements that fit the designed phase profile.¹⁸²

The other application is to improve the efficiency of certain aspects of optical devices. Efficiency is always the key factor for optical design. Recently, folded metasurface optics has attracted attention for its extended optical design space. Folded metasurface optics are metasurfaces developed in the folded optics architecture, as shown in Figure 20e. The light will be greatly reduced after multiple reflections. For devices composed of multiple metasurfaces, efficiency is of crucial importance. Faraji-Dana et al. used PSO to maximize the deflection efficiency of a spectrometer⁸⁴ (see Figure 20e) and a hyperspectral imager¹⁸³ (see Figure 20f) with folded metasurface optics.

An electrically tunable metasurface with liquid crystals has the potential to enhance the functionality of a device. The general electrically tunable metasurface was designed through physical intuition, resulting in a limited switching efficiency (<30%) and a small angle deflection (<25°).¹⁸⁴ Chung et al. achieved an electrically tunable metasurface with a high switching efficiency (80%) and a large angular deflection (144°) via inverse design, as shown in Figure 20g.¹⁸⁴ The key indicator of the design is the switching efficiency, which is the efficiency of the device when switching between different beam modes. In the comparison test, the switching efficiency with PSO is increased by 16%. Hu et al. also designed an electrically tunable metasurface with liquid crystals and demonstrated various polarization-dependent applications (see Figure 20h).¹⁸⁵ In this case, PSO was adopted to find the optimal nanostructure parameters of the meta-unit and give the evolution of polarization conversion efficiencies at different wavelengths.

Ant colony optimization (ACO), which is also a swarm intelligence algorithm, was first proposed by Dorigo in 1992.¹⁹⁶ This meta-heuristic mimics the foraging behavior of ant colonies. Figure 21a demonstrates the general workflow. Ants communicate through pheromones, which help ants identify their nestmates and transmit information. The most common application scenario is foraging. It is often seen that an ant appears near the fruit core, and soon a large number of ants cover the whole fruit core. After the ants find food, they leave pheromones on their way back to the nest. After other foraging ants perceive the pheromones, they will most likely stop searching randomly, find food along with the pheromone, and strengthen the concentration of the pheromone during the return journey. Among the multiple paths leading to food, the pheromones of the longer path evaporate for a longer time, becoming less attractive to the ant colony. Shorter paths are strengthened more frequently, forming a mechanism similar to positive feedback. In the end, the entire ant colony will follow the shortest path to quickly reach the food location in large numbers.

Figure 21.

Principle, performance, and application of MOLACO. (a) The flowchart of MOLACO.¹⁹³ (b) Diversity comparison between conventional ACO and MOLACO.¹⁹³ Reprinted with permission from ref (193). Copyright 2017 IEEE Antennas and Propagation Society. (c) Convergence curve of the adopted MOLACO and a schematic diagram example of a designed free-form meta-atom (inset).¹⁹⁴ Reprinted with permission from ref (194). Copyright 2019 Applied Computational Electromagnetics Society. (d) The path that ants walked was employed as the topology graph of the meta-atom (top). Three-dimensional grid mesh provides a mathematically reasonable, arbitrary, and easy-to-transform design space (bottom).¹⁹⁵ Reprinted with permission from ref (195). Copyright 2019 American Chemical Society.

The evaporation of the pheromones is essential for artificial intelligence systems that simulate ant colonies. If there is no evaporation, the initial random exploration of the first batch of ants is too attractive for subsequent foraging. Later, ants could hardly resist the advantages of the pioneers. The search for the optimal solution will be limited to the initial random search, and there is no guarantee that the optimal global solution will be found.

The nature of ACO is that ants can move arbitrarily in the design space. This feature makes ACO fit for topology design in a meander line pattern. ACO has had demonstrated success in electromagnetic design and optimization, such as meander line RFID antennas¹⁹⁷ and Raman fiber amplifiers.¹⁹⁸ In 2017, Zhu et al. introduced lazy ants into ACO and developed the multi-objective lazy ant colony optimization (MOLACO) algorithm, which is based on the previous analysis of the key role of lazy ants in the ant colony.¹⁹³ In conventional ACO, the optimization will only terminate after the conditions are met. There is one more possibility of termination in MOLACO. Each ant has a different endurancement factor. The endurance factor determines whether the ant is active or lazy. The longer the distance the ant searches, the more exhausted it will be. Lazier ants reach the fatigue limit more easily. The ants will stop moving when they are fatigued. The decision-making process model in MOLACO is closer to a natural ant colony than that in ACO. Compared with conventional ACO, AOLACO increased the diversity of the latent solution, as shown in Figure 21b. Under the same number of optimization iterations, more possible path candidates are generated with MOLACO. In addition to the evaporation of pheromones, this is another safeguard against the pioneer advantage.

In 2019, Zhu applied MOLACO in the inverse design of a metasurface. As shown in Figure 21c, MOLACO was used to generate three-dimensional free-form meta-atoms for a metasurface.¹⁹⁴ The specific design is not limited to the planar, canonical, and easy-to-manufacture structure. The design goal focuses on reaching peak performance. The final result presents complex and tortuous three-dimensional features. Zhu et al.¹⁹⁵ prove the capabilities of MOLACO with a beam-steering metasurface in the midwavelength infrared (MWIR) range. It is well know that metallic structures have more ohmic loss than all-dielectric ones. In some application scenarios, the all-dielectric metasurface also has its weaknesses, such as intercell coupling. To improve the transmission efficiency of metallic meta-atoms, the authors innovatively expanded the topology optimization from two-dimensional to three-dimensional. MOLACO uses the finally generated ant colony path as the topological diagram of the antenna, as shown in Figure 21d. Ants can walk freely in three-dimensional space to find the optimal three-dimensional topological structure. Optimized unintuitive geometries of the meta-atoms present very high transmission efficiencies, which are comparable to the state-of-art performance of their all-dielectric counterparts at that time. In this kind of three-dimensional topology optimization, MOLACO is better than traditional GA. With the same grid mesh design framework, the computation load of GA increases exponentially with the number of grids, while the load of MOLACO increases linearly. GA binarily codes each grid in the optimization model. The relationship between adjacent codes in GA is independent. It is difficult or impossible to fabricate discrete fragments in the final three-dimensional result, such as an abrupt fault or cavity.

3.3. Data Analysis

Besides mapping the relationship between the structure geometry and an optical property, AI also reveals unprecedented capabilities of data analysis in meta-optics. Such applications could be computer vision tasks on images that are captured from meta-lens.¹⁹⁹ More frequently, AI is employed to process unreadable data obtained from metasurfaces, such as image analysis,²⁰⁰ microwave signals,²⁰¹ and infrared spectrum information.²⁰²

Biomolecule detection and monitoring is extremely important for understanding the occurrence of diseases and the course of a drug treatment. High-sensitivity and label-free real-time detection using metasurfaces has attracted widespread attention. The metasurface enhances the infrared absorption of biomolecules so that it can detect the analytes down to trace amounts. However, in the detection of multitarget analytes, it is difficult to distinguish the responses from different analytes, especially when the net refractive index during the biochemical process is basically unchanged. With the help of artificial intelligence, the spectral data can be effectively post-processed to distinguish the analytes.

In 2018, Rodrigo et al. presented an infrared metasurface to dynamically analyze the molecular information in the lipid membrane processes, which is shown in Figure 22. The employed metasurface was Au arrays on an infrared transparent CaF₂ substrate. As the absorption spectrum is different for different analytes, they used linear least-squares regression, which plays an important role in the subfield of artificial intelligence known as machine learning, to obtain the response signals of proteins and lipids. In this way, they achieved label-free dynamic monitoring.

Figure 22.

Infrared metasurface augmented by deep learning for biomolecule sensing. (a) Dynamic analysis of peptide-induced neurotransmitter cargo release in synaptic vesicle mimics.²⁰³ Reprinted with permission from ref (203). Copyright 2018 Springer Nature. (b) Surface-enhanced infrared absorption (SEIRA) for glucose and fructose level sensing.²⁰⁴ Reprinted with permission from ref (204). Copyright 2019 American Chemical Society. (c) Workflow for dynamic monitoring involving lipid vesicles, sucrose, nucleotides, and melittin.²⁰⁵ Reprinted with permission from ref (205). Copyright 2021 Wiley-VCH.

Besides dynamic biochemical reactions, the monitoring of the blood glucose concentration also has important practical significance. The noninvasive sensing of human blood sugar levels can avoid pain and potential infection hazards. With the help of surface-enhanced infrared absorption (SEIRA) spectroscopy, Kühner et al. demonstrated reliable optical detection in the mid-infrared spectrum of pure glucose and fructose solutions and mixtures of the two in aqueous solutions. As shown in the upper left panel of Figure 22b, the authors fabricated arrays of linear gold nanoantennas on the top of the IR transparent CaF₂ wafer. The glucose (red sphere) and fructose (blue sphere) aqueous solution mixture flows into the cell of the sensor through a tube. Glucose and fructose have different characteristic vibrations in the SEIRA spectrum (see the upper right corner of Figure 22a). However, such unstable spectral data with the interference of noise are difficult to recognize using the naked eye or traditional linear analysis methods. The authors used principal component analysis (PCA)²⁰⁶ in machine learning to extract corresponding features and thus realize the noninvasive sensing of glucose and fructose.

In 2021, another metasurface that worked across a more broad mid-infrared spectrum range from below 1000 cm^–1 to above 3000 cm^–1 was developed by John-Herpin et al. for in situ SEIRA spectroscopy, as shown in Figure 22c.²⁰⁵ They fabricated a similar metasurface and integrated the metasurface into a polydimethylsiloxane (PDMS) microfluidic device to construct a multianalyte bioparticle system in water in real time. Besides the broader spectrum, they used a DNN to dynamically monitor more types of biomolecules. This work employed a two-hidden-layer DNN to transform the 1089 wavenumber data to the weights of four kinds of biomolecules. DNN obtains the dynamic changes in the concentrations of biomolecules from the subtle differences in the spectral data in a short response time. Supported by the data analysis of machine learning, SEIRA spectroscopy with a plasmonic metasurface has become a fascinating sensing platform with ultrasensitivity and high fidelity.²⁰⁷

Infrared absorption spectroscopy is also a preferred approach for chemical detection due to its in situ operation, fast response speed, and high fidelity. Recently, Meng et al. presented a microspectrometer for chemical identification, which was composed of metasurface spectral filters and a photodetector array, as shown in Figure 23a. The noisy raw data collected from the detector are processed by a machine learning algorithm, namely a support vector machine (SVM). With machine learning, the authors achieved the chemical identification of six gas-phase chemicals and six liquid-phase chemicals. Figure 23b demonstrates the response spectra for the 12 kinds of chemicals. From the visualization of classification results via PCA, as shown in Figure 23c, five analytes and the no-chemical case were clearly classified.

Figure 23.

Data analyzed by machine learning for a metasurface-based application. (a–c) Data analyzed for chemical analyte classification.²⁰⁸ Reprinted with permission from ref (208). Copyright 2021 American Chemical Society. (a) Schematic of the metasurface chemical classifier. (b) Transmittance spectra for each kind of chemical. (c) Visualization of classification results of the first two principal components (top) and the first three principal components via PCA. (d–g) Data analyzed for acoustic imaging.²⁰⁹ Reprinted with permission from ref (209). Copyright 2020 American Physical Society. (d) Schematic of the experiment configuration. (e) Propagation of waves with high=amplitude wave vector components containing subwavelength characteristic information without (left) and with (right) the meta-lens. (f) Data flow from the radiation source to back-end reconstruction and recognition. (g) Reconstruction and recognition results for far-field information without (top) and with (bottom) the meta-lens.

In addition to the sensing of biochemical molecules, AI can also perform image reconstruction and recognition for data collected from a meta-lens. Due to the diffraction limit, it is challenging for observers in the far-field to observe and identify objects that are much smaller than the illumination wavelength. However, a meta-lens that is placed in the near-field of the object can reradiate the waves to the far-field.²⁰⁹Figure 23d shows the experimental setup for far-field subwavelength imaging. Additionally, the propagation of waves is vividly depicted in Figure 23e. The meta-lens receives the subwavelength image information in the near-field of the source and then converts it into propagation field components that can reach the far-field. In the propagation, absorption loss and noise will increase the difficulty of data analysis. Deep-learning-based methods could easily handle this. Measured data from the receiver are processed by a U-Net-shaped neural network to reconstruct the image and perform digit recognition. The specific data flow is shown in Figure 23f. For the reconstruction and recognition results, as shown in Figure 23g, it is almost impossible to reconstruct information without the meta-lens, not to mention the identification task. This subwavelength imaging task is completed via the cooperation of both the meta-lens and deep learning.

Some metasurface-based applications, such as sensing and imaging, have high data complexity and experience easy interference, which hinder the actual application space. Artificial intelligence can learn potential laws from a large amount of data and provide robust and accurate intelligent feedback. From visible¹⁹⁹ to infrared²¹⁰ and microwave,²⁰¹ AI-empowered metasurfaces demonstrated the vitality of new applications.

3.4. Intelligent Programmable Meta-devices

In recent years, programmable and reconfigurable metasurfaces have been extensively studied for reconfiguration. However, the conventional trial-and-error mode of reconfiguration, which is time-consuming, hinders the real-time application. Without fast reconfiguration, a programmable metasurface is only a nondisposable device. With help from AI, systems based on a programmable metasurface are like a computer with a CPU installed. When a programmable or reconfigurable metasurface cooperates with AI, the data flow between them forms a loop. AI is responsible for obtaining and processing optical data and manipulates the reconfiguration of the programmable metasurface. This allows the metasurface to evolve from an ordinary optical diffraction element to an intelligent element that understands input data and gives real-time responses by itself.

In applications that have large data flows, such as a security check or pipeline monitoring, full-resolution imaging is low in efficiency and high in consumption. However microwave or millimeter-wave radars are useful for such cases, as they can reconstruct the scene without a loss of interest by recording only a small amount of the relevant data. Compared with imaging devices that have fixed with specific imaging scenarios due to their fixed properties after manufacturing, a programmable metasurface is much more flexible. As microwave imaging devices must be able to reconstruct the scene in real time, the radiation pattern of the coding metasurface should change instantly when the scene changes. Real-time in situ inverse scattering problem solution is difficult using conventional numerical methods. Li et al. developed an intelligent imager under the cooperation of a programmable metasurface and machine learning, as shown in Figure 24a.²¹¹ Two machine learning algorithms, random projection²¹⁴ and PCA, were adopted in this work. This intelligent imager can be reprogrammed in situ to generate the radiation patterns in real-time using machine learning. Figure 24b shows the radiation and corresponding coding patterns of the metasurface for 16 leading measurement modes. Therefore, this intelligent imager can handle both image reconstruction and recognition for moving-through-wall body gestures. As shown in Figure 22c, the imaging results become clearer as the number of measurements increases.

Figure 24.

Overview of AI-empowered intelligent programmable meta-devices. (a–c) An intelligent imager.²¹¹ Reprinted with permission from ref (211). Copyright 2019 Springer Nature. (a) Working principle of a real-time intelligent imager. (b) Sixteen radiation patterns and the corresponding patterns generated by machine learning. (c) Machine learning-driven imaging results of two cases at different numbers of measurements (100, 200, 400, and 600). (d) An intelligent imager and recognizer.²¹² Reprinted with permission from ref (212). Copyright 2019 Springer Nature. (e) An intelligent cloak.²¹³ Reprinted with permission from ref (213). Copyright 2019 Springer Nature.

Further unleashing the potential of the intelligent imager to the recognizer, Li et al. displayed fascinating in situ imaging results and recognition performance with the help of neural networks, as shown in Figure 24d.²¹² In this work, two CNNs were adopted sequentially to process the data collected by the programmable metasurface. The first CNN extracts key features from the disordered microwave data collected by the metasurface and converts features to readable optical images. Even if the specimen stands behind a 5 cm thick wooden wall, the imaging results are still accurate. The following CNN employs the well-known classifier, faster R-CNN, to recognize the region of interest (ROI). Given the location of the ROI, the metasurface is reprogrammed to focus radiation beams on the target position. In this way, filtered and enhanced EM data of the ROI are delivered to another CNN to perform recognition.

The cloak of invisibility is always a hot topic in meta-optics. Most metasurface cloaks should cooperate with outside intervention. Qian et al. proposed an intelligent metasurface cloak driven by AI (see Figure 24e) that responded to ambient changes within milliseconds. The neural-network-based AI in this work learned the relationships among the cloaking structure, incoming light, and ambient conditions and conveyed the understanding to the cloak. The working principle of the cloak is that each meta-atom provides a different local reflection spectrum. Therefore, the metasurface can generate a scattered wave similar to blank space. For this, a five-layer DNN was developed. After training, the proposed DNN achieved an accuracy of over 98%. Given the information about the surrounding background, the DNN provides the reflection property solution of the amplitude and the phase for each meta-atom independently. Then, each element is tuned by feeding the corresponding bias voltages. For a fast response, all unimportant intermediate variables should be avoided. The proposed DNN directly uses the bias voltages as the output. The intelligent cloak opens the way to facilitate programmable metasurfaces with AI processing cores as intelligent meta-devices.

Besides the intelligent meta-devices discussed above, an AI-empowered programmable metasurface can also dynamically manipulate EM waves for real-time complex beamforming²¹⁵ and to achieve 3D sensing.²¹⁶ A programmable metasurface can self-adaptively perform intelligent tasks with the help of AI, which is a fascinating direction to develop intelligent devices.

4. Meta-optics for AI

4.1. Optical Realization of Artificial Intelligence

Optical realizations of AI, such as all-optical or hybrid optoelectronic neural networks, are fascinating for their ultrahigh computing speeds and low energy consumption.^217,218 As the medium of information, light propagates in the optical neural network. Even for the computationally expensive tasks, the inference time cost is still tiny because of the speed of the light. In recent years, inspired by the development of neural networks (deep learning), researchers began to develop optical neural networks (ONN) wherein light diffraction mimicked the data transfer of the fully connected layer in a DNN.

The deep diffractive neural network (D²NN) was proposed by Lin et al. in 2018 with a fascinating demonstration of digit recognition and an imaging lens set, as shown in Figure 25a and b, respectively, using diffractive optical elements (DOE).²¹⁹ Since then, there has been an upsurge in the use of D²NN for recognition tasks, such as digit recognition²²⁴ and orbital angular momentum (OAM) detection.²²⁵ Further, Rahman improved the inference performance of D²NN, making accurate classification possible even only with part of the input.²²⁰ Additionally, as shown in Figure 25c, the D²NN system was developed via ensemble learning. In 2021, the D²NN could realize the single-pixel spectral encoding of spatial image information (see Figure 25d), which was fast and safe.²²¹ Besides DOE, an ONN was developed by Mach–Zehnder interferometers, as shown in Figure 25e.²²² Developed in a silicon-integrated circuit, this ONN demonstrated a vowel recognition application.

Figure 25.

Other nonmeta-optical realizations of AI. (a) All-optical diffractive neural network for digit recognition.²¹⁹ (b) All-optical imaging diffractive network.²¹⁹ Reprinted with permission from ref (219). Copyright 2018 American Association for the Advancement of Science. (c) An ensemble deep diffractive neural network system.²²⁰ Reprinted with permission from ref (220). Copyright 2020 Springer Nature. (d) Spectral encoding of spatial information via a diffractive optical neural network.²²¹ Reprinted with permission from ref (221). Copyright 2021 American Association for the Advancement of Science. (e) ONN with the photonic integrated circuit.²²² Reprinted with permission from ref (222). Copyright 2017 Springer Nature. (f) Reconfigurable diffractive optoelectronic processor.²²³ Reprinted with permission from ref (223). Copyright 2021 Springer Nature.

Despite the research enthusiasm in this area and excellent research fruits, there remain many unsolvable problems that hinder the development of all-optical AI. The first difficulty is the photonic nonlinearity. For example, the strength of current neural networks significantly depends on if the function law can be fit nonlinearly. Additionally, this nonlinearity is realized by the heavy implementation of the nonlinear activation function after each linear operation. However, the efficiency of optical nonlinearity is extremely low. The all-optical neural network cannot support much nonlinear operation due to their huge energy loss. Due to the congenital defect of optics, to meet the nonlinearity requirements, researchers searched for help from a hybrid opto-electronic system,^226−229 heat saturation absorption,²³⁰ or a new nanophotonic medium.²³¹ The working unit in Figure 25f leverages the nonlinearity in the conversion from the optical signal to the electrical signal in the imaging sensor. In addition to the network architecture that mimics a DNN, the exploration of the structure of the optical neural network is also a new developmental direction. Similar attempts include optical recurrent neural networks²³² and residual connections.²³³ Different from a digital computer program, optical AI is also faced with this controlling difficulty as an analogue device. Difficulties are also opportunities. These immature technologies have huge developmental potential in future research. The ability of a metasurface to freely manipulate light propagation proves that it has the potential to replace these optical components. The superstructure surface is flat and extremely thin, so it only takes up a small amount of system volume. An optical chip based on the metasurface has a high integration. All these applications are promising directions for meta-optics.

4.2. Functions of Meta-optics for Potential Optical Computation

At the present time, metasurface optical devices are used to demonstrate applications due to their advanced optical properties and novel functions. For AI technology, converting the deep learning neural network based on a programming language into an optical neural network can increase the computing speed from the speed of electrons to the speed of photons. Meta-optics for all-optical information processing have already been noticed.⁷⁵ In this section, we introduce several functional meta-devices that have the potential to be used in the development of optical neural networks or optical computation. The potential meta-devices for optical computation are categorized into wavefront engineering, nonlinear operation, and edge detection. The wavefront engineering metasurfaces are used to manipulate the phase, amplitude, and polarization of light. The nonlinear metasurfaces can convert the frequency of the incident light to achieve spectral control. The edge detection metasurfaces have important roles in image processing and machine vision due to the optical differential operation. These components can be arbitrarily combined, and the neural network can be realized through the optical method. A next-generation optical computing system based on meta-optics can be implemented to serve the Al technology.

4.2.1. Wavefront Engineering

Meta-optics has been demonstrated in various applications of classical optics. The functions of the meta-devices can be roughly classified into four types: phase-, amplitude-, and polarization- related types and mixed types. The purpose of the phase-related application is to realize a full 2π phase modulation to provide full control of the wavefront. The new wavefront can be reconstructed, and the propagating direction can be changed arbitrarily using phase tailoring. The representative applications of meta-optics with phase engineering are beam steering, holographic imaging, and focusing.

Light focusing is a typical function of optics. A metasurface used to focus the light is called a meta-lens. The phase distribution of the focusing can be obtained through the focusing lens equation, which is shown below.

9

where λ is the working wavelength, x and y are defined as the displacements calculated from the center of the lens, and f is the focal length of the lens. For monochrome focusing or imaging, this kind of meta-lens can work with a high efficiency. A TiO₂-based PB-phase meta-lens demonstrated circular polarization focusing with 86%, 73%, and 66% efficiencies at 405, 532, and 660 nm wavelengths, respectively.¹⁹ Once the wavelength and the focal length of the meta-lens are determined, the corresponding phase profile can be calculated. When this meta-lens is operated at other wavelengths, its focal length will change, which is called the chromatic aberration effect. For work in multiple or continuous wavelengths, this is a key issue for some application scenarios. One of the efficient ways to address the chromatic aberration in a continuous bandwidth is to use integrated resonant elements (IRU) to acquire the phase compensation.²³⁴ The above-mentioned focusing lens equation can be described and divided into two parts as follows:

10

11

where the designed working band is from λ_min to λ_max. To eliminate the chromatic aberration and achieve an achromatic meta-lens, φ(R, λ_max) is the basic focusing phase profile that is related to the λ_max, which can be realized by the PB-phase design method. Δφ(R,λ) is the required phase compensation of the whole continuous working band. The IRUs provide different phase compensations for various wavelengths. Using this achromatic meta-lens design method, Tsai et al. first demonstrated a Au-based reflective type metallic achromatic meta-lens that worked in the near-infrared (NIR) region.²⁰Figure 26a shows the Au-based achromatic meta-lens and the IRUs. The IRUs are rotated to fulfill the focusing phase profile, and the various antenna sets provide the phase compensation. The working band of this achromatic meta-lens is from 1200 to 1680 nm. For work at higher frequencies, the applied material has to be carefully considered. Aluminum (Al) has a higher plasma frequency than Au and low absorption in visible light. An Al-based reflective-type metallic achromatic meta-lens was implemented for operation over the whole visible spectrum.²³⁴ For practical usage, the GaN-based transmissive achromatic meta-lens, which works in the visible spectrum (from 400 to 660 nm), was proposed that demonstrated full-color imaging, as shown in Figure 26b.⁷⁰ The GaN-based transmission-type achromatic meta-lens is composed of inverse and solid GaN nanostructures with an 800 nm height and a 50 nm minimum width. The size of the GaN achromatic meta-lens is about 50 μm, which is on the same order as the size of human hair. The operation efficiency is about 40% on average for achromatic focusing. The achromatic meta-lenses mentioned above were designed using the PB phase, so they worked under the circular polarization of incident light. Xiao et al. demonstrated a polarization-insensitive TiO₂-based transmissive achromatic meta-lens with a recorded high operation efficiency, 77.1–88.5%, in the continuous working band (650 to 1000 nm).³³ To eliminate the polarization dependence, the building blocks of this achromatic meta-lens consisted of four types of nanopillars, namely circular, ring, square, and bipolar concentric ring-shaped cross sections (Figure 26c). Using these four as the basic unit structures and adjusting the feature sizes of these structures, the authors achieved 2π phase modulation and the phase compensation of the designed working band. The size of this achromatic meta-lens is 25 μm, and the nanostructures were 1500 nm in height. Such a high-aspect-ratio TiO₂ meta-lens dramatically enhances the group delay range and brings the improvement of the working efficiency.

Figure 26.

Meta-lenses. (a) Au-based reflective-type metallic achromatic meta-lens.²⁰ Reprinted with permission from ref (20). Copyright 2018 Springer Nature. (b) GaN-based transmissive achromatic meta-lens.⁷⁰ Reprinted with permission from ref (70). Copyright 2018 Springer Nature. (c) Polarization-insensitive TiO₂-based transmissive achromatic meta-lens.³³ Reprinted with permission from ref (33). Copyright 2021, Springer Nature. (d) GaN-based pixel-level full-color routing meta-lens.⁷⁷ Reprinted with permission from ref (77). Copyright 2017 American Chemical Society. (e) Synthetic-phase airy beam metasurfaces.²³⁸ Reprinted with permission from ref (238). Copyright 2021 American Chemical Society. (f) Full-Stokes imaging polarimetry by amorphous silicon-based meta-lenses.²³⁹ Reprinted with permission from ref (239). Copyright 2018 American Chemical Society.

In addition to on-axis focusing, the focus position can also be adjusted through a special design to achieve off-axis focusing. Figure 26d shows the GaN-based pixel-level full-color-routing meta-lens. To converge the focus light into an arbitrary position in free space, the focusing lens equation was rewritten as

12

where (r_p,θ_p) is the arbitrary position of the meta-lens surface, (f, δ_f, θ_f) is the 3D position of the focus point in free space, and λ is the designed working wavelength. A multiplex unit cell is composed of 16 GaN nanopillars, which are four nanopillars for 633 nm light (R), four nanopillars for 430 nm light (B), and eight nanopillars for 532 nm light (G1 and G2). The number of green nanopillars is two-times higher than the others because human eyes are more sensitive to green. This dielectric meta-lens can guide individual primary colors into the four quadrants, which matches the Bayer array configuration.⁷⁷ The color router was designed with a size of 50 μm × 50 μm, a polar angle (δ_f) of 8°, and a focal length of 100 μm. The efficiencies of R, G1, B, and G2 are 15.9%, 37.86%, 38.33%, and 27.56%, respectively. This meta-lens-empowered full-color router is promising for light manipulation at a pixel-level scale. The manipulation of the wavefront can be more intuitive through the superposition of phase profiles. The phase profile of the light beam can be directly synthesized with an additional phase to produce a new optical function.^235−237 Wen et al. demonstrated amorphous silicon-based synthetic phase metasurfaces able to generate a focal-plane-adjustable Airy beam by synthesizing the phase of a Fresnel holographic lens. The cubic phase of the Airy beam was also added a Dammann grating phase to generate an array of Airy optical beams.²³⁸Figure 26e shows the synthetic phase profiles for producing a new light wavefront of the Airy beam. In addition to multiple focal points, the generation of multiple polarization states is also very helpful for optical calculations and can provide another degree of freedom for the optical calculation. Figure 26f shows the three integrated amorphous silicon-based meta-lenses focusing on the six spots representing six different polarization states.²³⁹ There are Tthree independent polarization bases (H/V, +45°/ −45°, and RCP/LCP). A superpixel has six spots to focus on six pixels of the image sensor. The intensity of the light received by these six pixels of the image sensor is determined by to the polarization state of the incident light. The NIR full-Stokes polarization camera was implemented to analyze and image the scene with a complicated polarization. These meta-devices with a particular design can manipulate the light’s phase, amplitude, and polarization for use in optical computation.

4.2.2. Nonlinear Metasurface

Nonlinear operation is also the essential operation of the deep learning neural network. To realize the optical neural network, the nonlinear optical effect can play a role in the operation. The nonlinear optical effect generates a new frequency of the electromagnetic wave applied in laser, display, spectroscopy, data storage, signal processing, and quantum technology. Figure 27 shows the common nonlinear effects used in optical applications, such as second-harmonic generation (SHG), third-harmonic generation (THG), four-wave mixing (FWM), and spontaneous parametric down-conversion (SPDC). The energy diagram illuminates the frequency conversion between the fundamental light and the nonlinear light under the conservation of energy. SHG and THG generate nonlinear frequencies two or three times, respectively, that of the fundamental light frequency. FWM is the interaction between two (or three) frequencies to produce two (or one) new frequencies. In quantum optics, SPDC is an essential technology for preparing entangled photon pairs. The comprehensive lectures of nonlinear meta-optics report on fundamental concepts, design, and application.^240,241 In general, the nonlinear effect of meta-optics has occurred via the nonlinear susceptibilities of the selected material, resonant excitations, and the symmetry breaking of the meta-molecule. The material for nonlinear metasurfaces can be chosen with metal or dielectric materials. A metallic nonlinear metasurface can be realized by both gold planar and vertical split-ring resonators (PSRR and VSRR, respectively) for secon- harmonic generation.⁵⁴ The SHG signals of the VSRRs are shown in Figure 28a. Under the same feature size condition, the pumping wavelengths are 1035 and 858 nm for PSRR and VSRR, respectively, in accordance with the realistic resonant position. The nonlinear optical signal measured from the VSRR metasurface indicates an improvement of 2.6× in SHG compared to that of the PSRR metasurface. The VSRRs can couple well with the electric and magnetic fields of an incident wave. The localized field is confined between two vertical prongs of a VSRR, which are lifted away from the glass substrate. This well-designed feature provides an enhancement of the nonlinear process. The nonlinear metasurface can be designed with the focusing phase to realize nonlinear imaging. The gold C3 nanoantennas are rotated according to the phase requirement of focusing on the formation of a nonlinear meta-lens, as shown in Figure 28b.²⁴² The phase modulation has followed the concept of the nonlinear PB phase. When the right circularly polarized fundamental light with a 1085 nm wavelength is incident on the nonlinear meta-lens, the left circularly polarized SHG signal is generated and focused at the designed focal plane. The “L”-shaped aperture is used as the imaging target by a nonlinear meta-lens with the SHG signal. The nonlinear metasurface with the phase design can be implemented with nonlinear beam-shaping²⁴³ and holographic^49,244 and optical encryption.²⁴⁵

Figure 27.

Schematic diagram of the nonlinear optical effect and the frequency conversion. (a) Second-harmonic generation. (b) Third-harmonic generation. (c) Four-wave mixing. (d) Spontaneous parametric down-conversion.

Figure 28.

Nonlinear metasurfaces. (a) Second-order nonlinear generation by gold vertical split-ring resonators.⁵⁴ Reprinted with permission from ref (54). Copyright 2019 Wiley-VCH. (b) Second-order nonlinear generation and focusing by C3 gold meta-atoms.²⁴² Reprinted with permission from ref (242). Copyright 2018 Wiley-VCH (c) Vacuum ultraviolet light second-order nonlinear generation by ZnO.⁵⁷ Reprinted with permission from ref (57). Copyright 2018 American Chemical Society. (d) Vacuum ultraviolet light third-order nonlinear generation by TiO₂.⁵⁶ Reprinted with permission from ref (56). Copyright 2019 American Chemical Society. (e) Spontaneous parametric down-conversion by the lithium niobite metasurface.²⁴⁸ Reprinted with permission from ref (248). Copyright 2021 American Chemical Society. (f) High-dimensional entangled photon pairs generated by the GaN meta-lens array.⁹¹ Reprinted with permission from ref (91). Copyright 2020 American Association for the Advancement of Science.

The excitation of nonlinear effects usually requires high-powered pulsed lasers to improve the efficiency of occurrence. Under high-powered laser irradiation, the metallic nonlinear metasurface can be easily damaged due to the low damage threshold. This dramatically limits the further improvement of the nonlinear optical conversion efficiency. The dielectric material-based nonlinear metasurface was proposed to increase the nonlinear conversion efficiency. Semmlinger et al. demonstrated the generator of vacuum ultraviolet (VUV) light by a ZnO⁵⁷ and TiO₂⁵⁶ nonlinear metasurface. ZnO was selected for VUV light SHG because of its large second-order susceptibility for a significant second-order nonlinear optical response, as shown in Figure 28c.²⁴⁶ The ZnO nanodisk was designed to provide a magnetic dipole resonance mode at a fundamental wavelength of 394 nm for VUV light generation. At present, a prism-coupled potassium fluoroboratoberyllate (KBBF) crystal is the most commonly used crystal for coherent VUV generation. Compared with that of the prism-coupled potassium KBBF crystal, the effective nonlinear coefficient of the ZnO nonlinear metasurface is three times. ZnO nonlinear metasurface is compact and can be readily integrated into VUV light source ultrafast laser systems without the requirement of complex experimental configurations or phase matching. The phase-matching condition can be greatly relaxed on the metasurface because the effective nonlinear optical process only occurs in the material layer with a subwavelength thickness. In this case, the phase-matching condition is no longer as necessary as that in conventional nonlinear optical crystals. Figure 28d shows that TiO₂ was employed for VUV THG due to its large third-order susceptibility. The nonlinear signal with a 185 nm wavelength is generated by the strong light–matter interaction between the fundamental light and the TiO₂ nanodisk. The TiO₂ nanodisk has the dark anapole mode, which contributes to the large local field enhancement at the fundamental wavelength. The anapole mode is increased by the destructive interference of toroidal and electric dipole moments. These two kinds of dipoles have a similar strengths but a −π-phase difference. In addition, the TiO₂ nanodisk has a strong magnetic dipole that overlaps with the anapole mode to simultaneously contribute to the electromagnetic field confinement inside the disks. The measured nonlinear signal of the TiO₂ nonlinear metasurface can be enhanced by a factor of 180 compared to that of an unpatterned TiO₂ thin film of the same thickness. The nonlinear conversion efficiency of the metasurface is an important issue for practical use. To obtain nonlinear light with a high generation efficiency, we can focus on several directions, for example, the exploration of nonlinear materials with high nonlinear susceptibilities or the design of meta-atoms to support extremely high Q resonances at both the pump and nonlinear signal frequencies.²⁴⁷

SPDC is a nonlinear optical process for quantum-entangled photon pair generation. A pump photon of a frequency ω_p excites the nonlinear crystal and splits into two daughter photons, called signal and idler photons, with lower frequencies ω_s and ω_i, respectively. The SPDC generation efficiencies of common nonlinear crystals, such as lithium niobate (LN) and β-barium borate (BaB₂O₄, BBO), are extremely weak because SPDC is based on the parametric amplification of the vacuum field. Unlike classical nonlinear processes such as SHG, the generation rate of SPDC is linearly proportional to the pump power, so it is impossible to improve its efficiency through the use of pulsed or focused pumps. To improve the generation efficiency of the SPDC, the meta-atoms were designed with resonances at the signal and idler photon frequencies. A reflective type of LN-based quantum optical metasurface (QOM) was demonstrated to generate the SPDC photon pairs, as shown in Figure 28e.²⁴⁸ The QOM was excited by a laser with a wavelength of 788 nm , and the wavelength of the generated SPDC photon pairs was 1576 nm. The photon pairs emitted within the narrow wavelength range of the designed resonances. The generation rate enhancement of the QOM showed an enhancement of two orders of magnitude compared to that of an unpatterned LN film of the same thickness. In this era of big data, the amount of information that needs to be processed has increased significantly, and an artificial intelligence path was developed to meet this demand. For optical artificial intelligence systems, quantum-entangled photons are powerful carriers. In quantum optics, a pair of entangled photons can contain multiple quantum states. When the quantum states of multiple pairs of entangled photon pairs are coupled and superimposed on each other, new high-dimensional quantum superposition states can be generated. To meet the demand for high-dimensional optical quantum information technologies, a high-dimensional quantum entanglement optical chip with 100 entangled photon pairs was demonstrated by a 10 × 10 GaN meta-lens array, as shown in Figure 28f.⁹¹ This meta-lens array was integrated with a thin BaB₂O₄ crystal. The meta-lens array produced multifocused spots of pumping light inside the nonlinear crystal. This optical quantum chip could generate 100-path entangled photon pairs, and the multidimensional tomography with high fidelities was measured. This new high-dimensional quantum light source is compact, controllable, and has no need for a low-temperature environment.

4.2.3. Edge Detection

Edge detection is the primary method of image processing and machine vision. Edge detection technology is based on the optical differential operation. The traditional optical differentiator is composed of a lens and a spatial filter, which is bulky, complicated, and inconvenient for system integration. The metasurface has been developed as the optical differential operation device for the edge detection application. Figure 29a shows a Pancharatnam–Berry-phase metasurface, which was designed for and performed edge detection via analog spatial differentiation.²⁴⁹ The sample is made of form-birefringent nanostructured glass slabs with the PB-phase design. The PB-phase gradient metasurface was placed at the Fourier plane, and the output electrical field was placed at the image plane. Under the linear polarization of incident light, the left-handed and right-handed circularly polarized (LCP and RCP, respectively) components face opposite phase gradients. This leads output images with LCP and RCP that translate a distance with opposite directions. The overlap of these two output images presents the same linear polarization of the incident light. The edge image can be switched on and off by a linear polarization analyzer. The one-dimensional edge information is along the polarization direction of the incident light. Orientation-dependent edge detection is achieved by varying the linear polarization state of the analyzer. Using a similar concept, a PB-phase metasurface was also combined with the quantum-entangled photon pair to enable quantum edge detection.⁹²

Figure 29.

Edge detection using meta-optics. (a) PB-phase metasurface for orientation-dependent 1D edge detection.²⁴⁹ Reprinted with permission from ref (249). Copyright 2019 United States National Academy of Sciences. (b) The metasurface spatial filter for 2D edge detection.²⁵⁰ Reprinted with permission from ref (250). Copyright 2020 American Chemical Society. (c) The achromatic meta-lens array for 1D to 3D edge detection.⁷⁸ Reprinted with permission from ref (78). Copyright 2021 Walter de Gruyter.

The metasurface-driven optical spatial filter is formed by subwavelength TiO₂ nanopillars. The phase profile of the metasurface is designed to behave as a constant-phase profile for LCP incident light and a spiral-phase profile that changes from 0 to 2π for one turn for RCP incident light. The metasurface spatial filter was set in a Fourier transform set-up to provide the complex masking function. When the masking function includes a constant phase and amplitude (LCP light incident), a bright-field image at the image plane is generated. In this situation, this Fourier transform setup is like a conventional 4f imaging system. When the masking function is a spiral-phase plate (RCP light incident), the corresponding point spread function can be seen as a vortex with a doughnut-like intensity distribution. The vortex element has a π-phase difference along the opposite azimuth, resulting in destructive interference and a dark background for the uniform area of the input electrical field. The arbitrarily uneven amplitude and phase gradient areas in the region of integration remove the destructive interference and present bright areas that represent the edge information, as shown in Figure 29b. Ordinary diffraction imaging and isotropic edge-enhanced imaging can be dynamically switched by varying the circular polarization type of the incident light.

The achromatic meta-lens array enables 1D to 3D edge detection through its ability to acquire all light information.⁷⁸ The meta-lens array captures all the light field information. The light field raw image is processed by the first-order derivation and rendered as an algorithm of light field imaging to extract the edge information from 1D to 3D. The image data are discrete data, so their derivative can be obtained by the method of difference. For 1D edge detection, the derivative is to subtract the current pixel value from the next pixel value. The 2D edge is the sum of the differences in the two directions, either normal x–y directions or cross directions. When the depth information on the object is also obtained, the edge image containing the depth information can be realized as the detected 3D edge. In Figure 29c, the three objects are located at different distances, corresponding to the meta-lens array. After the image is computed, each letter object can have the sharpest edge image at its respective depth imaging position. Full-color edge imaging is also performed by the achromatic meta-lens array.

4.2.4. Optical Frontend for a Neural Network

Recently, it was reported that a metasurface can behave as the optical frontend of the deep learning-based image processing pipeline. The total pipeline of the optoelectronic neural network is divided into the physical layer based on the metasurface and digital layers developed on computers. As shown in Figure 30a, a reconfigurable metasurface emits microwave patterns to encode scene information for subsequent deep learning-based classification tasks.²⁵¹ Additionally, because of the reconfigurability of the adopted metasurface, it can participate in the in situ training of the photoelectric neural network as a trainable physical layer. In the actual training, the physical layer of the metasurface is described as an analytical model and cascaded with a simple DNN structure. The specific configuration of the metasurface is determined after training. With the front end of the metasurface, 10–15% classification accuracy gains were observed. Due to the lack of optical nonlinearity, the hybrid photonic–electronic system could be an alternative choice for cognitive optical computation tasks. Colburn et al. proposed an architecture that adopted the metasurface as the frontend of a CNN, as shown in Figure 30b.²⁵² CNN has advantages in image-based tasks because its operation can consider the spatial information on the image. The digital implementation of convolution spends many computational sources and leads to a high latency. The metasurface is the better choice for the linear processor, as it keeps the spatial information. The authors used the 4f system with Fourier transform to perform the convolution operation. Additionally, for parallel computing, the two lenses in the 4f system are replaced by two lens arrays. The metasurface is located on the Fourier plane and acts as the filter (kernel). The parts that take up the most computing resources are usually in the initial layers of the CNN. Especially for the high-resolution image, the optical convolution operation is ultrafast and energy-saving.

Figure 30.

Optical frontend of the neural network. (a) A configurable metasurface works as a physical trainable layer for a DNN.²⁵¹ Reprinted with permission from ref (251). Copyright 2020 Wiley-VCH. (b) A metasurface works as the optical frontend of a CNN.²⁵² Reprinted with permission from ref (252). Copyright 2019 Optical Society of America.

4.3. Meta-Based Optical Neural Network

A meta-based optical neural network (ONN) is used to realize an artificial neural network (ANN) in a physical way with a metasurface. In other words, the meta-based ONN adopts an ANN to model the light propagation in a multilayer meta-system. Additionally, the training strategy in an ANN is used to optimize the configuration of each meta-atom in the meta-based ONN. For brevity, we refer to the meta-based optical neural network as a meta-neural network. In addition to its light-speed processing and low energy consumption, the meta-neural network is compact and easy to visualize. The metasurface is composed of thin, flat, and chip-level optics that can freely control the light. A meta-neural network is a meta-system composed of cascaded multilayer metasurfaces. In terms of architecture, both meta-neural networks and an ANN are layered structures. The output of each metasurface layer can be visualized by adjusting the position of the sensor. In theory, a metasurface can mimic the functionality of a hidden layer in an ANN, as shown in Figure 31a and b, where the data flow is in the form of light. The metasurface consists of a dense array of subwavelength meta-atoms. According to Huygens’ Principle, each meta-atom at the metasurface can be regarded as a secondary source of the wave.²¹⁹ Each subwavelength meta-atom (see Figure 31c) behaves like an independent neuron (see Figure 31d) in the artificial neural network. In a multiple-layer metasurface system, every meta-atom in the l-layer metasurface connects to all meta-atoms in the l + 1 layer metasurface through the diffraction of light. Specifically, a meta-atom on the l-layer metasurface is a secondary source at the location r⃗_i^l = (x_i, y_i^l, z_i). According to Rayleigh–Sommerfeld diffraction,²⁵³ the Huygens wavelet of this source along the direction of propagation (z-axis) can be described as the z-derivative of the spherical wave, which is²⁵⁴

13

where u(r⃗_i^l) is the input wave and t(r⃗_i) is the transmission coefficient of the meta-atom.

14

15

where k is the wavevector of light. Additionally, the electric field u(r⃗_j^l+1) that reaches the meta-atom at the l + 1 layer metasurface at the location r⃗_j = (x_j^l+1, y_j, z_j^l+1) is the summation of the field excited by all meta-atoms at the l-layer metasurface, which is given by²⁵⁴

16

Figure 31.

Comparison between a metasurface-based optical neural network and an artificial neural network. (a) Schematic diagram of an optical neural network composed of metasurface layers. (b) Schematic diagram of the corresponding artificial neural network. (c) Light diffraction among meta-atoms of different layers. (d) Data flow among neurons of different layers.

The process of optical diffraction can be compared to the matrix operations in an ANN during the transmission or reflection coefficient t(r⃗_i^l) = A_ie^jϕ_il of each meta-atom, which corresponds to the neuron that carries the trainable weights. The loss function was constructed using the difference in the light intensity distribution between the output and the ground truth. In addition, the transmission coefficient or the reflection coefficient t(r⃗_i^l) of each meta-atom was optimized iteratively through the back-propagation algorithm. However, the diffraction and propagation of light are actually linear matrix multiplications, making it impossible to fit complex nonlinear functions. ANNs are used to model complex functions for complicated tasks, and the use of nonlinear activation functions enables them to approximate arbitrarily complex functions. In the current research, the nonlinearity in the ONN is still challenging to solve. Optical nonlinearity has always been a problem of low efficiency, which limits the application of the current ONN. It has been reported the use of the photoelectric combination can be an excellent way to provide nonlinearity. Normally, the output layer in the ONN is an image sensor and the capture of light intensity I = |u(r⃗)|² is a natural nonlinear operation in the photoelectric conversion process.²²³ However, this is far from the activation function in an ANN. Even the photoelectric implementation of the nonlinear activation function is still immature. All-optical nonlinearity is the key to realizing a truly deep ONN.

Qian et al. proposed a diffractive neural network based on the metasurface that was capable of all seven basic logic operations. The authors experimentally demonstrated the AND, OR, and NOT operations with a two-layer dielectric metasurface at microwave frequency (17 GHz). This optical neural network does not have a high requirement for the input light. Without an additional bulky apparatus for input light control, the whole photonic system is relatively compact. As shown by Figure 32a, the incident plane wave is spatially encoded at the input layer, which is an optical mask. The regions corresponding to the computing compositions have high light transmittance. The authors used a metasurface to mimic the functionality of hidden layers in the artificial neural network. As shown in the right bottom panel of Figure 32a, the magnitude changes slightly with the height of the meta-atom, while the phase changes smoothly. The proposed ONN uses the phase as the training variable. Each metasurface manipulates the phase compensation to control the propagation of light. The compound metasurfaces direct and focus the light into one of two specific areas at the output layer, representing the logic states 1 and 0, respectively. This work successfully demonstrated the potential of metasurface-based ONN for mathematical operation applications.

Figure 32.

Meta-neural network for logic operation.²⁵⁴ (a) Design scheme and experimental setup. (b) Schematic of the corresponding digital neural network. (c) Experimental setup with the fabricated meta-neural network. (d) Design space of the meta-atom. Reprinted with permission from ref (254). Copyright 2020 Springer Nature.

Besides basic mathematical operations, there are meta-neural networks for elementary cognitive tasks, such as the recognition of handwritten digits. With accessible images from the Modified National Institute of Standards and Technology (MNIST) data set,²⁵⁶ the recognition of handwritten digits is a universal pattern recognition test model. The meta-neural network acts as a classifier by directing the light from the object to the corresponding detector (0–9). Under the diffraction simulation model, each layer of the neuromorphic metasurface in the meta-neural network is designed through the training strategy of machine learning.

Wu et al. proposed a meta-neural network composed of cascaded neuromorphic metasurfaces for the recognition of handwritten digits.²⁵⁵Figure 33a demonstrates the design principle. Given the plane wave carrying the pattern information as the input, the multilayer metasurfaces controls the local amplitude and phase of the light. Each meta-atom acts as a neuron, and light interferes strongly with each neuron. After being processed by the three-layer meta-neural network, the light is focused and directed to the spatial location of the corresponding detector. As shown in Figure 33b, the authors used a near-to-far-field transformation to model the forward light propagation. Figure 33c provides the intensity distribution among the free space in the meta-neural network. The intensity of the focused beam directed toward the right detector is stronger than others. In this proposed meta-neural network, not only the phase but also the amplitude are considered. As shown in Figure 33d, the authors constructed the functions of amplitude-width and phase-width. Additionally, they directly optimized the specific design parameters and widths of the nanopillars in the iterative training process. The proposed meta-neural network works at the visible wavelength (700 nm). The accuracy tested on the MNIST data set reached 85%. However, this work vectorized the image as the input, which did not fully utilize the properties of the metasurface as a 2D material.

Figure 33.

Meta-neural network of cascaded metasurfaces for the recognition of handwritten digits. (a–d) Neuromorphic metasurface.²⁵⁵ Reprinted with permission from ref (255). Copyright 2020 Optical Society of America. (a) Schematic of the classifier. (b) Model of forwarding propagation in the meta-neural network. (c) Intensity distribution of light propogation in the neural network. (d) Amplitude and phase distribution over different meta-atom parameters (width).

Extending the vector input (1D) to the matrix input (2D), Weng et al. proposed a two-layer meta-neural network, as shown in Figure 34a. Without vectorization, the spatial information in the input is maintained. Additionally, the lack of preprocessing means that the whole system is purely passive and has a low latency. The monochromatic plane wave illuminates the input pattern and reaches the metasurface. The metasurface controls the local phase of the light. In the iterative training, the phase profile of each metasurface is optimized. Under certain phase compensations, the optical computation of the corresponding matrix operation for recognition is realized by diffraction in the 3D space. Detecting regions of ten digits are uniformly distributed on the output plane. The authors performed an experimental verification based on 3D printing fabrication, as shown in Figure 34b. They fabricated 20 steel plates of the digits shape. Panels c and d in Figure 34 demonstrate the energy distribution matrices of the simulation and the experiment, respectively. The experimental results show good consistency with the simulation. Further, the authors used a four-layer meta-neural network to distinguish the orbital angular momentum (OAM), which has a more complicated spatial pattern (see Figure 34e). The key advantage of the proposed meta-neural network is that there is no need for a strict alignment between the beam and the recognition device. For 255 combinations of 8 OAM orders, the proposed neural network could achieve a recognition accuracy of more than 90% even when the beam center was misaligned by six wavelengths (6λ). On the output plane, there are eight focused beams that denote the existence of different OAM states. Compared with the single focused beam in the primary classification task (e.g., the recognition of handwritten digits), this work demonstrates more computational power.

Figure 34.

Purely passive meta-neural network for object recognition.²⁵⁷ (a) Meta-neural network for the recognition of handwritten digits. (b) Picture of the fabricated meta-neural network. (c) Energy distribution matrix of the simulation for 20 selected samples. (d) Energy distribution matrix of the experiment for 20 selected samples. (e) Meta-neural network for OAM state recognition. “Y” and “N” denote the presence and absence of the corresponding OAM state, respectively. Reprinted with permission from ref (257). Copyright 2020 Springer Nature.

To reduce misalignment between different layers of the metasurface, a kind of 1D metasurface is proposed to achieve full integrity, which is called a metaline (see Figure 35a).²⁵⁸ In the proposed meta-neural network, each hidden layer in Figure 35b corresponds to a metaline in Figure 35a, and each metaline is a line of meta-atoms. In Figure 35a, each detector represents a digit. Additionally, the five-layer meta-neural network directs the propagation of the input signal to the corresponding detectors. Figure 35c demonstrates examples of digit recognition, where the light intensity of the corresponding position is higher than others. The on-chip meta-neural network was developed on the silicon-on-insulator (SOI) platform. As shown in Figure 35a, the meta-atom is an etched rectangle slot. With negligible transmission loss (see Figure 35d), the phase compensation of each meta-atom is selected as the trainable variable. The input image is converted to a vector signal. Tested on the MNIST data set, this meta-neural network reached 88.8% accuracy.

Figure 35.

Meta-neural network of 1D metalines for the recognition of handwritten digits.²⁵⁸ (a) Schematic of meta-lines. (b) Corresponding conventional neural network whose hidden layers are realized as meta-lines. (c) Verification results. (d) Design space of the meta-atom. Reprinted with permission from ref (258). Copyright 2020 Optical Society of America.

As shown in Table 3, the meta-neural networks can be composed of metasurfaces or metalines. The metasurface is more suitable for an image-based application, which can take spatial information into consideration. It is easy to align different layers in the metaline-based meta-neural network, which has more advantages in actual experiment implementation. Mostly, the phase changes linearly with the structure parameter of the meta-atom while the amplitude almost remains invariant. Therefore, the meta-neural network usually adopts the phase as the training variable. Current research of meta-neural networks is still immature. Additionally, most of the related literature remains in the simulation stage. Without actual manufacturing and experimental verification, the efficacy of a meta-neural network is difficult to prove. The high accuracy results (>85%) are verified by the simulation, but there must be a gap between reality and theory. Importantly, significant of light propagation loss is ignored in the modeling of the cascade system. Manufacturing defects will bring undesired absorption, which leads to the transmittance of the fabricated metasurface being lower than the theoretical value. The deviation caused by the quantification will also cause the actual object to deviate from the ideal model. For example, the trained variable, such as the phase, is not continuously realized during meta-atom selection and fabrication. Additionally, in practice, the diffraction loss of light in free space is very large. The amount of light that reaches the metasurface of the next layer is much less than the amount of light that reaches the previous layer. Due to the efficiency limit, all the literature listed in Table 3 used only a small number of layers. Therefore, efficiency is one of the keys to the development of a multilayer cascaded meta-neural network. In the application column, the number in brackets represents the number of cognitive categories. For example, (10) means there are 10 classes of the output. The current application is still limited to handwritten digit recognition, whose cognitive category is small. More complicated tasks, e.g., face recognition and object detection, would be more challenging and fascinating in future research.

Table 3. Summary of the Meta-neural Network.

type	training variables	layers	size	working bands	lattice period	input	training set	test set	application	accuracy (%)	ref
metasurface	phase	2	cascaded layers, 300 × 420 mm2	17.6 mm	10 mm	optical mask			logic operation (7)		(254)
metasurface	width of the meta-atom	3	cascaded layers, length 94 μm, distance 188 μm	700 nm	235 nm	vector 400 × 1	50 000	10 000	recognition of handwritten digits (10)	85	(255)
metasurface	phase	2	cascaded layers, 56 × 56 cm2, distance 17.5 cm	3 kHz	2 cm	matrix 28 × 28	55 000	10 000	recognition of handwritten digits (10)	93	(257)
metasurface	phase	4	cascaded layers			matrix 101 × 101			discerning misaligned OAM vortices (255)	90.20	(257)
meta-line	phase	5	on-chip, 400 × 800 μm2	1.55 μm	500 nm	vector 784 × 1	4000	10 000	recognition of handwritten digits (10)	88.80	(258)

The major deficiency of the meta-neural network is its nontunability. Due to the deficiency in the development of tunable metasurface, the all-optical and in situ training cannot currently be realized. Additionally, after fabrication, the metasurface is fixed and nonadjustable. Therefore, each meta-neural network is tailored for a specific task. Theoretically, the layout adopted in metasurface manufacturing is the optimal solution after training.

5. Summary and Prospects

In the forward problem, surrogate modeling is prominent for its ultrafast computation speed. The characteristics of real-time response and the neural network structure make it easy to incorporate into inverse design schemes. However, the performance of the surrogate model is limited to specific tasks, which are not flexible. Inaccurate prediction at resonances and the time-consuming preparation of training data also make it less practical. In the future, improvements to the deep learning-based algorithms and the development of composite multifunctional models will be promising directions.

For the inverse problem, automatic design is a fascinating topic. NN-based methods and evolutional computation-based algorithms are two representatives. NN-based methods are faster, while evolution computation has no restriction on the differentiability of the optimized function. Most current NN-based works focus on the design of meta-atoms. Given demanded optical responses, the inverse design scheme outputs the geometry of the candidates. Such on-demand design normally shows a high compliance with targets. Inverse design provides a reliable solution to the clueless problem. However, there is a lack of NN-based literature on supercell and metasurface scales. The overall system design method will be more intelligent and convenient. Gradient-free evolutionary computation is widely applied in optimization, such as maximizing the efficiency. Despite consuming relatively more time, evolutionary computation-based methods show capability in global optimization and various design tasks. Additionally, evolution computation does not require training data. However, the nature of iterative optimization determines that it is destined not to be applied directly in large-scale system design. Overall, AI lowers the threshold of meta-optics design. AI-based design requires less experience and optical background knowledge. Recently, AI has also shown powerful capabilities in the analysis of data from a metasurface, which is of great importance in extreme measurement and dynamic analysis. Furthermore, AI can control the readjustment of the metasurface using data previously acquired and processed. Such a loop of the data stream achieves system automation. AI-empowered meta-optics exhibits the capabilities of self-control. These smart and intelligent meta-devices based on AI could be a new hot topic in the cross-field between AI and meta-optics.

In the current development of electronic chips, Moore’s law is about to meet the physical limit as advanced semiconductor processes continue to approach the 3 nm process. It is becoming more and more difficult to shrink the line width of semiconductors, and the cost is getting higher and higher. The increase in the computing speed of electronic chips has gradually slowed down. Compared with computing with electrons as a medium, photonic computing naturally has the advantages of the high dimension of information transmission and the strong parallel transmission capability. In traditional optics, optical computations are limited by the unity of material properties, and traditional optical components are generally bulky and difficult to integrate. The emergence of meta-optics provides a new direction for optical computing. Through the specially designed artificial nanoantenna array, the characteristics of light can be arbitrarily adjusted to produce novel functions. Meta-devices have the advantages of being ultrathin, lightweight, compact, and easy to integrate, perfectly solving the shortcomings of traditional optics. At present, meta-optics has realized a variety of optical applications, which can control the phase, frequency, polarization, amplitude, and propagation direction of light. Through rigorous design, these optical properties can be adjusted not only individually but also multidimensionally. AI neural network computing realized by optics is a big trend in the future. At present, optical neural networks are generally implemented by optical fibers or silicon-based waveguides. In comparison, the number of research results on optical neural networks based on meta-optics is relatively small. The challenge that needs to be solved urgently is the work efficiency and tunability of meta-devices. Take the optical neural network as an example. As the number of computing layers increases, each layer of the metasurface provides computing information and loses a certain amount of signal strength. Once the computing architecture becomes more complex and the number of computing layers increases, the number of photons obtained at the end becomes very limited. Another challenge is the tunability of functions. The wavefront manipulation of light is the phase arrangement through the metasurface, and each phase pixel of the phase profile depends on the design of each meta-atom, such as shape, feature size, and effective refractive index. Meta-atoms are subwavelength structures. At present, the individual electric adjustment of the meta-atom to achieve real-time adjustable meta-devices has only been realized in the terahertz region. This is a large challenge for visible or near-infrared meta-devices because the feature size of the meta-atom size is very small (around a few tens of nanometers). It can only realize the control of the entire meta-devices, resulting in a monotonous function that is often just a switch function. With the continuous innovation of metasurface optics research and related fabrication technology improvements, these challenges will hopefully be overcome.

Artificial intelligence technology applied to meta-optics can solve complex optical designs and quickly obtain the best solution of the problem to meet the demands of novel functions. Currently, the amount of data processing is dramatically increasing, and photons that carry high-dimensional information can effectively expand the bandwidth of information operations. Optical neural network computing based on meta-optics can break through the bottleneck of electronic computing power and computing speed. Artificial intelligence and metasurface optics in synergy will help the research and development of advanced optical chips, which is conducive to the implementation of the next generation of optical devices and systems, and will bring human civilization to a higher level in the future.

Glossary

Abbreviations

1D

one-dimensional

2D

two-dimensional

3D

three-dimensional

ACO

ant colony optimization

AE

autoencoder

AAE

adversarial autoencoders

AI

artificial intelligence

ANN

artificial neural network

A-SBFA

advanced short backfire antenna

BBO

β-barium borate

BIC

bound state in the continuum

CD

circular dichroism

cGAN

conditional generative adversarial network

CMA-ES

covariance matrix adaptation evolution strategy

CMOS

complementary metal-oxide semiconductor

CNN

convolutional neural network

ConvNet

convolutional neural network

CPU

central processing unit

CV

computer vision

DNN

deep neural network

DOE

diffractive optical elements

DOF

degree of freedom

D²NN

deep diffractive neural network

EBL

electron beam lithography

EM

electromagnetic

ES

evolution strategy

FCN

fully convolutional network

FDTD

finite difference time domain

FEM

finite element method

FIB

focused ion beam

FPGA

field-programmable gate array

FWM

four-wave mixing

GA

genetic algorithm

GaN

gallium nitride

GAN

generative adversarial network

GPU

graphics processing unit

GPN

geometry-predicting-network

GRIN

gradient index

GS

Gerchberg–Saxton

GTTS

genetic-type tree search

IRU

integrated resonant elements

ITO

indium tin oxide

KBBF

fluoroboratoberyllate

KNN

k-nearest neighbor

LCP

left circular polarization

LN

lithium niobate

MCTS

Monte Carlo tree search

MIM

metal–insulator–metal

MLP

multilayer perceptron

MNIST

modified national institute of standards and technology

MOLACO

multiobjective lazy ant colony optimization

MSE

mean-squared error

MWIR

midwavelength infrared

NIR

near-infrared

NN

neural network

OAM

orbital angular momentum

ONN

optical neural network

PB

Pancharatnam–Berry

PCA

principal component analysis

PDMS

polydimethylsiloxane

PReLU

parametric rectified linear unit

PSNR

peak signal-to-noise ratio

PSO

particle swarm optimization

PSRR

planar split-ring resonators

QOM

quantum optical metasurface

RCWA

rigorous coupled wave analysis

RCP

right circular polarization

ReLU

rectified linear unit

ResNet

residual neural network

RCS

radar cross section

RNN

recurrent neural network

ROI

region of interest

SEIRA

surface-enhanced infrared absorption

SHG

second-harmonic generation

SOI

silicon-on-insulator

SPDC

spontaneous parametric down-conversion

SPN

spectrum-predicting-network

SRR

split ring resonator

SVD

singular value decomposition

SVM

support vector machine

Tanh

hyperbolic tangent

THG

third-harmonic generation

TiO₂

titanium dioxide

TNN

tandem neural network

UV

ultraviolet

VAE

variational autoencoder

VSRR

vertical split-ring resonators

VUV

vacuum ultraviolet

Biographies

Mu Ku Chen received his Ph.D. from the Department of Physics at National Taiwan University in 2019. He was a Research Assistant Professor in the Department of Electronic and Information Engineering at The Hong Kong Polytechnic University. At present, he is a Research Assistant Professor in the Department of Electrical Engineering at City University of Hong Kong. His research interests include photonic information, nanophotonics, the fabrication of micro- and nanoelectronics, and artificial nanoantenna array-based meta-devices for the photonic applications.

Xiaoyuan Liu is a Ph.D. student in the Department of Electrical Engineering at City University of Hong Kong. Her research interests focus on meta-lens arrays and deep learning for the design and application of meta-devices. She received her B.Eng. in electronic information engineering at Tianjin University (2020).

Yanni Sun received her B.S. and M.S. from Xi’an JiaoTong University (China) and her Ph.D. from Washington University in Saint Louis, MO, all in Computer Science. She was an Associate Professor with tenure at Michigan State University. Currently, she is an Associate Professor in the Department of Electrical Engineering at City University of Hong Kong. Her research interests include deep learning-based sequence analysis, computational biology, genomic data mining, algorithm design, and software development. In particular, she is interested in applying computational models, algorithms, and machine learning, deep learning, and data mining methods to analyze biological data.

Din Ping Tsai received his Ph.D. from the University of Cincinnati in 1990. He was the Head and Chair Professor of the Department of Electronic and Information Engineering at The Hong Kong Polytechnic University from 2019 to 2021. Currently, he is the Chair Professor of the Department of Electrical Engineering at City University of Hong Kong. He is an elected Fellow of AAAS, APS, EMA, IEEE, NAI, OSA, and SPIE. He was recognized by Web of Science (Clarivate Analytics) as a Highly Cited Researcher in the field of cross-field in both 2019 and 2020. His current research interests are meta-devices, quantum information technology, nanophotonics, and advanced micro- and nanofabrication and design.

Author Contributions

^# These authors contributed equally to this work.

The authors declare no competing financial interest.