A review of simulation optimization with connection to artificial intelligence

Abstract

We review simulation optimization methods and their connection with artificial intelligence (AI) techniques. In particular, we focus on two areas: stochastic gradient estimation, which plays a central role in training neural networks for deep learning and reinforcement learning; and ranking and selection, which can be used as the node selection policy in Monte Carlo tree search. We also review the literature on inventory management, which has been studied in both simulation optimization and AI.

1. Introduction

Simulation is a powerful tool for analyzing complex stochastic systems found in manufacturing and supply chains, transportation, healthcare, call centers, finance, and many other fields [1], [2], [3]. Simulating a complex stochastic system is computationally expensive, and accurate system performance evaluation often requires a large number of simulation replications. Optimization of such complex stochastic systems may involve a large number of alternatives and decision variables. In recent decades, a stream of literature, e.g., [4], [5], [6], [7], has attempted to combine methods in simulation and optimization efficiently.

Inventory management is a central area in operations management. In a large e-commerce platform, there could be millions of stock keeping units (SKUs) stored in a complex network of warehouses. Simulation can be used to evaluate the cost of an inventory system. How to make reordering decisions for numerous SKUs to minimize the inventory costs is a complicated optimization problem.

Optimizing the reordering decisions is a Markov decision problem (MDP), which typically suffers from the curse of dimensionality in computation. To address this computational difficulty, many works focus on a simple policy such as base-stock and (s,S) policies that are only optimal under strict conditions that may not hold in practice, e.g., identically and independently distributed (i.i.d.) demands. In this case, artificial intelligence (AI) techniques have the potential to solve the original MDP. A new paradigm for solving the MDP is deep reinforcement learning (DRL), which parameterizes the reordering policy by an artificial neural network (ANN), where numerous parameters can be tuned through simulation optimization.

In this paper, we review two classic areas in simulation optimization: stochastic gradient estimation [8] and ranking and selection (R&S) [9]. We will discuss how these methods underpin modern AI techniques and the different focuses of simulation optimization and modern AI studies. The central issues addressed in AI have differed from those in other more traditional applications of simulation optimization, which will be discussed in detail in the following sections. We illustrate the application background using inventory management, a central problem in supply chain management, which has been studied in simulation [10], [11], [12], [13] and recently in AI [14], [15], [16], [17].

Prior to the recent buzz of ChapGPT, the most celebrated success story in AI was the stunning success of AlphaGo and its successors, developed using ANNs and Monte Carlo tree search (MCTS) [18], [19], [20], and trained by DRL to approximate value and policy functions. For an introduction to MCTS, see Fu [21], and for more information on ANNs, see Goodfellow et al. [22]. To a large extent, ANNs, MCTS, and DRL used in AlphaGo define modern AI.

ANNs are foundational to modern AI, and their training can be formulated as a simulation optimization problem, where stochastic gradient estimation is key for training them in deep learning and reinforcement learning [23], [24]. The upper confidence bound algorithm for trees (UCT) [25], derived from [26], [27] based on the multi-armed bandit (MAB) problem, is a popular node selection policy in MCTS [28], [29], [30], and is used in AlphaGo.

The rest of this paper is organized as follows. Section 2 introduces the general problem setting in simulation optimization. An overview of simulation optimization approaches and their connection to AI is provided in Section 3. Sections 4 and 5 introduce the two areas of stochastic gradient estimation and simulation budget allocation and their applications to AI. The last section concludes and discusses future directions. Most of the material of this review paper is taken from [31], including many parts verbatim and Figs. 2, Fig. 5, Fig. 6, Fig. 7, Fig. 8–9.

Fig. 2.

99% confidence intervals for the means of five alternatives.

Fig. 5.

Two tic-tac-toe board configurations to be considered.

Fig. 6.

Game tree for the first tic-tac-toe board configuration (assuming “greedy optimal” play), where we note that if the opponent’s moves were completely randomized, “O” would have an excellent chance of winning!.

Fig. 7.

Decision tree for the first tic-tac-toe board configuration ofFig. 5.

Fig. 8.

Decision tree for the first tic-tac-toe board configuration with non-optimal greedy moves.

Fig. 9.

Four steps in Monte Carlo tree search.

2. Problem setting

A generic simulation optimization problem can be written as

$$\underset{\theta \in \Theta }{\min }J\left(\theta \right):=\mathbb{E}\left[L(\theta ;\omega )\right],$$

where $\Theta$ is the feasible space of controllable parameters $\theta$, and $\omega$ represents the randomness of the simulation model $L(\theta ;·)$. In a simulation study, $J\left(\theta \right)$ cannot be evaluated analytically but can be estimated by the average of simulation replications, i.e., $\frac{1}{n}{\sum }_{i=1}^{n}L(\theta ;{\omega }_i)$, where ${\omega }_i$, $i=1,\dots ,n$, represent the i.i.d. inputs of the simulation model. Given a parameter $\theta$, running a simulation $L(\theta ;\omega )$ would typically be computationally expensive. The feasible space $\Theta$ could be continuous or discrete. Continuous feasible spaces face computational challenges when the dimension of $\Theta$ is high; similarly for discrete feasible spaces when the cardinality is large. Thus, the main challenge in solving the simulation optimization problem arises from the computational burden for evaluating the expected performance of the simulation model and searching for the optima of the expected performance.

We consider a periodic-review serial supply chain system containing $K$ intermediate echelons between a manufacturer and customers. An example with $K=3$ is shown in Fig. 1. We assume that the inventory replenishment at any stage takes the same amount of a fixed lead time denoted by $L$. For echelon $i$ at period $t$, we use $I_t^i$, $B_t^i$, and $O_t^i$ to denote on-hand inventory at the beginning, backlogs (accumulative unmet demand) at the beginning and ordering quantity, respectively. The quantity of goods that are shipped is

$$P_t^i=\min \{B_t^i+O_t^{i-1},I_t^i+P_{t-L}^{i+1}\}.$$

Actor $i$ makes decision for echelon $i$. The update of the backlog $B_t^i$ and inventory $I_t^i$ for actor $i$ can be written as

$$B_{t+1}^i=(B_t^i+O_t^{i-1}-P_t^i){}^{+},I_{t+1}^i=(I_t^i+P_{t-L}^{i+1}-P_t^i){}^{+}.$$

Cost $C_t^i$ of echelon $i$ is given by

$$C_t^i=h^i{I}_t^i+b^i{B}_t^i,$$

where coefficients of backlog costs and holding costs are given by $b^i$ and $h^i$, respectively. The total costs of the system in period $t$ can be expressed by

$$C_t^{\text{total}}=\sum _{i=1}^K{C}_t^i=\sum _{i=1}^K(h^i{I}_t^i+b^i{B}_t^i).$$

Fig. 1.

A serial supply chain system with three intermediate echelons.

The randomness of the inventory system consists of customer demand $O_t^0$. A simple inventory policy is the $(s,S)$ policy. Under an $(s,S)$ inventory control policy, when the inventory position (which includes inventory on hand plus that on order) falls below $s$ at an order decision point (discrete points in time in a periodic review setting and any point in time in a continuous review setting), then an order is placed for the amount that would bring the inventory position up to $S$ [32]. For the $(s,S)$ policy, the controllable vector (set of decision variables) is two-dimensional, i.e., $\theta =(s,S)$. For DRL, the inventory policy $\pi$ can be represented by an ANN where there are numerous parameters, so the vector $\theta$ is high-dimensional. For the inventory system, the output of the simulation model is $L(\theta ;\omega )={\sum }_{t=0}^T{\eta }^t{C}_t^{\text{total}}$. How to optimize the inventory policy is transferred to optimize $\theta$. Detailed description about the inventory system of the serial supply chain and more complicated inventory systems can be found in [15] and [16].

Next we turn to a recent application of simulation optimization, i.e., training an ANN in AI. Examples of output include a multidimensional vector for a classification problem that groups subjects into classes, or scalar output from a regression problem that estimates relationships among variables. The input could be an image represented by a vector of pixels or text encoded by a certain rule. The inputs are typically randomly drawn from a large dataset, e.g., ImageNet [33]. An ANN is comprised of many layers of neurons, the basic computational unit. ANN models usually contain randomness such as dropouts that randomly cut off the connection between two neurons and neuronal processing noises injected into the computation of neurons (see e.g., Peng et al. [24]). The computational complexity of ANNs arises from the composition of a huge number of neurons that are layered, as the output vector of the neurons in the preceding layer is the input vector for the neurons in the subsequent layer. From the discussion above, an ANN can be viewed as a stochastic simulation model parameterized by synaptic weights. For deep learning, training an ANN requires solving the following simulation optimization problem:

$$\begin{array}{c}\hfill \underset{\theta \in \Theta }{\min }\mathbb{E}\left[L(X_{\theta }^{\left(\tau \right)}\left(n\right),\mathcal{O}\left(n\right))\right],\end{array}$$

where $L(·)$ is the loss function between the corresponding $n$th output of the ANN, $X_{\theta }^{\left(\tau \right)}\left(n\right)$, and the $n$th observation $\mathcal{O}\left(n\right)$, $\theta$ is the vector of parameters. For regression, $L(·)$ could be the quadratic loss function, and for classification, $L(·)$ could be the cross-entropy loss function.

ANNs are also used in reinforcement learning to parameterize the value function and action policy. An MDP, which is used to model the reinforcement learning problem, can be represented as a 5-tuple $(\mathcal{S},\mathcal{A},p,u,\eta )$, where $\mathcal{S}$ and $\mathcal{A}$ are state and action spaces, respectively; $p\left(s^{\prime }\right|s,a)$ is the transition probability with $s$ being the current state and $s^{\prime }$ being the future state one step ahead; $u(s^{\prime },a,s)$ is the reward function; $\eta \in (0,1)$ is the reward discount factor. One might aim to optimize a parameterized policy function $\pi \left(a\right|s;\theta )$, which is a distribution over the action space conditioned on the state, through interaction with the system. We denote the state and action at time $t\in \{0,1,2,\dots ,T\}$ by $s_t$ and $a_t$, where $s_t\sim p(·|s_{t-1},a_{t-1})$ and $a_t\sim \pi (·|s_t;\theta )$. The trajectory generated by the MDP can be thus defined as $\tau =\{s_0,a_0,s_1,\dots ,a_{T-1},s_T\}\sim \Pi (·;\theta )$. Here we have $\Pi (\tau ;\theta )=p\left(s_0\right){\prod }_{t=0}^{T-1}\pi \left(a_t\right|s_t;\theta )p\left(s_{t+1}\right|s_t,a_t)$, where $s_0$ is the initial state. The total reward of the trajectory is denoted as $U\left(\tau \right)={\sum }_{t=0}^{T-1}{\eta }^{t}u(s_{t+1},a_t,s_t)$. In an episode, a complete simulation trajectory of the action and state is generated by the interaction between the agent and environment. For reinforcement learning, training an ANN requires solving the following simulation optimization problem:

$$\begin{array}{c}\hfill \underset{\theta \in \Theta }{max}{\mathbb{E}}_{\tau \sim \Pi (·;\theta )}\left[U\left(\tau \right)\right].\end{array}$$ (1)

3. Overview of simulation optimization approaches

With the problem setting and application background introduced, we provide a brief overview of sample average approximation (SAA), stochastic approximation (SA), and R&S for solving simulation optimization problems.

3.1. Overview of SAA and SA

The first two approaches are for solving simulation optimization problems with continuous controllable parameters. For example, for the single item $(s,S)$ inventory control problem, the two parameters can be optimized on the real space with constraints $s,S\ge 0$ and $s<S$. When the dimension of the parameter space is high, gradient-based optimization techniques are computationally preferable for solving generic nonlinear optimization problems. Deterministic gradient search can be applied to solve the nonlinear optimization via SAA as follows [34], [35]:

$${\theta }_{k+1}={\theta }_k-{\lambda }_k\frac{1}{N}\sum _{i=1}^N{\nabla }_{\theta }L(\theta ;{\omega }_i){|}_{\theta ={\theta }_k},k\in \mathbb{N},$$

which closely resembles SA [36], [37]:

$${\theta }_{k+1}={\theta }_k-{\lambda }_k\widehat{\nabla }J\left({\theta }_k\right),k\in \mathbb{N},$$

where ${\nabla }_{\theta }L(\theta ;{\omega }_i)$, $i=1,2,\dots$, are unbiased (pathwise) stochastic gradient estimates given independently and identically distributed (i.i.d.) inputs of the simulation model, $\widehat{\nabla }J\left(\theta \right)$ is a gradient estimator for $J\left(\theta \right)$ (e.g., ${\nabla }_{\theta }L(\theta ;\omega )$), and ${\lambda }_k$ is the step size (learning rate). Comparisons between the performance of SAA and SA can be found in the literature, e.g., [38].

The SA approach with an unbiased stochastic gradient estimator (i.e., $\mathbb{E}\left[\widehat{\nabla }J\left(\theta \right)\right]=\nabla J\left(\theta \right)$) is referred to as the Robbins–Monro algorithm, and the SA with the (biased) finite-difference (FD) gradient estimator is called the Kiefer–Wolfowitz algorithm. The FD method for gradient estimation is always implementable without the need for information on the stochastic model. However, FD may have undesirable statistical properties, and its computational burden increases linearly with the dimension of the controllable parameters. The Kiefer–Wolfowitz algorithm requires tuning more parameters than the Robbins–Monro algorithm. These drawbacks make the Kiefer–Wolfowitz algorithm unfavorable for solving large-scale simulation optimization problems. For training an ANN, the Robbins–Monro algorithm and its variant are predominant. Recently, Hu et al. [39] provide a three-timescale stochastic approximation to optimize a quantile, based on two unbiased estimators for the gradient of the distribution function and density function, and Jiang et al. [40] develop a two-timescale stochastic approximation for optimizing a quantile in the context of DRL, based on the unbiased estimator for gradient of the distribution function. A weakness of gradient-based optimization methods is that they can only guarantee convergence to a local optimum. Some metaheuristics, such as simulation annealing, can guarantee convergence to a global optimum. In high-dimensional optimization problems such as training ANNs, convergence to a global optimum would often be computationally intractable, whereas convergence to a local optimum would usually be satisfactory enough.

Training an ANN typically involves the empirical minimization of a loss function by tuning the parameters. Factors such as the number of layers, the type of activation, and the layer sizes are chosen based on domain-specific applications [22]. However, SAA is generally not practical for training ANNs, because the input data may be huge (e.g., an image dataset) and the computational machinery of an ANN is heavy; hence, computing the summation of gradients of ANN over all data, i.e., ${\sum }_{i=1}^N{\nabla }_{\theta }L(\theta ;{\omega }_i)$, requires too much memory and computation to update the parameters. On the other hand, the update in SA requires much less memory and computational resources. However, to accelerate training the ANN, a mini-batch of inputs $X^{\left(1\right)}\left(n\right)$, $n=1,\dots ,l$, is put into a tensor, and a parallel computing platform can be leveraged to process the computation of ANN and the stochastic gradient estimate with the tensor as input. In practice, the entire input dataset is split into mini-batches with $l$ pieces of data, $\lceil N/l\rceil$ iterations of stochastic gradient descent using data in a mini-batch are run in an epoch, and random reshuffling of data is done at the end of an epoch before repeating the iterative process. The (global or local) convergence of SA is established under an assumption on step sizes (e.g., ${\sum }_{k=1}^{\infty }{\lambda }_k=\infty$, ${\sum }_{k=1}^{\infty }{\lambda }_k^2<\infty$) and an assumption on the randomness of the stochastic gradient estimator [36]. The random reshuffling of data at the end of an epoch would generally make the randomness of the stochastic gradient estimator violate the assumptions required to ensure the convergence of SA. Tuning the step sizes in SA for training a large-scale ANN on a large dataset is usually a time-consuming and labor-intensive manual task. The step size is often kept constant in the beginning of the training and decreased after the loss decreases to a certain level in the later stages of the training. AdaGrad and Adam, variants of SA, introduce momentum and scaling factors for different coordinates to facilitate training the ANN [41], [42].

3.2. Overview of ranking and selection

We next turn our attention to R&S, which aims to solve discrete simulation optimization problems via efficient sampling. Specifically, our goal is to solve maximization problem $\arg {\max }_{i=1,\dots ,k}{\mu }_i$, where the expectation of alternative $i$ is unknown but can be estimated by the average of independently and identically distributed simulated samples $X_{i\ell }$, $\ell \in {\mathbb{Z}}^{+}$. In a queueing network, scheduling servers at different queues to improve the performance is a combinatorial optimization problem with discrete controllable parameters. For discrete parameters where there is no information available on the local topology of a solution, finding the best alternative may require pairwise comparisons between different alternatives [9]. The problem of allocating computing resources to competing alternatives for finding the best alternative is a research question often called the R&S problem in the simulation literature, and it is closely related to the MAB problem in machine learning. In R&S, the primary goal has to do with correct selection of the best alternative, whereas traditional MAB work aims to maximize cumulative reward or minimize cumulative regret. Applications particularly suitable for methods in R&S include drug testing to guarantee a high probability that a lifesaving drug is effective. However, such methods may not be best suited for applications like designing a complex manufacturing system, where the possible number of alternative designs could be huge, running a simulation model would be expensive, and a managerial decision might need to be made quickly and adaptively to varying market conditions.

R&S techniques impact the efficiency of node selection in Monte Carlo tree search (MCTS). The successful Go-playing AI algorithm AlphaGo employs MCTS using the UCT node selection policy [18], where the number of possible states is of the order of ${10}^{170}$, more than the number of atoms in the universe. MCTS provides a more effective way than traditional deterministic tree search techniques for estimating the state-action value function(s) via Monte Carlo simulation by following procedures of node selection and rollout. To improve the node selection policies in MCTS, techniques such as the Optimal Computing Budget Allocation (OCBA) and the Asymptotically Optimal Allocation Policy (AOAP) from R&S have been applied to MCTS, demonstrating better performance than the original UCT node selection policy [29], [30].

For simulation-based reinforcement learning algorithms to perform well, the state space should be explored sufficiently. Q-learning [43] also requires thorough exploration of the action space. Since pure greedy policies may lead to an insufficient exploration of the state-action space, $ϵ$-greedy, Boltzmann machines [43], and other MAB-based methods have been used to encourage more exploration. R&S-based methods have also been used to better balance exploration and exploitation [44], [45].

4. Stochastic gradient estimation

In this section, we focus on stochastic gradient estimation, the key element in implementing gradient-based simulation optimization methods such as SA and SAA introduced in the last section. For an introduction, see Fu [8]. Traditional application areas for stochastic gradient estimation include discrete event systems [46], financial engineering, and risk management [47], [48], [49], [50]. Fu [10] studies the stochastic gradient estimation for the two parameters in the $(s,S)$ inventory system. Recently, this topic has attracted attention in AI and machine learning; see a review paper by DeepMind [51]. In Section 4.1, we introduce classic problems and methods in stochastic gradient estimation, and we show how the methods can be applied to deep learning and reinforcement learning in Sections 4.2 and 4.3, respectively.

4.1. Classic problems and methods

Stochastic gradient estimation methods aim to estimate the gradient of the expected performance of a stochastic model ${\nabla }_{\theta }\mathbb{E}\left[L(\theta ;\omega )\right]$. The infinitesimal perturbation analysis (IPA) or pathwise gradient is given by ${\nabla }_{\theta }L(\theta ;\omega )$, and under appropriate conditions justifying the interchange of gradient and expectation operators, it is unbiased, i.e., satisfies

$$\mathbb{E}\left[{\nabla }_{\theta }L(\theta ;\omega )\right]={\nabla }_{\theta }\mathbb{E}\left[L(\theta ;\omega )\right].$$

The required conditions for unbiasedness stem from applying the dominated convergence theorem, which for the IPA estimator generally translates to requiring that the performance function $L(·;\omega )$ be Lipschitz continuous almost surely [52]. When $L(·;\omega )$ is discontinuous, smoothed perturbation analysis (SPA) can be applied to derive an unbiased gradient estimator by conditioning on certain quantities to smooth out the discontinuities [53].

Suppose the density $f(·;\theta )$ of $X$ has support $\Omega$, which is independent of the parameter $\theta$. When the stochastic model depends on the parameter $\theta$ only through the input variables, i.e., $h\left(X\right)$, the gradient of the expected performance of a stochastic model can be given by

$$\begin{array}{c}\hfill {\nabla }_{\theta }\mathbb{E}\left[h\left(X\right)\right]=\mathbb{E}\left[h\left(X\right){\nabla }_{\theta }\log f(X;\theta )\right],\end{array}$$

under certain conditions to justify the interchange of the gradient and expectation. The likelihood ratio (LR) gradient estimator is $h\left(X\right){\nabla }_{\theta }\log f(X;\theta )$. Since the parameter does not directly appear in $h$, the LR estimator does not require the continuity of $h$ to be unbiased. When the parameter $\theta$ appears directly in $h$, the push-out LR could be applied, which requires an explicit change of variables to push out the parameter $\theta$. Interested readers may refer to [54] for details of push-out LR.

When both IPA and LR are unbiased, as a rule-of-thumb, IPA has a smaller variance than LR [46], [48]. Cui et al. [55] provide sufficient conditions under which IPA has smaller variance, as well as a counterexample demonstrating that LR has a smaller variance when the sufficient condition is not satisfied. L’Ecuyer [56] provides a framework to unify IPA and LR, which is referred to as the combined derivative estimator in a recent review by Glasserman [57]. When $\Omega$ is dependent on the parameter $\theta$, Wang et al. [50] offer a technique to push out the parameter using a change of variables. Peng et al. [58], [59] consider an explicit stochastic model $\varphi \left(g\right(X;\theta \left)\right)$, where $g$ is a vector of functions with second-order differentiability and $\phi$ is a measurable function not necessarily continuous, and derive an unbiased generalized likelihood ratio (GLR) estimator in this setting. GLR generalizes LR by allowing $\theta$ to directly appear in the stochastic model and generalizes IPA by allowing discontinuities in the stochastic model. When the parameter $\theta$ does not directly appear in the stochastic model, GLR reduces to LR; when there is an explicit change of variables to push out the parameter, GLR coincides with the push-out LR, so GLR can be viewed as a generalization of push-out LR, which “implicitly” pushes out the parameter. See also [60] for some recent development. Because $\varphi$ is only required to be a measurable function, which is very general, this flexibility allows GLR to handle many applications with discontinuities under a single umbrella.

Stochastic gradient estimation research for quantiles, conditional value-at-risk, and distortion risk measure is relatively recent [61], [62], [63], [64], [65], and is largely dependent on estimating the gradient of the distribution function [66], [67]. Stochastic gradient estimators for risk measures based on a single simulation run are generally biased. Much research attention has focused on analyzing the asymptotic properties of the estimators.

Traditional theoretical discussions in stochastic gradient estimation focus on the statistical properties of the estimator, such as unbiasedness and small variance. Less attention has been paid to the computational complexity, parallelization, and scalability of the estimator, which are critical for applications in AI. This may be due to the relatively lower-dimensional problems and the lack of well-accepted large-scale testbeds for analyzing the computational properties of estimators. In contrast, the AI community is built upon an open source ecosystem, where numerous ANN models, training algorithms, and datasets can be accessed online, some of which become benchmarks and testbeds of successful applications of AI such as computer vision (CV) and natural language processing (NLP) [68], [69], [70]. The testbed SimOpt, available at https://github.com/simopt-admin/simopt/wiki [71], [72], [73], [74], serves as an analogous valuable resource for the simulation optimization research community, with a diverse set of vetted examples, although relatively fewer large-scale industrial-level applications that are frequently found in AI testbeds.

4.2. Stochastic gradient estimation for deep learning

Training ANNs is central for deep learning, a prominent area in AI. Backpropagation (BP), an important gradient estimation technique embedded in most open source machine learning platforms for training ANNs, is shown by Peng et al. [24] to be pathwise equivalent to IPA, and backward propagation of errors can reduce computational complexity. Due to the nonlinearity of the activation function, tuning parameters in training ANNs is handled by SA algorithms. As there are often many parameters (the synaptic weights and biases to connect the neurons), the computational complexity in each descent step could be gigantic. In this regard, the BP method [75] provides an efficient way to compute the gradient at each step. It systematically applies the chain rule to propagate the perturbation effect of each parameter, by first making a forward pass to evaluate the function signals through the neuron layers, followed by a backward pass of the derivative components, or the so-called error signals, that ultimately allows an evaluation of the gradient with respect to all parameters simultaneously. Specifically, derivatives of all parameters can be decomposed into products of function signals and error signals, which results in reduced computational complexity relative to IPA without backward propagation of errors. Interested readers may refer to [24] for technical details.

Peng et al. [24] show that an LR-based (push-out) method can significantly improve the robustness of an ANN. The LR estimator is unbiased even if $\phi$ or $L$ is discontinuous. This provides substantially more flexibility in choosing $\phi$ and $L$. For example, we may choose $L$ to be the zero-one loss function and $\phi$ to be a threshold function. Moreover, LR avoids the issue of vanishing gradient, which may arise in BP. However, the LR estimator is demonstrably simpler, as it does not need a backward recursion as in BP. The calculations of the LR-based estimator for parameters in each neuron are performed completely inside the neuron, which can be parallelized and results in substantial computational savings for higher-order derivatives. Interested readers may refer to [24] for technical details. By applying variance reduction techniques such as antithetic variates, Jiang et al. [76] have extended the LR method in [24], which was developed for a multi-layer perceptron (MLP), a basic ANN structure, to more complicated ANNs including CNNs, RNNs, graph neural networks (GNNs), and spiking neural networks (SNNs) with applications including CV, NLP, and social network analysis. ANNs trained by the LR-based method are more robust to perturbations in the input data than those trained by BP. Because the LR method only requires the local information of the model and the final loss for training ANN, it has a significant computational advantage in fine-tuning a module of a large ANN model.

4.3. Stochastic gradient estimation for reinforcement learning

Reinforcement learning, particularly DRL, has recently achieved many breakthroughs in AI. Policy gradient methods, an important class of DRL algorithms, solve problem (1) using stochastic gradient ascent. The key to implementing policy gradient methods is to estimate the gradient. Specifically, the gradient can be expressed as follows:

$$\begin{array}{c}\hfill {\nabla }_{\theta }\mathbb{E}\left[U\left(\tau \right)\right]=\mathbb{E}\left[U\left(\tau \right)\sum _{t=0}^{T-1}{\nabla }_{\theta }\log \pi \left(a_t\right|s_t;\theta )\right],\end{array}$$ (2)

where $\tau =\{s_0,a_0,s_1,\dots ,a_{T-1},s_T\}$ is the trajectory of one episode in the MDP, and the interchange of the gradient and integral in the second equality can be justified by the dominated convergence theorem [77]. The term $U\left(\tau \right){\sum }_{t=0}^{T-1}{\nabla }_{\theta }\log \pi \left(a_t\right|s_t;\theta )$ inside the expectation on the right-hand side of (2) is the LR estimator. We rewrite (2) as

$$\begin{array}{c}\hfill \mathbb{E}\left[U\left(\tau \right)\sum _{t=0}^{T-1}{\nabla }_{\theta }\log \pi \left(a_t\right|s_t;\theta )\right]=\mathbb{E}\left[\sum _{t=0}^{T-1}{\nabla }_{\theta }\log \pi \left(a_t\right|s_t;\theta )G_t\left(\tau \right)\right],\end{array}$$

where $G_t\left(\tau \right)={\sum }_{t^{\prime }=t}^{T-1}{\eta }^{t^{\prime }}u(s_{t^{\prime }+1},a_{t^{\prime }},s_{t^{\prime }})$ is the cost-to-go. Intuitively, the equality holds because the action determined by $\pi \left(a_t\right|s_t;\theta )$ at time $t$ would not affect the returns before time $t$ but only affect the cost-to-go. The DRL algorithm that uses the gradient estimator above in the SA to optimize the parameter $\theta$ in ANN is called REINFORCE [78].

To improve data efficiency for the policy gradient method, we rewrite (2) as

$$\begin{array}{c}\hfill {\nabla }_{\theta }\mathbb{E}\left[U\left(\tau \right)\right]=\mathbb{E}\left[\sum _{t=0}^{T-1}\frac{{\nabla }_{\theta }\pi \left(a_t\right|s_t;\theta )}{\pi \left(a_t\right|s_t;{\theta }_0)}{G}_t\left(\tau \right)\right].\end{array}$$

To reduce the variance of gradient estimator, a baseline $b\left(s_t\right)$ is subtracted from the cost-to-go $G_t\left(\tau \right)$:

$$\frac{{\nabla }_{\theta }\pi \left(a_t\right|s_t;{\theta }_t)}{\pi \left(a_t\right|s_t;{\theta }_0)}(G_t\left(\tau \right)-b\left(s_t\right)).$$

This baseline $b\left(s_t\right)$ independent of $\theta$ does not affect the expectation of the stochastic gradient estimator. The term ${\nabla }_{\theta }\pi \left(a_t\right|s_t;{\theta }_t)b\left(s_t\right)/\pi \left(a_t\right|s_t;{\theta }_0)$ can be viewed as a control variate that has zero expectation when $b\left(s_t\right)$ is not controlled by $a_t$ and tends to be positively correlated with the stochastic gradient estimator, so it does not alter the expectation of the stochastic gradient estimator but can reduce its variance. For the actor-critic type of algorithms [43], the baseline is approximated by an ANN that is trained by the samples of the cost-to-go $G_t\left(\tau \right)$. The clip technique used in proximal policy optimization (PPO) can further reduce the variance of the policy gradient algorithm [69].

Jiang et al. [40] have developed policy gradient methods for quantile-based DRL by a two-time-scale SA algorithm. To improve the learning efficiency, Jiang et al. [40] use the bootstrap technique to allow multiple updates of parameters in one episode and truncate the trajectory of an episode into trajectories of different lengths, which are randomly picked by bootstrapping. Importance sampling and the clip technique are also applied in [40] to propose a quantile-based PPO (QPPO).

To apply DRL to the inventory management, the ordering quantity $O_t^i$ of the $i$th echelon is treated as actor $i$’s action $a_t^i$ and a discrete action space for each actor is ${\mathbb{A}}^i=\{0,\dots ,O^{\max }\}$, where $O^{\max }$ denotes the maximum ordering quantity in each period. The on-hand inventory, backlogs, downstream demand, and in-transit orders $(I_t^i,B_t^i,O_{t-1}^{i-1},P_{t-L}^{i+1},\dots ,P_{t-1}^{i+1})$ are included in the state of the MDP for optimizing the inventory ordering policy. When the action $a_t^i$ follows a policy modeled by a neural network ${\pi }_{\theta }^i$, the output is $L(\theta ;\omega )={\sum }_{t=0}^T{\eta }^t{C}_t^{\text{total}}$ parameterized by $\theta$ in the neural network. Rather than optimizing the two-dimensional parameter in the $(s,S)$ inventory policy, training the neural network requires optimizing the high-dimensional parameter $\theta$, which could be performed by DRL. The costs $C_t^i$ and $C_t^{\text{total}}$ are used in the reward of the DRL. Wang et al. [79] and Liu et al. [15] apply an DRL and multi-agents DLR to optimize the expected cost of an ordering policy for a multi-echelon inventory system with and without information sharing in execution, respectively. Liu et al. [16] apply PPO to solve a large-scale inventory management problem of an e-commerce company with a complicated warehouse network. Zheng et al. [17] propose a dual-agent DLR algorithm to solve a joint dynamic pricing and inventory problem. Jiang et al. [40] apply QPPO to optimize the quantile of an ordering policy for a multi-echelon inventory system.

5. Ranking and selection

In this section, we provide a brief overview of R&S, an actively studied field in simulation optimization in the discrete input setting. For more information, readers can consult established reference volumes such as Bechhofer et al. [80], Chen and Lee [81], and Powell and Ryzhov [82]. Traditional applications of R&S include optimizing complex discrete event dynamic systems (DEDS) that are computationally intensive to simulate [81], and finding the most effective drugs, where the economic cost of each sample for testing the effectiveness of the drug is expensive [82]. Recent applications in AI can be found in Xiang et al. [83]. In Section 5.1, we introduce classical problems and methods in R&S, and we show how an AOAP method is adapted to be the selection policy of MCTS in Section 5.2.

5.1. Classical problems and methods

The goal of a classical R&S problem is to select the best alternative with the smallest (or largest) mean from a finite number of alternatives with unknown means ${\mu }_1,\dots ,{\mu }_k$. In the context of simulation, each alternative is typically a complex stochastic model (e.g., a queueing network), so the mean of the random output of the model (e.g., average system time) does not have an analytical form but can be estimated by simulation. The unknown means of alternatives can be estimated by independent random samples (simulation replications) following a joint distribution, i.e., $(X_{1,\ell },\dots ,X_{k,\ell })\sim Q(·;\theta )$, $\ell \in {\mathbb{Z}}^{+}$, where $\theta$ contains all the unknown parameters of the sampling distribution, including ${\mu }_1,\dots ,{\mu }_k$. The most common assumption in the literature of R&S is that the samples of different alternatives, i.e., $X_{i,\ell },X_{j,\ell }$, $i\ne j$, follow independent normal distributions. In practice, the output of a stochastic model is often an average of many random variables (e.g., an average system time of 100 customers), so that the normal assumption of samples is justified by the central limit theorem. If the samples do not follow a normal distribution, then a macro-replication, which is the average of a batch of simulation replications, is used as a sample in the experiment; this macro-replication is approximately normally distributed. By the law of large numbers, sample means converge to means almost surely (a.s.), i.e., ${\sum }_{\ell =1}^n{X}_{i,\ell }/n\to {\mu }_i$ a.s., as $n\to \infty$, $i=1,\dots ,k$. Thus, as the simulation budget goes to infinity, the best alternative could be eventually selected. However, simulation replications are typically expensive, which often leads to a very limited simulation budget in practice.

Fig. 2 presents the corresponding 99% confidence intervals for the means of five alternatives. In this example, it is highly unlikely for alternatives 1, 4, and 5 to be the best alternative (with the lowest mean). Therefore, it seems unnecessary to spend a significant number of simulation replications on accurately learning the means of alternatives 1, 4, and 5 rather than focusing on learning the means of the other two more promising alternatives to distinguish which one is actually the best. This example motivates the study of a central issue in the research of R&S, i.e., how to allocate simulation replications intelligently to improve sampling efficiency. Intuitively, allocating more simulation replications to the alternatives with smaller means and higher variances seems to be reasonable for efficiently selecting the best alternative. In the literature, this intuition is summarized as the “mean-variance” or “exploitation-exploration” tradeoff. To formalize the study on the issue, we need to introduce a metric to measure sampling efficiency. In R&S, a popular metric is the probability of correct selection (PCS), i.e., $PCS\left(\theta \right)=P({\mu }_{\mathcal{S}}<{\mu }_i,i\ne \mathcal{S}\left|\theta \right)=\mathbb{E}\left[{\mathcal{L}}_{0-1}\left({\mathcal{E}}_T\right)\right|\theta ]$, where $\mathcal{S}$ is the selection decision made after allocating a total of $T$ simulation replications, ${\mathcal{E}}_t$ is the information set obtained after allocating $t$ simulation replications, and ${\mathcal{L}}_{0-1}\left({\mathcal{E}}_t\right)=\mathbf{1}\{\mathcal{S}=i^{*}\}$ is the zero-one loss after allocating $t$ simulation replications with $i^{*}=\arg {\min }_{i=1,\dots ,k}{\mu }_i$. Discussion of different metrics can be found in Eckman and Henderson [84]. Notice that for the metric commonly used in R&S, the loss only occurs at the end after allocating all $T$ simulation replications.

The MAB problem is closely related to R&S [85], [86], [87]. The goal of a classic MAB problem is similar to that of R&S, i.e., to play most often the best arm from a finite number of arms with unknown expected rewards (means). A major difference from R&S lies in the objective, which for traditional MAB is the total expected regret (or reward) accumulated each time an arm is pulled, although more recently MAB has also considered a setting termed best-arm identification (BAI) or the pure exploration problem, where a correct selection criterion is also used. A good balance of exploitation and exploration is also key to intelligently sampling each alternative in the MAB problem. Thompson sampling (TS) [88] is a popular sampling procedure in the traditional MAB setting, but the performance of TS is notoriously poor for R&S or BAI problem [87]. Recently, Russo [89] proposed a top-two Thompson sampling (TTTS) procedure that performs well for the R&S problem. Compared to TS, TTTS tends to explore more when allocating samples. Shi et al. [90] extend TTTS to select contextually dependent top-$m$ alternatives.

In R&S, numerous existing sampling procedures can be basically categorized into two branches developed from two distinct perspectives, which can be viewed as duals of each other. An earlier branch, studied under an indifference zone (IZ) paradigm [91], [92], aims to guarantee a predetermined PCS level for parameters satisfying an IZ assumption, i.e., ${\min }_{\theta \in \Xi }PCS\left(\theta \right)\ge \alpha$ with $\Xi =\{\theta \in \Theta :{\mu }_{i^{*}}+\delta \le {\mu }_i,i\ne i^{*}\}$. Since the sampling procedures derived in the IZ paradigm need to guarantee a PCS level for the worst-case scenario, called the slippage configuration [93], in practice these procedures tend to be overly conservative in that they often require allocating more simulation replications than necessary for guaranteeing a specified PCS level. A recent theoretical advance by Fan et al. [94] eliminates the need for the IZ assumption in guaranteeing PCS, although the drawback of conservatism still persists.

Rather than focusing on the worst-case scenario, Chen [95] (see also Chen et al. [96]) developed an alternative approach called the optimal computing budget allocation (OCBA) method. OCBA solves a static optimization problem, i.e.,

$$\underset{n_1+\cdots +n_k=T}{\max }PCS\left(\theta \right),$$

with $n_i$ being the number of simulation replications allocated to alternative $i$. With several approximations, an asymptotically optimal sampling ratio with an analytical formula can be obtained, given the unknown parameter $\theta$ [97], [98]. Subsequently, an estimate for $\theta$ is incorporated into the OCBA formula. This is done using initial samples that are usually equally distributed to each alternative in the first stage. As a result, the remaining samples can be allocated based on the sampling ratio suggested by OCBA, thereby enhancing sampling efficiency. Here the sampling efficiency should not be defined as the PCS, given a specific parameter $\theta$. For a given parameter, we can directly select the best alternative by sorting means which are a part of $\theta$ without running simulations. Therefore, a reasonable performance metric could be an integrated PCS (IPCS) over a weighting prior measure $F(·)$ on the parameter space $\Theta$, i.e., $IPCS={\int }_{\theta \in \Theta }PCS\left(\theta \right)F\left(d\theta \right)$. Under a Bayesian framework, a dynamic allocation and selection (A&S) policy to optimize IPCS is defined in Peng et al. [99], and a Bellman equation for solving the optimal A&S policy is rigorously established in Peng et al. [100]. Well-known Bayesian sequential sampling procedures include expected value of improvement (EVI) in Chick et al. [101], knowledge gradient (KG) in Frazier et al. [102], and expected improvement (EI) in Ryzhov [103]. KG and EI sequentially allocate a sample to optimize myopic surrogate criteria.

The dynamic decisions in the R&S problem can be formulated as an A&S policy [99]. The allocation policy is a sequence of mappings ${\mathcal{A}}_t(·)=(A_1(·),\dots ,A_t(·))$ that sequentially allocates each sample to an alternative based on collected information, and the selection policy $\mathcal{S}(·)$ makes a final decision to select the best alternative after exhausting all simulation replications. From Peng et al. [100], the optimal A&S policy satisfies a Bellman equation. The commonly assumed selection policy chooses the alternative with the largest posterior mean, even though it is not the optimal selection policy.

Peng et al. [100] propose a dynamic sampling procedure derived in an approximate dynamic programming (ADP) paradigm. The idea of ADP is to find a suitable value function approximation (VFA) for the Bellman equation. A simple treatment is to fix the selection policy to select the alternative with the largest posterior mean, and look one step ahead for allocating the next replication. Subsequently, VFA becomes an integral of the standard normal density over a region encompassed by $k-1$ hyperplanes. The VFA can be further simplified to be an analytical form. AOAP has an analytical form that reflects a similar (posterior) mean-variance tradeoff as the OCBA formula. Not only is AOAP consistent but it also sequentially achieves the asymptotically optimal sampling ratio.

Beyond the classic R&S problem, recent advances have been made in subset selection [104], [105], [106], feasibility identification [107], [108], quantile-based simulation sample allocation [109], [110], context-dependent simulation sample allocation [90], [111], [112], [113], and large-scale parallel computing simulation sampling allocation [114], [115], [116], [117]. Many of these recent advances have not yet been applied to AI, where they have the potential for improving computational efficiency.

The fixed-budget R&S problem as MDP can be characterized by the following Bellman equation: for $0\le t<T$,

$$a_{t+1}^{*}=\arg \underset{i=1,\cdots ,N}{\max }Q(s_t,i),$$

$$V_t\left(s_t\right)=Q(s_t,a_{t+1}^{*})=\mathbb{E}[Q\left(s_t,a_{t+2}^{*})\right|s_t,a_{t+1}^{*}]=\mathbb{E}[V_{t+1}\left(s_{t+1}\right)|s_t,a_{t+1}^{*}],$$

where $a_{t+1}^{*}$ is the optimal allocation policy at (t+1)-th step, $Q(s_t,i)$ is the state-action-value function, which represents the reward value of choosing alternative $i$, and $s_{t+1}$ is determined by $s_t$ and newly allocated observation. Zhou et al. [118] propose to solve the MDP via a rollout method depicted in Fig. 3, where the state-action-value function is estimated by an average of simulated sample paths following a rollout policy with the only reward received at the end, i.e., 0 if the best alternative is incorrectly selected and 1 if the best is selected correctly.

Fig. 3.

The decisions in the rollout process.

The rollout policy is much slower compared with the traditional R&S method. Zhou et al. [118] propose to use an offline-trained ANN for learning the value functions estimated by the rollout policy and employing the ANN to solve the SDP online. In Fig. 4, a neural network is trained to specify the allocation policy and base policy for rollout, and another neural network is trained to evaluate the PCS for implementing the stopping rule. In the pre-training procedure called AlphaRank, the input, output, and training objectives of the ANN are defined first; then labeled data is selected in the training process; finally reasonable evaluation criteria is defined to quantify the performance of the ANN model after training. This research demonstrates that not only simulation optimization contributes to recent advances in AI but also recent AI advancements can inspire novel approaches in simulation optimization.

Fig. 4.

NN training and evaluating architecture with the number of alternatives = 2.

5.2. Ranking and selection for Monte Carlo tree search

As mentioned earlier, AlphaGo employs MCTS, and simulation sample allocation is important for node selection in MCTS. We illustrate the MCTS for a board game using the simple example of tic-tac-toe, which is a well-known two-player game on a 3 $\times$ 3 grid. Players alternate turns, with the “X” player going first, aiming to get 3 marks in a row. Assuming the primary objective (for both players) is to win and the secondary objective is to avoid losing, the following “greedy” policy is optimal (for both players):

If a win move is available, then take it; else if a block move is available, then take it.

However, there remain numerous situations where neither a win move nor a block move is available. In traditional game tree search, these would be enumerated. We instead illustrate the MCTS approach, which samples the opposing moves rather than enumerating them.

When going first (as X), there are three unique first moves to consider – corner, side, and middle. If going second (as O), the possible moves depend of course on what X has selected. If the middle was chosen, then there are two unique moves available (corner or non-corner side); if a side (corner or non-corner) was chosen, then there are five unique moves available. Despite the game having a relatively small number of outcomes compared to most other games, such as checkers, enumerating them all for illustration purposes can be quite messy, so we further simplify by examining two different game situations that already have preliminary moves.

Assume henceforth that we are the “O” player. We begin by illustrating with two games that have already seen three moves, two by our opponent and one by us, so it is our turn to make a move. The two different board configurations are shown in Fig. 5. For the first game, by symmetry there are just two unique moves for us to consider: corner or (non-corner) side. In this particular situation, following the optimal policy above leads to an easy decision: corner move leads to a loss, and non-corner move leads to a draw; there is a unique “sample path” in both cases, thus eliminating the need to simulate. The trivial game tree and decision tree are provided in Figs. 6 and 7, respectively, with the game tree that allows non-optimal moves shown in Fig. 8. A more complicated game configuration can be found in Fu [21], and an online interactive Java demo of MCTS can be found at https://terpconnect.umd.edu/∼mfu/demo.

We view each configuration of a board game as a node and each move as an edge in MCTS. In general, MCTS consists of four steps: selection, expansion, simulation, and back-propagation, which are depicted in Fig. 9. The functions of the four steps are described below:

• •Selection: the MCTS algorithm traverses the current tree from the root node following a tree policy. The policy employs an evaluation function to optimally select nodes with the highest estimated value.
• •Expansion: a new child node is added to the tree to that node which was optimally reached during the selection step.
• •Simulation: a simulation is performed following a default policy until a result or predefined state is achieved.
• •Back-propagation: after determining the value of the newly added node, the remaining tree must be updated from the new node to the root node. The number of simulations stored in each node is incremented and if the new node’s simulation results in a win, then the number of wins is also incremented (similarly for a loss or draw).

Traditionally, the UCT algorithm in MAB has been used as the tree selection policy to balance the exploration and exploitation [25], whereas R&S procedures have been applied to the tree selection to further improve performance [29], [119]. In AlphaGo Zero, the policy network $f_{\pi }$ can assist the selection of actions. Specifically, to determine the next action $a_t^k$ at state $s_t^k$ in the $k$th rollout, UCT selects the action with the highest upper confidence bound as

$$a_t^k=\arg \underset{a\in \mathcal{A}}{\max }\{{\overline{Q}}^{k-1}(s_t^k,a)+c{f}_{\pi }(s_t^k,a)\frac{\sqrt{{\sum }_{a^{\prime }\in \mathcal{A}}{N}^{k-1}(s_t^k,a^{\prime })}}{1+N^{k-1}(s_t^k,a)}\},$$

where $c$ is a positive exogenous parameter that balances the exploration and exploitation during node selection, and ${\overline{Q}}^{k-1}(s_t^k,a_t)$ and $N^{k-1}(s_t^k,a_t)$ are the sample mean and the number of visit times of the state-action pair $(s_t^k,a_t)$ of the state-action value after $k-1$ rollouts, respectively. After all $K$ rollouts are conducted, UCT determines the final action ${\widehat{a}}_{t_0}^K$ under the root state $s_{t_0}$ as the action with the highest number of visit times, i.e.,

$${\widehat{a}}_{t_0}^K=\arg \underset{a\in \mathcal{A}}{\max }{N}^K(s_{t_0},a).$$

An application of an AlphaGo-like MCTS to adaptive bitrate control in video delivery can be found in [120]. Applications of MCTS to inventory management and computational finance can be found in [121] and [122], respectively.

Liu et al. [30] propose Asymptotically Optimal Allocation Policy for Trees (AOAT) by extending AOAP to multi-stage node selection problem in MCTS, which utilizes sample means, variances, and the number of node visits to balance the exploration of nodes with large variances and the exploitation of nodes with large state-action values. The value network could provide prior knowledge for AOAT. A variant of AOAT called AOAT-Pi includes the policy network for determining the final action.

6. Conclusions

We reviewed simulation optimization methods closely related to AI, focusing on two areas: stochastic gradient estimation and R&S. We discussed some classic techniques in AI from the perspective of simulation optimization and introduced new methods for advancing modern AI techniques. The differences between traditional simulation optimization applications and new applications in modern AI were also discussed. Traditional simulation optimization research focuses on statistical properties of stochastic estimators (e.g., for the gradient) and asymptotic analyses of optimization algorithms, whereas the computational complexity, parallelization, and scalability of the estimator and finite-sample behavior of the optimization algorithm are deemed as more critical for AI applications. By bridging the gap, advances in simulation optimization can be used to improve training speed, robustness, and resilience of decision making in AI (see [76] and [40]).

Beyond AlphaRank in [118] for solving classical R&S problems, future research direction includes extending the AlphaRank framework to subset selection, feasibility identification, quantile-based simulation sample allocation, and context-dependent simulation sample allocation. The training of an AI agent is largely a mean-based optimization, whereas empirical behavioral studies have demonstrated that human decision making under uncertainty is inconsistent with expected utility models, which are not able to adequately account for risk. Quantile-based DRL proposed by [40] can better capture the tail risk. Recent research incorporating risk includes Glynn et al. [61], who consider a distortion risk measure that is a Lebesgue-Stieltjes integral of quantiles. Integrating risk measures into the training of AI agents would result in better aligned decisions; see also [123].

Declaration of competing interest

The authors declare that they have no conflicts of interest in this work.

Biographies

Yijie Peng is currently an associate professor of the Department of Management Science and Information Systems in Guanghua School of Management at Peking University (PKU). He received his Ph.D. from the Department of Management Science at Fudan University and his BS degree from the School of Mathematics at Wuhan University. Many of his publications appear in high-quality journals including Operations Research, INFORMS Journal on Computing, and IEEE Transactions on Automatic Control. He is awarded the 2019 Outstanding Simulation Publication Award of INFORMS Simulation Society. He serves as an associate editor for Asia-Pacific Journal of Operational Research and IEEE Control Systems Society Conference Editorial Board. His research interests include stochastic modeling and analysis, simulation optimization, machine learning, and healthcare.

Chun-Hung Chen received his Ph.D. degree from Harvard University in 1994. He is currently a professor at George Mason University. Dr. Chen was an assistant professor at the University of Pennsylvania before joining GMU. Sponsored by NSF, NIH, DOE, NASA, FAA, and AFOSR in the U.S., he has worked on the development of very efficient methodology for simulation-based decision making and its applications. Dr. Chen has served on several editorial boards, such as IEEE Transactions on Automatic Control, IEEE Transactions on Automation Science and Engineering, IIE Transactions, Asia-Pacific Journal of Operational Research, Journal of Simulation Modeling Practice and Theory, International Journal of Simulation and Process Modeling, and Journal of Traffic and Transportation Engineering. Dr. Chen is an author of the popular book Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. He is an IEEE Fellow.

Michael C. Fu holds the Smith Chair of Management Science at the Robert H. Smith School of Business, with a joint appointment in the Institute for Systems Research, at the University of Maryland. He is the co-author of the books, Conditional Monte Carlo: Gradient Estimation and Optimization Applications, which received the INFORMS Simulation Society’s 1998 Outstanding Publication Award, and Simulation-Based Algorithms for Markov Decision Processes, and editor/co-editor of four volumes: Perspectives in Operations Research, Advances in Mathematical Finance, Encyclopedia of Operations Research and Management Science (3rd edition), and Handbook of Simulation Optimization. He served as the Operations Research NSF Program Director (2010–2012 & 2015) and as General Co-Chair for the 2020 INFORMS National Meeting. Recent awards include the INFORMS Simulation Society’s Distinguished Service Award (2019), the INFORMS Saul Gass Expository Writing Award (2021), and the INFORMS Kimball Medal (2022). He is a Fellow of IEEE and INFORMS.