Mean Field Games (MFG) have been introduced to tackle games with a large number of competing players. Considering the limit when the number of players is infinite, Nash equilibria are studied by considering the interaction of a typical player with the population's distribution. The situation in which the players cooperate corresponds to Mean Field Control (MFC) problems, which can also be viewed as optimal control problems driven by a McKean-Vlasov dynamics. These two types of problems have found a wide range of potential applications, for which numerical methods play a key role since most models do not have analytical solutions. In these notes, we review several aspects of numerical methods for MFG and MFC. We start by presenting some heuristics in a basic linear-quadratic setting. We then discuss numerical schemes for forward-backward systems of partial differential equations (PDEs), optimization techniques for variational problems driven by a Kolmogorov-Fokker-Planck PDE, an approach based on a monotone operator viewpoint, and stochastic methods relying on machine learning tools.
相關內容
Stochastic games combine controllable and adversarial non-determinism with stochastic behavior and are a common tool in control, verification and synthesis of reactive systems facing uncertainty. In this paper, we study turn-based stochastic two-player games on graphs where the winning condition is to satisfy at least one reachability or safety objective from a given set of alternatives with at least some desired probability. These objectives are also known as disjunctive queries (DQs). We present a fine-grained overview of strategy and computational complexity and provide new lower and upper bounds for several variants of the problem. These results extend the previous literature on DQs significantly. We also propose a novel value iteration-style algorithm for approximating the set of Pareto optimal thresholds for a given DQ.
Divergence-free discontinuous Galerkin (DG) finite element methods offer a suitable discretization for the pointwise divergence-free numerical solution of Borrvall and Petersson's model for the topology optimization of fluids in Stokes flow [Topology optimization of fluids in Stokes flow, International Journal for Numerical Methods in Fluids 41 (1) (2003) 77--107]. The convergence results currently found in literature only consider $H^1$-conforming discretizations for the velocity. In this work, we extend the numerical analysis of Papadopoulos and S\"uli to divergence-free DG methods with an interior penalty [I. P. A. Papadopoulos and E. S\"uli, Numerical analysis of a topology optimization problem for Stokes flow, arXiv preprint arXiv:2102.10408, (2021)]. We show that, given an isolated minimizer to the analytical problem, there exists a sequence of DG finite element solutions, satisfying necessary first-order optimality conditions, that strongly converges to the minimizer.
This paper presents and analyzes an immersed finite element (IFE) method for solving Stokes interface problems with a piecewise constant viscosity coefficient that has a jump across the interface. In the method, the triangulation does not need to fit the interface and the IFE spaces are constructed from the traditional $CR$-$P_0$ element with modifications near the interface according to the interface jump conditions. We prove that the IFE basis functions are unisolvent on arbitrary interface elements and the IFE spaces have the optimal approximation capabilities, although the proof is challenging due to the coupling of the velocity and the pressure. The stability and the optimal error estimates of the proposed IFE method are also derived rigorously. The constants in the error estimates are shown to be independent of the interface location relative to the triangulation. Numerical examples are provided to verify the theoretical results.
In this paper we introduce a procedure for identifying optimal methods in parametric families of numerical schemes for initial value problems in partial differential equations. The procedure maximizes accuracy by adaptively computing optimal parameters that minimize a defect-based estimate of the local error at each time-step. Viable refinements are proposed to reduce the computational overheads involved in the solution of the optimization problem, and to maintain conservation properties of the original methods. We apply the new strategy to recently introduced families of conservative schemes for the Korteweg-de Vries equation and for a nonlinear heat equation. Numerical tests demonstrate the improved efficiency of the new technique in comparison with existing methods.
In this paper, an important discovery has been found for nonconforming immersed finite element (IFE) methods using integral-value degrees of freedom for solving elliptic interface problems. We show that those IFE methods can only achieve suboptimal convergence rates (i.e., $O(h^{1/2})$ in the $H^1$ norm and $O(h)$ in the $L^2$ norm) if the tangential derivative of the exact solution and the jump of the coefficient are not zero on the interface. A nontrivial counter example is also provided to support our theoretical analysis. To recover the optimal convergence rates, we develop a new nonconforming IFE method with additional terms locally on interface edges. The unisolvence of IFE basis functions is proved on arbitrary triangles. Furthermore, we derive the optimal approximation capabilities of both the Crouzeix-Raviart and the rotated-$Q_1$ IFE spaces for interface problems with variable coefficients via a unified approach different from multipoint Taylor expansions. Finally, optimal error estimates in both $H^1$- and $L^2$- norms are proved and confirmed with numerical experiments.
This survey is meant to provide an introduction to linear models and the theories behind them. Our goal is to give a rigorous introduction to the readers with prior exposure to ordinary least squares. In machine learning, the output is usually a nonlinear function of the input. Deep learning even aims to find a nonlinear dependence with many layers which require a large amount of computation. However, most of these algorithms build upon simple linear models. We then describe linear models from different views and find the properties and theories behind the models. The linear model is the main technique in regression problems and the primary tool for it is the least squares approximation which minimizes a sum of squared errors. This is a natural choice when we're interested in finding the regression function which minimizes the corresponding expected squared error. This survey is primarily a summary of purpose, significance of important theories behind linear models, e.g., distribution theory, minimum variance estimator. We first describe ordinary least squares from three different points of view upon which we disturb the model with random noise and Gaussian noise. By Gaussian noise, the model gives rise to the likelihood so that we introduce a maximum likelihood estimator. It also develops some distribution theories via this Gaussian disturbance. The distribution theory of least squares will help us answer various questions and introduce related applications. We then prove least squares is the best unbiased linear model in the sense of mean squared error and most importantly, it actually approaches the theoretical limit. We end up with linear models with the Bayesian approach and beyond.
Semi-lagrangian schemes for discretization of the dynamic programming principle are based on a time discretization projected on a state-space grid. The use of a structured grid makes this approach not feasible for high-dimensional problems due to the curse of dimensionality. Here, we present a new approach for infinite horizon optimal control problems where the value function is computed using Radial Basis Functions (RBF) by the Shepard's moving least squares approximation method on scattered grids. We propose a new method to generate a scattered mesh driven by the dynamics and the selection of the shape parameter in the RBF using an optimization routine. This mesh will help to localize the problem and approximate the dynamic programming principle in high dimension. Error estimates for the value function are also provided. Numerical tests for high dimensional problems will show the effectiveness of the proposed method.
Variable steps implicit-explicit multistep methods for PDEs have been presented in [17], where the zero-stability is studied for ODEs; however, the stability analysis still remains an open question for PDEs. Based on the idea of linear multistep methods, we present a simple weighted and shifted BDF2 methods with variable steps for the parabolic problems, which serve as a bridge between BDF2 and Crank-Nicolson scheme. The contributions of this paper are as follows: we first prove that the optimal adjacent time-step ratios for the weighted and shifted BDF2, which greatly improve the maximum time-step ratios for BDF2 in [11,15]. Moreover, the unconditional stability and optimal convergence are rigorous proved, which make up for the vacancy of the theory for PDEs in [17]. Finally, numerical experiments are given to illustrate theoretical results.
This manuscript surveys reinforcement learning from the perspective of optimization and control with a focus on continuous control applications. It surveys the general formulation, terminology, and typical experimental implementations of reinforcement learning and reviews competing solution paradigms. In order to compare the relative merits of various techniques, this survey presents a case study of the Linear Quadratic Regulator (LQR) with unknown dynamics, perhaps the simplest and best studied problem in optimal control. The manuscript describes how merging techniques from learning theory and control can provide non-asymptotic characterizations of LQR performance and shows that these characterizations tend to match experimental behavior. In turn, when revisiting more complex applications, many of the observed phenomena in LQR persist. In particular, theory and experiment demonstrate the role and importance of models and the cost of generality in reinforcement learning algorithms. This survey concludes with a discussion of some of the challenges in designing learning systems that safely and reliably interact with complex and uncertain environments and how tools from reinforcement learning and controls might be combined to approach these challenges.
Existing multi-agent reinforcement learning methods are limited typically to a small number of agents. When the agent number increases largely, the learning becomes intractable due to the curse of the dimensionality and the exponential growth of agent interactions. In this paper, we present Mean Field Reinforcement Learning where the interactions within the population of agents are approximated by those between a single agent and the average effect from the overall population or neighboring agents; the interplay between the two entities is mutually reinforced: the learning of the individual agent's optimal policy depends on the dynamics of the population, while the dynamics of the population change according to the collective patterns of the individual policies. We develop practical mean field Q-learning and mean field Actor-Critic algorithms and analyze the convergence of the solution to Nash equilibrium. Experiments on Gaussian squeeze, Ising model, and battle games justify the learning effectiveness of our mean field approaches. In addition, we report the first result to solve the Ising model via model-free reinforcement learning methods.
注冊地址： 北京市海淀區羊坊店路18號2幢3層301-191