We prove closed-form equations for the exact high-dimensional asymptotics of a family of first order gradient-based methods, learning an estimator (e.g. M-estimator, shallow neural network, ...) from observations on Gaussian data with empirical risk minimization. This includes widely used algorithms such as stochastic gradient descent (SGD) or Nesterov acceleration. The obtained equations match those resulting from the discretization of dynamical mean-field theory (DMFT) equations from statistical physics when applied to gradient flow. Our proof method allows us to give an explicit description of how memory kernels build up in the effective dynamics, and to include non-separable update functions, allowing datasets with non-identity covariance matrices. Finally, we provide numerical implementations of the equations for SGD with generic extensive batch-size and with constant learning rates.
相關內容
In this paper, we formulate and analyse a geometric low-regularity integrator for solving the nonlinear Klein-Gordon equation in the $d$-dimensional space with $d=1,2,3$. The integrator is constructed based on the two-step trigonometric method and thus it has a simple form. Error estimates are rigorously presented to show that the integrator can achieve second-order time accuracy in the energy space under the regularity requirement in $H^{1+\frac{d}{4}}\times H^{\frac{d}{4}}$. Moreover, the time symmetry of the scheme ensures its good long-time energy conservation which is rigorously proved by the technique of modulated Fourier expansions. A numerical test is presented and the numerical results demonstrate the superiorities of the new integrator over some existing methods.
In this paper, we consider a numerical method for the multi-term Caputo-Fabrizio time-fractional diffusion equations (with orders $\alpha_i\in(0,1)$, $i=1,2,\cdots,n$). The proposed method employs a fast finite difference scheme to approximate multi-term fractional derivatives in time, requiring only $O(1)$ storage and $O(N_T)$ computational complexity, where $N_T$ denotes the total number of time steps. Then we use a Legendre spectral collocation method for spatial discretization. The stability and convergence of the scheme have been thoroughly discussed and rigorously established. We demonstrate that the proposed scheme is unconditionally stable and convergent with an order of $O(\left(\Delta t\right)^{2}+N^{-m})$, where $\Delta t$, $N$, and $m$ represent the timestep size, polynomial degree, and regularity in the spatial variable of the exact solution, respectively. Numerical results are presented to validate the theoretical predictions.
The most popular method for computing the matrix logarithm is a combination of the inverse scaling and squaring method in conjunction with a Pad\'e approximation, sometimes accompanied by the Schur decomposition. The main computational effort lies in matrix-matrix multiplications and left matrix division. In this work we illustrate that the number of such operations can be substantially reduced, by using a graph based representation of an efficient polynomial evaluation scheme. A technique to analyze the rounding error is proposed, and backward error analysis is adapted. We provide substantial simulations illustrating competitiveness both in terms of computation time and rounding errors.
We study discretizations of fractional fully nonlinear equations by powers of discrete Laplacians. Our problems are parabolic and of order $\sigma\in(0,2)$ since they involve fractional Laplace operators $(-\Delta)^{\sigma/2}$. They arise e.g.~in control and game theory as dynamic programming equations, and solutions are non-smooth in general and should be interpreted as viscosity solutions. Our approximations are realized as finite-difference quadrature approximations and are 2nd order accurate for all values of $\sigma$. The accuracy of previous approximations depend on $\sigma$ and are worse when $\sigma$ is close to $2$. We show that the schemes are monotone, consistent, $L^\infty$-stable, and convergent using a priori estimates, viscosity solutions theory, and the method of half-relaxed limits. We present several numerical examples.
This paper studies the convergence of a spatial semidiscretization of a three-dimensional stochastic Allen-Cahn equation with multiplicative noise. For non-smooth initial values, the regularity of the mild solution is investigated, and an error estimate is derived with the spatial $ L^2 $-norm. For smooth initial values, two error estimates with the general spatial $ L^q $-norms are established.
We present a novel discontinuous Galerkin finite element method for numerical simulations of the rotating thermal shallow water equations in complex geometries using curvilinear meshes, with arbitrary accuracy. We derive an entropy functional which is convex, and which must be preserved in order to preserve model stability at the discrete level. The numerical method is provably entropy stable and conserves mass, buoyancy, vorticity, and energy. This is achieved by using novel entropy stable numerical fluxes, summation-by-parts principle, and splitting the pressure and convection operators so that we can circumvent the use of chain rule at the discrete level. Numerical simulations on a cubed sphere mesh are presented to verify the theoretical results. The numerical experiments demonstrate the robustness of the method for a regime of well developed turbulence, where it can be run stably without any dissipation. The entropy stable fluxes are sufficient to control the grid scale noise generated by geostrophic turbulence, eliminating the need for artificial stabilisation.
This work focuses on the numerical approximations of random periodic solutions of stochastic differential equations (SDEs). Under non-globally Lipschitz conditions, we prove the existence and uniqueness of random periodic solutions for the considered equations and its numerical approximations generated by the stochastic theta (ST) methods with theta within (1/2,1]. It is shown that the random periodic solution of each ST method converges strongly in the mean square sense to that of SDEs for all step size. More precisely, the mean square convergence order is 1/2 for SDEs with multiplicative noise and 1 for SDEs with additive noise. Numerical results are finally reported to confirm these theoretical findings.
We develop a novel and efficient discontinuous Galerkin spectral element method (DG-SEM) for the spherical rotating shallow water equations in vector invariant form. We prove that the DG-SEM is energy stable, and discretely conserves mass, vorticity, and linear geostrophic balance on general curvlinear meshes. These theoretical results are possible due to our novel entropy stable numerical DG fluxes for the shallow water equations in vector invariant form. We experimentally verify these results on a cubed sphere mesh. Additionally, we show that our method is robust, that is can be run stably without any dissipation. The entropy stable fluxes are sufficient to control the grid scale noise generated by geostrophic turbulence without the need for artificial stabilisation.
Dirac delta distributionally sourced differential equations emerge in many dynamical physical systems from neuroscience to black hole perturbation theory. Most of these lack exact analytical solutions and are thus best tackled numerically. This work describes a generic numerical algorithm which constructs discontinuous spatial and temporal discretisations by operating on discontinuous Lagrange and Hermite interpolation formulae recovering higher order accuracy. It is shown by solving the distributionally sourced wave equation, which has analytical solutions, that numerical weak-form solutions can be recovered to high order accuracy by solving a first-order reduced system of ordinary differential equations. The method-of-lines framework is applied to the DiscoTEX algorithm i.e through discontinuous collocation with implicit-turned-explicit (IMTEX) integration methods which are symmetric and conserve symplectic structure. Furthermore, the main application of the algorithm is proved, for the first-time, by calculating the amplitude at any desired location within the numerical grid, including at the position (and at its right and left limit) where the wave- (or wave-like) equation is discontinuous via interpolation using DiscoTEX. This is shown, firstly by solving the wave- (or wave-like) equation and comparing the numerical weak-form solution to the exact solution. Finally, one shows how to reconstruct the scalar and gravitational metric perturbations from weak-form numerical solutions of a non-rotating black hole, which do not have known exact analytical solutions, and compare against state-of-the-art frequency domain results. One concludes by motivating how DiscoTEX, and related algorithms, open a promising new alternative Extreme-Mass-Ratio-Inspiral (EMRI)s waveform generation route via a self-consistent evolution for the gravitational self-force programme in the time-domain.
We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.
注冊地址： 北京市海淀區羊坊店路18號2幢3層301-191