Mixed-precision algorithms combine low- and high-precision computations in order to benefit from the performance gains of reduced-precision without sacrificing accuracy. In this work, we design mixed-precision Runge-Kutta-Chebyshev (RKC) methods, where high precision is used for accuracy, and low precision for stability. Generally speaking, RKC methods are low-order explicit schemes with a stability domain growing quadratically with the number of function evaluations. For this reason, most of the computational effort is spent on stability rather than accuracy purposes. In this paper, we show that a na\"ive mixed-precision implementation of any Runge-Kutta scheme can harm the convergence order of the method and limit its accuracy, and we introduce a new class of mixed-precision RKC schemes that are instead unaffected by this limiting behaviour. We present three mixed-precision schemes: a first- and a second-order RKC method, and a first-order multirate RKC scheme for multiscale problems. These schemes perform only the few function evaluations needed for accuracy (1 or 2 for first- and second-order methods respectively) in high precision, while the rest are performed in low precision. We prove that while these methods are essentially as cheap as their fully low-precision equivalent, they retain the convergence order of their high-precision counterpart. Indeed, numerical experiments confirm that these schemes are as accurate as the corresponding high-precision method.
相關內容
We develop an a posteriori error analysis for a novel quantity of interest (QoI) evolutionary partial differential equations (PDEs). Specifically, the QoI is the first time at which a functional of the solution to the PDE achieves a threshold value signifying a particular event, and differs from classical QoIs which are modeled as bounded linear functionals. We use Taylor's theorem and adjoint based analysis to derive computable and accurate error estimates for linear parabolic and hyperbolic PDEs. Specifically, the heat equation and linearized shallow water equations (SWE) are used for the parabolic and hyperbolic cases, respectively. Numerical examples illustrate the accuracy of the error estimates.
This paper introduces a new class of numerical methods for the time integration of evolution equations set as Cauchy problems of ODEs or PDEs. The systematic design of these methods mixes the Runge-Kutta collocation formalism with collocation techniques, in such a way that the methods are linearly implicit and have high order. The fact that these methods are implicit allows to avoid CFL conditions when the large systems to integrate come from the space discretization of evolution PDEs. Moreover, these methods are expected to be efficient since they only require to solve one linear system of equations at each time step, and efficient techniques from the literature can be used to do so. After the introduction of the methods, we set suitable definitions of consistency and stability for these methods. This allows for a proof that arbitrarily high order linearly implicit methods exist and converge when applied to ODEs. Eventually, we perform numerical experiments on ODEs and PDEs that illustrate our theoretical results for ODEs, and compare our methods with standard methods for several evolution PDEs.
A quasi-second order scheme is developed to obtain approximate solutions of the shallow water equationswith bathymetry. The scheme is based on a staggered finite volume scheme for the space discretization:the scalar unknowns are located in the discretisation cells while the vector unknowns are located on theedges (in 2D) or faces (in 3D) of the mesh. A MUSCL-like interpolation for the discrete convectionoperators in the water height and momentum equations is performed in order to improve the precisionof the scheme. The time discretization is performed either by a first order segregated forward Eulerscheme in time or by the second order Heun scheme. Both schemes are shown to preserve the waterheight positivity under a CFL condition and an important state equilibrium known as the lake at rest.Using some recent Lax-Wendroff type results for staggered grids, these schemes are shown to be Lax-consistent with the weak formulation of the continuous equations; besides, the forward Euler schemeis shown to be consistent with a weak entropy inequality. Numerical results confirm the efficiency andaccuracy of the schemes.
The sharpest known high probability generalization bounds for uniformly stable algorithms (Feldman, Vondr\'{a}k, 2018, 2019), (Bousquet, Klochkov, Zhivotovskiy, 2020) contain a generally inevitable sampling error term of order $\Theta(1/\sqrt{n})$. When applied to excess risk bounds, this leads to suboptimal results in several standard stochastic convex optimization problems. We show that if the so-called Bernstein condition is satisfied, the term $\Theta(1/\sqrt{n})$ can be avoided, and high probability excess risk bounds of order up to $O(1/n)$ are possible via uniform stability. Using this result, we show a high probability excess risk bound with the rate $O(\log n/n)$ for strongly convex and Lipschitz losses valid for \emph{any} empirical risk minimization method. This resolves a question of Shalev-Shwartz, Shamir, Srebro, and Sridharan (2009). We discuss how $O(\log n/n)$ high probability excess risk bounds are possible for projected gradient descent in the case of strongly convex and Lipschitz losses without the usual smoothness assumption.
In this paper we analyse full discretizations of an initial boundary value problem (IBVP) related to reaction-diffusion equations. To avoid possible order reduction, the IBVP is first transformed into an IBVP with homogeneous boundary conditions (IBVPHBC) via a lifting of inhomogeneous Dirichlet, Neumann or mixed Dirichlet-Neumann boundary conditions. The IBVPHBC is discretized in time via the deferred correction method for the implicit midpoint rule and leads to a time-stepping scheme of order $2p+2$ of accuracy at the stage $p=0,1,2,\cdots $ of the correction. Each semi-discretized scheme results in a nonlinear elliptic equation for which the existence of a solution is proven using the Schaefer fixed point theorem. The elliptic equation corresponding to the stage $p$ of the correction is discretized by the Galerkin finite element method and gives a full discretization of the IBVPHBC. This fully discretized scheme is unconditionally stable with order $2p+2$ of accuracy in time. The order of accuracy in space is equal to the degree of the finite element used when the family of meshes considered is shape-regular while an increment of one order is proven for quasi-uniform family of meshes. Numerical tests with a bistable reaction-diffusion equation having a strong stiffness ratio and a linear reaction-diffusion equation addressing order reduction are performed and demonstrate the unconditional convergence of the method. The orders 2,4,6,8 and 10 of accuracy in time are achieved.
A computationally efficient high-order solver is developed to compute the wall distances, which are typically used for turbulence modelling, peripheral flow simulations, Computer Aided Design (CAD) etc. The wall distances are computed by solving the differential equations namely: Eikonal, Hamilton-Jacobi (H-J) and Poisson. The computational benefit of using high-order schemes (explicit/compact schemes) for wall-distance solvers, both in terms of accuracy and computational time, has been demonstrated. A new H-J formulation based on the localized artificial diffusivity (LAD) approach has been proposed, which has produced results with an accuracy comparable to that of the Eikonal formulation. When compared to the baseline H-J solver using upwind schemes, the solution accuracy has improved by an order of magnitude and the calculations are $\approx$ 5 times faster using the modified H-J formulation. A modified curvature correction has also been implemented into the H-J solver to account for the near-wall errors due to concave/convex wall curvatures. The performance of the solver using different schemes has been tested both on the steady canonical test cases and the unsteady test cases like `piston-cylinder arrangement', `bouncing cube' and `burning of a star grain propellant' where the wall-distance evolves with time.
We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.
Interpretation of Deep Neural Networks (DNNs) training as an optimal control problem with nonlinear dynamical systems has received considerable attention recently, yet the algorithmic development remains relatively limited. In this work, we make an attempt along this line by reformulating the training procedure from the trajectory optimization perspective. We first show that most widely-used algorithms for training DNNs can be linked to the Differential Dynamic Programming (DDP), a celebrated second-order trajectory optimization algorithm rooted in the Approximate Dynamic Programming. In this vein, we propose a new variant of DDP that can accept batch optimization for training feedforward networks, while integrating naturally with the recent progress in curvature approximation. The resulting algorithm features layer-wise feedback policies which improve convergence rate and reduce sensitivity to hyper-parameter over existing methods. We show that the algorithm is competitive against state-ofthe-art first and second order methods. Our work opens up new avenues for principled algorithmic design built upon the optimal control theory.
When and why can a neural network be successfully trained? This article provides an overview of optimization algorithms and theory for training neural networks. First, we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum, and then discuss practical solutions including careful initialization and normalization methods. Second, we review generic optimization methods used in training neural networks, such as SGD, adaptive gradient methods and distributed methods, and theoretical results for these algorithms. Third, we review existing research on the global issues of neural network training, including results on bad local minima, mode connectivity, lottery ticket hypothesis and infinite-width analysis.
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.
注冊地址： 北京市海淀區羊坊店路18號2幢3層301-191