We propose a novel and efficient lifting approach for the optimal control of rigid-body systems with contacts to improve the convergence properties of Newton-type methods. To relax the high nonlinearity, we consider the state, acceleration, contact forces, and control input torques, as optimization variables and the inverse dynamics and acceleration constraints on the contact frames as equality constraints. We eliminate the update of the acceleration, contact forces, and their dual variables from the linear equation to be solved in each Newton-type iteration in an efficient manner. As a result, the computational cost per Newton-type iteration is almost identical to that of the conventional non-lifted Newton-type iteration that embeds contact dynamics in the state equation. We conducted numerical experiments on the whole-body optimal control of various quadrupedal gaits subject to the friction cone constraints considered in interior-point methods and demonstrated that the proposed method can significantly increase the convergence speed to more than twice that of the conventional non-lifted approach.

Cross-diffusion systems arise as hydrodynamic limits of lattice multi-species interacting particle models. The objective of this work is to provide a numerical scheme for the simulation of the cross-diffusion system identified in [J. Quastel, Comm. Pure Appl. Math., 45 (1992), pp. 623--679]. To simulate this system, it is necessary to provide an approximation of the so-called self-diffusion coefficient matrix of the tagged particle process. Classical algorithms for the computation of this matrix are based on the estimation of the long-time limit of the average mean square displacement of the particle. In this work, as an alternative, we propose a novel approach for computing the self-diffusion coefficient using deterministic low-rank approximation techniques, as the minimum of a high-dimensional optimization problem. The computed self-diffusion coefficient is then used for the simulation of the cross-diffusion system using an implicit finite volume scheme.

Imposition methods of interface conditions for the second-order wave equation with non-conforming grids is considered. The spatial discretization is based on high order finite differences with summation-by-parts properties. Previously presented solution methods for this problem, based on the simultaneous approximation term (SAT) method, have shown to introduce significant stiffness. This can lead to highly inefficient schemes. Here, two new methods of imposing the interface conditions to avoid the stiffness problems are presented: 1) a projection method and 2) a hybrid between the projection method and the SAT method. Numerical experiments are performed using traditional and order-preserving interpolation operators. Both of the novel methods retain the accuracy and convergence behavior of the previously developed SAT method but are significantly less stiff.

The existing discrete variational derivative method is only second-order accurate and fully implicit. In this paper, we propose a framework to construct an arbitrary high-order implicit (original) energy stable scheme and a second-order semi-implicit (modified) energy stable scheme. Combined with the Runge--Kutta process, we can build an arbitrary high-order and unconditionally (original) energy stable scheme based on the discrete variational derivative method. The new energy stable scheme is implicit and leads to a large sparse nonlinear algebraic system at each time step, which can be efficiently solved by using an inexact Newton type algorithm. To avoid solving nonlinear algebraic systems, we then present a relaxed discrete variational derivative method, which can construct second-order, linear, and unconditionally (modified) energy stable schemes. Several numerical simulations are performed to investigate the efficiency, stability, and accuracy of the newly proposed schemes.

The paper proposes a decoupled numerical scheme of the time-dependent Ginzburg-Landau equations under temporal gauge. For the order parameter and the magnetic potential, the discrete scheme adopts the second type Ned${\rm \acute{e}}$lec element and the linear element for spatial discretization, respectively, and a fully linearized backward Euler method and the first order exponential time differencing method for time discretization, respectively. The maximum bound principle of the order parameter and the energy dissipation law in the discrete sense are proved for this finite element-based scheme. This allows the application of the adaptive time stepping method which can significantly speed up long-time simulations compared to existing numerical schemes, especially for superconductors with complicated shapes. The error estimate is rigorously established in the fully discrete sense. Numerical examples verify the theoretical results of the proposed scheme and demonstrate the vortex motions of superconductors in an external magnetic field.

A simple generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed. The velocity field of this flow is inferred from the probability current of a time-dependent density that interpolates between the base and the target in finite time. Unlike conventional normalizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to a simple quadratic loss for the velocity itself which is expressed in terms of expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the likelihood at any time along the interpolant. In addition, the flow can be optimized to minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. The approach is also contextualized in its relation to diffusions. In particular, in situations where the base is a Gaussian density, we show that the velocity of our normalizing flow can also be used to construct a diffusion model to sample the target as well as estimating its score. This allows one to map methods based on stochastic differential equations to those using ordinary differential equations, simplifying the mechanics of the model, but capturing equivalent dynamics. Benchmarking on density estimation tasks illustrates that the learned flow can match and surpass maximum likelihood continuous flows at a fraction of the conventional ODE training costs.

Simulation of stochastic partial differential equations (SPDE) on a general domain requires a discretization of the noise. In this paper, the noise is discretized by a piecewise linear interpolation. The error caused by this is analyzed in the context of a fully discrete finite element approximation of a semilinear stochastic reaction-advection-diffusion equation on a convex polygon. The noise is Gaussian, white in time and correlated in space. It is modeled as a standard cylindrical Wiener process on the reproducing kernel Hilbert space associated to the covariance kernel. The noise is assumed to extend to a larger polygon than the SPDE domain to allow for sampling by the circulant embedding method. The interpolation error is analyzed under mild assumptions on the kernel. The main tools used are Hilbert--Schmidt bounds of multiplication operators onto negative order Sobolev spaces and an error bound for the finite element interpolant in fractional Sobolev norms. Examples with covariance kernels encountered in applications are illustrated in numerical simulations using the FEniCS finite element software. Conclusions from the analysis include that interpolation of noise with Mat\'ern kernels does not cause an additional error, that there exist kernels where the interpolation error dominates and that generation of noise on a coarser mesh than that of the SPDE discretization does not always result in a loss of accuracy.

Social networks affect the diffusion of information, and thus have the potential to reduce or amplify inequality in access to opportunity. We show empirically that social networks often exhibit a much larger potential for unequal diffusion across groups along paths of length 2 and 3 than expected by our random graph models. We argue that homophily alone cannot not fully explain the extent of unequal diffusion and attribute this mismatch to unequal distribution of cross-group links among the nodes. Based on this insight, we develop a variant of the stochastic block model that incorporates the heterogeneity in cross-group linking. The model provides an unbiased and consistent estimate of assortativity or homophily on paths of length 2 and provide a more accurate estimate along paths of length 3 than existing models. We characterize the null distribution of its log-likelihood ratio test and argue that the goodness of fit test is valid only when the network is dense. Based on our empirical observations and modeling results, we conclude that the impact of any departure from equal distribution of links to source nodes in the diffusion process is not limited to its first order effects as some nodes will have fewer direct links to the sources. More importantly, this unequal distribution will also lead to second order effects as the whole group will have fewer diffusion paths to the sources.

Diffusion models are a class of deep generative models that have shown impressive results on various tasks with dense theoretical founding. Although diffusion models have achieved impressive quality and diversity of sample synthesis than other state-of-the-art models, they still suffer from costly sampling procedure and sub-optimal likelihood estimation. Recent studies have shown great enthusiasm on improving the performance of diffusion model. In this article, we present a first comprehensive review of existing variants of the diffusion models. Specifically, we provide a first taxonomy of diffusion models and categorize them variants to three types, namely sampling-acceleration enhancement, likelihood-maximization enhancement and data-generalization enhancement. We also introduce in detail other five generative models (i.e., variational autoencoders, generative adversarial networks, normalizing flow, autoregressive models, and energy-based models), and clarify the connections between diffusion models and these generative models. Then we make a thorough investigation into the applications of diffusion models, including computer vision, natural language processing, waveform signal processing, multi-modal modeling, molecular graph generation, time series modeling, and adversarial purification. Furthermore, we propose new perspectives pertaining to the development of this generative model.

As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the numerical values in those computations arose. Strongly related to the problem of numerical representation is the problem of quantization: in what manner should a set of continuous real-valued numbers be distributed over a fixed discrete set of numbers to minimize the number of bits required and also to maximize the accuracy of the attendant computations? This perennial problem of quantization is particularly relevant whenever memory and/or computational resources are severely restricted, and it has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications. Thus, it is not surprising that quantization has emerged recently as an important and very active sub-area of research in the efficient implementation of computations associated with Neural Networks. In this article, we survey approaches to the problem of quantizing the numerical values in deep Neural Network computations, covering the advantages/disadvantages of current methods. With this survey and its organization, we hope to have presented a useful snapshot of the current research in quantization for Neural Networks and to have given an intelligent organization to ease the evaluation of future research in this area.

We propose a novel and efficient lifting approach for the optimal control of rigid-body systems with contacts to improve the convergence properties of Newton-type methods. To relax the high nonlinearity, we consider the state, acceleration, contact forces, and control input torques, as optimization variables and the inverse dynamics and acceleration constraints on the contact frames as equality constraints. We eliminate the update of the acceleration, contact forces, and their dual variables from the linear equation to be solved in each Newton-type iteration in an efficient manner. As a result, the computational cost per Newton-type iteration is almost identical to that of the conventional non-lifted Newton-type iteration that embeds contact dynamics in the state equation. We conducted numerical experiments on the whole-body optimal control of various quadrupedal gaits subject to the friction cone constraints considered in interior-point methods and demonstrated that the proposed method can significantly increase the convergence speed to more than twice that of the conventional non-lifted approach.

Cross-diffusion systems arise as hydrodynamic limits of lattice multi-species interacting particle models. The objective of this work is to provide a numerical scheme for the simulation of the cross-diffusion system identified in [J. Quastel, Comm. Pure Appl. Math., 45 (1992), pp. 623--679]. To simulate this system, it is necessary to provide an approximation of the so-called self-diffusion coefficient matrix of the tagged particle process. Classical algorithms for the computation of this matrix are based on the estimation of the long-time limit of the average mean square displacement of the particle. In this work, as an alternative, we propose a novel approach for computing the self-diffusion coefficient using deterministic low-rank approximation techniques, as the minimum of a high-dimensional optimization problem. The computed self-diffusion coefficient is then used for the simulation of the cross-diffusion system using an implicit finite volume scheme.

Imposition methods of interface conditions for the second-order wave equation with non-conforming grids is considered. The spatial discretization is based on high order finite differences with summation-by-parts properties. Previously presented solution methods for this problem, based on the simultaneous approximation term (SAT) method, have shown to introduce significant stiffness. This can lead to highly inefficient schemes. Here, two new methods of imposing the interface conditions to avoid the stiffness problems are presented: 1) a projection method and 2) a hybrid between the projection method and the SAT method. Numerical experiments are performed using traditional and order-preserving interpolation operators. Both of the novel methods retain the accuracy and convergence behavior of the previously developed SAT method but are significantly less stiff.

The existing discrete variational derivative method is only second-order accurate and fully implicit. In this paper, we propose a framework to construct an arbitrary high-order implicit (original) energy stable scheme and a second-order semi-implicit (modified) energy stable scheme. Combined with the Runge--Kutta process, we can build an arbitrary high-order and unconditionally (original) energy stable scheme based on the discrete variational derivative method. The new energy stable scheme is implicit and leads to a large sparse nonlinear algebraic system at each time step, which can be efficiently solved by using an inexact Newton type algorithm. To avoid solving nonlinear algebraic systems, we then present a relaxed discrete variational derivative method, which can construct second-order, linear, and unconditionally (modified) energy stable schemes. Several numerical simulations are performed to investigate the efficiency, stability, and accuracy of the newly proposed schemes.

The paper proposes a decoupled numerical scheme of the time-dependent Ginzburg-Landau equations under temporal gauge. For the order parameter and the magnetic potential, the discrete scheme adopts the second type Ned${\rm \acute{e}}$lec element and the linear element for spatial discretization, respectively, and a fully linearized backward Euler method and the first order exponential time differencing method for time discretization, respectively. The maximum bound principle of the order parameter and the energy dissipation law in the discrete sense are proved for this finite element-based scheme. This allows the application of the adaptive time stepping method which can significantly speed up long-time simulations compared to existing numerical schemes, especially for superconductors with complicated shapes. The error estimate is rigorously established in the fully discrete sense. Numerical examples verify the theoretical results of the proposed scheme and demonstrate the vortex motions of superconductors in an external magnetic field.

A simple generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed. The velocity field of this flow is inferred from the probability current of a time-dependent density that interpolates between the base and the target in finite time. Unlike conventional normalizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to a simple quadratic loss for the velocity itself which is expressed in terms of expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the likelihood at any time along the interpolant. In addition, the flow can be optimized to minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. The approach is also contextualized in its relation to diffusions. In particular, in situations where the base is a Gaussian density, we show that the velocity of our normalizing flow can also be used to construct a diffusion model to sample the target as well as estimating its score. This allows one to map methods based on stochastic differential equations to those using ordinary differential equations, simplifying the mechanics of the model, but capturing equivalent dynamics. Benchmarking on density estimation tasks illustrates that the learned flow can match and surpass maximum likelihood continuous flows at a fraction of the conventional ODE training costs.

Simulation of stochastic partial differential equations (SPDE) on a general domain requires a discretization of the noise. In this paper, the noise is discretized by a piecewise linear interpolation. The error caused by this is analyzed in the context of a fully discrete finite element approximation of a semilinear stochastic reaction-advection-diffusion equation on a convex polygon. The noise is Gaussian, white in time and correlated in space. It is modeled as a standard cylindrical Wiener process on the reproducing kernel Hilbert space associated to the covariance kernel. The noise is assumed to extend to a larger polygon than the SPDE domain to allow for sampling by the circulant embedding method. The interpolation error is analyzed under mild assumptions on the kernel. The main tools used are Hilbert--Schmidt bounds of multiplication operators onto negative order Sobolev spaces and an error bound for the finite element interpolant in fractional Sobolev norms. Examples with covariance kernels encountered in applications are illustrated in numerical simulations using the FEniCS finite element software. Conclusions from the analysis include that interpolation of noise with Mat\'ern kernels does not cause an additional error, that there exist kernels where the interpolation error dominates and that generation of noise on a coarser mesh than that of the SPDE discretization does not always result in a loss of accuracy.

Social networks affect the diffusion of information, and thus have the potential to reduce or amplify inequality in access to opportunity. We show empirically that social networks often exhibit a much larger potential for unequal diffusion across groups along paths of length 2 and 3 than expected by our random graph models. We argue that homophily alone cannot not fully explain the extent of unequal diffusion and attribute this mismatch to unequal distribution of cross-group links among the nodes. Based on this insight, we develop a variant of the stochastic block model that incorporates the heterogeneity in cross-group linking. The model provides an unbiased and consistent estimate of assortativity or homophily on paths of length 2 and provide a more accurate estimate along paths of length 3 than existing models. We characterize the null distribution of its log-likelihood ratio test and argue that the goodness of fit test is valid only when the network is dense. Based on our empirical observations and modeling results, we conclude that the impact of any departure from equal distribution of links to source nodes in the diffusion process is not limited to its first order effects as some nodes will have fewer direct links to the sources. More importantly, this unequal distribution will also lead to second order effects as the whole group will have fewer diffusion paths to the sources.

Diffusion models are a class of deep generative models that have shown impressive results on various tasks with dense theoretical founding. Although diffusion models have achieved impressive quality and diversity of sample synthesis than other state-of-the-art models, they still suffer from costly sampling procedure and sub-optimal likelihood estimation. Recent studies have shown great enthusiasm on improving the performance of diffusion model. In this article, we present a first comprehensive review of existing variants of the diffusion models. Specifically, we provide a first taxonomy of diffusion models and categorize them variants to three types, namely sampling-acceleration enhancement, likelihood-maximization enhancement and data-generalization enhancement. We also introduce in detail other five generative models (i.e., variational autoencoders, generative adversarial networks, normalizing flow, autoregressive models, and energy-based models), and clarify the connections between diffusion models and these generative models. Then we make a thorough investigation into the applications of diffusion models, including computer vision, natural language processing, waveform signal processing, multi-modal modeling, molecular graph generation, time series modeling, and adversarial purification. Furthermore, we propose new perspectives pertaining to the development of this generative model.

As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the numerical values in those computations arose. Strongly related to the problem of numerical representation is the problem of quantization: in what manner should a set of continuous real-valued numbers be distributed over a fixed discrete set of numbers to minimize the number of bits required and also to maximize the accuracy of the attendant computations? This perennial problem of quantization is particularly relevant whenever memory and/or computational resources are severely restricted, and it has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications. Thus, it is not surprising that quantization has emerged recently as an important and very active sub-area of research in the efficient implementation of computations associated with Neural Networks. In this article, we survey approaches to the problem of quantizing the numerical values in deep Neural Network computations, covering the advantages/disadvantages of current methods. With this survey and its organization, we hope to have presented a useful snapshot of the current research in quantization for Neural Networks and to have given an intelligent organization to ease the evaluation of future research in this area.

We propose a novel and efficient lifting approach for the optimal control of rigid-body systems with contacts to improve the convergence properties of Newton-type methods. To relax the high nonlinearity, we consider the state, acceleration, contact forces, and control input torques, as optimization variables and the inverse dynamics and acceleration constraints on the contact frames as equality constraints. We eliminate the update of the acceleration, contact forces, and their dual variables from the linear equation to be solved in each Newton-type iteration in an efficient manner. As a result, the computational cost per Newton-type iteration is almost identical to that of the conventional non-lifted Newton-type iteration that embeds contact dynamics in the state equation. We conducted numerical experiments on the whole-body optimal control of various quadrupedal gaits subject to the friction cone constraints considered in interior-point methods and demonstrated that the proposed method can significantly increase the convergence speed to more than twice that of the conventional non-lifted approach.

Cross-diffusion systems arise as hydrodynamic limits of lattice multi-species interacting particle models. The objective of this work is to provide a numerical scheme for the simulation of the cross-diffusion system identified in [J. Quastel, Comm. Pure Appl. Math., 45 (1992), pp. 623--679]. To simulate this system, it is necessary to provide an approximation of the so-called self-diffusion coefficient matrix of the tagged particle process. Classical algorithms for the computation of this matrix are based on the estimation of the long-time limit of the average mean square displacement of the particle. In this work, as an alternative, we propose a novel approach for computing the self-diffusion coefficient using deterministic low-rank approximation techniques, as the minimum of a high-dimensional optimization problem. The computed self-diffusion coefficient is then used for the simulation of the cross-diffusion system using an implicit finite volume scheme.

Imposition methods of interface conditions for the second-order wave equation with non-conforming grids is considered. The spatial discretization is based on high order finite differences with summation-by-parts properties. Previously presented solution methods for this problem, based on the simultaneous approximation term (SAT) method, have shown to introduce significant stiffness. This can lead to highly inefficient schemes. Here, two new methods of imposing the interface conditions to avoid the stiffness problems are presented: 1) a projection method and 2) a hybrid between the projection method and the SAT method. Numerical experiments are performed using traditional and order-preserving interpolation operators. Both of the novel methods retain the accuracy and convergence behavior of the previously developed SAT method but are significantly less stiff.

The existing discrete variational derivative method is only second-order accurate and fully implicit. In this paper, we propose a framework to construct an arbitrary high-order implicit (original) energy stable scheme and a second-order semi-implicit (modified) energy stable scheme. Combined with the Runge--Kutta process, we can build an arbitrary high-order and unconditionally (original) energy stable scheme based on the discrete variational derivative method. The new energy stable scheme is implicit and leads to a large sparse nonlinear algebraic system at each time step, which can be efficiently solved by using an inexact Newton type algorithm. To avoid solving nonlinear algebraic systems, we then present a relaxed discrete variational derivative method, which can construct second-order, linear, and unconditionally (modified) energy stable schemes. Several numerical simulations are performed to investigate the efficiency, stability, and accuracy of the newly proposed schemes.

The paper proposes a decoupled numerical scheme of the time-dependent Ginzburg-Landau equations under temporal gauge. For the order parameter and the magnetic potential, the discrete scheme adopts the second type Ned${\rm \acute{e}}$lec element and the linear element for spatial discretization, respectively, and a fully linearized backward Euler method and the first order exponential time differencing method for time discretization, respectively. The maximum bound principle of the order parameter and the energy dissipation law in the discrete sense are proved for this finite element-based scheme. This allows the application of the adaptive time stepping method which can significantly speed up long-time simulations compared to existing numerical schemes, especially for superconductors with complicated shapes. The error estimate is rigorously established in the fully discrete sense. Numerical examples verify the theoretical results of the proposed scheme and demonstrate the vortex motions of superconductors in an external magnetic field.

A simple generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed. The velocity field of this flow is inferred from the probability current of a time-dependent density that interpolates between the base and the target in finite time. Unlike conventional normalizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to a simple quadratic loss for the velocity itself which is expressed in terms of expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the likelihood at any time along the interpolant. In addition, the flow can be optimized to minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. The approach is also contextualized in its relation to diffusions. In particular, in situations where the base is a Gaussian density, we show that the velocity of our normalizing flow can also be used to construct a diffusion model to sample the target as well as estimating its score. This allows one to map methods based on stochastic differential equations to those using ordinary differential equations, simplifying the mechanics of the model, but capturing equivalent dynamics. Benchmarking on density estimation tasks illustrates that the learned flow can match and surpass maximum likelihood continuous flows at a fraction of the conventional ODE training costs.

Simulation of stochastic partial differential equations (SPDE) on a general domain requires a discretization of the noise. In this paper, the noise is discretized by a piecewise linear interpolation. The error caused by this is analyzed in the context of a fully discrete finite element approximation of a semilinear stochastic reaction-advection-diffusion equation on a convex polygon. The noise is Gaussian, white in time and correlated in space. It is modeled as a standard cylindrical Wiener process on the reproducing kernel Hilbert space associated to the covariance kernel. The noise is assumed to extend to a larger polygon than the SPDE domain to allow for sampling by the circulant embedding method. The interpolation error is analyzed under mild assumptions on the kernel. The main tools used are Hilbert--Schmidt bounds of multiplication operators onto negative order Sobolev spaces and an error bound for the finite element interpolant in fractional Sobolev norms. Examples with covariance kernels encountered in applications are illustrated in numerical simulations using the FEniCS finite element software. Conclusions from the analysis include that interpolation of noise with Mat\'ern kernels does not cause an additional error, that there exist kernels where the interpolation error dominates and that generation of noise on a coarser mesh than that of the SPDE discretization does not always result in a loss of accuracy.

Social networks affect the diffusion of information, and thus have the potential to reduce or amplify inequality in access to opportunity. We show empirically that social networks often exhibit a much larger potential for unequal diffusion across groups along paths of length 2 and 3 than expected by our random graph models. We argue that homophily alone cannot not fully explain the extent of unequal diffusion and attribute this mismatch to unequal distribution of cross-group links among the nodes. Based on this insight, we develop a variant of the stochastic block model that incorporates the heterogeneity in cross-group linking. The model provides an unbiased and consistent estimate of assortativity or homophily on paths of length 2 and provide a more accurate estimate along paths of length 3 than existing models. We characterize the null distribution of its log-likelihood ratio test and argue that the goodness of fit test is valid only when the network is dense. Based on our empirical observations and modeling results, we conclude that the impact of any departure from equal distribution of links to source nodes in the diffusion process is not limited to its first order effects as some nodes will have fewer direct links to the sources. More importantly, this unequal distribution will also lead to second order effects as the whole group will have fewer diffusion paths to the sources.

Diffusion models are a class of deep generative models that have shown impressive results on various tasks with dense theoretical founding. Although diffusion models have achieved impressive quality and diversity of sample synthesis than other state-of-the-art models, they still suffer from costly sampling procedure and sub-optimal likelihood estimation. Recent studies have shown great enthusiasm on improving the performance of diffusion model. In this article, we present a first comprehensive review of existing variants of the diffusion models. Specifically, we provide a first taxonomy of diffusion models and categorize them variants to three types, namely sampling-acceleration enhancement, likelihood-maximization enhancement and data-generalization enhancement. We also introduce in detail other five generative models (i.e., variational autoencoders, generative adversarial networks, normalizing flow, autoregressive models, and energy-based models), and clarify the connections between diffusion models and these generative models. Then we make a thorough investigation into the applications of diffusion models, including computer vision, natural language processing, waveform signal processing, multi-modal modeling, molecular graph generation, time series modeling, and adversarial purification. Furthermore, we propose new perspectives pertaining to the development of this generative model.

As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the numerical values in those computations arose. Strongly related to the problem of numerical representation is the problem of quantization: in what manner should a set of continuous real-valued numbers be distributed over a fixed discrete set of numbers to minimize the number of bits required and also to maximize the accuracy of the attendant computations? This perennial problem of quantization is particularly relevant whenever memory and/or computational resources are severely restricted, and it has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications. Thus, it is not surprising that quantization has emerged recently as an important and very active sub-area of research in the efficient implementation of computations associated with Neural Networks. In this article, we survey approaches to the problem of quantizing the numerical values in deep Neural Network computations, covering the advantages/disadvantages of current methods. With this survey and its organization, we hope to have presented a useful snapshot of the current research in quantization for Neural Networks and to have given an intelligent organization to ease the evaluation of future research in this area.

We propose a novel and efficient lifting approach for the optimal control of rigid-body systems with contacts to improve the convergence properties of Newton-type methods. To relax the high nonlinearity, we consider the state, acceleration, contact forces, and control input torques, as optimization variables and the inverse dynamics and acceleration constraints on the contact frames as equality constraints. We eliminate the update of the acceleration, contact forces, and their dual variables from the linear equation to be solved in each Newton-type iteration in an efficient manner. As a result, the computational cost per Newton-type iteration is almost identical to that of the conventional non-lifted Newton-type iteration that embeds contact dynamics in the state equation. We conducted numerical experiments on the whole-body optimal control of various quadrupedal gaits subject to the friction cone constraints considered in interior-point methods and demonstrated that the proposed method can significantly increase the convergence speed to more than twice that of the conventional non-lifted approach.

Cross-diffusion systems arise as hydrodynamic limits of lattice multi-species interacting particle models. The objective of this work is to provide a numerical scheme for the simulation of the cross-diffusion system identified in [J. Quastel, Comm. Pure Appl. Math., 45 (1992), pp. 623--679]. To simulate this system, it is necessary to provide an approximation of the so-called self-diffusion coefficient matrix of the tagged particle process. Classical algorithms for the computation of this matrix are based on the estimation of the long-time limit of the average mean square displacement of the particle. In this work, as an alternative, we propose a novel approach for computing the self-diffusion coefficient using deterministic low-rank approximation techniques, as the minimum of a high-dimensional optimization problem. The computed self-diffusion coefficient is then used for the simulation of the cross-diffusion system using an implicit finite volume scheme.

Imposition methods of interface conditions for the second-order wave equation with non-conforming grids is considered. The spatial discretization is based on high order finite differences with summation-by-parts properties. Previously presented solution methods for this problem, based on the simultaneous approximation term (SAT) method, have shown to introduce significant stiffness. This can lead to highly inefficient schemes. Here, two new methods of imposing the interface conditions to avoid the stiffness problems are presented: 1) a projection method and 2) a hybrid between the projection method and the SAT method. Numerical experiments are performed using traditional and order-preserving interpolation operators. Both of the novel methods retain the accuracy and convergence behavior of the previously developed SAT method but are significantly less stiff.

The existing discrete variational derivative method is only second-order accurate and fully implicit. In this paper, we propose a framework to construct an arbitrary high-order implicit (original) energy stable scheme and a second-order semi-implicit (modified) energy stable scheme. Combined with the Runge--Kutta process, we can build an arbitrary high-order and unconditionally (original) energy stable scheme based on the discrete variational derivative method. The new energy stable scheme is implicit and leads to a large sparse nonlinear algebraic system at each time step, which can be efficiently solved by using an inexact Newton type algorithm. To avoid solving nonlinear algebraic systems, we then present a relaxed discrete variational derivative method, which can construct second-order, linear, and unconditionally (modified) energy stable schemes. Several numerical simulations are performed to investigate the efficiency, stability, and accuracy of the newly proposed schemes.

The paper proposes a decoupled numerical scheme of the time-dependent Ginzburg-Landau equations under temporal gauge. For the order parameter and the magnetic potential, the discrete scheme adopts the second type Ned${\rm \acute{e}}$lec element and the linear element for spatial discretization, respectively, and a fully linearized backward Euler method and the first order exponential time differencing method for time discretization, respectively. The maximum bound principle of the order parameter and the energy dissipation law in the discrete sense are proved for this finite element-based scheme. This allows the application of the adaptive time stepping method which can significantly speed up long-time simulations compared to existing numerical schemes, especially for superconductors with complicated shapes. The error estimate is rigorously established in the fully discrete sense. Numerical examples verify the theoretical results of the proposed scheme and demonstrate the vortex motions of superconductors in an external magnetic field.

A simple generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed. The velocity field of this flow is inferred from the probability current of a time-dependent density that interpolates between the base and the target in finite time. Unlike conventional normalizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to a simple quadratic loss for the velocity itself which is expressed in terms of expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the likelihood at any time along the interpolant. In addition, the flow can be optimized to minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. The approach is also contextualized in its relation to diffusions. In particular, in situations where the base is a Gaussian density, we show that the velocity of our normalizing flow can also be used to construct a diffusion model to sample the target as well as estimating its score. This allows one to map methods based on stochastic differential equations to those using ordinary differential equations, simplifying the mechanics of the model, but capturing equivalent dynamics. Benchmarking on density estimation tasks illustrates that the learned flow can match and surpass maximum likelihood continuous flows at a fraction of the conventional ODE training costs.

Simulation of stochastic partial differential equations (SPDE) on a general domain requires a discretization of the noise. In this paper, the noise is discretized by a piecewise linear interpolation. The error caused by this is analyzed in the context of a fully discrete finite element approximation of a semilinear stochastic reaction-advection-diffusion equation on a convex polygon. The noise is Gaussian, white in time and correlated in space. It is modeled as a standard cylindrical Wiener process on the reproducing kernel Hilbert space associated to the covariance kernel. The noise is assumed to extend to a larger polygon than the SPDE domain to allow for sampling by the circulant embedding method. The interpolation error is analyzed under mild assumptions on the kernel. The main tools used are Hilbert--Schmidt bounds of multiplication operators onto negative order Sobolev spaces and an error bound for the finite element interpolant in fractional Sobolev norms. Examples with covariance kernels encountered in applications are illustrated in numerical simulations using the FEniCS finite element software. Conclusions from the analysis include that interpolation of noise with Mat\'ern kernels does not cause an additional error, that there exist kernels where the interpolation error dominates and that generation of noise on a coarser mesh than that of the SPDE discretization does not always result in a loss of accuracy.

Social networks affect the diffusion of information, and thus have the potential to reduce or amplify inequality in access to opportunity. We show empirically that social networks often exhibit a much larger potential for unequal diffusion across groups along paths of length 2 and 3 than expected by our random graph models. We argue that homophily alone cannot not fully explain the extent of unequal diffusion and attribute this mismatch to unequal distribution of cross-group links among the nodes. Based on this insight, we develop a variant of the stochastic block model that incorporates the heterogeneity in cross-group linking. The model provides an unbiased and consistent estimate of assortativity or homophily on paths of length 2 and provide a more accurate estimate along paths of length 3 than existing models. We characterize the null distribution of its log-likelihood ratio test and argue that the goodness of fit test is valid only when the network is dense. Based on our empirical observations and modeling results, we conclude that the impact of any departure from equal distribution of links to source nodes in the diffusion process is not limited to its first order effects as some nodes will have fewer direct links to the sources. More importantly, this unequal distribution will also lead to second order effects as the whole group will have fewer diffusion paths to the sources.

Diffusion models are a class of deep generative models that have shown impressive results on various tasks with dense theoretical founding. Although diffusion models have achieved impressive quality and diversity of sample synthesis than other state-of-the-art models, they still suffer from costly sampling procedure and sub-optimal likelihood estimation. Recent studies have shown great enthusiasm on improving the performance of diffusion model. In this article, we present a first comprehensive review of existing variants of the diffusion models. Specifically, we provide a first taxonomy of diffusion models and categorize them variants to three types, namely sampling-acceleration enhancement, likelihood-maximization enhancement and data-generalization enhancement. We also introduce in detail other five generative models (i.e., variational autoencoders, generative adversarial networks, normalizing flow, autoregressive models, and energy-based models), and clarify the connections between diffusion models and these generative models. Then we make a thorough investigation into the applications of diffusion models, including computer vision, natural language processing, waveform signal processing, multi-modal modeling, molecular graph generation, time series modeling, and adversarial purification. Furthermore, we propose new perspectives pertaining to the development of this generative model.

As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the numerical values in those computations arose. Strongly related to the problem of numerical representation is the problem of quantization: in what manner should a set of continuous real-valued numbers be distributed over a fixed discrete set of numbers to minimize the number of bits required and also to maximize the accuracy of the attendant computations? This perennial problem of quantization is particularly relevant whenever memory and/or computational resources are severely restricted, and it has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications. Thus, it is not surprising that quantization has emerged recently as an important and very active sub-area of research in the efficient implementation of computations associated with Neural Networks. In this article, we survey approaches to the problem of quantizing the numerical values in deep Neural Network computations, covering the advantages/disadvantages of current methods. With this survey and its organization, we hope to have presented a useful snapshot of the current research in quantization for Neural Networks and to have given an intelligent organization to ease the evaluation of future research in this area.

We propose a novel and efficient lifting approach for the optimal control of rigid-body systems with contacts to improve the convergence properties of Newton-type methods. To relax the high nonlinearity, we consider the state, acceleration, contact forces, and control input torques, as optimization variables and the inverse dynamics and acceleration constraints on the contact frames as equality constraints. We eliminate the update of the acceleration, contact forces, and their dual variables from the linear equation to be solved in each Newton-type iteration in an efficient manner. As a result, the computational cost per Newton-type iteration is almost identical to that of the conventional non-lifted Newton-type iteration that embeds contact dynamics in the state equation. We conducted numerical experiments on the whole-body optimal control of various quadrupedal gaits subject to the friction cone constraints considered in interior-point methods and demonstrated that the proposed method can significantly increase the convergence speed to more than twice that of the conventional non-lifted approach.

Cross-diffusion systems arise as hydrodynamic limits of lattice multi-species interacting particle models. The objective of this work is to provide a numerical scheme for the simulation of the cross-diffusion system identified in [J. Quastel, Comm. Pure Appl. Math., 45 (1992), pp. 623--679]. To simulate this system, it is necessary to provide an approximation of the so-called self-diffusion coefficient matrix of the tagged particle process. Classical algorithms for the computation of this matrix are based on the estimation of the long-time limit of the average mean square displacement of the particle. In this work, as an alternative, we propose a novel approach for computing the self-diffusion coefficient using deterministic low-rank approximation techniques, as the minimum of a high-dimensional optimization problem. The computed self-diffusion coefficient is then used for the simulation of the cross-diffusion system using an implicit finite volume scheme.

Imposition methods of interface conditions for the second-order wave equation with non-conforming grids is considered. The spatial discretization is based on high order finite differences with summation-by-parts properties. Previously presented solution methods for this problem, based on the simultaneous approximation term (SAT) method, have shown to introduce significant stiffness. This can lead to highly inefficient schemes. Here, two new methods of imposing the interface conditions to avoid the stiffness problems are presented: 1) a projection method and 2) a hybrid between the projection method and the SAT method. Numerical experiments are performed using traditional and order-preserving interpolation operators. Both of the novel methods retain the accuracy and convergence behavior of the previously developed SAT method but are significantly less stiff.

The existing discrete variational derivative method is only second-order accurate and fully implicit. In this paper, we propose a framework to construct an arbitrary high-order implicit (original) energy stable scheme and a second-order semi-implicit (modified) energy stable scheme. Combined with the Runge--Kutta process, we can build an arbitrary high-order and unconditionally (original) energy stable scheme based on the discrete variational derivative method. The new energy stable scheme is implicit and leads to a large sparse nonlinear algebraic system at each time step, which can be efficiently solved by using an inexact Newton type algorithm. To avoid solving nonlinear algebraic systems, we then present a relaxed discrete variational derivative method, which can construct second-order, linear, and unconditionally (modified) energy stable schemes. Several numerical simulations are performed to investigate the efficiency, stability, and accuracy of the newly proposed schemes.

The paper proposes a decoupled numerical scheme of the time-dependent Ginzburg-Landau equations under temporal gauge. For the order parameter and the magnetic potential, the discrete scheme adopts the second type Ned${\rm \acute{e}}$lec element and the linear element for spatial discretization, respectively, and a fully linearized backward Euler method and the first order exponential time differencing method for time discretization, respectively. The maximum bound principle of the order parameter and the energy dissipation law in the discrete sense are proved for this finite element-based scheme. This allows the application of the adaptive time stepping method which can significantly speed up long-time simulations compared to existing numerical schemes, especially for superconductors with complicated shapes. The error estimate is rigorously established in the fully discrete sense. Numerical examples verify the theoretical results of the proposed scheme and demonstrate the vortex motions of superconductors in an external magnetic field.

A simple generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed. The velocity field of this flow is inferred from the probability current of a time-dependent density that interpolates between the base and the target in finite time. Unlike conventional normalizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to a simple quadratic loss for the velocity itself which is expressed in terms of expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the likelihood at any time along the interpolant. In addition, the flow can be optimized to minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. The approach is also contextualized in its relation to diffusions. In particular, in situations where the base is a Gaussian density, we show that the velocity of our normalizing flow can also be used to construct a diffusion model to sample the target as well as estimating its score. This allows one to map methods based on stochastic differential equations to those using ordinary differential equations, simplifying the mechanics of the model, but capturing equivalent dynamics. Benchmarking on density estimation tasks illustrates that the learned flow can match and surpass maximum likelihood continuous flows at a fraction of the conventional ODE training costs.

Simulation of stochastic partial differential equations (SPDE) on a general domain requires a discretization of the noise. In this paper, the noise is discretized by a piecewise linear interpolation. The error caused by this is analyzed in the context of a fully discrete finite element approximation of a semilinear stochastic reaction-advection-diffusion equation on a convex polygon. The noise is Gaussian, white in time and correlated in space. It is modeled as a standard cylindrical Wiener process on the reproducing kernel Hilbert space associated to the covariance kernel. The noise is assumed to extend to a larger polygon than the SPDE domain to allow for sampling by the circulant embedding method. The interpolation error is analyzed under mild assumptions on the kernel. The main tools used are Hilbert--Schmidt bounds of multiplication operators onto negative order Sobolev spaces and an error bound for the finite element interpolant in fractional Sobolev norms. Examples with covariance kernels encountered in applications are illustrated in numerical simulations using the FEniCS finite element software. Conclusions from the analysis include that interpolation of noise with Mat\'ern kernels does not cause an additional error, that there exist kernels where the interpolation error dominates and that generation of noise on a coarser mesh than that of the SPDE discretization does not always result in a loss of accuracy.

Social networks affect the diffusion of information, and thus have the potential to reduce or amplify inequality in access to opportunity. We show empirically that social networks often exhibit a much larger potential for unequal diffusion across groups along paths of length 2 and 3 than expected by our random graph models. We argue that homophily alone cannot not fully explain the extent of unequal diffusion and attribute this mismatch to unequal distribution of cross-group links among the nodes. Based on this insight, we develop a variant of the stochastic block model that incorporates the heterogeneity in cross-group linking. The model provides an unbiased and consistent estimate of assortativity or homophily on paths of length 2 and provide a more accurate estimate along paths of length 3 than existing models. We characterize the null distribution of its log-likelihood ratio test and argue that the goodness of fit test is valid only when the network is dense. Based on our empirical observations and modeling results, we conclude that the impact of any departure from equal distribution of links to source nodes in the diffusion process is not limited to its first order effects as some nodes will have fewer direct links to the sources. More importantly, this unequal distribution will also lead to second order effects as the whole group will have fewer diffusion paths to the sources.

Diffusion models are a class of deep generative models that have shown impressive results on various tasks with dense theoretical founding. Although diffusion models have achieved impressive quality and diversity of sample synthesis than other state-of-the-art models, they still suffer from costly sampling procedure and sub-optimal likelihood estimation. Recent studies have shown great enthusiasm on improving the performance of diffusion model. In this article, we present a first comprehensive review of existing variants of the diffusion models. Specifically, we provide a first taxonomy of diffusion models and categorize them variants to three types, namely sampling-acceleration enhancement, likelihood-maximization enhancement and data-generalization enhancement. We also introduce in detail other five generative models (i.e., variational autoencoders, generative adversarial networks, normalizing flow, autoregressive models, and energy-based models), and clarify the connections between diffusion models and these generative models. Then we make a thorough investigation into the applications of diffusion models, including computer vision, natural language processing, waveform signal processing, multi-modal modeling, molecular graph generation, time series modeling, and adversarial purification. Furthermore, we propose new perspectives pertaining to the development of this generative model.

As soon as abstract mathematical computations were adapted to computation on digital computers, the problem of efficient representation, manipulation, and communication of the numerical values in those computations arose. Strongly related to the problem of numerical representation is the problem of quantization: in what manner should a set of continuous real-valued numbers be distributed over a fixed discrete set of numbers to minimize the number of bits required and also to maximize the accuracy of the attendant computations? This perennial problem of quantization is particularly relevant whenever memory and/or computational resources are severely restricted, and it has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications. Thus, it is not surprising that quantization has emerged recently as an important and very active sub-area of research in the efficient implementation of computations associated with Neural Networks. In this article, we survey approaches to the problem of quantizing the numerical values in deep Neural Network computations, covering the advantages/disadvantages of current methods. With this survey and its organization, we hope to have presented a useful snapshot of the current research in quantization for Neural Networks and to have given an intelligent organization to ease the evaluation of future research in this area.