Deep learning has enjoyed tremendous success in a variety of applications but its application to quantile regressions remains scarce. A major advantage of the deep learning approach is its flexibility to model complex data in a more parsimonious way than nonparametric smoothing methods. However, while deep learning brought breakthroughs in prediction, it often lacks interpretability due to the black-box nature of multilayer structure with millions of parameters, hence it is not well suited for statistical inference. In this paper, we leverage the advantages of deep learning to apply it to quantile regression where the goal to produce interpretable results and perform statistical inference. We achieve this by adopting a semiparametric approach based on the partially linear quantile regression model, where covariates of primary interest for statistical inference are modelled linearly and all other covariates are modelled nonparametrically by means of a deep neural network. In addition to the new methodology, we provide theoretical justification for the proposed model by establishing the root-$n$ consistency and asymptotically normality of the parametric coefficient estimator and the minimax optimal convergence rate of the neural nonparametric function estimator. Across several simulated and real data examples, our proposed model empirically produces superior estimates and more accurate predictions than various alternative approaches.
相關內容
Gaussian process regression (GPR) model is a popular nonparametric regression model. In GPR, features of the regression function such as varying degrees of smoothness and periodicities are modeled through combining various covarinace kernels, which are supposed to model certain effects. The covariance kernels have unknown parameters which are estimated by the EM-algorithm or Markov Chain Monte Carlo. The estimated parameters are keys to the inference of the features of the regression functions, but identifiability of these parameters has not been investigated. In this paper, we prove identifiability of covariance kernel parameters in two radial basis mixed kernel GPR and radial basis and periodic mixed kernel GPR. We also provide some examples about non-identifiable cases in such mixed kernel GPRs.
We propose several novel consistent specification tests for quantile regression models which generalize former tests by important characteristics. First, we allow the covariate effects to be quantile-dependent and nonlinear, bypassing estimation difficulties such as multicollinearity. Second, we allow for parameterizing the conditional quantile functions by appropriate basis functions, rather than parametrically. In the framework of splines, we can thus test for the order and number of knots. We are hence able to test for functional forms beyond linearity, while retaining the linear effects as special cases. In both cases, the induced class of conditional distribution functions is tested with a Cram\'{e}r-von Mises type test statistic for which we derive the theoretical limit distribution and propose a practical bootstrap method. To increase the power of the first test, we further suggest a modified test statistic using the $B$-spline approach from the second test. A detailed Monte Carlo experiment shows that the tests result in reasonable sized testing procedures with large power. Our first application to conditional income disparities between East and West Germany over the period 2001--2010 indicates that there are not only still significant differences between East and West but also across the quantiles of the conditional income distributions, when conditioning on age and year. The second application to data from the Australian national electricity market reveals the importance of using interaction effects for modelling the highly skewed and heavy-tailed distributions of energy prices conditional on day, time of day and demand.
We develop a Bayesian graphical modeling framework for functional data for correlated multivariate random variables observed over a continuous domain. Our method leads to graphical Markov models for functional data which allows the graphs to vary over the functional domain. The model involves estimation of graphical models that evolve functionally in a nonparametric fashion while accounting for within-functional correlations and borrowing strength across functional positions so contiguous locations are encouraged but not forced to have similar graph structure and edge strength. We utilize a strategy that combines nonparametric basis function modeling with modified Bayesian graphical regularization techniques, which induces a new class of hypoexponential normal scale mixture distributions that not only leads to adaptively shrunken estimators of the conditional cross-covariance but also facilitates a thorough theoretical investigation of the shrinkage properties. Our approach scales up to large functional datasets collected on a fine grid. We show through simulations and real data analysis that the Bayesian functional graphical model can efficiently reconstruct the functionally-evolving graphical models by accounting for within-function correlations.
We study empirical Bayes estimation of the effect sizes of $N$ units from $K$ noisy observations on each unit. We show that it is possible to achieve near-Bayes optimal mean squared error, without any assumptions or knowledge about the effect size distribution or the noise. The noise distribution can be heteroskedastic and vary arbitrarily from unit to unit. Our proposal, which we call Aurora, leverages the replication inherent in the $K$ observations per unit and recasts the effect size estimation problem as a general regression problem. Aurora with linear regression provably matches the performance of a wide array of estimators including the sample mean, the trimmed mean, the sample median, as well as James-Stein shrunk versions thereof. Aurora automates effect size estimation for Internet-scale datasets, as we demonstrate on data from a large technology firm.
We perform scalable approximate inference in a continuous-depth Bayesian neural network family. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infinite-parameter setting, producing arbitrarily-flexible approximate posteriors. We also derive a novel gradient estimator that approaches zero variance as the approximate posterior over weights approaches the true posterior. This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-efficient training and tunable precision of Neural ODEs.
We propose a principal components regression method based on maximizing a joint pseudo-likelihood for responses and predictors. Our method uses both responses and predictors to select linear combinations of the predictors relevant for the regression, thereby addressing an oft-cited deficiency of conventional principal components regression. The proposed estimator is shown to be consistent in a wide range of settings, including ones with non-normal and dependent observations; conditions on the first and second moments suffice if the number of predictors ($p$) is fixed and the number of observations ($n$) tends to infinity and dependence is weak, while stronger distributional assumptions are needed when $p \to \infty$ with $n$. We obtain the estimator's asymptotic distribution as the projection of a multivariate normal random vector onto a tangent cone of the parameter set at the true parameter, and find the estimator is asymptotically more efficient than competing ones. In simulations our method is substantially more accurate than conventional principal components regression and compares favorably to partial least squares and predictor envelopes. The method's practical usefulness is illustrated in a data example with cross-sectional prediction of stock returns.
Ordinary differential equations (ODEs) are a mathematical model used in many application areas such as climatology, bioinformatics, and chemical engineering with its intuitive appeal to modeling. Despite ODE's wide usage in modeling, the frequent absence of their analytic solutions makes it challenging to estimate ODE parameters from the data, especially when the model has lots of variables and parameters. This paper proposes a Bayesian ODE parameter estimating algorithm which is fast and accurate even for models with many parameters. The proposed method approximates an ODE model with a state-space model based on equations of a numeric solver. It allows fast estimation by avoiding computations of a complete numerical solution in the likelihood. The posterior is obtained by a variational Bayes method, more specifically, the approximate Riemannian conjugate gradient method (Honkela et al. 2010), which avoids samplings based on Markov chain Monte Carlo (MCMC). In simulation studies, we compared the speed and performance of the proposed method with existing methods. The proposed method showed the best performance in the reproduction of the true ODE curve with strong stability as well as the fastest computation, especially in a large model with more than 30 parameters. As a real-world data application, a SIR model with time-varying parameters was fitted to the COVID-19 data. Taking advantage of the proposed algorithm, more than 50 parameters were adequately estimated for each country.
This survey is meant to provide an introduction to linear models and the theories behind them. Our goal is to give a rigorous introduction to the readers with prior exposure to ordinary least squares. In machine learning, the output is usually a nonlinear function of the input. Deep learning even aims to find a nonlinear dependence with many layers which require a large amount of computation. However, most of these algorithms build upon simple linear models. We then describe linear models from different views and find the properties and theories behind the models. The linear model is the main technique in regression problems and the primary tool for it is the least squares approximation which minimizes a sum of squared errors. This is a natural choice when we're interested in finding the regression function which minimizes the corresponding expected squared error. This survey is primarily a summary of purpose, significance of important theories behind linear models, e.g., distribution theory, minimum variance estimator. We first describe ordinary least squares from three different points of view upon which we disturb the model with random noise and Gaussian noise. By Gaussian noise, the model gives rise to the likelihood so that we introduce a maximum likelihood estimator. It also develops some distribution theories via this Gaussian disturbance. The distribution theory of least squares will help us answer various questions and introduce related applications. We then prove least squares is the best unbiased linear model in the sense of mean squared error and most importantly, it actually approaches the theoretical limit. We end up with linear models with the Bayesian approach and beyond.
This article considers inference in linear instrumental variables models with many regressors, all of which could be endogenous. We propose the STIV estimator. Identification robust confidence sets are derived by solving linear programs. We present results on rates of convergence, variable selection, confidence sets which adapt to the sparsity, and analyze confidence bands for vectors of linear functions using bias correction. We also provide solutions to some instruments being endogenous. The application is to the EASI demand system.
Hahn et al. (2020) offers an extensive study to explicate and evaluate the performance of the BCF model in different settings and provides a detailed discussion about its utility in causal inference. It is a welcomed addition to the causal machine learning literature. I will emphasize the contribution of the BCF model to the field of causal inference through discussions on two topics: 1) the difference between the PS in the BCF model and the Bayesian PS in a Bayesian updating approach, 2) an alternative exposition of the role of the PS in outcome modeling based methods for the estimation of causal effects. I will conclude with comments on avenues for future research involving BCF that will be important and much needed in the era of Big data.
小貼士
登錄享
相關主題
注冊地址： 北京市海淀區羊坊店路18號2幢3層301-191