In traditional Machine Learning, the algorithms predictions are based on the assumption that the data follows the same distribution in both the training and the test datasets. However, in real world data this condition does not hold and, for instance, the distribution of the covariates changes whereas the conditional distribution of the targets remains unchanged. This situation is called covariate shift problem where standard error estimation may be no longer accurate. In this context, the importance is a measure commonly used to alleviate the influence of covariate shift on error estimations. The main drawback is that it is not easy to compute. The Kullback-Leibler Importance Estimation Procedure (KLIEP) is capable of estimating importance in a promising way. Despite its good performance, it fails to ignore target information, since it only includes the covariates information for computing the importance. In this direction, this paper explores the potential performance improvement if target information is considered in the computation of the importance. Then, a redefinition of the importance arises in order to be generalized in this way. Besides the potential improvement in performance, including target information make possible the application to a real application about plankton classification that motivates this research and characterized by its great dimensionality, since considering targets rather than covariates reduces the computation and the noise in the covariates. The impact of taking target information is also explored when Logistic Regression (LR), Kernel Mean Matching (KMM), Ensemble Kernel Mean Matching (EKMM) and the naive predecessor of KLIEP called Kernel Density Estimation (KDE) methods estimate the importance. The experimental results lead to a more accurate error estimation using target information, especially in case of the more promising method KLIEP.
The Fisher-Rao distance between two probability distributions of a statistical model is defined as the Riemannian geodesic distance induced by the Fisher information metric. In order to calculate the Fisher-Rao distance in closed-form, we need (1) to elicit a formula for the Fisher-Rao geodesics, and (2) to integrate the Fisher length element along those geodesics. We consider several numerically robust approximation and bounding techniques for the Fisher-Rao distances: First, we report generic upper bounds on Fisher-Rao distances based on closed-form 1D Fisher-Rao distances of submodels. Second, we describe several generic approximation schemes depending on whether the Fisher-Rao geodesics or pregeodesics are available in closed-form or not. In particular, we obtain a generic method to guarantee an arbitrarily small additive error on the approximation provided that Fisher-Rao pregeodesics and tight lower and upper bounds are available. Third, we consider the case of Fisher metrics being Hessian metrics, and report generic tight upper bounds on the Fisher-Rao distances using techniques of information geometry. Uniparametric and biparametric statistical models always have Fisher Hessian metrics, and in general a simple test allows to check whether the Fisher information matrix yields a Hessian metric or not. Fourth, we consider elliptical distribution families and show how to apply the above techniques to these models. We also propose two new distances based either on the Fisher-Rao lengths of curves serving as proxies of Fisher-Rao geodesics, or based on the Birkhoff/Hilbert projective cone distance. Last, we consider an alternative group-theoretic approach for statistical transformation models based on the notion of maximal invariant which yields insights on the structures of the Fisher-Rao distance formula which may be used fruitfully in applications.
As language models (LMs) become more capable, it is increasingly important to align them with human preferences. However, the dominant paradigm for training Preference Models (PMs) for that purpose suffers from fundamental limitations, such as lack of transparency and scalability, along with susceptibility to overfitting the preference dataset. We propose Compositional Preference Models (CPMs), a novel PM framework that decomposes one global preference assessment into several interpretable features, obtains scalar scores for these features from a prompted LM, and aggregates these scores using a logistic regression classifier. Through these simple steps, CPMs allow to control which properties of the preference data are used to train the preference model and to build it based on features that are believed to underlie the human preference judgment. Our experiments show that CPMs not only improve generalization and are more robust to overoptimization than standard PMs, but also that best-of-n samples obtained using CPMs tend to be preferred over samples obtained using conventional PMs. Overall, our approach demonstrates the benefits of endowing PMs with priors about which features determine human preferences while relying on LM capabilities to extract those features in a scalable and robust way.
The ParaDiag family of algorithms solves differential equations by using preconditioners that can be inverted in parallel through diagonalization. In the context of optimal control of linear parabolic PDEs, the state-of-the-art ParaDiag method is limited to solving self-adjoint problems with a tracking objective. We propose three improvements to the ParaDiag method: the use of alpha-circulant matrices to construct an alternative preconditioner, a generalization of the algorithm for solving non-self-adjoint equations, and the formulation of an algorithm for terminal-cost objectives. We present novel analytic results about the eigenvalues of the preconditioned systems for all discussed ParaDiag algorithms in the case of self-adjoint equations, which proves the favorable properties the alpha-circulant preconditioner. We use these results to perform a theoretical parallel-scaling analysis of ParaDiag for self-adjoint problems. Numerical tests confirm our findings and suggest that the self-adjoint behavior, which is backed by theory, generalizes to the non-self-adjoint case. We provide a sequential, open-source reference solver in Matlab for all discussed algorithms.
In logistic regression modeling, Firth's modified estimator is widely used to address the issue of data separation, which results in the nonexistence of the maximum likelihood estimate. Firth's modified estimator can be formulated as a penalized maximum likelihood estimator in which Jeffreys' prior is adopted as the penalty term. Despite its widespread use in practice, the formal verification of the corresponding estimate's existence has not been established. In this study, we establish the existence theorem of Firth's modified estimate in binomial logistic regression models, assuming only the full column rankness of the design matrix. We also discuss other binomial regression models obtained through alternating link functions and prove the existence of similar penalized maximum likelihood estimates for such models.
We demonstrate that Assembly Theory, pathway complexity, the assembly index, and the assembly number are subsumed and constitute a weak version of algorithmic (Kolmogorov-Solomonoff-Chaitin) complexity reliant on an approximation method based upon statistical compression, their results obtained due to the use of methods strictly equivalent to the LZ family of compression algorithms used in compressing algorithms such as ZIP, GZIP, or JPEG. Such popular algorithms have been shown to empirically reproduce the results of AT that were reported before in successful application to separating organic from non-organic molecules and in the context of the study of selection and evolution. We prove the connections and full equivalence of Assembly Theory to Shannon Entropy and statistical compression, and AT's disconnection as a statistical approach from causality. We demonstrate that formulating a traditional statistically compressed description of molecules, or the theory underlying it, does not imply an explanation or quantification of biases in generative (physical or biological) processes, including those brought about by selection and evolution, when lacking in logical consistency and empirical evidence. We argue that in their basic arguments, the authors of AT conflate how objects may assemble with causal directionality, and conclude that Assembly Theory does not explain selection or evolution beyond known and previously established connections, some of which are reviewed.
We demonstrate that Assembly Theory, pathway complexity, the assembly index, and the assembly number are subsumed and constitute a weak version of algorithmic (Kolmogorov-Solomonoff-Chaitin) complexity reliant on an approximation method based upon statistical compression, their results obtained due to the use of methods strictly equivalent to the LZ family of compression algorithms used in compressing algorithms such as ZIP, GZIP, or JPEG. Such popular algorithms have been shown to empirically reproduce the results of AT's assembly index and their use had already been reported in successful application to separating organic from non-organic molecules, and the study of selection and evolution. Here we exhibit and prove the connections and full equivalence of Assembly Theory to Shannon Entropy and statistical compression, and AT's disconnection as a statistical approach from causality. We demonstrate that formulating a traditional statistically compressed description of molecules, or the theory underlying it, does not imply an explanation or quantification of biases in generative (physical or biological) processes, including those brought about by selection and evolution, when lacking in logical consistency and empirical evidence. We argue that in their basic arguments, the authors of AT conflate how objects may assemble with causal directionality, and conclude that Assembly Theory does nothing to explain selection or evolution beyond known and previously established connections, some of which are reviewed.
LiDAR-based 3D perception algorithms have evolved rapidly alongside the emergence of large datasets. Nonetheless, considerable performance degradation often ensues when models trained on a specific dataset are applied to other datasets or real-world scenarios with different LiDAR. This paper aims to develop a unified model capable of handling different LiDARs, enabling continual learning across diverse LiDAR datasets and seamless deployment across heterogeneous platforms. We observe that the gaps among datasets primarily manifest in geometric disparities (such as variations in beams and point counts) and semantic inconsistencies (taxonomy conflicts). To this end, this paper proposes UniLiDAR, an occupancy prediction pipeline that leverages geometric realignment and semantic label mapping to facilitate multiple datasets training and mitigate performance degradation during deployment on heterogeneous platforms. Moreover, our method can be easily combined with existing 3D perception models. The efficacy of the proposed approach in bridging LiDAR domain gaps is verified by comprehensive experiments on two prominent datasets: OpenOccupancy-nuScenes and SemanticKITTI. UniLiDAR elevates the mIoU of occupancy prediction by 15.7% and 12.5%, respectively, compared to the model trained on the directly merged dataset. Moreover, it outperforms several SOTA methods trained on individual datasets. We expect our research to facilitate further study of 3D generalization, the code will be available soon.
We study strong approximation of scalar additive noise driven stochastic differential equations (SDEs) at time point $1$ in the case that the drift coefficient is bounded and has Sobolev regularity $s\in(0,1)$. Recently, it has been shown in [arXiv:2101.12185v2 (2022)] that for such SDEs the equidistant Euler approximation achieves an $L^2$-error rate of at least $(1+s)/2$, up to an arbitrary small $\varepsilon$, in terms of the number of evaluations of the driving Brownian motion $W$. In the present article we prove a matching lower error bound for $s\in(1/2,1)$. More precisely we show that, for every $s\in(1/2,1)$, the $L^2$-error rate $(1+s)/2$ can, up to a logarithmic term, not be improved in general by no numerical method based on finitely many evaluations of $W$ at fixed time points. Up to now, this result was known in the literature only for the cases $s=1/2-$ and $s=1-$. For the proof we employ the coupling of noise technique recently introduced in [arXiv:2010.00915 (2020)] to bound the $L^2$-error of an arbitrary approximation from below by the $L^2$-distance of two occupation time functionals provided by a specifically chosen drift coefficient with Sobolev regularity $s$ and two solutions of the corresponding SDE with coupled driving Brownian motions. For the analysis of the latter distance we employ a transformation of the original SDE to overcome the problem of correlated increments of the difference of the two coupled solutions, occupation time estimates to cope with the lack of regularity of the chosen drift coefficient around the point $0$ and scaling properties of the drift coefficient.
In logistic regression modeling, Firth's modified estimator is widely used to address the issue of data separation, which results in the nonexistence of the maximum likelihood estimate. Firth's modified estimator can be formulated as a penalized maximum likelihood estimator in which Jeffreys' prior is adopted as the penalty term. Despite its widespread use in practice, the formal verification of the corresponding estimate's existence has not been established. In this study, we establish the existence theorem of Firth's modified estimate in binomial logistic regression models, assuming only the full column rankness of the design matrix. We also discuss multinomial logistic regression models. Unlike the binomial regression case, we show through an example that the Jeffreys-prior penalty term does not necessarily diverge to negative infinity as the parameter diverges.
Non-probability survey samples are examples of data sources that have become increasingly popular in recent years, also in official statistics. However, statistical inference based on non-probability samples is much more difficult because they are biased and are not representative of the target population (Wu, 2022). In this paper we consider a method of joint calibration for totals (Deville & S\"arndal, 1992) and quantiles (Harms & Duchesne, 2006) and use the proposed approach to extend existing inference methods for non-probability samples, such as inverse probability weighting, mass imputation and doubly robust estimators. By including quantile information in the estimation process non-linear relationships between the target and auxiliary variables can be approximated the way it is done in step-wise (constant) regression. Our simulation study has demonstrated that the estimators in question are more robust against model mis-specification and, as a result, help to reduce bias and improve estimation efficiency. Variance estimation for our proposed approach is also discussed. We show that existing inference methods can be used and that the resulting confidence intervals are at nominal levels. Finally, we applied the proposed methods to estimate the share of vacancies aimed at Ukrainian workers in Poland using an integrated set of administrative and survey data about job vacancies. The proposed approaches have been implemented in two R packages (nonprobsvy and jointCalib), which were used to conduct the simulation and empirical study