Learning with Square Loss: Localization through Offset Rademacher Complexity
We introduce Offset Rademacher Complexity as a tool for analyzing the localization properties and fast rates of learning algorithms under square loss.
Statistical Learning
How Well Generative Adversarial Networks Learn Distributions
This paper studies the rates of convergence for learning distributions implicitly with the adversarial framework and Generative Adversarial Networks (GANs), which subsume Wasserstein, Sobolev, MMD GAN, and Generalized/Simulated Method of Moments (GMM/SMM) as special cases. We study a wide range of parametric and nonparametric target distributions under a host of objective evaluation metrics. We investigate how to obtain valid statistical guarantees for GANs through the lens of regularization.
Just Interpolate: Kernel Ridgeless Regression Can Generalize
In the absence of explicit regularization, interpolating kernel machine has the potential to fit the training data perfectly, at the same time, still generalizes well on test data. We isolate a phenomenon of implicit regularization for minimum-norm interpolated solutions.
Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits
What are the provable benefits of the adaptive representation by neural networks compared to the pre-specified fixed basis representation in the classical nonparametric literature? We answer the above questions via a dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the training process of neural networks.
On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels
We study the risk of minimum-norm interpolants of data in Reproducing Kernel Hilbert Spaces. Our upper bounds on the risk are of a multiple-descent shape. Empirical evidence supports our finding that minimum-norm interpolants in RKHS can exhibit this unusual non-monotonicity in sample size.
A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-L1-Norm Interpolated Classifiers
This paper establishes a precise high-dimensional asymptotic theory for boosting on separable data, taking statistical and computational perspectives.
Mehler’s Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
We utilize a connection between compositional kernels and branching processes via Mehler’s formula to study deep neural networks. This new probabilistic insight provides us a novel perspective on the mathematical role of activation functions in compositional neural networks. We study the unscaled and rescaled limits of the compositional kernels and explore the different phases of the limiting behavior, as the compositional depth increases.
Interpolating Classifiers Make Few Mistakes
This paper provides elementary analyses of the regret and generalization of minimum-norm interpolating classifiers.
Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria
Blessings and curses of covariate shifts, directional convergece, and the connection to experimental design.
Learning When the Concept Shifts: Confounding, Invariance, and Dimension Reduction
Confounding can obfuscate the definition of the best prediction model (concept shift) and shift covariates to domains yet unseen (covariate shift). Therefore, a model maximizing prediction accuracy in the source environment could suffer a significant accuracy drop in the target environment. We propose a new domain adaptation method for observational data in the presence of confounding, and characterize the the stability and predictability tradeoff leveraging a structural causal model.