Randomization Inference When N Equals One

A statistical theory for N-of-1 experiments, where a unit serves as its own control and treatment in different time windows.

October 2023 · Tengyuan Liang, Benjamin Recht

Detecting Weak Distribution Shifts via Displacement Interpolation

Detecting weak, systematic distribution shifts and quantitatively modeling individual, heterogeneous responses to policies or incentives have found increasing empirical applications in social and economic sciences. We propose a model for weak distribution shifts via displacement interpolation, drawing from the optimal transport theory.

May 2023 · YoonHaeng Hur, Tengyuan Liang

Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria

Blessings and curses of covariate shifts, directional convergece, and the connection to experimental design.

December 2022 · Tengyuan Liang

High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Models

We study Langevin dynamics for recovering the planted signal in the spiked matrix model. We provide a path-wise characterization of the overlap between the output of the Langevin algorithm and the planted signal. This overlap is characterized in terms of a self-consistent system of integro-differential equations, usually referred to as the Crisanti-Horner-Sommers-Cugliandolo-Kurchan (CHSCK) equations in the spin glass literature.

April 2022 · Tengyuan Liang, Subhabrata Sen, Pragya Sur

Online Learning to Transport via the Minimal Selection Principle

Motivated by robust dynamic resource allocation in operations research, we study the Online Learning to Transport (OLT) problem where the decision variable is a probability measure, an infinite-dimensional object. We draw connections between online learning, optimal transport, and partial differential equations through an insight called the minimal selection principle, originally studied in the Wasserstein gradient flow setting by Ambrosio et al. (2005).

February 2022 · Wenxuan Guo, YoonHaeng Hur, Tengyuan Liang, Christopher Ryan

Reversible Gromov-Monge Sampler for Simulation-Based Inference

Motivated by the seminal work on distance and isomorphism between metric measure spaces, we propose a new notion called the Reversible Gromov-Monge (RGM) distance and study how RGM can be used to design new transform samplers to perform simulation-based inference.

September 2021 · YoonHaeng Hur, Wenxuan Guo, Tengyuan Liang

Universal Prediction Band via Semi-Definite Programming

This paper proposes a computationally efficient method to construct nonparametric, heteroscedastic prediction bands for uncertainty quantification.

March 2021 · Tengyuan Liang

Interpolating Classifiers Make Few Mistakes

This paper provides elementary analyses of the regret and generalization of minimum-norm interpolating classifiers.

January 2021 · Tengyuan Liang, Benjamin Recht

Mehler’s Formula, Branching Process, and Compositional Kernels of Deep Neural Networks

We utilize a connection between compositional kernels and branching processes via Mehler’s formula to study deep neural networks. This new probabilistic insight provides us a novel perspective on the mathematical role of activation functions in compositional neural networks. We study the unscaled and rescaled limits of the compositional kernels and explore the different phases of the limiting behavior, as the compositional depth increases.

April 2020 · Tengyuan Liang, Hai Tran-Bach

A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-L1-Norm Interpolated Classifiers

This paper establishes a precise high-dimensional asymptotic theory for boosting on separable data, taking statistical and computational perspectives.

February 2020 · Tengyuan Liang, Pragya Sur

On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels

We study the risk of minimum-norm interpolants of data in Reproducing Kernel Hilbert Spaces. Our upper bounds on the risk are of a multiple-descent shape. Empirical evidence supports our finding that minimum-norm interpolants in RKHS can exhibit this unusual non-monotonicity in sample size.

August 2019 · Tengyuan Liang, Alexander Rakhlin, Xiyu Zhai

Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

What are the provable benefits of the adaptive representation by neural networks compared to the pre-specified fixed basis representation in the classical nonparametric literature? We answer the above questions via a dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the training process of neural networks.

January 2019 · Xialiang Dou, Tengyuan Liang

Deep Neural Networks for Estimation and Inference

Can deep neural networks with standard archtectures estimate treatment effects and perform downstream uncertainty quantification tasks?

September 2018 · Max H. Farrell, Tengyuan Liang, Sanjog Misra

Just Interpolate: Kernel Ridgeless Regression Can Generalize

In the absence of explicit regularization, interpolating kernel machine has the potential to fit the training data perfectly, at the same time, still generalizes well on test data. We isolate a phenomenon of implicit regularization for minimum-norm interpolated solutions.

August 2018 · Tengyuan Liang, Alexander Rakhlin

Local Optimality and Generalization Guarantees for the Langevin Algorithm via Empirical Metastability

We study the detailed path-wise behavior of the discrete-time Langevin algorithm for non-convex Empirical Risk Minimization (ERM) through the lens of metastability, adopting some techniques from Berglund and Gentz (2003).

February 2018 · Belinda Tzen, Tengyuan Liang, Maxim Raginsky

How Well Generative Adversarial Networks Learn Distributions

This paper studies the rates of convergence for learning distributions implicitly with the adversarial framework and Generative Adversarial Networks (GANs), which subsume Wasserstein, Sobolev, MMD GAN, and Generalized/Simulated Method of Moments (GMM/SMM) as special cases. We study a wide range of parametric and nonparametric target distributions under a host of objective evaluation metrics. We investigate how to obtain valid statistical guarantees for GANs through the lens of regularization.

December 2017 · Tengyuan Liang

Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradients

Modern statistical inference tasks often require iterative optimization methods to compute the solution. Convergence analysis from an optimization viewpoint only informs us how well the solution is approximated numerically but overlooks the sampling nature of the data. We introduce the moment-adjusted stochastic gradient descents, a new stochastic optimization method for statistical inference.

December 2017 · Tengyuan Liang, Weijie J. Su