Download:
Abstract:

Consider the problem: given data pair $(\mathbf{x}, \mathbf{y})$ drawn from a population with $f_\star(x) = \mathbb{E}[ \mathbf{y} | \mathbf{x} = x]$, specify a neural network and run gradient flow on the weights over time until any stationarity. How does $f_t$, the function computed by the neural network at time $t$, relate to $f_\star$, in terms of approximation and representation? What are the provable benefits of the adaptive representation by neural networks compared to the pre-specified fixed basis representation in the classical nonparametric literature? We answer the above questions via a dynamic reproducing kernel Hilbert space (RKHS) view indexed by the training process of neural networks. We show that when reaching any local stationarity, gradient flow learns an adaptive RKHS representation, and performs the global least squares projection onto the adaptive RKHS, simultaneously. In addition, we prove that as the RKHS is data-adaptive and task-specific, the residual for $f_\star$ lies in a subspace that is smaller than the orthogonal complement of the RKHS, formalizing the representation and approximation benefits of neural networks.

Citation

Xialiang Dou, and Tengyuan Liang. 2021. “Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits.” Journal of the American Statistical Association (Theory and Methods) 116 (535): 1507–1520.

@article{Dou_2020,
   title={Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits},
   volume={116},
   ISSN={1537-274X},
   url={http://dx.doi.org/10.1080/01621459.2020.1745812},
   DOI={10.1080/01621459.2020.1745812},
   number={535},
   journal={Journal of the American Statistical Association},
   author={Dou, Xialiang and Liang, Tengyuan},
   year={2020},
   month=apr, pages={1507–1520} }