Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

5-2019

Abstract

We give a new algorithm for learning a two-layer neural network under a very general class of input distributions. Assuming there is a ground-truth two-layer network $y = A \sigma(Wx) + \xi$, where A, W are weight matrices, $\xi$ represents noise, and the number of neurons in the hidden layer is no larger than the input or output, our algorithm is guaranteed to recover the parameters A, W of the ground-truth network. The only requirement on the input x is that it is symmetric, which still allows highly complicated and structured input. Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions. We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks. Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions.

Keywords

Neural network, Optimization, Symmetric inputs, Moment-of-moments

Discipline

Databases and Information Systems | OS and Networks

Research Areas

Data Science and Engineering; Intelligent Systems and Optimization

Publication

Proceedings of the 7th International Conference on Learning Representations (ICLR 2019), New Orleans, Louisiana, May 6-9

First Page

1

Last Page

53

Publisher

International Conference on Learning Representations

City or Country

New Orleans, USA

Additional URL

https://arxiv.org/abs/1810.06793

Download

Share

COinS