Publication Type

Conference Proceeding Article

Version

submittedVersion

Publication Date

2-2021

Abstract

We show generalisation error bounds for deep learning with two main improvements over the state of the art. (1) Our bounds have no explicit dependence on the number of classes except for logarithmic factors. This holds even when formulating the bounds in terms of the Frobenius-norm of the weight matrices, where previous bounds exhibit at least a squareroot dependence on the number of classes. (2) We adapt the classic Rademacher analysis of DNNs to incorporate weight sharing—a task of fundamental theoretical importance which was previously attempted only under very restrictive assumptions. In our results, each convolutional filter contributes only once to the bound, regardless of how many times it is applied. Further improvements exploiting pooling and sparse connections are provided. The presented bounds scale as the norms of the parameter matrices, rather than the number of parameters. In particular, contrary to bounds based on parameter counting, they are asymptotically tight (up to log factors) when the weights approach initialisation, making them suitable as a basic ingredient in bounds sensitive to the optimisation procedure. We also show how to adapt the recent technique of loss function augmentation to replace spectral norms by empirical analogues whilst maintaining the advantages of our approach.

Keywords

(Deep) Neural Network Learning Theory, Learning Theory

Discipline

Artificial Intelligence and Robotics | Theory and Algorithms

Research Areas

Intelligent Systems and Optimization

Publication

Proceedings of the 35th AAAI Conference on Artificial Intelligence 2021: February 2-9, Virtual

Volume

9

First Page

8279

Last Page

8287

ISBN

9781577358664

Publisher

AAAI Press

City or Country

Palo Alto, CA

Additional URL

https://ojs.aaai.org/index.php/AAAI/article/view/17007

Download

Share

COinS