Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

4-2021

Abstract

We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when computing mappings. Spectral properties are often subject to constraints in optimization problems, leading to better models and stability of optimization. We start by looking at the compact SVD parameterization of weight matrices and identifying redundancy sources in the parameterization. We further apply the Tensor Train (TT) decomposition to the compact SVD components, and propose a non-redundant differentiable parameterization of fixed TT-rank tensor manifolds, termed the Spectral Tensor Train Parameterization (STTP). We demonstrate the effects of neural network compression in the image classification setting, and both compression and improved training stability in the generative adversarial training setting.

Discipline

Artificial Intelligence and Robotics | Databases and Information Systems

Research Areas

Data Science and Engineering

Publication

Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, 2021 Apr 13-15

First Page

3547

Last Page

3555

Identifier

PMLR 130:3547-3555

Publisher

Proceedings of Machine Learning Research

City or Country

Virtual

Download

Share

COinS