Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
12-2018
Abstract
Future mobile devices are anticipated to perceive, understand and react to the world on their own by running multiple correlated deep neural networks on-device. Yet the complexity of these neural networks needs to be trimmed down both within-model and cross-model to fit in mobile storage and memory. Previous studies squeeze the redundancy within a single model. In this work, we aim to reduce the redundancy across multiple models. We propose Multi-Task Zipping (MTZ), a framework to automatically merge correlated, pre-trained deep neural networks for cross-model compression. Central in MTZ is a layer-wise neuron sharing and incoming weight updating scheme that induces a minimal change in the error function. MTZ inherits information from each model and demands light retraining to re-boost the accuracy of individual tasks. Evaluations show that MTZ is able to fully merge the hidden layers of two VGG-16 networks with a 3.18% increase in the test error averaged on ImageNet and CelebA, or share 39.61% parameters between the two networks with .5% increase in the test errors for both tasks. The number of iterations to retrain the combined network is at least 17.8 × lower than that of training a single VGG-16 network. Moreover, experiments show that MTZ is also able to effectively merge multiple residual networks.
Discipline
Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
Proceedings of the 32nd Annual Conference on Advances in Neural Information Processing Systems, Montréal, Canada, 2018 December 2-8
Volume
31
First Page
6019
Last Page
6029
City or Country
Montréal, Canada
Citation
HE, Xiaoxi; ZHOU, Zimu; and THIELE, Lothar.
Multi-task zipping via layer-wise neuron sharing. (2018). Proceedings of the 32nd Annual Conference on Advances in Neural Information Processing Systems, Montréal, Canada, 2018 December 2-8. 31, 6019-6029.
Available at: https://ink.library.smu.edu.sg/sis_research/4554
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://dl.acm.org/citation.cfm?id=3327501