Publication Type
Conference Proceeding Article
Version
submittedVersion
Publication Date
8-2022
Abstract
The lottery ticket hypothesis (LTH) states that a randomly initialized dense network contains sub-networks that can be trained in isolation to the performance of the dense network. In this paper, to achieve rapid learning with less computational cost, we explore LTH in the context of meta learning. First, we experimentally show that there are sparse sub-networks, known as meta winning tickets, which can be meta-trained to few-shot classification accuracy to the original backbone. The application of LTH in meta learning enables the adaptation of meta-trained networks on various IoT devices with fewer computation. However, the status quo to identify winning tickets requires iterative training and pruning, which is particularly expensive for finding meta winning tickets. To this end, then we investigate the inter- and intra-layer patterns among different meta winning tickets, and propose a scheme for early detection of a meta winning ticket. The proposed scheme enables efficient training in resource-limited devices. Besides, it also designs a lightweight solution to search the meta winning ticket. Evaluations on standard few-shot classification benchmarks show that we can find competitive meta winning tickets with 20% weights of the original backbone, while incurring only 8%-14% (Conv-4) and 19%-29% (ResNet-12) computation overhead (measured by FLOPs) of the standard winning ticket finding scheme.
Keywords
Meta Learning, Network Pruning, Lottery Ticket Hypothesis
Discipline
Software Engineering
Research Areas
Software and Cyber-Physical Systems
Publication
Proceedings 28th ACM SIGKDD Conference On Knowledge Discovery And Data Mining, Washington, DC, USA, August 14-18
First Page
411
Last Page
420
Identifier
10.1145/3534678.3539467
Publisher
Association for Computing Machinery
City or Country
New York
Citation
GAO, Dawei; XIE, Yuexiang; ZHOU, Zimu; WANG, Zhen; LI, Yaliang; and DING, Bolin..
Finding meta winning ticket to train your MAML. (2022). Proceedings 28th ACM SIGKDD Conference On Knowledge Discovery And Data Mining, Washington, DC, USA, August 14-18. 411-420.
Available at: https://ink.library.smu.edu.sg/sis_research/7256
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
https://doi.org/10.1145/3534678.3539467