Finding meta winning ticket to train your MAML

Author

Dawei GAO
Yuexiang XIE
Zimu ZHOU
Zhen WANG
Yaliang LI
Bolin. DING

Publication Type

Conference Proceeding Article

Version

submittedVersion

Publication Date

8-2022

Abstract

The lottery ticket hypothesis (LTH) states that a randomly initialized dense network contains sub-networks that can be trained in isolation to the performance of the dense network. In this paper, to achieve rapid learning with less computational cost, we explore LTH in the context of meta learning. First, we experimentally show that there are sparse sub-networks, known as meta winning tickets, which can be meta-trained to few-shot classification accuracy to the original backbone. The application of LTH in meta learning enables the adaptation of meta-trained networks on various IoT devices with fewer computation. However, the status quo to identify winning tickets requires iterative training and pruning, which is particularly expensive for finding meta winning tickets. To this end, then we investigate the inter- and intra-layer patterns among different meta winning tickets, and propose a scheme for early detection of a meta winning ticket. The proposed scheme enables efficient training in resource-limited devices. Besides, it also designs a lightweight solution to search the meta winning ticket. Evaluations on standard few-shot classification benchmarks show that we can find competitive meta winning tickets with 20% weights of the original backbone, while incurring only 8%-14% (Conv-4) and 19%-29% (ResNet-12) computation overhead (measured by FLOPs) of the standard winning ticket finding scheme.

Keywords

Meta Learning, Network Pruning, Lottery Ticket Hypothesis

Discipline

Software Engineering

Research Areas

Software and Cyber-Physical Systems

Publication

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14-18

First Page

411

Last Page

420

Identifier

10.1145/3534678.3539467

Publisher

Association for Computing Machinery

City or Country

New York

Citation

GAO, Dawei; XIE, Yuexiang; ZHOU, Zimu; WANG, Zhen; LI, Yaliang; and DING, Bolin.. Finding meta winning ticket to train your MAML. (2022). KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14-18. 411-420.
Available at: https://ink.library.smu.edu.sg/sis_research/7260

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.

Additional URL

https://doi.org/10.1145/3534678.3539467