Publication Type

Conference Proceeding Article

Version

acceptedVersion

Publication Date

12-2020

Abstract

Despite its high search efficiency, differential architecture search (DARTS) often selects network architectures with dominated skip connections which lead to performance degradation. However, theoretical understandings on this issue remain absent, hindering the development of more advanced methods in a principled way. In this work, we solve this problem by theoretically analyzing the effects of various types of operations, e.g. convolution, skip connection and zero operation, to the network optimization. We prove that the architectures with more skip connections can converge faster than the other candidates, and thus are selected by DARTS. This result, for the first time, theoretically and explicitly reveals the impact of skip connections to fast network optimization and its competitive advantage over other types of operations in DARTS. Then we propose a theory-inspired path-regularized DARTS that consists of two key modules: (i) a differential group-structured sparse binary gate introduced for each operation to avoid unfair competition among operations, and (ii) a path-depth-wise regularization used to incite search exploration for deep architectures that often converge slower than shallow ones as shown in our theory and are not well explored during search. Experimental results on image classification tasks validate its advantages.

Discipline

OS and Networks | Systems Architecture

Research Areas

Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

Proceedings of the 34th Conference on Neural Information Processing Systems, Virtual Conference, 2020 December 6-12

First Page

1

Last Page

12

Publisher

NeurIPS

City or Country

Virtual Conference

Additional URL

https://proceedings.neurips.cc/paper_files/paper/2020/hash/5e1b18c4c6a6d31695acbae3fd70ecc6-Abstract.html

Download

Share

COinS