Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
4-2024
Abstract
Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and non-stationary nature of continual learning data, especially in the online setting, amplify the discrepancy between training and testing in BN and hinder the performance of older tasks. In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting. This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect. Extensive experiments on different continual learning algorithms and online scenarios show that CN is a direct replacement for BN and can provide substantial performance improvements. Our implementation is available at https://github.com/phquang/Continual-Normalization.
Keywords
Continual learning, Generalisation, Learning data, Learning methods, Nonstationary, Normalisation, Normalization effect, On-line setting, Performance, Training and testing
Discipline
Databases and Information Systems
Research Areas
Information Systems and Management
Publication
Proceedings of the 10th International Conference on Learning Representations, ICLR 2022, April 25-29
First Page
1
Last Page
20
Publisher
ICLR
City or Country
Wisconsin, USA Wisconsin, USA
Citation
PHAM, Quang; LIU, Chenghao; and HOI, Steven.
Continual normalization: Rethinking batch normalization for online continual learning. (2024). Proceedings of the 10th International Conference on Learning Representations, ICLR 2022, April 25-29. 1-20.
Available at: https://ink.library.smu.edu.sg/sis_research/8238
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
