Publication Type
Journal Article
Version
acceptedVersion
Publication Date
9-2024
Abstract
Training neural networks (NNs) using combinatorial optimization solvers has gained attention in recent years. In low-data settings, the use of state-of-the-art mixed integer linear programming solvers, for instance, has the potential to exactly train an NN while avoiding computing-intensive training and hyperparameter tuning and simultaneously training and sparsifying the network. We study the case of few-bit discrete-valued neural networks, both binarized neural networks (BNNs) whose values are restricted to ±1 and integer-valued neural networks (INNs) whose values lie in the range {−P,…,P}. Few-bit NNs receive increasing recognition because of their lightweight architecture and ability to run on low-power devices: for example, being implemented using Boolean operations. This paper proposes new methods to improve the training of BNNs and INNs. Our contribution is a multiobjective ensemble approach based on training a single NN for each possible pair of classes and applying a majority voting scheme to predict the final output. Our approach results in the training of robust sparsified networks whose output is not affected by small perturbations on the input and whose number of active weights is as small as possible. We empirically compare this BeMi approach with the current state of the art in solver-based NN training and with traditional gradient-based training, focusing on BNN learning in few-shot contexts. We compare the benefits and drawbacks of INNs versus BNNs, bringing new light to the distribution of weights over the {−P,…,P} interval. Finally, we compare multiobjective versus single-objective training of INNs, showing that robustness and network simplicity can be acquired simultaneously, thus obtaining better test performances. Although the previous state-of-the-art approaches achieve an average accuracy of 51.1%51.1% on the Modified National Institute of Standards and Technology data set, the BeMi ensemble approach achieves an average accuracy of 68.4% when trained with 10 images per class and 81.8% when trained with 40 images per class while having up to 75.3% NN links removed.
Keywords
Binarized neural networks, Integer neural networks, Mixed-integer linear programming, Structured ensemble, Few-shot learning, Sparsity, Multi-objective optimisation
Discipline
Artificial Intelligence and Robotics | Computer Sciences
Research Areas
Intelligent Systems and Optimization
Publication
INFORMS Journal on Computing
First Page
1
Last Page
21
ISSN
1091-9856
Identifier
10.1287/ijoc.2023.0281
Publisher
INFORMS
Citation
BERNARDELLI, Ambrogio Maria; GUALANDI, Stefano; MILANESI, Simone; LAU, Hoong Chuin; and NEIL, Yorke-Smith.
Multiobjective linear ensembles for robust and sparse training of few-bit neural networks. (2024). INFORMS Journal on Computing. 1-21.
Available at: https://ink.library.smu.edu.sg/sis_research/9955
Copyright Owner and License
Authors
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Additional URL
https://doi.org/10.1287/ijoc.2023.0281
