Publication Type
Conference Proceeding Article
Version
publishedVersion
Publication Date
6-2022
Abstract
Convolutional Neural Networks (CNNs) have been re-garded as the go-to models for visual recognition. More re-cently, convolution-free networks, based on multi-head self-attention (MSA) or multi-layer perceptrons (MLPs), become more and more popular. Nevertheless, it is not trivial when utilizing these newly-minted networks for video recognition due to the large variations and complexities in video data. In this paper, we present MLP-3D networks, a novel MLP-like 3D architecture for video recognition. Specifically, the architecture consists of MLP-3D blocks, where each block contains one MLP applied across tokens (i.e., token-mixing MLP) and one MLP applied independently to each token (i.e., channel MLP). By deriving the novel grouped time mixing (GTM) operations, we equip the basic token-mixing MLP with the ability of temporal modeling. GTM divides the input tokens into several temporal groups and linearly maps the tokens in each group with the shared projection matrix. Furthermore, we devise several variants of GTM with different grouping strategies, and compose each vari-ant in different blocks of MLP-3D network by greedy ar-chitecture search. Without the dependence on convolutions or attention mechanisms, our MLP-3D networks achieves 68.5%/81.4% top-1 accuracy on Something-Something V2 and Kinetics-400 datasets, respectively. Despite with fewer computations, the results are comparable to state-of-the-art widely-used 3D CNNs and video transformers.
Discipline
Artificial Intelligence and Robotics | Graphics and Human Computer Interfaces
Research Areas
Intelligent Systems and Optimization
Publication
Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, June 18-24
First Page
3052
Last Page
3062
ISBN
9781665469463
Identifier
10.1109/CVPR52688.2022.00307
Publisher
Computer Vision Foundation
City or Country
New Orleans, USA
Citation
QIU, Zhaofan; YAO, Ting; NGO, Chong-wah; and MEI, Tao.
MLP-3D: A MLP-like 3D architecture with grouped time mixing. (2022). Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, June 18-24. 3052-3062.
Available at: https://ink.library.smu.edu.sg/sis_research/7505
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.
Additional URL
http://doi.org/10.1109/CVPR52688.2022.00307
Included in
Artificial Intelligence and Robotics Commons, Graphics and Human Computer Interfaces Commons