EEG-video emotion-based summarization: Learning with EEG auxiliary signals

Author

Wai-Cheong L. LEW
Di WANG
Kai-Keng ANG
Joo-Hwee LIM
Chai QUEK
Ah-hwee TAN, Singapore Management UniversityFollow

Publication Type

Journal Article

Version

publishedVersion

Publication Date

9-2022

Abstract

Video summarization is the process of selecting a subset of informative keyframes to expedite storytelling with limited loss of information. In this paper, we propose an EEG-Video Emotion-based Summarization (EVES) model based on a multimodal deep reinforcement learning (DRL) architecture that leverages neural signals to learn visual interestingness to produce quantitatively and qualitatively better video summaries. As such, EVES does not learn from the expensive human annotations but the multimodal signals. Furthermore, to ensure the temporal alignment and minimize the modality gap between the visual and EEG modalities, we introduce a Time Synchronization Module (TSM) that uses an attention mechanism to transform the EEG representations onto the visual representation space. We evaluate the performance of EVES on the TVSum and SumMe datasets. Based on the rank order statistics benchmarks, the experimental results show that EVES outperforms the unsupervised models and narrows the performance gap with supervised models. Furthermore, the human evaluation scores show that EVES receives a higher rating than the state-of-the-art DRL model DR-DSN by 11.4% on the coherency of the content and 7.4% on the emotion-evoking content. Thus, our work demonstrates the potential of EVES in selecting interesting content that is both coherent and emotion-evoking.

Keywords

Video summarization, EEG-video representation, emotion-evoking, multimodality

Discipline

Databases and Information Systems | Graphics and Human Computer Interfaces

Research Areas

Data Science and Engineering

Publication

IEEE Transactions on Affective Computing

Volume

13

First Page

1827

Last Page

1839

ISSN

1949-3045

Identifier

10.1109/TAFFC.2022.3208259

Publisher

IEEE

Citation

LEW, Wai-Cheong L.; WANG, Di; ANG, Kai-Keng; LIM, Joo-Hwee; QUEK, Chai; and TAN, Ah-hwee. EEG-video emotion-based summarization: Learning with EEG auxiliary signals. (2022). IEEE Transactions on Affective Computing. 13, 1827-1839.
Available at: https://ink.library.smu.edu.sg/sis_research/7567

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.

Additional URL

http://doi.org/10.1109/TAFFC.2022.3208259