Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

8-2024

Abstract

This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, EmpathyEar supports user inputs in any combination of text, sound, and vision, and produces multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized speeches. A series of emotion-aware instruction-tuning is performed for comprehensive emotional understanding and generation capabilities. In this way, EmpathyEar provides users with responses that achieve a deeper emotional resonance, closely emulating human-like empathy. The system paves the way for the next emotional intelligence, for which we open-source the code for public access.

Keywords

Multimodal chatbot, Empathetic response generation, Large language model, LLMs, Digital avatars

Discipline

Artificial Intelligence and Robotics | Graphics and Human Computer Interfaces

Research Areas

Data Science and Engineering; Intelligent Systems and Optimization

Areas of Excellence

Digital transformation

Publication

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) : Bangkok, Thailand, August 11-16

Volume

3

First Page

61

Last Page

71

Identifier

10.18653/v1/2024.acl-demos.7

Publisher

Association for Computational Linguistics

City or Country

Bangkok, Thailand

Additional URL

https://doi.org/10.18653/v1/2024.acl-demos.7

Download

Share

COinS