Publication Type

Journal Article

Version

publishedVersion

Publication Date

3-2025

Abstract

We present a multimodal instruction comprehension framework, called MImIC, that utilizes visual sensing (including LIDAR and 2D RGB sensing) & AI spatial reasoning capabilities to support more seamless and immersive interaction between humans and AI-driven situated assistive agents. MImIC's key new capability is to support disambiguation of a wider set of relative spatial references that users naturally employ while issuing spatially-situated instructions. To support enhanced visual grounding via a combination of both fully-qualified and relative attribute references, MImIC uses (a) a fine-tuned transformer-based language translation DNN to accurately convert natural verbal commands into a structured set of machine understandable constraints (BLEU score=92.5), (b) a set of modules that use RGB+LIDAR sensing data to convert any relative attribute preferences to fully-qualified attribute constraints (median height/width estimation errors

Keywords

Human-Agent Interaction, Multi-Modality Agent Comprehension, Neural Language Processing, Object Retrieval

Discipline

Artificial Intelligence and Robotics | Databases and Information Systems

Research Areas

Data Science and Engineering

Areas of Excellence

Digital transformation

Publication

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Volume

9

Issue

1

First Page

1

Last Page

34

ISSN

2474-9567

Identifier

10.1145/3712268

Publisher

Association for Computing Machinery (ACM)

Additional URL

https://doi.org/10.1145/3712268

Download

Share

COinS