Publication Type
Conference Proceeding Article
Version
publishedVersion
Publication Date
12-2020
Abstract
Natural human interactions for Mixed Reality Applications are overwhelmingly multimodal: humans communicate intent and instructions via a combination of visual, aural and gestural cues. However, supporting low-latency and accurate comprehension of such multimodal instructions (MMI), on resource-constrained wearable devices, remains an open challenge, especially as the state-of-the-art comprehension techniques for each individual modality increasingly utilize complex Deep Neural Network models. We demonstrate the possibility of overcoming the core limitation of latency–vs.–accuracy tradeoff by exploiting cross-modal dependencies–i.e., by compensating for the inferior performance of one model with an increased accuracy of more complex model of a different modality. We present a sensor fusion architecture that performs MMI comprehension in a quasi-synchronous fashion, by fusing visual, speech and gestural input. The architecture is reconfigurable and supports dynamic modification of the complexity of the data processing pipeline for each individual modality in response to contextual changes. Using a representative “classroom” context and a set of four common interaction primitives, we then demonstrate how the choices between low and high complexity models for each individual modality are coupled. In particular, we show that (a) a judicious combination of low and high complexity models across modalities can offer a dramatic 3-fold decrease in comprehension latency together with an increase ∼10-15% in accuracy, and (b) the right collective choice of models is context dependent, with the performance of some model combinations being significantly more sensitive to changes in scene context or choice of interaction.
Keywords
Sensor fusion, Mixed reality, Multimodal interactions
Discipline
Databases and Information Systems
Research Areas
Data Science and Engineering
Publication
Proceedings of the 17th IEEE International Conference on Mobile Ad Hoc and Smart Systems, Virtual, Delhi, India, 2020 December 10-13
First Page
309
Last Page
317
ISBN
9781728198668
Identifier
10.1109/MASS50613.2020.00046
Publisher
IEEE
City or Country
New Jersey
Citation
KANATTA GAMAGE, Ramesh Darshana Rathnayake; DE SILVA, Ashen; PUWAKDANDAWA, Dasun; MEEGAHAPOLA, Lakmal; MISRA, Archan; and PERERA, Indika.
Jointly optimizing sensing pipelines for multimodal mixed reality interaction. (2020). Proceedings of the 17th IEEE International Conference on Mobile Ad Hoc and Smart Systems, Virtual, Delhi, India, 2020 December 10-13. 309-317.
Available at: https://ink.library.smu.edu.sg/sis_research/6780
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.