Publication Type

Journal Article

Version

publishedVersion

Publication Date

3-2023

Abstract

User interfaces (UI) of desktop, web, and mobile applications involve a hierarchy of objects (e.g., applications, screens, view class, and other types of design objects) with multimodal (e.g., textual and visual) and positional (e.g., spatial location, sequence order, and hierarchy level) attributes. We can therefore represent a set of application UIs as a heterogeneous network with multimodal and positional attributes. Such a network not only represents how users understand the visual layout of UIs but also influences how users would interact with applications through these UIs. To model the UI semantics well for different UI annotation, search, and evaluation tasks, this article proposes the novel Heterogeneous Attention-based Multimodal Positional (HAMP) graph neural network model. HAMP combines graph neural networks with the scaled dot-product attention used in transformers to learn the embeddings of heterogeneous nodes and associated multimodal and positional attributes in a unified manner. HAMP is evaluated with classification and regression tasks conducted on three distinct real-world datasets. Our experiments demonstrate that HAMP significantly outperforms other state-of-the-art models on such tasks. To further provide interpretations of the contribution of heterogeneous network information for understanding the relationships between the UI structure and prediction tasks, we propose Adaptive HAMP (AHAMP), which adaptively learns the importance of different edgeslinking different UI objects. Our experiments demonstrate AHAMP’ssuperior performance over HAMP on a number of tasks, and its ability to provide interpretations of the contribution of multimodal and positional attributes, as well as heterogeneous network information to different tasks.

Keywords

Computing methodologies, Neural networks, Human-centered computing, User interface management systems, Computing methodologies, Artificial intelligence, Information systems, Multimedia information systems

Discipline

Artificial Intelligence and Robotics | Databases and Information Systems | OS and Networks

Research Areas

Data Science and Engineering

Publication

ACM Transactions on Interactive Intelligent Systems

Volume

13

Issue

3

First Page

1

Last Page

31

ISSN

2160-6455

Identifier

10.1145/3578522

Publisher

Association for Computing Machinery (ACM)

Additional URL

https://doi.org/10.1145/3578522

Download

Share

COinS