Publication Type

Conference Proceeding Article

Version

publishedVersion

Publication Date

4-2021

Abstract

Rich multi-modal information - text, code, images, categorical and numerical data - co-exist in the user interface (UI) design of mobile applications. UI designs are composed of UI entities supporting different functions which together enable the application. To support effective search and recommendation applications over mobile UIs, we need to be able to learn UI representations that integrate latent semantics. In this paper, we propose a novel unsupervised model - Multi-modal Attention-based Attributed Network Embedding (MAAN) model. MAAN is designed to capture both multi-modal and structural network information. Based on the encoder-decoder framework, MAAN aims to learn UI representations that allow UI design reconstruction. The generated embedding can be applied to a variety of tasks: predicting UI elements associated with UI screens, inferring missing UI screen and element attributes, predicting UI user ratings, and retrieving UIs. Extensive experiments, including user evaluations, conducted on two datasets from RICO, a rich real-world mobile UI repository, demonstrates that MAAN out-performs other state-of-the-art models.

Keywords

Network embedding, mobile application user interface, unsupervised retrieval, multi-modal

Discipline

Databases and Information Systems | OS and Networks

Research Areas

Data Science and Engineering

Publication

Proceedings of the 26th Annual Conference on Intelligent User Interfaces, College Station, Texas, 2021 April 14-17

First Page

366

Last Page

376

ISBN

9781450380171

Identifier

10.1145/3397481.3450693

Publisher

Association for Computing Machinery (ACM)

City or Country

United States of America

Embargo Period

4-6-2022

Copyright Owner and License

Authors

Download

Share

COinS