Publication Type
Conference Proceeding Article
Version
acceptedVersion
Publication Date
11-2025
Abstract
Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages. To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-theart (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generation of interaction compared with full page, (2) prone to ten types of failure, (3) poor performance on visually subtle interactions, and (4) insufficient undestanding on interaction when limited to single-modality visual descriptions. To address these limitations, we propose four enhancement strategies: Interactive Element Highlighting, Failureaware Prompting (FAP), Visual Saliency Enhancement, and Visual-Textual Descriptions Combination, all aiming at improving MLLMs’ performance on the Interaction-toCode task. The Interaction2Code benchmark and code are available in https://github. com/WebPAI/Interaction2Code.
Discipline
Software Engineering
Research Areas
Intelligent Systems and Optimization
Areas of Excellence
Digital transformation
Publication
Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul,November 16-20
First Page
1
Last Page
21
City or Country
Seoul, Korea
Citation
XIAO, Jingyu; WAN, Yuxuan; HUO, Yintong; WANG, Zixin; XU, Xinyi; WANG, Wenxuan; XU, Zhiyao; WANG, Yuhang; and LYU, Michael R..
Interaction2Code: Benchmarking MLLM-based interactive webpage code generation from interactive prototyping. (2025). Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul,November 16-20. 1-21.
Available at: https://ink.library.smu.edu.sg/sis_research/10733
Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.