Mulong Xie

Mulong Xie

Auto-generated profile, based on publicly available information

Mulong Xie

CSIRO’s Data61, Eveleigh, Australia

Research Focus & Metrics

Academic Career Length

5Years (2022-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
7 November 2022
Verified
Conference Paper
Open access
MXMulong Xie
Mulong Xie
ZXZhenchang Xing
Zhenchang Xing
SFSidong Feng
Sidong Feng
XXXiwei XU
Xiwei XU
LZLiming Zhu
Liming Zhu
CCChunyang Chen
Chunyang Chen
Graphical User Interface (GUI) is not merely a collection of individual and unrelated widgets, but rather partitions discrete widgets into groups by various visual cues, thus forming higher-order perceptual units such as tab, menu, card or list. The ability to automatically segment a GUI into perceptual groups of widgets constitutes a fundamental component of visual intelligence to automate GUI design, implementation and automation tasks. Although humans can partition a GUI into meaningful perceptual groups of widgets in a highly reliable way, perceptual grouping is still an open challenge for computational approaches. Existing methods rely on ad-hoc heuristics or supervised machine learning that is dependent on specific GUI implementations and runtime information. Research in psychology and biological vision has formulated a set of principles (i.e., Gestalt theory of perception) that describe how humans group elements in visual scenes based on visual cues like connectivity, similarity, proximity and continuity. These principles are domain-independent and have been widely adopted by practitioners to structure content on GUIs to improve aesthetic pleasantness and usability. Inspired by these principles, we present a novel unsupervised image-based method for inferring perceptual groups of GUI widgets. Our method requires only GUI pixel images, is independent of GUI implementation, and does not require any training data. The evaluation on a dataset of 1,091 GUIs collected from 772 mobile apps and 20 UI design mockups shows that our method significantly outperforms the state-of-the-art ad-hoc heuristics-based baseline. Our perceptual grouping method creates opportunities for improving UI-related software engineering tasks.

12 April 2024
Verified
Conference Paper
Open access
YSYanqi Su
Yanqi Su
DLDianshu Liao
Dianshu Liao
ZXZhenchang Xing
Zhenchang Xing
QHQing Huang
Qing Huang
MXMulong Xie
Mulong Xie
XXXiwei XU
Xiwei XU

25 February 2026 in ACM Transactions on Computer-Human Interaction
Verified
Journal Article
Open access
MYMingyue Yuan
Mingyue Yuan
JCJieshan Chen
Jieshan Chen
YHYongquan Hu
Yongquan Hu
SFSidong Feng
Sidong Feng
MXMulong Xie
Mulong Xie
AQAaron Quigley
Aaron Quigley
In automated UI design generation, a key challenge is the lack of support for iterative processes, as most systems focus solely on end-to-end output. This stems from limited capabilities in interpreting design intent and a lack of transparency for refining intermediate results. To better understand these challenges, we conducted a formative study that identified concrete and actionable requirements for supporting iterative design with Generative Tools. Guided by these findings, we propose PrototypeFlow, a human-centered system for automated UI generation that leverages multi-modal inputs and models. PrototypeFlow takes natural language descriptions and layout preferences as input to generate the high-fidelity UI design. At its core is a theme design module that clarifies implicit design intent through prompt enhancement and orchestrates sub-modules for component-level generation. Designers retain full control over inputs, intermediate results, and final prototypes, enabling flexible and targeted refinement by steering generation and directly editing outputs. Our experiments and user studies confirmed the effectiveness and usefulness of our proposed PrototypeFlow.