The Graphical User Interface (GUI) plays a critical role in the interaction between users and mobile applications (apps), aiming at facilitating the operation process. However, due to the variety of functions and non-standardized design, GUIs might have many accessibility issues, like the size of components being too small or their intervals being narrow. These issues would hinder the operation of low vision users, preventing them from obtaining information accurately and conveniently. Although several technologies and methods have been proposed to address these issues, they are typically confined to issue identification, leaving the resolution in the hands of developers. Moreover, it can be challenging to ensure that the color, size, and interval of the fixed GUIs are appropriately compared to the original ones. In this work, we propose a novel approach named AccessFixer ( Access ibility Issues Fix ing Method), which utilizes the Relational-Graph Convolutional Neural Network (R-GCN) to simultaneously fix three kinds of accessibility issues, including small sizes, narrow intervals, and low color contrast in GUIs. With AccessFixer, the fixed GUIs would have a consistent color palette, uniform intervals, and adequate size changes achieved through coordinated adjustments to the attributes of related components. Our experiments demonstrate the effectiveness and usefulness of AccessFixer in fixing GUI accessibility issues. After fixing 30 real-world apps, our approach solves an average of 81.2% of their accessibility issues. Compared with the baseline tool that can only fix size-related issues, AccessFixer not only fixes both the interval and color contrast of components, but also ensures that no new issues arise in the fixed results. Also, we apply AccessFixer to 10 open-source apps by submitting the fixed results with pull requests (PRs) on GitHub. The results demonstrate that developers approve of our submitted fixed GUIs, with 8 PRs being merged or under fixing. A user study examines that low vision users host a positive attitude toward the GUIs fixed by our method.

Malware detection is a critical issue in software engineering as it directly threatens user information security. Existing approaches often focus on individual modality (either source code or binary code) for the detection, but it ignores to effectively exploit the complementary information between them. This limits the detection performance, especially in complex and evasive malware scenarios. In this paper, we take Android applications written in Java as objects, and provide a novel fine-grained multimodal fusion method with large pre-trained models to combine the features from source and binary codes for the malware detection. For the source code modality, we employ the graphical user interface (GUI) as a framework to segment the source code into snippets, and use a pre-trained programming language model to extract feature representations. For the binary code modality, we convert binary code into grayscale images and fine-tune a pre-trained vision model to extract features indirectly. We then implement cross-modal attention and devise a contrastive loss to align features across modalities, supplementing this with supervised classification loss to refine the multimodal fusion process specifically for malware detection. Our experiments, conducted using the Data-MD and Data-MC benchmarks, demonstrate that our approach achieves a precision of 0.977 and a recall of 0.984 in detecting malware. This underscores the advantages of using large pre-trained models for feature representation and the fusion of information across different modalities for effective malware detection.

UI animations, such as card movement and menu slide in/out, provide appealing user experience and enhance the usability of mobile applications. In the process of UI animation implementation, it is difficult for developers to identify suitable APIs for the animation to be implemented from a large number of APIs. Fortunately, the huge app market contains millions of apps, and they can provide valuable data resources for solving this problem. By summarizing the API usage for the same or similar animations in apps, reusable knowledge can be mined for the API recommendation. In this paper, we propose a novel method Animation2API, which mines the knowledge about APIs from existing apps and recommends APIs for UI animations. Different from existing text-based API recommendation approaches, Animation2API takes the UI animation in GIF/video format as query input. Firstly, we construct a database containing mappings between UI animations and APIs by analyzing a broad set of apps. Then, we build a UI animation feature extractor, which can be used to gain temporal-spatial feature vectors of UI animations. By comparing the temporal-spatial feature vectors between UI animations, we identify animations that are similar to the query animation from the database. Finally, we summarize the APIs used for implementing these animations and recommend a list of APIs for developers. The empirical evaluation results show that our method can achieve 82.66% Success rate and outperform the baseline Guru by 230.77% and 184.95% in terms of Precision and Recall when considering twenty APIs. In the user study, we take the scenarios of using web search and ChatGPT to implement animations as baselines, and the results show that participants can complete animations faster (14.54%) after using Animation2API. Furthermore, participants’ positive feedbacks on the questionnaire indicate the usefulness of Animation2API.