Large Language Models (LLM) have rapidly affirmed in the latest years as a means to support or substitute human actors in a variety of tasks. LLM agents can generate valid software models, because of their inherent ability in evaluating textual requirements provided to them in the form of prompts. The goal of this work is to evaluate the capability of LLM agents to correctly generate UML class diagrams in activities of Requirements Modeling in the field of Software Engineering. Our aim is to evaluate LLMs in an educational setting, i.e., understanding how valuable are the results of LLMs when compared to results made by human actors, and how valuable can LLM be to generate sample solutions to provide to students. For that purpose, we collected 20 exercises from a diverse set of web sources and compared the models generated by a human and an LLM solver in terms of syntactic, semantic, pragmatic correctness, and distance from a provided reference solution. Our results show that the solutions generated by an LLM solver typically present a significantly higher number of errors in terms of semantic quality and textual difference against the provided reference solution, while no significant difference is found in syntactic and pragmatic quality. We can therefore conclude that, with a limited amount of errors mostly related to the textual content of the solution, UML diagrams generated by LLM agents have the same level of understandability as those generated by humans, and exhibit the same frequency in violating rules of UML Class Diagrams.

Gamification is an established practice in Software Engineering to increase effectiveness and engagement in many practices. This manuscript provides a characterization of the application of gamification to the Software Testing area. Such practice in fact reportedly suffers from low engagement by both personnel in industrial contexts and learners in educational contexts. Our goal is to identify the application areas and utilized gamified techniques and mechanics, the provided benefits and drawbacks, as well as the open challenges in the field. To this purpose, we conducted a Multivocal Literature Review to identify white and grey literature sources addressing gamified software testing. We analyzed 73 contributions and summarized the most common gamified mechanics, concepts, tools, and domains where they are mostly applied. We conclude that gamification in software testing is mostly applied to the test creation phase with simple white-box unit or mutation testing tools and is mostly used to foster good behaviors by promoting the testers’ accomplishment. Key research areas and main challenges in the field are: careful design of tailored gamified mechanics for specific testing techniques; the need for technological improvements to enable crowdsourcing, cooperation, and concurrency; the necessity for empirical and large-scale evaluation of the benefits delivered by gamification mechanics.

Android applications do not seem to be tested as thoroughly as desktop ones. In particular, graphical user interface (GUI) testing appears generally limited. Like web-based applications, mobile apps suffer from GUI test fragility, i.e., GUI test classes failing or needing updates due to even minor modifications in the GUI or in the application under test. The objective of our study is to estimate the adoption of GUI testing frameworks among Android open-source applications, the quantity of modifications needed to keep test classes up to date, and their amount due to GUI test fragility. We introduce a set of 21 metrics to measure the adoption of testing tools and the evolution of test classes and test methods, and to estimate the fragility of test suites. We computed our metrics for six GUI testing frameworks, none of which achieved a significant adoption among Android projects hosted on GitHub. When present, GUI test methods associated with the considered tools are modified often, and a relevant portion (70% on average) of those modifications is induced by GUI-related fragilities. On average, for the projects considered, more than 7% of the total modified lines of code between consecutive releases belong to test classes developed with the analyzed testing frameworks. The measured percentage was higher on average than the one required by other generic test code, based on the JUnit testing framework. Fragility of GUI tests constitutes a relevant concern, probably an obstacle for developers to adopt test automation. This first evaluation of the fragility of Android scripted GUI testing can constitute a benchmark for developers and testers leveraging the analyzed test tools and the basis for the definition of a taxonomy of fragility causes and guidelines to mitigate the issue.

Several techniques exist for test automation in the mobile domain, from script-based techniques to automated test generation based on GUI models. Most techniques, however, fall short in being adopted extensively by practitioners because of the very high cost for the definition (and maintainence) of test cases. Random testing tools decouple the test generation from the specific app (or version) to test, however they create unreproducible tests often with a low bug-finding ability. The objective of this work is to build a testing framework for Android apps, capable of allowing the user to write very simple test scripts without knowing details about the application implementation details and appearance. The main requirement for those tests is to generate adaptive tests that can be executed on a great number of apps without manual re-adaptations. We developed a testing framework that: (1) models the application by means of state graphs; (2) by using machine learning algorithms, classifies the type of application under test and each activity (i.e., screen) of the application; (3) allows to create general test cases, specific to application and activity categories, that are then automatically adapted to the AUT without effort from the tester. We created two separated classifiers (for application and activity type) that outperform available approaches in literature. On top of them, the developed testing framework was able to perfectly adapt high-level test cases to 28 of 32 tested applications. Such high-level tests allow a reduction of LOCs of the test suite of around 90%. Our findings prove that a machine learning-based approach can be derived for the creation of high-level, adaptive test cases for different typologies of Android applications and activities. Our prototype framework is modular in nature, and allows expansions by means of addition of new commands to be executed on the classified apps and activities.