Virtual devices based on device emulation have been widely used in lab research of mobile app testing for their efficiency and low cost. However, it remains controversial to use virtual devices for app testing in industry, given the inherent difficulties of high-fidelity emulation across diverse mobile systems and devices. Hence, mobile app companies still rely on physical device farms or services like AWS Device Farm. This paper presents our effort to analyze, improve, and effectively use virtual devices for large-scale testing of mobile apps like Douyin. Our goal is to understand the fidelity of virtual devices and to explore how to better utilize virtual devices to improve the efficiency and accessibility of mobile app testing in industrial settings. Our study is conducted on a massive commercial testing infrastructure that deploys a physical device farm and its virtualized counterpart. We show that high-fidelity app testing can be achieved by sensible design and implementation of virtual device farms. With that, we find that major discrepancies are no longer caused by commonly believed factors like hardware heterogeneity and system customizations, but due to non-standard, uncoordinated, and occasionally defective vendor-specific services and drivers, as well as defense mechanisms against malicious app behavior. We present effective solutions to address those problems to significantly improve testing fidelity. We also share our experiences of using virtual device farms to substantially improve the efficiency and accessibility of mobile app testing, without compromising safety.
13 April 2024 in ACM Transactions on Sensor Networks
Android system has been widely deployed in energy-constrained IoT devices for many practical applications, such as smart phone, smart home, healthcare, fitness, and beacons. However, Android users oftentimes suffer from app crashes, which directly disrupt user experience and could lead to data loss. Till now, the community have limited understanding of their prevalence, characteristics, and root causes. In this article, we make an in-depth study of the crash events regarding ten very popular apps of different genres, based on fine-grained system-level traces crowd-sourced from 93 million Android devices. We find that app crashes occur prevalently on the various hardware models studied, and better hardware does not seem to essentially relieve the problem. Most importantly, we unravel multi-fold root causes of app crashes, and pinpoint that the most crashes stem from the subtle yet crucial inconsistency between app developers’ supposed memory/process management model and Android’s actual implementations. We design practical approaches to addressing the inconsistency; after large-scale deployment, they reduce 40.4% of the app crashes with negligible system overhead. In addition, we summarize important lessons learned from this study, and have released our measurement code/data to the community.
With recent advances on cellular technologies pushing the boundary of cellular performance, cellular reliability has become a key concern of their adoption and deployment. To fully understand cellular reliability, we work with a major Android phone vendor, Xiaomi, to conduct a long-term (2020-2024) and large-scale (involving 123M users) measurement study in China, with coarse-grained general statistics and fine-grained sampling diagnostics. Our measurement reveals contrasting evolution trends of cellular failures in different stages of the data connection: in the past five years, failures after connection establishment decrease remarkably (by 29%), while failures during connection setup exhibit a sharp increase (by 38%). Our analysis illustrates that the contrast stems from the joint impact of multiple stakeholders, including ISPs’ increasing deployment of 5G base stations, 5G infrastructure upgrade from NSA (Non-Standalone) to SA (Standalone) mode, software defects coming from Android’s adaptation to new cellular technologies, and so forth. Our work provides actionable insights for improving cellular reliability at scale. More importantly, we have built on our insights to develop enhancements that effectively address cellular reliability issues with remarkable real-world impact—our optimizations have reduced 38% cellular connection failures for 5G phones and 31% failure recovery time across all phones.