15 March 2024 in ACM Transactions on Software Engineering and Methodology
Code summarization aims to generate short functional descriptions for source code to facilitate code comprehension. While Information Retrieval (IR) approaches that leverage similar code snippets and corresponding summaries have led the early research, Deep Learning (DL) approaches that use neural models to capture statistical properties between code and summaries are now mainstream. Although some preliminary studies suggest that IR approaches are more effective in some cases, it is currently unclear how effective the existing approaches can be in general, where and why IR/DL approaches perform better, and whether the integration of IR and DL can achieve better performance. Consequently, there is an urgent need for a comprehensive study of the IR and DL code summarization approaches to provide guidance for future development in this area. This article presents the first large-scale empirical study of 18 IR, DL, and hybrid code summarization approaches on five benchmark datasets. We extensively compare different types of approaches using automatic metrics, we conduct quantitative and qualitative analyses of where and why IR and DL approaches perform better, respectively, and we also study hybrid approaches for assessing the effectiveness of integrating IR and DL. The study shows that the performance of IR approaches should not be underestimated, that while DL models perform better in predicting tokens from method signatures and capturing structural similarities in code, simple IR approaches tend to perform better in the presence of code with high similarity or long reference summaries, and that existing hybrid approaches do not perform as well as individual approaches in their respective areas of strength. Based on our findings, we discuss future research directions for better code summarization.
17 June 2026 in ACM Transactions on Software Engineering and Methodology
Deep Learning (DL) has already played a significant role in numerous fields, making it crucial to ensure the stability of both training and inference in DL systems. The computation of DL models can be viewed as the execution of a series of DL operators, which are essential components that perform the core numerical computations. Therefore, the numerical stability of these operators is critical for the overall quality of these systems. Operators with numerical instabilities can cause degraded model performance or even numerical exceptions that disrupt the training process. A common practice to improve numerical stability is increasing the precision of specific floating-point computations. However, determining the appropriate data type for internal computations, referred to as precision tuning, remains a significant challenge. This article presents the first systematic study on precision tuning in the implementation of DL operators. We introduce OpScanner , capable of extracting precision casting operations critical to numerical stability of operators within existing DL frameworks. Using the captured operations, we conduct a comprehensive categorization and comparative analysis between two popular DL frameworks. Our findings can help developers understand precision tuning in operators and implement new operators. Based on the results of our comparative analysis, we conduct precision testing experiments on a set of operators from both frameworks to examine the impact of precision tuning. The test results show that tested TensorFlow operators produce significantly more floating-point errors compared to their PyTorch counterparts due to a lack of precision tuning. One of the most inaccurate operators has been confirmed to have numerical issues by the TensorFlow community. Additionally, our study examines the effect of precision tuning on weight updates during training. We found that larger errors, resulting from a lack of tuning, lead to greater weight deviations during training.
15 April 2026 in ACM Transactions on Software Engineering and Methodology
As third-party libraries evolve, some API usages become deprecated, requiring corresponding updates in client code. Although library maintainers often provide natural language (NL) descriptions through changelogs or warnings to assist migration, updating deprecated API usage remains labor-intensive and error-prone. In this paper, we present the first empirical study that investigates whether LLMs can effectively update deprecated API usages based on NL descriptions. To this end, we construct a comprehensive benchmark of real-world API deprecation updates. Specifically, we convert NL descriptions into static rules, and identify matching updates from candidate code modifications retrieved via keyword search. The benchmark covers diverse deprecation granularities and varying levels of updating difficulty. Based on this benchmark, we evaluate 12 representative LLMs from four model families with different types and sizes. Our findings reveal that models from different families exhibit varying effectiveness in updating deprecated API usages, and both model size and code-specific tuning influence performance. By examining results across different levels of difficulty, we find that LLMs perform well on simple cases that only require element replacement, whereas complex cases involving additional code adaptation remain challenging. According to our manual analysis, non-trivial parameter changes and required data transformations account for the majority of cases that all LLMs fail to handle effectively. We further assess whether providing additional information such as surrounding code context and API documentation can help improve the effectiveness. Overall, our study highlights that automatically updating deprecated API usages from NL descriptions remains a challenging problem, and provides practical implications for future improvements in LLM-based updating. Additionally, our automatic data collection process has the additional benefit of detecting inconsistencies between deprecation documentation and code. In this process, we identified six documentation bugs, all of which have been confirmed and fixed.