Software testing is a crucial yet time-consuming aspect of software development. Writing comprehensive unit tests that accurately verify whether a function or an entire program behaves as intended requires considerable effort from developers, particularly when handling numerous edge cases. This study explores how Large Language Models (LLMs) can streamline this process by automatically generating effective unit tests. We evaluate various LLMs on their capability to interpret problem specifications, analyze source code across multiple programming languages, and generate suitable test cases. The effectiveness of these test cases is assessed using the Pass@1 and line coverage metrics. Our findings reveal that LLMs perform significantly better when provided with both the problem description and the corresponding solution code, particularly in the C programming language. Additionally, we observe substantial performance improvements when example test cases are included in the prompt, leading to higher Pass@1 scores and enhanced code coverage, particularly with more advanced LLMs.

As the demand for more complex circuits increases, so does the duration of creating and testing them. The most time-consuming task in circuit development is notoriously the verification process, primarily due to the large number of simulations (hundreds or even thousands) required to ensure that the circuits adhere to the specifications regardless of the operating conditions. In order to decrease the number of required simulations, various verification algorithms have been proposed over the years, but this comes with an additional issue: the thorough validation of the algorithms. As simulations on real circuits are significantly time-consuming, synthetic circuits can offer precious insights into the capabilities of the verification algorithm. In this paper, we propose a benchmark of synthetic circuits that can be used to exhaustively validate pre-silicon (Pre-Si) verification algorithms. The newly created benchmark consists of 900 synthetic circuits (mathematical functions) with input dimensions (variables) ranging from 2 to 10. We design the benchmark to include functions of varying complexities, reflecting real-world circuit expectations. Eventually, we use this benchmark to evaluate a previously proposed state-of-the-art Pre-Si circuit verification algorithm. We show that this algorithm generally obtains relative verification errors below 2% with fewer than 150 simulations if the circuits have less than six to seven operating conditions. In addition, we demonstrate that some of the most complex circuits in the benchmark pose serious problems to the verification algorithm: the worst case is not found even when 200 simulations are used.

Large language models (LLMs) are increasingly used in programming tasks, including code generation, debugging, validation, optimization, test generation, and educational support. Their use as graders of student programming assignments is, however, still not fully understood, especially in object-oriented programming (OOP). In this context, correctness depends not only on the final output of a program but also on class structure, encapsulation, abstraction, and design decisions. This paper introduces a dataset of 271 C++ student solutions to eight OOP examination problems, together with the problem statements and the reference grades produced by the university auto-grader. We evaluate four locally served open-weight models—phi4-mini, gpt-oss:20b, qwen3.6:27b, and qwq:32b—under one endpoint, one prompt, and deterministic decoding, and we report four properties that a deployable grader must have. The first is output-format compliance, which ranges from 88.9% to 98.9% across models: qwq:32b failed to terminate on 30 of 271 submissions even at a doubled generation budget, spending 7.7 MJ, 29.9% of its total energy, to produce nothing. The second is agreement with the reference grades on the 30-point course scale, where qwen3.6:27b performs best (MAE 3.23 points, $r=0.85$) and phi4-mini worst (MAE 10.90, $r=0.17$); all four models grade systematically more strictly than the auto-grader. The third is reproducibility, which fails at two levels. Repeating an identical run, three models returned identical grades at temperature 0 with a fixed seed while gpt-oss:20b varied by up to 21 points on the same submission; restricting the serving daemon to one concurrent request makes it reproducible, locating that failure in request batching rather than in sampling. Grading the same submissions as part of a different set, however, no model is reproducible: agreement with the original run ranges from 13 to 33 of 40 submissions, and a control run with the original client excludes our instrumentation as the cause. The fourth is degeneracy: phi4-mini awarded an identical grade of zero to every submission in three of the eight examinations, 109 submissions in total, so its aggregate agreement statistics describe a model that is not grading. On a subset graded blind by two instructors, each instructor agreed with the auto-grader more closely than the two agreed with each other, so the residual error of the best-performing model is of the same order as ordinary disagreement between human graders. We also report a negative result: a declaration-level reference could be reconstructed for only five of the eight examinations, because the auto-grader merges missing declarations and wrong return values into single checks. Together these results indicate that agreement with a reference is an insufficient basis for choosing a local grading model, and that compliance, determinism, degeneracy, and the cost of failed generations must be measured alongside it.

Software testing is a crucial yet time-consuming aspect of software development. Writing comprehensive unit tests that accurately verify whether a function or an entire program behaves as intended requires considerable effort from developers, especially when handling numerous edge cases. This study explores how Large Language Models (LLMs) can streamline this process by automatically generating effective unit tests. We evaluate various LLMs on their capability to interpret problem specifications, analyze source code across multiple programming languages, and generate suitable test cases. The effectiveness of these test cases is assessed using the Pass@1 and Line Coverage metrics. Our findings reveal that LLMs perform significantly better when provided with both the problem description and the corresponding solution code, particularly in the C programming language. Additionally, we observe substantial performance improvements when example test cases are included in the prompt, leading to higher Pass@1 scores and enhanced code coverage, particularly with more advanced LLMs.