Vadim Liventsev

Vadim Liventsev

Auto-generated profile, based on publicly available information

Vadim Liventsev

Department of Mathematics and Computer Science, Eindhoven University of Technology, Eindhoven, NETHERLANDS

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
12 July 2023
Verified
Conference Paper
Open access
VLVadim Liventsev
Vadim Liventsev
Anastasiia GrishinaAnastasiia Grishina
Anastasiia Grishina
AHAki Härmä
Aki Härmä
LMLeon Moonen
Leon Moonen
Current approaches to program synthesis with Large Language Models (LLMs) exhibit a "near miss syndrome": they tend to generate programs that semantically resemble the correct answer (as measured by text similarity metrics or human evaluation), but achieve a low or even zero accuracy as measured by unit tests due to small imperfections, such as the wrong input or output format. This calls for an approach known as Synthesize, Execute, Debug (SED), whereby a draft of the solution is generated first, followed by a program repair phase addressing the failed tests. To effectively apply this approach to instruction-driven LLMs, one needs to determine which prompts perform best as instructions for LLMs, as well as strike a balance between repairing unsuccessful programs and replacing them with newly generated ones. We explore these trade-offs empirically, comparing replace-focused, repair-focused, and hybrid debug strategies, as well as different template-based and model-based prompt-generation techniques. We use OpenAI Codex as the LLM and Program Synthesis Benchmark 2 as a database of problem descriptions and tests for evaluation. The resulting framework outperforms both conventional usage of Codex without the repair phase and traditional genetic programming approaches.

1 November 2019
Verified
Conference Paper
Closed access
ASArtem Shelmanov
Artem Shelmanov
VLVadim Liventsev
Vadim Liventsev
DKDanil Kireev
Danil Kireev
NKNikita Khromov
Nikita Khromov
APAlexander Panchenko
Alexander Panchenko
DDDmitry V. Dylov
Dmitry V. Dylov
Active learning is a technique that helps to minimize the annotation budget required for the creation of a labeled dataset while maximizing the performance of a model trained on this dataset. It has been shown that active learning can be successfully applied to sequence tagging tasks of text processing in conjunction with deep learning models even when a limited amount of labeled data is available. Recent advances in transfer learning methods for natural language processing based on deep pre-trained models such as ELMo and BERT offer a much better ability to generalize on small annotated datasets compared to their shallow counterparts. The combination of deep pre-trained models and active learning leads to a powerful approach to dealing with annotation scarcity. In this work, we investigate the potential of this approach on clinical and biomedical data. The experimental evaluation shows that the combination of active learning and deep pre-trained models outperforms the standard methods of active learning. We also suggest a modification to a standard uncertainty sampling strategy and empirically show that it could be beneficial for annotation of very skewed datasets. Finally, we propose an annotation tool empowered with active learning and deep pre-trained models that could be used for entity annotation directly from Jupyter IDE.

29 March 2025 in ACM Transactions on Evolutionary Learning and Optimization
Verified
Journal Article
Open access
Anastasiia GrishinaAnastasiia Grishina
Anastasiia Grishina
VLVadim Liventsev
Vadim Liventsev
AHAki Härmä
Aki Härmä
LMLeon Moonen
Leon Moonen
Program synthesis with Large Language Models (LLMs) suffers from a “near-miss syndrome”: The generated code closely resembles a correct solution but fails unit tests due to minor errors. We address this with a multi-agent framework called Synthesize, Execute, Instruct, Debug, and Repair (SEIDR). Effectively applying SEIDR to instruction-tuned LLMs requires determining (a) optimal prompts for LLMs, (b) what ranking algorithm selects the best programs in debugging rounds, and (c) balancing the repair of unsuccessful programs with the generation of new ones. We empirically explore these tradeoffs by comparing replace-focused, repair-focused, and hybrid debug strategies. We also evaluate lexicase and tournament selection to rank candidates in each generation. On Program Synthesis Benchmark 2 (PSB2), our framework outperforms both conventional use of OpenAI Codex without a repair phase and traditional genetic programming approaches. SEIDR outperforms the use of an LLM alone, solving 18 problems in C++ and 20 in Python on PSB2 at least once across experiments. To assess generalizability, we employ GPT-3.5 and Llama 3 on the PSB2 and HumanEval-X benchmarks. Although SEIDR with these models does not surpass current state-of-the-art methods on the Python benchmarks, the results on HumanEval-C++ are promising. SEIDR with Llama 3-8B achieves an average pass@100 of 84.2%. Across all SEIDR runs, 163 of 164 problems are solved at least once with GPT-3.5 in HumanEval-C++, and 162 of 164 with the smaller Llama 3-8B. We conclude that SEIDR effectively overcomes the near-miss syndrome in program synthesis with LLMs.


13 February 2025 in IEEE Access
Verified
Journal Article
Open access
VLVadim Liventsev
Vadim Liventsev
SBSander de Bruin
Sander de Bruin
AHAki Härmä
Aki Härmä
MPMilan Petković
Milan Petković
Autoencoder models of source code are an emerging alternative to autoregressive large language models with important benefits for genetic improvement of software. We hypothesize that encoder-decoder architectures are suboptimal for source code because they ignore the grammatical structure that can be derived with an Abstract Syntax Tree parser. We propose a structured Variational Auto-Encoder based on TreeLSTM that operates directly on the AST. We train it along with a baseline sequence VAE on a dataset of competitive programming submissions We find the structured model to perform better in most tests, with some notable exceptions. These findings suggest structured autoencoder models could enable more effective generation and manipulation of source code for tasks like automated bug fixing and generative programming.