Alejandro Velasco

Alejandro Velasco

Auto-generated profile, based on publicly available information

Alejandro Velasco

William & Mary, Williamsburg, USA

Research Focus & Metrics

Academic Career Length

12Years (2015-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
4 July 2025 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
GCGiuseppe Crupi
Giuseppe Crupi
RTRosalia Tufano
Rosalia Tufano
AVAlejandro Velasco
Alejandro Velasco
AMAntonio Mastropaolo
Antonio Mastropaolo
DPDenys Poshyvanyk
Denys Poshyvanyk
GBGabriele Bavota
Gabriele Bavota
Large Language Models (LLMs) have been recently exploited as judges for complex natural language processing tasks, such as Q&A (Question & Answer). The basic idea is to delegate to an LLM the assessment of the “quality” of the output provided by an automated technique (often another LLM) for tasks for which: (i) quantitative metrics would only tell part of the story, and; (ii) a large-scale human-based evaluation would be too expensive. LLMs-as-a-judge, if proven effective for a specific task, can also unlock new possibilities for automation, with several LLMs proposing a solution for a given instance of the task (e.g., an answer to a question) and others judging and deciding what is the best output to show the user. We study the effectiveness of LLMs-as-a-judge for two code-related tasks, namely code generation and code summarization. The rationale for choosing these tasks is two-fold. First, quantitative metrics are usually not enough for the assessment of code summarizers/generators. For example, it is well documented that metrics such as BLEU are quite weak proxies for the quality of the generated summaries. Second, even state-of-the-art techniques still struggle with handling complex instances of these tasks (e.g., summarizing a quite long / complex function), making them good candidates for benefiting from more advanced solutions envisioning collaboration among LLMs. For code generation, we check whether eight LLMs are able to judge the correctness of 1,405 Java methods and 1,281 Python functions generated by the same LLMs or implemented by humans. For code summarization, we compare the judgment of five LLMs to those provided by nine humans for ç1.2k summaries, related to both Java and Python functions. Our findings show that GPT-4-turbo is the best LLM in terms of judging capabilities for both tasks, with “smaller” LLMs featuring tens of billions parameters not being able to cope with judging tasks. However, even the best-performing LLM frequently misjudges the correctness of the code and summary quality.

21 March 2024 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
DPDavid Nader Palacio
David Nader Palacio
AVAlejandro Velasco
Alejandro Velasco
NCNathan Cooper
Nathan Cooper
ARAlvaro Rodriguez
Alvaro Rodriguez
KMKevin Moran
Kevin Moran
DPDenys Poshyvanyk
Denys Poshyvanyk
Neural Language Models of Code, or Neural Code Models (NCMs), are rapidly progressing from research prototypes to commercial developer tools. As such, understanding the capabilities and limitations of such models is becoming critical. However, the abilities of these models are typically measured using automated metrics that often only reveal a portion of their real-world performance. While, in general, the performance of NCMs appears promising, currently much is unknown about how such models arrive at decisions. To this end, this paper introduces docode , a post hoc interpretability method specific to NCMs that is capable of explaining model predictions. docode is based upon causal inference to enable programming language-oriented explanations. While the theoretical underpinnings of docode are extensible to exploring different model properties, we provide a concrete instantiation that aims to mitigate the impact of spurious correlations by grounding explanations of model behavior in properties of programming languages. To demonstrate the practical benefit of docode , we illustrate the insights that our framework can provide by performing a case study on two popular deep learning architectures and ten NCMs. The results of this case study illustrate that our studied NCMs are sensitive to changes in code syntax. All our NCMs, except for the BERT-like model, statistically learn to predict tokens related to blocks of code (e.g., brackets, parenthesis, semicolon) with less confounding bias as compared to other programming language constructs. These insights demonstrate the potential of docode as a useful method to detect and facilitate the elimination of confounding bias in NCMs.

1 November 2017
Verified
Conference Paper
Closed access
DDDavid Delgado
David Delgado
AVAlejandro Velasco
Alejandro Velasco
JAJairo Aponte
Jairo Aponte
AMAndrian Marcus
Andrian Marcus
This paper presents the evolution of a project-based course in Software Engineering for undergraduate students at the Universidad Nacional de Colombia. We describe and explain the changes we have done over six semesters. In addition, we investigate the effects of the changes on the students' grades and their project activities, by analyzing the software project repositories and the student feedback. Most of the changes had positive and expected results, while some had unexpected consequences. We distill a set of lessons regarding the class evolution, which will guide the future improvement of the course and which could be useful for other educators developing a similar course.

14 April 2024
Verified
Conference Paper
Open access
AVAlejandro Velasco
Alejandro Velasco
DPDavid N. Palacio
David N. Palacio
DRDaniel Rodriguez-Cardenas
Daniel Rodriguez-Cardenas
DPDenys Poshyvanyk
Denys Poshyvanyk

1 January 2015
Verified
Book Chapter
Closed access
PDF file
VSVladimir Subotic
Vladimir Subotic
ACArturo Campos
Arturo Campos
AVAlejandro Velasco
Alejandro Velasco
EAEduard Ayguadé
Eduard Ayguadé
JLJesus Labarta
Jesus Labarta
MVMateo Valero
Mateo Valero