Mohammad Abdollahi

Mohammad Abdollahi

Auto-generated profile, based on publicly available information

Mohammad Abdollahi

York University, Canada

Research Focus & Metrics

Academic Career Length

1Year (2026-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
6 March 2026 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
MAMohammad Abdollahi
Mohammad Abdollahi
RZRuixin Zhang
Ruixin Zhang
NHNima Shiri Harzevili
Nima Shiri Harzevili
JSJiho Shin
Jiho Shin
Song WangSong Wang
Song Wang
HHHadi Hemmati
Hadi Hemmati
Since the release of early Large Language Models (LLMs) such as GPT-2 and GPT-3 around 2020, rapid advancements in LLM capabilities have significantly impacted the field of code intelligence, enabling automation across a wide range of coding tasks, including code generation, program repair, debugging, and software testing etc. As the LLMs are widely used in coding tasks, benchmarking their capabilities in a meticulous and meaningful way is essential. In this work, we survey 142 related papers published between January 2020 and June 2025, covering 156 unique benchmarks and 32 different coding tasks, to provide a comprehensive review of coding benchmarks to evaluate LLMs, exploring their characteristics, strengths, limitations, dataset construction, task coverage, alignment with real world challenges, and performance metrics. Our analysis indicates that Python was the most prevalent programming language, featured in 77% of the datasets, and GitHub was the most common data source, used in 46% of the cases. Also, there is an increasing trend in the publication of new benchmark datasets for coding intelligence over the past three years. Prominent publication venues are machine learning conferences such as NeurIPS, ICLR, and ICML, while comparatively fewer datasets have appeared in software engineering venues such as ICSE, ASE, and MSR. In addition, most of the benchmarks focused on the code generation task, with 86 datasets dedicated to this area. We identify key gaps and limitations in existing benchmarks, including bias in datasets, lack of standardized evaluation protocols, and insufficient coverage of real-world challenges. Finally, we present a vision for the future of LLM benchmark creation, advocating for more robust, adaptable, and better alignment with practical code intelligence demands that ensure the reliability and practicality of LLMs in coding workflows.

2 June 2026 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
MAMohammad Abdollahi
Mohammad Abdollahi
KTKhandaker Rifah Tasnia
Khandaker Rifah Tasnia
SSSoumit Kanti Saha
Soumit Kanti Saha
JYJinqiu Yang
Jinqiu Yang
Song WangSong Wang
Song Wang
HHHadi Hemmati
Hadi Hemmati
Understanding a program’s runtime reasoning behavior, i.e., how intermediate states and control flows lead to final execution results, is essential for reliable code generation, debugging, and automated reasoning. While large language models (LLMs) have demonstrated impressive capabilities in predicting program outputs, most prior studies have focused on output accuracy and performance comparisons, treating reasoning as a black box. As a result, models often struggle with complex control logic or subtle semantic nuances, and little is known about the structure, quality, or failure modes of their reasoning traces. Recent advances in reasoning LLMs, which explicitly generate intermediate reasoning steps before producing final answers, raise the expectation that models could not only produce correct code but also explain their reasoning in ways consistent with program semantics. This ability is particularly critical for tasks involving runtime behavior, where reasoning must capture both control flow and state changes. However, the lack of systematic evaluation of reasoning traces leaves open fundamental questions about how reasoning errors arise and how they influence execution correctness. To address this gap, we conduct the first empirical study on reasoning runtime behavior inference with reasoning LLMs to systematically uncover and characterize the errors in their reasoning traces. For our research, we curate an augmented benchmark constructed from three widely used datasets (i.e., HumanEval+, LiveCodeBench, and REPOEXEC), containing 477 code snippets. For each code snippet, we experiment with three different types of inputs, i.e., regular, edge, and invalid inputs. A total of 12 input values are selected per snippet, each paired with its ground-truth execution outcome. We evaluate four state-of-the-art reasoning LLMs, i.e., DeepSeek-R1, OpenAI o4-mini, Gemini 2.5 Flash, and Claude 4 Sonnet, and our experiment results show that the examined reasoning LLMs can achieve high accuracies ranging from 62% to 98% across all three input types. Additionally, we analyzed the errors in reasoning traces produced by LLMs and developed a comprehensive taxonomy that groups nine specific categories of inference errors under four high-level parent categories for code output inference tasks. We further investigate whether a tool-augmented reasoning approach can enhance the reasoning ability of LLMs. Using failures in the Computation Errors category as a case study, our experiments show that this method successfully corrects 69% of such errors, underscoring the promise of tool augmentation in improving LLM reasoning.