Transformer-based language models often achieve strong results on mathematical reasoning benchmarks while remaining fragile on basic numerical understanding and arithmetic operations. A central limitation is that numbers are processed as symbolic tokens whose embeddings do not explicitly encode numerical value, leading to systematic errors. We introduce a value-aware numerical representation that augments standard tokenized inputs with a dedicated prefix token whose embedding is explicitly conditioned on the underlying numerical value. This mechanism injects magnitude information directly into the model's input space while remaining compatible with existing tokenizers and decoder-only Transformer architectures. Evaluation on arithmetic tasks shows that the proposed approach outperforms baselines across numerical formats, tasks, and operand lengths. These results indicate that explicitly encoding numerical value is an effective and efficient way to improve fundamental numerical robustness in language models.
Automated generation of multiple-choice questions (MCQs) represents a viable strategy for scalable assessment and instructional support, but current systems exhibit persistent limitations in distractor quality, reasoning validity, and deployment accessibility. Building on our earlier short paper, YMCQ: Reasoning-Enhanced MCQ Generation, this journal article extends that work by providing a more detailed presentation and explanation of the MCQ generation and multi-stage filtering method introduced there. The method integrates reasoning-based answer generation with misconception-aware distractors, using open-source LLMs fine-tuned on publicly available corpora augmented with synthetic explanations for correct answers and distractors. These augmentations enable the generation of MCQs that are structurally valid, semantically coherent, and pedagogically aligned. The extended presentation further details the method and its implementation in QuizTools, a publicly available platform that supports automated MCQ generation from source materials and enables human-in-the-loop refinement. The pairwise performance evaluation of the generated questions was reported in the short paper; this journal article additionally includes an individual-question quality evaluation, analysis of user edits, and a usability study of QuizTools. The models, datasets, and source code, including over 300,000 quiz items with annotated explanations, are publicly available to facilitate reproducibility and further research in educational NLP.
Generating multiple text sequences and refining them through feedback is essential for improving the quality of outputs in many NLP tasks. While Large Language Models can leverage iterative feedback during inference, smaller models often lack this capability due to limited capacity and the absence of suitable training paradigms. In this paper, we propose a novel Feedback-Aware Inference approach that enables iterative sequence generation with integration of feedback signals. Our method allows models to generate multiple sequences, incorporate feedback from previous iterations, and refine outputs accordingly. This approach dynamically adjusts to different quality metrics, making it adaptable to various contexts and objectives. We evaluate our approach on two distinct tasks: Answer Selection for Question Generation and Keyword Generation, arguing for its generalizability and effectiveness. Results show that our method outperforms strong baselines, maintaining high performance across iterations and achieving superior results even with smaller, open-source models.
The broad adoption of Large Language Models (LLMs) has increased the need for human-curated datasets that serve as evaluation benchmarks. This need is particularly pronounced for non-English languages and for tasks that are inherently subjective and require multiple human perspectives. One such example is the development of benchmarks designed to assess the cultural awareness of LLMs. Statistics and data science courses offer a potential setting for developing such benchmarks while teaching students to apply LLM evaluation techniques using statistical inference. This paper presents a pilot project in which students in a statistics course within a data science engineering program created culturally diverse multiple-choice questions, generated answers using LLMs, and applied statistical methods to assess model accuracy. Student feedback indicated the project was engaging and useful for learning, while also highlighting a notable reliance on LLMs, particularly for interpreting statistical results. The resulting dataset comprises 1355 multiple-choice questions across 18 categories, including language, social media, and politics. After filtering valid items, the dataset was used to evaluate both closed- and open-source LLMs. Results show that the Gemini (closed-source) and Qwen (open-source) model families achieved the best performance, with improvements linked to model size, reasoning capabilities, and access to search tools. The best closed-source model achieved an accuracy of 97.66%, whereas the best open-source model achieved an accuracy of 79.07%. Qualitative analyses of errors in the filtering procedure and model reasoning process point to possible explanations into the challenges LLMs face when handling culturally specific content. Furthermore, results support a cultural injection hypothesis, whereby cultural knowledge is embedded during pretraining and accessed through instruction tuning. Through this work, we aim to demonstrate how statistics and data science courses can provide productive contexts for developing open-source benchmarks for non-English languages while also enriching students’ learning experiences. The dataset is publicly available.