Aninditha Ramesh

Aninditha Ramesh

Auto-generated profile, based on publicly available information

Aninditha Ramesh

Carnegie Mellon University, Pittsburgh, USA

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
5 December 2024
Verified
Conference Paper
Open access
ARAninditha Ramesh
Aninditha Ramesh
AAArav Agarwal
Arav Agarwal
JDJacob Arthur Doughty
Jacob Arthur Doughty
KRKetan Ramaneti
Ketan Ramaneti
JSJaromir Savelka
Jaromir Savelka
MSMajd Sakr
Majd Sakr
There has been a growing interest in utilizing large language models (LLMs) for numerous educational applications. Recent studies have focused on the use of LLMs for generating various educational artifacts for programming education, such as programming exercises, model solutions, or multiple-choice questions (MCQs). The ability to efficiently and reliably assess the quality of such artifacts, both automatically and human generated, has become of paramount importance. Hence, there is a pressing need to develop and make available robust benchmarks. In this paper, we investigate an example use case of assessing the quality of programming MCQs. To that end, we carefully curated a data set of 192 MCQs annotated with quality scores based on a rubric that evaluates crucial aspects such as, e.g., their clarity, the presence of a single correct answer, and the quality of distractors. The results show that the task presents a considerable challenge even to the state-of-the-art LLMs and, hence, further research is needed. To further such research efforts in this important area we release the dataset as well as the extensible evaluation pipeline to the public.