Param Biyani

Param Biyani

Auto-generated profile, based on publicly available information

Param Biyani

Microsoft, Bengaluru, India

Research Focus & Metrics

Representative Publications

View All Publications
10 July 2024
Verified
Conference Paper
Open access
PBParam Biyani
Param Biyani
YBYasharth Bajpai
Yasharth Bajpai
ARArjun Radhakrishna
Arjun Radhakrishna
GSGustavo Soares
Gustavo Soares
SGSumit Gulwani
Sumit Gulwani
Evaluating conversational assistants, such as GitHub Copilot Chat, poses a significant challenge for tool builders in the domain of Software Engineering. These assistants rely on language models and chat-based user experiences, rendering their evaluation with respect to the quality of the Human-AI conversations complicated. Existing general-purpose metrics for measuring conversational quality found in literature are inadequate for appraising domain-specific dialogues due to their lack of contextual sensitivity. In this paper, we present RUBICON, a technique for evaluating domain-specific Human-AI conversations. RUBICON leverages large language models to generate candidate rubrics for assessing conversation quality and employs a selection process to choose the subset of rubrics based on their performance in scoring conversations. In our experiments, RUBICON effectively learns to differentiate conversation quality, achieving higher accuracy and yield rates than existing baselines.