Soumyadeep Roy

Soumyadeep Roy

Auto-generated profile, based on publicly available information

Soumyadeep Roy

Indian Institute of Technology Kharagpur

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
10 July 2024
Verified
Conference Paper
Open access
SRSoumyadeep Roy
Soumyadeep Roy
AKAparup Khatua
Aparup Khatua
FGFatemeh Ghoochani
Fatemeh Ghoochani
UHUwe Hadler
Uwe Hadler
WNWolfgang Nejdl
Wolfgang Nejdl
NGNiloy Ganguly
Niloy Ganguly
GPT-4 demonstrates high accuracy in medical QA tasks, leading with an accuracy of 86.70%, followed by Med-PaLM 2 at 86.50%. However, around 14% of errors remain. Additionally, current works use GPT-4 to only predict the correct option without providing any explanation and thus do not provide any insight into the thinking process and reasoning used by GPT-4 or other LLMs. Therefore, we introduce a new domain-specific error taxonomy derived from collaboration with medical students. Our GPT-4 USMLE Error (G4UE) dataset comprises 4153 GPT-4 correct responses and 919 incorrect responses to the United States Medical Licensing Examination (USMLE) respectively. These responses are quite long (258 words on average), containing detailed explanations from GPT-4 justifying the selected option. We then launch a large-scale annotation study using the Potato annotation platform and recruit 44 medical experts through Prolific, a well-known crowdsourcing platform. We annotated 300 out of these 919 incorrect data points at a granular level for different classes and created a multi-label span to identify the reasons behind the error. In our annotated dataset, a substantial portion of GPT-4's incorrect responses is categorized as a "Reasonable response by GPT-4," by annotators. This sheds light on the challenge of discerning explanations that may lead to incorrect options, even among trained medical professionals. We also provide medical concepts and medical semantic predications extracted using the SemRep tool for every data point. We believe that it will aid in evaluating the ability of LLMs to answer complex medical questions. We make the resources available at https://github.com/roysoumya/usmle-gpt4-error-taxonomy.

26 October 2021
Verified
Conference Paper
Open access
SRSoumyadeep Roy
Soumyadeep Roy
SCSUDIP CHAKRABORTY
SUDIP CHAKRABORTY
AMAishik Mandal
Aishik Mandal
GBGunjan Balde
Gunjan Balde
PSPrakhar Sharma
Prakhar Sharma
NGNiloy Ganguly
Niloy Ganguly
Online medical forums have become a predominant platform for answering health-related information needs of consumers. However, with a significant rise in the number of queries and the limited availability of experts, it is necessary to automatically classify medical queries based on a consumer's intention, so that these questions may be directed to the right set of medical experts. Here, we develop a novel medical knowledge-aware BERT-based model (MedBERT) that explicitly gives more weightage to medical concept-bearing words, and utilize domain-specific side information obtained from a popular medical knowledge base. We also contribute a multi-label dataset for the Medical Forum Question Classification (MFQC) task. MedBERT achieves state-of-the-art performance on two benchmark datasets and performs very well in low resource settings.

1 August 2024
Verified
Conference Paper
Open access
PDF file
GBGunjan Balde
Gunjan Balde
SRSoumyadeep Roy
Soumyadeep Roy
MMMainack Mondal
Mainack Mondal
NGNiloy Ganguly
Niloy Ganguly
Low-rank Adaptation (LoRA) has demonstrated remarkable capabilities for task specific fine-tuning. However, in scenarios that involve multiple tasks, training a separate LoRA model for each one results in considerable inefficiency in terms of storage and inference. Moreover, existing parameter generation methods fail to capture the correlations among these tasks, making multi-task LoRA parameter generation challenging. To address these limitations, we propose In-Context Meta LoRA (ICM-LoRA), a novel approach that efficiently achieves task-specific customization of large language models (LLMs). Specifically, we use training data from all tasks to train a tailored generator, Conditional Variational Autoencoder (CVAE). CVAE takes task descriptions as inputs and produces task-aware LoRA weights as outputs. These LoRA weights are then merged with LLMs to create task-specialized models without the need for additional fine-tuning. Furthermore, we utilize in-context meta-learning for knowledge enhancement and task mapping, to capture the relationship between tasks and parameter distributions. As a result, our method achieves more accurate LoRA parameter generation for diverse tasks using CVAE. ICM-LoRA enables more accurate LoRA parameter reconstruction than current parameter reconstruction methods and is useful for implementing task-specific enhancements of LoRA parameters. At the same time, our method occupies 283MB, only 1% storage compared with the original LoRA. The code is available at https://github.com/YihuaJerry/ICM-LoRA.

1 January 2024
Verified
Conference Paper
Open access
PDF file
GBGunjan Balde
Gunjan Balde
SRSoumyadeep Roy
Soumyadeep Roy
MMMainack Mondal
Mainack Mondal
NGNiloy Ganguly
Niloy Ganguly
Gunjan Balde, Soumyadeep Roy, Mainack Mondal, Niloy Ganguly. Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.

10 March 2025
Verified
Conference Paper
Open access
NSNicolas Steiner
Nicolas Steiner
ZLZiteng Li
Ziteng Li
OVOmid Vosoughi
Omid Vosoughi
JSJohanna Schrader
Johanna Schrader
SRSoumyadeep Roy
Soumyadeep Roy
MTMing Tang
Ming Tang
This study presents a comprehensive benchmarking of three state-of-the-art single-cell foundation models scGPT, Geneformer, and scFoundation, on cell-type classification tasks. We evaluate the models on three datasets: myeloid, human pancreas, and multiple sclerosis, examining both standard fine-tuning and few-shot learning scenarios. Our work reveals that scFoundation consistently achieves the best performance while Geneformer performs poorly, yielding results sometimes even worse than those of the baseline models. Additionally, we demonstrate that a good foundation model can generalize well even when fine-tuned with out-of-distribution data, a capability that the baseline models lack. Our work highlights the potential of foundation models for addressing challenging biomedical questions, particularly in contexts where models are trained on one population but deployed on another.

1 January 2025
Verified
Conference Paper
Open access
PDF file
GBGunjan Balde
Gunjan Balde
SRSoumyadeep Roy
Soumyadeep Roy
MMMainack Mondal
Mainack Mondal
NGNiloy Ganguly
Niloy Ganguly
Gunjan Balde, Soumyadeep Roy, Mainack Mondal, Niloy Ganguly. Findings of the Association for Computational Linguistics: ACL 2025. 2025.