Arijit Ray

Arijit Ray

Auto-generated profile, based on publicly available information

Arijit Ray

Boston University

Research Focus & Metrics

Academic Career Length

12Years (2015-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
12 November 2021 in Applied AI Letters
Verified
Journal Article
Open access
PDF file
ARArijit Ray
Arijit Ray
MCMichael Cogswell
Michael Cogswell
XLXiao Lin
Xiao Lin
KAKamran Alipour
Kamran Alipour
ADAjay Divakaran
Ajay Divakaran
GBGiedrius Burachas
Giedrius Burachas
Attention maps, a popular heatmap‐based explanation method for Visual Question Answering, are supposed to help users understand the model by highlighting portions of the image/question used by the model to infer answers. However, we see that users are often misled by current attention map visualizations that point to relevant regions despite the model producing an incorrect answer. Hence, we propose Error Maps that clarify the error by highlighting image regions where the model is prone to err. Error maps can indicate when a correctly attended region may be processed incorrectly leading to an incorrect answer, and hence, improve users' understanding of those cases. To evaluate our new explanations, we further introduce a metric that simulates users' interpretation of explanations to evaluate their potential helpfulness to understand model correctness. We finally conduct user studies to see that our new explanations help users understand model correctness better than baselines by an expected 30% and that our proxy helpfulness metrics correlate strongly () with how well users can predict model correctness.

25 October 2021 in arXiv
Verified
Preprint
Open access
PDF file
ARArijit Ray
Arijit Ray
MCMichael Cogswell
Michael Cogswell
XLXiao Lin
Xiao Lin
KAKamran Alipour
Kamran Alipour
ADAjay Divakaran
Ajay Divakaran
GBGiedrius Burachas
Giedrius Burachas
Attention maps, a popular heatmap-based explanation method for Visual Question Answering (VQA), are supposed to help users understand the model by highlighting portions of the image/question used by the model to infer answers. However, we see that users are often misled by current attention map visualizations that point to relevant regions despite the model producing an incorrect answer. Hence, we propose Error Maps that clarify the error by highlighting image regions where the model is prone to err. Error maps can indicate when a correctly attended region may be processed incorrectly leading to an incorrect answer, and hence, improve users' understanding of those cases. To evaluate our new explanations, we further introduce a metric that simulates users' interpretation of explanations to evaluate their potential helpfulness to understand model correctness. We finally conduct user studies to see that our new explanations help users understand model correctness better than baselines by an expected 30\% and that our proxy helpfulness metrics correlate strongly ($\rho>0.97$) with how well users can predict model correctness.

30 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
DBDina Bashkirova
Dina Bashkirova
ARArijit Ray
Arijit Ray
RMRupayan Mallick
Rupayan Mallick
SBSarah Adel Bargal
Sarah Adel Bargal
JZJianming Zhang
Jianming Zhang
Kate SaenkoKate Saenko
Kate Saenko
Professional artists, photographers, and other visual content creators use object relighting to establish their photo's desired effect. Unfortunately, manual tools that allow relighting have a steep learning curve and are difficult to master. Although generative editing methods now enable some forms of image editing, relighting is still beyond today's capabilities; existing methods struggle to keep other aspects of the image -- colors, shapes, and textures -- consistent after the edit. We propose Lasagna, a method that enables intuitive text-guided relighting control. Lasagna learns a lighting prior by using score distillation sampling to distill the prior of a diffusion model, which has been finetuned on synthetic relighting data. To train Lasagna, we curate a new synthetic dataset ReLiT, which contains 3D object assets re-lit from multiple light source locations. Despite training on synthetic images, quantitative results show that Lasagna relights real-world images while preserving other aspects of the input image, outperforming state-of-the-art text-guided image editing methods. Lasagna enables realistic and controlled results on natural images and digital art pieces and is preferred by humans over other methods in over 91% of cases. Finally, we demonstrate the versatility of our learning objective by extending it to allow colorization, another form of image editing.

2 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
KDKatherine Deng
Katherine Deng
ARArijit Ray
Arijit Ray
RTReuben Tan
Reuben Tan
SGSaadia Gabriel
Saadia Gabriel
BPBryan A. Plummer
Bryan A. Plummer
Kate SaenkoKate Saenko
Kate Saenko
Existing emotion prediction benchmarks contain coarse emotion labels which do not consider the diversity of emotions that an image and text can elicit in humans due to various reasons. Learning diverse reactions to multimodal content is important as intelligent machines take a central role in generating and delivering content to society. To address this gap, we propose Socratis, a societal reactions benchmark, where each image-caption (IC) pair is annotated with multiple emotions and the reasons for feeling them. Socratis contains 18K free-form reactions for 980 emotions on 2075 image-caption pairs from 5 widely-read news and image-caption (IC) datasets. We benchmark the capability of state-of-the-art multimodal large language models to generate the reasons for feeling an emotion given an IC pair. Based on a preliminary human study, we observe that humans prefer human-written reasons over 2 times more often than machine-generated ones. This shows our task is harder than standard generation tasks because it starkly contrasts recent findings where humans cannot tell apart machine vs human-written news articles, for instance. We further see that current captioning metrics based on large vision-language models also fail to correlate with human preferences. We hope that these findings and our benchmark will inspire further research on training emotionally aware models.

18 June 2015 in Applied Artificial Intelligence
Verified
Journal Article
Closed access
ARArijit Ray
Arijit Ray
KGKishan Prudhvi Guddanti
Kishan Prudhvi Guddanti
CNChellammal Nallaperumal
Chellammal Nallaperumal
This article discusses an intelligent driving system (IDS) that uses a Polynomial Regression Network (PRN) and a Gaussian Anomaly Detection System (GADS). The PRN models the relation between the motor input and vehicle movement, thus, inherently taking nonlinear and noisy environmental factors into consideration. The anomaly detection system detects probable system output or sensor failure using a Gaussian pattern-match algorithm for taking proper corrective action. The time of computation for training the PRN is shown to be as low as 0.3 seconds and thus, the system can be used for instantaneous training in any environment at high speeds. Such intelligent driving systems will be useful for electric race-car designs, electric consumer vehicles, and robotic vehicles for stability on adverse road or track conditions.

3 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
ARArijit Ray
Arijit Ray
FRFilip Radenovic
Filip Radenovic
ADAbhimanyu Dubey
Abhimanyu Dubey
BPBryan A. Plummer
Bryan A. Plummer
RKRanjay Krishna
Ranjay Krishna
Kate SaenkoKate Saenko
Kate Saenko
Compositional reasoning is a hallmark of human visual intelligence. Yet, despite the size of large vision-language models, they struggle to represent simple compositions by combining objects with their attributes. To measure this lack of compositional capability, we design Cola, a text-to-image retrieval benchmark to Compose Objects Localized with Attributes. To solve Cola, a model must retrieve images with the correct configuration of attributes and objects and avoid choosing a distractor image with the same objects and attributes but in the wrong configuration. Cola contains about 1.2k composed queries of 168 objects and 197 attributes on around 30K images. Our human evaluation finds that Cola is 83.33% accurate, similar to contemporary compositionality benchmarks. Using Cola as a testbed, we explore empirical modeling designs to adapt pre-trained vision-language models to reason compositionally. We explore 6 adaptation strategies on 2 seminal vision-language models, using compositionality-centric test benchmarks - Cola and CREPE. We find the optimal adaptation strategy is to train a multi-modal attention layer that jointly attends over the frozen pre-trained image and language features. Surprisingly, training multimodal layers on CLIP performs better than tuning a larger FLAVA model with already pre-trained multimodal layers. Furthermore, our adaptation strategy improves CLIP and FLAVA to comparable levels, suggesting that training multimodal layers using contrastive attribute-object data is key, as opposed to using them pre-trained. Lastly, we show that Cola is harder than a closely related contemporary benchmark, CREPE, since simpler fine-tuning strategies without multimodal layers suffice on CREPE but not on Cola. However, we still see a significant gap between our best adaptation and human accuracy, suggesting considerable room for further research.