Arnab Dhua

Arnab Dhua

Auto-generated profile, based on publicly available information

Arnab Dhua

Amazon

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 August 2024
Verified
Conference Paper
Open access
XZXinliang Zhu
Xinliang Zhu
SHSheng-Wei Huang
Sheng-Wei Huang
HDHan Ding
Han Ding
JYJinyu Yang
Jinyu Yang
KCKelvin Chen
Kelvin Chen
ADArnab Dhua
Arnab Dhua
Image to image matching has been well studied in the computer vision community. Previous studies mainly focus on training a deep metric learning model matching visual patterns between the query image and gallery images. In this study, we show that pure image-to- image matching suffers from false positives caused by matching to local visual patterns. To alleviate this issue, we propose to leverage recent advances in vision-language pretraining research. Specifically, we introduce additional image-text alignment losses into deep metric learning, which serve as constraints to the image-to-image matching loss. With additional alignments between the text (e.g., product title) and image pairs, the model can learn concepts from both modalities explicitly, which avoids matching low-level visual features. We progressively develop two variants, a 3-tower and a 4-tower model, where the latter takes one more short text query input. Through extensive experiments, we show that this change leads to a substantial improvement to the image to image matching problem. We further leveraged this model for multimodal search, which takes both image and reformulation text queries to improve search quality. Both offline and online experiments show strong improvements on the main metrics. Specifically, we see 4.95% relative improvement on image matching click through rate with the 3-tower model and 1.13% further improvement from the 4-tower model.

10 July 2024
Verified
Conference Paper
Open access
XZXinliang Zhu
Xinliang Zhu
ADArnab Dhua
Arnab Dhua
DGDouglas Gray
Douglas Gray
IYI. Zeki Yalniz
I. Zeki Yalniz
TYTan Yu
Tan Yu
BPBryan Plummer
Bryan Plummer
Multimodal data is available in many applications like e-commerce production listings, social media posts and short videos. However, existing algorithms dealing with those types of data still focus on uni-modal representation learning by vision-language alignment and cross-modal retrieval. In this workshop, we target to bring a new retrieval problem where both queries and documents are multimodal. With the popularity of vision language modeling, large language models (LLMs), retrieval augmented generation (RAG), and multimodal LLM, we see a lot of new opportunities for multimodal representation and retrieval tasks. This event will be a comprehensive half-day workshop focusing on the subject of multimodal representation and retrieval. The agenda includes keynote speeches, oral presentations, and an interactive panel discussion.

16 September 2026
Verified
Book Chapter
Closed access
PDF file
VSVishwas Sathish
Vishwas Sathish
VRViresh Ranjan
Viresh Ranjan
XZXinliang Zhu
Xinliang Zhu
ADArnab Dhua
Arnab Dhua
DGDouglas Gray
Douglas Gray