Reuben Tan

Reuben Tan

Auto-generated profile, based on publicly available information

Reuben Tan

Boston University

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
29 October 2022
Verified
Book Chapter
Closed access
PDF file
RTReuben Tan
Reuben Tan
BPBryan A. Plummer
Bryan A. Plummer
Kate SaenkoKate Saenko
Kate Saenko
JLJp Lewis
Jp Lewis
ASAvneesh Sud
Avneesh Sud
TLThomas W Leung
Thomas W Leung

21 October 2020 in arXiv
Verified
Preprint
Open access
PDF file
RTReuben Tan
Reuben Tan
BPBryan A. Plummer
Bryan A. Plummer
Kate SaenkoKate Saenko
Kate Saenko
Large-scale dissemination of disinformation online intended to mislead or deceive the general population is a major societal problem. Rapid progression in image, video, and natural language generative models has only exacerbated this situation and intensified our need for an effective defense mechanism. While existing approaches have been proposed to defend against neural fake news, they are generally constrained to the very limited setting where articles only have text and metadata such as the title and authors. In this paper, we introduce the more realistic and challenging task of defending against machine-generated news that also includes images and captions. To identify the possible weaknesses that adversaries can exploit, we create a NeuralNews dataset composed of 4 different types of generated articles as well as conduct a series of human user study experiments based on this dataset. In addition to the valuable insights gleaned from our user study experiments, we provide a relatively effective approach based on detecting visual-semantic inconsistencies, which will serve as an effective first line of defense and a useful reference for future work in defending against machine-generated disinformation.

2 December 2021 in arXiv
Verified
Preprint
Open access
PDF file
RTReuben Tan
Reuben Tan
BPBryan A. Plummer
Bryan A. Plummer
Kate SaenkoKate Saenko
Kate Saenko
HJHailin Jin
Hailin Jin
BRBryan Russell
Bryan Russell
We introduce the task of spatially localizing narrated interactions in videos. Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations. To achieve this goal, we propose a multilayer cross-modal attention network that enables effective optimization of a contrastive loss during training. We introduce a divided strategy that alternates between computing inter- and intra-modal attention across the visual and natural language modalities, which allows effective training via directly contrasting the two modalities' representations. We demonstrate the effectiveness of our approach by self-training on the HowTo100M instructional video dataset and evaluating on a newly collected dataset of localized described interactions in the YouCook2 dataset. We show that our approach outperforms alternative baselines, including shallow co-attention and full cross-modal attention. We also apply our approach to grounding phrases in images with weak supervision on Flickr30K and show that stacking multiple attention layers is effective and, when combined with a word-to-region loss, achieves state of the art on recall-at-one and pointing hand accuracies.

2 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
KDKatherine Deng
Katherine Deng
ARArijit Ray
Arijit Ray
RTReuben Tan
Reuben Tan
SGSaadia Gabriel
Saadia Gabriel
BPBryan A. Plummer
Bryan A. Plummer
Kate SaenkoKate Saenko
Kate Saenko
Existing emotion prediction benchmarks contain coarse emotion labels which do not consider the diversity of emotions that an image and text can elicit in humans due to various reasons. Learning diverse reactions to multimodal content is important as intelligent machines take a central role in generating and delivering content to society. To address this gap, we propose Socratis, a societal reactions benchmark, where each image-caption (IC) pair is annotated with multiple emotions and the reasons for feeling them. Socratis contains 18K free-form reactions for 980 emotions on 2075 image-caption pairs from 5 widely-read news and image-caption (IC) datasets. We benchmark the capability of state-of-the-art multimodal large language models to generate the reasons for feeling an emotion given an IC pair. Based on a preliminary human study, we observe that humans prefer human-written reasons over 2 times more often than machine-generated ones. This shows our task is harder than standard generation tasks because it starkly contrasts recent findings where humans cannot tell apart machine vs human-written news articles, for instance. We further see that current captioning metrics based on large vision-language models also fail to correlate with human preferences. We hope that these findings and our benchmark will inspire further research on training emotionally aware models.

14 August 2022 in arXiv
Verified
Preprint
Open access
PDF file
RTReuben Tan
Reuben Tan
BPBryan A. Plummer
Bryan A. Plummer
Kate SaenkoKate Saenko
Kate Saenko
JLJp Lewis
Jp Lewis
ASAvneesh Sud
Avneesh Sud
TLThomas W Leung
Thomas W Leung
Recent self-supervised approaches have used large-scale image-text datasets to learn powerful representations that transfer to many tasks without finetuning. These methods often assume that there is one-to-one correspondence between its images and their (short) captions. However, many tasks require reasoning about multiple images and long text narratives, such as describing news articles with visual summaries. Thus, we explore a novel setting where the goal is to learn a self-supervised visual-language representation that is robust to varying text length and the number of images. In addition, unlike prior work which assumed captions have a literal relation to the image, we assume images only contain loose illustrative correspondence with the text. To explore this problem, we introduce a large-scale multimodal dataset containing over 31M articles, 22M images and 1M videos. We show that state-of-the-art image-text alignment methods are not robust to longer narratives with multiple images. Finally, we introduce an intuitive baseline that outperforms these methods on zero-shot image-set retrieval by 10% on the GoodNews dataset.

22 August 2019 in arXiv
Verified
Preprint
Open access
PDF file
RTReuben Tan
Reuben Tan
MVMariya I. Vasileva
Mariya I. Vasileva
Kate SaenkoKate Saenko
Kate Saenko
BPBryan A. Plummer
Bryan A. Plummer
Many real-world tasks require models to compare images along multiple similarity conditions (e.g. similarity in color, category or shape). Existing methods often reason about these complex similarity relationships by learning condition-aware embeddings. While such embeddings aid models in learning different notions of similarity, they also limit their capability to generalize to unseen categories since they require explicit labels at test time. To address this deficiency, we propose an approach that jointly learns representations for the different similarity conditions and their contributions as a latent variable without explicit supervision. Comprehensive experiments across three datasets, Polyvore-Outfits, Maryland-Polyvore and UT-Zappos50k, demonstrate the effectiveness of our approach: our model outperforms the state-of-the-art methods, even those that are strongly supervised with pre-defined similarity conditions, on fill-in-the-blank, outfit compatibility prediction and triplet prediction tasks. Finally, we show that our model learns different visually-relevant semantic sub-spaces that allow it to generalize well to unseen categories.