Tobias Höllerer

Tobias Höllerer

Auto-generated profile, based on publicly available information

Tobias Höllerer

University of California Santa Barbara

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 November 2021 in arXiv
Verified
Preprint
Open access
PDF file
YDYi Ding
Yi Ding
ARAlex Rich
Alex Rich
MWMason Wang
Mason Wang
NSNoah Stier
Noah Stier
MTMatthew Turk
Matthew Turk
THTobias Höllerer
Tobias Höllerer
Multimodal classification is a core task in human-centric machine learning. We observe that information is highly complementary across modalities, thus unimodal information can be drastically sparsified prior to multimodal fusion without loss of accuracy. To this end, we present Sparse Fusion Transformers (SFT), a novel multimodal fusion method for transformers that performs comparably to existing state-of-the-art methods while having greatly reduced memory footprint and computation cost. Key to our idea is a sparse-pooling block that reduces unimodal token sets prior to cross-modality modeling. Evaluations are conducted on multiple multimodal benchmark datasets for a wide range of classification tasks. State-of-the-art performance is obtained on multiple benchmarks under similar experiment conditions, while reporting up to six-fold reduction in computational cost and memory requirements. Extensive ablation studies showcase our benefits of combining sparsification and multimodal learning over naive approaches. This paves the way for enabling multimodal learning on low-resource devices.