5 June 2023 in IEEE Transactions on Knowledge and Data Engineering
With the explosive growth of multimedia contents, multimedia retrieval is facing unprecedented challenges on both storage cost and retrieval speed. Hashing technique can project the high-dimensional data into compact binary hash codes. With it, the most time-consuming semantic similarity computation during the multimedia retrieval process can be significantly accelerated with fast Hamming distance computation, and meanwhile the storage cost can be reduced greatly by the binary embedding. In the light of this, multi-modal hashing has recently received considerable attention to support large-scale multimedia retrieval. Different from uni-modal hashing, the multi-modal hashing focuses on modeling the multi-modal semantics and further preserving them into binary hash codes with hash learning. In this paper, we first systematically review the existing learning to hash methods for efficient multimedia retrieval, categorizing them according to the multimedia retrieval tasks, the specific multi-modal semantic modeling techniques, and hash learning strategies. Thereafter, we present the performance comparison results. We ultimately discuss the challenges and potential research directions that may require further investigation in multi-modal hash learning.
26 April 2019 in IEEE Transactions on Knowledge and Data Engineering
Many achievements have been made on learning to hash for uni-modal and cross-modal retrieval. However, it is still an unsolved problem that how to directly and efficiently learn discriminative discrete hash codes for the multimedia retrieval, where both query and database samples are represented with heterogeneous multi-modal features. With this motivation, we propose a Fast Discrete Collaborative Multi-modal Hashing (FDCMH) method in this paper. We first propose an efficient collaborative multi-modal mapping that first transforms heterogeneous multi-modal features into the unified factors to exploit the complementarity of multi-modal features and preserve the semantic correlations in multiple modalities with linear computation and space complexity. Such shared factors also bridge the heterogeneous modality gap and remove the inter-modality redundancy. Further, we develop an asymmetric hashing learning module to simultaneously correlate the learned hash codes with low-level data distribution and high-level semantics. In particular, this design could avoid the challenging symmetric semantic matrix factorization and $O (n^2s)$ memory cost ( $n$ is the number of training samples). It can support both computation and memory efficient discrete hash optimization. Experiments on several public multimedia retrieval datasets demonstrate the superiority of the proposed approach compared with state-of-the-art hashing techniques, in terms of both model learning efficiency and retrieval accuracy.
11 November 2020 in IEEE Transactions on Multimedia
Social networks allow users to actively upload images and the descriptive tags, which leads to an explosive growth of social images. Multi-view hashing is an efficient technique to support large-scale social image retrieval for its desirable capabilities of encoding the multi-view features into compact binary hash codes with extremely low storage cost and fast retrieval speed. However, existing methods require the multi-view features to be fully paired at both offline model training and online query stages. This requirement cannot be easily satisfied for social image retrieval, where social images lacking descriptive tags are common in social networks. In this paper, we propose an Unsupervised Adaptive Partial Multi-view Hashing (UAPMH) method to handle the partial-view hashing problem for efficient social image retrieval. Specifically, the shared and view-specific latent representations for fully-paired and partial-view images respectively are learned separately by adaptive partial multi-view matrix factorization within the identical semantic space. In particular, instead of adopting the simple fixed view combination weights, we develop a parameter-free weight learning scheme to adaptively learn the weights to capture the view variations and the discriminative capabilities of different views. With such design, our model can sufficiently exploit the available partial-view samples with separate hash code learning, and meanwhile, effectively preserve the latent relations of images and tags into the hash codes with the semantic space sharing. Moreover, to avoid the relaxing errors and improve the learning efficiency, the binary hash codes are directly learned in a fast mode by simple and efficient operations. Finally, we extend UAPMH to supervised learning paradigm as Supervised Adaptive Partial Multi-view Hashing (SAPMH) with the supervision of pair-wise semantic labels, to further enhance the discriminative capability of hash codes. Experiments demonstrate the state-of-the-art performance of the proposed approaches on the public social image retrieval datasets.
9 July 2020 in IEEE Signal Processing Letters
Unsupervised multi-modal hashing has recently attracted broad attention in research area of large-scale multimedia retrieval for its low storage cost, high retrieval speed, and independence on semantic labels. However, the model learning process of existing methods still suffer from the problem of low efficiency: 1) Many existing methods measure the contributions of different modalities using fixed modality weights. In order to avoid over-fitting, they need an inefficient hyper-parameter adjustment process. 2) Most existing methods adopt inefficient optimization strategies to learn hash codes. In this paper, we propose an unsupervised Efficient Parameter-free Adaptive Multimodal Hashing (EPAMH) model to adaptively capture the modality variations and preserve the discriminative semantics of multimodal features into the binary hash codes. Moreover, we directly learn the binary codes with simple and efficient operations, which prevents the relaxing quantization errors and improves the model learning efficiency. Experiments prove the superior performance of EPAMH on three public multimedia retrieval datasets. Our source codes and testing datasets can be obtained at https://github.com/ChaoqunZheng/EPAMH
2 January 2024 in Information Sciences
23 September 2024 in ACM Transactions on Multimedia Computing, Communications, and Applications
Cross-modal hashing is widely used for efficient similarity searches, improving data processing efficiency, and reducing storage costs. Existing cross-modal hashing methods primarily focus on centralized training scenarios, where fixed-scale and fixed-category multi-modal data is collected beforehand. However, these methods often face challenges associated with the potential risk of privacy breaches and high data communication costs during data transmission in real-world multimedia retrieval tasks. To tackle these challenges, in this article, we propose an efficient privacy-enhanced prototype-based federated cross-modal hashing (PEPFCH). In PEPFCH, we integrate local and global prototypes in order to effectively capture the distinctive traits of individual clients, while also harnessing the collective intelligence of the entire federated learning system. Moreover, to ensure the security of prototype information and prevent its disclosure during the aggregation process, we use a prototype encryption transmission mechanism to encrypt the prototype information before transmission, making it challenging for attackers to gain access to sensitive data. Additionally, to facilitate personalized federated learning and alleviate the issue of parametric catastrophic forgetting, we establish the image and text hyper-networks for each client and adopt a hyper-network extension strategy to selectively preserve and update previously acquired knowledge when acquiring new concepts or categories. Comprehensive experiments highlight the efficiency and superiority of our proposed method. To enhance research and accessibility, we have publicly released our source codes at: https://github.com/vindahi/PEPFCH .