Unsupervised cross-modal hashing retrieval across image and text modality is a challenging task because of the suboptimality of similarity guidance, i.e., the joint similarity matrix constructed by existing methods does not possess clear enough guiding significance. How to construct more robust similarity matrix is the key to solve this problem. The unsupervised cross-modal retrieval methods based on graph have a good performance in mining semantic information of input samples, but the graph hashing based on traditional affinity graph cannot capture the high-order semantic information of input samples effectively. In order to overcome the aforementioned limitations, this paper presents a novel hypergraph-based approach for unsupervised cross-modal retrieval that differs from previous works in two significant ways. Firstly, to address the ubiquitous redundant information present in current methods, this paper introduces a robust similarity matrix constructing method. Secondly, we propose a novel hypergraph enhanced module that produces embedding vectors by hypergraph convolution and attention mechanism for input data, capturing important high-order semantics. Our approach is evaluated on the NUS-WIDE and MIRFlickr datasets, and yields state-of-the-art performance for unsupervised cross-modal retrieval.

Data generated and collected from Cyber-Physical-Social Systems (CPSS) that usually in the forms of image, audio, video, and text, are complex and heterogeneous. How to deal with the cross-modal retrieval problem for heterogeneous CPSS data has drawn considerable interests recently. The hashing based methods have been widely studied in building bilateral semantic associations of binary codes for cross-model retrieval. However, most existing methods discard binary constraints and learn linear projections as hashing functions. Moreover, none of them consider the cross-modal retrieval application in the scenario of Cloud-Fog-Edge computing. Therefore, how to learn more compact and discriminative binary codes with discrete constraints and nonlinear hashing functions for CPSS data in the Cloud-Fog-Edge architecture is still an open problem. In this paper, we propose a nonlinear discrete cross-modal hashing (NDCMH) method based on concise binary classification for CPSS data which fully investigates the nonlinear relationship embedding, discrete optimization as well as the hashing functions learning. Different from previous methods, our work presents a concise but promising cross-modal hashing method that builds a direct connection between original CPSS data and binary codes, which can alleviate the impact of large quantization loss. Furthermore, we execute the cross-modal retrieval service at cloud and fog. Specifically, hashing functions are deployed at the fog plane to reduce the amount data transfer and storage need on the cloud. Extensive experiments carried out on typical CPSS datasets demonstrate that the proposed NDCMH significantly outperforms other state-of-the-art methods.

Ship image classification identifies the type of ships in an input image, which plays a significant role in the marine field. To enhance the ship classification performance, various research focuses on studying multi-modal ship classification, which aims at combining the advantages of visible images and infrared images to capture complementary information. However, the current methods simply concatenate features of different modalities to learn complementary information, which neglects the multi-level correlation between different modalities. Moreover, the existing methods require a large amount of labeled ship images to train the model. How to capture the multi-level cross-modal correlation between unlabeled and labeled data is still a challenge. In this paper, a novel semi-supervised multi-modal ship classification approach is proposed to solve these issues, which consists of two components, i.e., multi-level cross-modal interactive network and semi-supervised contrastive learning strategy. To learn comprehensive complementary information for classification, the multi-level cross-modal interactive network is designed to build local-level and global-level cross-modal feature correlation. Then, the semi-supervised contrastive learning strategy is employed to drive the optimization of the network with the intra-class consistency constraint based on supervision signals of unlabeled samples and prior label information. Extensive experiments on the public datasets demonstrate that our approach achieves state-of-the-art semi-supervised classification effectiveness.