As an increasingly popular task in multimedia information retrieval, video moment retrieval (VMR) aims to localize the target moment from an untrimmed video according to a given language query. Most previous methods depend heavily on numerous manual annotations ( i.e. , moment boundaries), which are extremely expensive to acquire in practice. In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop. In this paper, we focus on a novel task: cross-domain VMR, where fully-annotated datasets are available in one domain (“source domain”), but the domain of interest (“target domain”) only contains unannotated datasets. As far as we know, we present the first study on cross-domain VMR. To address this new task, we propose a novel Multi-Modal Cross-Domain Alignment (MMCDA) network to transfer the annotation knowledge from the source domain to the target domain. However, due to the domain discrepancy between the source and target domains and the semantic gap between videos and queries, directly applying trained models to the target domain generally leads to a performance drop. To solve this problem, we develop three novel modules: (i) a domain alignment module is designed to align the feature distributions between different domains of each modality; (ii) a cross-modal alignment module aims to map both video and query features into a joint embedding space and to align the feature distributions between different modalities in the target domain; (iii) a specific alignment module tries to obtain the fine-grained similarity between a specific frame and the given query for optimal localization. By jointly training these three modules, our MMCDA can learn domain-invariant and semantic-aligned cross-modal representations. Extensive experiments on three challenging benchmarks (ActivityNet Captions, Charades-STA and TACoS) illustrate that our cross-domain method MMCDA outperforms all state-of-the-art single-domain methods. Impressively, MMCDA raises the performance by more than 7% in representative cases, which demonstrates its effectiveness.

This paper studies the multimedia problem of temporal sentence grounding (TSG), which aims to accurately determine the specific video segment in an untrimmed video according to a given sentence query. Traditional TSG methods mainly follow the top-down or bottom-up framework and are not end-to-end. They severely rely on time-consuming post-processing to refine the grounding results. Recently, some transformer-based approaches are proposed to efficiently and effectively model the fine-grained semantic alignment between video and query. Although these methods achieve significant performance to some extent, they equally take frames of the video and words of the query as transformer input for correlating, failing to capture their different levels of granularity with distinct semantics. To address this issue, in this paper, we propose a novel H ierarchical L ocal- G lobal T ransformer (HLGT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities for learning more fine-grained multi-modal representations. Specifically, we first split the video and query into individual clips and phrases to learn their local context (adjacent dependency) and global correlation (long-range dependency) via a temporal transformer. Then, a global-local transformer is introduced to learn the interactions between the local-level and global-level semantics for better multi-modal reasoning. Besides, we develop a new cross-modal cycle-consistency loss to enforce interaction between two modalities and encourage the semantic alignment between them. Finally, we design a brand-new cross-modal parallel transformer decoder to integrate the encoded visual and textual features for final grounding. Extensive experiments on three challenging datasets (ActivityNet Captions, Charades-STA and TACoS) show that our proposed HLGT achieves a new state-of-the-art performance, demonstrating its effectiveness and computational efficiency.

Incomplete multi-view clustering is an important technique to deal with real-world incomplete multi-view data. Previous methods assume that all views have the same incompleteness, i.e., balanced incompleteness. However, different views often have distinct incompleteness, i.e., unbalanced incompleteness, which results in strong views (low-incompleteness views) and weak views (high-incompleteness views). The unbalanced incompleteness prevents us from directly using previous methods. In this paper, inspired by the effective biological evolution theory, we design the novel scheme of view evolution to cluster strong and weak views. Moreover, we propose an Unbalanced Incomplete Multi-view Clustering method (UIMC), which is the first effective method based on view evolution for unbalanced incomplete multi-view clustering. Compared with previous methods, UIMC has two unique advantages: 1) it proposes weighted multi-view subspace clustering to integrate unbalanced incomplete views, which effectively solves the unbalanced incomplete multi-view clustering problem; 2) it designs the low-rank representation to recover the data, which diminishes the impact of the incompleteness and noises. Extensive experimental results demonstrate that UIMC improves the clustering performance by up to 40% on three evaluation metrics over other state-of-the-art methods. We provide codes for all of our experiments in https://github.com/ZeusDavide/TETCI_UIMC .

Multiview clustering has wide real-world applications because it can process data from multiple sources. However, these data often contain missing instances and noises, which are ignored by most multiview clustering methods. Missing instances may make these methods difficult to use directly, and noises will lead to unreliable clustering results. In this article, we propose a novel autoweighted noisy and incomplete multiview clustering (ANIMC) approach via a soft autoweighted strategy and a doubly soft regular regression model. First, by designing adaptive semiregularized nonnegative matrix factorization, the soft autoweighted strategy assigns a proper weight to each view and adds a soft boundary to balance the influence of noises and incompleteness. Second, by proposing $\theta$ -norm, the doubly soft regularized regression model adjusts the sparsity of our model by choosing different $\theta$ . Compared with previous methods, ANIMC has three unique advantages: 1) it is a soft algorithm to adjust our approach in different scenarios, thereby improving its generalization ability; 2) it automatically learns a proper weight for each view, thereby reducing the influence of noises; 3) it performs doubly soft regularized regression that aligns the same instances in different views, thereby decreasing the impact of missing instances. Extensive experimental results show its superior advantages over other state-of-the-art works.

Real data often appear in the form of multiple incomplete views. Incomplete multiview clustering is an effective method to integrate these incomplete views. Previous methods only learn the consistent information between different views and ignore the unique information of each view, which limits their clustering performance and generalizations. To overcome this limitation, we propose a novel V iew V ariation and V iew H eredity approach (V $^3$ H). Inspired by the variation and the heredity in genetics, V $^3$ H first decomposes each subspace into a variation matrix for the corresponding view and a heredity matrix for all the views to represent the unique information and the consistent information respectively. Then, by aligning different views based on their cluster indicator matrices, V $^3$ H integrates the unique information from different views to improve the clustering performance. Finally, with the help of the adjustable low-rank representation based on the heredity matrix, V $^3$ H recovers the underlying true data structure to reduce the influence of the large incompleteness. More importantly, V $^3$ H presents possibly the first work to introduce genetics to clustering algorithms for learning simultaneously the consistent information and the unique information from incomplete multiview data. Extensive experimental results on fifteen benchmark datasets validate its superiority over other state-of-the-arts. Impact Statement—Incomplete multiview clustering is a popular technology to cluster incomplete datasets from multiple sources. The technology is becoming more significant due to the absence of the expensive requirement of labeling these datasets. However, previous algorithms cannot fully learn the information of each view. Inspired by variation and heredity in genetics, our proposed algorithm V$^3$H fully learns the information of each view. Compared with the state-of-the-art algorithms, V$^3$H improves clustering performance by more than 20% in representative cases. With the large improvement on multiple datasets, V$^3$H has wide potential applications including the analysis of pandemic, financial and election datasets. The DOI of our codes is 10.24 433/CO.2 119 636.v1.