General-purpose foundation models have led to recent breakthroughs in artificial intelligence. In remote sensing, self-supervised learning (SSL) and Masked Image Modeling (MIM) have been adopted to build foundation models. However, these models primarily learn low-level features and require annotated data for fine-tuning. Moreover, they are inapplicable for retrieval and zero-shot applications due to the lack of language understanding. To address these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics and aligned text embeddings for seamless downstream application. To address the scarcity of pre-training data, we leverage data scaling which converts heterogeneous annotations into a unified image-caption data format based on Box-to-Caption (B2C) and Mask-to-Box (M2B) conversion. By further incorporating UAV imagery, we produce a 12 × larger pretraining dataset than the combination of all available datasets. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, k -NN classification, few-shot classification, image-text retrieval, and object counting in remote sensing images. Evaluation on 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, shows that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP beats the state-of-the-art method by 9.14% mean recall on the RSITMD dataset and 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the CLIP baseline by up to 6.39% average accuracy on 12 downstream datasets.

The joint use of multi-source remote sensing data for earth observation missions has drawn much attention. Although the fusion of several data sources can improve the accuracy of land-cover identification, many technical obstacles exist, such as disparate data structures, irrelevant physical characteristics, and a lack of training data. In this article, a novel dual-branch method, consisting of a hierarchical convolutional neural network (CNN) and a transformer network, is proposed for fusing multi-source heterogeneous information and improving joint classification performance. First, by combining CNN with a transformer, the proposed dual-branch network can significantly capture and learn spectral-spatial features from hyperspectral image data and elevation features from light detection and ranging (LiDAR) data. Then, to fuse these two sets of data features, a cross-token attention fusion encoder is designed in a specialty. The well-designed deep hierarchical architecture takes full advantage of the powerful spatial context information extraction ability of CNN and the strong long-range dependency modeling ability of the transformer network based on self-attention mechanism. Four standard datasets are used in experiments to verify the effectiveness of the approach. The experimental results reveal that the proposed framework can perform noticeably better than state-of-the-art methods. The source code of the proposed method will be available publicly at https://github.com/zgr6010/Fusion_HCT.git.

Multiview learning (MVL), which enhances the learners’ performance by coordinating complementarity and consistency among different views, has attracted much attention. The multiview generalized eigenvalue proximal support vector machine (MvGSVM) is a recently proposed effective binary classification method, which introduces the concept of MVL into the classical generalized eigenvalue proximal support vector machine (GEPSVM). However, this approach cannot guarantee good classification performance and robustness yet. In this article, we develop multiview robust double-sided twin SVM (MvRDTSVM) with SVM-type problems, which introduces a set of double-sided constraints into the proposed model to promote classification performance. To improve the robustness of MvRDTSVM against outliers, we take L1-norm as the distance metric. Also, a fast version of MvRDTSVM (called MvFRDTSVM) is further presented. The reformulated problems are complex, and solving them are very challenging. As one of the main contributions of this article, we design two effective iterative algorithms to optimize the proposed nonconvex problems and then conduct theoretical analysis on the algorithms. The experimental results verify the effectiveness of our proposed methods.

As a core function of autonomous driving (AD) and the Internet of Vehicles (IoV), accurately predicting the trajectory of vehicles can significantly improve traffic safety and reduce crash injuries. In this article, we propose an intention-aware non-autoregressive Transformer model with multi-attention learning for multimodal vehicle trajectory prediction. We first present social attention learning (SAL) where graph attention is properly integrated with the Transformer encoder so as to model the social interaction between vehicles. Then, the social and temporal dependency across consecutive frames is captured by temporal attention learning (TAL). The above social and temporal attention modules can be interleaved and stacked to achieve the coupled modeling and thus extract abundant features from trajectory data. To implement precise prediction as well as efficient inference, we further put forward an intention-aware decoder query generation approach to produce multiple possible trajectories concurrently. Finally, cross-attention learning (CAL) is devised to make full use of the encoded features, therefore, yielding future predictions. The proposed model is evaluated on two large-scale vehicle trajectory datasets and the experimental results verify that our algorithm achieves better performance compared with some state-of-the-art models. The root-mean-square error (RMSE) of the predicted trajectory over 5-s time horizon for the next generation simulation (NGSIM) and HighD datasets is 3.43 and 1.10 m, respectively.

Of late, there are many studies on the robust discriminant analysis, which adopt L₁-norm as the distance metric, but their results are not robust enough to gain universal acceptance. To overcome this problem, the authors of this article present a nonpeaked discriminant analysis (NPDA) technique, in which cutting L₁-norm is adopted as the distance metric. As this kind of norm can better eliminate heavy outliers in learning models, the proposed algorithm is expected to be stronger in performing feature extraction tasks for data representation than the existing robust discriminant analysis techniques, which are based on the L₁-norm distance metric. The authors also present a comprehensive analysis to show that cutting L₁-norm distance can be computed equally well, using the difference between two special convex functions. Against this background, an efficient iterative algorithm is designed for the optimization of the proposed objective. Theoretical proofs on the convergence of the algorithm are also presented. Theoretical insights and effectiveness of the proposed method are validated by experimental tests on several real data sets.

Recent works have proposed two L1-norm distance measure-based linear discriminant analysis (LDA) methods, L1-LD and LDA-L1, which aim to promote the robustness of the conventional LDA against outliers. In LDA-L1, a gradient ascending iterative algorithm is applied, which, however, suffers from the choice of stepwise. In L1-LDA, an alternating optimization strategy is proposed to overcome this problem. In this paper, however, we show that due to the use of this strategy, L1-LDA is accompanied with some serious problems that hinder the derivation of the optimal discrimination for data. Then, we propose an effective iterative framework to solve a general L1-norm minimization-maximization (minmax) problem. Based on the framework, we further develop a effective L1-norm distance-based LDA (called L1-ELDA) method. Theoretical insights into the convergence and effectiveness of our algorithm are provided and further verified by extensive experimental results on image databases.