Accurate land cover classification and detection of objects in high-resolution electro-optical remote sensing imagery (RSI) have long been a challenging task. Recently, important new benchmark data sets have been released which are suitable for land cover classification and object detection research. Here, we present state-of-the-art results for four benchmark data sets using a variety of deep convolutional neural networks (DCNN) and multiple network fusion techniques. We achieve 99.70%, 99.66%, 97.74%, and 97.30% classification accuracies on the PatternNet, RSI-CB256, aerial image, and RESISC-45 data sets, respectively, using the Choquet integral with a novel data-driven optimization method presented in this letter. The relative reduction in classification errors achieved by this data driven optimization is 25%-45% compared with the single best DCNN results.

Information fusion is an essential part of numerous engineering systems and biological functions, e.g., human cognition. Fusion occurs at many levels, ranging from the low-level combination of signals to the high-level aggregation of heterogeneous decision-making processes. While the last decade has witnessed an explosion of research in deep learning, fusion in neural networks has not observed the same revolution. Specifically, most neural fusion approaches are ad hoc, are not understood, are distributed versus localized, and/or explainability is low (if present at all). Herein, we prove that the fuzzy Choquet integral (ChI), a powerful nonlinear aggregation function, can be represented as a multi-layer network, referred to hereafter as ChIMP. We also put forth an improved ChIMP (iChIMP) that leads to a stochastic gradient descent-based optimization in light of the exponential number of ChI inequality constraints. An additional benefit of ChIMP/iChIMP is that it enables eXplainable AI (XAI). Synthetic validation experiments are provided and iChIMP is applied to the fusion of a set of heterogeneous architecture deep models in remote sensing. We show an improvement in model accuracy and our previously established XAI indices shed light on the quality of our data, model, and its decisions.

Kernel methods for classification is a well-studied area in which data are implicitly mapped from a lower-dimensional space to a higher dimensional space to improve classification accuracy. However, for most kernel methods, one must still choose a kernel to use for the problem. Since there is, in general, no way of knowing which kernel is the best, multiple kernel learning (MKL) is a technique used to learn the aggregation of a set of valid kernels into a single (ideally) superior kernel. The aggregation can be done using weighted sums of the precomputed kernels, but determining the summation weights is not a trivial task. Furthermore, MKL does not work well with large datasets because of limited storage space and prediction speed. In this paper, we address all three of these multiple kernel challenges. First, we introduce a new linear feature level fusion technique and learning algorithm, GAMKLp. Second, we put forth three new algorithms, DeFIMKL, DeGAMKL, and DeLSMKL, for nonlinear fusion of kernels at the decision level. To address MKL's storage and speed drawbacks, we apply the Nystrom approximation to the kernel matrices. We compare our methods to a successful and state-of-the-art technique called MKL-group lasso (MKLGL), and experiments on several benchmark datasets show that some of our proposed algorithms outperform MKLGL when applied to support vector machine (SVM)-based classification. However, to no surprise, there does not seem to be a global winner but instead different strategies that a user can employ. Experiments with our kernel approximation method show that we can routinely discard most of the training data and at least double prediction speed without sacrificing classification accuracy. These results suggest that MKL-based classification techniques can be applied to big data efficiently, which is confirmed by an experiment using a large dataset.

The $Choquet integral$ (ChI) is a parametric nonlinear aggregation function defined with respect to the $fuzzy measure$ (FM). To date, application of the ChI has sadly been restricted to problems with relatively few numbers of inputs; primarily as the FM has 2$^{N}$ variables for $N$ inputs and $N$(2$^{N-1}$-1) monotonicity constraints. In return, the community has turned to density-based imputation (e.g, Sugeno λ-FM) or the number of $interactions$ (FM variables) are restricted (e.g., $k$-additivity). Herein, we propose a new scalable data-driven way to represent and learn the ChI, making learning computationally manageable for larger $N$. First, data supported variables are identified and used in optimization. Identification of these variables also allows us recognize future ill posed fusion scenarios; ChIs involving variable subsets not supported by data. Second, we outline an imputation function framework to address data unsupported variables. Third, we present a lossless way to compress redundant variables and associated monotonicity constraints. Last, we outline a lossy approximation method to further compress the ChI (if/when desired). Computational complexity analysis and experiments conducted on synthetic data sets with known FMs demonstrate the effectiveness and efficiency of the proposed theory.

A novel approach is proposed for combining multiple anomaly algorithm decisions with image space cell-structured features in a long wave infrared (LWIR) system in the context of forward looking (FL) buried explosive hazard detection along a road. A pre-screener is applied first, which is an ensemble of trainable size-contrast filters with mean shift clustering in Universal Transverse Mercator (UTM) space. Next, features from image chips representing anomaly decisions from different algorithms are extracted from UTM confidence maps based on maximally stable extremal regions (MSERs) and Gaussian mixture models (GMMs). Pre-screener hits in UTM space are back-projected into the video at multiple standoff distances and cell-structured local binary patterns (LBPs), histogram of gradients (HOGs) and mean-variance descriptors are extracted. Experiments are conducted using buried materials with varying metal contents and depths at a U.S. Army test site. Results are extremely encouraging for FL imaging and show a significant decrease in the number of false alarms (FAs). Targets not currently detected by our system are also not detected by a human under manual visual inspection.

The modern era of machine learning is focused on data-driven solutions. While this has resulted in astonishing leaps in numerous applications, explainability has not witnessed the same growth. The reality is, most machine learning solutions are black boxes. Herein, we focus on data/information fusion in machine learning. Specifically, we explore four eXplainable Artificial Intelligence (XAI) questions relative to Choquet integral; (i) what is the quality of our inputs and their interactions, (ii) how is the information being combined, (iii) what is the quality of our training data (and thus our learned models), and (iv) what trust do we place in an output? Previously, we derived an initial set of indices for (i)–(iv) on the premise of perfect knowledge. Herein, we make XAI more accurate by taking into consideration what the machine learned. A combination of synthetic data and real-world experiments from remote sensing for fusing deep learners in the context of classification are explored. Our approach leads to performance gain, insights into what was learned, and it helps us realize better future solutions.