As an essential approach in many Internet of Things (IoT) applications, multiview learning synthesizes multiple features to achieve more comprehensive descriptions of data items. Most of the previous studies on multiview learning have been dedicated to increasing the prediction accuracy, while ignoring the reliability of the decision. This would limit their deployment in high-risk IoT and industrial applications such as the automated vehicle. Although a trusted multiview classification model has been proposed recently, it cannot well deal with the highly complementary multiview data. In this work, we present an evidential multiview deep learning (EMDL) method to make reliable decisions. EMDL first seeks view-specific evidence of each category, which could be termed as the amount of support to each category collected from data. It then dynamically fuses different views at the evidence level to construct the multiview common evidence and makes reliable prediction accordingly (strong evidence indicates high prediction confidence). In particular, we establish a degradation layer to learn the mappings from the common evidence (comprehensive information) to view-specific evidences (partial information) for evidence fusion. It aims to explicitly model consistent and complementary relations in multiview data at the evidence level. We apply EMDL on a synthetic toy dataset and five real-world datasets (three datasets are related to industrial scenarios). Experiments show that EMDL outperforms state-of-the-art baseline methods.

Multiview representation learning (MVRL) leverages information from multiple views to obtain a common representation summarizing the consistency and complementarity in multiview data. Most previous matrix factorization-based MVRL methods are shallow models that neglect the complex hierarchical information. The recently proposed deep multiview factorization models cannot explicitly capture consistency and complementarity in multiview data. We present the deep multiview concept learning (DMCL) method, which hierarchically factorizes the multiview data, and tries to explicitly model consistent and complementary information and capture semantic structures at the highest abstraction level. We explore two variants of the DMCL framework, DMCL-L and DMCL-N, with respectively linear/nonlinear transformations between adjacent layers. We propose two block coordinate descent-based optimization methods for DMCL-L and DMCL-N. We verify the effectiveness of DMCL on three real-world data sets for both clustering and classification tasks.

Multiview clustering aims to leverage information from multiple views to improve the clustering performance. Most previous works assumed that each view has complete data. However, in real-world datasets, it is often the case that a view may contain some missing data, resulting in the problem of incomplete multiview clustering (IMC). Previous approaches to this problem have at least one of the following drawbacks: 1) employing shallow models, which cannot well handle the dependence and discrepancy among different views; 2) ignoring the hidden information of the missing data; and 3) being dedicated to the two-view case. To eliminate all these drawbacks, in this work, we present the adversarial IMC (AIMC) framework. In particular, AIMC seeks the common latent representation of multiview data for reconstructing raw data and inferring missing data. The elementwise reconstruction and the generative adversarial network are integrated to evaluate the reconstruction. They aim to capture the overall structure and get a deeper semantic understanding, respectively. Moreover, the clustering loss is designed to obtain a better clustering structure. We explore two variants of AIMC, namely: 1) autoencoder-based AIMC (AAIMC) and 2) generalized AIMC (GAIMC), with different strategies to obtain the multiview common representation. Experiments conducted on six real-world datasets show that AAIMC and GAIMC perform well and outperform the baseline methods.

As an essential role in smart city applications, personalized recommender systems help users to find their potentially interested items from their historically generated data. Recently, researchers have started to utilize the massive user-generated multi-modal contents to improve recommendation performance. However, previous methods have at least one of the following drawbacks: (1) employing shallow models, which cannot well capture high-level conceptual information; (2) failing to capture personalized user visual preference. In this work, we present a deep Users' Multi-modal Preferences based Recommendation (UMPR) method to capture the textual and visual matching of users and items for recommendation. We extract textual matching from historical reviews. We construct users' visual preference embeddings to model users' visual preference and match them with items' visual embeddings to obtain the visual matching. We apply UMPR on two applications related to smart city: restaurant recommendation and product recommendation. Experiments show that UMPR outperforms competitive baseline methods.

We are concerned with using user-tagged images to learn proper hashing functions for image retrieval. The benefits are two-fold: (1) we could obtain abundant training data for deep hashing models; (2) tagging data possesses richer semantic information which could help better characterize similarity relationships between images. However, tagging data suffers from noises, vagueness and incompleteness. Different from previous unsupervised or supervised hashing learning, we propose a novel weakly-supervised deep hashing framework which consists of two stages: weakly-supervised pre-training and supervised fine-tuning. The second stage is as usual. In the first stage, we propose two formulations Tag-basEd weakLy-supErvised Modally COoperative hashing Network (TelecomNet) and Generalized TelecomNet (GTelecomNet). Rather than performing supervision on tags, TelecomNet first learns an observed semantic embedding vector for each image from attached tags and then uses it to guide hashing learning. GTelecomNet introduces a novel semantic network to exploit more precise semantic information. By carefully designing the optimization problem, they can well leverage tagging information and image content for hashing learning. The framework is general and does not depend on specific deep hashing methods. Empirical results on real world datasets show that they significantly increase the performance of state-of-the-art deep hashing methods.