Capsule network is a novel architecture to encode the properties and spatial relationships of the feature in an image, which shows encouraging results on image classification. However, the original capsule network is not suitable for some classification tasks, where the target objects are complex internal representations. Hence, we propose a multi-scale capsule network that is more robust and efficient for feature representation in image classification. The proposed multi-scale capsule network consists of two stages. In the first stage, structural and semantic information are obtained by multi-scale feature extraction. In the second stage, the hierarchy of features is encoded to multi-dimensional primary capsules. Moreover, we propose an improved dropout to enhance the robustness of the capsule network. Experimental results show that our method has a competitive performance on FashionMNIST and CIFAR10 datasets.

Benefiting from Multi-view Video plus Depth (MVD) and Depth-Image-Based-Rendering (DIBR) technologies, only limited views of a real 3D scene need to be captured, compressed and transmitted. However, the quality assessment of synthesized views is very challenging since some new types of distortions, which are inherently different from the texture coding errors, are inevitably produced by view synthesis and depth map compression, and the corresponding original views (reference views) are usually not available. Thus the full-reference quality metrics cannot be used for synthesized views. In this paper, we propose a novel no-reference image quality assessment method for 3D synthesized views ( called NIQSV+). This blind metric can evaluate the quality of synthesized views by measuring the typical synthesis distortions: blurry regions, black holes and stretching, with access to neither the reference image nor the depth map. To evaluate the performance of the proposed method, we compare it with four full-reference 3D (synthesized view dedicated) metrics, five full-reference 2D metrics and three no-reference 2D metrics. In terms of their correlations with subjective scores, our experimental results show that the proposed no-reference metric approaches the best of the state-of-the-art full reference and no-reference 3D metrics; and outperforms the widely used no-reference and full-reference 2D metrics significantly. In terms of its approximation of human ranking, the proposed metric achieves the best performance in the experimental test.

With the wide use of visual sensors in the Internet of Things (IoT) in the past decades, huge amounts of images are captured in people's daily lives, which poses challenges to traditional deep-learning-based image retrieval frameworks. Most such frameworks need a large amount of annotated training data, which are expensive. Moreover, machines still lack human intelligence, as illustrated by the fact that they pay less attention to the interesting regions that humans generally focus on when searching for images. Hence, this paper proposes a novel unsupervised framework that focuses on the instance object in the image and integrates human intelligence into the deep-learning-based image retrieval. This framework is called adversarial instance-level image retrieval (AILIR). We incorporate adversarial training and an attention mechanism into this framework that considers human intelligence with artificial intelligence. The generator and discriminator are redesigned to guarantee that the generator retrieves similar images while the discriminator selects unmatched images and creates an adversarial reward for the generator. A minimax game is conducted by the adversarial reward retrieval mechanism until the discriminator is unable to judge whether the image sequence retrieved matches the query. Comparison and ablation experiments on four benchmark datasets prove that the proposed adversarial training framework indeed improves instance retrieval and outperforms the state-of-the-art methods focused on instance retrieval.

Depth-image-based rendering (DIBR) is a fundamental technology in several 3-D-related applications, such as free viewpoint video, virtual reality, and augmented reality. However, new challenges have also been brought in assessing the quality of DIBR-synthesized views since this process induces some new types of distortions, which are inherently different from the distortion caused by video coding. In this paper, we present a new DIBR-synthesized image database with the associated subjective scores. We also test the performances of the state-of-the-art objective quality metrics on this database. This paper focuses on the distortions only induced by different DIBR synthesis methods. Seven state-of-the-art DIBR algorithms, including inter-view synthesis and single-view-based synthesis methods, are considered in this database. The quality of synthesized views was assessed subjectively by 41 observers and objectively using 14 state-of-the-art objective metrics. Subjective test results show that the interview synthesis methods, having more input information, significantly outperform the single-view-based ones. Correlation results between the tested objective metrics and the subjective scores on this database reveal that further studies are still needed for a better objective quality metric dedicated to the DIBR-synthesized views.

Independent mobility poses a great challenge to the visually impaired individuals. This paper proposes a novel system to understand dynamic crosswalk scenes, which detects the key objects, such as crosswalk, vehicle, and pedestrian, and identifies pedestrian traffic light status. The indication of where and when to cross the road is provided to the visually impaired based on the crosswalk scene understanding. Our proposed system is implemented on a head-mounted mobile device (SensingAI G1) equipped with an Intel RealSense camera and a cellphone, and provides surrounding scene information to visually impaired individuals through audio signal. To validate the performance of the proposed system, we propose a crosswalk scene understanding dataset which contains three sub-datasets: a pedestrian traffic light dataset with 7447 images, a dataset of key objects on the crossroad with 1006 images and a crosswalk dataset with 3336 images. Extensive experiments demonstrated that the proposed system was robust and outperformed the state-of-the-art approaches. The experiment conducted with the visually impaired subjects shows that the system is practical useful.