Deep convolutional neural networks (CNNs) have achieved state-of-the-art performance in a wide range of applications. However, deeper CNN models, which are usually computation consuming, are widely required for complex artificial intelligence (AI) tasks. Though recent research progress on network compression, such as pruning, has emerged as a promising direction to mitigate computational burden, existing accelerators are still prevented from completely utilizing the benefits of leveraging sparsity due to the irregularity caused by pruning. On the other hand, field-programmable gate arrays (FPGAs) have been regarded as a promising hardware platform for CNN inference acceleration. However, most existing FPGA accelerators focus on dense CNN and cannot address the irregularity problem. In this article, we propose a sparsewise dataflow to skip the cycles of processing multiply-and-accumulates (MACs) with zero weights and exploit data statistics to minimize energy through zeros gating to avoid unnecessary computations. The proposed sparsewise dataflow leads to a low bandwidth requirement and high data sharing. Then, we design an FPGA accelerator containing a vector generator module (VGM) that can match the index between sparse weights and input activations according to the proposed dataflow. Experimental results demonstrate that our implementation can achieve 987-, 46-, and 57-imag/s performance for AlexNet, VGG-16, and ResNet-50 on Xilinx ZCU102, respectively, which provides 1.5x-6.7x speedup and 2.0x-6.0x energy efficiency over previous CNN FPGA accelerators.

The epileptic detection with Electroencephalography (EEG) has been deeply studied and developed. However, previous research gave little attention to the physical appearance and early onset warnings of seizure. When a seizure occurs, electrodes near the epileptic foci will exhibit significantly fluctuating and inconsistent voltages. In this paper, a novel approach to epileptic detection based on the Hierarchy Graph Convolution Network (HGCN) structure is proposed. Multiple features of time or frequency domains extracted from the raw EEG signals are taken as the input of HGCN. The topological relationship between every single electrode is utilized by HGCN. The Tree Classification (TC) and Preictal Fuzzification (PF) are proposed to adapt both multi-classification tasks and refine-classification tasks. Experiments are performed on the CHB-MIT and TUH datasets. Compared with the state-of-the-art, our proposed model achieves a 5.77% improvement of accuracy on the CHB-MIT dataset, and an improvement of 2.43% and 19.7% for sensitivity and specificity on the TUH dataset, respectively.

The Artificial Intelligence (AI) in edge computing is requesting new processing units with a much higher computing-power ratio. The emerging resistive Non-Volatile Memory (NVM) with the in-memory computing capability may greatly advance the AI hardware technologies. In this paper, we propose the use of binary resistive memory to form an 8-bit fixed-point data/weight for AI computing. A robust Computing-In-Memory (CIM) core with digital input and analog output Multiplication-and-Accumulation (MAC) circuit is proposed. The corresponding integration scheme and Successive Approximation Register Analog-to-Digital Converter (SAR ADC) based data conversion scheme are also presented. The simulation results show that the proposed CIM core achieves 7.26 bit of Effective Number of Bits (ENOB) with 0.78,mW (256*1) power consumption and 1.85,M/s computing speed. Compared with previously reported CIM implementations and Deep Learning Accelerators (DLAs) (without CIM ability), our design achieves 2.23-7.26x better energy efficiency in 8-bit input 8-bit weight pattern, and achieves relatively high accuracy with LeNet and AlexNet.

The separation of the data capture and analysis in modern vision systems has led to a massive amount of data transfer between the end devices and cloud computers, resulting in long latency, slow response, and high power consumption. Efficient hardware architectures are under focused development to enable Artificial Intelligence (AI) at the resource-limited sensing devices. One of the most promising solutions is to enable Processing-in-Pixel (PIP) scheme. However, the conventional schemes suffer from the low fill-factor issue. This paper proposes a PIP based Complementary Metal-Oxide-Semiconductor (CMOS) sensor architecture, which allows convolution operation before the column readout circuit to significantly reduce the overall power consumption while improving the resource utilization of the succeeding deep learning accelerator. The simulation results show that the proposed architecture could support the computing efficiency up to 3.37 TOPS/W at the 8-bit weight configuration, which is four times as high as the conventional schemes after normalization. The transistors required for each pixel are only 3.5T, significantly improving the fill-factor.

The data storage has been one of the bottlenecks in surveillance systems. The conventional video compression schemes such as H.264 and H.265 do not fully utilize the low information density characteristic of the surveillance video, and they attach equal importance to foreground and background when performing compression. In this article, we propose a novel video compression scheme that compresses the foreground and background of the surveillance video separately. The compression ratio is greatly improved by sharing background information among adjacent frames through an adaptive background updating and interpolation module. Besides, we present two different schemes to compress the foreground and compare their performance in the ablation study to show the importance of temporal information for video compression. In the decoding end, a coarse-to-fine two-stage module is applied to achieve the composition of the foreground and background and the enhancements of frame quality. The experimental results show that our proposed method requires 49.75% less bpp (bits per pixel) than the conventional algorithm H.265 to achieve the same PSNR (36 dB) on the HEVC dataset.

Human video motion transfer has a wide range of applications in multimedia, computer vision and graphics. Recently, due to the rapid development of Generative Adversarial Networks (GANs), there has been significant progress in the field. However, almost all existing GAN-based works are prone to address the mapping from human motions to video scenes, with scene appearances encoded individually in the trained models. Therefore, each trained model can only generate videos with a specific scene appearance, and new models are required to be trained to generate new appearances. Besides, existing works lack the capability of appearance control. For example, users have to provide video records of wearing new clothes or performing in new backgrounds to enable clothes or background changing in their synthetic videos, which greatly limits the application flexibility. In this paper, we propose General Appearance-Controllable GAN (GAC-GAN), a general method for appearance-controllable human video motion transfer. To enable general-purpose appearance synthesis, we propose to include appearance information in the conditioning inputs. Thus, once trained, our model can generate new appearances by altering the input appearance information. To achieve appearance control, we first obtain the appearance-controllable conditioning inputs and then utilize a two-stage GAC-GAN to generate the corresponding appearance-controllable outputs, where we utilize an Appearance-Consistency GAN (ACGAN) loss and a shadow extraction module for output foreground and background appearance control respectively. We further build a solo dance dataset containing a large number of dance videos for training and evaluation. Experimental results on our solo dance dataset and iPER dataset show that our proposed GAC-GAN can not only support appearance-controllable human video motion transfer but also achieve higher video quality than state-of-art methods.