The acceleration architecture of transposed convolution layers is essential since transposed convolution operations, as critical components in the generative model of generative adversarial networks, are computationally intensive inherently. In addition, the pre-processing of inserting and padding with zeros for input feature maps causes many ineffective operations. Most of the already known FPGA (Field Programmable Gate Array) based architectures for convolution layers cannot tackle these issues. In this paper, we firstly propose a novel dataflow exploration through splitting the filters and its corresponding input feature maps into four sets and then applying the Winograd algorithm for fast processing with a high efficiency. Secondly, we present an underlying FPGA-based accelerator architecture that features owning processing units, with embedded parallel, pipelined, and buffered processing flow. At last, a parallelism-aware memory partition technique and the hardware-based design space are explored coordinating, respectively, for the required parallel operations and optimal design parameters. Experiments of several state-of-the-art GANs by our methods achieve an average performance of 639.2 GOPS on Xilinx ZCU102 and 162.5 GOPS on Xilinx VC706. In reference to a conventional optimized accelerator baseline, this work demonstrates an 8.6× (up to 11.7×) increase in processing performance, compared to below 2.2× improvement by the prior studies in the literature.

Off-chip memory, such as DRAM, its access energy cost is orders of magnitude higher than other operations such as multiply and accumulate, thereby dominating the system energy consumption. Therefore, optimizing the access of the off-chip memory is crucial to further improve the energy efficiency of the deep neural network (DNN) accelerator. Towards this, this brief proposed an adaptive scheduling algorithm to minimize the DRAM access. Compared with the previous works, it can not only dynamically determine the data partition and the data type that will be reused, but also considered the constraints between adjacent layer, that is, if the output feature map of $i$ th layer is divided into N parts, the output feature map of $i+1$ th layer can only be divided into N parts or write back to the off-chip memory. Therefore, a minimize and realizable memory access solution can be obtained. Choosing 3 popular networks UNet, VGG-16 and MobileNet as benchmarks, the experiment results show that our scheduling algorithm can achieve a $34.46\% \sim 93.42\%$ reduction in energy consumption of DRAM access and a $34.34\% \sim 93.37\%$ reduction in DRAM access latency when compared to previous works.

A fast-locking digital delay-locked loop (DLL) is presented in this paper. By adopting a novel high resolution Time-to-Digital Converter (TDC), the time for generating fine-tuned codes is reduced to two clock cycles. Thus the total locking time is greatly reduced to 8 input reference clock cycles and remarkably shortened by 80% to 94.6% compared to previous closed-loop architectures. The proposed DLL has been fabricated in a 0.13μm CMOS technology and operates from 80MHz to 450MHz. The measured RMS and peak-to-peak jitters are 2.3ps and 10ps respectively.

The network on chip (NoC) used in AI chip is required to be able to deliver data to multiple destination endpoints simultaneously. The existing solutions such as hierarchial mesh (HMESH) and FATTREE all modify the connection between routers to achieve multicasting. Although this method can achieve delivering data to multiple destination endpoints at the same time, it significantly increases the number of routers used, and therefore brings huge additional area and power consumption overhead. In order to solve this problem, this work designed a customized multicast-enabled tree-based NoC, named mTREE, by modifying the microarchitecture of the router, including the routing calculation unit, credit counter, and switch allocator. When implemented on Arria10 FPGA, mTREE only brings a 1% increase in resource utilization compared with the traditional tree-topology NoC. Compared with two other multicast-enabled NoCs, FATTREE and HMESH, mTREE can reach a 3.19×∼3.50× and 1.67×∼3.02× reduction in resource utilization respectively, and a 4.81×∼8.01× and 2.85×∼3.03× reduction in energy consumption respectively.

In recent years, long short-term memory (LSTM) has been used in many speech recognition tasks, due to its excellent performance. Due to a large amount of calculation and complex data dependencies of LSTM, it is often not so efficient to deploy on the field-programmable gate array (FPGA) platform. This paper proposes an LSTM accelerator, driven by a specific instruction set. The accelerator consists of a matrix multiplication unit and a post-processing unit. The matrix multiplication unit uses staggered timing of read data to reduce register usage. The post-processing unit can complete various calculations with only a small amount of digital signal processing (DSP) slices, through resource sharing, and at the same time, the memory footprint is reduced, through the well-designed data flow design. The accelerator is batch-based and capable of computing data from multiple users simultaneously. Since the calculation process of LSTM is divided into a sequence of instructions, it is feasible to execute multi-layer LSTM networks as well as large-scale LSTM networks. Experimental results show that our accelerator can achieve a performance of 2036 GOPS at 16-bit data precision, while having higher hardware utilization compared to previous work.

Pruning and quantization are two commonly used approaches to accelerate the LSTM (Long Short-Term Memory) model. However, the traditional linear quantization usually suffers from the problem of gradient vanishing, and the existing pruning methods all have the problem of producing undesired irregular sparsity or large indexing overhead. To alleviate the problem of vanishing gradient, this work proposed a normalized linear quantization approach, which first normalize operands regionally and then quantize them in a local mix-max range. To overcome the problem of irregular sparsity and large indexing overhead, this work adopts the permuted block diagonal mask matrices to generate the sparse model. Due to the sparse model being highly regular, the position of non-zero weights can be obtained by a simple calculation, thus avoiding the large indexing overhead. Based on the sparse LSTM model generated from the permuted block diagonal mask matrices, this paper also proposed a high energy-efficiency accelerator, PermLSTM that comprehensively exploits the sparsity of weights, activations, and products regarding the matrix–vector multiplications, resulting in a 55.1% reduction in power consumption. The accelerator has been realized on Arria-10 FPGAs running at 150 MHz and achieved $2.19\times$∼$24.4\times$ energy efficiency compared with the other FPGA-based LSTM accelerators previously reported.