The softmax function has been widely used in deep neural networks (DNNs), and studies on efficient hardware accelerators for DNN have also attracted tremendous attention. However, it is very challenging to design efficient hardware architectures for softmax because of the expensive exponentiation and division calculations in it. In this paper, the softmax function is firstly simplified by exploring algorithmic strength reductions. Afterwards, a hardware-friendly and precision-adjustable calculation method for softmax is proposed, which can meet different precision requirements in various deep learning (DL) tasks. Based on the above innovations, an efficient architecture for softmax is presented. By tuning the parameter P, the system accuracy and complexity can be adjusted dynamically to achieve a good tradeoff between them. The proposed design is coded using hardware description language (HDL) and evaluated on two platforms, Xilinx Zynq-7000 ZC706 development board and TSMC 28-nm CMOS technology, respectively. Hardware implementation results show that our architecture significantly outperforms other works in speed and area, and that by adjusting P, the accuracy can be further increased with little hardware overhead.

To enable efficient deployment of convolutional neural networks (CNNs) on embedded platforms for different computer vision applications, several convolution variants have been introduced, such as depthwise convolution (DWCV), transposed convolution (TPCV), and dilated convolution (DLCV). To address the utilization degradation issue occurred in a general convolution engine for these emerging operators, a highly flexible and reconfigurable hardware accelerator is proposed to efficiently support various CNN-based vision tasks. Firstly, to avoid workload imbalance of TPCV, a zero transfer and skipping (ZTS) method is proposed to reorganize the computation process. To eliminate the redundant zero calculations of TPCV and DLCV, a sparsity-alike processing (SAP) method is proposed based on weight-oriented dataflow. Secondly, the DWCV or pooling layers are configured to be directly executed after standard convolutions without external memory accesses. Furthermore, a programmable execution schedule is introduced to gain better flexibility. Finally, the proposed accelerator is evaluated on Intel Arria 10 SoC FPGA. Experimental results show state-of-the-art performance on both large-scale and lightweight CNNs for image segmentation or classification. Specifically, the accelerator can achieve a processing speed up to 339.9 FPS and computational efficiency up to 0.58 GOPS/DSP, which is $3.3\times $ better than the prior art evaluated on the same network.

Long Short-Term Memory (LSTM) and its variants have been widely adopted in many sequential learning tasks, such as speech recognition and machine translation. Significant accuracy improvements can be achieved using complex LSTM model with a large memory requirement and high computational complexity, which is time consuming and energy demanding. The low-latency and energy-efficiency requirements of realworld applications make model compression and hardware acceleration for LSTM an urgent need. In this work, several hardware-efficient network compression schemes are introduced first, including structured top-k pruning, clipped gating, and multiplication-free quantization, to reduce the model size and the number of matrix operations by 32× and 21.6×, respectively, with negligible accuracy loss. Then, efficient hardware architectures for accelerating the compressed LSTM are proposed, which support inference of multi-layer and multiple time steps. The computation process is judiciously reorganized and the memory access pattern is well optimized, which alleviate the limited memory bandwidth bottleneck and enable higher throughput. Moreover, parallel processing strategy is carefully designed to make full use of the sparsity introduced by pruning and clipped gating with high hardware utilization efficiency. Implemented on Intel Arria10 SX660 FPGA running at 200MHz, the proposed design is able to achieve 1.4–2.2× energy efficiency and requires significantly less hardware resources compared with the state-ofthe- art LSTM implementations.

Deep neural networks (DNNs) with various nonlinear activation functions (NAFs) have achieved unprecedented successes, sparking interest in efficient DNN hardware implementation. However, most existing NAF implementations focus on one type of functions with dedicated architectures, which is not suited for supporting versatile DNN accelerators. In this brief, based on a proposed reciprocal approximation optimization (RAO) method, an efficient reconfigurable nonlinear activation function module (ReAFM) is devised to implement various NAFs. The computational logic and dataflow of certain NAFs are merged and reused to minimize hardware consumption by leveraging the correlations among different NAFs. In addition, a precision adjustable exponential unit (PAEU) is developed to obtain a good tradeoff between the approximation accuracy and hardware cost. Compared to the prior art, the experimental results demonstrate that the proposed ReAFM can support many more NAF types with comparable or even better performance. Furthermore, evaluation results on some prevalent neural networks show that the proposed approximation method causes negligible accuracy loss (< 0.1%).

Transformers have achieved significant success in deep learning, and training Transformers efficiently on resource-constrained platforms has been attracting continuous attention for domain adaptions and privacy concerns. However, deploying Transformers training on these platforms is still challenging due to its dynamic workloads, intensive computations, and massive memory accesses. To address these issues, we propose an Efficient Training Accelerator for TRansformers (TRETA) through a hardware-algorithm co-optimization strategy. First, a hardware-friendly mixed-precision training algorithm is presented based on a compact and efficient data format, which significantly reduces the computation and memory requirements. Second, a flexible and scalable architecture is proposed to achieve high utilization of computing resources when processing arbitrary irregular general matrix multiplication (GEMM) operations during training. These irregular GEMMs lead to severe under-utilization when simply mapped on traditional systolic architectures. Third, we develop training-oriented architectures for the crucial Softmax and layer normalization functions in Transformers, respectively. These area-efficient modules have unified and flexible microarchitectures to meet various computation requirements of different training phases. Finally, TRETA is implemented under Taiwan Semiconductor Manufacturing Company (TSMC) 28-nm technology and evaluated on multiple benchmarks. The experimental results show that our training framework achieves the same accuracy as the full precision baseline. Moreover, TRETA can achieve 14.71 tera operations per second (TOPS) and 3.31 TOPS/W in terms of throughput and energy efficiency, respectively. Compared with prior arts, the proposed design shows 1.4– $24.5\times $ speedup and 1.5– $25.4\times $ energy efficiency improvement.

Improving the performance of high-level computer vision tasks in adverse weather (e.g., haze) is highly critical for autonomous driving safety. However, collecting and annotating training sets for various high-level tasks in haze weather are expensive and time-consuming. To address this issue, we propose a novel haze generation model called HazeGEN by coupling the variational autoencoder and the generative adversarial network to automatically generate annotated datasets. The proposed HazeGEN leverages a shared latent space assumption based on an optimized encoder–decoder architecture, which guarantees high fidelity in the cross-domain image translations. To ensure that the generated image can truly facilitate high-level vision task performance, a semisupervised learning strategy is developed for HazeGEN to efficiently learn the useful knowledge from both the real-world images (with unsupervised losses) and the synthetic images generated following the atmosphere scattering model (with supervised losses). Extensive experiments and ablation studies demonstrate that training the model with our generated haze dataset greatly improves accuracy in high-level tasks such as semantic segmentation and object detection. Furthermore, one important but under-exploited issue is investigated to find out whether the developed dataset can be a good substitute for the real ones. Results show that the generated dataset has the most similar performance to the real-world collected haze dataset on multiple challenging industrial scenarios compared with prior works.