28 February 2020 in Frontiers in Neuroscience
GSGopalakrishnan Srinivasan
Gopalakrishnan Srinivasan
Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN, and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.
30 December 2019 in IEEE Access
Deep convolutional neural network (DCNN) based supervised learning is a widely practiced approach for large-scale image classification. However, retraining these large networks to accommodate new, previously unseen data demands high computational time and energy requirements. Also, previously seen training samples may not be available at the time of retraining. We propose an efficient training methodology and incrementally growing DCNN to learn new tasks while sharing part of the base network. Our proposed methodology is inspired by transfer learning techniques, although it does not forget previously learned tasks. An updated network, for learning new set of classes, is formed using previously learned convolutional layers (shared from initial part of base network) with addition of few newly added convolutional kernels included in the later layers of the network. We employed a `clone-and-branch' technique with calibration, which allows the network to learn new tasks (containing classes with similar features as old tasks) one after another without any performance loss in old tasks. We evaluated the proposed scheme on several recognition applications. The classification accuracy achieved by our approach is comparable to the regular incremental learning approach (where networks are updated with new training samples only, without any network sharing), while achieving energy efficiency, reduction in storage requirements, memory access and training time.
14 May 2018 in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
Deep Neural Networks (DNNs) have emerged as the state-of-the-art technique in a wide range of machine learning tasks for analytics and computer vision in the next generation of embedded (mobile, IoT, wearable) devices. Despite their success, they suffer from high energy requirements. In recent years, the inherent error resiliency of DNNs has been exploited by introducing approximations at either the algorithmic or the hardware levels (individually) to obtain energy savings while incurring tolerable accuracy degradation. However, there is a need for investigating the overall energy-accuracy trade-offs arising from the introduction of approximations at different levels in complex DNNs. We perform a comprehensive analysis to determine the effectiveness of cross-layer approximations for the energy-efficient realization of large-scale DNNs. The approximations considered are as follows: (i) use of lower complexity networks (containing lesser number of layers and/or neurons per layer), (ii) pruning of synaptic weights, (iii) approximate multiplication operation in the neuronal MAC (Multiply-and-Accumulate) computation, and (iv) approximate write/read operations to/from the synaptic memory. Our experiments on recognition benchmarks (MNIST, CIFAR10) show that cross-layer approximation provides substantial improvements in energy efficiency for different accuracy/quality requirements. Furthermore, we propose a synergistic framework for combining the approximation techniques to achieve maximal energy benefits from approximate DNNs.
23 May 2024 in IEEE Journal of Solid-State Circuits
Extended reality (XR) applications are machine learning (ML)-intensive, featuring deep neural networks (DNNs) with millions of weights, tightly latency-bound (10–20 ms end-to-end), and power-constrained (low tens of mW average power). While ML performance and efficiency can be achieved by introducing neural engines within low-power systems-on-chip (SoCs), system-level power for nontrivial DNNs depends strongly on the energy of non-volatile memory (NVM) access for network weights. This work introduces Siracusa , a near-sensor heterogeneous SoC for next-generation XR devices manufactured in 16 nm CMOS. Siracusa couples an octa-core cluster of RISC-V digital signal processing (DSP) cores with a novel tightly coupled “At-Memory” integration between a state-of-the-art digital neural engine called and an on-chip NVM based on magnetoresistive random access memory (MRAM), achieving 1.7 $\times$ higher throughput and 3 $\times$ better energy efficiency than XR SoCs using NVM as background memory. The fabricated SoC prototype achieves an area efficiency of 65.2 GOp/s/mm ${}^\text{2}$ and a peak energy efficiency of 8.84 TOp/J for DNN inference while supporting complex, heterogeneous application workloads, which combine ML with conventional signal processing and control.
1 February 2018 in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
Intermittently powered systems represent a new class of batteryless devices that operate solely on energy harvested from their environment. Due to the unreliable nature of ambient energy sources, these devices experience frequent intervals of power loss, leading to sudden reboots. Tolerating such power supply disruptions require the ability to rapidly checkpoint/save system state when power loss is imminent and restore it at the start of the next power cycle to continue computations in a seamless manner. A typical microcontroller used in these systems consists of a fast nonvolatile SRAM and a nonvolatile Flash storage. Prior work has shown how emerging nonvolatile memory technologies such as STT-MRAM can improve the energy efficiency of these systems, either by using STT-MRAM as a drop-in replacement for Flash (henceforth referred to as the SRAM+STT-MRAM memory configuration) or using STT-MRAM as unified memory (henceforth referred to as the unified STT-MRAM memory configuration). However, both these configurations have significant drawbacks. Using the SRAM+STT-MRAM configuration leads to high checkpointing overhead due to the inefficient write operations of STT-MRAM whereas using the unified STT-MRAM configuration is inefficient due to executing every program instruction directly from STT-MRAM. This paper proposes a novel Spin Hall Effect-based nonvolatile-SRAM (SNVRAM) bit-cell that combines the nonvolatility of spin devices with the speed and energy efficiency of conventional 6T SRAM cells. We explore the use of the proposed SNVRAM to replace the SRAM in a transiently powered system to mitigate the drawbacks of the aforementioned memory configurations. Simulation results using a set of evaluation benchmarks demonstrate that the SNVRAM+STT-MRAM configuration leads to significant memory energy benefits of $2.6\times $ and $2.8\times $ on average, compared to the SRAM+STT-MRAM and unified STT-MRAM memory configurations, respectively.