Reinforcement Learning (RL) has opened up new opportunities to enhance existing smart systems that generally include a complex decision-making process. However, modern RL algorithms, e.g., Deep Q-Networks (DQN), are based on deep neural networks, resulting in high computational costs. In this paper, we propose QHD, an off-policy value-based Hyperdimensional Reinforcement Learning, that mimics brain properties toward robust and real-time learning. QHD relies on a lightweight brain-inspired model to learn an optimal policy in an unknown environment. On both desktop and power-limited embedded platforms, QHD achieves significantly better overall efficiency than DQN while providing higher or comparable rewards. QHD is also suitable for highly-efficient reinforcement learning with great potential for online and real-time learning. Our solution supports a small experience replay batch size that provides 12.3 times speedup compared to DQN while ensuring minimal quality loss. Our evaluation shows QHD capability for real-time learning, providing 34.6 times speedup and significantly better quality of learning than DQN.
Applications in the Internet of Things (IoT) utilize machine learning to analyze sensor-generated data. However, a major challenge lies in the lack of targeted intelligence in current sensing systems, leading to vast data generation and increased computational and communication costs. To address this challenge, we propose a novel sensing framework to equip sensing systems with intelligent data transmission capabilities by integrating a highly efficient machine learning model placed near the sensor. This model provides prompt feedback for the sensing system to transmit only valuable data while discarding irrelevant information by regulating the frequency of data transmission. The near-sensor model is quantized and optimized for real-time sensor control. To enhance the framework’s performance, the training process is customized and a “lazy” sensor deactivation strategy utilizing temporal information is introduced. The suggested framework is orthogonal to other IoT frameworks and can be considered as a plug-in for selective data transmission. The framework is implemented, encompassing both software and hardware components. The experiments demonstrate that the framework utilizing the suggested module achieves over 85% system efficiency in terms of energy consumption and storage, with negligible impact on performance. This framework has the potential to significantly reduce data output from sensors, benefiting a wide range of IoT applications.
Integrating deep learning with Reinforcement Learning (RL) results in algorithms that achieve human-like learning in complex yet unknown environments via a process of trial and error. Despite the advancements, the computational costs associated with deep learning become a major drawback. This paper proposes a revamped Q-learning algorithm powered by Hyperdimensional Computing (HDC), targeting more efficient and adaptive exploration. We introduce a solution leveraging model uncertainty to navigate agent exploration. Our evaluation shows that the proposed algorithm is a significant enhancement in learning quality and efficiency compared to previous HDC-based algorithms, achieving more than 330 more rewards with small overheads in computation. In addition, it maintains an edge over DNN-based alternatives by ensuring reduced runtime costs and improved policy learning, achieving up to 6.9 × faster learning.
15 July 2024 in IEEE Transactions on Biomedical Engineering
Objective: Develop a novel and highly efficient framework that decodes Inferior Colliculus (IC) neural activities for phoneme recognition. Methods: We propose using Hyperdimensional Computing (HDC) to support an efficient phoneme recognition algorithm, in contrast to widely applied Deep Neural Networks (DNN). The high-dimensional representation and operations in HDC are rooted in human brain functionalities and naturally parallelizable, showing the potential for efficient neural activity analysis. Our proposed method includes a spatial and temporal-aware HDC encoder that effectively captures global and local patterns. As part of our framework, we deploy the lightweight HDC-based algorithm on a highly customizable and flexible hardware platform, i.e., Field Programmable Gate Arrays (FPGA), for optimal algorithm speedup. To evaluate our method, we record IC neural activities on gerbils while playing the sound of different phonemes. Results: We compare our proposed method with multiple baseline machine learning algorithms in recognition quality and learning efficiency, across different hardware platforms. The results show that our method generally achieves better classification quality than the best-performing baseline. Compared to the Deep Residual Neural Network (i.e., ResNet), our method shows a speedup up to 74×, 67×, 210× on CPU, GPU, and FPGA respectively. We achieve up to 15% (10%) higher accuracy in consonant (vowel) classification than ResNet. Conclusion: By leveraging brain-inspired HDC for IC neural activity encoding and phoneme classification, we achieve orders of magnitude runtime speedup while improving accuracy in various challenging task settings. Significance: Decoding IC neural activities is an important step to enhance understanding about human auditory system. However, these responses from the central auditory system are noisy and contain high variance, demanding large-scale datasets and iterative model fine-tuning. The proposed HDC-based framework is more scalable and viable for future real-world deployment thanks to its fast training and overall better quality.
In wireless networks, dynamic spectrum access is the key to improving spectrum utilization and increasing channel capacity. Since the channels in wireless networks are highly correlated, they require intelligent algorithms to dynamically handle multi-channel access. Reinforcement Learning (RL) algorithms are introduced as effective techniques to optimize network performance. However, current RL methods heavily rely on a computationally intensive deep neural network (DNN) that is not friendly for edge devices. In this paper, we propose HD-RL, a dynamic wireless channel-sharing solution that utilizes brain-inspired lightweight hyperdimensional computing as the learning engine. HD-RL mimics important brain functionalities towards high-efficiency and noise-tolerant computation. HD-RL naturally encodes and memorizes prior knowledge to provide the near-optimal policy for channel throughput and α-fairness in the wireless network. Our evaluation shows that HD-RL achieves maximum throughput and fairness while significantly improving efficiency compared to the state-of-the-art DNN-based RL algorithms. In particular, HD-RL achieves more than 20× speedup for reaching the fairness objective. On average, the speedup of convergence time is more than 10× compared to the baseline. Our results also indicate that HD-RL has substantially higher robustness against possible hardware failure, e.g., up to 40% dimension loss in the model.