Recurrent neural networks (RNNs) are widely recognized for their proficiency in modeling temporal dependencies, making them highly prevalent in sequential data processing applications. Nevertheless, vanilla RNNs are confronted with the well-known issue of gradient vanishing and exploding, posing a significant challenge for learning and establishing long-range dependencies. Additionally, gated RNNs tend to be over-parameterized, resulting in poor computational efficiency and network generalization. To address these challenges, this article proposes a novel delayed memory unit (DMU). The DMU incorporates a delay line structure along with delay gates into vanilla RNN, thereby enhancing temporal interaction and facilitating temporal credit assignment. Specifically, the DMU is designed to directly distribute the input information to the optimal time instant in the future, rather than aggregating and redistributing it over time through intricate network dynamics. Our proposed DMU demonstrates superior temporal modeling capabilities across a broad range of sequential modeling tasks, utilizing considerably fewer parameters than other state-of-the-art gated RNN models in applications such as speech recognition, radar gesture recognition, ECG waveform segmentation, and permuted sequential (PS) image classification.

The surrogate gradient descent (SGD) algorithm enabled spiking neural networks (SNNs) to be trained to carry out challenging sensory processing tasks, an important step in understanding how spikes contribute to neural computations. However, it is unclear the extent to which these algorithms fully explore the space of possible spiking solutions to problems. We investigated whether spiking networks trained with SGD can learn to make use of information that is only encoded in the timing and not the rate of spikes. We constructed synthetic datasets with a range of types of spike timing information (interspike intervals, spatio-temporal spike patterns or polychrony, and coincidence codes). We find that SGD training can extract all of these types of information. In more realistic speech-based datasets, both timing and rate information is present. We therefore constructed variants of these datasets in which all rate information is removed, and find that SGD can still perform well. We tested all networks both with and without trainable axonal delays. We find that delays can give a significant increase in performance, particularly for more challenging tasks. To determine what types of spike timing information are being used by the networks trained on the speech-based tasks, we test these networks on time-reversed spikes which perturb spatio-temporal spike patterns but leave interspike intervals and coincidence information unchanged. We find that when axonal delays are not used, networks perform well under time reversal, whereas networks trained with delays perform poorly. This suggests that SNNs with delays are better able to exploit temporal structure. To facilitate further studies of temporal coding, we have released our modified speech-based datasets.

Spiking neural networks excel at event-driven sensing. Yet, maintaining task-relevant context over long timescales both algorithmically and in hardware, while respecting both tight energy and memory budgets, remains a core challenge in the field. Here we address this challenge through an algorithm–hardware co-design effort. At the algorithm level, inspired by the cortical fast–slow organization in the brain, we introduce a neural network with an explicit slow memory pathway that, combined with fast spiking activity, enables a dual memory pathway architecture in which each layer maintains a compact low-dimensional state that summarizes recent activity and modulates spiking dynamics. This explicit memory stabilizes learning while preserving event-driven sparsity, achieving competitive accuracy on long-sequence benchmarks with 40–60% fewer parameters than equivalent state-of-the-art spiking neural networks. At the hardware level, we introduce a near-memory-compute architecture that fully leverages the advantages of the dual memory pathway architecture by retaining its compact shared state while optimizing data flow, across heterogeneous sparse-spike and dense-memory pathways. We show experimental results that demonstrate more than a fourfold increase in throughput and over a fivefold improvement in energy efficiency compared with state-of-the-art implementations. Together, these contributions demonstrate that biological principles can guide functional abstractions that are both algorithmically effective and hardware-efficient, establishing a scalable co-design framework for real-time neuromorphic computation and learning.